What is consumer health informatics? A systematic review of published definitions.

Inform Health Soc Care, Early Online: 1–22, 2014 ! Informa UK Ltd. ISSN: 1753-8157 print / 1753-8165 online DOI: 10.3109/17538157.2014.907804

REVIEW

Inform Health Soc Care Downloaded from informahealthcare.com by University of Liverpool on 12/31/14 For personal use only.

What is consumer health informatics? A systematic review of published definitions David Flaherty, Laurie Hoffman-Goetz, and Jose F. Arocha School of Public Health and Health Systems, University of Waterloo, Waterloo, Ontario, Canada Background: Consumer health informatics (CHI) is an emerging field that utilizes technology to provide health information to enhance health-care decision making by the public. There is, however, no widely accepted or uniform definition of CHI. A consensus definition would be important for pedagogical reasons, to build capacity and to reduce confusion about what the discipline consists of. Aim: We undertook a systematic review of published definitions of CHI and evaluated them using five quality assessment criteria and measures of similarity. Methods: Five databases were searched (Embase, Web of Science, MEDLINE, CINAHL and Business Source Complete) resulting in 1101 citations. Twenty-three studies met the inclusion criteria. Definitions were appraised using five criteria (with each scoring out of one): use of published citation, multi-disciplinarity, journal impact, definition comprehensibility, text readability. Results: Most definitions scored low on citation (Mean ± SD: 0.22 ± 0.42), multidisciplinarity (0.15 ± 0.28) and readability (0.04 ± 0.21) and somewhat higher on IF (0.35 ± 0.45) and definition comprehensibility (idea density) (0.87 ± 0.34) criteria. Overall, the quality of the published definitions was low 1.63 ± 0.80 (out of five). Conclusions: The definitions of CHI were variable in terms of the quality assessment criteria. This suggests the need for continued discussion amongst consumer health informaticians to develop a clear consensus definition about CHI. Keywords Consumer, definitions, health informatics, medical informatics, systematic review

INTRODUCTION Since its inception nearly two decades ago, an increasing number of professionals have focused on creating innovative health applications for the consumer. These professionals have popularized the term ‘‘consumer health informatics’’, a phrase introduced by Ferguson (1). Consumer health informatics (CHI), an important subfield of health informatics, addresses the needs of health consumers and medical and public health practitioners in order to bolster healthcare. However, since CHI is extremely broad in reach and applications can be targeted towards a variety of health issues

Correspondence: Dr. Laurie Hoffman-Goetz, Professor, School of Public Health and Health Systems, University of Waterloo, Waterloo, 200 University Avenue W, Ontario N2L 3G1, Canada. E-mail: [email protected]


2

L. Hoffman-Goetz et al.

(i.e. from smoking cessation to cancer information to suicide prevention), key challenges, such as effectively reaching special and underserved populations (2), assessing the accuracy of health information (3), improving end-user uptake and ease-of-use (4), are difficult to resolve. Still, there are examples of successful applications of CHI that have demonstrated its benefits for patients and lay people (5). One such example is the Comprehensive Health Enhancement Support System (CHESS), which is a computer-based system of integrated services structured to help individuals deal with health crises or medical concerns (6). In several studies focusing on the efficacy of the CHESS software for women with breast cancer, Gustafson and colleagues (7,8) have shown it significantly improved their information competence, social support, participation in health and confidence in their doctor compared with women who only used the Internet (7,8). The CHESS program is just one example of a successful and important CHI application. Consumer health informatics has the potential to improve outcomes across a spectrum of diseases and health problems via an application (or electronic tool, system or technology) that provides tailored information for the individual (9). According to the American Medical Informatics Association (10), CHI provides ‘‘information structures and processes that empower consumers to manage their own health: for example, health information literacy, consumer-friendly language, personal health records and Internet-based strategies and resources’’ [paragraph1]. Given the rapid growth of CHI by the health conscious public and the diverse nature of the field, it is critical that researchers agree on what CHI encompasses relative to other informatics sub-specialties. Indeed, there appears to be a lack of agreement in the definition of CHI (11–13). This lack of a standard definition of CHI raises a number of potential issues. Without a clear definition it will be difficult to develop core competencies for the training of consumer health informatics professionals. CHI professionals can help the public (consumer) with accessing health information via Internetbased technologies (e.g. health websites, applications). This is vitally important at the present time, as the Internet has brought about a huge array of health information that is frequently, but not always accurately, accessed by the population (14). Also, several fields related to CHI, such as medical informatics and public health informatics, have clearly articulated core competencies. For example, COACH: Canada’s Health Informatics Association recently developed guidelines that outline the core competencies needed to perform as a health informatics professional (as well as a biomedical informatics professional). The COACH expert panel identified three general areas and seven advanced corresponding sub-areas that are needed: Information Sciences (information management, information technology); Health Sciences (clinical and health services, Canadian health system); and Management Sciences (organizational and behavioral management, project management and analysis and evaluation) (15). Similarly, the Centers for Disease Control and Prevention and the U.S. Department of Health and Human Services developed a list of 14 core competencies for public health informaticians and senior public health informaticians to help establish training and the advancement of


Consumer health informatics definitions

the field (16). In both cases, professionals from their respective fields agreed on the core competencies for both health informatics and public health informatics to build the foundation for education and training in these fields. However, it is difficult to arrive at agreement about core competencies and specific skills needed for consumer health informaticians if there is no consensus about what constitutes CHI. Bernstam et al. (17) provides an important rationale for why it is necessary to develop definitions in the informatics field using the example of bioinformatics. Arising from formally defining bio-informatics, these researchers suggested that there were a number of practical implications including (1) educational program design, which allows for a clear vision of the field to students; guide curriculum development and evaluation within training programs; (2) administrative decisions, which allows administrators to make the case for resources to guide informatics units (academic and serviceoriented) with respect to hiring faculty or staff, relationship to other organizational units and performance metrics; and (3) communication, which allows internal communication among informaticians and external communication with those outside of the field (page 11) (17). In other words, having a consensus definition about a field can help match current and potential collaborators, guide informatics societies such as the American and International Medical Informatics Associations (AMIA and IMIA, respectively) and help funding agencies and members of the general public understand their roles and contributions. To date, there has been no unified definition of CHI put forward by an expert organization, despite reference to CHI in the literature. This study aimed first to systematically review published definitions of consumer health informatics in peer-reviewed publications and assess them for definitional quality. Second, using latent semantic analysis (LSA), we also explored how similar the definitions are to one another. This latter objective serves the purpose of providing further evidence of the key concepts or themes included in the definitions. We asked five questions to guide the systematic review in terms of assessment of the CHI definitions: (1) What is the likely impact or reach of the definition in the academic and practice communities? (2) Does the CHI definition emphasize a multi-disciplinary, inter-disciplinary or uni-disciplinary approach to the field? (3) Does the CHI definition contain references to other published literature? (4) Are the underlying ideas or concepts in the CHI definition difficult to comprehend to most readers? And (5), Is the CHI definition text readability appropriate for the general population? The underlying motivation of this review is to stimulate scholarly discussion amongst researchers, practitioners and consumer end users about the CHI components necessary to enhance consumer and patient-oriented health care decision making.

METHODS Search strategy The systematic review methodology used in this study is informed by similar studies that aim to clarify the definitions of Health 2.0 (18), e-Health (19), competency based education (20) and biomedical informatics (17). A systematic

3


4


review was undertaken in January 2013 using the guidelines of Petticrew and Roberts (21) and the Cochrane Collaboration (22) to obtain peer-reviewed articles with definitions of consumer health informatics. Systematic reviews involve mapping out areas of uncertainty, identifying where little or no relevant research has been done and indicating where new studies are needed. For this study, we chose to include peer-reviewed articles only, as these are more widely accessed by health informatics researchers and professionals and are most likely to define a field of consumer health informatics. For this reason, we did not include definitions of CHI in the grey literature (e.g. Mednar) although further research may include these and other deep web resources. The databases searched and the number of articles screened for each database included Embase (n ¼ 467); Web of Science and PubMed, via Web of Knowledge (n ¼ 447); CINAHL (n ¼ 175); and Business Source Complete, via EBSCO (n ¼ 12). Search terms were generated from the topic of the systematic review, keywords from articles of relevance, MeSH terms and consultation with the university health librarian. Reference lists from the identified articles were hand searched for relevant studies that were not captured from the databases. ‘‘Consumer Health Informatics’’ is not listed as an MeSH term. Medical Subject Headings, or MeSH terms, consist of a thesaurus of medical terms developed by the U.S. National Library of Medicine. It is a set of synonyms and related descriptors organized in a hierarchical structure that allows one to search at various levels for an identified concept or topic. The search conducted for this study captured both the fields of consumer health and related areas of informatics and thus two search strings were combined. The first string used the following terms: consumer, consumer health information and patient participation. The second string (combined with Boolean operators) included the terms informatics and medical informatics. Inclusion/exclusion criteria Eligible journal articles were included if they were written in English (either in print or in online journals) and contained text including a definition of consumer health informatics in explicit terms. Articles were identified by conducting title, abstract and keyword searches. The abstracts were then read for relevance to consumer health informatics. If the article was deemed relevant, it was retrieved and read in full in search for a definition of CHI. All article types were included in the review (i.e. empirical studies, editorials, reviews, clinical investigations). Furthermore, the lists of references within the identified articles were hand-searched for additional relevant citations. An article was excluded if it did not contain a previously published definition (i.e. a fully quoted definition from another author) or did not provide a new, not previously cited, definition. Data collection Data extraction included author, title, source, date of publication, article type (e.g. review, meta-analysis, clinical report) and the exact definition used in the article. These variables were transcribed into a matrix modified from Van De Belt et al. (18).



Critical appraisal of CHI definitions Each definition was evaluated against a set of criteria, two of which were adapted from published work about the components necessary for good definitions (23). The other three criteria were based on objective measures or metrics (described below) that reflect features of a good CHI definition. A total of five assessment criteria were used, with all criteria weighted as a maximum of one. A score of zero was assigned for ‘‘no’’ or ‘‘not applicable’’ or ‘‘condition not met’’, and a score of one was assigned for ‘‘yes’’ or ‘‘condition fully met’’. A score of 0.5 was given where partial fulfillment of the criterion occurred. A maximum score of five was possible for each definition summing across all criteria. The criteria and rationale for the criteria are described below. Criterion 1 (Citation): Does the article provide a cited source for the CHI definition? A cited definition suggests the author has reviewed the literature, and is making an informed decision on what CHI encompasses. Furthermore, science is built on the foundation and work of others. By citing previously published work, the author shows where his understanding of a given topic comes from (24) and fundamentally, it is about building knowledge about CHI. Hence, each piece of cited research adds to the collective body of work about a newer, emerging field. CHI definitions that had a published or referenced source were scored as one. Definitions that did not provide a reference to a published source were scored zero. Criterion 2 (Multidisciplinary): Does the definition indicate that CHI is a multidisciplinary field? Consumer Health Informatics lies at the inter-section of several fields: medical informatics, nursing informatics, social care, public health, health promotion, health education, marketing and communication science (25). Furthermore, in a consensus discussion amongst members of the American Medical Informatics Association (AMIA), they concluded that a working definition of CHI should stress the multi-disciplinary nature of the field (12). Therefore, we considered it important that the definition provided in a published article either identified this significant characteristic or provided an explanation of other disciplines contributing to CHI. A score of one was given if the definition mentioned the multi-disciplinary nature of CHI and explicitly named other fields relevant for CHI. A score of 0.5 was given if the definition stated that CHI is multidisciplinary, but failed to identify any disciplines or knowledge areas that contribute to the field or, alternatively, mentioned the contributing domains but failed to mention multi-disciplinary nature of CHI. A score of zero was given if the definition did not mention the multi-disciplinary nature of CHI and also did not identify other contributing fields. Criterion 3 (Impact): ‘‘Is the definition of CHI found in a journal with high scientific impact?’’ Impact factor (IF), although imperfect, is often used as a proxy measure of the scholarly prestige of a journal (26). It is a single statistic that captures the average citation performance of articles in a specific journal over a period of one, two or three years. Generally speaking, the higher the IF, the more

5

6



attention is given to articles in the journal. Although there has been criticism about the overall value and calculation of IF (27), journals with high impact factors tend to have greater reach and influence in the scientific and public media (28). Discourse in the public media would be particularly relevant given the focus of CHI is on the consumer. The two year Impact factor (IF) for each journal in which a definition of CHI was found were retrieved from the ISI Web of Knowledge Journal Citation Report. If the definition was published in a journal with an IF of 51, it was scored as zero; an IF between one and two was scored as 0.5; and an IF42 was assigned a score of one. Articles (definitions) were scored as zero if the journal was not included in the ISI database. Criteria 4 and 5 Critical appraisal of what constitutes a good definition in the social sciences (into which consumer health informatics falls as a discipline) was described by Hart (23). He identified a number of criteria in sociology, two of which were modified for appraisal of consumer health informatics definitions. These criteria were comprehensibility and simplicity of the definitions. Comprehensibility was determined by text complexity and simplicity by text readability. These are described below. Criterion 4 (Comprehensibility): ‘‘Is the definition of CHI comprehensible for researchers and practitioners from other cognate fields and for the educated ‘lay’ public?’’ Comprehensibility is a key factor for researchers and professionals to adopt a proposed definition. If the definition is too complex, it will likely not be easily understood, remembered or adopted for use. As a measure of comprehensibility, propositional density (P-D) which is based on propositional analysis (a method for the investigation of discourse comprehension) was used to score the definitions for their underlying complexity. Both reading time and recall are key measures of readability and more propositions per unit of text require more reading time (29). Therefore, a higher P-D score suggests the more difficult a definition will be to read. The definition received a score of one if P-D was 50.50 (brief, clear, concise, with fewer propositions per unit of text), and a score of zero if it is40.50, indicating a semantically dense and prolix definition. Criterion 5 (Simplicity): ‘‘Are the terms used to describe the topic simple and without jargon?’’ A good definition should be clear and written in a way that is easy for the reader to understand (23). It should be free of jargon and use language that an educated layperson can understand. In other words, a good definition should be understood not just by the specialized expert but also by the public. There are a number of readability formulas to assess text simplicity. Readability formulas measure structural difficulty of text (i.e. vocabulary, word length) at the sentence level, but do not consider other factors related to reading difficulty (i.e. reader characteristics, material organization, text coherence) (30). One of the most widely used is the Simplified Measure of Gobbledygook (SMOG) (31) and has been used to assess text difficulty of printed health information (32). SMOG provides a reading grade level (RGL) metric. Definitions of CHI were scored as one if the RGL was 5 or ¼ to Grade 12



(high school) of formal education (i.e. simpler and easier to read) and a score of zero if the RGL was 4Grade 13 (post-secondary education; greater text difficulty). The underlying assumption was that good definitions are written in clear, plain language that is easy to understand by the non-technical, but educated reader (i.e. a librarian working with the public, but not necessarily a medical librarian specializing in informatics) as well as the public with an attained high school education. To further compare the published definitions, Latent Semantic Analysis (LSA) was applied to provide a score of similarity among definitions. LSA is a statistical method for determining the meaning of texts in context, (e.g. wordword, word-sentence, word-text passage) including the semantic similarity of words and passages through analysis of a body of text (33,34). Analysis All studies were scored by one of the authors using the criteria described above. To ensure reliability, a second author independently scored six out of twenty three articles (26.1%), which were chosen at random from the full sample. There was near complete agreement on the criteria scoring and any discrepancies were resolved by discussion between the coders. Descriptive statistics for the five criteria were conducted using IBM SPSS Statistics 21. For criteria 4 (propositional density) and 5 (text readability), the following analyses were also used. The analysis for propositional density (criterion 4 for comprehensibility) was done using the Computerized Propositional Idea Density Rater (CPIDR) V5.1 software (35). CPIDR works by measuring the idea density of text by using a part-of-speech tagger, then counting the appropriate parts of speech and applying corrective rules to adjust the count in certain situations (36). Each definition was broken up into ‘‘propositions’’ or ‘‘idea units’’, these units are then divided by the number of words in the sentence to deliver a propositional density score. The software has also shown to be in agreement with the consensus of a panel of trained P-D raters better than the raters agreed with each other (37), proving its reliability. For criterion 5 (simplicity/readability) the Reading Grade Level (RGL) using SMOG score was calculated; the calculation was based on the formula for text using fewer than 30 sentences since none of the definitions were longer than 30 sentences (31). Using software created by the Science and Applications of the Latent Semantic Analysis Group at University of Colorado at Boulder (http:// lsa.colorado.edu), the definitions were entered into a ‘‘Matrix Comparison’’ LSA web-based tool and a corresponding score, ranging from 0.00 (unrelated definitions) to 1.00 (related definitions), for each definition was given depending on the similarities between them.

RESULTS In total, 1101 records were identified through database searching (Embase, CINAHL, Web of Knowledge, MEDLINE and Business Source) and an additional eight records were identified through reference list scan of relevant articles (Figure 1). Nine-hundred and fourteen records were screened (after

7


8


Figure 1. Reference list scan of relevant articles.

duplicates removed) and 808 were excluded as not relevant and did not meet predetermined criteria. The remaining 106 full text articles were read and assessed for eligibility of which 83 were excluded for not containing a definition of consumer health informatics. In total, twenty-three articles that had definitions of consumer health informatics were included in the review (see Table 1 for definitions) (11,12,38–58). Definitions ranged in length between 12 and 72 words (M ± SD: 33.9 ± 15.9) and the majority were one sentence long, with the exception of two definitions (38,56), which were two sentences long (1.1 ± 0.3). The year of publication spanned from 1995 to 2012.

Jimison and Sher (40)

Motley et al. (41)

Bader and Barude (42)

Anonymous (43)

Bouhaddou et al. Brennan (1999) (44) (45)

Rhodes (46)

(Continued )

An exciting new Consumer health Consumer health Information CHI generally CHI can be Special-purpose Consumer informatfield of medical informatics repreinformatics is the supplied to encompasses defined as any computer tools ics is defined as computing, sents a diverse union of health patients using two areas: the information referred to as the use of comconsumer health field devoted to care content with advanced use of Internet that enables Consumer puter to support informatics is the developthe speed and information technology to individuals to Health consumers in devoted to the ment, implemenease of and communiconnect to online understand Informatics obtaining inforstudy and develtation and technology. cation services for health their health (CHI) represent mation, analyzing opment of a new research on teletechnologies. care information, and make the applicaunique care breed of comcommunication such as that health-related tion of comneeds and helpputer and teleand computer contained in decisions for puter and ing to make communication applications medical journals themselves or information decisions about systems designed designed to be and professional their family. technologies healthcare and for use by used by conpublications; and specifically to health promotion. laypersons. sumers to access systems and softsupport the information on a ware provided by health inforwide variety of clinicians to mation and health care patients to help in communicainformation. the diagnosis and tion needs of treatment of spepatients and cific conditions lay persons. and diseases.

Ferguson (39)

Table 1. Consumer health informatics definitions and citations included in the review.



9

Comm. on enhancing the Internet for health apps. (47)

Houston et al. (12)

Lewis and Pesut (48)

Gustafson et al. (49)

Eysenbach (50)

Perry and Weldon (51)

Bakker et al. (52)

Consumer health Consumer health . . . the set of activ- . . . a sub-specialty of ‘‘Consumer Consumer Health ’Consumer A branch of informatics (CHI) informatics is the ities aimed at medical informatHealth Informatics health informscholarship, is a rapidly evolbranch of medgiving consumer ics which studies Informatics’’ is Systems (CHIS) atics’ is the referred to as ving sub-discipical informatics a more profrom a patient/ an emergent include patientemerging consumer line of medical that analyses nounced role in consumer perproperty of the oriented interscience at the health informinformatics. consumers’ their own health spective the use seamless netactive computercross-roads of atics (CHI), has needs for inforand healthcare, of electronic works of data, based programs health informrecently develmation; studies ranging from the information and services, inforthat provideatics and oped with the and implements development of communication mation and information, decipublic health purpose of methods of tools for selfto improve medconnectivity sion, behavior which deals understanding making informaassessment of ical outcomes available change and with investigathow contion accessible to health risks and and the healththrough the emotional suping determinsumers use consumers; and management of care decisionInternet. port for health ants, condiadvanced models and intechronic disease making process. issues. tions, eleminformation grates conto home-based ents, models management sumers’ prefermonitoring of and processes and delivery ences into health status and to design, technologies medical informadelivery of care. implement such as the tion systems. and maximize Internet in Consumer informthe effectiveorder to atics stands at the ness of comgather and crossroads of puterized inforultimately act on information other disciplines, mation and about health– such as nursing tele-communifor themselves informatics, cation and or for those public health, network systhey care for. health promotion, tems for health education, consumers. library science and communication science, and is perhaps the most challenging and rapidly expanding field in medical informatics.

Eysenbach (38)

Table 1. Continued


10 L. Hoffman-Goetz et al.

Keselman et al. (54)

Gibbons et al. (55)

Alamantariotou (56)

Ho (57)

Shaikh (58)

Arocha and HoffmanGoetz (11)

Consumer health Consumer health Consumer health Consumer health Consumer health Consumer health An important informatics has information informatics is informatics is a informatics is no informatics component of emerged as a resources provide defined as any part of meddifferent to the refers to health health informstrategy to inform health informaelectronic tool, ical informatics branches of information atics, conand empower tion to lay users, technology or that has as first informatics in technology sumer health patients for selfhopefully to electronic applipriority to anahealthcare other that utilizes informatics, management of empower cation that is lyse the interthan it primarily data enabled has been their health. patients, caredesigned to action represents the by cyber infradefined as a givers, families interact directly between consumer interstructure, or in field that and consumers; with consumers, Information ests and is about other words ‘analyses improve deciwith or without Technology (IT) providing the the computer, consumers’ sions; and ultimthe presence of a and health consumer with mobile, and needs for ately foster better health care consumers. the right tools, Internet platinformation; public health professional that Consumer skills, support and forms necesstudies and outcomes. provides or uses health informknowledge to sary for implements individualized atics applicabetter manage coordinating methods of (personal) infortions are their health care. care delivery making information and designed to by health mation provides the interact systems and accessible to consumer with directly with clinical and consumers; individualized the customer public health and models assistance, to with or without professionals, and integrates help the patient the essential as well as for consumers’ better manage presence of consumers to preferences their health or healthcare. be empowinto medical health care. ered to information manage their systems’ own health. (Eysenbach 2000).

Khan et al. (53)



11

12

L. Hoffman-Goetz et al. Table 2. List of definitions and corresponding scores for five assessment criteria.


Definition source

Criterion1 Criterion2 Criterion3 Criterion4 Criterion5 Score (Citation) (MD) (IF) (PD) (RGL) out of 5

Ferguson (39) 0 0 0 1 0 1 Jimison and Sher (40) 0 0 1 1 0 2 Motley et al. (41) 0 0 0 1 1 2 Bader and Barude (42) 0 0 1 1 0 2 Anonymous (43) 0 0.5 0 1 0 1.5 Bouhaddou et al. (44) 1 0 0 1 0 2 Brennan (45) 0 0 0.5 1 0 1.5 Rhodes (46) 1 0 0 1 0 2 Eysenbach (38) 0 1 1 0 0 2 Comm. on enhancing 0 0.5 0 0 0 0.5 the Internet for health apps. (47) Houston et al. (12) 0 0 0 1 0 1 Lewis andPesut (48) 0 0 1 1 0 2 Gustafson et al. (49) 1 0 1 1 0 3 Eysenbach (50) 0 0.5 0 1 0 1.5 Perry and Weldon (51) 0 0 0 1 0 1 Bakker et al. (52) 0 0 0 1 0 1 Khan et al. (53) 0 0 0 1 0 1 Keselman et al. (54) 0 0 1 1 0 2 Gibbons et al. (55) 0 0 0 0 0 0 Alamantariotou (56) 0 0 0 1 0 1 Ho (57) 0 0 0 1 0 1 Shaikh (58) 1 0.5 1 1 0 3.5 Arocha and 1 0 0.5 1 0 2.5 Hoffman-Goetz (11) Mean, Standard M ¼ 0.22, M ¼ 0.13, M ¼ 0.35, M ¼ 0.87, M ¼ 0.04, M ¼ 1.63, Deviation, Total SD ¼ 0.42, SD ¼ 0.27, SD ¼ 0.46, SD ¼ 0.34, SD ¼ 0.21, SD ¼ 0.80 T ¼ 9/23 T ¼ 20/23 T ¼ 1/23 T ¼ 5/23 T ¼ 5/23 MD ¼ multi-disciplinary; IF ¼ impact factor; PD ¼ propositional density; RGL ¼ reading grade level using SMOG

Appraisal of CHI definitions A breakdown of how each definition scored in the evaluation, along with the mean and standard deviation, are included in Table 2. None of the published definitions fully met all criteria used in the evaluation. The mean score overall was one point seven out of five, and the most frequently met criterion was propositional density (criterion 4) with 20/23 definitions (87%) meeting the requirements for this criterion. The key results are highlighted below. Criterion 1 (citation) Eighteen of the 23 definitions (78.3%) were scored as zero on the first criterion while five (21.7%) of the 23 definitions achieved a score of one (11,44,46,49,58) meaning the definition cited published literature. None of the definitions cited the same paper. Criterion 2 (multi-disciplinary focus) Five definitions (21.7%) met the requirements for criterion 2 (multi-disciplinary). Of these, four (43,47,50,58) received a score of 0.5, and one (38) received a



full score. Of the four definitions which received a partial score of 0.5, these stated the fields encompassed within CHI but not that the area was specifically ‘‘multi-disciplinary’’. The definition provided by Eysenbach (38) was the only one to receive a full score. CHI was specifically identified as a multidisciplinary field and provided examples (e.g. nursing informatics, public health, health promotion, health education, library science and communication science). The remaining 17 definitions (73.9%) received a score of zero and failed to mention the multi-disciplinary focus of CHI and give examples of component disciplines. Criterion 3 (impact factor) Seven definitions (30.4%) achieved a score of one, being published in journals with high impact (38,40,42,48,49,54,58). Two definitions scored 0.5 (11,45) while the remaining 14 definitions received a score of zero being published in journals with impact factors of less than one or no impact factor (39,41,43,44,46–48,50–53,55–57). The definition published in a journal with the highest impact factor was that of Eysenbach appearing in the British Medical Journal in 2000. Criterion 4 (idea density) The overall mean of the idea density (ID) score was zero point four five out of one with a standard deviation of zero point zero five. It was the most frequently met criterion with twenty (87.0%) definitions (11,12,39–46,48–54, 56–58) meeting the requirements (at least partially) having an ID score of 0.50 or less. The highest scoring definition was Eysenbach (38) with an ID score of 0.55 (24 propositions/44 words). The lowest scoring was the Motley et al. definition (41) with an ID score of 0.35 (six propositions/seventeen words). Criterion 5 (readability) The overall mean RGL for the 23 definitions was very high at 20.6 (±3.5). It was the least met criterion with only one (4.3%) definition scoring one (a text reading grade level of high school or less) (41). The definition of Motley et al. (41) had the lowest readability score at 12.5 whereas the highest readability score was tied between Arocha and Hoffman-Goetz (11) and an anonymous author (43) at a RGL of 25.6. Definition with the highest/lowest overall score on assessment criteria The definition with the highest overall score was that of Shaikh and colleagues (58). This definition fulfilled four of the five criteria, with an assessment score of 3.5/5. This definition included a score of one for citation, 0.5 for multidisciplinary focus (mentioning three contributing fields but not stating that CHI is multi-disciplinary), one on impact factor (being published in a journal with a impact factor of 3.95), one on propositional density with a score of 0.500, and a zero on the text readability with a SMOG reading grade level of 24.9. One definition (55) received an overall score of zero out of five indicating poor quality on all assessment criteria. This definition scored zero on the citation criterion as it did not reference any published literature for the source of the definition; zero on the criterion of multi-disciplinary as there was no identification of either the multi-disciplinary nature of CHI or identification of

13

14



contributing fields; zero on the criterion of Impact Factor as it was published in a journal which did not have an IF listed in ISI; zero on the criterion of idea density with a propositional density score of 0.52 which exceeded the 0.50 threshold, and zero on the criterion of text simplicity with a RGL of 25, far exceeding the threshold of high school (grade 12) or less. Latent semantic analysis As the ‘‘matrix comparison’’ matches each definition to one another, a total of 529 LSA scores (23 23) were produced. The mean LSA score was 0.79 out of a possible 1.00 with a standard deviation of 0.11. The median was 0.81 and the most frequent score of similarity was 0.86. The range of LSA scores were from a low of 0.37 (definitions of Motley et al. (41), Bader and Barude (42)) to a high of 0.94 (for the definitions of Anonymous (43) and Gibbons et al. (55); and of Eysenbach (38), Arocha and Hoffman-Goetz (11)). Table 3 shows the most similar and least similar definitions along with their corresponding LSA scores. Table 4 presents a list of the most frequently occurring words that appeared in the definitions identified in the 23 articles (excluding prepositions such as ‘‘of’’, ‘‘and’’, ‘‘the’’, ‘‘for’’, etc.). The five most frequently appearing words in the definitions were: ‘‘health’’, ‘‘informatics’’, ‘‘information’’, ‘‘consumer’’ and ‘‘consumers’’. Other less frequently included terms were ‘‘medical’’ ‘‘care’’, ‘‘systems’’ and ‘‘computer’’, emphasizing the health informatics origins.

DISCUSSION This is the first systematic review of peer-reviewed published definitions of consumer health informatics. We used a set of assessment criteria about what makes a ‘‘good definition’’ to more objectively evaluate the existing CHI definitions. However, we did not ‘‘judge’’ which definition, if any, is ‘‘best’’ for adoption by consumer health informatics researchers, practitioners and the public. The results of this study indicated that none of the 23 published definitions met all five criteria used to evaluate the features of good definitions. According to our analysis, in general, the definitions would be strengthened by referencing other work (criterion 1), mentioning the multi-disciplinary nature of CHI (criterion 2) and rewriting in simpler language (criterion 5). There was an interesting dichotomy between the complexity and readability of the definitions included in this review. Twenty of 23 definitions (87.0%) had a maximum score of one on the idea density criterion. A score of one meant that the idea density was low and the definition was easier to comprehend (i.e. few ideas per unit of text). In contrast, criterion five assessed the readability of the definition text and showed that virtually all definitions used difficult and complex words and jargon, and were written at a post-secondary reading grade level. These two measures of textual complexity provide insight into potential usefulness of the definitions for adoption by a wide spectrum of users. Whereas propositional density assesses the underlying conceptual complexity, readability measures, the linguistic complexity of a body of text. Most definitions were not particularly idea-dense, making them understandable and not difficult to

Consumer health informatics definitions Table 3. Definitions with the highest/lowest Latent semantic analysis scores.


Gibbons et al. (55)

Anonymous (43)

Eysenbach (38)

Arocha and Hoffman-Goetz (11)

Bader and Braude (42)

Motley et al. (41)

Consumer health informatics is defined as any electronic tool, technology or electronic application that is designed to interact directly with consumers, with or without the presence of a health care professional that provides or uses individualized (personal) information and provides the consumer with individualized assistance, to help the patient better manage their health or health care. CHI generally encompasses two areas: the use of Internet technology to connect to online services for health care information, such as that contained in medical journals and professional publications; and systems and software provided by clinicians to patients to help in the diagnosis and treatment of specific conditions and diseases. Consumer health informatics is the branch of medical informatics that analyses consumers’ needs for information; studies and implements methods of making information accessible to consumers; and models and integrates consumers’ preferences into medical information systems. Consumer informatics stands at the cross-roads of other disciplines, such as nursing informatics, public health, health promotion, health education, library science and communication science, and is perhaps the most challenging and rapidly expanding field in medical informatics. An important component of health informatics, consumer health informatics, has been defined as a field that ‘analyses consumers’ needs for information; studies and implements methods of making information accessible to consumers; and models and integrates consumers’ preferences into medical information systems’ (Eysenbach, 2000). "Patient informatics": information supplied to patients using advanced information and communication technologies. Consumer health informatics is the union of health care content with the speed and ease of technology.

LSA Score 0.94

LSA Score 0.94

LSA Score 0.37

15

16



Table 4. Most frequently used words in CHI definitions. Health Informatics Information Consumer Consumers Medical Care Systems Computer Technology Patients Healthcare Support Public Patient Needs Internet

48 28 26 25 12 10 10 7 7 5 5 5 4 4 4 4 4

Designed Defined Communication Tools Tele-communication Technologies Studies Science Models Management Making Electronic Development Delivery Decisions Better

4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3

remember. However, this was partially offset by the finding that most definitions (22/23 or 96%) were written in complex language. Ideally, for the definitions to be optimally understandable by diverse users ranging from experts to interested consumers, they would be written so that both the underlying propositional structure (PD) and the surface linguistic form (RGL) are at a low complexity level. Although both of these measures deal with the better understanding of the written definition, they are independent of one another and work complementary to one another (59). In other words, one cannot assume that in writing a definition with a low RGL that it will have a low PD (and vice versa). An interesting finding from this systematic review was that about one-third (9/23) of the definitions appeared in articles published in journals with an impact factor (IF) greater than one, potentially giving them a further reach and more credibility in the academic field than the definitions published in journals with low or no IF. This is exemplified by Eysenbach’s (38) definition of CHI, published in the British Medical Journal, which has an IF of 17.2. This definition has been cited 571 times according to Google Scholar; by comparison, the second highest Google Scholar citation was 175 for the definition of Gustafson (49) which was published in the International Journal of Medical Informatics with an IF of 2.4. Impact factor is important when considering the potential reach of the definitions. Journals with very high IFs (e.g. British Medical Journal or Journal of the American Medical Association) have media relation teams and subsequently produce press releases with their most recent/relevant articles that are sent to news journalists. Articles included in these releases are likely to be presented in the news media and therefore to reach the public (28). This is important for the field of Consumer Health Informatics. Indeed, once a consensus single definition is in place, publishing the definition in a journal with a high IF could help to widen dissemination across the component



disciplines and the interested public. For example, the article by Eysenbach (38) was published in the British Medical Journal, which has an IF of 417 and regularly produces press releases. Their articles are consistently picked up by the mass media such as The Huffington Post and the Los Angeles Times (60,61) and they keep track of the ‘‘latest BMJ articles in the news’’ on their website (62). Although Eysenbach’s (38) article (and definition) was not, the chances of it being reported in the news was undoubtedly greater than if it were published in a smaller, less prestigious journal (e.g. Alamantariotou’s definition (56) which was published in the International Journal of Electronic Healthcare, not listed in ISI and also not picked up by the media). Latent Semantic Analysis (33,34) is not a commonly used research methodology in health research, but it was uniquely suited to gain a better understanding of the extent to which these definitions were congruent or similar overall. The high LSA mean for all definitions indicates that the subject matter among the 23 definitions is quite similar. This finding suggests that although there may not be a formally agreed upon definition about CHI, there is an implicit consensus amongst the authors of what the field entails. We included a count of the most frequent words that appeared in the 23 definitions and by doing so provide an initial insight as to the most prominent aspects of CHI. We suggest that this list of core terms can provide a tentative starting point for the development of a future consensus definition. The words ‘‘consumer’’, ‘‘health’’ and "informatics’’ are obviously featured the most. Technology based words, such as ‘‘Internet’’ (four times), ‘‘communication’’ (four times), ‘‘telecommunication’’ (three times) are not as common, but may serve a vital purpose in a potential consensus definition as they are foundational to CHI. Landauer, Foltz, and Laham (34) have outlined several unique uses for LSA, which could be directly applicable to the development of a consensus definition building upon what has been published. LSA has been used in estimating passage coherence, the learnability of text passages by individual students and estimating the quality and quantity of knowledge contained in a document (34). Although some of these methods are not applicable to single sentence analysis (as was used here) they are applicable for making comparisons between documents and also before and after testing of written passages. Overall, the criteria proposed in this research are useful as a starting point for examining the core components of consumer health informatics and developing a definition that would encompass the major areas and activities of the discipline. Agreement about a single definition of CHI could potentially result in better use of health information by consumers. Future research by our team will be directed towards (a) a thematic analysis of the definitions to elucidate the most reoccurring or prominent themes in the existing body of text; (b) a conceptual analysis to uncover the core concepts in those definitions; (c) an investigation of the ‘‘grey literature’’ where other, less well-known definitions may be found; and (d) a survey of key players and stakeholders in the consumer health informatics domain (e.g. academic researchers, as well as public health practitioners) to find out what they believe to be the key components of the CHI field, which would be beneficial in both the development of core competencies and a unified definition.

17


18


Study limitations There are several limitations to this study. First, it has been suggested that systematic reviews may have less validity when a field or discipline is new (63), such as CHI, because of only a few studies in this discipline having been published. However, Petticrew and Roberts (21) state that ‘‘even when a field is immature, it is important to cumulate prospectively rather than wait for some later date when ‘‘enough’’ evidence has accumulated and consolidation can occur’’ (p. 35). A second limitation is in regards to the criteria we used to evaluate the definitions of CHI. Three of the five criteria have been applied and statistically validated elsewhere: the Simplified Measure of Gobbledygook readability index (64) and the Computerized Propositional Idea Density Rater calculator (37) have been validated for concurrent validity. The ISI journal impact factor rating has also been appraised for content validity (65). However, none of these three criteria have been directly applied to assessment of definitional quality. The other two criteria, citation/referencing and multi-disciplinary focus, were intended to be as objective as possible. Third, it is important to note the limitations in the use of SMOG when applying it to short text and medical terminology, as both of these factors can heavily influence the resulting RGL. Health informatics terminology contains terms, which are often multisyllabic words (e.g. informatics, technology); definitions are generally short in nature, giving the text a large number of ‘‘difficult’’ words, which inflates the RGL score. Still, the SMOG test is useful despite these limitations. It gives the authors of definitions and designers of health information a benchmark for before and after testing, and also a method of comparing their work to other definitions. It is also the only formula that is appropriate for analyzing materials with fewer than 100 words (66). A fourth limitation is that two of the elements we used as necessary for a good definition are drawn from the sociological rather than the informatics literature (23). A definition is a thorough description of the meaning of a term (e.g. consumer health informatics). Hart’s criteria were originally developed to improve sociological definitions and were adapted in this study for health definitions. Because health and sociology are different fields of study (with different assumptions, conceptual foundations, history, emphasis etc.) specific aspects of Hart’s criteria might be less relevant for CHI definitions. However, we believe that the core meaning of his ‘‘comprehensibility’’ and ‘‘simplicity’’ criteria translate well and are captured by the propositional density and SMOG analyses. Finally, there is a certain amount of subjectivity in the five criteria chosen, insofar that using different criteria could have resulted in different assessments of quality. It is also important to distinguish that this review was aimed at assessing the quality of CHI definitions, as opposed to identifying a specific definition that should be adopted for use in the field.

CONCLUSIONS None of the 23 definitions included in this review completely fulfilled the five criteria used to assess definitional quality. The results suggest the need for dialogue amongst professionals within the field in order to develop a unified



definition. This unified definition would in turn inform the development of core competencies. The American Medical Informatics Association has recently developed a definition and set of core competencies to guide curriculum development in biomedical informatics (67). AMIA proposed a core definition ‘‘that could be adopted for ongoing use by AMIA and would capture the field’s scope and focus’’ (67, p. 8). The discipline of consumer health informatics would benefit from a similar study as that done for biomedical informatics. Houston et al. (12) began the work in 2001 by seeking out a consensus description of the field, but CHI was still an emerging area and it was difficult to present a consensus definition at that point. We suggest that continued discussion by consumer health informaticians and other key players to define the field and establish core competencies will be an important step for improving the delivery, access and usefulness of health care information for the public.

DECLARATION OF INTEREST One of the definitions reviewed was from an article published by two of the contributing authors. We attempted to minimize bias by having the remaining (non-contributing) author do all of the evaluations and by having a secondary party grade a random sample.

REFERENCES 1. Nelson R, Ball MJ. Consumer informatics: applications and strategies in cyber health care. New York: Springer; 2004. 2. Jimison H, Gorman P, Woods S, et al. Barriers and drivers of health information technology use for the elderly, chronically ill and underserved. Evid Rep Technol Assess (Full Rep) 2008;175:1–1422. 3. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the World Wide Web. JAMA 2002;287:2691–700. 4. Hung M, Conrad J, Hon SD, et al. Uncovering patterns of technology use in consumer health informatics. Wiley Interdiscipl Rev: Comput Stat 2013;5:432–47. 5. Gibbons MC, Wilson RF, Samal L, et al. Impact of consumer health informatics applications. Evid Rep Technol Assess (Full Rep) 2009;188:1–546. 6. The Center for Health Enhancement System Studies. University of Wisconsin [Internet]. Available from: www.chess.wisc.edu/chess/projects/about_chess.aspx [last accessed 25 Jan 2014]. 7. Gustafson DH, Hawkins R, Pingree S, et al. Effect of computer support on younger women with breast cancer. J Gen Intern Med 2001;16:435–45. 8. Gustafson DH, Hawkins R, Mctavish F, et al. Internet-based interactive support for cancer patients: are integrated systems better? J Commun 2008;58:238–57. 9. Eyler AA. Consumer health informatics: improving patient engagement. Transl Behav Med 2011;1:10. 10. American Medical Informatics Association [Internet]. Available from: www. amia.org/applications-informatics/consumer-health-informatics [last accessed 5 Aug 2012]. 11. Arocha JF, Hoffman-Goetz L. A survey of public health and consumer health informatics programmes and courses in Canadian universities and colleges. Inform Health Soc Care 2012;37:242–52. 12. Houston TK, Chang BL, Brown S, Kukafka R. Consumer health informatics: a consensus description and commentary from American Medical Informatics Association members. Proceedings of the AMIA Symp 2001;2:269–73. Available

19

20


13. 14.

15.


16.

17. 18.

19. 20.

21. 22.

23. 24. 25. 26. 27. 28. 29. 30. 31. 32.

33. 34. 35.

36.

from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2243697/ [last accessed 27 Sept 2013]. McDaniel AM, Schutte DL, Keller LO. Consumer health informatics: from genomics to population health. Nurs Outlook 2008;56:216–23. Hanif F, Read JC, Goodacre JA, et al. The role of quality tools in assessing reliability of the internet for health information. Inform Health Soc Care 2009;34:231–43. COACH. Health Informatics Professional Core Competencies v3.0. Toronto Ontario 2012. Available from: www.coachorg.com. Centers for disease control and prevention. Competencies for public health informaticians 2009. Atlanta Georgia 2009. Available from: www.cdc.gov/informaticscompetencies/downloads/phi_competencies.pdf. Bernstam EV, Smith JW, Johnson TR. What is biomedical informatics? J Biomed Inform 2010;43:104–10. Van De Belt TH, Engelen LJ, Berben SA, Schoonhoven L. Definition of Health 2.0 and Medicine 2.0: a systematic review. J Med Internet Res 2010;12:e18. doi: 10.2196/jmir.1350. Oh H, Rizo C, Enkin M, Jadad A. What is eHealth? A systematic review of published definitions. J Med Internet Res 2005;7:e1. doi:10.2196/jmir.7.1.e1. Frank JR, Mungroo R, Ahmad Y, et al. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach 2010;32:631–7. Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Malden (MA): Blackwell; 2006. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.0.2. The Cochrane Collaboration 2009. Available from: www.cochranehandbook.org. Hart H. Some Methods for Improving Sociological Methods. Am Sociol Rev 1943;8:333–51. Misser E, Bell S. Why we cite sources in academic papers. Writing Centre, Wilfrid Laurier University; 2013. Available from: www.wlu.ca/writing. Mantas J. Education and consumer health informatics. IMIA yearbook of medical informatics. Stuttgart, Germany: Schattauer; 2007. Garfield E. The history and meaning of the journal impact factor. JAMA 2006;295:90–3. Seglen PO. Why the impact factor of journals should not be used for evaluating. BMJ 1997;314:498–502. Arnold K. Journals, the press and press releases: a cozy relationship. Sci Ed 2003;26:82–4. Kintsch W. The representation of meaning in memory. Hillsdale (NJ): Lawrence Erlbaum Associates (distributed by Halsted Press Division, New York: Wiley); 1974. Woods B, Moscardo G, Greenwood T. A critical review of readability and comprehensibility tests. J Tourism Stud 1998;9:49–61. McLaughlin GH. SMOG Grading: a new readability formula. J Reading 1969;12:639–46. Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav 2006;33:352–73. Landauer TK, McNamara DS, Dennis S, Kintsch W. Handbook of latent semantic analysis. New York: Routledge; 2011. Landauer TK, Foltz PW, Laham D. An introduction to latent semantic analysis. Discour Proc 1998;25:259–84. Computerized Propositional Idea Density Rater (CPDR) version 5.1. Computer Analysis of Speech for Psychological Research, Institute for Artificial Intelligence, University of Georgia 2012. Available from: www.ai.uga.edu/caspr/. Covington MA. Idea density: a potentially informative characteristic of retrieved documents. Paper presented at IEEE SoutheastCon; 2009 March 5–8;2009.


37. 38. 39. 40.


41.

42. 43. 44.

45. 46. 47.

48. 49.

50. 51.

52. 53.

54.

55.

56. 57. 58.

Available from: www.ai.uga.edu/caspr/Covington-2009-Idea-Density-paper-SEC09060.pdf [last accessed 4 Oct 2013]. Brown C, Snodgrass T, Kemper SJ, et al. Automatic measurement of propositional idea density from part-of-speech tagging. Behav Res Methods 2008;40:540–5. Eysenbach G. Recent advances: consumer health informatics. BMJ 2000;320:1713–16. Ferguson T. Consumer health informatics. Heal Forum J 1995;38:28–33. Jimison H, Sher P. Consumer health informatics: health information technology for consumers. J Am Soc Inf Sci 1995;46:783–90. Motley CM, Heyl NF, Ricks RL, et al. Consumer health informatics: emerging issues. Report to the Chairman, Subcommittee on human resources and intergovernmental relations, House committee on government reform and oversight, US General Accounting Office 1996. Available from: www.gao.gov/archive/1996/ ai96086.pdf [last accessed 4 Oct 2013]. Bader SA, Braude RM. ‘Patient informatics’: creating new partnerships in medical decision making. Acad Med 1998;73:408–11. [Anonymous]. Online services facilitate self-care practice. Empl Benef Plan Rev 1998;52:51. Bouhaddou O, Lambert JG, Miller S. Consumer health informatics: knowledge engineering and evaluation studies of medical HouseCall. Annual AMIA Symposium Proceedings, 1998;612–16. Brennan PF. Health informatics and community health: support for patients as collaborators in care. Methods Inf Med 1999;38:274–8. Rhodes E. Consumer informatics: helping patients to access health information via the Internet. Nurs Connect 2000;13:33–42. Committee on enhancing the Internet for health applications: technical requirements and implementation strategies CSaTB, National Research Council. Networking Health: Prescriptions for the Internet. Washington, DC: National Academic Press; 2000. Available from: www.ncbi.nlm.nih.gov/books/NBK44724/ [last accessed 5 Oct 2013]. Lewis D, Pesut D. Emergence of consumer health care informatics. Nurs Outlook 2001;49:7. Gustafson DH, Hawkins RP, Boberg EW, et al. CHESS: 10 years of research and development in consumer health informatics for broad populations, including the underserved. Int J Med Inform 2002;65:169–77. Eysenbach G. The semantic web and healthcare consumers: a new challenge and opportunity on the horizon? Int J Technol Manag 2003;5:194–212. Perry G, Weldon S. Consumer health informatics research: implications for consumers, health information professionals and researchers. J Med Internet Res 2005;9:1–10. Bakker TA, Ryce AN, Logan RA, et al. A consumer health informatics (CHI) toolbox: challenges and implications. AMIA Ann Sympos Proc 2005;2:21–5. Khan SA, McFarlane DJ, Li, J, et al. Healthy Harlem: empowering health consumers through social networking, tailoring and web 2.0 technologies. AMIA Annual Symposium Proceedings; 2007:1007. Keselman A, Logan R, Smith CA, et al. Developing informatics tools and strategies for consumer-centered health communication. J Am Med Inform Assoc 2008;15:473–83. Gibbons MC, Wilson RF, Samal L, et al. Consumer health informatics: results of a systematic evidence review and evidence based recommendations. Transl Behav Med 2009;1:72–82. Alamantariotou K. Consumer health informatics and interactive visual learning tools for health. Int J Electron Healthc 2010;5:414–24. Ho J. Consumer health informatics. (Health Informatics: an overview). Stud Health Technol Inform 2010;151:185–94. Shaikh AR, Prabhu Das I, Vinson CA, Spring B. Cyber-infrastructure for consumer health. Am J Prevent Med 2011;40:91–6.

21


22


59. Ta-Min R, Arocha JF, Hoffman-Goetz L. Assessing readability and comprehensibility of web-based cancer information. J Inform Technol Heal 2007;5:300–12. 60. The Huffington Post Canada. War On Drugs Catastrophic Failure, BMJ Open Study Finds. [Internet]. Available from: www.huffingtonpost.ca/2013/10/01/war-ondrugs-study_n_4023486.html#slide¼887965 [last accessed 10 Oct 2013]. 61. Los Angeles Times. Time to end the war against saturated fat? [Internet]. Available from: www.latimes.com/science/sciencenow/la-sci-saturated-fat-20131022,0,2193813. story#axzz2jEiqIJyS [last accessed 24 Oct 2013]. 62. British Medical Journal. Latest BMJ Headlines in the News. [Internet]. Available from: www.bmj.com/news [last accessed 24 Oct 2013]. 63. Ravetz J. Scientific knowledge and its social problems. Middlesex: Penguin University Books; 1973. 64. Ley P, Florio T. The use of readability formulas in health care. Psychol Health Med 1996;1:7–28. 65. Saha S. Impact factor: a valid measure of journal quality? J Med Libr Assoc 2003; 91:42–6. 66. Bastable, S. Essentials of patient education. Sudbury (MA): Jones and Bartlett; 2006. 185p. 67. Kulikowski CA, Shortliffe EH, Currie LM, et al. AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. J Am Med Inform Assoc 2012; 19:9318. doi: 10.1136/amiajnl-2012-001053.

What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products.

What Is Fitness Training? Definitions and Implications: A Systematic Review Article.

Uncovering patterns of technology use in consumer health informatics.

What is open peer review? A systematic review.

What is open peer review? A systematic review.

What is actually measured in process evaluations for worksite health promotion programs: a systematic review.

What is the Relationship between Risky Outdoor Play and Health in Children? A Systematic Review.

International telepsychiatry: a review of what has been published.

Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review.

What Factors Work in Mental Health Court?: A Consumer Perspective.

Definitions, Foundations and Associations of Physical Literacy: A Systematic Review.

Suicide contagion: a systematic review of definitions and research utility.

Social media and patient health outcomes. Findings from the yearbook 2014 section on consumer health informatics.

Transforming consumer health informatics through a patient work framework: connecting patients to context.

Transforming consumer health informatics: connecting CHI applications to the health-IT ecosystem.

Peri-implantitis: a systematic review of recently published papers.

A review of analytics and clinical informatics in health care.

What is kangaroo mother care? Systematic review of the literature.

'Spin' in published biomedical literature: A methodological systematic review.

2016 Year-in-Review of Clinical and Consumer Informatics: Analysis and Visualization of Keywords and Topics.

What Are the Real Procedural Costs of Bariatric Surgery? A Systematic Literature Review of Published Cost Analyses.

Systematic Review of Medical Informatics-Supported Medication Decision Making.

Acadian Mental Health: A Review of Published Literature.

What information is used in treatment decision aids? A systematic review of the types of evidence populating health decision aids.