Gap Analysis of Biomedical Informatics Graduate Education Competencies Anna L. Ritko, MA, MPhil1, Michelle Odlum, MPH, EdD2 1 University of Oklahoma, Tulsa, OK; 2City University of New York, NY, NY Abstract Graduate training in biomedical informatics (BMI) is evolving rapidly. BMI graduate programs differ in informatics domain, delivery method, degrees granted, as well as breadth and depth of curricular competencies. Using the current American Medical Informatics Association (AMIA) definition of BMI core competencies as a framework, we identified and labeled course offerings within graduate programs. From our qualitative analysis, gaps between defined competencies and curricula emerged. Topics missing from existing graduate curricula include community health, translational and clinical research, knowledge representation, data mining, communication and evidence-based practice. Introduction Definitions of what is and what is not informatics continue to be discussed and debated1,2. Based on survey data collected in 2008, biomedical education accounted for 177 programs within the US3. As of 2011, a comparison of program curricula identified 73 Masters-level graduate programs.4 Recent work on informatics curricula has defined educational competencies and requirements based on literature reviews, Internet searches, needs assessments and analysis of health information technology workforce data sources5-7. The International Medical Informatics Association (IMIA) informs a global perspective on educational competencies with a three-dimensional framework of educational needs and recommended course content8. Making sense of what is and what is not being taught proves increasingly difficult. In 2012, AMIA published a white paper defining core competencies for BMI graduate education as fundamental skills in science, discipline-focused concepts, theories and methods, application of technologies, and human and social context9. These broad definitions encompass the knowledge and skills required to investigate, analyze, evaluate and disseminate results within BMI. The scope of our work serves to evaluate these competency definitions within the context of current graduate education, and assess saturation of curricula to identify areas for innovation in development of a new academic informatics program. To compare existing BMI graduate curricula and core competencies, we used a grounded theory approach to data analysis of education10,11. Methods To collect, analyze and evaluate US-based BMI training, we employed qualitative analysis to perform gap analysis of current graduate programs. Initially, we identified all BMI-related training programs by searching online using both keywords as well as graduate program websites. Using Google, we searched using terms ‘health informatics programs’, ‘clinical informatics programs’ and ‘biomedical informatics’. We also examined lists of graduate programs identified on education websites affiliated with academic institutions, accreditation associations, government agencies, professional associations and marketing services12-15. We compiled a list of university names, program titles, school, college or department names affiliated with each program, the degree types offered and curriculum as defined by each program’s website. Post-doctoral training was excluded from our data collection due to study emphasis on post-baccalaureate and pre-doctoral training. We also did not collect nor analyze data on graduate programs centered exclusively on bioinformatics due to several existing studies on bioinformatics programs16. The resulting list comprises 205 unique offerings of programs and degrees granted. From this list, we identified all courses associated with each program by reviewing the curriculum detailed on each program’s website. From the program list identified, we collected 1722 course titles. Using AMIA’s recent white paper on BMI graduate education as a guide, we developed a 2-dimensional coding framework (Figure 1) consisting of four main categories of competencies and related topics9. Based on our reading of AMIA’s definition as well as thematic analysis of graduate course titles, we defined topics related to each core competency.
1214
Scope and Breadth of Discipline Biology Biotechnology Consumer Health Healthcare Systems Nursing Population Health Translational and Clinical Research
Theory and Methodology Frameworks Methods and Processes Knowledge Management Knowledge Representation Theories Typology
Technological Approach Database Systems Data Mining Imaging and Signal Analysis Language Processing Networking, Security and Privacy Software Engineering System Analysis and Design
Human and Social Context Communication Decision Support Tools Evidence-Based Practice Human Computer Interaction Legal, Ethical and Social Issues Organizational Behavior and Management Research Design and Evaluation
Figure 1. Framework of BMI competencies and topics. Using this framework as a guide, we (AR, MO) labeled all 1722 course titles. Cohen’s kappa was 0.43, a moderate level of agreement. We deemed this level of agreement as satisfactory for an initial round of analysis given our contrasting perspectives as informaticists: one with a background in nursing and the other as a computer scientist. Differences in coding also demonstrate the interdisciplinary nature of BMI, with differences in opinion on what topics constitute fundamental concepts of BMI versus methods and technology. Results Within the US, 91 universities provide BMI programs, excluding programs specific to bioinformatics and postdoctoral training. From these 91 universities, there are 60 distinct program titles, and 205 unique program-degree offerings. This variety in graduate training demonstrates multiple practice-level perspectives on informatics, enabling students with multiple backgrounds and interests to pursue applied, research or clinical-focused training. The variations in degrees illuminate different student populations of BMI. Within US-based BMI training, 77 programs grant graduate-level certificates, 98 programs grant masters degrees, and 30 grant doctoral degrees. Of these degrees, 24 types of master’s degrees are offered, and 7 types of certificates are offered. Several masters programs focus on professional experience, with a Professional Science Masters, an Executive Masters of Science and an MBA with a specialty in Health Information. Certificate programs demonstrate options in levels of experience, with one program offering an Educator’s Certificate for current instructors of information technology. As seen in Figure 2, 36 states offer BMI graduate education. California, Massachusetts, North Carolina and Texas offer the highest number of degrees granted within BMI. Certain areas of the country are saturated, while other regions offer minimum to moderate levels of educational options.
ME
WA AK
VT
ND
MT
MN NY
OR
WI SD
ID
MI
WY
PA IA
NE
OH
NV
IL
IN
WV
UT CO
CA
KS
VA
MO
KY
NC
TN
HI AZ
OK NM
MS TX
SC
AR AL
GA
LA FL
1-5 Programs 6-10 Programs 11-15 Programs 16+ Programs
Figure 2. US map of BMI graduate education training.
1215
NH MA CT NJ DE MD
RI
Table 1. Concentration (%) of competencies across all programs within BMI. Core Competency Scope and Breadth of Discipline Theory and Methodology Technological Approach Human and Social Context
Average Program Concentration 24 23 19 34
BMI Competency Concentration Graduate curricula varies in focus of competencies. To compare graduate curricula, we averaged the percentage of course content for topics and competencies across all programs. We analyzed the courses associated with 198 programs from the 205 unique degree programs compiled. Six programs were not used within the analysis because the curricula was not available from their websites. Table 1 highlights differences in program concentration of competencies for BMI graduate programs. Programs average higher focus on course content on the human and social context (34%) of BMI. Discipline (24%), and theory and methodology (23%) average similarly with almost one-quarter each of program competencies. Technological approaches (19%), however, comprise the least focus of graduate competencies. These program averages demonstrate the dominance of fundamental knowledge applied within the context of the healthcare environment for BMI graduate training. Information technologies also form one of the core foundations of BMI, but curricula demonstrates a lesser depth in focus on this competency. Prerequisite advanced knowledge in computer science methods and applications could be considered a core requirement within many of these programs, thereby influencing increased curricula development towards other competencies and topics17. Domain-specific analysis of competencies demonstrates different levels of focus. Programs identified within this study represent seven domains of informatics. Each program’s curriculum of BMI competencies demonstrates the differences in each domain (Table 2). Programs in consumer health informatics show the highest concentration in context-focused curricula (46%) and the lowest in technological approach (9%). Other informatics domains also show greater focus on human and social context-related curricula: nursing (41%), health (36%), public health (35%) and clinical (34%). These same domains focus less on technologies. Biomedical informatics, as research-focused domain, focuses on the competencies of discipline (31%) and theory and methodology (28%). Imaging informatics highlights theory and methodology (38%) as the core competency of curricula. Proliferation of BMI programs demonstrates the focus of graduate education on applied practice-levels. Health informatics accounts for 111 of 195 training programs analyzed within this study. Differing applications of health informatics with an emphasis on technologies and context comprise subcategories of this domain, with degree programs emphasizing security and privacy, software engineering, information technology leadership, health administration and applied research. Many of these programs result from associations between two or more departments, schools or colleges within the academic institution, with focus of competencies and topics dependent on the departments offering the program. Table 2. Concentration (%) of competencies for program domains within BMI. Program Domain Biomedical Informatics Clinical Informatics Consumer Health Informatics Health Informatics Imaging Informatics Nursing Informatics Public Health Informatics
Discipline 31 20 22 22 33 25 26
Theory and Methodology 28 24 22 22 38 21 26
1216
Technological Approach 17 22 9 21 24 13 13
Human and Social Context 23 34 46 36 5 41 35
Scope and Breadth of Discipline Biology, as the study of organisms and organic systems, is considered a foundation of all BMI domains3,5,9,18. Course content includes bioinformatics, pathology informatics, biochemistry, pathophysiology, biomedical physics, molecular informatics, genetics, molecular phylogenetics, neuroscience and genomics. Biotechnology, the application and use of biology to learn about living systems, is a required competency of bioinformatics16. Biotechnology focuses on introductory courses in biomedicine, electrical engineering, biomedical imaging, medical instrumentation and imaging informatics. Consumer, or personal, health represents development and evaluation of patient-centered technologies. Two programs offer degrees in consumer health, with one doctoral program in ‘Personal Health Informatics’ and one masters program in ‘Clinical Informatics and Patient-Centered Technologies’19,20. Key concepts of courses offered cover e-health, e-commerce, health services marketing and technology within developing countries. Healthcare system encompasses all perspectives of patient care, with courses covering the history of US healthcare systems, telemedicine and telehealth, pervasive technologies, integrated health networks, health organization and administration, electronic health records (EHR), health economics and healthcare delivery systems. Nursing represents the nurse as facilitator and promoter of health resources and references, and as communication mediator between clinicians, individuals, families and communities21 . Courses on this topic covers introductory and advanced foundations to nursing informatics. Population health studies the distribution of health outcomes amongst individuals and communities, evaluating the multi-level effects of environment, social structures, medical practice and other enabling systems on health22,23. Course titles such as ‘Behavioral Sciences in Public Health’, ‘Ecological Information Systems’ and ‘GIS Applications in Public Health’ demonstrate the broad reach of public health informatics. Translational and clinical research focuses on evaluation and interpretation of research into clinical practice, and is now considered a competency of all informatics domains17,24. Clinical trials, pharmacology and personalized care represent key themes of this topic within graduate education. Topics defined within the scope and breadth of discipline represent prerequisite and fundamental knowledge of BMI. Figure 3 highlights the differences in distribution of topics within this competency. Topics with the lowest concentration include nursing (2%), biotechnology (3%), and translational and clinical research (3.5%). Consumer health (6%), biology (9.5%) and population health (12%) comprise almost a third of discipline-related topics. However, healthcare system (64%) concepts comprise the majority of course content taught on the discipline of BMI. Discipline-related Topic Distribution Biology Biotechnology Consumer Health Healthcare System Nursing Population Health Translational/Clinical Research 0
20
40
60
80
100 %
Figure 3. Topic distribution within the competency of scope and breadth of discipline.
1217
Theory and Methodology Frameworks represent the varying approaches to framing and conceptualizing knowledge within BMI9. This topic includes analysis of service delivery models, e-health business models, decision models and psychosocial factors influencing health behavior. Knowledge management is the organization and administration of information, with an emphasis on the design, architecture, development and maintenance of information technologies25. We separate knowledge representation from knowledge management due to continued discussion on the inclusiveness of health information management as a discipline of BMI1,26. Knowledge representation is the abstraction and encoding of data into knowledge, with emphasis on metadata and ontologies to enable syntactic and semantic interoperability between information systems25. The Leadership in Health Information Technology for Health Professionals program offers a course exploring the analysis of standards, ethics and stakeholder values of knowledge representation and interoperability27. Methods and processes consist of qualitative, quantitative, mixed and computational subjects. Cognitive and social science methods include health promotion, health services program planning, decision analysis and q-methodology. Methods common to all sciences include empirical methods, quality appraisal and evaluation, and survey sampling. Population health methods include biostatistics, disease surveillance, epidemiology, survival analysis and outcomes analysis, stochastic methods for biology, and genetic analysis and discovery. Computer-based methods of simulation modeling, statistical learning, dynamical models and visualization are also categorized into this topic. Lastly, business process and methodology is included in this topic with courses on computer-supported collaborative work, lean six sigma and methods of performance measurement. Typology classifies knowledge into types, and consists of course content focused on existing medical terminologies, standards, classification systems and vocabularies within BMI. Theories form the principles of BMI, with an interdisciplinary cross-section of theoretical foundations of multiple sciences. This topic provides principles, foundations, perspectives and concepts of discipline-related topics of each domain. Course titles include ‘Perspectives in Environmental Health’, ‘Health Data: Theory and Practice’, ‘Principles of Clinical and Translational Science’, and ‘Theoretical Foundations in Personal Health Informatics’. The theories, methods and processes of BMI form the underlying principles to investigating and solving problems within the field. Figure 4 demonstrates the dominance of methods and processes within this competency. Frameworks of BMI (2.5%), knowledge representation (3.5%), typology (9%), theories (9.5%), and knowledge management (21%) comprise almost half of all topics taught within the theory and methodology competency. Methods and processes (54.5%) have the highest concentration within this competency. Theory and Methodology-related Topic Distribution Frameworks Knowledge Management Knowledge Representation Methods and Processes Theories Typology 0
20
40
60
80
Figure 4. Topic distribution within the competency of theory and methodology.
1218
100 %
Technological Approach Database systems courses within BMI graduate education focus on clinical analytics, data evaluation, data modeling for implementation, data structures and algorithms, data warehousing, business intelligence and database management, organization, file structure, performance tuning, querying, and security. Data mining of large data sets has gained recognition as an invaluable approach to investigation within many fields. Many aspects of data mining can be seen in current graduate programs, with courses on R, predictive analytics, web mining, pattern recognition, machine learning and aggregating health data. Imaging and signal analysis focuses on the processing and interpretation of images, speech and other medically relevant inputs. For graduate education, course content includes understanding, processing and analyzing images. Language processing is the processing of natural or spoken language into meaning. Applications of natural language processing include ontological development from EHRs and development of question and answering systems. Course titles include ‘Introduction to Medical Language Processing’, ‘Shallow Processing Techniques for Natural Language Processing’ and ‘Text Understanding and Information Retrieval’. Networking, security and privacy issues are an important aspect of patient-care, and consequently, have become a subcategory of health informatics with several programs offering a focus on security and privacy as a degree. This topic covers telecommunications, telehealth, telemedicine, software-as-a-service (SaaS), health information exchange issues, compliance, software and hardware assurance, cloud computing and HL7. Software engineering encompasses the many programming languages, design architectures, development strategies and applications of computer science and information technologies. Course content covers both existing and emerging trends in design and development of software systems, including XML, virtual reality, robotics, SQL, object-oriented design, as well as problem-oriented programming. System analysis and design covers courses on EHRs, clinical data acquisition, design, selection and management of health care systems, enterprise information systems, workflow mapping and reengineering. The theory and methodology of computer science intersects with BMI to define approaches taken to investigating and solving problems within this field. Figure 5 demonstrates the evolving technological approaches of BMI, with the emerging technologies of imaging and signal analysis (2.5%), language processing (3.5) and data mining (6%) representing the least amount of focus within this competency. System analysis and design (21%), database systems (23.5%), and networking, security and privacy (22.5%) comprise the main categories of technology applications to BMI. These results highlight computer science technologies with a history of long-term use as main topics of this competency. Technological Approach-related Topic Distribution Database Systems Data Mining Imaging/Signal Analysis Language Processing Networking, Security, Privacy Software Engineering System Analysis & Design 0
20
40
60
80
Figure 5. Topic distribution within the competency of technological approach.
1219
100 %
Human and Social Context Communication enables dissemination and application of results into practice, within the clinical, organizational or research environments. Course content focuses on communication through teaching, marketing, management, program planning, grant writing, scientific and strategic writing as well as publication and presentation skills. Decision support tools represent existing information systems and technologies developed to assist with the cognitive process in making a choice. Course titles include ‘Administrative and Clinical Decision Support Systems’ and ‘Decision Support and Intelligent Systems’. Evidence-based practice is the application of evidence-based research into medicine and clinical practice. Minimal course content exists on this topic, with evidence-based practice situated within the context of nursing or research design. Human computer interaction, as a topic within BMI, emphasizes the design and evaluation of technology based on it’s downstream influence within healthcare. Course content covers user interface design and development, usability, needs assessment, cognitive science, medical device strategy and clinician information needs. Legal, ethical and social issues encompass the implications of applying technology within healthcare, with course content focused on organizational context of healthcare computing, and policy development arising from health law. Organizational behavior and management covers a broad range of courses on multiple aspects of healthcare culture arising from an organizational context. Courses include management of health care practice, reimbursement systems, global enterprise, change leadership, financial accounting, project management, negotiation and conflict resolution, strategic planning and risk management. Research design and evaluation is the application of knowledge and skills to implementing, evaluating and then reiteratively improving upon solutions within the human and social context of healthcare and research. This topic covers courses in applied research projects, residency in health informatics, quality improvement, clinical trials and case studies in health information management. The topics of the competency of human and social context represent different perspectives of applying and practicing BMI within the healthcare environment. Evidence-based practice (1.5%) demonstrates the least focus of all topics within this competency. Communication (4.5%), decision-support tools (6.5%), and human-computer interaction (6.5) are amongst the next topics receiving low concentration of course content. Legal, ethical and social issues (13.5%) receive a moderate amount of human and social contextual content within graduate curricula. Research design and evaluation (32.5%), and organizational behavior and management (35%) comprise the majority of course content within this competency. Human and Social Context-related Topic Distribution Communication Decision Support Tools Evidence-Based Practice Human Computer Interaction Legal, Ethical, Social Issues Organizational Behavior/Mgmt Research Design & Evaluation 0
20
40
60
80
Figure 6. Topic distribution within the competency of human and social context.
1220
100 %
Discussion BMI graduate curricula must be in line with the ever-changing workforce needs. Labor market surveys for BMI indicate 63% of doctoral graduates conduct research after graduation, with 31% developing informatics tools or health information technology28. Graduates of advanced study in BMI continue to pursue multiple career paths not traditionally considered in BMI graduate education. Increasing numbers of training programs of health informatics, nursing informatics, public health informatics and consumer health informatics show the demand for training at multiple practice-levels. In fact, graduates of advanced study in BMI continue to pursue multiple career paths not traditionally considered in BMI graduate education. Minimal studies have mapped the needs of healthcare organizations and information systems departments to the educational requirements of informatics curricula7,17,22. Several studies on educational competencies and requirements within BMI have identified the breadth of BMI graduate education2,3,6-8,16,17,22. No recent work has examined the depth of curricula available in existing graduate education. The intent of this study is to identify existing and emerging topics not fully covered by existing degree programs. Our gap analysis demonstrates minimal coverage of the following topics: population health, translational research, knowledge representation, data mining, communication and evidence-based practice. Community Health Population health, with a focus on prevention, is the foundation for degree programs centered on public health informatics23. However, we found only three courses focused on community health amongst all BMI graduate education. The distinction of community health from population health relies on characteristics of a community. Improving community health requires a system-wide perspective on the multi-level effects of physical, social and cultural factors as well as the manner in which a community organizes itself29. Evaluation of health delivery models attuned to community care must incorporate measures of environment, primary care, specialty care, beliefs, insurance coverage, community health partners, care coordination and costs of care. With minimal evidence on methods to improve health outcomes and reduce health disparities amongst the medically underserved in our communities, we continue to look for mechanisms of change30. Increased training for the regional healthcare workforce helps to build a foundation for future efforts in improving public health at the community-level31. Education on health delivery models, mining administrative and managerial data sources, and cost improvement measures is required to train and support workforce within community medicine. Evaluation and maintenance of community medicine is one aspect of public health. Dissemination of results from improved systems of care, by removing regional information silos, is necessary in reducing health disparities nationally. Translational Research The ability to disseminate research results from community health into actionable health care delivery models relies increasingly on translational and clinical research methods. Federal government initiatives in research and healthcare workforce development place great emphasis on translating evidence-based knowledge into working and sustaining improvement in population health. Evidence-based practice is also gaining traction within the research areas of health services, comparative effectiveness and patient-centered outcomes, in an effort to evaluate and disseminate medical theory into practice. Subspecialties of care remain absent from graduate curricula as well. As systems of information become attune to specifics of specialty care, there is increased demand for training on subdomains of health informatics. Subdomains include behavioral health, dermatology, oncology, long-term care, obstetrics, gynecology, osteopathic and gerontology. Job listings for oncology informatics, pathology informatics and cheminformatics demonstrate the need to understand the requirements of these specializations and skillsets. For example, EHR systems differ significantly across specialties with specific requirements for security, privacy, reimbursement, compliance, documentation and workflows. Comprehension and application of specialty medical practice is essential to providing value within varying healthcare environments.
1221
Data Mining and Interoperability Data silos continue to exist, even within a single healthcare organization, with most informatics development completed by a mix of internal staff, external consultants and vendors. Integration and interoperability between multiple data sources becomes an issue of knowledge management and representation of such knowledge; thus emphasizing the need for data mining techniques. An expansive approach to education is required to enable data mining of knowledge organization systems. In fact, with the growing demand for analysis and evaluation of massive data sets within the medical domain, the emerging methods of data mining and language processing require increased instruction. Translation of such technologies to existing problems in health may assist in informing health economics and policy, as it has in existing data sources with genomics and social media-based communication. Dissemination Through Communication Communication as a mechanism for translation and dissemination is also largely absent from graduate education. Multiple avenues exist to communicate with peers, and increased focus on leveraging and expanding networks of researchers, working professionals, clinicians and the public must be part of the future of informatics education, research and practice. Many career pathways exist within BMI, but instruction on long-term employability is absent from current graduate curricula. The interdisciplinary field of BMI necessitates a continuous evaluation of mapping education to the needs of the healthcare workforce. Conclusion To understand the current status of BMI graduate education within the US, we conducted thematic analysis of 1722 courses from 205 unique degree programs. Our methodology was dependent on the accuracy and completeness of course descriptions listed on websites associated to each degree program. From this analysis, we identified gaps in accessibility as well as availability of course content supporting AMIA’s outline of core competencies for BMI graduate education. The states of Connecticut, Idaho, Maine, Nebraska, Nevada, New Hampshire, New Mexico, Oklahoma, Rhode Island, South Carolina, Vermont and Wyoming do not currently offer a single program with course content in BMI. Additionally, minimal course content is offered on community health, translational research, knowledge representation, data mining, communication for dissemination as well as evidence-based practice. Workforce skills required to support patient safety, accountable care, meaningful use and quality improvement may not be served by the availability and accessibility of current graduate education. In the future, we will evaluate how local workforce needs map to BMI graduate education competencies. References 1. 2.
Friedman CP. What informatics is and isn't. J Am Med Inform Assoc. 2013;20(2):224–6. Hersh W. Who are the informaticians? What we know and should know. J Am Med Inform Assoc. 2006;13(2):166–70. 3. Kampov-Polevoi J, Hemminger BM. Survey of biomedical and health care informatics programs in the United States. J Med Libr Assoc. 2010;98(2):178. 4. Kampov-Polevoi J, Hemminger BM. A curricula-based comparison of biomedical and health informatics programs in the USA. J Am Med Inform Assoc. 2011;18(2):195–202. 5. Huang QR. Competencies for graduate curricula in health, medical and biomedical informatics: a framework. Health Informatics J. 2007;13(2):89–103. 6. Hersh W. The health information technology workforce: estimations of demands and a framework for requirements. Appl Clin Inform. 2010;1(2):197–212. 7. Hersh W, Wright A, Garets D, Davis M, Pilolla L. Characterizing the health information technology workforce: Analysis from the HIMSS Analytics™ database. Portland, OR: http://www. billhersh. info/# workforce. 2008. 8. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med. 2010;49(2):105–120. 9. Kulikowski CA, Shortliffe EH, Currie LM, Elkin PL, Hunter LE, Johnson TR, et al. AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. J Am Med Inform Assoc. 2012;19(6):931-8. 10. Kennedy TJ, Lingard LA. Making sense of grounded theory in medical education. Med Educ. 2006;40(2):101–
1222
8. 11. Glaser BG, Strauss AL. The discovery of grounded theory: Strategies for qualitative research. Aldine de Gruyter; 1967. 12. Commission on Accreditation for Health Informatics and Information Management Education (CAHIIM) [Internet]. CAHIIM; 2013 [cited 2013 Mar 14]. Available from: http://www.cahiim.org/applyaccred_HI_grad.html 13. NLM's University-based Biomedical Informatics Research Training programs [Internet]. Bethesda (MD): U.S. National Library of Medicine; 2013 [cited 2013 Mar 14]. Available from: http://www.nlm.nih.gov/ep/GrantTrainInstitute.html 14. HIMSS. Healthcare Information and Management Systems Society (HIMSS) [Internet]. Chicago: HIMSS; 2013 [cited 2013 Mar 14]. Available from: http://www.himss.org 15. Paton C. Health Informatics Forum [Internet]. 2013 [cited 2013 Mar 14]. Available from: http://www.healthinformaticsforum.com/health-informatics-degrees-and-certificates 16. Hemminger BM, Losi T, Bauers A. Survey of bioinformatics programs in the United States. J Am Soc Inf Sci Tec. 2005;56(5):529–37. 17. Bernstam EV, Hersh WR, Johnson SB, Chute CG, Nguyen H, Sim I, et al. Synergies and distinctions between computational disciplines in biomedical research: perspective from the Clinical and Translational Science Award programs. Acad Med. 2009;84(7):964. 18. Johnson SB. A framework for the biomedical informatics curriculum. AMIA Annu Symp Proc. 2003:331–5. 19. Ph.D. in Personal Health Informatics [Internet]. Boston: Northeastern University; 2013 [cited 2013 Mar 12]. Available from: http://phi.ccs.neu.edu 20. Clinical Informatics & Patient-Centered Technologies [Internet]. Seattle: University of Washington; 2013 [cited 2013 Mar 12]. Available from: http://clinical-informatics.uw.edu 21. Alexander GL, Hull SC. Longitudinal Care Planning and Coordination in Long-Term and Post–Acute Care Settings. Comput Inform Nurs. LWW; 2012;30(9):454–5. 22. Soto Mas F, Hsu CE. Understanding public health informatics competencies for mid-tier public health practitioners-a Web-based Survey. Health Informatics J. 2012;18(1):66–76. 23. Joshi A, Perin DMP. Gaps in the Existing Public Health Informatics Training Programs: A Challenge to the Development of a Skilled Global Workforce. Perspect Health Inf Manag. 2012. 24. Sarkar IN. Biomedical informatics and translational medicine. J Transl Med. 2010;8(1):22. 25. Stock WG, Peters I, Weller K. Social semantic corporate digital libraries: Joining knowledge representation and knowledge management. Advances in Librarianship. Emerald Group Publishing Limited; 2010;32:137–58. 26. Hersh W. A stimulus to define informatics and health information technology. BMC Med Inform Decis Mak. 2009;9(1):24. 27. Leadership in Health Information Technology for Health Professionals Certificate [Internet]. Minneapolis: University of Minnesota School of Nursing; 2013 [cited 2013 Mar 14]. Available from: http://www.nursing.umn.edu/cphli/ 28. Dugan S, Saunders J. The Labor Market for Individuals with Advanced Degrees in Biomedical Informatics. 2011. 29. McKenzie J, Pinger R, Kotecki J. An introduction to community health. Jones & Bartlett Learning; 2011. 30. Jackson GL, Powers BJ, Chatterjee R, Bettger JP, Kemper AR, Hasselblad V, et al. The Patient-Centered Medical Home: A Systematic Review. Ann Intern Med. 2013;158(3):169–78. 31. Rosenthal EL, Brownstein JN, Rush CH, Hirsch GR, Willaert AM, Scott JR, et al. Community health workers: part of the solution. Health Aff. 2010;29(7):1338–42.
1223
1214
Scope and Breadth of Discipline Biology Biotechnology Consumer Health Healthcare Systems Nursing Population Health Translational and Clinical Research
Theory and Methodology Frameworks Methods and Processes Knowledge Management Knowledge Representation Theories Typology
Technological Approach Database Systems Data Mining Imaging and Signal Analysis Language Processing Networking, Security and Privacy Software Engineering System Analysis and Design
Human and Social Context Communication Decision Support Tools Evidence-Based Practice Human Computer Interaction Legal, Ethical and Social Issues Organizational Behavior and Management Research Design and Evaluation
Figure 1. Framework of BMI competencies and topics. Using this framework as a guide, we (AR, MO) labeled all 1722 course titles. Cohen’s kappa was 0.43, a moderate level of agreement. We deemed this level of agreement as satisfactory for an initial round of analysis given our contrasting perspectives as informaticists: one with a background in nursing and the other as a computer scientist. Differences in coding also demonstrate the interdisciplinary nature of BMI, with differences in opinion on what topics constitute fundamental concepts of BMI versus methods and technology. Results Within the US, 91 universities provide BMI programs, excluding programs specific to bioinformatics and postdoctoral training. From these 91 universities, there are 60 distinct program titles, and 205 unique program-degree offerings. This variety in graduate training demonstrates multiple practice-level perspectives on informatics, enabling students with multiple backgrounds and interests to pursue applied, research or clinical-focused training. The variations in degrees illuminate different student populations of BMI. Within US-based BMI training, 77 programs grant graduate-level certificates, 98 programs grant masters degrees, and 30 grant doctoral degrees. Of these degrees, 24 types of master’s degrees are offered, and 7 types of certificates are offered. Several masters programs focus on professional experience, with a Professional Science Masters, an Executive Masters of Science and an MBA with a specialty in Health Information. Certificate programs demonstrate options in levels of experience, with one program offering an Educator’s Certificate for current instructors of information technology. As seen in Figure 2, 36 states offer BMI graduate education. California, Massachusetts, North Carolina and Texas offer the highest number of degrees granted within BMI. Certain areas of the country are saturated, while other regions offer minimum to moderate levels of educational options.
ME
WA AK
VT
ND
MT
MN NY
OR
WI SD
ID
MI
WY
PA IA
NE
OH
NV
IL
IN
WV
UT CO
CA
KS
VA
MO
KY
NC
TN
HI AZ
OK NM
MS TX
SC
AR AL
GA
LA FL
1-5 Programs 6-10 Programs 11-15 Programs 16+ Programs
Figure 2. US map of BMI graduate education training.
1215
NH MA CT NJ DE MD
RI
Table 1. Concentration (%) of competencies across all programs within BMI. Core Competency Scope and Breadth of Discipline Theory and Methodology Technological Approach Human and Social Context
Average Program Concentration 24 23 19 34
BMI Competency Concentration Graduate curricula varies in focus of competencies. To compare graduate curricula, we averaged the percentage of course content for topics and competencies across all programs. We analyzed the courses associated with 198 programs from the 205 unique degree programs compiled. Six programs were not used within the analysis because the curricula was not available from their websites. Table 1 highlights differences in program concentration of competencies for BMI graduate programs. Programs average higher focus on course content on the human and social context (34%) of BMI. Discipline (24%), and theory and methodology (23%) average similarly with almost one-quarter each of program competencies. Technological approaches (19%), however, comprise the least focus of graduate competencies. These program averages demonstrate the dominance of fundamental knowledge applied within the context of the healthcare environment for BMI graduate training. Information technologies also form one of the core foundations of BMI, but curricula demonstrates a lesser depth in focus on this competency. Prerequisite advanced knowledge in computer science methods and applications could be considered a core requirement within many of these programs, thereby influencing increased curricula development towards other competencies and topics17. Domain-specific analysis of competencies demonstrates different levels of focus. Programs identified within this study represent seven domains of informatics. Each program’s curriculum of BMI competencies demonstrates the differences in each domain (Table 2). Programs in consumer health informatics show the highest concentration in context-focused curricula (46%) and the lowest in technological approach (9%). Other informatics domains also show greater focus on human and social context-related curricula: nursing (41%), health (36%), public health (35%) and clinical (34%). These same domains focus less on technologies. Biomedical informatics, as research-focused domain, focuses on the competencies of discipline (31%) and theory and methodology (28%). Imaging informatics highlights theory and methodology (38%) as the core competency of curricula. Proliferation of BMI programs demonstrates the focus of graduate education on applied practice-levels. Health informatics accounts for 111 of 195 training programs analyzed within this study. Differing applications of health informatics with an emphasis on technologies and context comprise subcategories of this domain, with degree programs emphasizing security and privacy, software engineering, information technology leadership, health administration and applied research. Many of these programs result from associations between two or more departments, schools or colleges within the academic institution, with focus of competencies and topics dependent on the departments offering the program. Table 2. Concentration (%) of competencies for program domains within BMI. Program Domain Biomedical Informatics Clinical Informatics Consumer Health Informatics Health Informatics Imaging Informatics Nursing Informatics Public Health Informatics
Discipline 31 20 22 22 33 25 26
Theory and Methodology 28 24 22 22 38 21 26
1216
Technological Approach 17 22 9 21 24 13 13
Human and Social Context 23 34 46 36 5 41 35
Scope and Breadth of Discipline Biology, as the study of organisms and organic systems, is considered a foundation of all BMI domains3,5,9,18. Course content includes bioinformatics, pathology informatics, biochemistry, pathophysiology, biomedical physics, molecular informatics, genetics, molecular phylogenetics, neuroscience and genomics. Biotechnology, the application and use of biology to learn about living systems, is a required competency of bioinformatics16. Biotechnology focuses on introductory courses in biomedicine, electrical engineering, biomedical imaging, medical instrumentation and imaging informatics. Consumer, or personal, health represents development and evaluation of patient-centered technologies. Two programs offer degrees in consumer health, with one doctoral program in ‘Personal Health Informatics’ and one masters program in ‘Clinical Informatics and Patient-Centered Technologies’19,20. Key concepts of courses offered cover e-health, e-commerce, health services marketing and technology within developing countries. Healthcare system encompasses all perspectives of patient care, with courses covering the history of US healthcare systems, telemedicine and telehealth, pervasive technologies, integrated health networks, health organization and administration, electronic health records (EHR), health economics and healthcare delivery systems. Nursing represents the nurse as facilitator and promoter of health resources and references, and as communication mediator between clinicians, individuals, families and communities21 . Courses on this topic covers introductory and advanced foundations to nursing informatics. Population health studies the distribution of health outcomes amongst individuals and communities, evaluating the multi-level effects of environment, social structures, medical practice and other enabling systems on health22,23. Course titles such as ‘Behavioral Sciences in Public Health’, ‘Ecological Information Systems’ and ‘GIS Applications in Public Health’ demonstrate the broad reach of public health informatics. Translational and clinical research focuses on evaluation and interpretation of research into clinical practice, and is now considered a competency of all informatics domains17,24. Clinical trials, pharmacology and personalized care represent key themes of this topic within graduate education. Topics defined within the scope and breadth of discipline represent prerequisite and fundamental knowledge of BMI. Figure 3 highlights the differences in distribution of topics within this competency. Topics with the lowest concentration include nursing (2%), biotechnology (3%), and translational and clinical research (3.5%). Consumer health (6%), biology (9.5%) and population health (12%) comprise almost a third of discipline-related topics. However, healthcare system (64%) concepts comprise the majority of course content taught on the discipline of BMI. Discipline-related Topic Distribution Biology Biotechnology Consumer Health Healthcare System Nursing Population Health Translational/Clinical Research 0
20
40
60
80
100 %
Figure 3. Topic distribution within the competency of scope and breadth of discipline.
1217
Theory and Methodology Frameworks represent the varying approaches to framing and conceptualizing knowledge within BMI9. This topic includes analysis of service delivery models, e-health business models, decision models and psychosocial factors influencing health behavior. Knowledge management is the organization and administration of information, with an emphasis on the design, architecture, development and maintenance of information technologies25. We separate knowledge representation from knowledge management due to continued discussion on the inclusiveness of health information management as a discipline of BMI1,26. Knowledge representation is the abstraction and encoding of data into knowledge, with emphasis on metadata and ontologies to enable syntactic and semantic interoperability between information systems25. The Leadership in Health Information Technology for Health Professionals program offers a course exploring the analysis of standards, ethics and stakeholder values of knowledge representation and interoperability27. Methods and processes consist of qualitative, quantitative, mixed and computational subjects. Cognitive and social science methods include health promotion, health services program planning, decision analysis and q-methodology. Methods common to all sciences include empirical methods, quality appraisal and evaluation, and survey sampling. Population health methods include biostatistics, disease surveillance, epidemiology, survival analysis and outcomes analysis, stochastic methods for biology, and genetic analysis and discovery. Computer-based methods of simulation modeling, statistical learning, dynamical models and visualization are also categorized into this topic. Lastly, business process and methodology is included in this topic with courses on computer-supported collaborative work, lean six sigma and methods of performance measurement. Typology classifies knowledge into types, and consists of course content focused on existing medical terminologies, standards, classification systems and vocabularies within BMI. Theories form the principles of BMI, with an interdisciplinary cross-section of theoretical foundations of multiple sciences. This topic provides principles, foundations, perspectives and concepts of discipline-related topics of each domain. Course titles include ‘Perspectives in Environmental Health’, ‘Health Data: Theory and Practice’, ‘Principles of Clinical and Translational Science’, and ‘Theoretical Foundations in Personal Health Informatics’. The theories, methods and processes of BMI form the underlying principles to investigating and solving problems within the field. Figure 4 demonstrates the dominance of methods and processes within this competency. Frameworks of BMI (2.5%), knowledge representation (3.5%), typology (9%), theories (9.5%), and knowledge management (21%) comprise almost half of all topics taught within the theory and methodology competency. Methods and processes (54.5%) have the highest concentration within this competency. Theory and Methodology-related Topic Distribution Frameworks Knowledge Management Knowledge Representation Methods and Processes Theories Typology 0
20
40
60
80
Figure 4. Topic distribution within the competency of theory and methodology.
1218
100 %
Technological Approach Database systems courses within BMI graduate education focus on clinical analytics, data evaluation, data modeling for implementation, data structures and algorithms, data warehousing, business intelligence and database management, organization, file structure, performance tuning, querying, and security. Data mining of large data sets has gained recognition as an invaluable approach to investigation within many fields. Many aspects of data mining can be seen in current graduate programs, with courses on R, predictive analytics, web mining, pattern recognition, machine learning and aggregating health data. Imaging and signal analysis focuses on the processing and interpretation of images, speech and other medically relevant inputs. For graduate education, course content includes understanding, processing and analyzing images. Language processing is the processing of natural or spoken language into meaning. Applications of natural language processing include ontological development from EHRs and development of question and answering systems. Course titles include ‘Introduction to Medical Language Processing’, ‘Shallow Processing Techniques for Natural Language Processing’ and ‘Text Understanding and Information Retrieval’. Networking, security and privacy issues are an important aspect of patient-care, and consequently, have become a subcategory of health informatics with several programs offering a focus on security and privacy as a degree. This topic covers telecommunications, telehealth, telemedicine, software-as-a-service (SaaS), health information exchange issues, compliance, software and hardware assurance, cloud computing and HL7. Software engineering encompasses the many programming languages, design architectures, development strategies and applications of computer science and information technologies. Course content covers both existing and emerging trends in design and development of software systems, including XML, virtual reality, robotics, SQL, object-oriented design, as well as problem-oriented programming. System analysis and design covers courses on EHRs, clinical data acquisition, design, selection and management of health care systems, enterprise information systems, workflow mapping and reengineering. The theory and methodology of computer science intersects with BMI to define approaches taken to investigating and solving problems within this field. Figure 5 demonstrates the evolving technological approaches of BMI, with the emerging technologies of imaging and signal analysis (2.5%), language processing (3.5) and data mining (6%) representing the least amount of focus within this competency. System analysis and design (21%), database systems (23.5%), and networking, security and privacy (22.5%) comprise the main categories of technology applications to BMI. These results highlight computer science technologies with a history of long-term use as main topics of this competency. Technological Approach-related Topic Distribution Database Systems Data Mining Imaging/Signal Analysis Language Processing Networking, Security, Privacy Software Engineering System Analysis & Design 0
20
40
60
80
Figure 5. Topic distribution within the competency of technological approach.
1219
100 %
Human and Social Context Communication enables dissemination and application of results into practice, within the clinical, organizational or research environments. Course content focuses on communication through teaching, marketing, management, program planning, grant writing, scientific and strategic writing as well as publication and presentation skills. Decision support tools represent existing information systems and technologies developed to assist with the cognitive process in making a choice. Course titles include ‘Administrative and Clinical Decision Support Systems’ and ‘Decision Support and Intelligent Systems’. Evidence-based practice is the application of evidence-based research into medicine and clinical practice. Minimal course content exists on this topic, with evidence-based practice situated within the context of nursing or research design. Human computer interaction, as a topic within BMI, emphasizes the design and evaluation of technology based on it’s downstream influence within healthcare. Course content covers user interface design and development, usability, needs assessment, cognitive science, medical device strategy and clinician information needs. Legal, ethical and social issues encompass the implications of applying technology within healthcare, with course content focused on organizational context of healthcare computing, and policy development arising from health law. Organizational behavior and management covers a broad range of courses on multiple aspects of healthcare culture arising from an organizational context. Courses include management of health care practice, reimbursement systems, global enterprise, change leadership, financial accounting, project management, negotiation and conflict resolution, strategic planning and risk management. Research design and evaluation is the application of knowledge and skills to implementing, evaluating and then reiteratively improving upon solutions within the human and social context of healthcare and research. This topic covers courses in applied research projects, residency in health informatics, quality improvement, clinical trials and case studies in health information management. The topics of the competency of human and social context represent different perspectives of applying and practicing BMI within the healthcare environment. Evidence-based practice (1.5%) demonstrates the least focus of all topics within this competency. Communication (4.5%), decision-support tools (6.5%), and human-computer interaction (6.5) are amongst the next topics receiving low concentration of course content. Legal, ethical and social issues (13.5%) receive a moderate amount of human and social contextual content within graduate curricula. Research design and evaluation (32.5%), and organizational behavior and management (35%) comprise the majority of course content within this competency. Human and Social Context-related Topic Distribution Communication Decision Support Tools Evidence-Based Practice Human Computer Interaction Legal, Ethical, Social Issues Organizational Behavior/Mgmt Research Design & Evaluation 0
20
40
60
80
Figure 6. Topic distribution within the competency of human and social context.
1220
100 %
Discussion BMI graduate curricula must be in line with the ever-changing workforce needs. Labor market surveys for BMI indicate 63% of doctoral graduates conduct research after graduation, with 31% developing informatics tools or health information technology28. Graduates of advanced study in BMI continue to pursue multiple career paths not traditionally considered in BMI graduate education. Increasing numbers of training programs of health informatics, nursing informatics, public health informatics and consumer health informatics show the demand for training at multiple practice-levels. In fact, graduates of advanced study in BMI continue to pursue multiple career paths not traditionally considered in BMI graduate education. Minimal studies have mapped the needs of healthcare organizations and information systems departments to the educational requirements of informatics curricula7,17,22. Several studies on educational competencies and requirements within BMI have identified the breadth of BMI graduate education2,3,6-8,16,17,22. No recent work has examined the depth of curricula available in existing graduate education. The intent of this study is to identify existing and emerging topics not fully covered by existing degree programs. Our gap analysis demonstrates minimal coverage of the following topics: population health, translational research, knowledge representation, data mining, communication and evidence-based practice. Community Health Population health, with a focus on prevention, is the foundation for degree programs centered on public health informatics23. However, we found only three courses focused on community health amongst all BMI graduate education. The distinction of community health from population health relies on characteristics of a community. Improving community health requires a system-wide perspective on the multi-level effects of physical, social and cultural factors as well as the manner in which a community organizes itself29. Evaluation of health delivery models attuned to community care must incorporate measures of environment, primary care, specialty care, beliefs, insurance coverage, community health partners, care coordination and costs of care. With minimal evidence on methods to improve health outcomes and reduce health disparities amongst the medically underserved in our communities, we continue to look for mechanisms of change30. Increased training for the regional healthcare workforce helps to build a foundation for future efforts in improving public health at the community-level31. Education on health delivery models, mining administrative and managerial data sources, and cost improvement measures is required to train and support workforce within community medicine. Evaluation and maintenance of community medicine is one aspect of public health. Dissemination of results from improved systems of care, by removing regional information silos, is necessary in reducing health disparities nationally. Translational Research The ability to disseminate research results from community health into actionable health care delivery models relies increasingly on translational and clinical research methods. Federal government initiatives in research and healthcare workforce development place great emphasis on translating evidence-based knowledge into working and sustaining improvement in population health. Evidence-based practice is also gaining traction within the research areas of health services, comparative effectiveness and patient-centered outcomes, in an effort to evaluate and disseminate medical theory into practice. Subspecialties of care remain absent from graduate curricula as well. As systems of information become attune to specifics of specialty care, there is increased demand for training on subdomains of health informatics. Subdomains include behavioral health, dermatology, oncology, long-term care, obstetrics, gynecology, osteopathic and gerontology. Job listings for oncology informatics, pathology informatics and cheminformatics demonstrate the need to understand the requirements of these specializations and skillsets. For example, EHR systems differ significantly across specialties with specific requirements for security, privacy, reimbursement, compliance, documentation and workflows. Comprehension and application of specialty medical practice is essential to providing value within varying healthcare environments.
1221
Data Mining and Interoperability Data silos continue to exist, even within a single healthcare organization, with most informatics development completed by a mix of internal staff, external consultants and vendors. Integration and interoperability between multiple data sources becomes an issue of knowledge management and representation of such knowledge; thus emphasizing the need for data mining techniques. An expansive approach to education is required to enable data mining of knowledge organization systems. In fact, with the growing demand for analysis and evaluation of massive data sets within the medical domain, the emerging methods of data mining and language processing require increased instruction. Translation of such technologies to existing problems in health may assist in informing health economics and policy, as it has in existing data sources with genomics and social media-based communication. Dissemination Through Communication Communication as a mechanism for translation and dissemination is also largely absent from graduate education. Multiple avenues exist to communicate with peers, and increased focus on leveraging and expanding networks of researchers, working professionals, clinicians and the public must be part of the future of informatics education, research and practice. Many career pathways exist within BMI, but instruction on long-term employability is absent from current graduate curricula. The interdisciplinary field of BMI necessitates a continuous evaluation of mapping education to the needs of the healthcare workforce. Conclusion To understand the current status of BMI graduate education within the US, we conducted thematic analysis of 1722 courses from 205 unique degree programs. Our methodology was dependent on the accuracy and completeness of course descriptions listed on websites associated to each degree program. From this analysis, we identified gaps in accessibility as well as availability of course content supporting AMIA’s outline of core competencies for BMI graduate education. The states of Connecticut, Idaho, Maine, Nebraska, Nevada, New Hampshire, New Mexico, Oklahoma, Rhode Island, South Carolina, Vermont and Wyoming do not currently offer a single program with course content in BMI. Additionally, minimal course content is offered on community health, translational research, knowledge representation, data mining, communication for dissemination as well as evidence-based practice. Workforce skills required to support patient safety, accountable care, meaningful use and quality improvement may not be served by the availability and accessibility of current graduate education. In the future, we will evaluate how local workforce needs map to BMI graduate education competencies. References 1. 2.
Friedman CP. What informatics is and isn't. J Am Med Inform Assoc. 2013;20(2):224–6. Hersh W. Who are the informaticians? What we know and should know. J Am Med Inform Assoc. 2006;13(2):166–70. 3. Kampov-Polevoi J, Hemminger BM. Survey of biomedical and health care informatics programs in the United States. J Med Libr Assoc. 2010;98(2):178. 4. Kampov-Polevoi J, Hemminger BM. A curricula-based comparison of biomedical and health informatics programs in the USA. J Am Med Inform Assoc. 2011;18(2):195–202. 5. Huang QR. Competencies for graduate curricula in health, medical and biomedical informatics: a framework. Health Informatics J. 2007;13(2):89–103. 6. Hersh W. The health information technology workforce: estimations of demands and a framework for requirements. Appl Clin Inform. 2010;1(2):197–212. 7. Hersh W, Wright A, Garets D, Davis M, Pilolla L. Characterizing the health information technology workforce: Analysis from the HIMSS Analytics™ database. Portland, OR: http://www. billhersh. info/# workforce. 2008. 8. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med. 2010;49(2):105–120. 9. Kulikowski CA, Shortliffe EH, Currie LM, Elkin PL, Hunter LE, Johnson TR, et al. AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. J Am Med Inform Assoc. 2012;19(6):931-8. 10. Kennedy TJ, Lingard LA. Making sense of grounded theory in medical education. Med Educ. 2006;40(2):101–
1222
8. 11. Glaser BG, Strauss AL. The discovery of grounded theory: Strategies for qualitative research. Aldine de Gruyter; 1967. 12. Commission on Accreditation for Health Informatics and Information Management Education (CAHIIM) [Internet]. CAHIIM; 2013 [cited 2013 Mar 14]. Available from: http://www.cahiim.org/applyaccred_HI_grad.html 13. NLM's University-based Biomedical Informatics Research Training programs [Internet]. Bethesda (MD): U.S. National Library of Medicine; 2013 [cited 2013 Mar 14]. Available from: http://www.nlm.nih.gov/ep/GrantTrainInstitute.html 14. HIMSS. Healthcare Information and Management Systems Society (HIMSS) [Internet]. Chicago: HIMSS; 2013 [cited 2013 Mar 14]. Available from: http://www.himss.org 15. Paton C. Health Informatics Forum [Internet]. 2013 [cited 2013 Mar 14]. Available from: http://www.healthinformaticsforum.com/health-informatics-degrees-and-certificates 16. Hemminger BM, Losi T, Bauers A. Survey of bioinformatics programs in the United States. J Am Soc Inf Sci Tec. 2005;56(5):529–37. 17. Bernstam EV, Hersh WR, Johnson SB, Chute CG, Nguyen H, Sim I, et al. Synergies and distinctions between computational disciplines in biomedical research: perspective from the Clinical and Translational Science Award programs. Acad Med. 2009;84(7):964. 18. Johnson SB. A framework for the biomedical informatics curriculum. AMIA Annu Symp Proc. 2003:331–5. 19. Ph.D. in Personal Health Informatics [Internet]. Boston: Northeastern University; 2013 [cited 2013 Mar 12]. Available from: http://phi.ccs.neu.edu 20. Clinical Informatics & Patient-Centered Technologies [Internet]. Seattle: University of Washington; 2013 [cited 2013 Mar 12]. Available from: http://clinical-informatics.uw.edu 21. Alexander GL, Hull SC. Longitudinal Care Planning and Coordination in Long-Term and Post–Acute Care Settings. Comput Inform Nurs. LWW; 2012;30(9):454–5. 22. Soto Mas F, Hsu CE. Understanding public health informatics competencies for mid-tier public health practitioners-a Web-based Survey. Health Informatics J. 2012;18(1):66–76. 23. Joshi A, Perin DMP. Gaps in the Existing Public Health Informatics Training Programs: A Challenge to the Development of a Skilled Global Workforce. Perspect Health Inf Manag. 2012. 24. Sarkar IN. Biomedical informatics and translational medicine. J Transl Med. 2010;8(1):22. 25. Stock WG, Peters I, Weller K. Social semantic corporate digital libraries: Joining knowledge representation and knowledge management. Advances in Librarianship. Emerald Group Publishing Limited; 2010;32:137–58. 26. Hersh W. A stimulus to define informatics and health information technology. BMC Med Inform Decis Mak. 2009;9(1):24. 27. Leadership in Health Information Technology for Health Professionals Certificate [Internet]. Minneapolis: University of Minnesota School of Nursing; 2013 [cited 2013 Mar 14]. Available from: http://www.nursing.umn.edu/cphli/ 28. Dugan S, Saunders J. The Labor Market for Individuals with Advanced Degrees in Biomedical Informatics. 2011. 29. McKenzie J, Pinger R, Kotecki J. An introduction to community health. Jones & Bartlett Learning; 2011. 30. Jackson GL, Powers BJ, Chatterjee R, Bettger JP, Kemper AR, Hasselblad V, et al. The Patient-Centered Medical Home: A Systematic Review. Ann Intern Med. 2013;158(3):169–78. 31. Rosenthal EL, Brownstein JN, Rush CH, Hirsch GR, Willaert AM, Scott JR, et al. Community health workers: part of the solution. Health Aff. 2010;29(7):1338–42.
1223
