Trends in Publication of Nursing Informatics Research Hyeoneui Kim, RN, MPH, PhD, Lucila Ohno-Machado, MD, MBA, PhD, Janet Oh, BS, Xiaoqian Jiang, PhD Division of Biomedical Informatics, University of California San Diego, La Jolla, CA Abstract We analyzed 741 journal articles on nursing informatics published in 7 biomedical/nursing informatics journals and 6 nursing journals from 2005 to 2013 to begin to understand publication trends in nursing informatics research and identify gaps. We assigned a research theme to each article using AMIA 2014 theme categories and normalized the citation counts using time from publication. Overall, nursing informatics research covered a broad spectrum of research topics in biomedical informatics and publication topics seem to be well aligned with the high priority research agenda identified by the nursing informatics community. The research themes with highest volume of publication were Clinical Workflow and Human Factors, Consumer Informatics and Personal Health Records, and Clinical Informatics, for which an increasing trend in publication was noted. Articles on Informatics Education and Workforce Development; Data Mining, NLP, Information Extraction; and Clinical Informatics showed steady and high volume of citations. Introduction Information technologies were identified as key factors in achieving improved patient safety and quality of care and the published literature is starting to confirm this role(1–3). Biomedical informatics, including nursing informatics, is a fast moving field that is heavily influenced by healthcare policy and clinical practice. Professional and/or academic organizations have been dedicating substantial effort to facilitate nursing informatics research and training that address current needs. For example, many nursing education programs offer training opportunities specialized in nursing informatics and also mandate an introductory nursing informatics class to all students as part of their core curriculum(4–7). The Nursing Informatics Working Group (NIWG) at AMIA addresses various issues in policy, research, and educational aspects of nursing informatics. The NIWG also offers two award programs to recognize noteworthy research presented in the annual AMIA symposium to encourage participation of nurse scholars (8). Scientific journals are a venue for dissemination of research to a large community of readers. The proportion of peer-reviewed articles published in a certain area of research (and their citations) can provide interesting insights on topic trends in informatics(9,10). Although it may not constitute a perfect measure, the number of citations is considered a de facto standard for measuring the impact of a scientific publication. Based on this idea, we have previously analyzed the publication and citation volumes in biomedical informatics that were published relatively recently (2009-2012) in the J Amer Med Inform Assoc (JAMIA) to better understand research topic trends for biomedical informatics in general. We conducted a similar analysis to describe the trends in published nursing informatics research by analyzing the articles published on selected major informatics and nursing journals during the past 9 years. We also compared the active research areas reflected in these articles against the nursing informatics research agenda proposed by nursing informatics leaders(11–13). Through this analysis, we aimed at (1) checking the trajectory of published nursing informatics research for the past 9 years, and (2) identifying potential gaps in particular research areas. Background In 1993, the National Institute of Nursing Research (NINR) of National Institutes of Health (NIH) sponsored a group of nursing informatics scholars in the investigation of research needs in nursing informatics with the purpose of developing a nursing informatics research agenda(14). Healthcare and biomedical research are changing rapidly due to the availability of advanced technologies to collect and analyze large volumes of data. Many nursing informatics leaders have proposed an updated research agenda over the past decade. In 2007, the AMIA NIWG proposed comprehensive nursing informatics agenda in nursing practice, education, and research (13). A year later, Bakken et, al. identified the areas that nursing informatics research needed to expand further to better
805
accommodate fast moving biomedical sciences(12). In 2012, the Nursing Informatics International Research Network (NIIRN) conducted an international survey of 468 nursing informatics researchers across the globe soliciting opinions on research topics that should be prioritized(11). The research agenda identified in these works is summarized in Table 1. Recently, Carrington et, al. reported on informatics topics that were most actively researched, the types of research, and research settings by reviewing 69 peer-reviewed articles with a nurse as the first author, published between Aug 2011 and Aug 2012(15). They identified three broad topics of research: (1) clinical informatics research that deals with various clinical information applications such as Electronic Medical Records (EMR) and bar-code medication administration, (2) human factors such as human computer interaction and communication, and (3) data interoperability such as terminology/standardization and care transition/handoffs(15). This work provided an important snapshot view of the research trends in that given year and provided valuable insights on the status of nursing informatics research. AMIA-NIWG (2007)(13) • Secondary use of clinical data • Use of aggregated & deidentified data • Data privacy and confidentiality • Data security and related technical infrastructure • Population health • Development and use of standards
Table 1. Nursing informatics research agenda Bakken, et al. (2008)(12) • User friendly tools for data analysis visualization, and modeling • Evaluation methods of human and organizational factors on health IT • Consumer empowerment • Use of genomics and environmental data • Reengineering of nursing practice
NIIRN (2012)*(11)
• Clinical information systems that provide real-time feedback • Evaluation of the impact of health IT system on patient outcome • Nursing decision support systems • Evaluation of the impact of health IT system on nursing workflow • Management of nursing data for research and patient care • Training nurses in health IT • Identifying nursing outcomes that are important to patients
* Only the top 7 highest priority items out of 20 are listed. Materials and Methods Article collection We retrieved articles relevant to nursing informatics published between 2005 and 2013 using ISI’s Web of Science(16). Here, the “articles relevant to nursing informatics” was defined according to the disciplines covered by the selected journals, keywords found in the articles, and authors’ affiliations. Articles published in nursing informatics journals were considered relevant. Articles published in general biomedical informatics journals were considered relevant when at least one of the authors was affiliated with a nursing institution (e.g., nursing schools, nursing research institutes, centers of nursing excellence, etc.). Additionally, articles published in non-informatics nursing journals were considered relevant if they contained the keyword “informatics” in the title or abstract. Nursing informatics research is published in a large number of biomedical informatics and/or engineering journals. We included one nursing informatics journal (CIN: Computers, Informatics, Nursing) and seven general biomedical informatics journals (Journal of Medical Internet Research, Journal of American Medical Informatics Association, Medical Decision Making, International Journal of Medical Informatics, Journal of Biomedical Informatics, BMC Medical Informatics and Decision Making, Method of Informatics in Medicine) in this analysis. These journals encompass a wide range of informatics topics and are among the top 20 journals with highest 5-year impact factors in the category “Medical Informatics” in ISI’s Web of Science (16). Four1 of these eight journals were also recognized in a prior work as the journals in which the most nursing informatics works were published (15). As many nursing informatics scholars publish informatics relevant works on non-informatics journals, we also included six nursing journals (i.e., International Journal of Nursing Studies, Journal of Advanced Nursing, Nursing Research, Journal of Nursing Scholarship, Nursing Outlook, Journal of Nursing Administration) that publish 1
CIN: Computers, Informatics, Nursing; Journal of American Medical Informatics Association; International Journal of Medical Informatics; Journal of Biomedical Informatics
806
F
general clinical nursing research as well as administrative, and managerial topics. These journals are among the top 20 nursing journals with highest 5-year impact scores under the category “Nursing” of ISI’s Web of Science(16). This journal selection approach undoubtedly has limitations, which are described in the discussion section. We retrieved citation information of selected articles from the ISI’s Web of Science(16) using its citation report function. A total of 741 articles were included in this analysis, which excluded white papers, opinion papers, duplicated articles, and articles without citation reports. The largest number of articles (N=406) was collected from the nursing informatics journal, 306 from biomedical informatics journals, and 29 from nursing journals. The paper selection process is summarized in Figure 1. Citation score normalization The citation report obtained from the ISI’s Web of Science(16) shows the annual citation counts of an article for the indicated time period. Using the same method reported in the prior work(9,10) we produced normalized citation scores to each paper to minimize the bias introduced by the differences in publication dates. For instance, an article published in January of 2006 has more chances of being read and cited than an article published in December of 2011 as it has been available to research communities for a longer period of time. Therefore we used a unit of measure that reflects the number of citation per month (CPM) since publication, as adopted in the prior work (9,10). The CPM scores were calculated separately for each year (i.e., annual CPM = number of citation in a given year / number of months an article was available in a given year) as well as for the entire period (i.e., overall CPM = total number of citations / total Figure 1. Article selection process number of months an article was available). The two calculations were performed to account for the fact that articles that are old accrue a relatively small number of citations per month after several years of publication, and articles that are recent also accrue a relatively small number of citations in the first two years after publication. Hence articles published in the middle of the accounted period (i.e, 2009-2010) would account for higher overall CPMs than articles at the extremes. However, they should result in comparable annual CPM when the number of years after publication is taken into account. Research theme assignment We assigned a research topic category to each article using the AMIA themes proposed for the 2014 annual symposium(17). Although the articles provide a list of keywords and/or are indexed with the terms from Medical Subject Heading (MeSH), they may not provide a single category label that represents the main topic of the study with sufficient expressivity for subdomains of informatics. We decided to use the AMIA 2014 themes after investigating various potential alternatives, such as chapter headings from biomedical informatics textbooks and subjects listed in informatics training curricula(18,19). Although AMIA themes may focus on the contemporary topics of biomedical informatics research, they were deemed viable options for this work, as they are expressive labels designed to capture a single main topic of a research work for the purposes of directing a diverse set of attendees to the presentations that are more relevant to their interests. We also developed annotation rules and a process shown in Figure 2 to maximize annotation consistency and accuracy. The theme categories that were deemed straightforward to use (i.e., Public Health Informatics and Biosurveillance, Translational Bioinformatics and Bioinformatics, Global eHealth, Imaging Informatics) are not shown in Figure 2 to simplify the display of the process. In addition, we did not use the theme category Meaningful Use as it can directly relate to many other informatics themes and was not relevant to older articles.
807
Two authors (HK, XJ) independently annotated the main research topics of randomly selected 100 articles using the AMIA 2014 themes. The annotation results were collaboratively reviewed and the discrepancies were resolved by reaching consensus. Also, the theme assignment rules and process were refined as necessary. One author (HK) who is trained in nursing informatics assigned themes to the remaining 641 articles following the refined rules and process. Metadata for the 741 articles with themes are available at https://idash-data.ucsd.edu/folder/504.
Figure 2. Topic assignment process Trend analysis We generated descriptive statistics on distributions of the research topics and their citation scores. We also tested whether these distributions differ significantly among journal types and whether they changed significantly over the years. SPSS (version 22) and Matlab were used to analyze the data. Results Among 741 articles collected, 701 were research articles and 40 were review articles. A total of 117 articles reported on work that targeted specific clinical information systems. The most frequently studied areas referred to Electronic Medical Records (EMR, N=61), Clinical Decision Support System (CDSS, N=33), Computerized Provider Order Entry (CPOE, N=10), and bar-coding medication administration (N=10).
808
Figure 3 Annual article volume and Citation per Month (CPM), and overall CPM by Themes
809
Distribution of research themes Most of the themes were utilized in the categorization of articles, except for the themes Imaging Informatics and Translational Bioinformatics and Biomedicine for which there were no articles. No article was left unassigned. The research theme most frequently appearing for the past 9 years was Clinical Workflow and Human Factors. Global eHealth was the least frequent (Figure 3). We analyzed the trend of publication frequencies for each research theme over the years using a parametric method. The article frequencies were used as the predictor variables and the publication year was used as the response variable to fit a linear regression model. Two parameters (constant and slope) were calculated and we conducted a t-test to check whether they were statistically significant. In Figure 3, the starred (*) themes are the ones that show statistically significant positive trends (i.e., increment of the article counts). Publications focusing on Consumer Informatics and PHR and Mobile Health have been continuously growing. In addition to these two themes, Clinical Informatics and Clinical Workflow and Human Factors are the areas where the high volume of nursing informatics articles has been published for the past 3 years. Also it is noteworthy that the nursing informatics articles on Biomedical Data Visualization and Global eHealth started showing up relatively recently. The articles published in informatics journals represented the broadest theme areas. Two most frequent themes from the nursing informatics journal were Informatics in Health Professional Education, and Clinical Workflow and Human Factor. Consumer Informatics and PHRs, and Clinical Workflow and Human Factor are the themes most frequently appearing in general biomedical informatics journals. Half of the articles from nursing journals were on Informatics Education and Workforce Development (Figure 4). The differences in theme distributions among journal types were statistically significant when tested with the Pearson Chi-Square test (p
805
accommodate fast moving biomedical sciences(12). In 2012, the Nursing Informatics International Research Network (NIIRN) conducted an international survey of 468 nursing informatics researchers across the globe soliciting opinions on research topics that should be prioritized(11). The research agenda identified in these works is summarized in Table 1. Recently, Carrington et, al. reported on informatics topics that were most actively researched, the types of research, and research settings by reviewing 69 peer-reviewed articles with a nurse as the first author, published between Aug 2011 and Aug 2012(15). They identified three broad topics of research: (1) clinical informatics research that deals with various clinical information applications such as Electronic Medical Records (EMR) and bar-code medication administration, (2) human factors such as human computer interaction and communication, and (3) data interoperability such as terminology/standardization and care transition/handoffs(15). This work provided an important snapshot view of the research trends in that given year and provided valuable insights on the status of nursing informatics research. AMIA-NIWG (2007)(13) • Secondary use of clinical data • Use of aggregated & deidentified data • Data privacy and confidentiality • Data security and related technical infrastructure • Population health • Development and use of standards
Table 1. Nursing informatics research agenda Bakken, et al. (2008)(12) • User friendly tools for data analysis visualization, and modeling • Evaluation methods of human and organizational factors on health IT • Consumer empowerment • Use of genomics and environmental data • Reengineering of nursing practice
NIIRN (2012)*(11)
• Clinical information systems that provide real-time feedback • Evaluation of the impact of health IT system on patient outcome • Nursing decision support systems • Evaluation of the impact of health IT system on nursing workflow • Management of nursing data for research and patient care • Training nurses in health IT • Identifying nursing outcomes that are important to patients
* Only the top 7 highest priority items out of 20 are listed. Materials and Methods Article collection We retrieved articles relevant to nursing informatics published between 2005 and 2013 using ISI’s Web of Science(16). Here, the “articles relevant to nursing informatics” was defined according to the disciplines covered by the selected journals, keywords found in the articles, and authors’ affiliations. Articles published in nursing informatics journals were considered relevant. Articles published in general biomedical informatics journals were considered relevant when at least one of the authors was affiliated with a nursing institution (e.g., nursing schools, nursing research institutes, centers of nursing excellence, etc.). Additionally, articles published in non-informatics nursing journals were considered relevant if they contained the keyword “informatics” in the title or abstract. Nursing informatics research is published in a large number of biomedical informatics and/or engineering journals. We included one nursing informatics journal (CIN: Computers, Informatics, Nursing) and seven general biomedical informatics journals (Journal of Medical Internet Research, Journal of American Medical Informatics Association, Medical Decision Making, International Journal of Medical Informatics, Journal of Biomedical Informatics, BMC Medical Informatics and Decision Making, Method of Informatics in Medicine) in this analysis. These journals encompass a wide range of informatics topics and are among the top 20 journals with highest 5-year impact factors in the category “Medical Informatics” in ISI’s Web of Science (16). Four1 of these eight journals were also recognized in a prior work as the journals in which the most nursing informatics works were published (15). As many nursing informatics scholars publish informatics relevant works on non-informatics journals, we also included six nursing journals (i.e., International Journal of Nursing Studies, Journal of Advanced Nursing, Nursing Research, Journal of Nursing Scholarship, Nursing Outlook, Journal of Nursing Administration) that publish 1
CIN: Computers, Informatics, Nursing; Journal of American Medical Informatics Association; International Journal of Medical Informatics; Journal of Biomedical Informatics
806
F
general clinical nursing research as well as administrative, and managerial topics. These journals are among the top 20 nursing journals with highest 5-year impact scores under the category “Nursing” of ISI’s Web of Science(16). This journal selection approach undoubtedly has limitations, which are described in the discussion section. We retrieved citation information of selected articles from the ISI’s Web of Science(16) using its citation report function. A total of 741 articles were included in this analysis, which excluded white papers, opinion papers, duplicated articles, and articles without citation reports. The largest number of articles (N=406) was collected from the nursing informatics journal, 306 from biomedical informatics journals, and 29 from nursing journals. The paper selection process is summarized in Figure 1. Citation score normalization The citation report obtained from the ISI’s Web of Science(16) shows the annual citation counts of an article for the indicated time period. Using the same method reported in the prior work(9,10) we produced normalized citation scores to each paper to minimize the bias introduced by the differences in publication dates. For instance, an article published in January of 2006 has more chances of being read and cited than an article published in December of 2011 as it has been available to research communities for a longer period of time. Therefore we used a unit of measure that reflects the number of citation per month (CPM) since publication, as adopted in the prior work (9,10). The CPM scores were calculated separately for each year (i.e., annual CPM = number of citation in a given year / number of months an article was available in a given year) as well as for the entire period (i.e., overall CPM = total number of citations / total Figure 1. Article selection process number of months an article was available). The two calculations were performed to account for the fact that articles that are old accrue a relatively small number of citations per month after several years of publication, and articles that are recent also accrue a relatively small number of citations in the first two years after publication. Hence articles published in the middle of the accounted period (i.e, 2009-2010) would account for higher overall CPMs than articles at the extremes. However, they should result in comparable annual CPM when the number of years after publication is taken into account. Research theme assignment We assigned a research topic category to each article using the AMIA themes proposed for the 2014 annual symposium(17). Although the articles provide a list of keywords and/or are indexed with the terms from Medical Subject Heading (MeSH), they may not provide a single category label that represents the main topic of the study with sufficient expressivity for subdomains of informatics. We decided to use the AMIA 2014 themes after investigating various potential alternatives, such as chapter headings from biomedical informatics textbooks and subjects listed in informatics training curricula(18,19). Although AMIA themes may focus on the contemporary topics of biomedical informatics research, they were deemed viable options for this work, as they are expressive labels designed to capture a single main topic of a research work for the purposes of directing a diverse set of attendees to the presentations that are more relevant to their interests. We also developed annotation rules and a process shown in Figure 2 to maximize annotation consistency and accuracy. The theme categories that were deemed straightforward to use (i.e., Public Health Informatics and Biosurveillance, Translational Bioinformatics and Bioinformatics, Global eHealth, Imaging Informatics) are not shown in Figure 2 to simplify the display of the process. In addition, we did not use the theme category Meaningful Use as it can directly relate to many other informatics themes and was not relevant to older articles.
807
Two authors (HK, XJ) independently annotated the main research topics of randomly selected 100 articles using the AMIA 2014 themes. The annotation results were collaboratively reviewed and the discrepancies were resolved by reaching consensus. Also, the theme assignment rules and process were refined as necessary. One author (HK) who is trained in nursing informatics assigned themes to the remaining 641 articles following the refined rules and process. Metadata for the 741 articles with themes are available at https://idash-data.ucsd.edu/folder/504.
Figure 2. Topic assignment process Trend analysis We generated descriptive statistics on distributions of the research topics and their citation scores. We also tested whether these distributions differ significantly among journal types and whether they changed significantly over the years. SPSS (version 22) and Matlab were used to analyze the data. Results Among 741 articles collected, 701 were research articles and 40 were review articles. A total of 117 articles reported on work that targeted specific clinical information systems. The most frequently studied areas referred to Electronic Medical Records (EMR, N=61), Clinical Decision Support System (CDSS, N=33), Computerized Provider Order Entry (CPOE, N=10), and bar-coding medication administration (N=10).
808
Figure 3 Annual article volume and Citation per Month (CPM), and overall CPM by Themes
809
Distribution of research themes Most of the themes were utilized in the categorization of articles, except for the themes Imaging Informatics and Translational Bioinformatics and Biomedicine for which there were no articles. No article was left unassigned. The research theme most frequently appearing for the past 9 years was Clinical Workflow and Human Factors. Global eHealth was the least frequent (Figure 3). We analyzed the trend of publication frequencies for each research theme over the years using a parametric method. The article frequencies were used as the predictor variables and the publication year was used as the response variable to fit a linear regression model. Two parameters (constant and slope) were calculated and we conducted a t-test to check whether they were statistically significant. In Figure 3, the starred (*) themes are the ones that show statistically significant positive trends (i.e., increment of the article counts). Publications focusing on Consumer Informatics and PHR and Mobile Health have been continuously growing. In addition to these two themes, Clinical Informatics and Clinical Workflow and Human Factors are the areas where the high volume of nursing informatics articles has been published for the past 3 years. Also it is noteworthy that the nursing informatics articles on Biomedical Data Visualization and Global eHealth started showing up relatively recently. The articles published in informatics journals represented the broadest theme areas. Two most frequent themes from the nursing informatics journal were Informatics in Health Professional Education, and Clinical Workflow and Human Factor. Consumer Informatics and PHRs, and Clinical Workflow and Human Factor are the themes most frequently appearing in general biomedical informatics journals. Half of the articles from nursing journals were on Informatics Education and Workforce Development (Figure 4). The differences in theme distributions among journal types were statistically significant when tested with the Pearson Chi-Square test (p
