This article was downloaded by: [Georgian Court University] On: 21 February 2015, At: 08:27 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Medical Reference Services Quarterly Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wmrs20
Development and Examination of a Rubric for Evaluating Point-of-Care Medical Applications for Mobile Devices a
b
c
Robyn Butcher , Martin MacKinnon , Kathleen Gadd & Denise d
LeBlanc-Duchin a
Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada b
Department of Nephrology, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada c
Click for updates
Library Services, Horizon Health Network, Miramichi Regional Hospital, Miramichi, New Brunswick, Canada d
Research Services, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada Published online: 22 Jan 2015.
To cite this article: Robyn Butcher, Martin MacKinnon, Kathleen Gadd & Denise LeBlanc-Duchin (2015) Development and Examination of a Rubric for Evaluating Point-of-Care Medical Applications for Mobile Devices, Medical Reference Services Quarterly, 34:1, 75-87, DOI: 10.1080/02763869.2015.986794 To link to this article: http://dx.doi.org/10.1080/02763869.2015.986794
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [Georgian Court University] at 08:27 21 February 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Medical Reference Services Quarterly, 34(1):75–87, 2015 Published with license by Taylor & Francis ISSN: 0276-3869 print=1540-9597 online DOI: 10.1080/02763869.2015.986794
Development and Examination of a Rubric for Evaluating Point-of-Care Medical Applications for Mobile Devices
Downloaded by [Georgian Court University] at 08:27 21 February 2015
ROBYN BUTCHER Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada
MARTIN MACKINNON Department of Nephrology, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada
KATHLEEN GADD Library Services, Horizon Health Network, Miramichi Regional Hospital, Miramichi, New Brunswick, Canada
DENISE LEBLANC-DUCHIN Research Services, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada
The rapid development and updates of mobile medical resource applications (apps) highlight the need for an evaluation tool to assess the content of these resources. The purpose of the study was to develop and test a new evaluation rubric for medical resource apps. The evaluation rubric was designed using existing literature and through a collaborative effort between a hospital and an academic librarian. Testing found scores ranging from 23% to 88% for the apps. The evaluation rubric proved able to distinguish levels of quality within each content component of the apps, demonstrating potential for standardization of medical resource app evaluations.
# Robyn Butcher, Martin MacKinnon, Kathleen Gadd, and Denise LeBlanc-Duchin Received: September 12, 2014; Revised: October 21, 2014; Accepted: November 3, 2014. Address correspondence to Robyn Butcher, Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, 500 University Avenue, Toronto, ON, Canada M5G 1V7. E-mail: [email protected] 75
76
R. Butcher et al.
KEYWORDS Clinical decisions, content quality, diagnosis and treatment, evaluation rubric, medical resources, mobile devices, mobile medical apps, point-of-care tools, usability
Downloaded by [Georgian Court University] at 08:27 21 February 2015
INTRODUCTION Library services, students in medical education, and practicing physicians are moving away from traditional medical resources (i.e., books, print journals, etc.), and moving toward the use of reference platforms for mobile devices.1 Studies have shown that more than 70% of physicians use smartphones.2,3 Handheld devices such as smartphones, iPod Touches, iPads, and tablets offer the advantages of being portable, immediately accessible, and able to download applications (apps). Medical reference apps are self-contained software applications which can store medical reference information, and once downloaded, can usually be accessed offline. If the reference information on medical apps is comparable to desktop point-of-care tools, use of apps could be beneficial to medical staff, students and residents, and for patient care. The U.S. Food and Drug Administration (FDA) distinguishes mobile medical apps as regulated medical devices based on their intended use which includes, among others, diagnosis of disease, treatment, or prevention of disease.4 Use of medical apps for these reasons in health care, particularly if integrated with patient information, needs research that evaluates clinical outcomes and potential consequences to help guide regulations that will protect and benefit patients.5 Other medical apps, such as those used to access electronic reference material, have no regulatory requirements.4 Ensuring the quality of the information provided in medical apps is left up to the user; however, a recent survey of 416 nurses and doctors in the United Kingdom noted that, despite 77% of them using apps, only 23% did some sort of informal evaluation of the apps’ credibility.3 To date, there is no formal evaluation process to ensure the sources of medical information available on these resource apps are peer reviewed for quality, validity, and reliability.3 Unlike resources such as Cochrane Reviews or peer-reviewed journal articles, there is no standard process to develop and publish a medical resource app. There are articles on individual apps6,7 and some general reviews of using mobile technology in health care8–13 or the accuracy of medical calculation apps.14 These articles evaluate various types of apps, usually for usability; most are lacking in information regarding the quality of the resources available on the recommended apps. Certain criteria must be met in order for Apple, Android, or Blackberry to sell a medical app in their virtual stores. These criteria include technical, design, and content specifications. Although the content specifications may include sources of cited references and authorship, they are not always readily available to the
Downloaded by [Georgian Court University] at 08:27 21 February 2015
Rubric for Evaluating Point-of-Care Applications
77
public and need not report any validation process prior to use by medical professionals. To add to this lack of a validation process, there is currently no standardized evaluation tool that could effectively handle the constant release of new mobile medical resource apps or updates to existing apps. Although evaluating the quality of the sources of the information on medical reference apps is of the utmost importance, preference with respect to usability is also a factor. Students and residents on internal medicine clinical teaching units and the cardiac care units are involved in the care of hospitalized patients. They also may spend significant time in the emergency room assessing a variety of patients with medical issues. During these times, they typically need to formulate clinical questions, including but not limited to disease descriptions, laboratory parameters, drug dosing, and drug interactions. In many medical teaching units, the students use mobile medical apps to answer those types of questions.15,16 The current study presents a model of collaboration between a hospital library and practitioners and consists of two parts. The first and primary focus of the study was the development and testing of an evaluation tool for medical reference apps. It was expected that competing medical apps would have different features and some would score higher on the evaluation tool and consequently be more suitable for use as a mobile point-of-care reference tool. However, these higher-scoring apps may not be the best for usability. Medical apps that include information regarding the quality of the resources could be more difficult to use. The second part of the study was to evaluate the four highest scoring apps for usability with a small sample of medical students and residents.
PART 1: DEVELOPMENT AND TESTING OF THE EVALUATION RUBRIC Methods DEVELOPMENT OF THE EVALUATION RUBRIC The evaluation tool was devised following literature searches and discussions regarding components for assessment of the sources of medical app information. There have not been any other studies which evaluate the information sources of medical apps. The numerical and relative value of each facet of the evaluation was based on scoring of similar evaluations for desktop point-of-care tools.17,18 The levels of evidence sources were based on the evidence-based pyramid from the University of Washington.19 The tool, in the form of a rubric, and the method of evaluation were also sent to medical librarians across Canada for feedback. TESTING THE EVALUATION RUBRIC Six mobile medical resource apps were chosen to test the new evaluation rubric. At the time of the study (fall 2011), these were the only six medical reference
Downloaded by [Georgian Court University] at 08:27 21 February 2015
78
R. Butcher et al.
apps available for the iOS operating system in North America that included both treatment and diagnosis information: Epocrates Essentials, DynaMed, PEPID Primary Care Plus, First Consult, Lexi Complete, and Medscape. Year-long subscriptions to these apps (one per device) were obtained either through purchasing or unconditional donation by the publishers. One librarian evaluated all six apps using the evaluation rubric. To assign scores, the information used had to be included within the app itself or available on the app’s publicly accessible website. Companies were not contacted for further information if the answers were not readily accessible. This method was chosen to approximate an average user’s approach. A second librarian independently evaluated the same apps. Screen shots were kept of the information sought. Scores for the mobile medical resource app evaluations using the rubric were calculated from a total score of 100%.
Results DEVELOPMENT OF THE EVALUATION RUBRIC The evaluation rubric is outlined in Table 1. The evaluation rubric was divided into three categories: content, transparency, and evidence. The weighing of each category was based on relative importance where the scoring for content totaled 20%, transparency totaled 20%, and evidence totaled 60% of the final score. The evidence category was given more weight because determining the quality, validity, and reliability of the medical resources for each app was the primary purpose of the evaluation tool. Content represented the volume of material available on the app. It was measured by the number of topics and the frequency of updates. Transparency referred to the openness of information. It was measured by references cited and authors listed. For transparency, it was not the number of references or authors listed that was significant, but rather if they were listed. Evidence was associated with quality of the sources used to create the disease monographs. It was measured by presence of peer review, evidence sources (i.e., textbooks, journals, and systematic reviews), evidence grade, and explanation of evidence grade. In terms of the peer review, just the disease monographs themselves were examined and whether they had been peer reviewed. The sources that created the disease monographs were not examined. The time to examine every reference for each disease monograph to ensure that it was peer reviewed would have been prohibitive. Yet peer review is the minimum for a monograph to be considered academic or evidence-based. Since a more rigorous evaluation of peer review was not possible, the weight of this component in the evaluation was relatively low. Evidence grade referred to a scale the app might have used to rate the level of evidence. Different apps used different scales. For example, one might have used a scale of one to three; another might have used letter
79
60
20
Transparency
Evidence
20
Content
%
12 30
12 6
Evidence source
Evidence grade
Evidence grade explanation
8
Peer review
Authorship
12
10
Updated
References
10
%
Number of topics
TABLE 1 Evaluation Rubric
Does this app provide an explanation for its grading system either in the app or on their website?
Does this app use an evidence grading system?
What is the method of evidence collection the app uses to create the information in their topics? (Look for the method on the website)
Does the app indicate if it is peer-reviewed either in the app or on their website?
Do the topics list personal authors, contributors, editors, or committee name?
Do topics provide references for their information either in text or at the end of the monograph?
How frequently are the topics in this app updated?
How many topics does this app indicate it contains on their website?
Yes No
Yes No
The editors or content authors have done a systematic review of the literature The editors or content authors have done a review of evidence guidelines or health technology assessments Other reviews of the literature Journals or government documents Textbooks or created internally No explanation
Yes No
Yes No
Yes No
More than once a week Weekly More than once a month Monthly Quarterly Annually or less No number given
5000þ 4000–4999 3000–3999 2000–2999 1000–1999 999 or No number given
Downloaded by [Georgian Court University] at 08:27 21 February 2015
6 0
12 0
18 12 6 0
24
30
12 0
8 0
12 0
10 8 6 4 2 1 0
10 8 6 4 2 1 0
Points
80
R. Butcher et al.
grades. No one scale was necessarily superior to another. There was an expectation that the app would explain how its scale was derived, and as such, scored separately as explanation of evidence grade.
Downloaded by [Georgian Court University] at 08:27 21 February 2015
TESTING THE EVALUATION RUBRIC The evaluations of the six medical resource apps from the two librarians had few discrepancies in the scores and once the screen shots were reviewed, they were quickly resolved. Each app evaluation took approximately 45 minutes. Results of the six apps using the rubric are in Table 2. The content category showed variation in number of topics, from less than 1,000 to more than 5,000. The apps were nearly equivalent in transparency. All apps included references and nearly all had authors listed. Scores for evidence ranged from very poor (6=60) to perfect (60=60). The total scores out of 100% ranged from 23% to 88%. Only four apps scored over 50%: DynaMed (88%), First Consult (83%), Medscape (72%), and PEPID Primary Care Plus (54%). As the highest-scoring apps, these four were chosen to be evaluated for usability by the medical students and residents in Part 2 of the study.
PART 2: USABILITY OF THE FOUR HIGHEST-SCORING MEDICAL APPS Methods Approximately 53 residents=students complete rotations in a given academic year at the Saint John Regional Hospital in the cardiac care unit or medical teaching unit. All potential students on these rotations were sent an e-mail invitation, describing the study, approximately two weeks prior to the beginning of their rotations. Participants who voluntarily agreed to participate signed an informed consent form. The study and consent was approved by the Horizon Health Network Ethics Review Board (Reference number: 20111640). The participants were asked to utilize and evaluate each of the selected mobile medical reference apps for a one-week period. iPod Touch handheld devices were provided preloaded with software to be used each week. The iPod Touch was selected because it allows for the download of apps without cellular abilities and it was not as large as tablets and easier for the participants to carry. The participants were asked to utilize the apps to help answer questions and inform clinical care. They were not restricted to one app but were encouraged to use the weekly app as the first go-to point for information. At the beginning of the study, the participants were asked to fill in an entrance questionnaire to collect general demographic information and to determine previous experience with handheld devices and apps. At the
81
Lexi complete
Medscape
First consult
Epocrates essentials
DynaMed
PEPID primary care plus
Application
Monthly 4
Medical Reference Services Quarterly Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wmrs20
Development and Examination of a Rubric for Evaluating Point-of-Care Medical Applications for Mobile Devices a
b
c
Robyn Butcher , Martin MacKinnon , Kathleen Gadd & Denise d
LeBlanc-Duchin a
Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada b
Department of Nephrology, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada c
Click for updates
Library Services, Horizon Health Network, Miramichi Regional Hospital, Miramichi, New Brunswick, Canada d
Research Services, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada Published online: 22 Jan 2015.
To cite this article: Robyn Butcher, Martin MacKinnon, Kathleen Gadd & Denise LeBlanc-Duchin (2015) Development and Examination of a Rubric for Evaluating Point-of-Care Medical Applications for Mobile Devices, Medical Reference Services Quarterly, 34:1, 75-87, DOI: 10.1080/02763869.2015.986794 To link to this article: http://dx.doi.org/10.1080/02763869.2015.986794
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [Georgian Court University] at 08:27 21 February 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Medical Reference Services Quarterly, 34(1):75–87, 2015 Published with license by Taylor & Francis ISSN: 0276-3869 print=1540-9597 online DOI: 10.1080/02763869.2015.986794
Development and Examination of a Rubric for Evaluating Point-of-Care Medical Applications for Mobile Devices
Downloaded by [Georgian Court University] at 08:27 21 February 2015
ROBYN BUTCHER Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada
MARTIN MACKINNON Department of Nephrology, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada
KATHLEEN GADD Library Services, Horizon Health Network, Miramichi Regional Hospital, Miramichi, New Brunswick, Canada
DENISE LEBLANC-DUCHIN Research Services, Horizon Health Network, Saint John Regional Hospital, Saint John, New Brunswick, Canada
The rapid development and updates of mobile medical resource applications (apps) highlight the need for an evaluation tool to assess the content of these resources. The purpose of the study was to develop and test a new evaluation rubric for medical resource apps. The evaluation rubric was designed using existing literature and through a collaborative effort between a hospital and an academic librarian. Testing found scores ranging from 23% to 88% for the apps. The evaluation rubric proved able to distinguish levels of quality within each content component of the apps, demonstrating potential for standardization of medical resource app evaluations.
# Robyn Butcher, Martin MacKinnon, Kathleen Gadd, and Denise LeBlanc-Duchin Received: September 12, 2014; Revised: October 21, 2014; Accepted: November 3, 2014. Address correspondence to Robyn Butcher, Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, 500 University Avenue, Toronto, ON, Canada M5G 1V7. E-mail: [email protected] 75
76
R. Butcher et al.
KEYWORDS Clinical decisions, content quality, diagnosis and treatment, evaluation rubric, medical resources, mobile devices, mobile medical apps, point-of-care tools, usability
Downloaded by [Georgian Court University] at 08:27 21 February 2015
INTRODUCTION Library services, students in medical education, and practicing physicians are moving away from traditional medical resources (i.e., books, print journals, etc.), and moving toward the use of reference platforms for mobile devices.1 Studies have shown that more than 70% of physicians use smartphones.2,3 Handheld devices such as smartphones, iPod Touches, iPads, and tablets offer the advantages of being portable, immediately accessible, and able to download applications (apps). Medical reference apps are self-contained software applications which can store medical reference information, and once downloaded, can usually be accessed offline. If the reference information on medical apps is comparable to desktop point-of-care tools, use of apps could be beneficial to medical staff, students and residents, and for patient care. The U.S. Food and Drug Administration (FDA) distinguishes mobile medical apps as regulated medical devices based on their intended use which includes, among others, diagnosis of disease, treatment, or prevention of disease.4 Use of medical apps for these reasons in health care, particularly if integrated with patient information, needs research that evaluates clinical outcomes and potential consequences to help guide regulations that will protect and benefit patients.5 Other medical apps, such as those used to access electronic reference material, have no regulatory requirements.4 Ensuring the quality of the information provided in medical apps is left up to the user; however, a recent survey of 416 nurses and doctors in the United Kingdom noted that, despite 77% of them using apps, only 23% did some sort of informal evaluation of the apps’ credibility.3 To date, there is no formal evaluation process to ensure the sources of medical information available on these resource apps are peer reviewed for quality, validity, and reliability.3 Unlike resources such as Cochrane Reviews or peer-reviewed journal articles, there is no standard process to develop and publish a medical resource app. There are articles on individual apps6,7 and some general reviews of using mobile technology in health care8–13 or the accuracy of medical calculation apps.14 These articles evaluate various types of apps, usually for usability; most are lacking in information regarding the quality of the resources available on the recommended apps. Certain criteria must be met in order for Apple, Android, or Blackberry to sell a medical app in their virtual stores. These criteria include technical, design, and content specifications. Although the content specifications may include sources of cited references and authorship, they are not always readily available to the
Downloaded by [Georgian Court University] at 08:27 21 February 2015
Rubric for Evaluating Point-of-Care Applications
77
public and need not report any validation process prior to use by medical professionals. To add to this lack of a validation process, there is currently no standardized evaluation tool that could effectively handle the constant release of new mobile medical resource apps or updates to existing apps. Although evaluating the quality of the sources of the information on medical reference apps is of the utmost importance, preference with respect to usability is also a factor. Students and residents on internal medicine clinical teaching units and the cardiac care units are involved in the care of hospitalized patients. They also may spend significant time in the emergency room assessing a variety of patients with medical issues. During these times, they typically need to formulate clinical questions, including but not limited to disease descriptions, laboratory parameters, drug dosing, and drug interactions. In many medical teaching units, the students use mobile medical apps to answer those types of questions.15,16 The current study presents a model of collaboration between a hospital library and practitioners and consists of two parts. The first and primary focus of the study was the development and testing of an evaluation tool for medical reference apps. It was expected that competing medical apps would have different features and some would score higher on the evaluation tool and consequently be more suitable for use as a mobile point-of-care reference tool. However, these higher-scoring apps may not be the best for usability. Medical apps that include information regarding the quality of the resources could be more difficult to use. The second part of the study was to evaluate the four highest scoring apps for usability with a small sample of medical students and residents.
PART 1: DEVELOPMENT AND TESTING OF THE EVALUATION RUBRIC Methods DEVELOPMENT OF THE EVALUATION RUBRIC The evaluation tool was devised following literature searches and discussions regarding components for assessment of the sources of medical app information. There have not been any other studies which evaluate the information sources of medical apps. The numerical and relative value of each facet of the evaluation was based on scoring of similar evaluations for desktop point-of-care tools.17,18 The levels of evidence sources were based on the evidence-based pyramid from the University of Washington.19 The tool, in the form of a rubric, and the method of evaluation were also sent to medical librarians across Canada for feedback. TESTING THE EVALUATION RUBRIC Six mobile medical resource apps were chosen to test the new evaluation rubric. At the time of the study (fall 2011), these were the only six medical reference
Downloaded by [Georgian Court University] at 08:27 21 February 2015
78
R. Butcher et al.
apps available for the iOS operating system in North America that included both treatment and diagnosis information: Epocrates Essentials, DynaMed, PEPID Primary Care Plus, First Consult, Lexi Complete, and Medscape. Year-long subscriptions to these apps (one per device) were obtained either through purchasing or unconditional donation by the publishers. One librarian evaluated all six apps using the evaluation rubric. To assign scores, the information used had to be included within the app itself or available on the app’s publicly accessible website. Companies were not contacted for further information if the answers were not readily accessible. This method was chosen to approximate an average user’s approach. A second librarian independently evaluated the same apps. Screen shots were kept of the information sought. Scores for the mobile medical resource app evaluations using the rubric were calculated from a total score of 100%.
Results DEVELOPMENT OF THE EVALUATION RUBRIC The evaluation rubric is outlined in Table 1. The evaluation rubric was divided into three categories: content, transparency, and evidence. The weighing of each category was based on relative importance where the scoring for content totaled 20%, transparency totaled 20%, and evidence totaled 60% of the final score. The evidence category was given more weight because determining the quality, validity, and reliability of the medical resources for each app was the primary purpose of the evaluation tool. Content represented the volume of material available on the app. It was measured by the number of topics and the frequency of updates. Transparency referred to the openness of information. It was measured by references cited and authors listed. For transparency, it was not the number of references or authors listed that was significant, but rather if they were listed. Evidence was associated with quality of the sources used to create the disease monographs. It was measured by presence of peer review, evidence sources (i.e., textbooks, journals, and systematic reviews), evidence grade, and explanation of evidence grade. In terms of the peer review, just the disease monographs themselves were examined and whether they had been peer reviewed. The sources that created the disease monographs were not examined. The time to examine every reference for each disease monograph to ensure that it was peer reviewed would have been prohibitive. Yet peer review is the minimum for a monograph to be considered academic or evidence-based. Since a more rigorous evaluation of peer review was not possible, the weight of this component in the evaluation was relatively low. Evidence grade referred to a scale the app might have used to rate the level of evidence. Different apps used different scales. For example, one might have used a scale of one to three; another might have used letter
79
60
20
Transparency
Evidence
20
Content
%
12 30
12 6
Evidence source
Evidence grade
Evidence grade explanation
8
Peer review
Authorship
12
10
Updated
References
10
%
Number of topics
TABLE 1 Evaluation Rubric
Does this app provide an explanation for its grading system either in the app or on their website?
Does this app use an evidence grading system?
What is the method of evidence collection the app uses to create the information in their topics? (Look for the method on the website)
Does the app indicate if it is peer-reviewed either in the app or on their website?
Do the topics list personal authors, contributors, editors, or committee name?
Do topics provide references for their information either in text or at the end of the monograph?
How frequently are the topics in this app updated?
How many topics does this app indicate it contains on their website?
Yes No
Yes No
The editors or content authors have done a systematic review of the literature The editors or content authors have done a review of evidence guidelines or health technology assessments Other reviews of the literature Journals or government documents Textbooks or created internally No explanation
Yes No
Yes No
Yes No
More than once a week Weekly More than once a month Monthly Quarterly Annually or less No number given
5000þ 4000–4999 3000–3999 2000–2999 1000–1999 999 or No number given
Downloaded by [Georgian Court University] at 08:27 21 February 2015
6 0
12 0
18 12 6 0
24
30
12 0
8 0
12 0
10 8 6 4 2 1 0
10 8 6 4 2 1 0
Points
80
R. Butcher et al.
grades. No one scale was necessarily superior to another. There was an expectation that the app would explain how its scale was derived, and as such, scored separately as explanation of evidence grade.
Downloaded by [Georgian Court University] at 08:27 21 February 2015
TESTING THE EVALUATION RUBRIC The evaluations of the six medical resource apps from the two librarians had few discrepancies in the scores and once the screen shots were reviewed, they were quickly resolved. Each app evaluation took approximately 45 minutes. Results of the six apps using the rubric are in Table 2. The content category showed variation in number of topics, from less than 1,000 to more than 5,000. The apps were nearly equivalent in transparency. All apps included references and nearly all had authors listed. Scores for evidence ranged from very poor (6=60) to perfect (60=60). The total scores out of 100% ranged from 23% to 88%. Only four apps scored over 50%: DynaMed (88%), First Consult (83%), Medscape (72%), and PEPID Primary Care Plus (54%). As the highest-scoring apps, these four were chosen to be evaluated for usability by the medical students and residents in Part 2 of the study.
PART 2: USABILITY OF THE FOUR HIGHEST-SCORING MEDICAL APPS Methods Approximately 53 residents=students complete rotations in a given academic year at the Saint John Regional Hospital in the cardiac care unit or medical teaching unit. All potential students on these rotations were sent an e-mail invitation, describing the study, approximately two weeks prior to the beginning of their rotations. Participants who voluntarily agreed to participate signed an informed consent form. The study and consent was approved by the Horizon Health Network Ethics Review Board (Reference number: 20111640). The participants were asked to utilize and evaluate each of the selected mobile medical reference apps for a one-week period. iPod Touch handheld devices were provided preloaded with software to be used each week. The iPod Touch was selected because it allows for the download of apps without cellular abilities and it was not as large as tablets and easier for the participants to carry. The participants were asked to utilize the apps to help answer questions and inform clinical care. They were not restricted to one app but were encouraged to use the weekly app as the first go-to point for information. At the beginning of the study, the participants were asked to fill in an entrance questionnaire to collect general demographic information and to determine previous experience with handheld devices and apps. At the
81
Lexi complete
Medscape
First consult
Epocrates essentials
DynaMed
PEPID primary care plus
Application
Monthly 4
