i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
journal homepage: www.ijmijournal.com
An internet based approach to improve general practitioners’ knowledge and practices: The development and pilot testing of the “ABC’s of vitamin D” program B. Bonevski a,∗ , P. Magin b , G. Horton b , J. Bryant c , M. Randell a , M.G. Kimlin d a
School of Medicine & Public Health, Faculty of Health & Medicine, University of Newcastle, Newcastle, NSW, Australia b Discipline of General Practice, Faculty of Health & Medicine, University of Newcastle, Newcastle, NSW, Australia c Priority Research Centre for Health Behaviour, School of Medicine & Public Health, Faculty of Health & Medicine, University of Newcastle, Newcastle, NSW, Australia d NHMRC Centre for Sun and Health, Queensland University of Technology, Brisbane, QLD, Australia
a r t i c l e
i n f o
a b s t r a c t
Article history:
Background: Online continuing medical education (CME) offers a number of advantages for
Received in revised form
physicians including flexibility with regards to location and timing of use. In order to effect
11 January 2015
physician practices and improve patient outcomes, it is important that the development of
Accepted 12 January 2015
online CME is theory and evidence-based.
Keywords:
practising general practitioners (GPs) on vitamin D and sun health called “The ABC’s of
Objectives: This paper aims to describe the development of an online CME program for Education medical continuing
Vitamin D for GPs” using elements of design principles for physician-education web sites as
Online
a framework. The paper will also report the program’s usability and acceptability pilot test
General practitioners
results.
Vitamin D deficiency
Methods: The ABC’s of Vitamin D program was developed following nine principles: needs assessment; evidence-based content development; multimodal program and modularisation; clinical cases; tailoring and interactivity; audit and feedback; credibility of the web site host; patient education materials; ease of use and navigation. Among the 20 GPs invited, acceptability and useability was tested with 12 GPs (60%) who agreed to participate and were interviewed following use of the program. The study was conducted between 2011 and 2013. Results: An online CME program consisting of eight modules was constructed. Of the 12 participating GPs, most (n = 11) reported that the program was clear and easy to understand, logical, easy to navigate, and took a reasonable amount of time (estimated between 1 and 3 h) to complete. Eleven of 12 participants said they would use the program as an accredited CME activity and all participants indicated that the program was ‘very or somewhat’ likely to lead to changes in the advice patients are given.
∗
Corresponding author. Tel.: +61 02 4940335710. E-mail addresses: [email protected], [email protected] (B. Bonevski).
http://dx.doi.org/10.1016/j.ijmedinf.2015.01.006 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
414
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Conclusion: This study found that a theory and evidence based approach for the development of an online CME program for GPs was acceptable to users. Further research is needed to examine whether the online CME program is effective at changing GP practices and improving patient outcomes. © 2015 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
There is considerable evidence that more traditional, passive forms of continuing medical education (CME) for practicing physicians, such as didactic lectures or distribution of written materials, are ineffective at changing clinical practice [1,2]. As a result, more interactive, engaging and educational approaches to CME have been sought. The use of the internet as a mechanism to deliver CME has been increasing in popularity. In 1997, 18 Northern American based sites were offering online CME activities, which grew to 300 in 2009 [3]. Not surprisingly in the United States (US) online CME use has also increased from 0.34% in 1997 to 41% of all CME in 2009 [3]. The online platform for delivering CME programs presents a number of advantages [4]: barriers of distance, cost and time are reduced since most online programs can be used anywhere and at any time, content can be updated readily with emerging new information, and there is significant potential to incorporate interactive elements, such as videos, podcasts and the provision of active learning questions with immediate feedback. As use of online CME grows, it is increasingly important that the online programs offered are well designed and grounded in sound education theory and evidence. Casebeer et al. have developed a list of principles for the design of online CME programs aimed at increasing the likelihood that they will change clinical practice and improve patient outcomes [5]. The principles are derived from literature reviews of learning theories, physician change theories, and evidence of effectiveness of CME interventions. According to Casebeer et al., physician-education web sites should include the following design principles: needs assessment, evidence-based content, multimodal strategies, use of modules (rather than a single education event), clinical cases, tailoring, interactivity, audit and feedback, credibility of the web site host, patient-education materials, ease of use and navigation, and simple hardware requirements. However, most CME online programs do not use these principles. In a review of online CME programs, Sklar [3] found that 31% were text only, only 1% included evidence based guidelines for practice, only 4% had question and answer format, 17% included clinical cases and 8% included multimodal strategies (podcasts). There is a clear need to improve the design of online CME programs. One area emerging as an important topic for CME is physician knowledge and practices relating to vitamin D deficiency. In particular, general practitioners (GPs) are at the frontline of delivering vitamin D related advice and care for the majority of the population [6]. Estimates suggest that one billion people worldwide are vitamin D deficient [7]. Initially thought to play a limited role in bone and skeletal health [8], benefits now
being ascribed to vitamin D include the prevention of some cancers, autoimmune disorders and possibly cardiovascular diseases [9–11]. Vitamin D deficiency may be independently associated with all-cause mortality in the general population [12,13], suggesting it plays an important role in overall health and may be an important public health measure. Vitamin D status is determined using blood testing. Although there is debate regarding optimal vitamin D levels [14,15], a serum 25-hydroxy vitamin D3 (25(OH)D) level of 50 nmol/L and above is generally recognised as sufficient [16,17]. Prevalence of vitamin D insufficiency is reported to be high [7,18]. While there is greater insufficiency of vitamin D at high latitudes [18–20], it is also found in fair-skinned people living at low latitudes like Australia, where ambient UV irradiance is high and the rates of skin cancer are amongst the highest in the world [21,22]. This co-occurrence of vitamin D insufficiency and skin cancer has led to the development of guidelines for a balanced approach to sun exposure for both the public [8,23] and doctors [24]. Achieving adequate vitamin D with their patients presents a conundrum for physicians as it is a nutrient most effectively derived from exposure to ultraviolet radiation (UVR) [6,7]. Physicians are likely to see patients who are at higher risk of vitamin D insufficiency including pregnant women, children, and the elderly, and can discuss vitamin D opportunistically during routine skin checks. Over-exposure to UVR is a primary risk factor for skin cancer [8]. Providing patients with advice that balances enough sun exposure to achieve adequate vitamin D status alongside the sun protection message is challenging and little guidance for GPs is currently available. There is evidence from general practice in Germany, Australia and New Zealand that indicates there are gaps in GP knowledge of vitamin D, that GPs are providing patients with advice that is not recommended, and that vitamin D testing rates may be inappropriately high [6,9,10]. In addition, we were unable to identify any published reports of existing vitamin D and sun protection education for GPs. This suggests that education for practicing physicians on the topic of vitamin D deficiency testing, management and sun exposure is urgently needed. This paper aims to describe the development of an online CME program for GPs on vitamin D and sun health called “The ABC’s of Vitamin D for GPs” using nine elements of Casebeer et al.’s [5] 12 design principles for physician-education web sites as a framework. Given online CME is a relatively recent innovation, and following Rogers Diffusion of Innovation theory [25] which proposes that individuals’ perceptions of an innovation determine its rate of adoption, the paper will also report the program’s usability, acceptability and perceived effectiveness as reported by the intended users of the program, GPs.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
2.
Methods
The ABC’s of Vitamin D program was developed and pilot tested between 2011 and 2013. Approval for the study was provided by the University of Newcastle Human Research Ethics Committee (H-2011-0040).
2.1.
Development of the online program
2.1.1.
Needs assessment
A baseline survey was conducted with a large sample of Australian general practitioners (n = 500) assessing knowledge, attitudes and practices relating to vitamin D and sun protection advice [6]. The survey identified a number of gaps in knowledge and a need for further information and education for practicing doctors. Results suggested that over-testing of vitamin D status may be occurring and that testing was not targeting high risk patients. There was confusion about the amount of UVR exposure necessary for adequate vitamin D synthesis and few GPs had read the national position statement released by a collaboration of peak health authorities regarding vitamin D deficiency. These results were used as a basis for developing the content areas of the CME program.
2.1.2.
Evidence-based content
The content of the online program was developed drawing on three evidence-based sources. First, a scoping literature review was conducted to identify all published evidence on vitamin D and optimal care. One of the publications identified was a comprehensive systematic review published by the Institute of Medicine which was extensively used for content development [26]. Second, current international peak health organisation recommendations and guidelines regarding sun exposure and vitamin D deficiency (less than 50 nm/L) prevention were consulted [8,24]. These guidelines provide recommendations for practising physicians for vitamin D related issues for the general population, those at increased risk of vitamin D deficiency and those at increased risk of skin cancer. Once developed, the gaps in knowledge identified in the baseline survey of GPs outlined above were examined against developed content to ensure that the content of the program adequately addressed these gaps. Iterative drafts of the content areas were developed based on these sources and considered by an advisory panel of 12 experts. The experts included representatives from general practice (N = 3), endocrinologists (N = 2), vitamin D specialists (N = 1), dermatologists (N = 1), representatives of the (Australian) national skin cancer committee (N = 2), a UVR epidemiologist (N = 1), and researchers and practitioners in the fields of skin cancer prevention and vitamin D deficiency (N = 2). The feedback from the expert advisory panel was incorporated into the final version of the program content. The program contains information for GPs classified according to eight modules (see Section 2.1.3 for more content information within each module). Initially, when users enter the program, they are provided with a background information section “Why is this program important?” which provides a rationale for the need for information and education on the topic of vitamin D and its relationship with UV exposure, and
415
GP over-testing of vitamin D status. The program also contains the objectives of the education specified as: “This web tool will help you: 1. Improve your overall knowledge of vitamin D. 2. Identify who may be at increased risk of vitamin D deficiency or skin cancer. 3. Provide patients with appropriate advice on balancing sun exposure and sun protection. 4. Decide when and how to test for vitamin D deficiency. 5. Manage vitamin D deficiency safely and effectively”. Additional components of the content include instructions about how to use the web program, and links to further reading and other resources.
2.1.3.
Multimodal program and modularisation
There is evidence that any effect of a single CME event decays over time and is less likely to cause long term changes in physician practices [27]. As a result, we constructed a program which enabled ongoing use, over multiple occasions, and through a number of ‘modules’. Figs. 1–4 show examples of screens from various program sections and modules. If interested in viewing the full program, please contact the corresponding author for access. Using a side bar, users navigated through eight modules which contain the following information: 1. Module 1 A brief background of vitamin D: provides information about the sources of vitamin D, its production and metabolism in the skin and body and activation of vitamin D, and its role in the functioning of body. 2. Module 2 Causes and health consequences of vitamin D deficiency: describes the causes and consequences of vitamin D deficiency including effects on bone health. 3. Module 3 Measuring vitamin D status: describes current vitamin D testing techniques and their limitations, information about the clinical indicators of vitamin D deficiency and information to help GPs identify when testing for vitamin D is indicated and when it is not. 4. Module 4 Achieving adequate vitamin D in Australia: provides information about the dietary recommendations for, and sources of, vitamin D, limitations of dietary sources, and current guidelines regarding sun exposure for vitamin D. 5. Module 5 Balancing UVR exposure: provides information about ways to advise patients about UVR exposure for vitamin D, ways to provide patients with advice about balancing sun exposure and sun protection and maintaining adequate vitamin D, and the effect of sunscreen on vitamin D. 6. Module 6 Groups at high risk of vitamin D deficiency: describes factors that may affect vitamin D synthesis in some people, and the groups most at risk of vitamin D deficiency. 7. Module 7 Managing vitamin D deficiency: provides an overview of treatment options for vitamin D deficiency and current recommendations for specific groups. 8. Module 8 Safety considerations: provides information about the interactions between vitamin D and other medications and vitamin D toxicity.
416
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Fig. 1 – Example screen shot of an opening screen stating program objectives. Illustrates the left column navigational bar listing the modules as well as additional program components.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
417
Fig. 2 – Example screen shot of Module 3 “Key Messages” program content.
Each module contains reference to the evidence base with links to additional reading, a summary screen of the ‘key messages’, and active learning questions.
A practising general practitioner who is also an academic involved in research and teaching was engaged to develop two patient case studies which addressed diagnosis and testing for vitamin D status (Module 3) and management of vitamin D deficiency (Module 7). Once case studies are completed, feedback and an outline of best practice care, supported by references to the evidence, are provided to users.
able to seamlessly manoeuvre from one module to another of their choice, or repeat a module a number of times until mastered. Interactive features contained in the online program included a “My Notes” function, whereby users could type in their thoughts or points of interest, pop-up definitions within modules which users could access by ‘hovering’ over certain words in the content, a resources page which contains web links to further reading and useful tools, a ‘downloads’ feature for downloading printable information on each module, and an audio ‘podcast’ obtained with permission from the Australian national radio broadcaster on the topic of vitamin D.
2.1.5.
2.1.6.
2.1.4.
Clinical cases
Tailoring and interactivity
Tailoring is based on health behaviour and communication theory [28]. A number of features ensured that the program was tailored to each user’s needs and was interactive. To enhance tailoring, the program was designed to adjust to the informational needs of the user by branching, a mechanism allowing users to skip to sections of interest and re-enter the program multiple times [27]. Furthermore, users were
Audit and feedback
Each module contained active learning questions with immediate feedback regarding accuracy of responses. In addition, for users wanting to complete the entire program and test themselves pre and post, the program contained questions to be completed before they used the program, and then again after they have completed the entire program. Finally, the program contained a ‘Reinforcing Activity’ asking users to review
418
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Fig. 3 – Example screen shot of content of information in Module 6.
the patient education materials they currently use in their practice on any topic relating to this program and create a resource folder of quality information and useful referral contacts in their local community. They were encouraged to use the programs’ links and resources page.
2.1.9.
2.1.7.
2.1.10. Program architecture
Credibility of the web site host
The program is hosted on the Royal Australian College of General Practice (RACGP) Continuing Medical Education and Quality Assurance (CME & QA) site and accessed using a web link supplied by the first author. The RACGP is Australia’s peak general practice training and education body and a highly credible host [29]. Due to its educational and interactive components, the program has been accredited and GPs completing the program receive six Category 2 CME & QA points.
2.1.8.
Patient-education materials
GPs had the option to download patient information sheets in pdf format within each module to print and distribute to patients during the consultation.
Ease of use and navigation
The program was designed to be easy and appealing to use. Where possible, graphics and illustrations were used and information was presented in an easy to read format using simple language. Acceptability and ease of use were established during pilot testing.
The software tool used to implement the ABC’s of vitamin D program was the outcome of a bespoke software development project run by the Distributed Computing Research Group at the University of Newcastle. The website was implemented as a traditional Web 2.0 application where GP’s used a standard web browser to access the program which resided on a remote server. The deployment of the program was broken into two parts—the server component and the client component. The server component was written in the Java programming language and utilised open source middleware packages such as Apache Tomcat, for the web application layer and MySQL for the data persistence layer. The client component was written in standards compliant HTML and CSS, with dynamic components written in the Javascript programming language.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
419
Fig. 4 – Example screen shot of “pop up” information when user “hovers” mouse over highlighted terms in the text.
2.2.
Testing usability and acceptability to users
2.2.1.
Procedure
Using a process described by Faulkner [30] with components recommended by Norman and Panizzi [31], a purposive sampling technique was used to recruit an independent sample of GPs from a local general practice research network to test the usability of the program and its appropriateness for a range of ages, gender and part time or full time status. The goal of the usability testing was to identify any interface problems that may cause user dissatisfaction; reduce completion time; improve program performance and efficiency; and increase user acceptability [32]. Twenty GPs involved in the GP research network were sent letters inviting them to participate in a study about an online CME program for vitamin D. GPs were offered a $250 retail voucher as reimbursement for their time. The letter was followed by a phone call from a member of the research team who explained the study and sought informed consent. Consenting GPs were given the hyperlink to the online program and asked to complete as much of the program as they were able within the next two weeks, following which a research team member telephoned them to
conduct a semi-structured interview. Telephone interviews, as opposed to web or paper surveys, were conducted for two main reasons: (1) the interviews were in-depth and qualitative in nature employing open ended questions, and offering participants opportunity to describe their experiences with the program and interact with the interviewer and (2) conducting the interviews by telephone was more acceptable to participants who were busy GPs than face to face interviews. An experienced independent researcher engaged specifically for conducting the telephone interviews conducted the interviews and an interview schedule with open-ended items and prompts for further discussion was followed.
2.2.2.
Outcome measures
A brief user interview was administered which assessed:
1. Respondent characteristics: age, gender and sessions worked per week. 2. Overall acceptability: of the online program using a 6 item scale with a 5-point response scale (Strongly agree-strongly disagree).
420
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Table 1 – Overall acceptability ratings of the ABC’s of vitamin D online CME program (n = 12 GPs). Strongly agree n Course material was presented at a satisfactory level of detail The time taken to complete the program was reasonable The time spent viewing the content was worthwhile The presentation of information was clear and easily understandable The information was presented in a logical manner The program was easy to navigate ∗
Agree n
Neutral n
Disagree* n
1 1 2 0 0 0
0 0 1 0 1 1
2 2 0 1 0 1
9 9 9 11 11 10
There were no ‘Strongly disagree’ responses.
Table 2 – Participants ratings of the web-program modules (n = 12 GPs). Module
1 2 3 4 5 6 7 8
Topic
In addition, open-ended questions asked respondents to offer suggestions for improving the program, and whether they experienced technical difficulties using the program.
3.
Results
3.1.
Sample
Twelve GPs participated in the pilot study (60% response rate). Participants were aged between 31 and 60 years, 50% were male and worked between two and eight sessions per week (where each session corresponds to half a day).
Overall acceptability
Table 1 shows the overall acceptability rating. The results suggest high acceptability.
3.1.2.
Extremely
Very
moderately
Not at all
8 9 9 9 10 5 4 8
2 1 0 0 1 1 0 0
2 2 3 3 1 6 8 4
0 0 0 0 0 0 0 0
A brief background of vitamin D Causes and health consequences of vitamin D deficiency Measuring vitamin D deficiency Achieving adequate vitamin D in Australia Balancing UV exposure Groups at high risk of vitamin D deficiency Managing vitamin D deficiency Safety considerations
3. Module preferences: respondents were asked to rate the informativeness of each module on a 5-point response scale (extremely-not at all). 4. Perceived effectiveness and use: respondents were asked to rate the usefulness of the content and how likely they thought the program was to result in changes in patient care (very likely, somewhat likely, not very likely, not at all likely). Respondents were also asked whether they would use the program if it was an accredited CME activity (yes/no) and whether they would refer to the program when talking to patients about skin cancer and/or vitamin D related issues (yes/no). 5. Completion time: both self-reported and program autorecorded (in min).
3.1.1.
N of participants ratings of each module as informative
Module preferences
Participants (n = 12) ratings of each module as informative (on a scale of ‘extremely’, ‘very’, ‘moderately’, ‘somewhat’ and ‘not at all’) are presented in Table 2.
3.1.3.
No opinion 0 0 0 0 0 0 0 0
Perceived effectiveness
Ten of the 12 (83%) participants rated the program content as ‘very useful’ and two as ‘somewhat useful’. Five participants believed that the program was ‘very likely’ to result in changes to advice being given to patients and seven participants rated it as ‘somewhat likely’ to cause change in advice given to patients. Eleven of the 12 participants (92%) said they would use the program if it was an accredited CME activity and nine participants (75%) reported that they would refer to the program when talking to patients about related topics.
3.1.4.
Completion time
All participants completed the entire program and reported that it took between 1 and 3 h to complete the eight modules. According to the program timer, the average length of use was 124 min, with a range of 46 min to 170 min.
3.1.5. Qualitative feedback—General comments and technical issues Suggestions for improvements included reducing the introduction and Module 1, less detail in Modules 1, 3 and 7, higher quality graphics and illustrations, provision of an indication of how long the program takes, and reduced repetitiveness of content across some modules. Some technical issues were identified (such as the program ‘freezing’ when answers submitted, not being able to progress from one module to another, a case study not functioning at all, difficulty logging on using certain internet browsers) which were referred back to the software programmer who corrected the problems.
4.
Discussion
This study described the development and pilot testing of a new online CME program for GPs in the area of vitamin D and sun health. The study highlights the importance of pilot-testing, with target users, new online programs. Results
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
showed that development of a CME online program for GPs can be robust following a theory and evidence based design framework for physician education web site development [5]. Usability and acceptability testing with GPs indicated high ratings of ease of use, perceived effectiveness and potential use as a CME activity, acceptable completion time and useful information.
4.1.
Limitations
The findings and limitations of this study are being used to guide future research with the ‘ABC’s of vitamin D’ program. The program was pilot tested with a small sample. A sample size of between 10 and 20 users has been identified as adequate for software usability testing [30–32]. With fewer users, fewer problems are uncovered during testing, and including more than 20 users becomes redundant. However, it should be acknowledged that as members of a research network, the current group of physicians may have different learning capabilities, interests or familiarity with computer-based instruction than the broader population of primary care physicians. Another limitation with the pilot testing is that users were not observed using the program, as they were interviewed by telephone. An important question is whether the program will be effective at improving physician knowledge, practice or patient outcomes. Future research is planned to examine the effectiveness of the ABC’s of Vitamin D program using a randomised controlled trial with a large representative sample of GPs.
4.2.
Lessons learnt
Pilot studies offer opportunity to test programs and identify potential problems during development which may negatively affect adoption and implementation. Some participants expressed uncertainties regarding the vitamin D and sun exposure guidelines used in the current program. Defining the accurate amount of UVR exposure necessary for achieving adequate vitamin D status is a complex and emerging science. The existing recommendations and guidelines note the limits of the evidence base [8,24]. This uncertainty may limit the effectiveness of the program at changing the advice GPs provide to patients regarding sun exposure for vitamin D deficiency [27]. Nonetheless, this version of the program used the most current evidence available which seemed to be validated by high acceptability ratings of the content of the program. One of the advantages of an online CME program is that as the evidence firms up over time, the program can be updated to reflect advances in knowledge and understanding. Ratings of each module suggested that participants found the modules covering sun and UVR exposure for vitamin D and effectively communicating to patients a health message for balancing sun exposure with sun protection ‘most’ informative. No modules were rated as ‘not at all’ informative. Unlike all other modules which were rated as being ‘extremely’ or ‘very’ informative by the majority of participants, the module on the topic of ‘managing vitamin D deficiency’ was rated by most participants as being ‘moderately’ or ‘somewhat’ informative. Some of the comments received on this module suggest that the way the information was presented may not
421
Summary points What was already known on the topic? • The use of the internet as a mechanism to deliver continuing medical education (CME) has been increasing in popularity. • As use of online CME grows, it is increasingly important that the online programs offered are well designed and grounded in sound education theory and evidence. What this study added to our knowledge? • This study applied a set of principles for the design of online CME programs aimed at increasing the likelihood that they will change clinical practice and improve patient outcomes in the development of an online CME program for general practitioners (GPs) on the topic of vitamin D. • Currently, most CME online programs do not use these principles. • This study showed that a program using online design principles was rated highly acceptable and easy to use by GPs. • Greater acceptability and ease of use increases the likelihood that the program will be used by GPs and that it will translate into changes in clinical practice.
have been practical, for example the module did not present dosage recommendations of supplements as monthly doses, which is the dose most often used. Other comments on Module 7 included repetitiveness with other modules, that it was “longer” than other modules and that it was more “confusing” than other modules. Changes to the module were made based on participant comments to ensure it was practical to users. In conclusion, a theory and evidence based approach may facilitate the development of an acceptable and informative online CME program for GPs. Further research is needed to examine whether the online CME program is effective at changing GP practices and improving patient outcomes.
Funding The research was funded by a Cancer Institute NSW Research Innovation Grant (10/RFG/2-38). At the time of the study BB was supported by a Cancer Institute NSW Career Development Fellowship (10/CDF/2-40).
Conflict of interest The authors declare no conflicts of interest.
Author contributions BB conceived of the study, MP and HG recruited participants, all authors contributed to data collection, analysis and write up of the manuscript.
422
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Acknowledgements Thanks to Mr Mark Wallis for the computer programming and Ms Raelene Monahan for assistance with data collection. Thanks to the National Prescribing Service (NPS) Quality Use of Medicines for assistance developing the vitamin D related content and our experts and GPs who participated in pilot testing for giving their time.
references
[1] L. Forsetlund, A. Bjørndal, A. Rashidian, G. Jamtvedt, M.A. O’Brien, F.M. Wolf, D. Davis, J. Odgaard-Jensen, A.D. Oxman, Continuing education meetings and workshops: effects on professional practice and health care outcomes, Cochrane Database Syst. Rev. (2) (2009) (Art. No.: CD003030. PMID: 19370580). [2] M. Mansouri, J. Lockyer, A meta-analysis of continuing medical education effectiveness, J. Contin. Educ. Health Prof. 27 (1) (2007) 6–15. [3] B. Sklar, Online Continuing Medical Education, July 2009, Available at www.cmelist.com/mastersthesis/. Archived at http://www.webcitation.org/6KtRtAXC3 (accessed 23 November 2013). [4] G. Ryan, P. Lyon, K. Kumar, J. Bell, S. Barnet, T. Shaw, Online CME: an effective alternative to face-to-face delivery, Med. Teach. 29 (8) (2007) e251–e257. [5] L.L. Casebeer, S.M. Strasser, C.M. Spettell, T.C. Wall, N. Weissman, M.N. Ray, J.J. Allison, Designing tailored web-based instruction to improve practising physicians’ preventive practices, J. Med. Internet Res. 5 (3) (2003) e20. [6] B. Bonevski, A. Girgis, P. Magin, G. Horton, I. Brozek, B. Armstrong, Prescribing sunshine: a survey of general practitioners’ knowledge, attitudes and practices relating to the sun and vitamin D, Int. J. Cancer 130 (2012) 2138–2145. [7] M.F. Holick, Vitamin D deficiency, N. Engl. J. Med. 357 (2007) 266–281. [8] Working Group of the Australian and New Zealand Bone and Mineral Society, Endocrine Society, and Osteoporosis Australia et al., Vitamin D and adult bone health in Australia and New Zealand: a position statement, Med. J. Aust. 182 (6) (2005) 281–285. [9] M.F. Holick, Sunlight and vitamin D for bone health and prevention of autoimmune diseases, cancers and cardiovascular disease, Am. J. Clin. Nutr. 80 (6) (2004) 1678S–1688S. [10] C.F. Garland, F.C. Garland, E.D. Gorham, M. Lipkin, et al., The role of vitamin D in cancer prevention, Am. J. Public Health 96 (2006) 252–261. [11] S.V. Ramagopalan, A. Heger, A.J. Berlanga, et al., A ChIP-seq defined genome-wide map of vitamin D receptor binding: associations with disease and evolution, Genome Res. 20 (2010) 1352–1360. [12] M.L. Melamed, E.D. Michos, W. Post, B. Astor, 25-Hydroxyvitamin D levels and the risk of mortality in the general population, Arch. Intern. Med. 168 (15) (2008) 1629–1637. [13] P. Autier, S. Gandini, Vitamin D supplementation and total mortality: a meta-analysis of randomized controlled trials, Arch. Intern. Med. 167 (16) (2007) 1730–1737 (10). [14] A.R. Webb, B.R. de Costa, M.F. Holick, Sunlight regulates the cutaneous production of vitamin D3 by causing its photodegradation, J. Clin. Endocrinol. Metab. 68 (1989) 882–887.
[15] B. Dawson-Hughes, R.P. Heaney, M.F. Holick, P. Lips, et al., Estimates of optimal vitamin D status, Osteoporos. Int. 16 (7) (2005) 713–716. [16] B.W. Hollis, C.L. Wagner, Normal serum vitamin D levels, N. Engl. J. Med. 352 (5) (2005), 515–515. [17] B. Diffey, Do we need a revised public health policy on sun exposure, Br. J. Dermatol. 154 (2006) 1046–1051. [18] D. Gillie, A new government policy is needed for sunlight and vitamin D, Br. J. Dermatol. 154 (2006) 1052–1061. [19] J. Moan, A.C. Porojnicu, A. Dahlback, R.B. Setlow, Addressing the health benefits and risks, involving vitamin D or skin cancer, of increased sun exposure, Proc. Natl. Acad. Sci. 105 (2) (2008) 668–673. [20] T. Vik, K. Try, J.H. Stromme, The vitamin D status of man at 70 degrees north, Scand. J. Clin. Lab. Invest. 40 (1980) 227–232. [21] M. Brustad, et al., Vitamin D status of middle-aged women at 65–71 degrees north in relation to dietary intake and exposure to ultraviolet radiation, Public Health Nutr. 7 (2004) 327–335. [22] J.J. McGrath, M.G. Kimlin, S. Saha, et al., Vitamin D insufficiency in south-east Queensland, Med. J. Aust. 174 (2001) 150–151. [23] J.A. Pasco, M.J. Henry, G.C. Nicholson, et al., Vitamin D status of women in the Geelong Osteoporosis Study: association with diet and casual exposure to sunlight, Med. J. Aust. 175 (2001) 401–405. [24] Australian New Zealand Bone and Mineral Society, Osteoporosis Australia, The Australasian College of Dermatologists and the Cancer Council Australia, Risks and benefits of sun exposure position statement, updated 2007, in: Osteoporosis Australia. Calcium, Vitamin D and Osteoporosis. A Guide for GPs, second ed., Australian New Zealand Bone and Mineral Society, Osteoporosis Australia, The Australasian College of Dermatologists and the Cancer Council Australia, 2008, Available online at http:/www.osteoporosis.org.au/files/factsheets/OTH-7665ENG.pdf (accessed 23 November 2013. [25] E.M. Rogers, Diffusion of Innovations, Free Press, New York, NY, 1995, ISBN 1451602472. [26] A.C. Ross, C.L. Taylor, A.L. Yaktine, H.B Del Valle (Eds.), Committee to Review Dietary Reference Intakes for Vitamin D and Calcium, Institute of Medicine, The National Academies, Washington, 2011, ISBN 9780309163941. [27] D. Davis, M.A. Thomson, A.D. Oxman, R.B. Haynes, Changing physician performance. A systematic review of the effect of continuing medical education strategies, J. Am. Med. Assoc. 274 (9) (1995) 700–705. [28] D.W. Farrell, L. Olevitch, L.K. Brennan, M. Kreuter, in: M. Kreuter (Ed.), Tailoring Health Messages: Customizing Communication with Computer Technology, Psychology Press, Mahwah, NJ, 1999, ISBN 1410603318. [29] Royal Australian College of General Practice. http://www.racgp.org.au/. Archived at http://www.webcitation.org/6KtRdxrzd (accessed 23 November 2013). [30] L. Faulkner, Beyond the five-user assumption: benefits of increased sample sizes in usability testing, Behav. Res. Methods Instrum. Comput. 35 (3) (2003) 379–383. [31] K.L. Norman, E. Panizzi, Levels of automation and user participation in usability testing, Int. Comput. 18 (2006) 246–264. [32] B. Shneiderman, Designing the User Interface: Strategies for Effective Human–Computer Interaction, fourth ed., Addison-Wesley, Reading, MA, 2005, ISBN 0321197860.
journal homepage: www.ijmijournal.com
An internet based approach to improve general practitioners’ knowledge and practices: The development and pilot testing of the “ABC’s of vitamin D” program B. Bonevski a,∗ , P. Magin b , G. Horton b , J. Bryant c , M. Randell a , M.G. Kimlin d a
School of Medicine & Public Health, Faculty of Health & Medicine, University of Newcastle, Newcastle, NSW, Australia b Discipline of General Practice, Faculty of Health & Medicine, University of Newcastle, Newcastle, NSW, Australia c Priority Research Centre for Health Behaviour, School of Medicine & Public Health, Faculty of Health & Medicine, University of Newcastle, Newcastle, NSW, Australia d NHMRC Centre for Sun and Health, Queensland University of Technology, Brisbane, QLD, Australia
a r t i c l e
i n f o
a b s t r a c t
Article history:
Background: Online continuing medical education (CME) offers a number of advantages for
Received in revised form
physicians including flexibility with regards to location and timing of use. In order to effect
11 January 2015
physician practices and improve patient outcomes, it is important that the development of
Accepted 12 January 2015
online CME is theory and evidence-based.
Keywords:
practising general practitioners (GPs) on vitamin D and sun health called “The ABC’s of
Objectives: This paper aims to describe the development of an online CME program for Education medical continuing
Vitamin D for GPs” using elements of design principles for physician-education web sites as
Online
a framework. The paper will also report the program’s usability and acceptability pilot test
General practitioners
results.
Vitamin D deficiency
Methods: The ABC’s of Vitamin D program was developed following nine principles: needs assessment; evidence-based content development; multimodal program and modularisation; clinical cases; tailoring and interactivity; audit and feedback; credibility of the web site host; patient education materials; ease of use and navigation. Among the 20 GPs invited, acceptability and useability was tested with 12 GPs (60%) who agreed to participate and were interviewed following use of the program. The study was conducted between 2011 and 2013. Results: An online CME program consisting of eight modules was constructed. Of the 12 participating GPs, most (n = 11) reported that the program was clear and easy to understand, logical, easy to navigate, and took a reasonable amount of time (estimated between 1 and 3 h) to complete. Eleven of 12 participants said they would use the program as an accredited CME activity and all participants indicated that the program was ‘very or somewhat’ likely to lead to changes in the advice patients are given.
∗
Corresponding author. Tel.: +61 02 4940335710. E-mail addresses: [email protected], [email protected] (B. Bonevski).
http://dx.doi.org/10.1016/j.ijmedinf.2015.01.006 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
414
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Conclusion: This study found that a theory and evidence based approach for the development of an online CME program for GPs was acceptable to users. Further research is needed to examine whether the online CME program is effective at changing GP practices and improving patient outcomes. © 2015 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
There is considerable evidence that more traditional, passive forms of continuing medical education (CME) for practicing physicians, such as didactic lectures or distribution of written materials, are ineffective at changing clinical practice [1,2]. As a result, more interactive, engaging and educational approaches to CME have been sought. The use of the internet as a mechanism to deliver CME has been increasing in popularity. In 1997, 18 Northern American based sites were offering online CME activities, which grew to 300 in 2009 [3]. Not surprisingly in the United States (US) online CME use has also increased from 0.34% in 1997 to 41% of all CME in 2009 [3]. The online platform for delivering CME programs presents a number of advantages [4]: barriers of distance, cost and time are reduced since most online programs can be used anywhere and at any time, content can be updated readily with emerging new information, and there is significant potential to incorporate interactive elements, such as videos, podcasts and the provision of active learning questions with immediate feedback. As use of online CME grows, it is increasingly important that the online programs offered are well designed and grounded in sound education theory and evidence. Casebeer et al. have developed a list of principles for the design of online CME programs aimed at increasing the likelihood that they will change clinical practice and improve patient outcomes [5]. The principles are derived from literature reviews of learning theories, physician change theories, and evidence of effectiveness of CME interventions. According to Casebeer et al., physician-education web sites should include the following design principles: needs assessment, evidence-based content, multimodal strategies, use of modules (rather than a single education event), clinical cases, tailoring, interactivity, audit and feedback, credibility of the web site host, patient-education materials, ease of use and navigation, and simple hardware requirements. However, most CME online programs do not use these principles. In a review of online CME programs, Sklar [3] found that 31% were text only, only 1% included evidence based guidelines for practice, only 4% had question and answer format, 17% included clinical cases and 8% included multimodal strategies (podcasts). There is a clear need to improve the design of online CME programs. One area emerging as an important topic for CME is physician knowledge and practices relating to vitamin D deficiency. In particular, general practitioners (GPs) are at the frontline of delivering vitamin D related advice and care for the majority of the population [6]. Estimates suggest that one billion people worldwide are vitamin D deficient [7]. Initially thought to play a limited role in bone and skeletal health [8], benefits now
being ascribed to vitamin D include the prevention of some cancers, autoimmune disorders and possibly cardiovascular diseases [9–11]. Vitamin D deficiency may be independently associated with all-cause mortality in the general population [12,13], suggesting it plays an important role in overall health and may be an important public health measure. Vitamin D status is determined using blood testing. Although there is debate regarding optimal vitamin D levels [14,15], a serum 25-hydroxy vitamin D3 (25(OH)D) level of 50 nmol/L and above is generally recognised as sufficient [16,17]. Prevalence of vitamin D insufficiency is reported to be high [7,18]. While there is greater insufficiency of vitamin D at high latitudes [18–20], it is also found in fair-skinned people living at low latitudes like Australia, where ambient UV irradiance is high and the rates of skin cancer are amongst the highest in the world [21,22]. This co-occurrence of vitamin D insufficiency and skin cancer has led to the development of guidelines for a balanced approach to sun exposure for both the public [8,23] and doctors [24]. Achieving adequate vitamin D with their patients presents a conundrum for physicians as it is a nutrient most effectively derived from exposure to ultraviolet radiation (UVR) [6,7]. Physicians are likely to see patients who are at higher risk of vitamin D insufficiency including pregnant women, children, and the elderly, and can discuss vitamin D opportunistically during routine skin checks. Over-exposure to UVR is a primary risk factor for skin cancer [8]. Providing patients with advice that balances enough sun exposure to achieve adequate vitamin D status alongside the sun protection message is challenging and little guidance for GPs is currently available. There is evidence from general practice in Germany, Australia and New Zealand that indicates there are gaps in GP knowledge of vitamin D, that GPs are providing patients with advice that is not recommended, and that vitamin D testing rates may be inappropriately high [6,9,10]. In addition, we were unable to identify any published reports of existing vitamin D and sun protection education for GPs. This suggests that education for practicing physicians on the topic of vitamin D deficiency testing, management and sun exposure is urgently needed. This paper aims to describe the development of an online CME program for GPs on vitamin D and sun health called “The ABC’s of Vitamin D for GPs” using nine elements of Casebeer et al.’s [5] 12 design principles for physician-education web sites as a framework. Given online CME is a relatively recent innovation, and following Rogers Diffusion of Innovation theory [25] which proposes that individuals’ perceptions of an innovation determine its rate of adoption, the paper will also report the program’s usability, acceptability and perceived effectiveness as reported by the intended users of the program, GPs.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
2.
Methods
The ABC’s of Vitamin D program was developed and pilot tested between 2011 and 2013. Approval for the study was provided by the University of Newcastle Human Research Ethics Committee (H-2011-0040).
2.1.
Development of the online program
2.1.1.
Needs assessment
A baseline survey was conducted with a large sample of Australian general practitioners (n = 500) assessing knowledge, attitudes and practices relating to vitamin D and sun protection advice [6]. The survey identified a number of gaps in knowledge and a need for further information and education for practicing doctors. Results suggested that over-testing of vitamin D status may be occurring and that testing was not targeting high risk patients. There was confusion about the amount of UVR exposure necessary for adequate vitamin D synthesis and few GPs had read the national position statement released by a collaboration of peak health authorities regarding vitamin D deficiency. These results were used as a basis for developing the content areas of the CME program.
2.1.2.
Evidence-based content
The content of the online program was developed drawing on three evidence-based sources. First, a scoping literature review was conducted to identify all published evidence on vitamin D and optimal care. One of the publications identified was a comprehensive systematic review published by the Institute of Medicine which was extensively used for content development [26]. Second, current international peak health organisation recommendations and guidelines regarding sun exposure and vitamin D deficiency (less than 50 nm/L) prevention were consulted [8,24]. These guidelines provide recommendations for practising physicians for vitamin D related issues for the general population, those at increased risk of vitamin D deficiency and those at increased risk of skin cancer. Once developed, the gaps in knowledge identified in the baseline survey of GPs outlined above were examined against developed content to ensure that the content of the program adequately addressed these gaps. Iterative drafts of the content areas were developed based on these sources and considered by an advisory panel of 12 experts. The experts included representatives from general practice (N = 3), endocrinologists (N = 2), vitamin D specialists (N = 1), dermatologists (N = 1), representatives of the (Australian) national skin cancer committee (N = 2), a UVR epidemiologist (N = 1), and researchers and practitioners in the fields of skin cancer prevention and vitamin D deficiency (N = 2). The feedback from the expert advisory panel was incorporated into the final version of the program content. The program contains information for GPs classified according to eight modules (see Section 2.1.3 for more content information within each module). Initially, when users enter the program, they are provided with a background information section “Why is this program important?” which provides a rationale for the need for information and education on the topic of vitamin D and its relationship with UV exposure, and
415
GP over-testing of vitamin D status. The program also contains the objectives of the education specified as: “This web tool will help you: 1. Improve your overall knowledge of vitamin D. 2. Identify who may be at increased risk of vitamin D deficiency or skin cancer. 3. Provide patients with appropriate advice on balancing sun exposure and sun protection. 4. Decide when and how to test for vitamin D deficiency. 5. Manage vitamin D deficiency safely and effectively”. Additional components of the content include instructions about how to use the web program, and links to further reading and other resources.
2.1.3.
Multimodal program and modularisation
There is evidence that any effect of a single CME event decays over time and is less likely to cause long term changes in physician practices [27]. As a result, we constructed a program which enabled ongoing use, over multiple occasions, and through a number of ‘modules’. Figs. 1–4 show examples of screens from various program sections and modules. If interested in viewing the full program, please contact the corresponding author for access. Using a side bar, users navigated through eight modules which contain the following information: 1. Module 1 A brief background of vitamin D: provides information about the sources of vitamin D, its production and metabolism in the skin and body and activation of vitamin D, and its role in the functioning of body. 2. Module 2 Causes and health consequences of vitamin D deficiency: describes the causes and consequences of vitamin D deficiency including effects on bone health. 3. Module 3 Measuring vitamin D status: describes current vitamin D testing techniques and their limitations, information about the clinical indicators of vitamin D deficiency and information to help GPs identify when testing for vitamin D is indicated and when it is not. 4. Module 4 Achieving adequate vitamin D in Australia: provides information about the dietary recommendations for, and sources of, vitamin D, limitations of dietary sources, and current guidelines regarding sun exposure for vitamin D. 5. Module 5 Balancing UVR exposure: provides information about ways to advise patients about UVR exposure for vitamin D, ways to provide patients with advice about balancing sun exposure and sun protection and maintaining adequate vitamin D, and the effect of sunscreen on vitamin D. 6. Module 6 Groups at high risk of vitamin D deficiency: describes factors that may affect vitamin D synthesis in some people, and the groups most at risk of vitamin D deficiency. 7. Module 7 Managing vitamin D deficiency: provides an overview of treatment options for vitamin D deficiency and current recommendations for specific groups. 8. Module 8 Safety considerations: provides information about the interactions between vitamin D and other medications and vitamin D toxicity.
416
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Fig. 1 – Example screen shot of an opening screen stating program objectives. Illustrates the left column navigational bar listing the modules as well as additional program components.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
417
Fig. 2 – Example screen shot of Module 3 “Key Messages” program content.
Each module contains reference to the evidence base with links to additional reading, a summary screen of the ‘key messages’, and active learning questions.
A practising general practitioner who is also an academic involved in research and teaching was engaged to develop two patient case studies which addressed diagnosis and testing for vitamin D status (Module 3) and management of vitamin D deficiency (Module 7). Once case studies are completed, feedback and an outline of best practice care, supported by references to the evidence, are provided to users.
able to seamlessly manoeuvre from one module to another of their choice, or repeat a module a number of times until mastered. Interactive features contained in the online program included a “My Notes” function, whereby users could type in their thoughts or points of interest, pop-up definitions within modules which users could access by ‘hovering’ over certain words in the content, a resources page which contains web links to further reading and useful tools, a ‘downloads’ feature for downloading printable information on each module, and an audio ‘podcast’ obtained with permission from the Australian national radio broadcaster on the topic of vitamin D.
2.1.5.
2.1.6.
2.1.4.
Clinical cases
Tailoring and interactivity
Tailoring is based on health behaviour and communication theory [28]. A number of features ensured that the program was tailored to each user’s needs and was interactive. To enhance tailoring, the program was designed to adjust to the informational needs of the user by branching, a mechanism allowing users to skip to sections of interest and re-enter the program multiple times [27]. Furthermore, users were
Audit and feedback
Each module contained active learning questions with immediate feedback regarding accuracy of responses. In addition, for users wanting to complete the entire program and test themselves pre and post, the program contained questions to be completed before they used the program, and then again after they have completed the entire program. Finally, the program contained a ‘Reinforcing Activity’ asking users to review
418
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Fig. 3 – Example screen shot of content of information in Module 6.
the patient education materials they currently use in their practice on any topic relating to this program and create a resource folder of quality information and useful referral contacts in their local community. They were encouraged to use the programs’ links and resources page.
2.1.9.
2.1.7.
2.1.10. Program architecture
Credibility of the web site host
The program is hosted on the Royal Australian College of General Practice (RACGP) Continuing Medical Education and Quality Assurance (CME & QA) site and accessed using a web link supplied by the first author. The RACGP is Australia’s peak general practice training and education body and a highly credible host [29]. Due to its educational and interactive components, the program has been accredited and GPs completing the program receive six Category 2 CME & QA points.
2.1.8.
Patient-education materials
GPs had the option to download patient information sheets in pdf format within each module to print and distribute to patients during the consultation.
Ease of use and navigation
The program was designed to be easy and appealing to use. Where possible, graphics and illustrations were used and information was presented in an easy to read format using simple language. Acceptability and ease of use were established during pilot testing.
The software tool used to implement the ABC’s of vitamin D program was the outcome of a bespoke software development project run by the Distributed Computing Research Group at the University of Newcastle. The website was implemented as a traditional Web 2.0 application where GP’s used a standard web browser to access the program which resided on a remote server. The deployment of the program was broken into two parts—the server component and the client component. The server component was written in the Java programming language and utilised open source middleware packages such as Apache Tomcat, for the web application layer and MySQL for the data persistence layer. The client component was written in standards compliant HTML and CSS, with dynamic components written in the Javascript programming language.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
419
Fig. 4 – Example screen shot of “pop up” information when user “hovers” mouse over highlighted terms in the text.
2.2.
Testing usability and acceptability to users
2.2.1.
Procedure
Using a process described by Faulkner [30] with components recommended by Norman and Panizzi [31], a purposive sampling technique was used to recruit an independent sample of GPs from a local general practice research network to test the usability of the program and its appropriateness for a range of ages, gender and part time or full time status. The goal of the usability testing was to identify any interface problems that may cause user dissatisfaction; reduce completion time; improve program performance and efficiency; and increase user acceptability [32]. Twenty GPs involved in the GP research network were sent letters inviting them to participate in a study about an online CME program for vitamin D. GPs were offered a $250 retail voucher as reimbursement for their time. The letter was followed by a phone call from a member of the research team who explained the study and sought informed consent. Consenting GPs were given the hyperlink to the online program and asked to complete as much of the program as they were able within the next two weeks, following which a research team member telephoned them to
conduct a semi-structured interview. Telephone interviews, as opposed to web or paper surveys, were conducted for two main reasons: (1) the interviews were in-depth and qualitative in nature employing open ended questions, and offering participants opportunity to describe their experiences with the program and interact with the interviewer and (2) conducting the interviews by telephone was more acceptable to participants who were busy GPs than face to face interviews. An experienced independent researcher engaged specifically for conducting the telephone interviews conducted the interviews and an interview schedule with open-ended items and prompts for further discussion was followed.
2.2.2.
Outcome measures
A brief user interview was administered which assessed:
1. Respondent characteristics: age, gender and sessions worked per week. 2. Overall acceptability: of the online program using a 6 item scale with a 5-point response scale (Strongly agree-strongly disagree).
420
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Table 1 – Overall acceptability ratings of the ABC’s of vitamin D online CME program (n = 12 GPs). Strongly agree n Course material was presented at a satisfactory level of detail The time taken to complete the program was reasonable The time spent viewing the content was worthwhile The presentation of information was clear and easily understandable The information was presented in a logical manner The program was easy to navigate ∗
Agree n
Neutral n
Disagree* n
1 1 2 0 0 0
0 0 1 0 1 1
2 2 0 1 0 1
9 9 9 11 11 10
There were no ‘Strongly disagree’ responses.
Table 2 – Participants ratings of the web-program modules (n = 12 GPs). Module
1 2 3 4 5 6 7 8
Topic
In addition, open-ended questions asked respondents to offer suggestions for improving the program, and whether they experienced technical difficulties using the program.
3.
Results
3.1.
Sample
Twelve GPs participated in the pilot study (60% response rate). Participants were aged between 31 and 60 years, 50% were male and worked between two and eight sessions per week (where each session corresponds to half a day).
Overall acceptability
Table 1 shows the overall acceptability rating. The results suggest high acceptability.
3.1.2.
Extremely
Very
moderately
Not at all
8 9 9 9 10 5 4 8
2 1 0 0 1 1 0 0
2 2 3 3 1 6 8 4
0 0 0 0 0 0 0 0
A brief background of vitamin D Causes and health consequences of vitamin D deficiency Measuring vitamin D deficiency Achieving adequate vitamin D in Australia Balancing UV exposure Groups at high risk of vitamin D deficiency Managing vitamin D deficiency Safety considerations
3. Module preferences: respondents were asked to rate the informativeness of each module on a 5-point response scale (extremely-not at all). 4. Perceived effectiveness and use: respondents were asked to rate the usefulness of the content and how likely they thought the program was to result in changes in patient care (very likely, somewhat likely, not very likely, not at all likely). Respondents were also asked whether they would use the program if it was an accredited CME activity (yes/no) and whether they would refer to the program when talking to patients about skin cancer and/or vitamin D related issues (yes/no). 5. Completion time: both self-reported and program autorecorded (in min).
3.1.1.
N of participants ratings of each module as informative
Module preferences
Participants (n = 12) ratings of each module as informative (on a scale of ‘extremely’, ‘very’, ‘moderately’, ‘somewhat’ and ‘not at all’) are presented in Table 2.
3.1.3.
No opinion 0 0 0 0 0 0 0 0
Perceived effectiveness
Ten of the 12 (83%) participants rated the program content as ‘very useful’ and two as ‘somewhat useful’. Five participants believed that the program was ‘very likely’ to result in changes to advice being given to patients and seven participants rated it as ‘somewhat likely’ to cause change in advice given to patients. Eleven of the 12 participants (92%) said they would use the program if it was an accredited CME activity and nine participants (75%) reported that they would refer to the program when talking to patients about related topics.
3.1.4.
Completion time
All participants completed the entire program and reported that it took between 1 and 3 h to complete the eight modules. According to the program timer, the average length of use was 124 min, with a range of 46 min to 170 min.
3.1.5. Qualitative feedback—General comments and technical issues Suggestions for improvements included reducing the introduction and Module 1, less detail in Modules 1, 3 and 7, higher quality graphics and illustrations, provision of an indication of how long the program takes, and reduced repetitiveness of content across some modules. Some technical issues were identified (such as the program ‘freezing’ when answers submitted, not being able to progress from one module to another, a case study not functioning at all, difficulty logging on using certain internet browsers) which were referred back to the software programmer who corrected the problems.
4.
Discussion
This study described the development and pilot testing of a new online CME program for GPs in the area of vitamin D and sun health. The study highlights the importance of pilot-testing, with target users, new online programs. Results
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
showed that development of a CME online program for GPs can be robust following a theory and evidence based design framework for physician education web site development [5]. Usability and acceptability testing with GPs indicated high ratings of ease of use, perceived effectiveness and potential use as a CME activity, acceptable completion time and useful information.
4.1.
Limitations
The findings and limitations of this study are being used to guide future research with the ‘ABC’s of vitamin D’ program. The program was pilot tested with a small sample. A sample size of between 10 and 20 users has been identified as adequate for software usability testing [30–32]. With fewer users, fewer problems are uncovered during testing, and including more than 20 users becomes redundant. However, it should be acknowledged that as members of a research network, the current group of physicians may have different learning capabilities, interests or familiarity with computer-based instruction than the broader population of primary care physicians. Another limitation with the pilot testing is that users were not observed using the program, as they were interviewed by telephone. An important question is whether the program will be effective at improving physician knowledge, practice or patient outcomes. Future research is planned to examine the effectiveness of the ABC’s of Vitamin D program using a randomised controlled trial with a large representative sample of GPs.
4.2.
Lessons learnt
Pilot studies offer opportunity to test programs and identify potential problems during development which may negatively affect adoption and implementation. Some participants expressed uncertainties regarding the vitamin D and sun exposure guidelines used in the current program. Defining the accurate amount of UVR exposure necessary for achieving adequate vitamin D status is a complex and emerging science. The existing recommendations and guidelines note the limits of the evidence base [8,24]. This uncertainty may limit the effectiveness of the program at changing the advice GPs provide to patients regarding sun exposure for vitamin D deficiency [27]. Nonetheless, this version of the program used the most current evidence available which seemed to be validated by high acceptability ratings of the content of the program. One of the advantages of an online CME program is that as the evidence firms up over time, the program can be updated to reflect advances in knowledge and understanding. Ratings of each module suggested that participants found the modules covering sun and UVR exposure for vitamin D and effectively communicating to patients a health message for balancing sun exposure with sun protection ‘most’ informative. No modules were rated as ‘not at all’ informative. Unlike all other modules which were rated as being ‘extremely’ or ‘very’ informative by the majority of participants, the module on the topic of ‘managing vitamin D deficiency’ was rated by most participants as being ‘moderately’ or ‘somewhat’ informative. Some of the comments received on this module suggest that the way the information was presented may not
421
Summary points What was already known on the topic? • The use of the internet as a mechanism to deliver continuing medical education (CME) has been increasing in popularity. • As use of online CME grows, it is increasingly important that the online programs offered are well designed and grounded in sound education theory and evidence. What this study added to our knowledge? • This study applied a set of principles for the design of online CME programs aimed at increasing the likelihood that they will change clinical practice and improve patient outcomes in the development of an online CME program for general practitioners (GPs) on the topic of vitamin D. • Currently, most CME online programs do not use these principles. • This study showed that a program using online design principles was rated highly acceptable and easy to use by GPs. • Greater acceptability and ease of use increases the likelihood that the program will be used by GPs and that it will translate into changes in clinical practice.
have been practical, for example the module did not present dosage recommendations of supplements as monthly doses, which is the dose most often used. Other comments on Module 7 included repetitiveness with other modules, that it was “longer” than other modules and that it was more “confusing” than other modules. Changes to the module were made based on participant comments to ensure it was practical to users. In conclusion, a theory and evidence based approach may facilitate the development of an acceptable and informative online CME program for GPs. Further research is needed to examine whether the online CME program is effective at changing GP practices and improving patient outcomes.
Funding The research was funded by a Cancer Institute NSW Research Innovation Grant (10/RFG/2-38). At the time of the study BB was supported by a Cancer Institute NSW Career Development Fellowship (10/CDF/2-40).
Conflict of interest The authors declare no conflicts of interest.
Author contributions BB conceived of the study, MP and HG recruited participants, all authors contributed to data collection, analysis and write up of the manuscript.
422
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 4 ( 2 0 1 5 ) 413–422
Acknowledgements Thanks to Mr Mark Wallis for the computer programming and Ms Raelene Monahan for assistance with data collection. Thanks to the National Prescribing Service (NPS) Quality Use of Medicines for assistance developing the vitamin D related content and our experts and GPs who participated in pilot testing for giving their time.
references
[1] L. Forsetlund, A. Bjørndal, A. Rashidian, G. Jamtvedt, M.A. O’Brien, F.M. Wolf, D. Davis, J. Odgaard-Jensen, A.D. Oxman, Continuing education meetings and workshops: effects on professional practice and health care outcomes, Cochrane Database Syst. Rev. (2) (2009) (Art. No.: CD003030. PMID: 19370580). [2] M. Mansouri, J. Lockyer, A meta-analysis of continuing medical education effectiveness, J. Contin. Educ. Health Prof. 27 (1) (2007) 6–15. [3] B. Sklar, Online Continuing Medical Education, July 2009, Available at www.cmelist.com/mastersthesis/. Archived at http://www.webcitation.org/6KtRtAXC3 (accessed 23 November 2013). [4] G. Ryan, P. Lyon, K. Kumar, J. Bell, S. Barnet, T. Shaw, Online CME: an effective alternative to face-to-face delivery, Med. Teach. 29 (8) (2007) e251–e257. [5] L.L. Casebeer, S.M. Strasser, C.M. Spettell, T.C. Wall, N. Weissman, M.N. Ray, J.J. Allison, Designing tailored web-based instruction to improve practising physicians’ preventive practices, J. Med. Internet Res. 5 (3) (2003) e20. [6] B. Bonevski, A. Girgis, P. Magin, G. Horton, I. Brozek, B. Armstrong, Prescribing sunshine: a survey of general practitioners’ knowledge, attitudes and practices relating to the sun and vitamin D, Int. J. Cancer 130 (2012) 2138–2145. [7] M.F. Holick, Vitamin D deficiency, N. Engl. J. Med. 357 (2007) 266–281. [8] Working Group of the Australian and New Zealand Bone and Mineral Society, Endocrine Society, and Osteoporosis Australia et al., Vitamin D and adult bone health in Australia and New Zealand: a position statement, Med. J. Aust. 182 (6) (2005) 281–285. [9] M.F. Holick, Sunlight and vitamin D for bone health and prevention of autoimmune diseases, cancers and cardiovascular disease, Am. J. Clin. Nutr. 80 (6) (2004) 1678S–1688S. [10] C.F. Garland, F.C. Garland, E.D. Gorham, M. Lipkin, et al., The role of vitamin D in cancer prevention, Am. J. Public Health 96 (2006) 252–261. [11] S.V. Ramagopalan, A. Heger, A.J. Berlanga, et al., A ChIP-seq defined genome-wide map of vitamin D receptor binding: associations with disease and evolution, Genome Res. 20 (2010) 1352–1360. [12] M.L. Melamed, E.D. Michos, W. Post, B. Astor, 25-Hydroxyvitamin D levels and the risk of mortality in the general population, Arch. Intern. Med. 168 (15) (2008) 1629–1637. [13] P. Autier, S. Gandini, Vitamin D supplementation and total mortality: a meta-analysis of randomized controlled trials, Arch. Intern. Med. 167 (16) (2007) 1730–1737 (10). [14] A.R. Webb, B.R. de Costa, M.F. Holick, Sunlight regulates the cutaneous production of vitamin D3 by causing its photodegradation, J. Clin. Endocrinol. Metab. 68 (1989) 882–887.
[15] B. Dawson-Hughes, R.P. Heaney, M.F. Holick, P. Lips, et al., Estimates of optimal vitamin D status, Osteoporos. Int. 16 (7) (2005) 713–716. [16] B.W. Hollis, C.L. Wagner, Normal serum vitamin D levels, N. Engl. J. Med. 352 (5) (2005), 515–515. [17] B. Diffey, Do we need a revised public health policy on sun exposure, Br. J. Dermatol. 154 (2006) 1046–1051. [18] D. Gillie, A new government policy is needed for sunlight and vitamin D, Br. J. Dermatol. 154 (2006) 1052–1061. [19] J. Moan, A.C. Porojnicu, A. Dahlback, R.B. Setlow, Addressing the health benefits and risks, involving vitamin D or skin cancer, of increased sun exposure, Proc. Natl. Acad. Sci. 105 (2) (2008) 668–673. [20] T. Vik, K. Try, J.H. Stromme, The vitamin D status of man at 70 degrees north, Scand. J. Clin. Lab. Invest. 40 (1980) 227–232. [21] M. Brustad, et al., Vitamin D status of middle-aged women at 65–71 degrees north in relation to dietary intake and exposure to ultraviolet radiation, Public Health Nutr. 7 (2004) 327–335. [22] J.J. McGrath, M.G. Kimlin, S. Saha, et al., Vitamin D insufficiency in south-east Queensland, Med. J. Aust. 174 (2001) 150–151. [23] J.A. Pasco, M.J. Henry, G.C. Nicholson, et al., Vitamin D status of women in the Geelong Osteoporosis Study: association with diet and casual exposure to sunlight, Med. J. Aust. 175 (2001) 401–405. [24] Australian New Zealand Bone and Mineral Society, Osteoporosis Australia, The Australasian College of Dermatologists and the Cancer Council Australia, Risks and benefits of sun exposure position statement, updated 2007, in: Osteoporosis Australia. Calcium, Vitamin D and Osteoporosis. A Guide for GPs, second ed., Australian New Zealand Bone and Mineral Society, Osteoporosis Australia, The Australasian College of Dermatologists and the Cancer Council Australia, 2008, Available online at http:/www.osteoporosis.org.au/files/factsheets/OTH-7665ENG.pdf (accessed 23 November 2013. [25] E.M. Rogers, Diffusion of Innovations, Free Press, New York, NY, 1995, ISBN 1451602472. [26] A.C. Ross, C.L. Taylor, A.L. Yaktine, H.B Del Valle (Eds.), Committee to Review Dietary Reference Intakes for Vitamin D and Calcium, Institute of Medicine, The National Academies, Washington, 2011, ISBN 9780309163941. [27] D. Davis, M.A. Thomson, A.D. Oxman, R.B. Haynes, Changing physician performance. A systematic review of the effect of continuing medical education strategies, J. Am. Med. Assoc. 274 (9) (1995) 700–705. [28] D.W. Farrell, L. Olevitch, L.K. Brennan, M. Kreuter, in: M. Kreuter (Ed.), Tailoring Health Messages: Customizing Communication with Computer Technology, Psychology Press, Mahwah, NJ, 1999, ISBN 1410603318. [29] Royal Australian College of General Practice. http://www.racgp.org.au/. Archived at http://www.webcitation.org/6KtRdxrzd (accessed 23 November 2013). [30] L. Faulkner, Beyond the five-user assumption: benefits of increased sample sizes in usability testing, Behav. Res. Methods Instrum. Comput. 35 (3) (2003) 379–383. [31] K.L. Norman, E. Panizzi, Levels of automation and user participation in usability testing, Int. Comput. 18 (2006) 246–264. [32] B. Shneiderman, Designing the User Interface: Strategies for Effective Human–Computer Interaction, fourth ed., Addison-Wesley, Reading, MA, 2005, ISBN 0321197860.
