Practical Radiation Oncology (2014) xx, xxx–xxx
www.practicalradonc.org
Original Report
Usability study of the EduMod eLearning Program for contouring nodal stations of the head and neck Rohan Deraniyagala MD a , Robert J. Amdur MD a,⁎, Arthur L. Boyer PhD b , Scott Kaylor MEd b a
Department of Radiation Oncology, University of Florida College of Medicine, Gainesville, Florida Radiation Oncology eLearning Center, Inc, Fremont, California
b
Received 3 October 2014; revised 22 October 2014; accepted 23 October 2014
Abstract Purpose: A major strategy for improving radiation oncology education and competence evaluation is to develop eLearning programs that reproduce the real work environment. A valuable measure of the quality of an eLearning program is “usability,” which is a multidimensional endpoint defined from the end user’s perspective. The gold standard for measuring usability is the Software Usability Measurement Inventory (SUMI). The purpose of this study is to use the SUMI to measure usability of an eLearning course that uses innovative software to teach and test contouring of nodal stations of the head and neck. Methods and materials: This is a prospective institutional review board–approved study in which all participants gave written informed consent. The study population was radiation oncology residents from 8 different programs across the United States. The subjects had to pass all sections of the same 2 eLearning modules and then complete the SUMI usability evaluation instrument. We reached the accrual goal of 25 participants. Usability results for the EduMod eLearning course, “Nodal Stations of the Head and Neck,” were compared with a large database of scores of other major software programs. Results were evaluated in 5 domains: Affect, Helpfulness, Control, Learnability, and Global Usability. Results: In all 5 domains, usability scores for the study modules were higher than the database mean and statistically superior in 4 domains. Conclusions: This is the first study to evaluate usability of an eLearning program related to radiation oncology. Usability of 2 representative modules related to contouring nodal stations of the head and neck was highly favorable, with scores that were superior to the industry standard in multiple domains. These results support the continued development of this type of eLearning program for teaching and testing radiation oncology technical skills. © 2014 Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
Conflicts of interest: R.J.A., A.L.B., and S.K. are partners in the company that owns and administrates the EduMod eLearning Program. ⁎ Corresponding author. 2000 SW Archer Road, PO Box 100385, Gainesville, FL 32610–0385. E-mail address: [email protected] (R.J. Amdur).
Introduction Multiple advisory organizations are directing changes to strengthen medical education and testing during training
http://dx.doi.org/10.1016/j.prro.2014.10.008 1879-8500/© 2014 Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
2
R. Deraniyagala et al
as well as after training to maintain specialty certifications. 1-3 Probably the most comprehensive evaluation of this subject is the Carnegie Foundation report, in which a major recommendation is to “[s]tandardize learning outcomes through objective assessment of competencies and to individualize the learning process within and across levels." 2 The Accreditation Council for Graduate Medical Education (ACGME) has embraced these recommendations by revising the new Program Requirements for Radiation Oncology residency training to include multiple items that require the resident to “demonstrate competence." 4 For radiation oncologists who have completed residency, the American Board of Radiology has embraced these recommendations in a major initiative to expand the Maintenance of Certification program so that physicians must demonstrate competence in a wide range of skills provided they are active in clinical practice. 3 To respond to these directives to improve medical education and competence testing, many fields are using innovative technology and eLearning platforms that model the real-work environment. In this article, we use the term “eLearning" to mean an education program in which the text, images, and audio content are delivered via a computer in electronic format. The field of anesthesiology is probably the most advanced in documenting the value of creating virtual environments to teach and test technical skills, but other specialties are publishing on this subject. 5 The potential is high for radiation oncology to benefit from eLearning teaching and testing because much of our work relies on a computer monitor—especially image analysis and treatment planning. Several groups are developing interactive eLearning programs related to radiation oncology: Radiotherap-e for technical support staff skills (Oncology Systems Limited, Shrewsbury, Shropshire, UK), the McGill University module on gynecologic anatomy, 6 and the TaCTICS program to test target contouring. 7 A description of these projects is beyond the scope of this article. The bottom line is that all are promising but none are in widespread use such that there is room for additional approaches. In 2011, we established a collaboration to develop a comprehensive software platform for teaching and testing radiation oncology knowledge and skills in an environment that is similar to clinical practice. The name of the program is EduMod (Rad Onc eLearning Center, Inc, Fremont, CA) because the platform is organized as education modules. The foundation of the EduMod program is the EduCase (Rad Onc eLearning Center, Inc) software that makes it possible to contour on, and work with, a wide range of image sets in a web-based environment that is similar to what clinicians experience in clinical practice. Many radiation oncologists are familiar with the EduCase software because it is what clinicians use in most of the eContouring courses at meetings of the American Society for Radiation Oncology and the European Society for Radiotherapy and Oncology. A
Practical Radiation Oncology: Month 2014
video tutorial about the modules that are the subject of this study can be accessed at http://vimeo.com/61734844. The first course in the EduMod program was released for commercial use in April 2013 and is limited to 11 modules related to nodal stations of the head and neck (www.educase.com). Since April 2013, this course has been used by physicians in all levels of training from all over the United States. From this experience, we have modified technical elements of the program in response to user feedback. This article reports the results of a formal study of the current version of the program. Education scientists have struggled for decades with how to evaluate the quality of an educational activity in general and eLearning programs in particular. 8,9 The few publications that evaluate eLearning programs specifically designed for radiation oncology use test performance as the main indication of the value of the activity. 6,7 The limitation of a test performance endpoint is related to 3 main factors. (1) The act of testing improves performance (testing is teaching). (2) Almost any activity designed to prepare a student for a test will improve test performance, yet what we want to know is how a given activity compares with alternatives in terms of a wide range of factors such as time, difficulty, and all the qualities that determine user-friendliness. (3) It is very difficult to conduct comparative studies that control for factors that may have an important influence on test performance—such as study time, learner ability, and performance motivation. In view of the limitations of the test performance endpoint, there is now a large body of literature suggesting that the best way to evaluate the quality of an eLearning program is to measure a psychological property called “usability,” which is a multidimensional endpoint defined from the end user’s perspective. 10-12 There are multiple approaches to measuring usability, but the “gold standard" is the Software Usability Measurement Inventory (SUMI) because it has been extensively validated with a large database of software programs that have a long track record of success in university courses and the commercial market. 12 The specific purpose of this paper is to report the results of a prospective study of SUMI-score usability for a set of eLearning modules that teach and test concepts and contouring of nodal stations of the head and neck. A more general purpose is to introduce radiation oncology educators to usability evaluation of eLearning programs that teach or test radiation oncology skills.
Methods and materials This is a prospective, institutional review board– approved study in which all participants gave written informed consent. The study population is 25 resident physicians from 8 ACGME-accredited radiation oncology residency programs in the United States. This number of
Practical Radiation Oncology: Month 2014 Table 1
Usability study of the EduMod eLearning Program
3
Study population (25 participants)
Residency program
Residency program year First
University of Miami (Miami, FL) Moffitt Cancer Center (Tampa, FL) University of North Carolina (Chapel Hill, NC) University of Chicago (Chicago, IL) Beaumont Health System (Royal Oak, MI) Mayo Clinic (Rochester, MN) University of Utah (Salt Lake City, UT) University of Southern California (Los Angeles, CA)
Second
Third
Fourth
3
1
1
2
1
1 1 1
1 1
2
participants was determined by the power analysis described later. A synopsis of the study is that participants completed all sections of the same 2 modules in the EduMod eLearning course about nodal stations of the head and neck and then recorded their evaluation with the SUMI survey of usability.
Study population The potential incentives for participating in this study were the educational value of the modules and contributing to the improvement of education programs in radiation oncology. There was no financial compensation for participating in this study. Participants were given access to the 2 study modules without having to pay a subscription fee. As these modules are available through the educase.com website for a subscription fee of $50, a potential incentive to participate in this study was to have access to the modules without having to pay this fee. However, this financial issue was not explained in the solicitation information other than to clarify that there was no cost, or remuneration, for participating in the study. Participation in the study was solicited by email in a 2-step process: First, Dr Amdur e-mailed the residency program director a brief description of the study, the informed consent document that resident participants would need to sign, and a letter requesting approval to contact residents directly to solicit participation. After Dr Amdur received e-mail approval from the program director, Dr Deraniyagala contacted as many residents in the program as he could obtain e-mails for to request study participation. In some cases, residents arranged a phone conversation with Dr Deraniyagala to understand the details of study participation better before deciding
2
2 1
1
2 2
whether to participate. Residents who had previously worked with an EduMod module and all residents from our program were prohibited from participating. The goal was to sample residents from various programs and from all years of residency. There was no specific goal for the number of programs or number of residents in a given program year. We targeted programs in most geographic regions of the country with representation from both private and public institutions. We solicited participation from 8 programs, and the program director at all of these programs approved us to contact their residents. Table 1 lists the programs and participants by program year. With 6 of the 8 programs, Dr Amdur knew the program directors and with 2 programs they had never met. None of the program directors from the programs that participated had any financial or professional interest in the outcome of our study; the program directors did not participate in the study; and the program directors did not get information on the performance of their residents on the study modules. Neither Dr Amdur nor Dr Deraniyagala knew the residents who participated in this study beyond superficial acquaintance in a few cases.
Study modules All participants completed the same 2 modules: Introduction to Nodal Stations of the Head and Neck and Level 1 Nodal Station. These modules were chosen because they are the first 2 modules in the course and because they contain testing of both factual knowledge and contouring skills. Before accessing the modules, Dr Deraniyagala gave each participant a brief overview by phone conversation or web conference of his or her requirements and how to access the modules and SUMI evaluation.
4
R. Deraniyagala et al
Figure 1
Practical Radiation Oncology: Month 2014
Examples of statements in the Software Usability Measurement Inventory (SUMI).
Study participation meant that each participant completed all 3 sections of both modules consisting of an Overview Lecture, Practice Section, and Examination(s). In the Introduction to Nodal Stations module, the only examination is a test with 50 multiple choice questions— many with associated images that test radiographic anatomy. In the level 1 module, there are 2 examinations: a multiple choice test and a contouring examination that is graded with a similarity metric program that quantitates how closely the student contour matches the reference contour. Completion of each module required the participant to score at least 70% on all examinations. The Lecture and Practice components of the modules have no time limit. The examinations, however, are time-limited. It usually takes 1.5-3 hours of work time to complete all sections of a module, but we did not attempt to record the time a participant spent working with a module. Participants were not allocated time by their residency programs to complete these modules. Review of the study modules was not supervised by the program director or any other person. Participants were permitted to work on the study modules on their home or work network, whichever they found most convenient. We asked each participant to complete the study modules (and evaluation) within 2 weeks of signing the consent document; all participants complied with this request.
SUMI evaluation After completing the 2 study modules, each participant evaluated usability of the modules as a group by completing the SUMI survey. The SUMI survey consists of 50 statements in which the participants respond with “agree,” “undecided,” or “disagree” (Fig 1).
The SUMI is copyrighted, and investigators must pay a fee to use the SUMI survey and to get benchmarked results. All activities related to the SUMI evaluation are handled by the company that administers the SUMI. We sent the SUMI administrators the e-mail addresses of the participants and they, in turn, e-mailed the participants a link to the online SUMI evaluation to complete. The SUMI administrator converts the survey responses into a usability score that is benchmarked to the scores of more than 50 software programs in the SUMI database. The programs in the SUMI database have a track record of success in university courses—mainly in engineering and the biologic sciences—or the general use market, such as the Microsoft Office Suite (Microsoft Corporation, Redmond, WA). The composition of the database is such that a favorable result for the software being studied is to have a SUMI score that is close to the database mean (set at a score of 50). A SUMI score is reported for 5 separate domains of usability: Affect is the user’s general emotional reaction to the software. Helpfulness is the degree to which the software is self-explanatory. Control measures how much the user feels in control of the navigation through the program. Learnability measures the facility of mastering the system, that is, how easy it is to learn how to use the software. Last, Global Usability is a kind of bottom-line composite measure of usability that describes the user’s generalized perceived quality of use. The SUMI process allowed us to add a nonvalidated statement to the 50-item SUMI survey. The statement that we added focused on the relative value of the study modules to other educational resources: “Rate the educational value of these modules compared to other resources you are familiar with,” with the response choices being, “less valuable,” “same value,” and “more valuable.”
Practical Radiation Oncology: Month 2014
Usability study of the EduMod eLearning Program
5
Statistical analyses The SUMI questionnaire contains 5 subscales that were extracted using factor analytic methods. 12 Each subscale was standardized using a “basket of goods" approach: taking a relatively fixed set of consumer products and services valued and used on a regular basis (eg, word processors, databases) to determine global user satisfaction levels with information technology. 13 SUMI data are presented as T-scores. A T-score is a standardized score that is calculated from the total distribution of scores from the basket of goods standardization. Scores are rescaled so that T-scores have a mean of 50 and a standard deviation of 10. Scores within 1 standard deviation (ie, a T-score N 50 and ≤ 60) on any dimension are regarded as being within the normal range on that dimension. Scores N 60 or b 40 are regarded as exceptional. Student t-test was used to compare a study result within each domain to the database mean of 50 in that domain. Participant number was set at 25 based on the following assumptions: SUMI scales are T-scores, which have a 50/10 distribution (mean = 50; standard deviation = 10). Assuming an effect size of ~ 0.5, the sample size would need to be between 20 and 25 to have 95% confidence intervals that do not include the population mean of 50. On that basis, we would be able to reject the null hypothesis that our sample is only of average satisfaction (ie, no difference between the sample average and the population average).
Results No participant reported a technical error with any of the websites or software associated with this study. Figure 2 shows the SUMI usability scores for the 2 study modules. In this plot, usability scores are normalized to a database mean of 50. In all domains, the usability score of the study modules was higher than the database mean. The difference between the usability score of the study modules and the database mean was statistically significant in 4 of the 5 domains: Affect, Helpfulness, Learnability, and Global Usability. Results for the statement, “Rate the educational value of these modules compared to other resources you are familiar with” were as follows: less valuable, 0%; same value, 24%; more value, 76%.
Discussion The main contribution of our project is that it is the first study to evaluate usability of an eLearning program in the field of radiation oncology. We studied the usability of 2 representative modules in the EduMod course, Nodal
Figure 2 Usability scores for the 2 EduMod modules compared with other eLearning programs in the Software Usability Measurement Inventory (SUMI) database. Scores are normalized relative to a scale value of 50, which is the mean score of all software programs in the SUMI data. P values comparing the SUMI scale score of the study modules to the database mean: Efficiency, .145; Affect, .003; Helpfulness, .001; Control, .828; Learnability, .007; and Global Usability, .043.
Stations of the Head and Neck. Usability results for these modules were highly favorable, with scores that were superior to the industry standard in multiple domains. These results support the continued development of this type of eLearning program for teaching and testing radiation oncology technical skills. A limitation of our study is that the 25 residents who participated are not a random sample and therefore may not be representative of residents in general. Studies such as ours are important because the path to progress in radiation oncology education and competence verification will likely involve eLearning programs that use innovative software to reproduce the real-work environment. To develop new programs, or evaluate the performance of existing programs in specific settings, radiation oncology educators will need to understand the role of usability testing. In the words of eLearning expert Adem Kilavuz, “From the point of view of people who need to use an interactive software system, usability is the most important aspect of the system.” 14 Usability goes beyond the concept of user-friendliness. The International Organization for Standardization defines usability as “the extent to which a product can be used by specified users to achieve specified goals, with effectiveness, efficiency, and satisfaction in a specified context of use.” 15 Usability summarizes how specific users, in a specific setting, work with a specific program. Usability evaluates the program–user interaction but it is not a direct measure of the content of the program being studied. A program may have the potential to accomplish the intended purpose, yet it will not accomplish this purpose if it has poor usability. In our study, we chose to evaluate usability with the SUMI because it has a mature questionnaire that has been validated
6
R. Deraniyagala et al Table 2
Practical Radiation Oncology: Month 2014
Resources about evaluating the usability of eLearning programs
Program
Comment a
General (public) usabilitynet.org/tools/r_questionnaire.htm SUMI (private) a sumi.ucc.ie WAMMI (private) a wammi.com QUIS (private) a cs.umd.edu/hcil/quis/ SUS (public) a measuringusability.com/sus.php CSUQ (public) a hcibib.org/perlman/question.cgi
Short paragraphs summarizing contact information and features of the main usability evaluation programs. Mature questionnaire with extensive standardization database that has been validated in multiple published studies. Relatively new questionnaire backed up by an extensive standardization database related to that of the SUMI. Minimal standardization and validation data but frequently used in the literature. Mature questionnaire but minimal standardization data. This is probably the most widely used public domain questionnaire. Respected public domain tool but minimal standardization data.
CSUQ, Computer System Usability Questionnaire; QUIS, Questionnaire for User Interaction Satisfaction; SUMI, Software Usability Measurement Inventory; SUS, System Usability Scale; WAMMI, Web Site Analysis and Measurement Inventory. a Programs in the private domain require written permission, and usually a fee, to use. Questionnaires in the public domain may be used without permission, for free.
in multiple studies 16-18 and a sophisticated database that allows user results to be benchmarked against industry standards. 12 But the SUMI is not the only option for evaluating the usability of an eLearning program. Table 2 lists major resources that radiation oncology educators may find useful in learning about usability testing programs. The 5 programs in this table are the main ones that have been used in published studies, but a web search will show many more. Major distinguishing features of the different programs are the extent of database standardization and if the program is in the private or public domain. Database standardization is important because it establishes a benchmark for comparing the results of the study program. Commercial domain is important because programs in the private domain usually require a fee to use the questionnaire and to get access to benchmark results. The cost to us for using the SUMI in the study described in this article was $2700. We did not evaluate other usability testing programs before choosing the SUMI, so we do not have additional comparison data or comments.
Conclusion It is important to evaluate the usability of eLearning programs in the setting in which they will be used. This study demonstrates the evaluation of usability of 2 eLearning modules related to Nodal Stations of the Head and Neck. The favorable results in our study support the continued development of these types of modules for the teaching and testing of radiation oncology skills, including image interpretation and contouring. The information in this article will be useful for radiation oncology educators who want to understand the role of usability testing in evaluating eLearning programs and to decide which tools to use in future projects.
References 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—Rationale and benefits. N Engl J Med. 2012;366:1051-1056. 2. Cooke M, Irby D, O'Brien B. Educating Physicians: A Call for Reform of Medical School and Residency. New York, NY: Jossey-Bass; 2010. 3. Becker GJ, Bosma JL, Guiberteau MJ, Gerdeman AM, Frush DP, Borgstede JP. ABR examinations: The why, what, and how. Int J Radiat Oncol Biol Phys. 2013;87:237-245. 4. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Radiation Oncology. Available at: http://www.acgme.org/acgmeweb/tabid/149/ ProgramandInstitutionalAccreditation/Hospital-BasedSpecialties/ RadiationOncology.aspx. Accessed September 15, 2014. 5. Taekman JM, Shelley K. Virtual environments in healthcare: Immersion, disruption, and flow. Int Anesthesiol Clin. 2010;48:101-121. 6. Alfieri J, Portelance L, Souhami L, et al. Development and impact evaluation of an e-learning radiation oncology module. Int J Radiat Oncol Biol Phys. 2012;82:e573-e580. 7. Fuller CD, Kalpathy-Cramer J, Choi J, et al. Prospective evaluation of an online atlas-based educational intervention on head and neck organ-at-Risk (OAR) and lymph node level (LNL) contouring: A pilot feasibility study using web-based feedback and analytic software. Int J Radiat Oncol Biol Phys. 2011;81:S161. 8. Squire D, Preece J. Usability and learning: Evaluating the potential of educational software. Comp Educ. 1996;27:15-22. 9. McDaniel MA, Roediger III HL, McDermott KB. Generalizing test-enhanced learning from the laboratory to the classroom. Psychon Bull Rev. 2007;14:200-206. 10. Ardito C, Costabile M, De Marsico M, et al. An approach to usability evaluation of e-learning applications. Univ Access Inf Soc. 2006;4:270-283. 11. Zaharias P, Poylymenakou A. Developing a usability evaluation method for e-learning applications: Beyond functional usability. Int J Hum-Comput Int. 2009;25:75-98. 12. Kirakowski J, Corbett M. SUMI: The Software Usability Measurement Inventory. Br J Educ Technol. 1993;24:210-212. 13. Kirakowski J. The software usability measurement inventory:Background and usage. Usability Eval Industry. 1996:169-178. 14. Kilavuz A. EDPC 5012: Evaluating ICT-based learning innovation assignment III. Available at: http://www.academia.edu/1285380/ A_Usability_Evaluation_Guide_for_eLearning_Courses. Accessed September 15, 2014.
Practical Radiation Oncology: Month 2014 15. Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. Geneva, Switzerland: International Organization for Standardization; 1998. 16. Laugwitz B, Held T, Schrepp M. Construction and evaluation of a user experience questionnaire. 2008:63-76.
Usability study of the EduMod eLearning Program
7
17. Kirakowski J, Claridge N. Taking the measure of user-centered design. Cutter IT J. 2003;16:21-27. 18. Nunes I, Kirakowski J. Interfaces usability–Portuguese version of the Software Usability Measurement Inventory (SUMI). Proc Occup Safety Hygiene (SHO10). 2010:972-978.
www.practicalradonc.org
Original Report
Usability study of the EduMod eLearning Program for contouring nodal stations of the head and neck Rohan Deraniyagala MD a , Robert J. Amdur MD a,⁎, Arthur L. Boyer PhD b , Scott Kaylor MEd b a
Department of Radiation Oncology, University of Florida College of Medicine, Gainesville, Florida Radiation Oncology eLearning Center, Inc, Fremont, California
b
Received 3 October 2014; revised 22 October 2014; accepted 23 October 2014
Abstract Purpose: A major strategy for improving radiation oncology education and competence evaluation is to develop eLearning programs that reproduce the real work environment. A valuable measure of the quality of an eLearning program is “usability,” which is a multidimensional endpoint defined from the end user’s perspective. The gold standard for measuring usability is the Software Usability Measurement Inventory (SUMI). The purpose of this study is to use the SUMI to measure usability of an eLearning course that uses innovative software to teach and test contouring of nodal stations of the head and neck. Methods and materials: This is a prospective institutional review board–approved study in which all participants gave written informed consent. The study population was radiation oncology residents from 8 different programs across the United States. The subjects had to pass all sections of the same 2 eLearning modules and then complete the SUMI usability evaluation instrument. We reached the accrual goal of 25 participants. Usability results for the EduMod eLearning course, “Nodal Stations of the Head and Neck,” were compared with a large database of scores of other major software programs. Results were evaluated in 5 domains: Affect, Helpfulness, Control, Learnability, and Global Usability. Results: In all 5 domains, usability scores for the study modules were higher than the database mean and statistically superior in 4 domains. Conclusions: This is the first study to evaluate usability of an eLearning program related to radiation oncology. Usability of 2 representative modules related to contouring nodal stations of the head and neck was highly favorable, with scores that were superior to the industry standard in multiple domains. These results support the continued development of this type of eLearning program for teaching and testing radiation oncology technical skills. © 2014 Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
Conflicts of interest: R.J.A., A.L.B., and S.K. are partners in the company that owns and administrates the EduMod eLearning Program. ⁎ Corresponding author. 2000 SW Archer Road, PO Box 100385, Gainesville, FL 32610–0385. E-mail address: [email protected] (R.J. Amdur).
Introduction Multiple advisory organizations are directing changes to strengthen medical education and testing during training
http://dx.doi.org/10.1016/j.prro.2014.10.008 1879-8500/© 2014 Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
2
R. Deraniyagala et al
as well as after training to maintain specialty certifications. 1-3 Probably the most comprehensive evaluation of this subject is the Carnegie Foundation report, in which a major recommendation is to “[s]tandardize learning outcomes through objective assessment of competencies and to individualize the learning process within and across levels." 2 The Accreditation Council for Graduate Medical Education (ACGME) has embraced these recommendations by revising the new Program Requirements for Radiation Oncology residency training to include multiple items that require the resident to “demonstrate competence." 4 For radiation oncologists who have completed residency, the American Board of Radiology has embraced these recommendations in a major initiative to expand the Maintenance of Certification program so that physicians must demonstrate competence in a wide range of skills provided they are active in clinical practice. 3 To respond to these directives to improve medical education and competence testing, many fields are using innovative technology and eLearning platforms that model the real-work environment. In this article, we use the term “eLearning" to mean an education program in which the text, images, and audio content are delivered via a computer in electronic format. The field of anesthesiology is probably the most advanced in documenting the value of creating virtual environments to teach and test technical skills, but other specialties are publishing on this subject. 5 The potential is high for radiation oncology to benefit from eLearning teaching and testing because much of our work relies on a computer monitor—especially image analysis and treatment planning. Several groups are developing interactive eLearning programs related to radiation oncology: Radiotherap-e for technical support staff skills (Oncology Systems Limited, Shrewsbury, Shropshire, UK), the McGill University module on gynecologic anatomy, 6 and the TaCTICS program to test target contouring. 7 A description of these projects is beyond the scope of this article. The bottom line is that all are promising but none are in widespread use such that there is room for additional approaches. In 2011, we established a collaboration to develop a comprehensive software platform for teaching and testing radiation oncology knowledge and skills in an environment that is similar to clinical practice. The name of the program is EduMod (Rad Onc eLearning Center, Inc, Fremont, CA) because the platform is organized as education modules. The foundation of the EduMod program is the EduCase (Rad Onc eLearning Center, Inc) software that makes it possible to contour on, and work with, a wide range of image sets in a web-based environment that is similar to what clinicians experience in clinical practice. Many radiation oncologists are familiar with the EduCase software because it is what clinicians use in most of the eContouring courses at meetings of the American Society for Radiation Oncology and the European Society for Radiotherapy and Oncology. A
Practical Radiation Oncology: Month 2014
video tutorial about the modules that are the subject of this study can be accessed at http://vimeo.com/61734844. The first course in the EduMod program was released for commercial use in April 2013 and is limited to 11 modules related to nodal stations of the head and neck (www.educase.com). Since April 2013, this course has been used by physicians in all levels of training from all over the United States. From this experience, we have modified technical elements of the program in response to user feedback. This article reports the results of a formal study of the current version of the program. Education scientists have struggled for decades with how to evaluate the quality of an educational activity in general and eLearning programs in particular. 8,9 The few publications that evaluate eLearning programs specifically designed for radiation oncology use test performance as the main indication of the value of the activity. 6,7 The limitation of a test performance endpoint is related to 3 main factors. (1) The act of testing improves performance (testing is teaching). (2) Almost any activity designed to prepare a student for a test will improve test performance, yet what we want to know is how a given activity compares with alternatives in terms of a wide range of factors such as time, difficulty, and all the qualities that determine user-friendliness. (3) It is very difficult to conduct comparative studies that control for factors that may have an important influence on test performance—such as study time, learner ability, and performance motivation. In view of the limitations of the test performance endpoint, there is now a large body of literature suggesting that the best way to evaluate the quality of an eLearning program is to measure a psychological property called “usability,” which is a multidimensional endpoint defined from the end user’s perspective. 10-12 There are multiple approaches to measuring usability, but the “gold standard" is the Software Usability Measurement Inventory (SUMI) because it has been extensively validated with a large database of software programs that have a long track record of success in university courses and the commercial market. 12 The specific purpose of this paper is to report the results of a prospective study of SUMI-score usability for a set of eLearning modules that teach and test concepts and contouring of nodal stations of the head and neck. A more general purpose is to introduce radiation oncology educators to usability evaluation of eLearning programs that teach or test radiation oncology skills.
Methods and materials This is a prospective, institutional review board– approved study in which all participants gave written informed consent. The study population is 25 resident physicians from 8 ACGME-accredited radiation oncology residency programs in the United States. This number of
Practical Radiation Oncology: Month 2014 Table 1
Usability study of the EduMod eLearning Program
3
Study population (25 participants)
Residency program
Residency program year First
University of Miami (Miami, FL) Moffitt Cancer Center (Tampa, FL) University of North Carolina (Chapel Hill, NC) University of Chicago (Chicago, IL) Beaumont Health System (Royal Oak, MI) Mayo Clinic (Rochester, MN) University of Utah (Salt Lake City, UT) University of Southern California (Los Angeles, CA)
Second
Third
Fourth
3
1
1
2
1
1 1 1
1 1
2
participants was determined by the power analysis described later. A synopsis of the study is that participants completed all sections of the same 2 modules in the EduMod eLearning course about nodal stations of the head and neck and then recorded their evaluation with the SUMI survey of usability.
Study population The potential incentives for participating in this study were the educational value of the modules and contributing to the improvement of education programs in radiation oncology. There was no financial compensation for participating in this study. Participants were given access to the 2 study modules without having to pay a subscription fee. As these modules are available through the educase.com website for a subscription fee of $50, a potential incentive to participate in this study was to have access to the modules without having to pay this fee. However, this financial issue was not explained in the solicitation information other than to clarify that there was no cost, or remuneration, for participating in the study. Participation in the study was solicited by email in a 2-step process: First, Dr Amdur e-mailed the residency program director a brief description of the study, the informed consent document that resident participants would need to sign, and a letter requesting approval to contact residents directly to solicit participation. After Dr Amdur received e-mail approval from the program director, Dr Deraniyagala contacted as many residents in the program as he could obtain e-mails for to request study participation. In some cases, residents arranged a phone conversation with Dr Deraniyagala to understand the details of study participation better before deciding
2
2 1
1
2 2
whether to participate. Residents who had previously worked with an EduMod module and all residents from our program were prohibited from participating. The goal was to sample residents from various programs and from all years of residency. There was no specific goal for the number of programs or number of residents in a given program year. We targeted programs in most geographic regions of the country with representation from both private and public institutions. We solicited participation from 8 programs, and the program director at all of these programs approved us to contact their residents. Table 1 lists the programs and participants by program year. With 6 of the 8 programs, Dr Amdur knew the program directors and with 2 programs they had never met. None of the program directors from the programs that participated had any financial or professional interest in the outcome of our study; the program directors did not participate in the study; and the program directors did not get information on the performance of their residents on the study modules. Neither Dr Amdur nor Dr Deraniyagala knew the residents who participated in this study beyond superficial acquaintance in a few cases.
Study modules All participants completed the same 2 modules: Introduction to Nodal Stations of the Head and Neck and Level 1 Nodal Station. These modules were chosen because they are the first 2 modules in the course and because they contain testing of both factual knowledge and contouring skills. Before accessing the modules, Dr Deraniyagala gave each participant a brief overview by phone conversation or web conference of his or her requirements and how to access the modules and SUMI evaluation.
4
R. Deraniyagala et al
Figure 1
Practical Radiation Oncology: Month 2014
Examples of statements in the Software Usability Measurement Inventory (SUMI).
Study participation meant that each participant completed all 3 sections of both modules consisting of an Overview Lecture, Practice Section, and Examination(s). In the Introduction to Nodal Stations module, the only examination is a test with 50 multiple choice questions— many with associated images that test radiographic anatomy. In the level 1 module, there are 2 examinations: a multiple choice test and a contouring examination that is graded with a similarity metric program that quantitates how closely the student contour matches the reference contour. Completion of each module required the participant to score at least 70% on all examinations. The Lecture and Practice components of the modules have no time limit. The examinations, however, are time-limited. It usually takes 1.5-3 hours of work time to complete all sections of a module, but we did not attempt to record the time a participant spent working with a module. Participants were not allocated time by their residency programs to complete these modules. Review of the study modules was not supervised by the program director or any other person. Participants were permitted to work on the study modules on their home or work network, whichever they found most convenient. We asked each participant to complete the study modules (and evaluation) within 2 weeks of signing the consent document; all participants complied with this request.
SUMI evaluation After completing the 2 study modules, each participant evaluated usability of the modules as a group by completing the SUMI survey. The SUMI survey consists of 50 statements in which the participants respond with “agree,” “undecided,” or “disagree” (Fig 1).
The SUMI is copyrighted, and investigators must pay a fee to use the SUMI survey and to get benchmarked results. All activities related to the SUMI evaluation are handled by the company that administers the SUMI. We sent the SUMI administrators the e-mail addresses of the participants and they, in turn, e-mailed the participants a link to the online SUMI evaluation to complete. The SUMI administrator converts the survey responses into a usability score that is benchmarked to the scores of more than 50 software programs in the SUMI database. The programs in the SUMI database have a track record of success in university courses—mainly in engineering and the biologic sciences—or the general use market, such as the Microsoft Office Suite (Microsoft Corporation, Redmond, WA). The composition of the database is such that a favorable result for the software being studied is to have a SUMI score that is close to the database mean (set at a score of 50). A SUMI score is reported for 5 separate domains of usability: Affect is the user’s general emotional reaction to the software. Helpfulness is the degree to which the software is self-explanatory. Control measures how much the user feels in control of the navigation through the program. Learnability measures the facility of mastering the system, that is, how easy it is to learn how to use the software. Last, Global Usability is a kind of bottom-line composite measure of usability that describes the user’s generalized perceived quality of use. The SUMI process allowed us to add a nonvalidated statement to the 50-item SUMI survey. The statement that we added focused on the relative value of the study modules to other educational resources: “Rate the educational value of these modules compared to other resources you are familiar with,” with the response choices being, “less valuable,” “same value,” and “more valuable.”
Practical Radiation Oncology: Month 2014
Usability study of the EduMod eLearning Program
5
Statistical analyses The SUMI questionnaire contains 5 subscales that were extracted using factor analytic methods. 12 Each subscale was standardized using a “basket of goods" approach: taking a relatively fixed set of consumer products and services valued and used on a regular basis (eg, word processors, databases) to determine global user satisfaction levels with information technology. 13 SUMI data are presented as T-scores. A T-score is a standardized score that is calculated from the total distribution of scores from the basket of goods standardization. Scores are rescaled so that T-scores have a mean of 50 and a standard deviation of 10. Scores within 1 standard deviation (ie, a T-score N 50 and ≤ 60) on any dimension are regarded as being within the normal range on that dimension. Scores N 60 or b 40 are regarded as exceptional. Student t-test was used to compare a study result within each domain to the database mean of 50 in that domain. Participant number was set at 25 based on the following assumptions: SUMI scales are T-scores, which have a 50/10 distribution (mean = 50; standard deviation = 10). Assuming an effect size of ~ 0.5, the sample size would need to be between 20 and 25 to have 95% confidence intervals that do not include the population mean of 50. On that basis, we would be able to reject the null hypothesis that our sample is only of average satisfaction (ie, no difference between the sample average and the population average).
Results No participant reported a technical error with any of the websites or software associated with this study. Figure 2 shows the SUMI usability scores for the 2 study modules. In this plot, usability scores are normalized to a database mean of 50. In all domains, the usability score of the study modules was higher than the database mean. The difference between the usability score of the study modules and the database mean was statistically significant in 4 of the 5 domains: Affect, Helpfulness, Learnability, and Global Usability. Results for the statement, “Rate the educational value of these modules compared to other resources you are familiar with” were as follows: less valuable, 0%; same value, 24%; more value, 76%.
Discussion The main contribution of our project is that it is the first study to evaluate usability of an eLearning program in the field of radiation oncology. We studied the usability of 2 representative modules in the EduMod course, Nodal
Figure 2 Usability scores for the 2 EduMod modules compared with other eLearning programs in the Software Usability Measurement Inventory (SUMI) database. Scores are normalized relative to a scale value of 50, which is the mean score of all software programs in the SUMI data. P values comparing the SUMI scale score of the study modules to the database mean: Efficiency, .145; Affect, .003; Helpfulness, .001; Control, .828; Learnability, .007; and Global Usability, .043.
Stations of the Head and Neck. Usability results for these modules were highly favorable, with scores that were superior to the industry standard in multiple domains. These results support the continued development of this type of eLearning program for teaching and testing radiation oncology technical skills. A limitation of our study is that the 25 residents who participated are not a random sample and therefore may not be representative of residents in general. Studies such as ours are important because the path to progress in radiation oncology education and competence verification will likely involve eLearning programs that use innovative software to reproduce the real-work environment. To develop new programs, or evaluate the performance of existing programs in specific settings, radiation oncology educators will need to understand the role of usability testing. In the words of eLearning expert Adem Kilavuz, “From the point of view of people who need to use an interactive software system, usability is the most important aspect of the system.” 14 Usability goes beyond the concept of user-friendliness. The International Organization for Standardization defines usability as “the extent to which a product can be used by specified users to achieve specified goals, with effectiveness, efficiency, and satisfaction in a specified context of use.” 15 Usability summarizes how specific users, in a specific setting, work with a specific program. Usability evaluates the program–user interaction but it is not a direct measure of the content of the program being studied. A program may have the potential to accomplish the intended purpose, yet it will not accomplish this purpose if it has poor usability. In our study, we chose to evaluate usability with the SUMI because it has a mature questionnaire that has been validated
6
R. Deraniyagala et al Table 2
Practical Radiation Oncology: Month 2014
Resources about evaluating the usability of eLearning programs
Program
Comment a
General (public) usabilitynet.org/tools/r_questionnaire.htm SUMI (private) a sumi.ucc.ie WAMMI (private) a wammi.com QUIS (private) a cs.umd.edu/hcil/quis/ SUS (public) a measuringusability.com/sus.php CSUQ (public) a hcibib.org/perlman/question.cgi
Short paragraphs summarizing contact information and features of the main usability evaluation programs. Mature questionnaire with extensive standardization database that has been validated in multiple published studies. Relatively new questionnaire backed up by an extensive standardization database related to that of the SUMI. Minimal standardization and validation data but frequently used in the literature. Mature questionnaire but minimal standardization data. This is probably the most widely used public domain questionnaire. Respected public domain tool but minimal standardization data.
CSUQ, Computer System Usability Questionnaire; QUIS, Questionnaire for User Interaction Satisfaction; SUMI, Software Usability Measurement Inventory; SUS, System Usability Scale; WAMMI, Web Site Analysis and Measurement Inventory. a Programs in the private domain require written permission, and usually a fee, to use. Questionnaires in the public domain may be used without permission, for free.
in multiple studies 16-18 and a sophisticated database that allows user results to be benchmarked against industry standards. 12 But the SUMI is not the only option for evaluating the usability of an eLearning program. Table 2 lists major resources that radiation oncology educators may find useful in learning about usability testing programs. The 5 programs in this table are the main ones that have been used in published studies, but a web search will show many more. Major distinguishing features of the different programs are the extent of database standardization and if the program is in the private or public domain. Database standardization is important because it establishes a benchmark for comparing the results of the study program. Commercial domain is important because programs in the private domain usually require a fee to use the questionnaire and to get access to benchmark results. The cost to us for using the SUMI in the study described in this article was $2700. We did not evaluate other usability testing programs before choosing the SUMI, so we do not have additional comparison data or comments.
Conclusion It is important to evaluate the usability of eLearning programs in the setting in which they will be used. This study demonstrates the evaluation of usability of 2 eLearning modules related to Nodal Stations of the Head and Neck. The favorable results in our study support the continued development of these types of modules for the teaching and testing of radiation oncology skills, including image interpretation and contouring. The information in this article will be useful for radiation oncology educators who want to understand the role of usability testing in evaluating eLearning programs and to decide which tools to use in future projects.
References 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—Rationale and benefits. N Engl J Med. 2012;366:1051-1056. 2. Cooke M, Irby D, O'Brien B. Educating Physicians: A Call for Reform of Medical School and Residency. New York, NY: Jossey-Bass; 2010. 3. Becker GJ, Bosma JL, Guiberteau MJ, Gerdeman AM, Frush DP, Borgstede JP. ABR examinations: The why, what, and how. Int J Radiat Oncol Biol Phys. 2013;87:237-245. 4. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Radiation Oncology. Available at: http://www.acgme.org/acgmeweb/tabid/149/ ProgramandInstitutionalAccreditation/Hospital-BasedSpecialties/ RadiationOncology.aspx. Accessed September 15, 2014. 5. Taekman JM, Shelley K. Virtual environments in healthcare: Immersion, disruption, and flow. Int Anesthesiol Clin. 2010;48:101-121. 6. Alfieri J, Portelance L, Souhami L, et al. Development and impact evaluation of an e-learning radiation oncology module. Int J Radiat Oncol Biol Phys. 2012;82:e573-e580. 7. Fuller CD, Kalpathy-Cramer J, Choi J, et al. Prospective evaluation of an online atlas-based educational intervention on head and neck organ-at-Risk (OAR) and lymph node level (LNL) contouring: A pilot feasibility study using web-based feedback and analytic software. Int J Radiat Oncol Biol Phys. 2011;81:S161. 8. Squire D, Preece J. Usability and learning: Evaluating the potential of educational software. Comp Educ. 1996;27:15-22. 9. McDaniel MA, Roediger III HL, McDermott KB. Generalizing test-enhanced learning from the laboratory to the classroom. Psychon Bull Rev. 2007;14:200-206. 10. Ardito C, Costabile M, De Marsico M, et al. An approach to usability evaluation of e-learning applications. Univ Access Inf Soc. 2006;4:270-283. 11. Zaharias P, Poylymenakou A. Developing a usability evaluation method for e-learning applications: Beyond functional usability. Int J Hum-Comput Int. 2009;25:75-98. 12. Kirakowski J, Corbett M. SUMI: The Software Usability Measurement Inventory. Br J Educ Technol. 1993;24:210-212. 13. Kirakowski J. The software usability measurement inventory:Background and usage. Usability Eval Industry. 1996:169-178. 14. Kilavuz A. EDPC 5012: Evaluating ICT-based learning innovation assignment III. Available at: http://www.academia.edu/1285380/ A_Usability_Evaluation_Guide_for_eLearning_Courses. Accessed September 15, 2014.
Practical Radiation Oncology: Month 2014 15. Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. Geneva, Switzerland: International Organization for Standardization; 1998. 16. Laugwitz B, Held T, Schrepp M. Construction and evaluation of a user experience questionnaire. 2008:63-76.
Usability study of the EduMod eLearning Program
7
17. Kirakowski J, Claridge N. Taking the measure of user-centered design. Cutter IT J. 2003;16:21-27. 18. Nunes I, Kirakowski J. Interfaces usability–Portuguese version of the Software Usability Measurement Inventory (SUMI). Proc Occup Safety Hygiene (SHO10). 2010:972-978.
