e-Learning

Evaluating PLATO: postgraduate teaching and learning online Menna Brown, College of Medicine, Swansea University, UK Alison Bullock, The Cardiff Unit for Research and Evaluation in Medical and Dental Education (CUREMeDE), School of Social Sciences, Cardiff University, UK

This evaluation aimed to identify the costs and benefits of PLATO (postgraduate learning and teaching online)

SUMMARY Background: The use of the Internet as a teaching medium has increased rapidly over the last decade. PLATO (postgraduate learning and teaching online) was launched in 2008 by the e-learning unit (ELU) of Wales Deanery. Located within Learning@NHSWales, a Moodle virtual learning environment (VLE), it hosts a wide range of freely available courses and resources tailored to support the education, training and continuing professional development (CPD) needs of health care professionals working across the National Health Service (NHS) Wales. The evaluation aimed to identify the costs and benefits of PLATO, report

its value as attributed by users, identify potential cost savings and make recommendations. Methods: Five courses (case studies) were selected, representing the range of available elearning resources: e-induction; fetal heart monitoring; cervical screening; GP prospective trainers; and tools for trainers. Mixed methods were used: one-to-one qualitative interviews, focus group discussions and surveys explored user views, and identified individual and organisational value. Results: Qualitative findings identified six key areas of value for users: ELU support and guidance; avoidance of duplication and standardisation; central

10  2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 10–14

reference; local control; flexibility for learners; and specific features. Survey results (n = 72) indicated 72 per cent of consultants reported that PLATO was easy to access and user friendly. E-learning was rated as ‘very ⁄ important’ for CPD by 79 per cent of respondents. Key challenges were: access, navigation, user concerns, awareness and support. Discussion: PLATO supports education and helps deliver UK General Medical Council standards. Future plans should address the suggested recommendations to realise cost savings for NHS Wales and the Wales Deanery. The findings have wider applicability to others developing or using VLEs.

INTRODUCTION

Mixed approaches, which include e-learning, can better facilitate learning preferences

I

t is commonplace nowadays to use technology to deliver and enhance learning. In a meta-analysis of the effectiveness of Internet-based educational interventions for health care professionals, although there was no single message, Cook et al. reported studies that revealed gains from e-learning interventions.1 More recently, Cook, Levinson and Garside found ‘that Internetbased instruction promotes greater learning efficiency than non-computer methods’.2 However, they noted that efficiency can only be examined within a given course and context, and that efficiency is dependent on learner aptitude and course design. Nevertheless, there is considerable literature to indicate that e-learning is no less efficient or effective than traditional approaches.3,4 Thus, it can be more cost-effective, creating savings in terms of the costs of travel, training the trainers and infrastructure; however, these ‘savings’ are diminished in blended approaches, which include face-to-face activity. Genischen et al. concluded that mixed approaches, which include e-learning, can better facilitate learning preferences.5 The rapid increase in the use of e-resources for continuing medical education led Shortt et al. to call for ‘quality criteria for accredited online modules’.6 From their review they identified key themes (Box 1). In terms of impediments, the development of e-learning material can be costly, and requires specific skills and expertise.5 Access is important, and users can be limited by the availability of the Internet. Users, particularly adult learners, are only likely to use resources that match their learning needs. Quality assessment is an issue for both individuals and institutions.

OBJECTIVES The purpose of the evaluation was to identify the benefits and costs of PLATO (postgraduate learning and teaching online), thereby establishing its value to users (learners, course facilitators and managers), the Wales Deanery and NHS Wales.

the evaluators to effectively reach a variety of potential respondents, and afforded them a practical opportunity to share their experiences of using (or not) PLATO. Participants in the focus groups and interviews were asked to discuss their involvement with the course from their perspective as a

METHOD Following ethics approval from Cardiff University, five active courses (case studies) were selected that highlighted the range of resources available in PLATO and the variety of users (see Table 1). Recruitment of participants in the evaluation varied. Trainee doctors (case study 1) were invited to participate via e-mail, electronic flyer and face-to-face in a focus group discussion. Consultants, general practitioners (GPs), nurses and administrators (case studies 2, 3 and 4) were invited to participate via e-mail, and were then followed-up with a telephone conversation. Educational supervisors (case study 5) were invited via e-mail to complete the online survey. This mixed-methods approach generated specific examples of use, benefit and barriers, and enabled an in-depth exploration of how users interacted with PLATO and the value that they assigned to this engagement. This variety of data collection methods allowed

Box 1. Key themes identified by Shortt et al.6 Fits with the way physicians prefer to learn • Learn at own pace • Opportunity to reflect on change and practice implications • Permits needs-based, goaldirected learning Fits with physician content preferences • Systematic approach to content • Clinical problem- or case-based • Interactive with content, experts and peers • Facilitates change and practice improvement Technical aspects • User-friendly • Effective use of multimedia • Includes linked resources Evaluation component • Allows for learner self-assessment • Course evaluates itself and outcomes

 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 10–14 11

This variety of data collection methods allowed the evaluators to reach a variety of potential respondents

Table 1. Case studies and data collection method Case study

Users

Data collection method

1. e-Induction

Trainee doctors

Focus group and one one-to-one interview

2. CTG foetal heart monitoring pilot course

Midwives and consultants

One one-to-one interview

3. Cervical screening Wales (CSW) update

GPs and practice nurses

Focus group and one one-to-one interview

4. GP Prospective Trainer course

GPs who wish to become GP trainers

Four one-to-one interviews

5. Tools for Trainers (a suite of modules)

Educational supervisors (NHS consultants) of trainee doctors

Online survey (n = 72) and two one-to-one interviews

Box 2. Quotations from case studies to illustrate each theme Theme: ELU support and guidance Absolutely brilliant, absolutely brilliant. They are a credit to the deanery actually because they are a fantastic team and any time you can contact them and they will help you as much as they can. Postgraduate centre manager I think having him listen to what we wanted and then tell us what he could offer was a really helpful dialogue. Course facilitator Theme: Standardisation and avoidance of duplication The associate dean for (role) has been looking at improving the educational supervisors’ course for educators in hospitals, and we have been able to tell her that we have a load of information on PLATO and she can just have anything that she wants because she can then access that. So we can share between the disciplines as well as using it for GP training. Course facilitator Theme: A central point of reference The participants know that even after the course is finished that if there is something they want to check on they can go back and they can look on PLATO, and they can find whatever it was that they need to refresh themselves on. I think that the fact that it is an ongoing resource is helpful. Course administrator Theme: Local control It’s really simple, it’s just thinking of the scenarios, thinking through the scenarios really, and going through our smear taker guide to see what policies have changed, what they need to know and basically we are delivering the same information face-to-face as we are online. In fact, I actually use the scenarios for the teaching session face-to-face. Course facilitator ‘‘There is a lot of centres out there who may not have somebody who is like myself, who would need even more support from the e-learning guys. To make sure that the e-induction passport works you need that unit.’’ PG centre manager Theme: Flexibility for learner In today’s climate, where staff are finding it difficult to juggle work and education or to attend courses in work time, then e-learning has an important role in offering an alternative to staff. Educational supervisor

12  2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 10–14

learner, educator or organisational representative, and to provide scenarios and examples to illustrate their experiences of PLATO. Trigger questions included: ‘Could you describe your involvement with PLATO’, ‘What gains in learning (if any) have you seen as a result of using PLATO?’, ‘What benefits (if any) have arisen from using PLATO?’ and ‘What challenges or barriers (if any) have you encountered in using PLATO?’ Data were audio recorded and transcribed for analysis. An inductive thematic analysis framework was applied to the data, informed by the work of Robinson.7 Common themes were identified and developed through a process of triangulation across the data and researcher discussion. Findings were fed back to a meeting of users, facilitated by the National Programme Delivery Group for e-learning. The group validated the analysis and contributed to the final development of themes. The e-learning unit (ELU) reviewed the financial costs of providing PLATO, which informed final conclusions and recommendations.

RESULTS Through inductive thematic analysis, six areas of value for users were identified. Quotations from case study participants illustrate each theme (Box 2). These included

benefits both for organisations (the Wales Deanery and NHS Wales) and for individual learners.

a pace, place and time of one’s choosing, which online learning in general provides.

ELU support and guidance The support and innovation of ELU representatives during the consultation and development process of new courses, coupled with continued technical expertise, were considered to be crucial aspects of the service provided by the unit.

A number of challenges were also noted during the discussions, and areas of difficulty that inhibited use were identified. These included: varied connectivity; difficulties accessing PLATO or Learning@NHSWales, including access to required e-mail accounts; navigation difficulties, specifically user ability to locate individual courses within PLATO; and enrolment key confusion. A general lack of awareness of the range of services and resources offered by ELU and PLATO inhibited uptake.

Standardisation and avoidance of duplication PLATO has the capacity to reach an all-Wales, all-National Health Service (NHS) audience. The inclusion of courses for use by any health care professional in Wales, regardless of location, facilitated the central development of a range of e-resources, produced either by representatives from across Wales working together or by a local team, and shared. The approach means information is produced once, standardised and, as a result, avoids any duplication of effort. A central point of reference Course documents, materials, assessments and communications for courses are housed within one area, which all users are able to access from any location with an Internet connection. Local control PLATO offers flexibility and control, and encourages users to take full responsibility for facilitating, maintaining, monitoring and further developing their courses once they are ‘live’; however, with local control and administration rights comes the added responsibility and dependency upon staff with IT knowledge, ability and willingness to drive programmes forward at a local level, so as to continue to develop features, use media options and advertise PLATO to reach its intended audience. Flexibility for the learner: pace, place and time PLATO can be accessed from anywhere with Internet access, supporting the concept of learning at

A number of cost savings were identified. In summary, these related to: • low-cost resource to deanery and NHS staff; • time saving for deanery and NHS staff who used PLATO for online learning; • reduced travel costs, compared with face-to-face delivery; • reduced running costs, compared with face-to-face courses; • as a central resource, PLATO saves health boards duplicating materials across Wales; • reduced costs from information sharing;

Six areas of value for users were identified

• potential reduction in legal costs for NHS Wales.

SURVEY RESULTS Two hundred and sixteen NHS educational supervisors were e-mailed a short survey and 72 were returned: a response rate of 33 per cent. Of those who had used one or more of the modules within Tools for Trainers, 72 per cent indicated that PLATO was easy to access and user-friendly, and 68 per cent judged it easy to navigate. PLATO was rated ‘good’ or ‘very good’ by 87 per cent of respondents, and 87 per cent also agreed that they liked the structure of the course. Overall (all respondents), 79 per cent rated e-learning as either an ‘important’ or ‘very important’ tool for continuing professional development (CPD) (Table 2). These results indicate that educational supervisors engaging with the Tools for Trainer course view their experience and use positively.

Table 2. Survey results case study 5 Question

Response

Was PLATO easy to access?

Yes, 72%

Was PLATO easy to navigate?

Yes, 72%

Was PLATO user friendly?

Yes, 68%

Overall how would you rate PLATO?

Good or very good, 87%

How would you rate the course structure of Tools for Trainers?

Good or very good, 87%

Did you require support to use PLATO?

Yes, 30%

How important is e-learning for CPD?

Important or very important, 79%

 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 10–14 13

The availability of PLATO can contribute to the quality of the learning experience

vide continuing support postlaunch. Review courses periodically to ensure the resources available within PLATO are fully used and up to date. 3. Complete the e-induction passport scheme. This should ensure standardisation across Wales and avoid duplication, thus creating cost savings. Addressing these recommendation should direct and improve the future use of PLATO.

CONCLUSION AND RECOMMENDATIONS The combination of PLATO and the Wales Deanery ELU provides a unique service to doctors in NHS Wales. Interviews and focus group discussions generated positive feedback on PLATO and the service provided by ELU. By providing complementary education and training that can be undertaken at the learner’s own pace, place and time, the availability of PLATO can contribute to the quality of the learning experience. It supports the continuing professional development of consultants and doctors in training, and provides certificates for learning portfolios. From our analysis, the data suggest that in the last 2-year period, usage and interest in PLATO and technologyenhanced learning has grown considerably. PLATO enables the fast and accurate reporting of training and education, which can be provided to regulating bodies as evidence that the Wales Deanery is meeting its educational duties and responsibilities. In comparison with commercial equivalents, the indications are that PLATO, hosted in its current location (within the larger platform of Learning@NHSWales)

provides the Wales Deanery with a low-cost solution to education and training in the electronic context. The strength of this study is its use of mixed methods, which generated rich data exploring the users’ engagement with PLATO and its perceived value to individual learners, educators and the organisation. Limitations include the response rate to the online survey, which means that data should be interpreted with caution. Financial and time restraints limited the number of participants, which were few in some of the selected case studies. The case study approach enabled the evaluators to gain focused and content-specific data, which assisted with meaningful interpretation; however, although varied, the case studies represent only a partial picture of PLATO usage. Based on the evaluation, we make three recommendations. 1. Address the access and navigation issues identified by users that currently act as barriers to use. 2. Provide continued support for facilitators during the course development stage, and pro-

REFERENCES 1. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA 2008;300:1181–1196. 2. Cook DA, Levinson AJ, Garside S. Time and learning efficiency in Internet-based learning: a systematic review and meta-analysis. Adv Health Sci Educ 2010;15:755–770. 3. Kulier R, Coppus SF, Zamora J, et al. The effectiveness of a clinically integrated e-learning course in evidence-based medicine: a cluster randomised controlled trial. BMC Med Educ 2009;9:21. 4. Hadley J, Kulier R, Zamora J, et al. Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training. J R Soc Med 2010;103:288–294. 5. Gensichen J, Vollmar HC, So¨nnichsen A, Waldmann UM, Sandars J. E-learning for education in primary healthcare – turning the hype into reality: a Delphi study. Eur J Gen Pract 2009;15:11–14. 6. Shortt SED, Guillemette JM, Duncan AM, Kirby F. Defining Quality Criteria for Online Continuing Medical Education Modules using Modified Nominal Group Technique. J Contin Educ Health Prof 2010;30:246–250. 7. Robinson C. Real World Research, 2nd edn. London: Blackwell, 2002.

Corresponding author’s contact details: Mrs Menna Brown, College of Medicine, Swansea University, Swansea SA2 8PP, UK. E-mail: [email protected]

Funding: None. Conflict of interest: None. Ethical approval: The study received a favourable ethics review from Cardiff Univesity ethics committee. doi: 10.1111/tct.12052

14  2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 10–14

Evaluating PLATO: postgraduate teaching and learning online.

  The use of the Internet as a teaching medium has increased rapidly over the last decade. PLATO (postgraduate learning and teaching online) was launc...
222KB Sizes 0 Downloads 0 Views