Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Perspectives Views in Psychiatric Care
ISSN 0031-5990
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views Mari E. Lahti, RN, MNSc, PhD, Raija M. Kontio, RN, PhD, and Maritta Välimäki, RN, PhD Mari E. Lahti, RN, MNSc, PhD, is a Midwife, Department of Nursing Science, University of Turku, Turku, Finland; Raija M. Kontio, RN, PhD, is an Assistant Chief, Department of Psychiatry, Hospital District of Helsinki and Uusimaa, Hyvinkää Hospital Region, Tuusula, Finland; and Maritta Välimäki, RN, PhD, is a Professor and Nursing Director, Department of Nursing Science, University Hospital of Turku, University of Turku, Turku, Finland.
Search terms: Education, e-learning, professional competence, qualitative methodology Author contact: melaht@utu.fi, [email protected], with a copy to the Editor: [email protected] Conflict of Interest Statement The authors report no actual or potential conflicts of interest.
BACKGROUND: e-Learning is one way to provide continuing education and has been shown to be an effective method. However, little is known about how psychiatric organizations benefit from continuing education. PURPOSE: The aim of this study was to use Kirkpatrick’s four-level model to evaluate an e-learning continuing education course from the perspectives of nursing managers in psychiatric hospital organizations. DESIGN AND METHODS: This qualitative design evaluated one e-learning course using Kirkpatrick’s model as an evaluation frame. FINDINGS: Nursing managers noticed positive, neutral, and negative reactions after the course. Nursing managers also reported that the nurses’ learning evolved.
First Received February 12, 2014; Final Revision received November 12, 2014; Accepted for publication December 9, 2014. doi: 10.1111/ppc.12100
Nurses constitute the largest healthcare professionals group (Organization for Economic Co-operation and Development [OECD], 2011). General hospitals and psychiatric hospitals benefit from competent nurses (Wright, Lavoie-Tremblay, Drevniok, Racine, & Savignac, 2011). Healthcare organizations are experiencing a shortage of nurses (World Health Organization [WHO], 2007) and their turnover is a serious challenge due in part to a lack of continuing education (Choi, Cheung, & Pang, 2012). e-Learning may prove to be an integral solution to offering sufficient continuing education (Liu, Rong, & Liu, 2014). The term e-learning could be considered an umbrella term for all aspects of education that employ the usage of information and communication technologies in learning and instructing (Maxwell & Mucklow, 2012; Moore, Dickson-Deane, & Galyen, 2011). Most commonly, e-learning refers to the broad range of platforms for learning and teaching that require the use of various electronic devices and the Internet (Maxwell & Mucklow, 2012). Moreover, online learning refers to the simultaneous use of a variety of media devices and the resources of the World Wide Web to achieve learning. It has been stated that in order for these efforts to count as online learning, students need to spend at least 25% of their course time online (Sajeva, 2009). 40
Continuing education aims to improve professional competence (Nalle, Wyatt, & Myers, 2010). Benefits from continuing education include motivation to learn (Joyce & Cowman, 2007), commitment to organization (Gould & Fontenla, 2006), thirst for knowledge (Cook, Levinson, & Garside, 2008), better communication (Wright et al., 2011), improved healthcare practice (Forsetlund et al., 2009), and better patient symptom handling (Covell, 2009). However, less is known about the possible disadvantages of continuing education, such as increased costs (Bjørk, Tørstad, Hansen, & Samdal, 2009) and time used for education (Nalle et al., 2010). The effectiveness of continuing education has generally been researched by focusing on levels of satisfaction and knowledge (Curran & Fleet, 2005) rather than on real reaction such as feelings or by exploring organizational implications (Brown, 2005) or implications for practice (Parker, Burrows, Nash, & Rosenblum, 2011). This would be important as organizations invest heavily in healthcare continuing education (Covell, 2009). Healthcare professionals in Finland have an obligation to participate in continuing education courses. The number of continuing education days depends on educational level, work requirements, and job description (Ministry of Social Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
Affairs and Health, 2004). However, nurses in Finland are not required to earn credits from these courses in order to qualify for license renewal. In the year 2011, 77% of nurses in Finland participated in some type of continued education program which on average lasted about 4 days (Hotti, 2012). Employers in Finland have the responsibility to organize continuing education for healthcare personnel. However, employees can also obtain continuing education on their own (Ministry of Social Affairs and Health, 2004). Kirkpatrick’s model (Kirkpatrick, 1996) was used to evaluate the implications of an e-learning course for clinical practice. Kirkpatrick’s model was developed in the late 1950s (Kirkpatrick & Kirkpatrick, 2006) to evaluate training programs and learning (Kirkpatrick, 1996). The main purpose of the model was to determine the effectiveness of a training program (Kirkpatrick & Kirkpatrick, 2006). The model is divided into four levels describing different aspects of learning evaluations: (a) student reaction, (b) student learning, (c) behavioral changes after training, and (d) impacts of the training from the organization’s perspective (Kirkpatrick & Kirkpatrick, 2006). The evaluation of training carefully considered those four aspects (Kirkpatrick & Kirkpatrick, 2006). More recently, Kirkpatrick’s model has been used to evaluate course outcomes (Galloway, 2005), for example, related to higher education (Praslova, 2010), transfer of knowledge (Lahti, Kontio, Pitkänen, & Välimäki, 2013), and online continuing education programs (Sears, Cohen, & Drope, 2008). The model is also tested for e-learning (Hamtini, 2008). This model has been shown to be suitable for evaluating training implications and to see if a training program meets organizational requirements (Smidt, Balandin, Sigafoos, & Reed, 2009). It is vitally important that the nursing managers are committed to the idea of systematic education continuation, and are willing to support other nurses who participate in such courses (Gould & Fontenla, 2006). To the best of the authors’ knowledge, this is the first study using all levels of Kirkpatrick’s model to evaluate an e-learning course in mental health. Aim Our aim was to use Kirkpatrick’s model to evaluate nurses’ reactions, learning, behavior, and course implications of an e-learning continuing education course from the perspectives of nursing managers in psychiatric hospitals. Nursing managers were asked the following questions: • How did the nurses react to the e-learning course? • How did the participation in the e-learning course improve the nurses’ knowledge and support their attitudes to psychiatric care? • What activities have the nurses transferred from the course content into clinical practice after the e-learning course? • What were the impacts of the course on clinical work? Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Methods Design This study used a qualitative descriptive design (Sandelowski, 2000) appropriate to understanding mental health organizations (Gillard, Simons, Turner, Lucock, & Edwards, 2012). Inductive content analysis was used to categorize the nursing managers’ written texts. Although Kirkpatrick’s model was used as an evaluation frame, the data categorization was conducted inductively based on the data provided by the nursing managers. Setting and Participants The study was conducted in six psychiatric hospital districts and nine psychiatric hospitals in Finland, which participated in the same e-learning continuing education course from 2008 to 2011. One of the six hospital districts had two hospitals in participation, and another district had three hospitals participating. All of the nurses from the organizations who participated in the e-learning course were working in inpatient wards. One main criterion for participation was that the nursing managers, nursing directors, and deputy nurse managers of wards where the e-learning course was conducted were Finnish speaking. Sampling was done according to the purposive method (Higginbottom, 2004). Exclusion criteria included being on a fixed-term contract (under 3 months). Of those invited nursing managers (N = 48), 28 nurse managers (61%) participated in the study. There are two kinds of nurses working in psychiatric hospitals in Finland: registered nurses, who have a higher level of education (210 European Credit Transfer and Accumulation System [ECTS]), and mental health nurses/practical nurses, who have a lower level of education (120 ECTS). Between the years 2008 and 2011, up to 294 nurses throughout 21 wards had participated in this e-learning course and 16 nurse managers from 28 participating nurse managers in this study participated also to the e-learning course. The e-learning course was advertised to all psychiatric hospitals in Finland, and either nurses themselves or the managers decided on the participation. In the 21 inpatient wards, the patient populations were similar; the wards were mixed sex, 24 hr, with 10–20 beds, mostly with locked doors. The wards treat patients with serious mental illnesses. Questionnaire The data collection was conducted by a structured, openended questionnaire developed specifically for this study. The open-ended questionnaire was pretested with three nursing managers and corrections to the questionnaire language and understandability were made based on their comments. The questionnaire, which included nine open-ended questions, 41
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
was derived from four essential ideas within Kirkpatrick’s model (Kirkpatrick, 1996): (a) Reactions (one question) “What kind of reactions did the e-learning course raise among nurses?”; (b) Learning (two questions) “How did the e-learning course affect the nurses’ knowledge related to the treatment of distressed and aggressive patients?” and “How did the e-learning course affect the nurses’ attitudes relating to the treatment of distressed and aggressive patients?”; (c) Behavior (two questions) “What kind of knowledge has been transferred to the nurses’ daily practice?” and “What kind of knowledge had managers hoped to be transferred to the nurses’ daily practice?”; and (d) Results (four questions) “What kind of advantages did managers notice were gained from the e-learning course and added to the nurses’ daily practice?”, “What kind of disadvantages did the managers notice resulted from the e-learning course and affected the nurses’ daily practice?”, “How has the e-learning course, in general, affected the nurses’ daily practice?”, and “What kind of overall impact did the e-learning course have on the nurses’ daily practice?” In addition, the questionnaire included six background questions (sex, age, organization, work position, years of managerial experience, and participation in e-learning course). Data Collection Contact persons in each psychiatric organization were asked to locate all nursing managers fulfilling the inclusion criteria and to forward their e-mail addresses to the researcher. The data were collected electronically using Webropol 2.0 online survey tool (https://www.webropolsurveys.com/). The researcher e-mailed the managers information about the study (purpose, aims, methods, practical information, and voluntary nature of the study). A link to the electronic questionnaire was also included. In one organization, contact persons were responsible for sending the information and the questionnaire to the participants due to hospital policy. Participants also gave written informed consent via the Webropol questionnaire. Response time was 2 weeks; after that, two reminders were sent. The e-Learning Course The e-learning continuing education course (ePsychNurse .Net) was intended to enable nurses to manage distressed and disturbed patients in psychiatric hospitals. It was originally developed and its effectiveness was tested on European Commission funding (2006–2009) together with six European countries (Leonardo da Vinci; FI-06-B-F-PP-16070; Välimäki, Lahti, Scott, & Chambers, 2008). The theoretical background of the course was based on reflective learning (Lowe, Rappolt, Jaglal, & Macdonald, 2007), which is used in mental health education (Cleary, Horsfall, Happell, & Hunt, 42
2013). The structure of the course was designed to proceed toward more detailed information from more general one, and transferring knowledge to support the clinical practice and deepening the students’ reflective thinking. The course aimed to support the students’ reflective thinking by asking them to reflect on critical incidents from practice in every unit of the course. There were seven constructed modules which are referred to as units. The units of the course were devoted to specific topics including orientation for the course, legal and ethical issues, internal and external factors associated with aggressive behavior, the self-awareness of the staff, teamwork, and the integration of evidence-based knowledge into practice. Each unit had specific reading material that included PowerPoint presentations, links, and research articles. Video lectures, discussion forums, patient scenarios, and additional reading material were also included. The course required approximately 120 hr, took 3–6 months to complete, and allowed for five ECTS credits. Each unit included reflective assignments and self-awareness exercises. Students had to pass each exercise (passed/need to complete); they were evaluated by their tutors based on how the students had fulfilled the specific criteria. Experienced nurses with MNS and/or PhD degree were educated to work as a course tutors and they commented on the reflective assignments after each unit. No structured knowledge tests were used. In accordance with Finnish standards, the course was not mandatory for all nurses because there were no compulsory credits needed in order to keep a nursing license. Nurses were also able to participate in the course partially within their working hours (40 out of 120 hr). For more details on this education, see Kontio et al. (2011) and Lahti et al. (2013). Data Analysis The qualitative written data were analyzed according to Kirkpatrick’s (Galloway, 2005) levels. The analysis was done manually using Hsieh and Shannon’s (2005) conventional content analysis, namely inductive content analysis, which is a useful approach when there is limited information available on given phenomena. The questions posed to the nurse managers were divided according to Kirkpatrick’s model referring to Hsieh and Shannon’s direct analysis, namely deductive analysis, but the responses to each question were categorized according to the conventional approach. This offered an opportunity to look at the phenomenon from the participants’ own perspective (Graneheim & Lundman, 2004). All written material was read to obtain an understanding of the text. The codes were formed from the text capturing the key thoughts. The codes were sorted into categories according to how they were related and linked (Graneheim & Lundman, 2004). We used Microsoft Word as data management and analysis software (La Pelle, 2004); the data were coded manually by making notes in printed documents. The coding was Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
conducted by one person experienced in qualitative analysis (M. L.) and was confirmed by a second researcher (M.V.). The credibility of the analysis process was confirmed using representative quotations from the original text in the results, and “Qualitative studies—COREQ—Consolidated criteria for reporting qualitative research” also guided data reporting (Tong, Sainsbury, & Craig, 2007). Ethics Ethical permission for this research was obtained from the local university ethics committee (dated March 15, 2013, University of Turku) and permission for the collection of data was obtained from the research permission committees of six hospital districts. In effect, one of the hospital districts granted permission for two hospitals, another district provided permission for three, and the remaining four hospital districts held one permission for one psychiatric hospital. Participation was voluntary, and the data were treated in confidence. Results
Nurses’ ReactionsAfterTheir e-Learning Course—Managers’ Perspective (Reactions; Level 1 in Kirkpatrick’s Model). This varied from positive, neutral, or negative. A reaction refers to how the students reacted to the e-learning course from the nurse managers’ perspective.Positive reactions were described as enthusiasm, motivation, and fascination. A few of the participating nurses were really enthusiastic about the course and tried to raise extensive discussion in their work communities. (ID 23) The psychiatric nursing managers described that the nurses discussed the content of nursing more and they reflected more on their work. Moreover, from the nursing managers’ point of view, the ward culture was built up to lean toward favoring alternative methods that avoided coercive practices. The culture was also geared toward creating an environment that encouraged more positive attitudes toward the development of the work itself. The nursing managers also reported neutral reactions among nurses, such as neutral or expectant feelings about the course. The nursing managers reported that there were also negative reactions, namely anger, fear, doubt, resistance, irritation, underestimation, feelings of being coerced, distraction, and laziness.
Participants’ Characteristics Most participants (n = 28) were female (68%). The biggest age group was 51–60 years (36%). Two thirds (68%) had 1–10 years of work experience at the managerial level. Half of the respondents (50%) were ward managers and over half (57%) had participated in the e-learning course. See Table 1.
Table 1. Characteristics of Study Participants (N = 28)
Sex Female Male Age (years) 31–40 41–50 51–60 61+ Work position Nursing director Ward manager Ward deputy manager Other Managerial experience (years) 1–10 11–20 21–30 Participated in ePsychNurse.Net course Yes No
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
N
%
19 9
68 32
8 9 10 1
28 32 36 4
6 14 5 3
21 50 18 11
18 5 5
64 18 18
16 12
57 43
Some nurses were angry and frustrated and did not see why they should participate in this course. Was their professional competence in doubt because they were “forced” into training? (ID 12) The nursing managers reported that some nurses were apprehensive about the course and its assignments. Moreover, the nursing managers noticed that the nurses initially resisted the course and the new learning method raised anxiety. Some nursing managers also thought that the nurses felt their professional knowledge was underestimated. In addition, the nursing managers pointed out that some nurses felt coerced to participate, and they had doubts about the course. The nursing managers also felt that some nurses were lazy during the course. Nurse’s Knowledge Levels and Attitudes Related to Psychiatric Care—the Managers’ Perspective (Learning; Level 2 in Kirkpatrick’s Model). This had seven categories. The categories were internationality, mental health act, ethical issues, treatment of aggressive patients, alternative methods, knowledge level, and discussion about the course content. Learning means how much the students learned from the e-learning course from the nursing managers’ perspective. The nursing managers reported that the nurses learned more about the relevant legislation, ethical issues, and how to cope with aggressive patients. Knowledge of the legislation was increased, and teamwork became a part of aggressive patients’ treatment. (ID 2) 43
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
The nursing managers felt that after the course, the nurses were knowledgeable about alternative methods related to avoiding coercion. Again, general discussion about the course topic supported knowledge transfer between nurses. In my opinion, the course content provided good information on the care of aggressive patients. (ID 5) The nursing managers’ views on the nurses’ learning on the course raised four issues related to the nurses’ attitudes: the nurses’ awareness of their own attitudes, change in attitudes, attitudes toward patient care, and reduction in the use of coercive methods. First, the nursing managers reported that the nurses’ awareness of their own attitudes changed through self-reflection exercises. Knowing your own attitudes through reflection, and scrutinizing your own actions is perhaps the most important thing considering attitudes. (ID 2) The nursing managers also described how the overall atmosphere and attitudes toward aggressive patients became more positive. They felt that the nurses’ attitudes toward patient care and treating patients became more individualistic and collaborative. The nurses were also more willing to use more alternative methods instead of physical restraints. The nursing managers felt that the use of coercion was reduced after the course. Attitudes are seen in discussions in the work unit and the use of coercion has diminished. (ID 28) Nurses’ Knowledge Transfer to Clinical Practice—Managers’ Perspective (Behavior; Level 3 in Kirkpatrick’s Model). This had seven categories: legislation on coercion, nursing distressed and disturbed patients, alternative methods to reduce coercion, anticipation of aggressive situations, debriefing, teamwork, and common discussions. Behavior refers to how much the students changed their behavior as a result of the e-learning course, and how they were able to transfer the knowledge to practice. The nursing managers noted that the nurses’ better understanding of the legislation on coercion was apparent in clinical practice. Attention has begun to be paid to measures preventing aggression and seclusion has clearly diminished. (ID 5) The nursing managers reported that the nurses cooperated more with distressed and disturbed patients; they added outdoor activities and involved patients more in decisionmaking. Moreover, the nurses tried to anticipate aggressive situations more. The nursing managers reported that they noticed more nurse–nurse and nurse–patient discussion, debriefing was more systematic, and teamwork changed after the course. 44
In my opinion, de-briefing after aggressive situations has become more systematic among the personnel. (ID 20) The nursing managers also hoped that what was learned could be transferred more effectively to clinical practice. Seven categories were formed: prevention of aggressive situations, holistic patient care, evidence-based care, understanding of mental illnesses, use of reflection, development in attitudes toward coercion, and increase in discussion. The nursing managers wanted the nurses to use different preventive methods in aggressive incidents, and that, in the future, nursing of distressed and disturbed patients could be more holistic. I would like to see greater inclusion of patients in decision-making related to their own care. (ID 12) The nursing managers hoped that nursing interventions could rely on evidence-based care, and that nurses could have a better understanding of mental illnesses. All the articles from the course are printed out to make a file for the personnel to read, and I wish that use could be made of the information, especially in our practical work. (ID 15) The nursing managers also pointed out that nurses could use more reflection in their work and that nurses’ attitudes toward use of coercion could be revised. Moreover, nurses could discuss more often with both patients and team members. Impacts of the e-Learning Course on Clinical Work—the Managers’ Perspective (Impact; Level 4 in Kirkpatrick’s Model). Impacts as a final result refer to the outcomes of the e-learning course observed by the organization. There were six main categories related to the advantages of the e-learning course. The categories were cooperation, anticipation of aggression, overall difference in attitudes, acknowledgment of patients, awareness of one’s own work, and new knowledge. Better cooperation among nurses, and between patients and nurses, was one advantage of the e-learning course. Anticipation of aggressive situations was seen to be related to patient treatment and handling. Overall changes in attitudes toward patients were seen as an advantage of the course; better understanding of patients’ needs and more humane treatment result from changed knowledge. The nursing managers reported that nurses were more aware of their own work. Anticipation of aggression is used more to avoid coercive practices, thereby, the use of seclusion has the potential to be diminished. (ID 1) Nursing managers also reported disadvantages of the e-learning continuing education course. These were the time used and the need for substitute nurses. Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Development of coercive practices Nurses’ work development
Patient treatment
Common discussions Discussion about the course content
Development in attitudes toward coercion Increase in discussion Teamwork
Alternative methods
Ethical issues Negative feelings
Treatment of aggressive patients
Mental Health Act Neutral feelings
Attitudes toward patient care Reduce in coercion
Knowledge level
Use of reflection
Understanding of mental illness
Evidence-based care
Awareness of one’s own work New knowledge
Organizational resources Anticipation of aggression Overall difference in attitudes Acknowledge patient
Implications Disadvantages
Time consuming Cooperation
Advantages Used knowledge
Internationality Positive feelings
Attitudes Knowledge Reaction
This study aimed to evaluate the e-learning continuing education course from the perspective of nursing managers in psychiatric hospital organizations. Kirkpatrick’s evaluation
Learning
Discussion
Reaction
See summary of categories (Table 2).
Table 2. Summary of Categories
The knowledge and skills of the personnel improved, which is apparent in how they encounter the patient and in care. (ID 17)
Behavior
Moreover, the nursing managers noticed that the use of coercion had become more humane, for example, the nurses wanted to shorten the period of seclusion, and they wanted to redecorate the seclusion rooms so that they were more comfortable for the patients. In addition, the nurses were more aware of their own work and more positive about development projects. The nursing managers felt that the organization’s input into e-learning continuing education course was one advantage. Moreover, the nursing managers pointed out that the nurses’ competence was better after the e-learning course, and communication among nurses was growing.
Prevention of aggressive situation Holistic patient care
Impact
Knowledge development in the future
The nurses’ interactive skills for dealing with aggressive patients improved. (ID 23)
Legislation on coercion Care of distressed and disturbed patients Alternative methods to reduce coercion Anticipation of aggressive situations Debriefing
Four categories were identified as relative to implications of the e-learning course’s effectiveness. The categories were cooperation, patient treatment, development of coercive practices, and nurses’ work development. The nursing managers described that in the category of cooperation, there were growing numbers of discussions among team members which increased positive communication. There were also reports from nursing managers which suggested that they felt as if the nurses were given new tools for clinical practice and patient treatment. They explained that at close observation, it seemed that these changes were a result of the e-learning course. The managers noticed that the coercive practices of the nurses after the e-learning course seemed less frequent than what they were before the course. They noted that any observed use of coercion with a patient was acted out in a more humane manner. The nursing managers stated that they were able to identify and calculate a nurse’s “work development” after the e-learning course. They explained that the nurses became “aware of their work,” and that the idea of developing and maintaining an internal clinical work ethic was more positive. They exclaimed that these quick results were an indicator that through this course model, the organization’s resources for education could be put to good use.
Nurses’ awareness of their own attitudes Change in attitudes
The course took a lot of time and there were substitute nurses on the ward. (ID 1)
Cooperation
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
45
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
model was used as an evaluation frame. This study showed that the nursing managers noticed that the nurses had different reactions to the e-learning course, which were mostly positive; the result is partially supported by earlier research among participants in the e-learning courses (Cook et al., 2008). Neutral and negative feelings by the nurses were also described by the managers, which is fairly understandable as e-learning is a new learning method in psychiatric hospitals and there may be some resistance to it among nurses (Kontio et al., 2011). In this study, the managers noticed that learning did indeed take place, especially in the knowledge and attitudes in the nurses’ clinical practice. They also reported some knowledge transfer from the course to clinical practice. This is similar to the findings of Lahti et al.’s (2013) study, in which the nurses’ perceptions of knowledge transfer to daily practice were examined, and the nurses reported a clear transfer of knowledge from an e-learning course to clinical practice. In contrast to earlier studies among nurses (Kontio et al., 2011), in this study there was a change in attitudes noticed by the nursing managers toward distressed and disturbed patients. It seemed that the nurses and nursing managers shared the same opinion about the effectiveness of knowledge transfer. Interestingly, the nurses did not experience the change in the atmosphere and attitudes the same way as the nursing managers had observed. This study brings a novel approach to evaluating the e-learning course from the perspective of the nursing managers, as in Finland, most of the continuing education in health care is paid for by the employees (Hotti, 2012). In this study, the nursing managers identified several issues where the nurses’ behavior in clinical practice had changed. This is also supported by Kontio et al. (2011), who found that the nurses’ knowledge about the legislation on coercion increased. This study also looked into understanding what the nursing managers hoped to see their nurses implementing in clinical practice after the e-learning course. It was noted that the nursing managers were hoping to observe holistic patient care and evidence-based practice. This was similar to what Lahti et al. (2013) found regarding the nurses’ own perspective of what knowledge they were hoping to transfer from an e-learning course to daily practice. The nursing managers reported advantages of the e-learning course for clinical practice, stating that this kind of an e-learning course with a lot of evidence-based material raised the nurses’ willingness to develop their work and approve research. Eizenberg (2011) has also suggested that nurses who think more positively about research are more likely to implement it in their own work. Our findings raised the important issue that the managers deemed discussion and communication among nurses important. In most respects, the nursing managers found that discussion was broadening among the nurses. This finding is also supported by Wright et al. (2011). 46
Implications for Psychiatric Organizations Given the shortage of nurses and the high turnover rates, the competition for nurses has begun (WHO, 2007). We therefore need to know more about how education might support nurses’ daily work (Lahti et al., 2013) but also motivate them and increase their commitment (Gould & Fontenla, 2006). Continuing education also enhances nurses’ professional skills, which will also affect how they are retained in the profession (Choi et al., 2012). Implications for Research More evaluation on levels 3 and 4 is needed to really understand the impacts of e-learning continuing education on clinical practice, that is, what kind of knowledge transfer took place after the e-learning course and how permanent this knowledge transfer is. Parker et al. (2011) call for more emphasis to be placed on the real change and its implications for practice. More emphasis could be placed on the use of quantitative designs and the probability of predicting certain outcomes that are more difficult to measure. Such things as cost-effectiveness, patient improvement, and wellness should be heavily considered when evaluating continuing education courses such as the one used in this study. However, use of qualitative research methods could help quantify the results, and bring to light more of the information related to certain observable behavioral changes that result from knowledge transfer. Moreover, qualitative research could help identify and verify possible final impacts on clinical practices that could develop from continuing education courses. Methodological Considerations The results were based solely on the respondents’ subjective perceptions. However, we managed to reach quite an informative group of respondents among psychiatric nursing managers in our study hospitals. The participation rate in this research was very low. This may be due to nursing managers’ demanding work (Lee & Cummings, 2008) and lack of time. Voluntary participation may also cause bias as those with more positive attitudes participate more actively than those with more negative attitudes (Kisely & Kendall, 2011). Moreover, those nursing managers who had participated themselves in the e-learning course may hold more positive opinions related to the e-learning course than nursing managers who had not attended such a course. However, all nursing managers had been informed about the e-learning course when they decided on the nurses’ participation in the course. Moreover, the coding was done by one researcher, and this may likewise cause bias. In addition, the nature of the data was narrow due to the participants’ brief responses, which may affect the analysis and results. Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
Conclusion We showed that Kirkpatrick’s evaluation model was a usable way to evaluate an e-learning continuing education course in a psychiatric organization from the nursing managers’ point of view. These findings are novel and promising within various continuing education situations where Finnish nurses participate approximately 4 days out of a year in continuing education efforts (Hotti, 2012). The e-learning course discussed in this study was 3–6 months long. Even though the transference of knowledge and the revision of daily practices did take time to be acclimated into the psychiatric nurses’ daily work flow, the nursing managers were able to identify positive reactions to the course and were able to observe considerable changes during that time. There were also findings which determined that the clinical practices among the nurses who participated in this e-learning course had sufficiently improved. However, there is still a need to evaluate e-learning continuing education courses in the field of psychiatric nursing more systematically to ensure appropriate education for psychiatric nurses. Acknowledgment This study was supported by EduMental, Tekes—the Finnish Funding Agency for Technology and Innovation (40245/12). References Bjørk, I., Tørstad, S., Hansen, B., & Samdal, G. (2009). Estimating the cost of professional developmental activities in health organizations. Nursing Economics, 27(4), 239–244. Brown, K. (2005). An examination of the structure and nomological network of trainee reactions: A closer look at “Smile Sheets.” Journal of Applied Psychology, 90(5), 991–1001. doi:10.1037/0021-9010.90.5.991 Choi, P., Cheung, K., & Pang, C. (2012). Attributes of nursing work environment as predictors of registered nurses’ job satisfaction and intention to leave. Journal of Nursing Management, 21(3), 429–439. Cleary, M., Horsfall, J., Happell, B., & Hunt, G. (2013). Reflective components in undergraduate mental health nursing curricula: Some issues for consideration. Issues in Mental Health Nursing, 34(2), 69–74. Cook, D., Levinson, A., & Garside, S. (2008). Internet-based learning in the health professions. Journal of the American Medical Association, 30(10), 1181–1196. Covell, C. (2009). Outcomes achieved from organizational investment in nursing continuing professional development. Journal of Nursing Administration, 39(10), 438–443. Curran, V., & Fleet, L. (2005). A review of evaluation outcomes of web-based continuing medical education. Medical Education, 39(6), 561–567. doi:10.1111/j.1365-2929.2005.02173.x
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Eizenberg, M. (2011). Implementation of evidence-based nursing practice: Nurses’ personal and professional factors? Journal of Advanced Nursing, 67, 33–42. doi:10.1111/j.1365-2648.2010.05488.x Forsetlund, L., Bjørndal, A., Rashidian, A., Jamtvedt, G., O’Brien, M., Wolf, F., . . . Oxman, A. (2009). Continuing education meetings and workshops: Effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews, (2), CD003030. doi:10.1002/14651858.CD003030.pub2 Galloway, D. (2005). Evaluating distance delivery and e-learning. Performance Improvement, 44(4), 21–27. Gillard, S., Simons, L., Turner, K., Lucock, M., & Edwards, C. (2012). Patient and public involvement in the coproduction of knowledge: Reflection on the analysis of qualitative data in a mental health study. Qualitative Health Research, 22(8), 1126–1137. Gould, D., & Fontenla, M. (2006). Commitment to nursing: Results of a qualitative interview study. Journal of Nursing Management, 14, 213–221. doi:10.1111/j.1365-2934.2006.00577.x Graneheim, U., & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105–112. Hamtini, T. (2008). Evaluating e-learning programs: An adaptation of Kirkpatrick’s model to accommodate e-learning environments. Journal of Computer Science, 4(8), 693–698. Higginbottom, G. (2004). Sampling issues in qualitative research. Nurse Researcher, 12(1), 7–19. Hotti. (2012). Municipality continuing education in health and social sector in year 2011. Local government employers. Retrieved May 22, 2014, from http://www.kuntatyonantajat.fi/ fi/ajankohtaista/tilastot/tiedustelut-ja-selvitykset/ taydennyskoulutus/sosiaali–ja-terveydenhuollon -taydennyskoulutus-011/Documents/sosiaali-%20ja%20 terveydenhuollon%20t%C3%A4ydennyskoulutus%202011.pdf Hsieh, H.-F., & Shannon, S. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. Joyce, P., & Cowman, S. (2007). Continuing professional development: Investment or expectation? Journal of Nursing Management, 15, 626–633. doi:10.1111/j.1365-2834.2007.00683.x Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, January, 54–59. Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs (3rd ed.). San Francisco: Berrett-Koehler Publishers. Kisely, S., & Kendall, E. (2011). Critically appraising qualitative research: A guide for clinicians more familiar with quantitative techniques. Australasian Psychiatry, 19(4), 364–367. Kontio, R., Lahti, M., Pitkänen, A., Joffe, G., Putkonen, H., Hätönen, H., . . . Välimäki, M. (2011). Impact of eLearning course on nurses’ professional competence in seclusion and restraint practices: A randomised controlled study
47
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
(ISRCTN32869544). Journal of Psychiatric and Mental Health Nursing, 18, 813–821. La Pelle, N. (2004). Simplifying qualitative data analysis using general purpose software. Field Methods, 16(1), 85–108. Lahti, M., Kontio, R., Pitkänen, A., & Välimäki, M. (2013). Knowledge transfer from an e-learning course to clinical practice. Nurse Education Today, 34, 842–847. Lee, H., & Cummings, G. (2008). Factors influencing job satisfaction of front line nurse managers: A systematic review. Journal of Nursing Management, 16, 768–783. doi:10.1111/j.1365-2834.2008.00879.x Liu, W.-I., Rong, J.-R., & Liu, C.-Y. (2014). Using evidence-integrated e-learning to enhance case management continuing education for psychiatric nurses: A randomised controlled trial with follow-up. Nurse Education Today, 34(11), 1361–1367. doi:10.1016/j.nedt.2014.03.004 Lowe, M., Rappolt, S., Jaglal, S., & Macdonald, G. (2007). The role of reflection in implementing learning from continuing education into practice. Journal of Continuing Education in the Health Professions, 27(3), 143–148. Maxwell, S., & Mucklow, J. (2012). e-Learning initiatives to support prescribing. British Journal of Clinical Pharmacology, 74, 621–631. Ministry of Social Affairs and Health. (2004). Recommendation for further education foe health care staff. Handbooks of the Ministry of Social Affairs and Health, Finland, Helsinki, 3/2004. ISBN: 952-00-1478-0. Moore, J., Dickson-Deane, C., & Galyen, K. (2011). E-Learning, online learning and distance learning environments: Are they the same? Internet and Higher Education, 14, 129–135. Nalle, M., Wyatt, H., & Myers, R. (2010). Continuing education needs of nurses in a voluntary continuing nursing education state. Journal of Continuing Education in Nursing, 41(3), 107–115. Organization for Economic Co-operation and Development. (2011). Health at a glance 2011: OECD indicators. Paris: Author. doi:10.1787/health_glance-2011-en Parker, K., Burrows, G., Nash, H., & Rosenblum, N. (2011). Going beyond Kirkpatrick in evaluating a clinician scientist program:
48
It’s not “If it works” but “How it works.” Academic Medicine, 86(11), 1389–1396. Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability, 22(3), 215–225. Sajeva, M. (2009). E-Learning: Web-based education. Current Opinion in Anaesthesiology, 19(6), 645–649. Sandelowski, M. (2000). Focus on research methods whatever happened to qualitative description? Research in Nursing & Health, 23, 334–340. Sears, K., Cohen, J., & Drope, J. (2008). Comprehensive evaluation of an online tobacco control continuing education course in Canada. Journal of Continuing Education in the Health Professions, 28(4), 235–240. Smidt, A., Balandin, S., Sigafoos, J., & Reed, V. (2009). The Kirkpatrick model: A useful tool for evaluating training outcomes. Journal of Intellectual & Developmental Disability, 34(3), 266–274. doi:10.1080/13668250903093125 Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349–357. Välimäki, M., Lahti, M., Scott, A., & Chambers, M. (2008). The changing face of psychiatric nursing—care or control. ePsychNurse.Net. Towards improved quality developing nurses’ continuing vocational training in psychiatric hospitals and inpatient units. ISBN: 978-951-29-3576-5. University of Turku, Finland. World Health Organization. (2007). Atlas: Nurses in mental health 2007. ISBN 978 92 4 156345 1. Retrieved April 1, 2013, from http://www.who.int/mental_health/evidence/nursing_atlas _2007.pdf Wright, D., Lavoie-Tremblay, M., Drevniok, U., Racine, H., & Savignac, H. (2011). Relational dimensions of a positive integration experience for new mental health nurses. Archives of Psychiatric Nursing, 25(3), 164–173. doi:10.1016/j.apnu.2010.07.008
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
ISSN 0031-5990
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views Mari E. Lahti, RN, MNSc, PhD, Raija M. Kontio, RN, PhD, and Maritta Välimäki, RN, PhD Mari E. Lahti, RN, MNSc, PhD, is a Midwife, Department of Nursing Science, University of Turku, Turku, Finland; Raija M. Kontio, RN, PhD, is an Assistant Chief, Department of Psychiatry, Hospital District of Helsinki and Uusimaa, Hyvinkää Hospital Region, Tuusula, Finland; and Maritta Välimäki, RN, PhD, is a Professor and Nursing Director, Department of Nursing Science, University Hospital of Turku, University of Turku, Turku, Finland.
Search terms: Education, e-learning, professional competence, qualitative methodology Author contact: melaht@utu.fi, [email protected], with a copy to the Editor: [email protected] Conflict of Interest Statement The authors report no actual or potential conflicts of interest.
BACKGROUND: e-Learning is one way to provide continuing education and has been shown to be an effective method. However, little is known about how psychiatric organizations benefit from continuing education. PURPOSE: The aim of this study was to use Kirkpatrick’s four-level model to evaluate an e-learning continuing education course from the perspectives of nursing managers in psychiatric hospital organizations. DESIGN AND METHODS: This qualitative design evaluated one e-learning course using Kirkpatrick’s model as an evaluation frame. FINDINGS: Nursing managers noticed positive, neutral, and negative reactions after the course. Nursing managers also reported that the nurses’ learning evolved.
First Received February 12, 2014; Final Revision received November 12, 2014; Accepted for publication December 9, 2014. doi: 10.1111/ppc.12100
Nurses constitute the largest healthcare professionals group (Organization for Economic Co-operation and Development [OECD], 2011). General hospitals and psychiatric hospitals benefit from competent nurses (Wright, Lavoie-Tremblay, Drevniok, Racine, & Savignac, 2011). Healthcare organizations are experiencing a shortage of nurses (World Health Organization [WHO], 2007) and their turnover is a serious challenge due in part to a lack of continuing education (Choi, Cheung, & Pang, 2012). e-Learning may prove to be an integral solution to offering sufficient continuing education (Liu, Rong, & Liu, 2014). The term e-learning could be considered an umbrella term for all aspects of education that employ the usage of information and communication technologies in learning and instructing (Maxwell & Mucklow, 2012; Moore, Dickson-Deane, & Galyen, 2011). Most commonly, e-learning refers to the broad range of platforms for learning and teaching that require the use of various electronic devices and the Internet (Maxwell & Mucklow, 2012). Moreover, online learning refers to the simultaneous use of a variety of media devices and the resources of the World Wide Web to achieve learning. It has been stated that in order for these efforts to count as online learning, students need to spend at least 25% of their course time online (Sajeva, 2009). 40
Continuing education aims to improve professional competence (Nalle, Wyatt, & Myers, 2010). Benefits from continuing education include motivation to learn (Joyce & Cowman, 2007), commitment to organization (Gould & Fontenla, 2006), thirst for knowledge (Cook, Levinson, & Garside, 2008), better communication (Wright et al., 2011), improved healthcare practice (Forsetlund et al., 2009), and better patient symptom handling (Covell, 2009). However, less is known about the possible disadvantages of continuing education, such as increased costs (Bjørk, Tørstad, Hansen, & Samdal, 2009) and time used for education (Nalle et al., 2010). The effectiveness of continuing education has generally been researched by focusing on levels of satisfaction and knowledge (Curran & Fleet, 2005) rather than on real reaction such as feelings or by exploring organizational implications (Brown, 2005) or implications for practice (Parker, Burrows, Nash, & Rosenblum, 2011). This would be important as organizations invest heavily in healthcare continuing education (Covell, 2009). Healthcare professionals in Finland have an obligation to participate in continuing education courses. The number of continuing education days depends on educational level, work requirements, and job description (Ministry of Social Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
Affairs and Health, 2004). However, nurses in Finland are not required to earn credits from these courses in order to qualify for license renewal. In the year 2011, 77% of nurses in Finland participated in some type of continued education program which on average lasted about 4 days (Hotti, 2012). Employers in Finland have the responsibility to organize continuing education for healthcare personnel. However, employees can also obtain continuing education on their own (Ministry of Social Affairs and Health, 2004). Kirkpatrick’s model (Kirkpatrick, 1996) was used to evaluate the implications of an e-learning course for clinical practice. Kirkpatrick’s model was developed in the late 1950s (Kirkpatrick & Kirkpatrick, 2006) to evaluate training programs and learning (Kirkpatrick, 1996). The main purpose of the model was to determine the effectiveness of a training program (Kirkpatrick & Kirkpatrick, 2006). The model is divided into four levels describing different aspects of learning evaluations: (a) student reaction, (b) student learning, (c) behavioral changes after training, and (d) impacts of the training from the organization’s perspective (Kirkpatrick & Kirkpatrick, 2006). The evaluation of training carefully considered those four aspects (Kirkpatrick & Kirkpatrick, 2006). More recently, Kirkpatrick’s model has been used to evaluate course outcomes (Galloway, 2005), for example, related to higher education (Praslova, 2010), transfer of knowledge (Lahti, Kontio, Pitkänen, & Välimäki, 2013), and online continuing education programs (Sears, Cohen, & Drope, 2008). The model is also tested for e-learning (Hamtini, 2008). This model has been shown to be suitable for evaluating training implications and to see if a training program meets organizational requirements (Smidt, Balandin, Sigafoos, & Reed, 2009). It is vitally important that the nursing managers are committed to the idea of systematic education continuation, and are willing to support other nurses who participate in such courses (Gould & Fontenla, 2006). To the best of the authors’ knowledge, this is the first study using all levels of Kirkpatrick’s model to evaluate an e-learning course in mental health. Aim Our aim was to use Kirkpatrick’s model to evaluate nurses’ reactions, learning, behavior, and course implications of an e-learning continuing education course from the perspectives of nursing managers in psychiatric hospitals. Nursing managers were asked the following questions: • How did the nurses react to the e-learning course? • How did the participation in the e-learning course improve the nurses’ knowledge and support their attitudes to psychiatric care? • What activities have the nurses transferred from the course content into clinical practice after the e-learning course? • What were the impacts of the course on clinical work? Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Methods Design This study used a qualitative descriptive design (Sandelowski, 2000) appropriate to understanding mental health organizations (Gillard, Simons, Turner, Lucock, & Edwards, 2012). Inductive content analysis was used to categorize the nursing managers’ written texts. Although Kirkpatrick’s model was used as an evaluation frame, the data categorization was conducted inductively based on the data provided by the nursing managers. Setting and Participants The study was conducted in six psychiatric hospital districts and nine psychiatric hospitals in Finland, which participated in the same e-learning continuing education course from 2008 to 2011. One of the six hospital districts had two hospitals in participation, and another district had three hospitals participating. All of the nurses from the organizations who participated in the e-learning course were working in inpatient wards. One main criterion for participation was that the nursing managers, nursing directors, and deputy nurse managers of wards where the e-learning course was conducted were Finnish speaking. Sampling was done according to the purposive method (Higginbottom, 2004). Exclusion criteria included being on a fixed-term contract (under 3 months). Of those invited nursing managers (N = 48), 28 nurse managers (61%) participated in the study. There are two kinds of nurses working in psychiatric hospitals in Finland: registered nurses, who have a higher level of education (210 European Credit Transfer and Accumulation System [ECTS]), and mental health nurses/practical nurses, who have a lower level of education (120 ECTS). Between the years 2008 and 2011, up to 294 nurses throughout 21 wards had participated in this e-learning course and 16 nurse managers from 28 participating nurse managers in this study participated also to the e-learning course. The e-learning course was advertised to all psychiatric hospitals in Finland, and either nurses themselves or the managers decided on the participation. In the 21 inpatient wards, the patient populations were similar; the wards were mixed sex, 24 hr, with 10–20 beds, mostly with locked doors. The wards treat patients with serious mental illnesses. Questionnaire The data collection was conducted by a structured, openended questionnaire developed specifically for this study. The open-ended questionnaire was pretested with three nursing managers and corrections to the questionnaire language and understandability were made based on their comments. The questionnaire, which included nine open-ended questions, 41
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
was derived from four essential ideas within Kirkpatrick’s model (Kirkpatrick, 1996): (a) Reactions (one question) “What kind of reactions did the e-learning course raise among nurses?”; (b) Learning (two questions) “How did the e-learning course affect the nurses’ knowledge related to the treatment of distressed and aggressive patients?” and “How did the e-learning course affect the nurses’ attitudes relating to the treatment of distressed and aggressive patients?”; (c) Behavior (two questions) “What kind of knowledge has been transferred to the nurses’ daily practice?” and “What kind of knowledge had managers hoped to be transferred to the nurses’ daily practice?”; and (d) Results (four questions) “What kind of advantages did managers notice were gained from the e-learning course and added to the nurses’ daily practice?”, “What kind of disadvantages did the managers notice resulted from the e-learning course and affected the nurses’ daily practice?”, “How has the e-learning course, in general, affected the nurses’ daily practice?”, and “What kind of overall impact did the e-learning course have on the nurses’ daily practice?” In addition, the questionnaire included six background questions (sex, age, organization, work position, years of managerial experience, and participation in e-learning course). Data Collection Contact persons in each psychiatric organization were asked to locate all nursing managers fulfilling the inclusion criteria and to forward their e-mail addresses to the researcher. The data were collected electronically using Webropol 2.0 online survey tool (https://www.webropolsurveys.com/). The researcher e-mailed the managers information about the study (purpose, aims, methods, practical information, and voluntary nature of the study). A link to the electronic questionnaire was also included. In one organization, contact persons were responsible for sending the information and the questionnaire to the participants due to hospital policy. Participants also gave written informed consent via the Webropol questionnaire. Response time was 2 weeks; after that, two reminders were sent. The e-Learning Course The e-learning continuing education course (ePsychNurse .Net) was intended to enable nurses to manage distressed and disturbed patients in psychiatric hospitals. It was originally developed and its effectiveness was tested on European Commission funding (2006–2009) together with six European countries (Leonardo da Vinci; FI-06-B-F-PP-16070; Välimäki, Lahti, Scott, & Chambers, 2008). The theoretical background of the course was based on reflective learning (Lowe, Rappolt, Jaglal, & Macdonald, 2007), which is used in mental health education (Cleary, Horsfall, Happell, & Hunt, 42
2013). The structure of the course was designed to proceed toward more detailed information from more general one, and transferring knowledge to support the clinical practice and deepening the students’ reflective thinking. The course aimed to support the students’ reflective thinking by asking them to reflect on critical incidents from practice in every unit of the course. There were seven constructed modules which are referred to as units. The units of the course were devoted to specific topics including orientation for the course, legal and ethical issues, internal and external factors associated with aggressive behavior, the self-awareness of the staff, teamwork, and the integration of evidence-based knowledge into practice. Each unit had specific reading material that included PowerPoint presentations, links, and research articles. Video lectures, discussion forums, patient scenarios, and additional reading material were also included. The course required approximately 120 hr, took 3–6 months to complete, and allowed for five ECTS credits. Each unit included reflective assignments and self-awareness exercises. Students had to pass each exercise (passed/need to complete); they were evaluated by their tutors based on how the students had fulfilled the specific criteria. Experienced nurses with MNS and/or PhD degree were educated to work as a course tutors and they commented on the reflective assignments after each unit. No structured knowledge tests were used. In accordance with Finnish standards, the course was not mandatory for all nurses because there were no compulsory credits needed in order to keep a nursing license. Nurses were also able to participate in the course partially within their working hours (40 out of 120 hr). For more details on this education, see Kontio et al. (2011) and Lahti et al. (2013). Data Analysis The qualitative written data were analyzed according to Kirkpatrick’s (Galloway, 2005) levels. The analysis was done manually using Hsieh and Shannon’s (2005) conventional content analysis, namely inductive content analysis, which is a useful approach when there is limited information available on given phenomena. The questions posed to the nurse managers were divided according to Kirkpatrick’s model referring to Hsieh and Shannon’s direct analysis, namely deductive analysis, but the responses to each question were categorized according to the conventional approach. This offered an opportunity to look at the phenomenon from the participants’ own perspective (Graneheim & Lundman, 2004). All written material was read to obtain an understanding of the text. The codes were formed from the text capturing the key thoughts. The codes were sorted into categories according to how they were related and linked (Graneheim & Lundman, 2004). We used Microsoft Word as data management and analysis software (La Pelle, 2004); the data were coded manually by making notes in printed documents. The coding was Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
conducted by one person experienced in qualitative analysis (M. L.) and was confirmed by a second researcher (M.V.). The credibility of the analysis process was confirmed using representative quotations from the original text in the results, and “Qualitative studies—COREQ—Consolidated criteria for reporting qualitative research” also guided data reporting (Tong, Sainsbury, & Craig, 2007). Ethics Ethical permission for this research was obtained from the local university ethics committee (dated March 15, 2013, University of Turku) and permission for the collection of data was obtained from the research permission committees of six hospital districts. In effect, one of the hospital districts granted permission for two hospitals, another district provided permission for three, and the remaining four hospital districts held one permission for one psychiatric hospital. Participation was voluntary, and the data were treated in confidence. Results
Nurses’ ReactionsAfterTheir e-Learning Course—Managers’ Perspective (Reactions; Level 1 in Kirkpatrick’s Model). This varied from positive, neutral, or negative. A reaction refers to how the students reacted to the e-learning course from the nurse managers’ perspective.Positive reactions were described as enthusiasm, motivation, and fascination. A few of the participating nurses were really enthusiastic about the course and tried to raise extensive discussion in their work communities. (ID 23) The psychiatric nursing managers described that the nurses discussed the content of nursing more and they reflected more on their work. Moreover, from the nursing managers’ point of view, the ward culture was built up to lean toward favoring alternative methods that avoided coercive practices. The culture was also geared toward creating an environment that encouraged more positive attitudes toward the development of the work itself. The nursing managers also reported neutral reactions among nurses, such as neutral or expectant feelings about the course. The nursing managers reported that there were also negative reactions, namely anger, fear, doubt, resistance, irritation, underestimation, feelings of being coerced, distraction, and laziness.
Participants’ Characteristics Most participants (n = 28) were female (68%). The biggest age group was 51–60 years (36%). Two thirds (68%) had 1–10 years of work experience at the managerial level. Half of the respondents (50%) were ward managers and over half (57%) had participated in the e-learning course. See Table 1.
Table 1. Characteristics of Study Participants (N = 28)
Sex Female Male Age (years) 31–40 41–50 51–60 61+ Work position Nursing director Ward manager Ward deputy manager Other Managerial experience (years) 1–10 11–20 21–30 Participated in ePsychNurse.Net course Yes No
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
N
%
19 9
68 32
8 9 10 1
28 32 36 4
6 14 5 3
21 50 18 11
18 5 5
64 18 18
16 12
57 43
Some nurses were angry and frustrated and did not see why they should participate in this course. Was their professional competence in doubt because they were “forced” into training? (ID 12) The nursing managers reported that some nurses were apprehensive about the course and its assignments. Moreover, the nursing managers noticed that the nurses initially resisted the course and the new learning method raised anxiety. Some nursing managers also thought that the nurses felt their professional knowledge was underestimated. In addition, the nursing managers pointed out that some nurses felt coerced to participate, and they had doubts about the course. The nursing managers also felt that some nurses were lazy during the course. Nurse’s Knowledge Levels and Attitudes Related to Psychiatric Care—the Managers’ Perspective (Learning; Level 2 in Kirkpatrick’s Model). This had seven categories. The categories were internationality, mental health act, ethical issues, treatment of aggressive patients, alternative methods, knowledge level, and discussion about the course content. Learning means how much the students learned from the e-learning course from the nursing managers’ perspective. The nursing managers reported that the nurses learned more about the relevant legislation, ethical issues, and how to cope with aggressive patients. Knowledge of the legislation was increased, and teamwork became a part of aggressive patients’ treatment. (ID 2) 43
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
The nursing managers felt that after the course, the nurses were knowledgeable about alternative methods related to avoiding coercion. Again, general discussion about the course topic supported knowledge transfer between nurses. In my opinion, the course content provided good information on the care of aggressive patients. (ID 5) The nursing managers’ views on the nurses’ learning on the course raised four issues related to the nurses’ attitudes: the nurses’ awareness of their own attitudes, change in attitudes, attitudes toward patient care, and reduction in the use of coercive methods. First, the nursing managers reported that the nurses’ awareness of their own attitudes changed through self-reflection exercises. Knowing your own attitudes through reflection, and scrutinizing your own actions is perhaps the most important thing considering attitudes. (ID 2) The nursing managers also described how the overall atmosphere and attitudes toward aggressive patients became more positive. They felt that the nurses’ attitudes toward patient care and treating patients became more individualistic and collaborative. The nurses were also more willing to use more alternative methods instead of physical restraints. The nursing managers felt that the use of coercion was reduced after the course. Attitudes are seen in discussions in the work unit and the use of coercion has diminished. (ID 28) Nurses’ Knowledge Transfer to Clinical Practice—Managers’ Perspective (Behavior; Level 3 in Kirkpatrick’s Model). This had seven categories: legislation on coercion, nursing distressed and disturbed patients, alternative methods to reduce coercion, anticipation of aggressive situations, debriefing, teamwork, and common discussions. Behavior refers to how much the students changed their behavior as a result of the e-learning course, and how they were able to transfer the knowledge to practice. The nursing managers noted that the nurses’ better understanding of the legislation on coercion was apparent in clinical practice. Attention has begun to be paid to measures preventing aggression and seclusion has clearly diminished. (ID 5) The nursing managers reported that the nurses cooperated more with distressed and disturbed patients; they added outdoor activities and involved patients more in decisionmaking. Moreover, the nurses tried to anticipate aggressive situations more. The nursing managers reported that they noticed more nurse–nurse and nurse–patient discussion, debriefing was more systematic, and teamwork changed after the course. 44
In my opinion, de-briefing after aggressive situations has become more systematic among the personnel. (ID 20) The nursing managers also hoped that what was learned could be transferred more effectively to clinical practice. Seven categories were formed: prevention of aggressive situations, holistic patient care, evidence-based care, understanding of mental illnesses, use of reflection, development in attitudes toward coercion, and increase in discussion. The nursing managers wanted the nurses to use different preventive methods in aggressive incidents, and that, in the future, nursing of distressed and disturbed patients could be more holistic. I would like to see greater inclusion of patients in decision-making related to their own care. (ID 12) The nursing managers hoped that nursing interventions could rely on evidence-based care, and that nurses could have a better understanding of mental illnesses. All the articles from the course are printed out to make a file for the personnel to read, and I wish that use could be made of the information, especially in our practical work. (ID 15) The nursing managers also pointed out that nurses could use more reflection in their work and that nurses’ attitudes toward use of coercion could be revised. Moreover, nurses could discuss more often with both patients and team members. Impacts of the e-Learning Course on Clinical Work—the Managers’ Perspective (Impact; Level 4 in Kirkpatrick’s Model). Impacts as a final result refer to the outcomes of the e-learning course observed by the organization. There were six main categories related to the advantages of the e-learning course. The categories were cooperation, anticipation of aggression, overall difference in attitudes, acknowledgment of patients, awareness of one’s own work, and new knowledge. Better cooperation among nurses, and between patients and nurses, was one advantage of the e-learning course. Anticipation of aggressive situations was seen to be related to patient treatment and handling. Overall changes in attitudes toward patients were seen as an advantage of the course; better understanding of patients’ needs and more humane treatment result from changed knowledge. The nursing managers reported that nurses were more aware of their own work. Anticipation of aggression is used more to avoid coercive practices, thereby, the use of seclusion has the potential to be diminished. (ID 1) Nursing managers also reported disadvantages of the e-learning continuing education course. These were the time used and the need for substitute nurses. Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Development of coercive practices Nurses’ work development
Patient treatment
Common discussions Discussion about the course content
Development in attitudes toward coercion Increase in discussion Teamwork
Alternative methods
Ethical issues Negative feelings
Treatment of aggressive patients
Mental Health Act Neutral feelings
Attitudes toward patient care Reduce in coercion
Knowledge level
Use of reflection
Understanding of mental illness
Evidence-based care
Awareness of one’s own work New knowledge
Organizational resources Anticipation of aggression Overall difference in attitudes Acknowledge patient
Implications Disadvantages
Time consuming Cooperation
Advantages Used knowledge
Internationality Positive feelings
Attitudes Knowledge Reaction
This study aimed to evaluate the e-learning continuing education course from the perspective of nursing managers in psychiatric hospital organizations. Kirkpatrick’s evaluation
Learning
Discussion
Reaction
See summary of categories (Table 2).
Table 2. Summary of Categories
The knowledge and skills of the personnel improved, which is apparent in how they encounter the patient and in care. (ID 17)
Behavior
Moreover, the nursing managers noticed that the use of coercion had become more humane, for example, the nurses wanted to shorten the period of seclusion, and they wanted to redecorate the seclusion rooms so that they were more comfortable for the patients. In addition, the nurses were more aware of their own work and more positive about development projects. The nursing managers felt that the organization’s input into e-learning continuing education course was one advantage. Moreover, the nursing managers pointed out that the nurses’ competence was better after the e-learning course, and communication among nurses was growing.
Prevention of aggressive situation Holistic patient care
Impact
Knowledge development in the future
The nurses’ interactive skills for dealing with aggressive patients improved. (ID 23)
Legislation on coercion Care of distressed and disturbed patients Alternative methods to reduce coercion Anticipation of aggressive situations Debriefing
Four categories were identified as relative to implications of the e-learning course’s effectiveness. The categories were cooperation, patient treatment, development of coercive practices, and nurses’ work development. The nursing managers described that in the category of cooperation, there were growing numbers of discussions among team members which increased positive communication. There were also reports from nursing managers which suggested that they felt as if the nurses were given new tools for clinical practice and patient treatment. They explained that at close observation, it seemed that these changes were a result of the e-learning course. The managers noticed that the coercive practices of the nurses after the e-learning course seemed less frequent than what they were before the course. They noted that any observed use of coercion with a patient was acted out in a more humane manner. The nursing managers stated that they were able to identify and calculate a nurse’s “work development” after the e-learning course. They explained that the nurses became “aware of their work,” and that the idea of developing and maintaining an internal clinical work ethic was more positive. They exclaimed that these quick results were an indicator that through this course model, the organization’s resources for education could be put to good use.
Nurses’ awareness of their own attitudes Change in attitudes
The course took a lot of time and there were substitute nurses on the ward. (ID 1)
Cooperation
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
45
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
model was used as an evaluation frame. This study showed that the nursing managers noticed that the nurses had different reactions to the e-learning course, which were mostly positive; the result is partially supported by earlier research among participants in the e-learning courses (Cook et al., 2008). Neutral and negative feelings by the nurses were also described by the managers, which is fairly understandable as e-learning is a new learning method in psychiatric hospitals and there may be some resistance to it among nurses (Kontio et al., 2011). In this study, the managers noticed that learning did indeed take place, especially in the knowledge and attitudes in the nurses’ clinical practice. They also reported some knowledge transfer from the course to clinical practice. This is similar to the findings of Lahti et al.’s (2013) study, in which the nurses’ perceptions of knowledge transfer to daily practice were examined, and the nurses reported a clear transfer of knowledge from an e-learning course to clinical practice. In contrast to earlier studies among nurses (Kontio et al., 2011), in this study there was a change in attitudes noticed by the nursing managers toward distressed and disturbed patients. It seemed that the nurses and nursing managers shared the same opinion about the effectiveness of knowledge transfer. Interestingly, the nurses did not experience the change in the atmosphere and attitudes the same way as the nursing managers had observed. This study brings a novel approach to evaluating the e-learning course from the perspective of the nursing managers, as in Finland, most of the continuing education in health care is paid for by the employees (Hotti, 2012). In this study, the nursing managers identified several issues where the nurses’ behavior in clinical practice had changed. This is also supported by Kontio et al. (2011), who found that the nurses’ knowledge about the legislation on coercion increased. This study also looked into understanding what the nursing managers hoped to see their nurses implementing in clinical practice after the e-learning course. It was noted that the nursing managers were hoping to observe holistic patient care and evidence-based practice. This was similar to what Lahti et al. (2013) found regarding the nurses’ own perspective of what knowledge they were hoping to transfer from an e-learning course to daily practice. The nursing managers reported advantages of the e-learning course for clinical practice, stating that this kind of an e-learning course with a lot of evidence-based material raised the nurses’ willingness to develop their work and approve research. Eizenberg (2011) has also suggested that nurses who think more positively about research are more likely to implement it in their own work. Our findings raised the important issue that the managers deemed discussion and communication among nurses important. In most respects, the nursing managers found that discussion was broadening among the nurses. This finding is also supported by Wright et al. (2011). 46
Implications for Psychiatric Organizations Given the shortage of nurses and the high turnover rates, the competition for nurses has begun (WHO, 2007). We therefore need to know more about how education might support nurses’ daily work (Lahti et al., 2013) but also motivate them and increase their commitment (Gould & Fontenla, 2006). Continuing education also enhances nurses’ professional skills, which will also affect how they are retained in the profession (Choi et al., 2012). Implications for Research More evaluation on levels 3 and 4 is needed to really understand the impacts of e-learning continuing education on clinical practice, that is, what kind of knowledge transfer took place after the e-learning course and how permanent this knowledge transfer is. Parker et al. (2011) call for more emphasis to be placed on the real change and its implications for practice. More emphasis could be placed on the use of quantitative designs and the probability of predicting certain outcomes that are more difficult to measure. Such things as cost-effectiveness, patient improvement, and wellness should be heavily considered when evaluating continuing education courses such as the one used in this study. However, use of qualitative research methods could help quantify the results, and bring to light more of the information related to certain observable behavioral changes that result from knowledge transfer. Moreover, qualitative research could help identify and verify possible final impacts on clinical practices that could develop from continuing education courses. Methodological Considerations The results were based solely on the respondents’ subjective perceptions. However, we managed to reach quite an informative group of respondents among psychiatric nursing managers in our study hospitals. The participation rate in this research was very low. This may be due to nursing managers’ demanding work (Lee & Cummings, 2008) and lack of time. Voluntary participation may also cause bias as those with more positive attitudes participate more actively than those with more negative attitudes (Kisely & Kendall, 2011). Moreover, those nursing managers who had participated themselves in the e-learning course may hold more positive opinions related to the e-learning course than nursing managers who had not attended such a course. However, all nursing managers had been informed about the e-learning course when they decided on the nurses’ participation in the course. Moreover, the coding was done by one researcher, and this may likewise cause bias. In addition, the nature of the data was narrow due to the participants’ brief responses, which may affect the analysis and results. Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
Conclusion We showed that Kirkpatrick’s evaluation model was a usable way to evaluate an e-learning continuing education course in a psychiatric organization from the nursing managers’ point of view. These findings are novel and promising within various continuing education situations where Finnish nurses participate approximately 4 days out of a year in continuing education efforts (Hotti, 2012). The e-learning course discussed in this study was 3–6 months long. Even though the transference of knowledge and the revision of daily practices did take time to be acclimated into the psychiatric nurses’ daily work flow, the nursing managers were able to identify positive reactions to the course and were able to observe considerable changes during that time. There were also findings which determined that the clinical practices among the nurses who participated in this e-learning course had sufficiently improved. However, there is still a need to evaluate e-learning continuing education courses in the field of psychiatric nursing more systematically to ensure appropriate education for psychiatric nurses. Acknowledgment This study was supported by EduMental, Tekes—the Finnish Funding Agency for Technology and Innovation (40245/12). References Bjørk, I., Tørstad, S., Hansen, B., & Samdal, G. (2009). Estimating the cost of professional developmental activities in health organizations. Nursing Economics, 27(4), 239–244. Brown, K. (2005). An examination of the structure and nomological network of trainee reactions: A closer look at “Smile Sheets.” Journal of Applied Psychology, 90(5), 991–1001. doi:10.1037/0021-9010.90.5.991 Choi, P., Cheung, K., & Pang, C. (2012). Attributes of nursing work environment as predictors of registered nurses’ job satisfaction and intention to leave. Journal of Nursing Management, 21(3), 429–439. Cleary, M., Horsfall, J., Happell, B., & Hunt, G. (2013). Reflective components in undergraduate mental health nursing curricula: Some issues for consideration. Issues in Mental Health Nursing, 34(2), 69–74. Cook, D., Levinson, A., & Garside, S. (2008). Internet-based learning in the health professions. Journal of the American Medical Association, 30(10), 1181–1196. Covell, C. (2009). Outcomes achieved from organizational investment in nursing continuing professional development. Journal of Nursing Administration, 39(10), 438–443. Curran, V., & Fleet, L. (2005). A review of evaluation outcomes of web-based continuing medical education. Medical Education, 39(6), 561–567. doi:10.1111/j.1365-2929.2005.02173.x
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
Eizenberg, M. (2011). Implementation of evidence-based nursing practice: Nurses’ personal and professional factors? Journal of Advanced Nursing, 67, 33–42. doi:10.1111/j.1365-2648.2010.05488.x Forsetlund, L., Bjørndal, A., Rashidian, A., Jamtvedt, G., O’Brien, M., Wolf, F., . . . Oxman, A. (2009). Continuing education meetings and workshops: Effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews, (2), CD003030. doi:10.1002/14651858.CD003030.pub2 Galloway, D. (2005). Evaluating distance delivery and e-learning. Performance Improvement, 44(4), 21–27. Gillard, S., Simons, L., Turner, K., Lucock, M., & Edwards, C. (2012). Patient and public involvement in the coproduction of knowledge: Reflection on the analysis of qualitative data in a mental health study. Qualitative Health Research, 22(8), 1126–1137. Gould, D., & Fontenla, M. (2006). Commitment to nursing: Results of a qualitative interview study. Journal of Nursing Management, 14, 213–221. doi:10.1111/j.1365-2934.2006.00577.x Graneheim, U., & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105–112. Hamtini, T. (2008). Evaluating e-learning programs: An adaptation of Kirkpatrick’s model to accommodate e-learning environments. Journal of Computer Science, 4(8), 693–698. Higginbottom, G. (2004). Sampling issues in qualitative research. Nurse Researcher, 12(1), 7–19. Hotti. (2012). Municipality continuing education in health and social sector in year 2011. Local government employers. Retrieved May 22, 2014, from http://www.kuntatyonantajat.fi/ fi/ajankohtaista/tilastot/tiedustelut-ja-selvitykset/ taydennyskoulutus/sosiaali–ja-terveydenhuollon -taydennyskoulutus-011/Documents/sosiaali-%20ja%20 terveydenhuollon%20t%C3%A4ydennyskoulutus%202011.pdf Hsieh, H.-F., & Shannon, S. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. Joyce, P., & Cowman, S. (2007). Continuing professional development: Investment or expectation? Journal of Nursing Management, 15, 626–633. doi:10.1111/j.1365-2834.2007.00683.x Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, January, 54–59. Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs (3rd ed.). San Francisco: Berrett-Koehler Publishers. Kisely, S., & Kendall, E. (2011). Critically appraising qualitative research: A guide for clinicians more familiar with quantitative techniques. Australasian Psychiatry, 19(4), 364–367. Kontio, R., Lahti, M., Pitkänen, A., Joffe, G., Putkonen, H., Hätönen, H., . . . Välimäki, M. (2011). Impact of eLearning course on nurses’ professional competence in seclusion and restraint practices: A randomised controlled study
47
Impact of an e-Learning Course on Clinical Practice in Psychiatric Hospitals: Nurse Managers’ Views
(ISRCTN32869544). Journal of Psychiatric and Mental Health Nursing, 18, 813–821. La Pelle, N. (2004). Simplifying qualitative data analysis using general purpose software. Field Methods, 16(1), 85–108. Lahti, M., Kontio, R., Pitkänen, A., & Välimäki, M. (2013). Knowledge transfer from an e-learning course to clinical practice. Nurse Education Today, 34, 842–847. Lee, H., & Cummings, G. (2008). Factors influencing job satisfaction of front line nurse managers: A systematic review. Journal of Nursing Management, 16, 768–783. doi:10.1111/j.1365-2834.2008.00879.x Liu, W.-I., Rong, J.-R., & Liu, C.-Y. (2014). Using evidence-integrated e-learning to enhance case management continuing education for psychiatric nurses: A randomised controlled trial with follow-up. Nurse Education Today, 34(11), 1361–1367. doi:10.1016/j.nedt.2014.03.004 Lowe, M., Rappolt, S., Jaglal, S., & Macdonald, G. (2007). The role of reflection in implementing learning from continuing education into practice. Journal of Continuing Education in the Health Professions, 27(3), 143–148. Maxwell, S., & Mucklow, J. (2012). e-Learning initiatives to support prescribing. British Journal of Clinical Pharmacology, 74, 621–631. Ministry of Social Affairs and Health. (2004). Recommendation for further education foe health care staff. Handbooks of the Ministry of Social Affairs and Health, Finland, Helsinki, 3/2004. ISBN: 952-00-1478-0. Moore, J., Dickson-Deane, C., & Galyen, K. (2011). E-Learning, online learning and distance learning environments: Are they the same? Internet and Higher Education, 14, 129–135. Nalle, M., Wyatt, H., & Myers, R. (2010). Continuing education needs of nurses in a voluntary continuing nursing education state. Journal of Continuing Education in Nursing, 41(3), 107–115. Organization for Economic Co-operation and Development. (2011). Health at a glance 2011: OECD indicators. Paris: Author. doi:10.1787/health_glance-2011-en Parker, K., Burrows, G., Nash, H., & Rosenblum, N. (2011). Going beyond Kirkpatrick in evaluating a clinician scientist program:
48
It’s not “If it works” but “How it works.” Academic Medicine, 86(11), 1389–1396. Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability, 22(3), 215–225. Sajeva, M. (2009). E-Learning: Web-based education. Current Opinion in Anaesthesiology, 19(6), 645–649. Sandelowski, M. (2000). Focus on research methods whatever happened to qualitative description? Research in Nursing & Health, 23, 334–340. Sears, K., Cohen, J., & Drope, J. (2008). Comprehensive evaluation of an online tobacco control continuing education course in Canada. Journal of Continuing Education in the Health Professions, 28(4), 235–240. Smidt, A., Balandin, S., Sigafoos, J., & Reed, V. (2009). The Kirkpatrick model: A useful tool for evaluating training outcomes. Journal of Intellectual & Developmental Disability, 34(3), 266–274. doi:10.1080/13668250903093125 Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349–357. Välimäki, M., Lahti, M., Scott, A., & Chambers, M. (2008). The changing face of psychiatric nursing—care or control. ePsychNurse.Net. Towards improved quality developing nurses’ continuing vocational training in psychiatric hospitals and inpatient units. ISBN: 978-951-29-3576-5. University of Turku, Finland. World Health Organization. (2007). Atlas: Nurses in mental health 2007. ISBN 978 92 4 156345 1. Retrieved April 1, 2013, from http://www.who.int/mental_health/evidence/nursing_atlas _2007.pdf Wright, D., Lavoie-Tremblay, M., Drevniok, U., Racine, H., & Savignac, H. (2011). Relational dimensions of a positive integration experience for new mental health nurses. Archives of Psychiatric Nursing, 25(3), 164–173. doi:10.1016/j.apnu.2010.07.008
Perspectives in Psychiatric Care 52 (2016) 40–48 © 2015 Wiley Periodicals, Inc.
