Evaluation and Program Planning 49 (2015) 76–85

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Piloting online WellnessRx learning modules: Demonstration of developmental evaluation Katharina Kovacs Burns *, Leah Gramlich, Lana Bistritz, Linda McCargar, Karin Olson, Melita Avdagovska Edmonton Clinic Health Academy, Health Sciences Council, 3-398 ECHA, University of Alberta, 11405–87 Avenue, Edmonton, AB, T6G 1C9, Canada

A R T I C L E I N F O

A B S T R A C T

Article history: Received 26 April 2014 Received in revised form 26 September 2014 Accepted 18 November 2014 Available online 18 December 2014

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules. ß 2014 Elsevier Ltd. All rights reserved.

Keywords: Pilot Learning modules Online Curriculum integration Nutrition Physical activity Health sciences students Developmental evaluation

1. Introduction Health systems are being forced to address the six main risk factors responsible for most of the preventable diseases: dietary risks, tobacco smoking, high body-mass index, high blood pressure, physical inactivity and high cholesterol (World Health Alliance, 2004; World Health Organization, 2009; Health Metrics and Evaluation, 2010). Strategies are needed to prepare health professionals and health systems to better monitor and reduce these health risks (World Health Alliance, 2004). This includes training health professionals to counsel patients/public on reducing their risk factors through behavior modification. The literature on health professional education programs shows deficiencies or gaps around two specific risk factor areas–nutrition

* Corresponding author. Tel.: +1 780 920 896. E-mail addresses: [email protected] (K. Kovacs Burns), [email protected] (L. Gramlich), [email protected] (L. Bistritz), [email protected] (L. McCargar), [email protected] (K. Olson), [email protected] (M. Avdagovska). http://dx.doi.org/10.1016/j.evalprogplan.2014.11.009 0149-7189/ß 2014 Elsevier Ltd. All rights reserved.

and physical activity (Adams, Lindell, Kohlmeier, & Zeisel, 2006; Crogan, Shultz, & Massey, 2001; Garry, Diamond, & Whitley, 2002). In Alberta, Canada, a joint study between the University of Alberta Health Sciences Council (UA) and Alberta Health Services (AHS) found that the knowledge and skills of health professional practitioners and students were inconsistent, deficient or lacking in nutrition, physical activity and wellness (NPAW), which impacted their confidence to advise patients and the public. In addition, the study found deficiencies and gaps in NPAW content within health professional students’ curricula (Mackenzie et al., 2009; Kovacs Burns & Gramlich, 2012). Recommendations included the development of education components focusing primarily on NPAW. The result was WellnessRx, a health initiative focusing on healthy living, primarily nutrition and physical activity, through education, knowledge translation and community engagement. The priority for the WellnessRx education component was to produce learning modules on nutrition and physical activity, which could be integrated into health professional students’ education curricula. Paralleling this development of WellnessRx and its education component was a developmental evaluation

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

framework(Gamble, 2008; Patton, 2009, 2011), which aligned evaluation of processes and outcomes for all aspects of WellnessRx as developed to meet complex strategic priorities within and across the three components of education, knowledge translation and community engagement. For the education component, this meant applying a robust evaluation at an early pilot phase of the learning modules, to gather real-time and real-world results from targeted audiences regarding their perceived challenges, benefits or values, and overall suggestions for clearly defining or refining the development of the WellnessRx learning modules. In addition, the developmental evaluation results are intended to guide the further development or refinement of the WellnessRx initiative and its preliminary process and outcomes plan outlined in a formative logic model. The intent of this paper is to showcase the application of the developmental evaluation with two learning module pilots as part of the WellnessRx education component. The objectives of this paper are to: (1) demonstrate the effectiveness of applying the developmental evaluation framework with the WellnessRx learning module pilots; and (2) describe the relevance of the evaluation results for informing, defining or altering the development of the education modules and their proposed pathway within the WellnessRx logic model. 2. Overview of WellnessRx and its significance in addressing a prescribed need The origin and development of WellnessRx as a health initiative are detailed in other papers (Kovacs Burns & Gramlich, 2012). WellnessRx had its origin in Alberta, Canada in response to an issue identified by Alberta Health Services (AHS) in 2007 regarding the lack of confidence of their health professionals to talk with their patients about NPAW. A joint study between AHS and the University of Alberta (UA) Health Sciences Council of eight Faculties was conducted in 2008–2009 (Mackenzie, Kovacs Burns & Gramlich) and resulted in findings which supported a key recommendation to develop education components to address the identified deficiencies with knowledge and skills regarding NPAW. Responding to this identified need, UA faculty and researchers along with AHS staff pooled their expertise, and created WellnessRx. The vision of this initiative was that the Alberta society embraces healthy lifestyles by enhancing nutrition and physical activity. The model of WellnessRx encompassed three components (education, knowledge translation and community engagement) all informed by research evidence and experience. The aim of the education component was ‘‘to enhance the knowledge, skills, and attitudes of health professional students and practitioners in the areas of nutrition and physical activity. This will be achieved by providing an integrated curriculum of online learning modules and resources’’ (WellnessRx Strategic Plan, 2013). Priority initiatives for education were identified by broad stakeholder groups (i.e. faculty, health professionals, researchers, educators, and students), and included the identification of content areas around NPAW to be developed into five education learning modules, which were proposed as: (1) The Role of Nutrition and Physical Activity in Wellness, (2) Nutrition and Physical Activity in Self-Health, (3) Nutrition and Physical Activity across the Life-Cycle, (4) Nutrition and Physical Activity in Disease, and (5) Nutrition and Physical Activity in Policies and Practices. The WellnessRx Logic Model presented in the Inline Supplementary Fig. 1 (Readers are requested to refer online for the supplementary materials. The online link is provided in the Appendix section at the back of the article) highlights the preliminary concepts, processes and outcomes anticipated for the education, knowledge translation and community engagement components. Developing the logic model concepts for only the

77

education component, required that an education working group and other experts on nutrition, physical activity, wellness, education and curriculum development, deliberate and identify learning objectives along with knowledge, skills and attitudes (KSAs) for each module. These would be the minimum for health professional students and professionals to learn in order to be prepared and confident to advise clients/patients about nutrition and physical activity as well as for their own self-health decisions. The details of this process involving the planning and development of these learning modules for the WellnessRx education component are published elsewhere (Bistritz et al., 2015). The intent was to begin implementing the WellnessRx logic model; developing and evaluating the learning modules for delivery, content and outcomes; and assessing students’ knowledge and skills related to the module content and their challenges and benefits. 3. Developmental evaluation framework for WellnessRx The details of the evolution of the WellnessRx developmental evaluation framework are described in detail in a companion methodology paper (Kovacs Burns et al., 2014). In brief, the approach applied includes the larger complex and formative logic model (Fig. 1) outlining anticipated aspects of the WellnessRx initiative to be developed, and a parallel developmental evaluation process adapted from Gamble (2008) and Patton (2009; 2011). Developmental evaluation was intended for evaluating complex innovative initiatives such as WellnessRx, where there is no precedent innovation to compare with, where diverse stakeholders have divergent views on content and uptake by targeted audiences, and where or how the learning modules will be integrated into curricula or training and expectations or outcomes of diverse users will be achieved. The framework was proposed to ensure that the evaluation was comprehensive, including formative evaluations where the results would inform or be directly applied in the practical development of the WellnessRx education component and its learning modules. This approach offers flexibility to allow for realtime and real-world experiences, activities and results to be incorporated in different ways to support, enhance or alter the development plan for WellnessRx and its intended outcomes, either as proposed in Fig. 1 or as redefined, based on the evaluation results. The developmental evaluation steps adapted from those described by Patton (2009) included: (1) evaluation of need for initiative or activities; (2) identification of stakeholders to be involved with different aspects of the initiative and its evaluation; (3) evolution of the initiative or activities based on evaluation results; (4) adaptation of the WellnessRx initiative including its logic model and developmental evaluation, based on initiative or activity outcomes and measures. These developmental evaluation steps will be similarly applied to the education component of WellnessRx and its learning modules. This paper specifically describes a demonstration of the developmental evaluation process with the WellnessRx education component as related to its learning module pilots. The resulting outcomes and recommendations will be summarized to inform the further development of the proposed WellnessRx initiative and its education component plan including the learning modules, as identified within Fig. 1. 4. Methodology—Piloting and evaluating the learning modules A multi-phased, multiple methods design was chosen for the developmental evaluation of the learning modules. The objectives were to: (1) Pilot the two developed WellnessRx education learning modules with health sciences Faculties at the University of Alberta; (2) Assess change in students’ knowledge and evaluate the learning module for its value or benefit in the curriculum, delivery approach, strengths, weaknesses and areas needing

78

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

improvement; and (3) Synthesize evaluation comments and recommendations to facilitate planning and application in further development of the two modules and others proposed. Pilot methodology was incorporated into the developmental evaluation as the two modules were still in early development phases but were in need of testing on as small-scale with a sub-set or convenience sample of the targeted audience such as students within Faculties (Arain, Campbell, Cooper, & Lancaster, 2010; van Teijlingen & Hundley, 2002). Pilots are small-scale implementations of innovations appropriate for developmental evaluation. Sample size is not the focus of the pilot, so data must be interpreted with caution (Lancaster, Dodd, & Williamson, 2004). The purpose and value of pilots is to acquire users’ general perceptions of the modules and trends which could be used to guide decisions of the Education Working Group as to whether or not the modules are feasible (Morin, 2013). This also determines if the development of the modules continues for a larger scale implementation with more diverse audiences (Kilanowski, 2011; Thabane et al., 2010; van Teijlingen & Hundley, 2002). As well, others interested in doing similar evaluations with pilots as part of complex innovations may benefit from this study and its findings. Following the approved ethics protocol for participants consenting to take part in this study, the Evaluation Working Group of the WellnessRx initiative representing the Health Sciences Faculties at the University of Alberta (UA) invited Faculties to participate in the pilots, which included identification of ‘champion’ instructors, their targeted courses for pilot integration and student groups. There were three parts to the evaluation of each module: (1) pre-post module assessment of students regarding their knowledge of the content, to determine if there was any gain in students’ knowledge around nutrition and physical activity, attributable to each module; (2) integrated mixed methods (Morse & Niehaus, 2009; Ostland, Kidd, Wengstrom, & Rowa-Dewar, 2011) in which interviews or focus groups with students would confirm and elaborate on their responses to the end-of-module evaluation surveys; and (3) interviews/focus groups with instructors to provide their understandings or perceptions of student responses and experiences. The specific student pre-post tests and evaluation surveys were designed for online anonymous access and completion by students. The pre-post surveys assessed students’ knowledge on the subject matter of the specific module. The evaluation surveys contained original questions as no validated instruments were located for this type of evaluation. For this survey, questions explored perceptions regarding (1) ‘satisfaction’, (2) ‘course and course content’, (3) ‘course delivery, processes and technology’, (4) ‘assessing learning’, and (5) ‘conclusion/recommendations’. The survey also contained a question regarding student interest in participating in interviews or focus groups. Follow-up contact was made by a person external to the study. Instructors were also invited to participate in interviews/focus groups. Interview and focus group questions for both groups paralleled those from the evaluation survey but were more exploratory in seeking clarification or understanding of survey responses, and in facilitating discussions around recommendations regarding the modules—for example, whether to keep or alter the module objectives and content, resources, activities, online format and so on. Data from the pre-post tests and evaluation surveys were collected into central online depots, accessible by only two individuals on the evaluation team. For both, descriptive analysis (frequencies and percentages) was done with an online statistical program. For open-ended student evaluation survey questions as well as the interviews and focus groups, data were thematically analyzed. Three raters reviewed and coded the interview and focus group transcripts. A coding framework was developed in which common themes and sub-themes emerged. The three raters approved the coding framework and applied it in the analysis of

the qualitative data. Inter-rater reliability pertaining to the coding of the data was ensured through this process. The survey openended comments and the interview and focus group transcripts were manually analyzed and coded. This general methodology applied to both phases involving the two pilots and their evaluations. Additional details for each pilot phase follow: 4.1. Phase 1 pilot study and evaluation Learning module 2 on ‘Self-Health’ was the first to be completed for online delivery and the pilot implementation. The learning objectives for this module against which the pre-post knowledge assessment of students and the post-course evaluation were aligned were as follows: (1) perform a history and self-assessment with respect to nutrition and physical activity; (2) conduct a selfhealth intervention and consider how to relate this to working with patients/clients; and (3) plan for current and future success on own health initiatives and relate this to how it will work with patients/clients. 4.2. Phase 2 pilot study and evaluation This second pilot was with module 3 on ‘Nutrition and Physical Activity across the Life-Cycle’. Findings including recommendations made in the Module 2 pilot (discussed in Section 5.1) were analyzed and summarized to facilitate discussion and plans for the development of Module 3. Again, as with the previous module, this one was developed for online access including the reading resources, pre-post assessment and post course evaluation survey. The objectives of this module used for the assessment and evaluation were to: (1) Recognize that nutrition and physical activity play a role in the health and wellness of pre-and post-natal mothers and infants; (2) Recognize that nutrition and physical activity play a role in the health and wellness of children and teenagers (ages 2–18 years); (3) Recognize that nutrition and physical activity play a role in the health and wellness of adults (ages 19–65); and (4) Recognize that nutrition and physical activity play a role in the health and wellness of seniors (over the age of 65). 5. Results 5.1. Applying findings of first pilot to second pilot Module 2 was the first pilot completed, and results were analyzed with the intent that the findings, including key recommendations, would inform the Education and Evaluation Working Groups and assist with the facilitated discussions on what needed to be altered in the planning and development of not only Module 2 but also Module 3 which was the second pilot. Some of these results and recommendations (described in more detail throughout the following results), included having more focused objectives and expectations, more relevant context for the module for personal or patient focus, less activities and making them more varied including some case-based, less redundancy in activities, less resources/information or more summarized information to read because of time limitations, and more integration of the module with actual curriculum course work as opposed to it being an add-on. Recommendations to keep certain aspects of Module 2 included resource links or websites students could use as references for themselves or patients, online format of module which made access flexible with time schedules, and generally the WellnessRx education content as most felt it was missing from their curriculum. Most of these recommendations could be applied in the development of Module 3. One exception was the

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

recommended integration of the module into the curriculum. The obstacle was that the WellnessRx education component with its learning modules was evolving and the evaluation results of one module pilot was inadequate to facilitate this type of discussion or business case with Faculties.

79

Learning Modules, Personal and Professional Learning from Modules, Module Format and Content, Module Delivery, Challenges/Difficulties and Recommendations. Indicative of qualitative studies, quotes were captured under each theme, supporting the commonly expressed or at times unique views, regarding evaluated aspects of the module pilots.

5.2. Response rates for module pilots Reported student responses for the pre-post assessments were from medical and dental (M/D) students for both modules. For module 2, 110 of 151 M/D students took the pre-module assessment and 31 took the post-module assessment (about 35% response rate). For module 3, 198 of 199 M/D students did the pre-module assessment, while 57 completed the post-module assessment (about 35%). For module 2 on ‘Self-Health’ 30 out of 151 (20% response rate) M/D students participated in the post-course online evaluation survey, while 14 out of 47 (30%) nursing (N) students responded to the post-course evaluation survey. One M/D student came for an interview and three M/D students came for a focus group. No N students agreed to interviews or focus groups, which was their choice as per ethics. Two instructors participated in a focus group. In the pilot of module 3, 24 out of 199 (12% response rate) M/D students completed the post-course evaluation survey. There were three focus groups (n = 8 M/D students) and one interview (M/D student). One instructor was interviewed. 5.3. Students’ pre-post module knowledge assessment The Inline Supplementary Fig. 2 shows the differences in prepost grade results of M/D students for both modules. For module 2, grades for the pre-test ranged from the 26% to 95% with an average of 72% (n = 110 students). The post-module assessment shows more students with higher grades, with 61% as the lowest grade and 89% the highest. The average was 77% (n = 31). The limitation for this comparison is that the pre-post assessments were not mandatory, so 79 students chose not to complete the post test. For module 3, Fig. 2 shows that over 50% of the 196 students who completed the pre-module test achieved 60% grades (highest grade was 85% and lowest was 30%). Again, less students completed the posttest but as shown in Fig. 2, over two-thirds of the 57 students who completed the post-test received 60% as grades with about half of the 57 receiving 75% for their grades (the highest grade was 95% and the lowest was 40%). 5.4. Common quantitative and qualitative module evaluation responses from students Module 2 on ‘Self-Health’ was the first pilot completed, and results that included key recommendations, informed the discussions and decisions of the Education and Evaluation Working Groups to alter the planning and development of Module 3 for the next pilot. The responses from M/D and N students for the postmodule evaluation surveys and the interviews/focus groups for the two pilots were coded, analyzed and clustered into common themes. These themes are listed in the Inline Supplementary Table 1 (left and middle columns, respectively). The detailed codes and themes for the interviews and focus groups with students are provided in the Inline Supplementary Table 2. Because the thematic clusters for both the student surveys (left column of Table 1) and the interviews/focus groups (middle column) were very similar, these themes were further analyzed and integrated into the common themes, shown in the right column in Table 1. These latter themes were verified by the coding raters as reflective of the students’ responses for the two module pilots, and are as follows: Impressions/Satisfaction regarding WellnessRx

5.4.1. Impressions/satisfaction regarding WellnessRx learning module(s) Overall impressions and satisfaction with the WellnessRx learning modules set the tone for understanding what students perceived about them. For module 2, the survey evaluation results indicated a clear difference in perspectives between medical/dental (M/D) and nursing (N) students, but overall students’ satisfactions were low. Over 57% of the participating N students were ‘dissatisfied’/‘very dissatisfied’ while M/D students were more ‘neutral’ about their overall satisfaction with both modules (37% module 2; 46% module 3). About one-third of M/D students were ‘satisfied/very satisfied’. These responses were confirmed in the qualitative components. Responses regarding the modules were more negative reflecting that the modules were ‘‘add-ons to their regular coursework’’, taking time from their main studies, being part of their grade and therefore mandatory, and not seen necessarily as worthwhile or having utility for them personally or professionally. More students expressed that they were ‘‘confused with the expectations of the modules’’ related to their regular gastroenterology and nutrition course. They felt that the modules need introductions and their importance and relevance in context with their regular courses had to be stressed. Other key words or phrases identified included: ‘confusing’, ‘too much work/effort’, ‘time consuming’, ‘too many assignments’, ‘little value to training as a doctor’, ‘redundant exercises’, ‘too heavy’, ‘lots of resources’, ‘lengthy’, ‘waste of time’, ‘lacked direction’, ‘overwhelming’ and ‘better off in earlier part of program’. The interviews and focus group responses substantiate some of the key words identified in the survey responses, as shown in Table 2. For example, students said that the modules presented them with some competing interests, because they were ‘‘outside of the curriculum and we have the core thing that we have to learn which is our lectures on which we are going to be tested.’’ In another example, students indicated that they were overwhelmed with reading in the modules, as ‘‘there were lots of resources. Certainly if you wanted to dive into anything or go gung ho, you could do that.’’ Specifically, for module 2, less than one-third of M/D students said they were most satisfied with the’ resources and information’ (30% respondents) followed closely by ‘that it was an online course’ (27%). N students were most satisfied that it was an online course (36%), but there were over 21% who were unsure what they were satisfied with. The Inline Supplementary Fig. 3(a) illustrates the details of what both the M/D and N students were ‘most satisfied’ with. Regarding module 3, 71% M/D students indicated that they were most satisfied with ‘resources and information’. Another 13% were most satisfied that ‘it was an online course’. During the interviews/focus groups, many mentioned the large number of resources, which they appreciated, but did not have the time to open. One student gave this perspective, which many students shared: All those resources, I am sure they are good and I opened quite a bit of them, but I will be honest that I didn’t open all of them in the assignments because there were so many. I liked that there was a section that said that these are for sure the resources that you need to look at and there was a section with all the others, but it would be nice to have just like one resource that kind of would be the go to resource for any type of a practitioner that would refer to the major points. Likewise, students were asked what they were least satisfied with regarding the WellnessRx learning modules. The Inline

80

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

Supplementary Fig. 3(b) shows the results. For module 2, M/D and N students were least satisfied with ‘activities’ followed by ‘resources and information’. One N student felt that ‘‘the activities took a very long time to complete, and the questions could have been much less redundant.’’ A M/D student said: ‘‘There were so many separate and not cohesive things to do that most people rushed through the assignments and did not get as much value out of it.’’ For module 3, M/D students also identified ‘activities’ as least satisfying (58%), and almost one-third were ‘not sure’, as shown in Fig. 3(b). During focus groups regarding module 3, one student indicated that ‘‘. . . it was good. . .. It encouraged a lot of learning rather than just completing the task. I thought the case based tutorials were very good. . ..’’ A large number of M/D and N students said that they were satisfied with a number of things. One M/D student summed it up: I did appreciate the layout of the module but I think that having it to be a self-directed activity is something I learn well from. Because of lectures and small group sessions, there is not a lot of self-directed learning and this was a good way to do that. I think the online component also helped because you could do it on your own time. You could fit it wherever you could. The Moodle platform was easy to navigate. And I did not have any issues with submissions or downloading any of the documents. I think in terms of the format it worked. There was less agreement that students’ expectations had been met. 43% of M/D students were ‘neutral’ that this module met their expectations. Another 23% ‘disagreed/strongly disagreed’. One M/D student said: ‘‘I understand the intention of WellnessRx and I can appreciate what it is trying to teach us, but I felt that the course/ module was too heavy in conjunction with everything else in the GI [GastroIntestinal] block.’’ However, another student saw this module as meeting an expectation that otherwise would not have been met had the module not been incorporated into the curriculum: I think the curriculum was structured appropriately. . ..I think it was also a good time in our training. We are post second year medical students and we are now starting to get a sense of a holistic view of health. Things are repeating from first year and coming up again and again, and nutrition and physical activity I think it’s not emphasized as much as it should be in our curriculum given its importance to health. So, I think it was placed kind of appropriately and it gives an extra emphasis on it compared to if we did not have this module. In addition to expectations, 50% of M/D students ‘agreed/ strongly agreed’ that their learning needs had been met, versus 23% who felt the opposite. Because this module was on ‘Self-Health’, there was a more personal learning outcome, and each student viewed their personal benefits differently. Some learning needs were met even when the student did not realize it. Knowledge wise I didn’t learn anything, but it forced me to pay attention to it. I know that pirogues are not healthy but now I am aware of what I was eating. When you start thinking critically then you do stop eating things that are not good for you. I was not happy in doing the module, but . . . I have awareness. I think I have done a couple of changes that are healthier. Similar results were found for module 3.

5.4.2. Personal and professional learning from modules—Satisfaction with learning, relevance and usefulness For both modules, students indicated their satisfaction about learning something that could be relevant personally, as well as

professionally. For module 2, around 75% M/D and 57% N students indicated this was true at a personal level. Some interview comments reflected this: ‘‘Self-evaluation is always relevant’’; or ‘‘I think having a good opportunity to try new things . . .and to see if it would work.’’ Regarding module 3, over 87% M/D students indicated that they were satisfied they learned about something relevant that they could apply personally, including ‘‘How to read the Canada Food Guide’’. No one said it was not relevant. Professionally, 80% M/D and 57% N students said ‘yes’ regarding module 2, and 88% M/D students said ‘yes’ for module 3. It was confirmed in the interviews and focus groups that the modules gave students something they would be able to use professionally. One M//D student said: ‘‘I did gain some knowledge, but the most important thing was being introduced to resources that I could refer patients to’’. Another M/D student said: ‘‘The ability to adequately take a history will be important. To actually get into the details on how a person eats, and even cooks, will help me understand more fully the kind of health someone is in. I will be able to help them with their diet and help them become more physically active.’’ Although students generally agreed that they were satisfied they learned relevant things from module 2, which they could apply personally and professionally, less agreed that the information was useful in making them change personally. More agreed that the information was useful professionally (73% of M/D and 79% N students). One student said: Professionally it was nice to know that there are many places on the internet that has good reliable articles. Some of the resources were not scientifically based articles but parenting sites that were easy to read and I know that they will help professionally. Good for referring people to. In addition to health professionals learning something from these modules, one student felt that module 2 ‘‘could be useful to someone who is not studying to become a healthcare professional, in terms of learning about the literature and that you cannot trust everything you read on websites.’’ For module 3, 81% M/D students ‘agreed/strongly agreed’ that after taking this module, they could see making changes personally. For 87% of M/D students, the information in this module will have practical applications to future practice. Some of the interview/focus group comments included previously also applied here.

5.4.3. Module format and content Less M/D students (53%) than N students (71%) ‘agreed/strongly agreed’ that the ‘learning module objectives’ were clearly stated. During the interviews, one M/D student felt that the objectives were very broad. Good. I think it was our journey to wellness. I think having them broad was useful and prompting us to think of what specifically we want to work on when sifting through the modules. For module 3, 75% of M/D students ‘agreed’ that the objectives were clearly stated, and that they were most likely able to achieve each one. In their comments, generally, the M/D students felt the objectives were appropriate. However, other responses indicated that their comfort levels with each objective varied in proportion to the amount of time they spent on each, as part of their course work. Some students indicated enhanced comfort with the information pertaining to the objectives of this module. One student said: ‘‘I feel like I have more knowledge to be able to help patients with their health and wellness. I will feel more comfortable doing so because I know more.’’ Another student was less confident and was ‘‘still unsure as to how I would advise patients.’’ As indicated previously, the majority of students (73% M/D and 57% N students for module 2; 95% M/D students for module 3)

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

found the modules relevant. However, their perspectives on specific approaches applied within each module show a different picture. Inline Supplementary Fig. 4 illustrates the student perspectives for overall module relevance, clarity of learning objectives and relevance of specific approaches applied within each of the two modules. Less N students, compared with M/D students, found most of the approaches for Module 2 to be relevant, except ‘on-line discussions’ and ‘combination activities’. As illustrated in Fig. 4, 50% or more of M/D students found ‘learning activities’, ‘independent reading’, and ‘narrative histories’ as useful approaches for module 2. For module 3, 63% or more of M/D students found all approaches relevant except ‘on-line discussions’. 79% of M/D students also found ‘tools/guides’ as part of module 3, to be relevant. When asked what approaches they found to be ‘most useful’, 43% of N and 57% of M/D students identified ‘learning activities’ as being most useful for module 2. Other approaches received less than 25% of student responses. For module 3, response rates were generally low with the highest response being 29% M/D students selecting a ‘combination of activities’ as most useful. These responses are illustrated in the Inline Supplementary Fig. 5(a). In the Inline Supplementary Fig. 5(b), the rankings of ‘least useful’ approaches to modules 2 and 3 are depicted. ‘Narrative histories’ was seen as the least useful approach for module 2, by N and M/D students. Others varied by student groups and modules. During the interviews/focus groups, there were mixed opinions on approaches to use. Students suggested ‘‘The case studies were good because they were encouraging you to think critically, rather than just trying to memorize’’; ‘‘The resources should be embedded into the questions’’; ‘‘online but integrate small group discussions’’; and ‘‘a combination of methods would be good. I do like online things in general and having things accessible to you so you can go and look up things online.’’ The surveys also had questions on four ‘content topics’ and their relevancy. These were ‘determinants of behavior change’, ‘nutrition and physical activity guidelines’, diet history’, and ‘physical assessment’. For module 2, 50% of N and 30% M/D students felt that ‘nutrition and physical activity guidelines’ were important to learn. Another 43% N and 33% M/D students felt that ‘determinants of behavior change’ was important. ‘Diet history’ was felt to be important to learn by another 30% of M/D students. Other items identified by students included: ‘‘More disease statistics broken down according to risk factors, gender, and hereditary factors. Also, maybe offering more information about potential strategies’’ For module 3, almost 60% of M/D students felt that ‘nutrition and physical activity guidelines’ were most important to learn. Students also agreed that module 3 was interesting (65% M/D students) and that the information was appropriate for their level of understanding (87% M/D students). Students would like to see ‘‘areas such as self-assessment or behavioral counselling’’ added. 5.4.4. Module delivery, processes and technology Survey responses on ‘course set-up and delivery’ indicated that 43% N students and 47% M/D students felt this was ‘good’ to ‘excellent’. Less than 30% of both groups felt the opposite. During one focus group, most of the students agreed with the statement that ‘‘It was well laid out. I remember that I was thinking that it is a nicely designed site.’’ For module 3, 67% felt it was ‘good’ to ‘satisfactorily’ set-up and delivered. More than 70% of both groups of students were ‘comfortable/very comfortable’ in using the webbased module 2 resources, and with navigating through the course without any difficulties (72% N students and 77% M/D students). When students were asked about their preference for the modules to remain online or delivered in another way, most preferred the online delivery because it offered more flexibility, easy access to resources and information, and can be worked on

81

individually or small groups, but online also has its challenges for students. One medical student liked the idea of ‘‘online because as a doctor we will learn things from online.’’ Another M/D said: I think online adds flexibilities over a term when you want to look at stuff. This is helpful for our schedules. I personally have difficulties in scheduling things as I do not have much free time. It will really help, just leaving it online. About 20% N and M students indicated that they had technical difficulties as well but very few called for technical support (7% N and 3% M/D students). Of those who called, 50% N and 25% M/D students had their problems resolved. For module 3, over 82% M/D students were ‘comfortable/very comfortable’ using this webbased resource or course and 58% navigated through it without any difficulties. Of those that encountered difficulties, only 4% called for technical support. Some resolved the problem by themselves using some other routes to access what they needed. 5.4.5. Challenges/difficulties Key areas identified early on as having potential challenges in the development and implementation of the module pilots included determining how the learning would fit with curricula, program and time constraints. Hence, many of the previous themes identified challenges with content, volume of information and resources, expectations, and so on. Other challenges were identified, including ‘time’ to do all that was asked in parallel with regular course requirements. For module 2, there were variations among the two student groups–57% N students spent 6 to 9 h per week on the module, and another 36% spent 2 to 5 h per week. About 7% spent more than 9 h per week on the module. The M/D students were the opposite–50% spent more than 9 h per week on the module, and another 33% spent 6 to 9 h per week. About 17% spent 2 to 5 h per week. Both groups clearly stated that this module was time consuming. For module 3, the M/D students spent different amounts of time than those for module 2. About 46% M/D students spent between 2 and 4 h per week on module 3, and another 38% spent less than 2 h per week. Very few (8%) spent more than 6 h per week. Over two thirds (67%) of students felt their time spent was ‘about right’, and another one-third (33%) felt it was too much. Many students said: ‘‘Yes, you can make the time. I felt like I was wasting time, sometimes.’’ Another confirmed this by saying: ‘‘I would argue that time is not the biggest challenge for us. We have lots of time but students are going to complain if you give them something extra to do.’’ A few other students suggested that both modules 2 and 3 would be better in year one of their programs when students actually do have more time in their curricula to incorporate these modules. 5.4.6. Recommendations regarding learning modules Two areas are discussed here—‘students recommending this module to others’ and ‘recommendations by students for module improvement’. 64% N students and 60% M/D students said they would not recommend module 2 to others, and 58% M/D students said they would recommend module 3 to other students and health professionals. Regarding module 2, one student would recommend this module to others, ‘‘but it should be made more of an interesting activity rather than an absolute requirement. Yes, I would recommend this course but it has to be altered. It needs to decrease the burden on the students.’’ Another student said ‘no’ ‘‘because it was very stressful in getting it all done with the rest of the course. . .. Content wise, I think so. It provided a nice framework of how you make changes, instead of just asking how you become healthier. You can actually try. This was a very systematic way of thinking about why it did not work and encourage yourself to try something new. I think that part was very useful.

82

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

To improve the modules, suggestions were many including: having appropriate introductions to the modules, providing overviews and expectations for module activities, better learning outcomes, finding and evaluating resources and tools that students could use themselves or recommend to patients, having more group work and less written reflections, providing down loadable worksheets or a workbook, incorporating resources into case studies, integrating the modules into curriculum and exams and looking at first year curriculum when students have more time. There were mixed views on integrating the modules into the curriculum as course material. Some said: ‘‘I guess that it would have been better if it was not only incorporated into your course activities but also your exam.’’ Others felt that the modules should be left as stand-alone, so people could access them as they needed to and still have them linked with another course like GI. ‘‘It might seem more important if it was stand-alone. People might take it as its own thing. It should be more focused. It helped us relate things to GI as we are learning in GI about clinically relevant lifestyle choices.’’ And finally, students felt the WellnessRx should be retained as the information was important and should be emphasized in curricula more than it has been, but maybe just not attached along with other courses. It should be linked with students’ exams as this is important information for students to know. 5.5. Faculty instructors’ perceptions and experiences with the WellnessRx learning module pilots Faculty instructors who were involved with the piloting of the modules were invited for interviews or focus groups. Three instructors were involved in the pilots—two with module 2 (SelfHealth) and one with module 3 (Nutrition and Physical Activity across the Lifecycle). One focus group of two instructors was held regarding module 2 and one interview on module 3. The transcripts were coded and thematically analyzed by three independent raters. Common themes and sub-themes were identified, as presented in the Inline Supplementary Table 3. The main themes covering the results were:     

Overall experiences and impressions of the pilots. Interpretation of student comments. Online delivery approach. Recommended modifications. Mentoring other faculty.

5.5.1. Overall experiences and impressions of the pilot Each instructor had unique experiences with the piloting of these newly developed modules. The instructors involved with the pilot of module 2, discussed the challenges they faced with incorporating a new web-based or online module with their regular classes. From my point of view, there are two elements. The one is the online delivery method component and the other is the curriculum component itself. We were the first ones to do it. It went live two days after my course had started, so there were a lot of growing pains with the online delivery component of it and I think that was a negative way for the students to get involved. . .. The other is the curriculum itself. And I think it did serve a need. . .. I think that they liked that component, but maybe this might not have been the right module to pilot in the medical curriculum. I think they were looking for more like what advice do I give a patient that is sitting across from me and it seemed hard for them to relate to what they were doing in this module. . .. The other instructor had a class of students who were well versed on the topic and were involved in the module 2 pilot as a ‘research initiative’. The instructor asked this group of students to

be participants in the research. The instructor reiterated a lot of concerns the students had with the module, but that ‘‘they valued the idea that it was included in the curriculum which previously it had not been.’’ This second group did not encounter the technical challenges that the other group had, as most had been dealt with before this second phase pilot started. For module 3, the instructor felt it was a better fit with the curriculum and the case-based style of learning was more appropriate for the students. Students in medicine are familiar with working on cases when they are doing small group activities. The instructor reflected that the online format of the module is not one of its strengths, but that ‘‘the initiative, just to get it into the curriculum is the major strength.’’ The biggest time commitment from the faculty was the grading of the module assignments, but otherwise there were no challenges for the instructors. Instructors said doing this module again with students would not be a problem. Mentoring other faculty was also viewed as positive. 5.5.2. Interpretation of student comments Regarding module 2, the instructors said that their students had similar expectations regarding activities and self-assessments, including narrative reflection activities which students felt were a waste of time. ‘‘They felt they were reflecting on top of reflection. After reading it, I have to agree that we had a lot of additional questions of why did you choose to do this. . ..It was a barrier for students and a waste of time.’’ Time was a big issue raised by students to their instructors. One instructor felt that it was difficult to compromise on content for time ‘‘as different people will like different things.’’ In response, the other instructor said: ‘‘I think they want the knowledge, they want the skills but they don’t want to put in the time so I think we need to hot sync with knowledge. There has to be a process.’’ For module 3, the instructor was told that this module would be more relevant in small group settings, as the content and case studies were similar to some of the group work the students were doing with dietitians. Students did not read all the resources listed because of time. Some said they would prefer to have only one major resource or a hand-book. The 7% grade attached to completing the assigned parts of the module was also not viewed as appropriate by students, and suggested the module be done in a 2 h block, or the other option was to have questions from the module on the final exam. One of the issues raised by the students was that they wished they could have had feedback on their case studies. The challenge for the instructor was marking 200 student assignments. The instructor was not surprised by any of the student evaluation comments and felt they made sense. However, the instructor was pleasantly surprised by the results of the prepost assessments. ‘‘Some students thought that this is old information and they know it all but in fact the pre-test was not very high and they did make significant gains from the pre-test to the post-test. To me that validated all the effort that we have been putting into this project’’ 5.5.3. Online delivery approach There was mixed thought on the delivery of the module. For one instructor, the online seemed to meet the need but that it could be integrated with other approaches. ‘‘. . .the online way is an easy way to disseminate, but I think if you did them in small group sessions you can get a lot out of it. Have a paper case that you can talk about.’’ The other instructor felt this module with a bit more development of specialized examples would be better integrated in with an interdisciplinary course offered to all health sciences students. This latter idea was viewed as an opportunity to consider along with other suggestions for a ‘‘summer nutrition boot camp’’ or ‘‘workshop or a summer course that people can take for credit.’’ The one advantage of online student activities is that the faculty can also go online to read and assess the assignments electronically.

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

For module 3, there were no online technical difficulties noted as these were addressed after the first pilot. Students told the instructor they appreciated that resources were linked within the module. The instructor felt that this module could be left online as a stand-alone unit. 5.5.4. Recommendations Based on previous comments, the instructors felt that the modules should be more case-based if possible and that they could be developed as stand-alone, or integrated into curricula or programs as groups wished. The flexibility was important to allow for a combination of delivery formats which might be of interest to a broader student or other type of audience. Essentially, ‘‘you have to show that there are equal ways to do it, especially in Faculties that are trying to create more opportunities for students.’’ 5.5.5. Mentoring other faculty members Other instructors are needed. Recruitment needs to happen with faculty who can find room in their curricula to implement one or all five of the WellnessRx learning modules. The views of the instructors were that this was more about marketing than mentoring. The implementation of the modules was seen to be relatively easy.

6. Discussion Generally, the developmental evaluation approach used for the education component of WellnessRx, or the two learning module pilots on ‘Self-Health’ and ‘Nutrition and Physical Activity across the Life-Cycle’, provided the necessary guidance through the evaluation steps, but also the flexibility needed for evaluating the complex process and outcomes related to developing and piloting the learning modules for targeted audiences (Gamble, 2008; Patton, 2011). Similar aspects for each of the modules were explored through this evaluation, including their value or benefit to students’ knowledge and skills, and their content, resources, activities and online delivery. The intent of the evaluation was to capture two areas of measures: (1) the change in students’ knowledge with the completion of the modules, as determined through the pre-post knowledge assessments, and (b) the students’ and instructors’ perceptions, experiences and recommendations regarding the learning module pilots. The added approach of piloting the modules complemented the developmental evaluation framework particularly as the purpose of pilots is to use a smallscale study with small sample sizes to determine feasibility of the modules (Arain et al., 2010; Thabane et al., 2010; van Teijlingen & Hundley, 2002). Together with the developmental evaluation considerations, the pilot of the modules provided the means for gathering essential findings including recommendations which informed the module developers about the feasibility and other qualities or challenges needing to be considered in the further development, implementation and evaluation of the modules. Pilot data still needs to be considered cautiously (Lancaster et al., 2004) but the combination of developmental evaluation and pilots proved to be worthwhile for this type of study. Others interested in this type of study will appreciate the intentions of pilots as part of developmental evaluation (Kilanowski, 2011). The one challenge is that the response rates for the two pilots were low. Although generally perceived to be a limitation for any statistical analysis, low response rates are often anticipated with pilots since their focus is not on large numbers of participants and statistically significant findings but rather on the module itself and its strengths and weaknesses to be considered in the larger scale development and implementation (Arain et al., 2010; Lancaster et al., 2004).

83

The results indicated that these pilots were a good means of testing and evaluating the learning modules to see if they made a difference to students in terms of enhancing their knowledge regarding nutrition and physical activity. The results for the prepost knowledge assessments of the M/D students for modules 1 and 2 as depicted in Fig. 2, clearly indicated some improvements in knowledge, and found by Bistritz et al. (2015) to be statistically significant (p < 0.01). We can assume from the results that the modules were effective in making some difference in knowledge outcomes, although this would need further correlational studies. The knowledge change experienced, as an indicator, could be viewed as a general benchmark for when the module is revised and implemented fully as part of a program or curriculum, with student assessments and module evaluations incorporated (Adams et al., 2006). Since the module is made up of many items including information, resources and tools and guides, as well as assessments, specific aspects of the module contributing to changes in knowledge would need to be further explored with the students. What were the main aspects of the learning module which contributed to the change in knowledge and attitudes of students? However, the findings also indicated the differences between user groups or targeted audiences completing the module. The medical and dental students preferred to have the module more integrated into their curriculum with marks or grades allocated, or in other words, made mandatory as opposed to an add-on to their other mandatory studies. They also did not mind being assessed on their knowledge, skills and attitudes—something expected of them. The nursing students, however, used the modules in a different way within their course, and therefore, the pre-post knowledge assessment and many activities did not fit for their purposes. These students may have felt uncomfortable doing the interviews/focus groups as a result. Therefore, curricula must be adaptable to accommodate different learners and their instructors. This flexibility is something to keep in mind with the development of the learning modules. More specific to the evaluation of the modules, results clearly indicated that not all students were satisfied with the amount of content, resources, activities, expectations or the web-based online delivery format for either module. For example, some students did not appreciate the content or the amount of it, and indicated that even though they learned a lot about themselves, resources available and tools to assess themselves and others, they found it overwhelming for the time they had, and also expected more of it to be relevant for patient care. Some disliked the online delivery even though it gave them flexibility to complete the tasks on their own time when they had time. The challenges for students were clearly stated, and time to complete all the activities was one of them, especially since the module was an addition to their regular program of studies. From this evaluation process we learned that there is no one way to develop or deliver education components on nutrition and physical education focused on self-health or across the lifespan, to satisfy or meet the learning needs and expectations of all learners. In addition to students’ perspectives, this pilot also captured those of the instructor who would no doubt receive direct feedback from students in the form of complaints or praises. Their recommendations were also clear in terms of seeing the relevance of the learning modules, but integrating them into appropriate curriculum areas within specific years of an education program. Students provided suggestions for making the modules more accommodating and relevant to student expectations—e.g. more case studies with direct application to personal or patient/person orientation. These and other recommendations could be implemented to alter or improve the two piloted modules and frame the development of the other modules planned for the WellnessRx education component.

84

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85

Generally, the developmental evaluation of the two pilots facilitated discussions that were more than just improving the modules as formative evaluation would do. The different data sources acquired through the application of developmental evaluation provided key findings for the developers, adopters and users of the modules. These findings included profiles of the adopters (i.e. Faculties) and users (i.e. students) of the modules along with assessments of different groups of students regarding their knowledge pre- and post-module, student quantitative evaluations and the triangulation of student and faculty interview/focus group responses into common themes. Although there was limitation in terms of the evaluation being applied to pilots and not having large numbers of participants, the advantage was that the purpose for the pilots was to not worry about sample size but to rather focus on testing out the modules before further development and full implementation. This combination pilot and developmental evaluation provided the necessary flexibility to explore the modules from a 360 degree perspective. In the end the developmental evaluation of the pilots not only resulted in better design and development of the learning modules and overall logic model for the WellnessRx initiative, but it saved time and development effort on the part of the working groups and stakeholders involved, as well as money and other resources needed for online module development. The next phases of development and implementation of the revised modules and the proposed ones will be crucial as their development and implementation will be based on some of the findings from this developmental evaluation. At some point in this journey, a business case for the WellnessRx learning modules will need to be developed to promote them to different audiences, including recommendations for either stand-alone online learning modules, or integrated into existing or revised curricula and being innovatively implemented as part of class assignments or discussions or some combination. 7. Conclusion: Lessons learned Based on the developmental evaluation related to the module pilots, several lessons were learned. Because of the complexity of the WellnessRx initiative including the evolving education component with its evolving learning modules, it is natural to think of many more challenges than enablers for setting up an evaluation framework rather than applying only formative and summative evaluation. There were many aspects of the modules to evaluate as experienced by different adopters (i.e. champion faculty) and targeted users (i.e. students, health professionals, others). It was because of the characteristics of the innovative initiative including its learning modules (i.e. complexity, innovation, the unknown, evolutionary and exploratory nature) that developmental evaluation was a fitting framework (Gamble, 2008; Patton, 2009, 2011). Developmental evaluation provided the flexibility needed to explore beyond only formative and possibly summative evaluation, but could include them, just as pilots were included. It was possible to incorporate the steps of developmental evaluation as part of the initiative and modules (i.e. logic model), exploring different perspectives through different methods and gaining valuable information about how adopters and users of the online learning modules felt and responded to the various aspects that made up the modules. This means that the developers of the education modules will have evaluation results from the pilots to inform them of what further planning and development or changes are needed with the two pilot modules and the others proposed. The latter is part of the developmental evaluation process and therefore a necessary component needing to be followed up and studied. Another lesson learned reflects on the value of developmental evaluation built into a new initiative’s logic model to gauge what

works, what needs to change, and what outcomes were achieved or not. This latter is what the evaluation must measure within the preliminary or formative logic model in order to inform what changes are essential to improve the WellnessRx initiative, as well as the education component and modules for the next diverse groups of learners, whether students or practicing health professionals. Changes directly applicable to the logic module may include the pilots of each of the five learning modules as shortterm outcomes, delaying many of the other short-term outcomes currently identified in the formative logic model. The pilots as an approach has proven to be invaluable in determining feasibility of the modules as part of the short-term outcomes for the further development or changes needed to the modules and any further work with stakeholder groups. Based on this demonstration, further application of the developmental evaluation framework with the education component learning modules will occur, as it will with the other components of the WellnessRx initiative. Such studies are needed to begin building the case for supporting more integration of these innovative and needed modules into the curriculum of health sciences students. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.evalprogplan. 2014.11.009. References Adams, K. M., Lindell, K. C., Kohlmeier, M., & Zeisel, S. H. (2006). Status of nutrition education in medical schools. American Journal of Clinical Nutrition, 83(Apr (4)), 941S–944S. Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology, 10. DOI 10.1186?1471-2288-10-67. (Article 67). Bistritz, L., Kovacs Burns, K., Gramlich, L., McCargar, L., Olson, K., & Avdagovska, M. (2015). WellnessRxTM Education Initiative: development and pilot study of nutrition and physical activity education for health sciences students. Pedagogy in Health Promotion. Crogan, N. L., Shultz, J. A., & Massey, L. K. (2001). Nutrition knowledge of nurses in long-term care facilities. Journal of Continuing Education in Nursing, 32(Jul–Aug (4)), 171–176. Gamble, J. (2008). A developmental evaluation primer. Canada: The J W McConnell Family Foundation. Garry, J. P., Diamond, J. J., & Whitley, T. W. (2002). Physical activity curricula in medical schools. Academic Medicine, 77(Aug (8)), 818–820. Health Metrics and Evaluation (2010). Global Burden of Disease Study 2010: GBD Profile Canada. Seattle, WA, USA Available at hwww.healthmetricsandevaluation.orgiRTD. Kilanowski, J. (2011). Pilot studies: Helmsman on the ship of research design. American Nurse Today, 6 http://www.americannursetoday.com/pilot-studies-helmsman-onthe-ship-of-research-design. Kovacs Burns, K., Gramlich, L., McCargar, L., Bistritz, L., Olson, K., & Avdagovska, M. (2014). Developmental Evaluation Framework for an Evolving WellnessRx Innovation. Canadian Journal of Evaluation (In Review). Kovacs Burns, K., & Gramlich, L. (2012). Preparing health professional students to counsel and advise on nutrition, physical activity and wellness: Determining education gaps and needs. Creative Education, 1, 564–568. Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004). Design and analysis of pilot studies: Recommendations for good practice. Journal of Evaluation in Clinical Practice, 10, 307–312. Mackenzie, M., Kovacs Burns, K., Gramlich, L., et al. (2009). Initiative for nutrition, physical activity and wellness—Final report. Edmonton, AB: University of Alberta and Alberta Health Services. Morin, K. H. (2013). Value of a pilot study. Journal of Nursing Education, 52, 547–548 DOI 10.3928/01484834-20130422-10. Morse, J., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press. Ostland, U., Kidd, L., Wengstrom, Y., & Rowa-Dewar, N. (2011). Combining qualitative and quantitative research within mixed method research designs: A methodological review. International Journal of Nursing Studies, 48(3), 369–383. Patton, M. (2011). Developmental evaluation: applying complexity concepts to enhance innovation and use. New York: Gilford Press. Patton, M. (2009). Developmental evaluation. Presentation. Canadian Evaluation Studies, 1(June). Thabane, L., Ma, J., Chu, R., Cheng, J., Esmaila, A., Rios, L. P., et al. (2010). A tutorial on pilot studies: The what, why and how. BMC Medical Research Methodology, 10. DOI 10.1186/1471-2288-10-1. (Article 1).

K. Kovacs Burns et al. / Evaluation and Program Planning 49 (2015) 76–85 van Teijlingen, E., & Hundley, V. (2002). The importance of pilot studies. Nursing Standard, 16(40), 33–36. WellnessRx (2013). WellnessRx strategic plan.. Available at: hwww.wellnessrx.cai. World Health Alliance World Health Organization (2004). Global strategy on diet, physical activity and health. Geneva, Switzerland: World Health Organization. World Health Organization (2009). Global Health Risks: Mortality and burden of disease attributable to selected major risks. Geneva: WHO: World Health Organization. Katharina Kovacs Burns, MSc, MHSA, PhD With a background in Kinesiology (health promotion and risk prevention), Health services administration and Policy analysis (including public engagement and knowledge management), as applied in her positions with government, community organizations and the University of Alberta (Associate Director, Health Sciences Council, and Director, Interdisciplinary Health Research Academy), Dr. Kovacs Burns provides expertise in interdisciplinary and interprofessional health research, education and practices. Her research in health and social programs and policies includes evaluation using mixed methods. Leah Gramlich, MD: Dr. Leah Gramlich is a physician nutritionist specialist and Gastroenterologist. She is a Professor in the Faculty of Medicine and department of Medicine at the University of Alberta with a cross appointment in Agriculture Life and Environmental Science. She is also Provincial Medical Advisor for Nutrition services in Alberta Health Services. Lana Bistritz, MD: Dr. Bistritz is a clinical gastroenterologist at the Royal Alexandra Hospital and assistant professor at the University of Alberta. She had additional training in medical education in the MHPE program at the University of Illinois,

85

Chicago. Her education interests include simulation training to develop residents’ procedural skills, and undergraduate curriculum development and evaluation.

Linda McCargar, RD, PhD: Dr. Linda McCargar is a Professor of Human Nutrition in the Department of Agricultural Food and Nutritional Sciences at the University of Alberta. She is also a Registered Dietitian. Her research program investigates the effects of nutrition, physical activity, disease states and other factors on energy metabolism and body composition.

Karin Olson, MSc, PhD: Dr. Karin Olson is a Professor in the Faculty of Nursing at the University of Alberta, where she teaches research methods and oncology/palliative care nursing. Dr. Olson maintains an active research program in symptom management using both qualitative and quantitative research methods and works part-time as a nurse consultant with the Edmonton Zone Palliative Care Program.

Melita Avdagovska, MSc Melita Avdagovska is the Project Manager for the WellnessRx initiative - interdisciplinary health initiative designed to address gaps in knowledge and skills regarding nutrition and physical activity. She coordinates the curriculum development and the community engagement components. Previously, Melita has held leadership roles in the not-for-profit sector in which she has addressed a wealth of issues, including respite care, education, and employment for people with disabilities. Melita has a Masters degree in political science-international affairs from the University of Central Oklahoma. Born in Macedonia, she has also lived in the United States and Morocco, and she speaks several languages. She has an interest in research in the areas of human rights, justice, conflict resolution, and healthcare.