509272
HPPXXX10.1177/1524839913509272He alth Promotion Practice / MonthMillery et al. / Innovative Instructional Technology Design Process 2013
Using Innovative Instructional Technology to Meet Training Needs in Public Health: A Design Process Mari Millery, PhD1 Michelle Hall, MA2 Joanna Eisman, MPH1 Marita Murrman, EdD1
Technology and distance learning can potentially enhance the efficient and effective delivery of continuing education to the public health workforce. Public Health Training Centers collaborate with instructional technology designers to develop innovative, competency-based online learning experiences that meet pressing training needs and promote best practices. We describe one Public Health Training Center’s online learning module design process, which consists of five steps: (1) identify training needs and priority competencies; (2) define learning objectives and identify educational challenges; (3) pose hypotheses and explore innovative, technology-based solutions; (4) develop and deploy the educational experience; and (5) evaluate feedback and outcomes to inform continued cycles of revision and improvement. Examples illustrate the model’s application. These steps are discussed within the context of design practices in the fields of education, engineering, and public health. They incorporate key strategies from across these fields, including principles of programmatic design familiar to public health professionals, such as backward design. The instructional technology design process we describe provides a structure for the creativity, collaboration, and systematic strategies needed to develop online learning products that address critical training needs for the public health workforce. Keywords: workforce development; continuing education; distance education; technology
Health Promotion Practice March 2014 Vol. 15, Suppl 1 39S–47S DOI: 10.1177/1524839913509272 © 2014 Society for Public Health Education
Introduction >> Use of technology, and of distance-based learning in particular, offers innovative solutions to problems associated with the delivery of training to working professionals. The public health workforce has a variety of pressing training needs (Institute of Medicine, 2003), which the Public Health Training Centers (PHTCs) are tasked with addressing. PHTC needs assessments clearly indicate that convenient access to training is necessary to reach public health workers who are continually asked to do more with less time and fewer resources. Web-based educational experiences have many advantages. They negate the need for off-site travel to training centers, can be pursued by learners when time permits, reach large numbers of learners, and use multimedia and innovative instructional methods (Cook,
1
Mailman School of Public Health of Columbia University, New York, NY, USA 2 Columbia Center for New Media Teaching and Learning, New York, NY, USA Authors’ Note: The work described in the article was supported by Health Resources and Services Administration Grant No. UB6HP20788. All article contributors received support from the grant. Address correspondence to Mari Millery, Mailman School of Public Health of Columbia University, 722 W 168th Street, Room 543, New York, NY 10032, USA; e-mail: mm994@columbia. edu. Supplement Note: This article is published in the supplement “Public Health Training Centers,” which was supported by the U.S. Department of Health and Human Services (HHS) Health Resources and Services Administration (HRSA) through a Cooperative Agreement (UBGHP20199) to Rollins School of Public Health, Emory University.
39S
2007). They also have disadvantages, including social isolation of learners, a need for high-quality design expertise and resources, and potential technical problems. Overall, however, evidence reviews show that online learning is effective compared with face-to-face learning (U.S. Department of Education, 2010). As PHTCs have begun producing online learning modules, they have developed design processes and learned many lessons. There is a need to document these design models for their own utility and to supplement the sparse literature on the topic. Below, we examine the literature on the design process, present the online module design process of the New York City-Long Island-Lower Tri-County (NYC-LI-LTC) PHTC, and illustrate the five-step process with examples from the Advanced Program Evaluation module (http://www.lowernysphtc.org/content/advanced-program-evaluation) and other projects.
Background >>
The federal funding guidance to PHTCs encourages use of distance learning modalities to deliver competency-based trainings consistent with the needs of the public health workforce (Health Resources and Services Administration, 2011). Over the past decade, PHTCs have developed a large number of online learning programs and continue to design new distance-based trainings in order to meet emerging needs and reach more trainees. The design process established by the NYC-LILTC PHTC for online learning modules makes innovative use of technology to meet identified training needs with content and pedagogy that effectively promote best practices reflecting the latest public health advances. The NYC-LI-LTC PHTC seeks to enhance the public health workforce’s ability to effectively deliver essential public health services in New York City and Suffolk, Nassau, Westchester, Rockland, and Putnam Counties. To design and build its online learning modules, the NYC-LI-LTC PHTC collaborates with the Columbia Center for New Media Teaching and Learning (CCNMTL), a service organization with a mission to support Columbia University in the purposeful use of new media technology. CCNMTL has completed more than 300 custom-designed learning projects, including web-based training environments, visualization and modeling tools, media archives, and simulations. Close collaboration between the PHTC and CCNMTL is essential to the success of the design process. Since public health workers in the United States have an average of 12 years of experience and an average age of 47 (Association of State and Territorial
40S HEALTH PROMOTION PRACTICE / March 2014
Health Officials, 2008), development of all NYC-LILTC PHTC trainings is guided by the key principles of adult learning theory (Merriam, Caffarella, & Baumgartner, 2007). Specifically, we use needs assessment results to identify the breadth and depth of learning public health workers need to perform their jobs. We engage workforce representatives in both the design and learning processes we choose for each training program. When presenting theoretical concepts, we do so by having workers apply the concepts to relevant public health problems. Finally, we use participatory techniques, like case studies and problem-solving activities, whenever possible. Herbert Simon’s (1969) influential definition of design reflects our team’s realization that the design process is essentially the same across fields: Everyone designs who devises courses of action aimed at changing existing situations into preferred ones. The intellectual activity that produces material artifacts is no different fundamentally from the one that prescribes remedies for a sick patient or the one that devises a new sales plan for a company or a social welfare policy for a state. (p 130)
Similarly, in his book, Designerly Ways of Knowing (2006), Nigel Cross suggests that although we do not know enough about design processes across professions, there is likely to be a “deep structure” to the process of translating needs to products. Across fields, the design process is a craft that practitioners hone through experience. It can be difficult to describe because it relies on implicit knowledge of expert practitioners. Edelson (2002) further points out, “The process of design is complex. Its open-endedness and reliance on creativity have made it a challenge for researchers to characterize and explain.” In an effort to take on the challenge, Edelson discusses three kinds of general design decisions: design process (steps and people involved), problem analysis (needs assessment), and design solution (resulting product). The specific steps outlined below encompass the progression from problem analysis to design solution, and constitute our approach to the design process. Design is essential for a variety of human endeavors. Design approaches across different fields have commonalities and differences. In the field of education, the tools of design include competencies and learning objectives. Core competency sets have been defined for many professions, including public health (The Council on Linkages Between Academia and Public Health Practice, 2010). Crafting of learning objectives is guided
Step
1. Assess needs and identify priority competencies.
2. Define learning objectives and identify educational challenges.
===>
===>
===>
Training Needs
Learning Objectives
Hypotheses
Competencies
Educational Challenges
Innovative Solutions
Key Concepts
3. Pose hypotheses and explore innovative technology-based solutions.
4. Develop and deploy educational experience.
5. Collect feedback and evaluate to inform cycles of revision and improvement.
===>
Development
Feedback
Deployment
Evaluation
Figure 1 Steps and Key Concepts of PHTC Online Learning Module Design Process NOTE: PHTC = Public Health Training Center.
by the tradition of Bloom’s taxonomy (Bloom & Krathwohl, 1956). The design approach to information technology (IT) applications follows engineering practices, with an emphasis on requirements analysis and requirements engineering (Robertson & Robertson, 2006). Just as learning objectives define what learners are expected to do as a result of the learning experience, IT requirements specify what the finished IT product is expected to do for the users. In health promotion and public health, professionals are trained to use systematic program design models, such as PRECEDEPROCEED (Green & Kreuter, 2005) and Intervention Mapping (Bartholomew, Parcel, Kok, Gottlieb, & Fernandez, 2011). The design approach uses theories and research evidence to inform the process. The design processes used in education, IT, and public health share several features. First is the fundamental principle of backward design—planning with the “end in mind” (Wiggins & McTighe, 2005). Backward design posits that the starting point of design is the end goal to be achieved, although designers in different fields may use different tools to define the goal and the path to it. Second is the use of audience feedback to inform the design, whether the audience is defined as learners (education), end users (IT), or stakeholders (public health). And third is the use of innovation, by which we mean the new, often creative and unique, solutions that emerge during the design process. Innovation involves the notion of a creative leap (Cross, 2006) and the cognitive process of creativity in general, as outlined in Wallas’s (1926) classic model of preparation, incubation,
illumination, and verification. CCNMTL follows the design research approach (Edelson, 2002), which features elements of academic inquiry and empirical research and emphasizes innovation (Bereiter, 2002). The process of designing interactive online tools and learning experiences is not well documented. Elwyn et al. (2011) describe a process for developing web-based decision support for patients. Their process, consisting of content specification and creative design, envisions design as a team effort requiring various kinds of expertise, captured, in part, in management and advisory groups. Cook and Dupras (2004) present a 10-step model for designing web-based learning modules, with an emphasis on active learning. Below, we describe the NYC-LI-LTC PHTC online learning module design process, which incorporates the design principles discussed above and progresses from perceived training needs in the field to convenient and innovative online trainings that support best practices in public health.
Method and Application >> The PHTC design process for online learning modules consists of five general steps: (1) assess needs and identify priority competencies, (2) define learning objectives and identify educational challenges, (3) pose hypotheses and explore innovative technology-based solutions, (4) develop and deploy the educational experience, and (5) collect feedback and evaluate to inform continued cycles of revision and improvement. Figure 1
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
41S
lists the sequence of the steps and associated key concepts. However, although the steps are depicted in sequence, it is important to note that they do not typically unfold in a strictly linear process; work often fluctuates among steps. Collaboration among participating organizations and team members, in their various roles, is integral to the design process. According to Edelson (2002), design methodology specifies required expertise and the roles of individuals providing the expertise. As Schon (1990) points out, design is a social process. In addition to the PHTC and content expert, there is an important role for the primary audience (learners), whose input is critical at various stages. CCNMTL staff fill a range of design and technical roles, such as instructional designer, web designer, interface designer, interactive designer, user experience designer, and developer/programmer. However, an effective online learning design team may not always require all of these roles. Because of team members’ diverse backgrounds, we allocate time for everyone to learn about potentially unfamiliar methods, and we clearly define team members’ roles. Our process was developed for the design of interactive, online, self-study learning modules, as distinguished from other types of online learning experiences, such as instructor-facilitated sessions, blended experiences combining online and face-to-face learning, or noninteractive recordings of live trainings. The typical module takes less than 1 hour to finish and can be completed at the learner’s convenience in one or more sessions. On completion, a certificate is offered. Although the design process was developed for a specific public health application, it can be used to guide the design of any technology-based educational experience. Step 1. Assess Needs and Identify Priority Competencies Step 1 is basically problem analysis: characterizing the goals, needs, or opportunities the design is intended to address (Edelson, 2002). Problem analysis may require systematic use of applied social science research methods to conduct a thorough needs assessment, an activity familiar to health promotion practitioners (Gilmore, 2012). Our training needs assessment uses key informant interviews and surveys to identify training needs among public health workers in the defined geographic area. Training needs are based on standard core competencies for public health professionals (The Council on Linkages Between Academia and Public Health Practice, 2010). Once a competency, or set of competencies, is identified as a priority training need, the team considers whether an online module is a
42S HEALTH PROMOTION PRACTICE / March 2014
potential solution. A purposeful rationale for the use of technology is required, as it may not be the best modality for all instruction. The core competency is then further analyzed to identify subcompetencies that closely correspond with the skills workers wish to acquire and the tasks they want to accomplish. For example, one of our highpriority competencies was Identify mechanisms to monitor and evaluate programs for their effectiveness and quality. After consulting with stakeholders and a program evaluation expert, we identified a related, high-priority subcompetency: Design rigorous and feasible evaluation plans. The next step was to specify the primary audience, in this case, midlevel local health department staff with program evaluation responsibilities. Figure 2 shows how training needs become translated into learning objectives, using the Advanced Program Evaluation module as an example. This translation process spans Step 1 and into Step 2. Key local health department informants highlighted the need for better program evaluation, addressing the corresponding core competency. Input from multiple team members and stakeholders was used to refine the subcompetency and to develop the learning objectives. The PHTC facilitates the connections among stakeholders representing the primary audience, instructional design/technical staff, and the content expert. Step 2. Define Learning Objectives and Identify Educational Challenges Learning objectives are central to instructional design (see Figure 2). They specify what learners are expected to accomplish as a result of the learning experience, thereby enabling “backward design” of the experience and assessment of learning outcomes. In Step 2, the content expert and instructional designer play key roles. Selection of an appropriate content expert to write the module content is critical to assure that the PHTC disseminates high-quality content on best practices. Learning objectives provide the context for identifying educational challenges learners might face. For example, what are some challenging concepts for learners to grasp? Input from stakeholders and the primary audience is important here. Sometimes a broader educational challenge motivates the design. For example, when CCNMTL worked with an environmental science professor to develop an inquiry-based multimedia simulation called Brownfield Action (Bower, Kelsey, & Moretti, 2011), the educational challenge was lack of
NEEDS ASSESSMENT
TRANSLATION OF NEED
• PHTC
NEED IN THE FIELD
PHTC
• Public health workers and their managers
Sub-Competency: Design rigorous and feasible evaluation plans
“To do better program evaluation” COL Competency: Identify mechanisms to monitor and evaluate programs for their effectiveness and quality
• PHTC • Content Expert • Instructional Technologist • Input from primary audience (further needs assessment)
EXPERTISE • Content expert Best practices in program evaluation
LEARNING OBJECTIVES − Appreciate that a comprehensive evaluation plan addresses a program’s logic, stakeholders, evaluation questions, and evaluation design. − Assess advantages and limitations of evaluation designs, including randomized, quasiexperimental and prepost designs. − Appraise and compare options for data collection methods, measures, and sampling strategies. − Identify and address associated threats to validity. − Identify strategies for addressing budget, time, data, and political constraints in evaluation practice.
Figure 2 Translating Training Needs Into Learning Objectives: Advanced Program Evaluation Module Example NOTE: PHTC = Public Health Training Center; COL = Council on Linkages.
excitement and engagement in science among nonscience undergraduates. A specific educational challenge of the Advanced Program Evaluation project was how to introduce logic models. Although knowledge of logic models was a prerequisite, feedback from primary audience representatives indicated the need for a brief review. Furthermore, we wanted to structure the learning experience as a case study and needed a way of presenting the logic model in this context. In Step 3, the solution was found through discussions with primary audience representatives and other team members: a brief, narrated video presentation of logic model components, using the case study example. Work often crosses back and forth between Steps 2 and 3. In design language, the identification of functional requirements frequently blurs with the search for solutions (Cross, 2006). Step 3. Pose Hypotheses and Explore Innovative Technology-Based Solutions Step 3 moves from problem analysis to solution construction (Edelson, 2002). It has been characterized as
a systematic search through a solution space (Simon, 1969) and a process of synthesis with considerable complexity (Schon, 1990). Following the model of design research (Edelson, 2002), Step 3 prompts the team to pose hypotheses about potential solutions. In the Advanced Program Evaluation project, we found that learners have difficulty grasping the elements of evaluation study designs, such as preintervention/ postintervention, comparison group, and randomized designs. We hypothesized that a traditional notation system often used to depict study designs with Xs for interventions and Os for observation points can serve as a learning tool, if learners apply the notation system to different examples. This hypothesis led to the design of an interactive notation exercise (see Figure 3). These types of interactive exercises are an important part of the online modules and are developed iteratively, as primary audience representatives and other team members provide ongoing feedback on a series of prototypes. Most NYC-LI-LTC PHTC modules are structured around case stories with protagonists. This is based on the hypothesized effectiveness of cases in organizing
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
43S
solutions. Sometimes, an off-the-shelf solution is available and selected instead of a custom design. Consistent with Cross’s (2006) account of designerly thinking, we find that moments of design “illumination” are in fact the result of the following: systematic exploration of existing solutions; deliberate use of cognitive processes, such as analogies; and sustained focus on the most critical functional requirements. These processes can yield unique, technology-based solutions— such as the logic model video and notation exercise described above—to educational challenges. Step 4. Develop and Deploy the Educational Experience Figure 3 Example of Interactive Component in Learning Module: Evaluation Design Notation Exercise NOTE: X = intervention/program/treatment; O = observation/ outcome measurement point (e.g., pretest or posttest); R = randomly assigned; NR = not randomly assigned. The two rows represent the two study groups to be compared. In the interactive exercise, the learner composes the design elements for a given scenario by selecting them from the blue boxes and dragging them into the design rows.
learning content. The team spends time selecting and defining an appropriate case, which the content expert writes up. The content expert and instructional designer develop a detailed outline of module content and search for innovative solutions to the challenges posed by the selected learning objectives. The outline is circulated to all team members, including primary audience representatives, for feedback. As module components are defined, we explore a variety of media to determine what would best support a particular type of content, following design principles from cognitive theory of multimedia learning (Mayer, 2002). Online learning modules can easily incorporate various media, including text, images, video, audio, interactive exercises, and quizzes. Each medium has unique affordances and constraints to be considered (Norman, 1988). Sometimes excitement about a technology may lure designers to add “bells and whistles” that may not add real value; it is important to avoid overengineering. The exploration of technology-based solutions draws on existing solutions, including those previously designed by CCNMTL. When exploring the mechanics of the notation exercise described above, we investigated digital tic-tac-toe and Sudoku game user interfaces. Investigating previous solutions to similar problems is an essential part of design thinking (Cross, 2006) and makes the process more efficient. Experienced designers are skilled at borrowing and adapting proven 44S HEALTH PROMOTION PRACTICE / March 2014
Designers are trained to translate functional user requirements into formal patterns of design structure (Cross, 2006). Development is the process of creating the design structure. In practice, it fluctuates back and forth between Steps 3 and 4, as ideas are sent back to the “drawing board” to be refined or replaced. Creative design continues during Step 4, with special focus on creating a visually attractive “look” and intuitive online navigation. Representation of the design concept is important at this step. Information architecture diagrams, website wire frames, and storyboards are commonly used representational tools. Figure 4 depicts a wireframe and the final design of the website housing our online learning modules. In this case, the programming challenge was to allow learners to leave multiple modules unfinished and to return later in order to complete them. The eventual solution was a detailed table of contents for each module, with icons indicating completion of module components. Step 4 may include a series of prototypes iteratively tested and developed. Focus groups, think-aloud techniques, and usability laboratories may be used. At the very least, the team wants to identify usability problems and content errors before wider dissemination. Deployment and dissemination are critically important steps, although not fully addressed here. Interested readers can view some of the products of our design process at http://www.lowernysphtc.org. In 2012, we launched two online modules: Advanced Program Evaluation and Introduction to Qualitative Analysis with ATLAS.ti. Additional modules were under development while this article was in press, with a 2013 launch date. Step 5. Collect Feedback and Evaluate to Inform Continued Cycles of Revision and Improvement After deployment, the online module remains a work in progress. Previously designed modules should be periodically reviewed and updated. We collect
Figure 4 Example of Representational Tool: Website Wireframe and Final Design Product
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
45S
systematic user feedback and evaluation data to assess learning and behavioral outcomes (Kirkpatrick & Kirkpatrick, 2006). Brief surveys are built into the modules as pre- and posttests to measure knowledge and self-efficacy. We also collect feedback on the learning experience and ask about intent to apply the learning content in practice. Additional follow-up evaluation surveys are e-mailed 6 weeks after module completion.
Discussion >> The NYC-LI-LTC PHTC has established a successful five-step design process that begins with competencybased needs assessment, moves through systematic stages of instructional technology design, and ends with evaluation and dissemination of a high-quality online learning experience. It is consistent with the 10-step model Cook and Dupras (2004) developed in the field of medical education. The process incorporates key design principles, including backward design, audience feedback, and innovation. It brings together a synergy of design strategies from the fields of education, public health, and IT engineering. Design is a team effort requiring effective collaboration among many individuals and organizations. As entities linked with academic and practice settings, PHTCs are well positioned to form effective design teams. Academic public health faculty can be recruited to serve as content experts. University-based instructional technology groups are ideally suited to partner with PHTCs to design technology-based learning experiences. And close linkages with public health practice settings provide a mechanism for meaningful input from the primary audience. Such linkages are integral to the PHTC program. When an online module “translates” academic content expertise for practitioners and is accessible to public health workers who have matching training needs, the PHTC has achieved its mandate. In practice, the design process does not unfold in five, distinct linear steps. For example, ideas for solutions start emerging during the problem definition stage. An initial idea for the notation exercise in the evaluation module was formulated as early as the learning objectives stage. The team needs to document early ideas but not act on them until the appropriate stage of the process. Design is inherently solution oriented, but it is important to let the process unfold and to avoid premature decisions. As we apply the design process as currently defined, some challenges remain. To implement the process well, sufficient resources and expertise must be available,
46S HEALTH PROMOTION PRACTICE / March 2014
including access to expertise in instructional technology design and development. Additionally, the role of theory could be expanded in this type of work.
Conclusions >> Our design process was influenced by strategies from several fields. It illustrates the positive cross-fertilization of ideas achieved when partners with different expertise collaborate. We learned firsthand that basic design processes are the same, whether one is trained as an educator, IT designer, or public health professional. However, although much can be learned by breaching the “silos” of practice, there also are good reasons for fields to adopt unique approaches. Although variations of effective design models, such as those described by Elwyn et al. (2011) and Cook and Dupras (2004), should be documented and disseminated, each design team should adopt a process that works within their context and modify it as needed. New technologies, such as distance learning, offer promising solutions. However, more work lies ahead as we seek to discover how to use these solutions and how to use them well. Some remaining challenges include effective dissemination, strategies to motivate learners, and strategies to address the social isolation of the online learner. Still, the design process begins to unravel the mystery and uncertainty that exist between needs identification and deployment of a solution. The process offers guideposts, but patience is still required. Some expert designers have noted that tolerance for uncertainty and trust in the process are their greatest strengths (Cross, 2006). References Association of State and Territorial Health Officials. (2008). 2007 State Public Health Workforce Survey results. Retrieved from http://www.astho.org/Programs/Workforce-and-LeadershipDevelopment/2007-State-Public-Health-Workforce-SurveyResults/ Bartholomew, L. K., Parcel, G. S., Kok, G., Gottlieb, N. H., & Fernandez, M. E. (2011). Planning health promotion programs: An intervention mapping approach. San Francisco, CA: Wiley. Bereiter, C. (2002). Design research for sustained innovation. Cognitive Studies, Bulletin of the Japanese Cognitive Science Society, 9, 321-327. Bloom, B., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals by a committee of college and university examiners: Handbook 1. Cognitive domain. London, England: Longmans. Bower, P., Kelsey, R., & Moretti, F. (2011). Brownfield action: An inquiry based multimedia simulation for teaching and learning environmental science. Science Education & Civic Engagement, 3, 7-14.
Cook, D. (2007). Web-based learning: Pros, cons and controversies. Clinical Medicine, 7(1), 37-42. Cook, D., & Dupras, D. (2004). A practical guide to developing effective web-based learning. Journal of General Internal Medicine, 19, 698-707. doi:10.1111/j.1525-1497.2004.30029.x The Council on Linkages Between Academia and Public Health Practice. (2010). Core competencies for public health professionals. Retrieved from http://www.phf.org/resourcestools/Documents/ Core_Public_Health_Competencies_III.pdf Cross, N. (2006). Designerly ways of knowing. London, England: Springer. Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11, 105-121. doi:10.2307/1466722 Elwyn, G., Kreuwel, I., Durand, M. A., Sivell, S., Joseph-Williams, N., Evans, R., & Edwards, A. (2011). How to develop web-based decision support interventions for patients: A process map. Patient Education and Counseling, 82, 260-265. doi:10.1016/j. pec.2010.04.034 Gilmore, G. (2012). Needs and capacity assessment strategies for health education and health promotion (4th ed.). Burlington, MA: Jones & Bartlett Learning. Green, L., & Kreuter, M. (2005). Health program planning: An educational and ecological approach (4th ed.). Boston, MA: McGraw-Hill. Health Resources and Services Administration. (2011). Funding opportunity announcement: Public Health Training Centers program (Catalog of Federal Domestic Assistance No. 93.516). Washington, DC: Author.
Institute of Medicine. (2003). Who will keep the public healthy: Educating public health professionals for the 21st century. Washington, DC: National Academies Press. Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler. Mayer, R. E. (2002). Multimedia learning. In H. R. Brian (Ed.), Psychology of learning and motivation (Vol.41, pp. 85-139). New York, NY: Academic Press. Merriam, S. B., Caffarella, R. S., & Baumgartner, L. M. (2007). Learning in adulthood: A comprehensive guide (3rd ed.). San Francisco, CA: Jossey-Bass. Norman, D. (1988). The design of everyday things. New York, NY: Basic Books. Robertson, S., & Robertson, J. (2006). Mastering the requirements process (2nd ed.). Upper Saddle River, NJ: Pearson Education. Schon, D. (1990). The design process. In V. A. Howard (Ed.), Varieties of thinking: Essays from Harvard’s Philosophy of Education Research Center (pp. 110-141). New York, NY: Routledge. Simon, H. (1969). The sciences of the artificial. Cambridge, MA: MIT Press. U.S. Department of Education. (2010). Evaluation of evidencebased practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Author. Wallas, G. (1926). The art of thought. New York, NY: Harcourt Brace Jovanovich. Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Upper Saddle River, NJ: Pearson.
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
47S
HPPXXX10.1177/1524839913509272He alth Promotion Practice / MonthMillery et al. / Innovative Instructional Technology Design Process 2013
Using Innovative Instructional Technology to Meet Training Needs in Public Health: A Design Process Mari Millery, PhD1 Michelle Hall, MA2 Joanna Eisman, MPH1 Marita Murrman, EdD1
Technology and distance learning can potentially enhance the efficient and effective delivery of continuing education to the public health workforce. Public Health Training Centers collaborate with instructional technology designers to develop innovative, competency-based online learning experiences that meet pressing training needs and promote best practices. We describe one Public Health Training Center’s online learning module design process, which consists of five steps: (1) identify training needs and priority competencies; (2) define learning objectives and identify educational challenges; (3) pose hypotheses and explore innovative, technology-based solutions; (4) develop and deploy the educational experience; and (5) evaluate feedback and outcomes to inform continued cycles of revision and improvement. Examples illustrate the model’s application. These steps are discussed within the context of design practices in the fields of education, engineering, and public health. They incorporate key strategies from across these fields, including principles of programmatic design familiar to public health professionals, such as backward design. The instructional technology design process we describe provides a structure for the creativity, collaboration, and systematic strategies needed to develop online learning products that address critical training needs for the public health workforce. Keywords: workforce development; continuing education; distance education; technology
Health Promotion Practice March 2014 Vol. 15, Suppl 1 39S–47S DOI: 10.1177/1524839913509272 © 2014 Society for Public Health Education
Introduction >> Use of technology, and of distance-based learning in particular, offers innovative solutions to problems associated with the delivery of training to working professionals. The public health workforce has a variety of pressing training needs (Institute of Medicine, 2003), which the Public Health Training Centers (PHTCs) are tasked with addressing. PHTC needs assessments clearly indicate that convenient access to training is necessary to reach public health workers who are continually asked to do more with less time and fewer resources. Web-based educational experiences have many advantages. They negate the need for off-site travel to training centers, can be pursued by learners when time permits, reach large numbers of learners, and use multimedia and innovative instructional methods (Cook,
1
Mailman School of Public Health of Columbia University, New York, NY, USA 2 Columbia Center for New Media Teaching and Learning, New York, NY, USA Authors’ Note: The work described in the article was supported by Health Resources and Services Administration Grant No. UB6HP20788. All article contributors received support from the grant. Address correspondence to Mari Millery, Mailman School of Public Health of Columbia University, 722 W 168th Street, Room 543, New York, NY 10032, USA; e-mail: mm994@columbia. edu. Supplement Note: This article is published in the supplement “Public Health Training Centers,” which was supported by the U.S. Department of Health and Human Services (HHS) Health Resources and Services Administration (HRSA) through a Cooperative Agreement (UBGHP20199) to Rollins School of Public Health, Emory University.
39S
2007). They also have disadvantages, including social isolation of learners, a need for high-quality design expertise and resources, and potential technical problems. Overall, however, evidence reviews show that online learning is effective compared with face-to-face learning (U.S. Department of Education, 2010). As PHTCs have begun producing online learning modules, they have developed design processes and learned many lessons. There is a need to document these design models for their own utility and to supplement the sparse literature on the topic. Below, we examine the literature on the design process, present the online module design process of the New York City-Long Island-Lower Tri-County (NYC-LI-LTC) PHTC, and illustrate the five-step process with examples from the Advanced Program Evaluation module (http://www.lowernysphtc.org/content/advanced-program-evaluation) and other projects.
Background >>
The federal funding guidance to PHTCs encourages use of distance learning modalities to deliver competency-based trainings consistent with the needs of the public health workforce (Health Resources and Services Administration, 2011). Over the past decade, PHTCs have developed a large number of online learning programs and continue to design new distance-based trainings in order to meet emerging needs and reach more trainees. The design process established by the NYC-LILTC PHTC for online learning modules makes innovative use of technology to meet identified training needs with content and pedagogy that effectively promote best practices reflecting the latest public health advances. The NYC-LI-LTC PHTC seeks to enhance the public health workforce’s ability to effectively deliver essential public health services in New York City and Suffolk, Nassau, Westchester, Rockland, and Putnam Counties. To design and build its online learning modules, the NYC-LI-LTC PHTC collaborates with the Columbia Center for New Media Teaching and Learning (CCNMTL), a service organization with a mission to support Columbia University in the purposeful use of new media technology. CCNMTL has completed more than 300 custom-designed learning projects, including web-based training environments, visualization and modeling tools, media archives, and simulations. Close collaboration between the PHTC and CCNMTL is essential to the success of the design process. Since public health workers in the United States have an average of 12 years of experience and an average age of 47 (Association of State and Territorial
40S HEALTH PROMOTION PRACTICE / March 2014
Health Officials, 2008), development of all NYC-LILTC PHTC trainings is guided by the key principles of adult learning theory (Merriam, Caffarella, & Baumgartner, 2007). Specifically, we use needs assessment results to identify the breadth and depth of learning public health workers need to perform their jobs. We engage workforce representatives in both the design and learning processes we choose for each training program. When presenting theoretical concepts, we do so by having workers apply the concepts to relevant public health problems. Finally, we use participatory techniques, like case studies and problem-solving activities, whenever possible. Herbert Simon’s (1969) influential definition of design reflects our team’s realization that the design process is essentially the same across fields: Everyone designs who devises courses of action aimed at changing existing situations into preferred ones. The intellectual activity that produces material artifacts is no different fundamentally from the one that prescribes remedies for a sick patient or the one that devises a new sales plan for a company or a social welfare policy for a state. (p 130)
Similarly, in his book, Designerly Ways of Knowing (2006), Nigel Cross suggests that although we do not know enough about design processes across professions, there is likely to be a “deep structure” to the process of translating needs to products. Across fields, the design process is a craft that practitioners hone through experience. It can be difficult to describe because it relies on implicit knowledge of expert practitioners. Edelson (2002) further points out, “The process of design is complex. Its open-endedness and reliance on creativity have made it a challenge for researchers to characterize and explain.” In an effort to take on the challenge, Edelson discusses three kinds of general design decisions: design process (steps and people involved), problem analysis (needs assessment), and design solution (resulting product). The specific steps outlined below encompass the progression from problem analysis to design solution, and constitute our approach to the design process. Design is essential for a variety of human endeavors. Design approaches across different fields have commonalities and differences. In the field of education, the tools of design include competencies and learning objectives. Core competency sets have been defined for many professions, including public health (The Council on Linkages Between Academia and Public Health Practice, 2010). Crafting of learning objectives is guided
Step
1. Assess needs and identify priority competencies.
2. Define learning objectives and identify educational challenges.
===>
===>
===>
Training Needs
Learning Objectives
Hypotheses
Competencies
Educational Challenges
Innovative Solutions
Key Concepts
3. Pose hypotheses and explore innovative technology-based solutions.
4. Develop and deploy educational experience.
5. Collect feedback and evaluate to inform cycles of revision and improvement.
===>
Development
Feedback
Deployment
Evaluation
Figure 1 Steps and Key Concepts of PHTC Online Learning Module Design Process NOTE: PHTC = Public Health Training Center.
by the tradition of Bloom’s taxonomy (Bloom & Krathwohl, 1956). The design approach to information technology (IT) applications follows engineering practices, with an emphasis on requirements analysis and requirements engineering (Robertson & Robertson, 2006). Just as learning objectives define what learners are expected to do as a result of the learning experience, IT requirements specify what the finished IT product is expected to do for the users. In health promotion and public health, professionals are trained to use systematic program design models, such as PRECEDEPROCEED (Green & Kreuter, 2005) and Intervention Mapping (Bartholomew, Parcel, Kok, Gottlieb, & Fernandez, 2011). The design approach uses theories and research evidence to inform the process. The design processes used in education, IT, and public health share several features. First is the fundamental principle of backward design—planning with the “end in mind” (Wiggins & McTighe, 2005). Backward design posits that the starting point of design is the end goal to be achieved, although designers in different fields may use different tools to define the goal and the path to it. Second is the use of audience feedback to inform the design, whether the audience is defined as learners (education), end users (IT), or stakeholders (public health). And third is the use of innovation, by which we mean the new, often creative and unique, solutions that emerge during the design process. Innovation involves the notion of a creative leap (Cross, 2006) and the cognitive process of creativity in general, as outlined in Wallas’s (1926) classic model of preparation, incubation,
illumination, and verification. CCNMTL follows the design research approach (Edelson, 2002), which features elements of academic inquiry and empirical research and emphasizes innovation (Bereiter, 2002). The process of designing interactive online tools and learning experiences is not well documented. Elwyn et al. (2011) describe a process for developing web-based decision support for patients. Their process, consisting of content specification and creative design, envisions design as a team effort requiring various kinds of expertise, captured, in part, in management and advisory groups. Cook and Dupras (2004) present a 10-step model for designing web-based learning modules, with an emphasis on active learning. Below, we describe the NYC-LI-LTC PHTC online learning module design process, which incorporates the design principles discussed above and progresses from perceived training needs in the field to convenient and innovative online trainings that support best practices in public health.
Method and Application >> The PHTC design process for online learning modules consists of five general steps: (1) assess needs and identify priority competencies, (2) define learning objectives and identify educational challenges, (3) pose hypotheses and explore innovative technology-based solutions, (4) develop and deploy the educational experience, and (5) collect feedback and evaluate to inform continued cycles of revision and improvement. Figure 1
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
41S
lists the sequence of the steps and associated key concepts. However, although the steps are depicted in sequence, it is important to note that they do not typically unfold in a strictly linear process; work often fluctuates among steps. Collaboration among participating organizations and team members, in their various roles, is integral to the design process. According to Edelson (2002), design methodology specifies required expertise and the roles of individuals providing the expertise. As Schon (1990) points out, design is a social process. In addition to the PHTC and content expert, there is an important role for the primary audience (learners), whose input is critical at various stages. CCNMTL staff fill a range of design and technical roles, such as instructional designer, web designer, interface designer, interactive designer, user experience designer, and developer/programmer. However, an effective online learning design team may not always require all of these roles. Because of team members’ diverse backgrounds, we allocate time for everyone to learn about potentially unfamiliar methods, and we clearly define team members’ roles. Our process was developed for the design of interactive, online, self-study learning modules, as distinguished from other types of online learning experiences, such as instructor-facilitated sessions, blended experiences combining online and face-to-face learning, or noninteractive recordings of live trainings. The typical module takes less than 1 hour to finish and can be completed at the learner’s convenience in one or more sessions. On completion, a certificate is offered. Although the design process was developed for a specific public health application, it can be used to guide the design of any technology-based educational experience. Step 1. Assess Needs and Identify Priority Competencies Step 1 is basically problem analysis: characterizing the goals, needs, or opportunities the design is intended to address (Edelson, 2002). Problem analysis may require systematic use of applied social science research methods to conduct a thorough needs assessment, an activity familiar to health promotion practitioners (Gilmore, 2012). Our training needs assessment uses key informant interviews and surveys to identify training needs among public health workers in the defined geographic area. Training needs are based on standard core competencies for public health professionals (The Council on Linkages Between Academia and Public Health Practice, 2010). Once a competency, or set of competencies, is identified as a priority training need, the team considers whether an online module is a
42S HEALTH PROMOTION PRACTICE / March 2014
potential solution. A purposeful rationale for the use of technology is required, as it may not be the best modality for all instruction. The core competency is then further analyzed to identify subcompetencies that closely correspond with the skills workers wish to acquire and the tasks they want to accomplish. For example, one of our highpriority competencies was Identify mechanisms to monitor and evaluate programs for their effectiveness and quality. After consulting with stakeholders and a program evaluation expert, we identified a related, high-priority subcompetency: Design rigorous and feasible evaluation plans. The next step was to specify the primary audience, in this case, midlevel local health department staff with program evaluation responsibilities. Figure 2 shows how training needs become translated into learning objectives, using the Advanced Program Evaluation module as an example. This translation process spans Step 1 and into Step 2. Key local health department informants highlighted the need for better program evaluation, addressing the corresponding core competency. Input from multiple team members and stakeholders was used to refine the subcompetency and to develop the learning objectives. The PHTC facilitates the connections among stakeholders representing the primary audience, instructional design/technical staff, and the content expert. Step 2. Define Learning Objectives and Identify Educational Challenges Learning objectives are central to instructional design (see Figure 2). They specify what learners are expected to accomplish as a result of the learning experience, thereby enabling “backward design” of the experience and assessment of learning outcomes. In Step 2, the content expert and instructional designer play key roles. Selection of an appropriate content expert to write the module content is critical to assure that the PHTC disseminates high-quality content on best practices. Learning objectives provide the context for identifying educational challenges learners might face. For example, what are some challenging concepts for learners to grasp? Input from stakeholders and the primary audience is important here. Sometimes a broader educational challenge motivates the design. For example, when CCNMTL worked with an environmental science professor to develop an inquiry-based multimedia simulation called Brownfield Action (Bower, Kelsey, & Moretti, 2011), the educational challenge was lack of
NEEDS ASSESSMENT
TRANSLATION OF NEED
• PHTC
NEED IN THE FIELD
PHTC
• Public health workers and their managers
Sub-Competency: Design rigorous and feasible evaluation plans
“To do better program evaluation” COL Competency: Identify mechanisms to monitor and evaluate programs for their effectiveness and quality
• PHTC • Content Expert • Instructional Technologist • Input from primary audience (further needs assessment)
EXPERTISE • Content expert Best practices in program evaluation
LEARNING OBJECTIVES − Appreciate that a comprehensive evaluation plan addresses a program’s logic, stakeholders, evaluation questions, and evaluation design. − Assess advantages and limitations of evaluation designs, including randomized, quasiexperimental and prepost designs. − Appraise and compare options for data collection methods, measures, and sampling strategies. − Identify and address associated threats to validity. − Identify strategies for addressing budget, time, data, and political constraints in evaluation practice.
Figure 2 Translating Training Needs Into Learning Objectives: Advanced Program Evaluation Module Example NOTE: PHTC = Public Health Training Center; COL = Council on Linkages.
excitement and engagement in science among nonscience undergraduates. A specific educational challenge of the Advanced Program Evaluation project was how to introduce logic models. Although knowledge of logic models was a prerequisite, feedback from primary audience representatives indicated the need for a brief review. Furthermore, we wanted to structure the learning experience as a case study and needed a way of presenting the logic model in this context. In Step 3, the solution was found through discussions with primary audience representatives and other team members: a brief, narrated video presentation of logic model components, using the case study example. Work often crosses back and forth between Steps 2 and 3. In design language, the identification of functional requirements frequently blurs with the search for solutions (Cross, 2006). Step 3. Pose Hypotheses and Explore Innovative Technology-Based Solutions Step 3 moves from problem analysis to solution construction (Edelson, 2002). It has been characterized as
a systematic search through a solution space (Simon, 1969) and a process of synthesis with considerable complexity (Schon, 1990). Following the model of design research (Edelson, 2002), Step 3 prompts the team to pose hypotheses about potential solutions. In the Advanced Program Evaluation project, we found that learners have difficulty grasping the elements of evaluation study designs, such as preintervention/ postintervention, comparison group, and randomized designs. We hypothesized that a traditional notation system often used to depict study designs with Xs for interventions and Os for observation points can serve as a learning tool, if learners apply the notation system to different examples. This hypothesis led to the design of an interactive notation exercise (see Figure 3). These types of interactive exercises are an important part of the online modules and are developed iteratively, as primary audience representatives and other team members provide ongoing feedback on a series of prototypes. Most NYC-LI-LTC PHTC modules are structured around case stories with protagonists. This is based on the hypothesized effectiveness of cases in organizing
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
43S
solutions. Sometimes, an off-the-shelf solution is available and selected instead of a custom design. Consistent with Cross’s (2006) account of designerly thinking, we find that moments of design “illumination” are in fact the result of the following: systematic exploration of existing solutions; deliberate use of cognitive processes, such as analogies; and sustained focus on the most critical functional requirements. These processes can yield unique, technology-based solutions— such as the logic model video and notation exercise described above—to educational challenges. Step 4. Develop and Deploy the Educational Experience Figure 3 Example of Interactive Component in Learning Module: Evaluation Design Notation Exercise NOTE: X = intervention/program/treatment; O = observation/ outcome measurement point (e.g., pretest or posttest); R = randomly assigned; NR = not randomly assigned. The two rows represent the two study groups to be compared. In the interactive exercise, the learner composes the design elements for a given scenario by selecting them from the blue boxes and dragging them into the design rows.
learning content. The team spends time selecting and defining an appropriate case, which the content expert writes up. The content expert and instructional designer develop a detailed outline of module content and search for innovative solutions to the challenges posed by the selected learning objectives. The outline is circulated to all team members, including primary audience representatives, for feedback. As module components are defined, we explore a variety of media to determine what would best support a particular type of content, following design principles from cognitive theory of multimedia learning (Mayer, 2002). Online learning modules can easily incorporate various media, including text, images, video, audio, interactive exercises, and quizzes. Each medium has unique affordances and constraints to be considered (Norman, 1988). Sometimes excitement about a technology may lure designers to add “bells and whistles” that may not add real value; it is important to avoid overengineering. The exploration of technology-based solutions draws on existing solutions, including those previously designed by CCNMTL. When exploring the mechanics of the notation exercise described above, we investigated digital tic-tac-toe and Sudoku game user interfaces. Investigating previous solutions to similar problems is an essential part of design thinking (Cross, 2006) and makes the process more efficient. Experienced designers are skilled at borrowing and adapting proven 44S HEALTH PROMOTION PRACTICE / March 2014
Designers are trained to translate functional user requirements into formal patterns of design structure (Cross, 2006). Development is the process of creating the design structure. In practice, it fluctuates back and forth between Steps 3 and 4, as ideas are sent back to the “drawing board” to be refined or replaced. Creative design continues during Step 4, with special focus on creating a visually attractive “look” and intuitive online navigation. Representation of the design concept is important at this step. Information architecture diagrams, website wire frames, and storyboards are commonly used representational tools. Figure 4 depicts a wireframe and the final design of the website housing our online learning modules. In this case, the programming challenge was to allow learners to leave multiple modules unfinished and to return later in order to complete them. The eventual solution was a detailed table of contents for each module, with icons indicating completion of module components. Step 4 may include a series of prototypes iteratively tested and developed. Focus groups, think-aloud techniques, and usability laboratories may be used. At the very least, the team wants to identify usability problems and content errors before wider dissemination. Deployment and dissemination are critically important steps, although not fully addressed here. Interested readers can view some of the products of our design process at http://www.lowernysphtc.org. In 2012, we launched two online modules: Advanced Program Evaluation and Introduction to Qualitative Analysis with ATLAS.ti. Additional modules were under development while this article was in press, with a 2013 launch date. Step 5. Collect Feedback and Evaluate to Inform Continued Cycles of Revision and Improvement After deployment, the online module remains a work in progress. Previously designed modules should be periodically reviewed and updated. We collect
Figure 4 Example of Representational Tool: Website Wireframe and Final Design Product
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
45S
systematic user feedback and evaluation data to assess learning and behavioral outcomes (Kirkpatrick & Kirkpatrick, 2006). Brief surveys are built into the modules as pre- and posttests to measure knowledge and self-efficacy. We also collect feedback on the learning experience and ask about intent to apply the learning content in practice. Additional follow-up evaluation surveys are e-mailed 6 weeks after module completion.
Discussion >> The NYC-LI-LTC PHTC has established a successful five-step design process that begins with competencybased needs assessment, moves through systematic stages of instructional technology design, and ends with evaluation and dissemination of a high-quality online learning experience. It is consistent with the 10-step model Cook and Dupras (2004) developed in the field of medical education. The process incorporates key design principles, including backward design, audience feedback, and innovation. It brings together a synergy of design strategies from the fields of education, public health, and IT engineering. Design is a team effort requiring effective collaboration among many individuals and organizations. As entities linked with academic and practice settings, PHTCs are well positioned to form effective design teams. Academic public health faculty can be recruited to serve as content experts. University-based instructional technology groups are ideally suited to partner with PHTCs to design technology-based learning experiences. And close linkages with public health practice settings provide a mechanism for meaningful input from the primary audience. Such linkages are integral to the PHTC program. When an online module “translates” academic content expertise for practitioners and is accessible to public health workers who have matching training needs, the PHTC has achieved its mandate. In practice, the design process does not unfold in five, distinct linear steps. For example, ideas for solutions start emerging during the problem definition stage. An initial idea for the notation exercise in the evaluation module was formulated as early as the learning objectives stage. The team needs to document early ideas but not act on them until the appropriate stage of the process. Design is inherently solution oriented, but it is important to let the process unfold and to avoid premature decisions. As we apply the design process as currently defined, some challenges remain. To implement the process well, sufficient resources and expertise must be available,
46S HEALTH PROMOTION PRACTICE / March 2014
including access to expertise in instructional technology design and development. Additionally, the role of theory could be expanded in this type of work.
Conclusions >> Our design process was influenced by strategies from several fields. It illustrates the positive cross-fertilization of ideas achieved when partners with different expertise collaborate. We learned firsthand that basic design processes are the same, whether one is trained as an educator, IT designer, or public health professional. However, although much can be learned by breaching the “silos” of practice, there also are good reasons for fields to adopt unique approaches. Although variations of effective design models, such as those described by Elwyn et al. (2011) and Cook and Dupras (2004), should be documented and disseminated, each design team should adopt a process that works within their context and modify it as needed. New technologies, such as distance learning, offer promising solutions. However, more work lies ahead as we seek to discover how to use these solutions and how to use them well. Some remaining challenges include effective dissemination, strategies to motivate learners, and strategies to address the social isolation of the online learner. Still, the design process begins to unravel the mystery and uncertainty that exist between needs identification and deployment of a solution. The process offers guideposts, but patience is still required. Some expert designers have noted that tolerance for uncertainty and trust in the process are their greatest strengths (Cross, 2006). References Association of State and Territorial Health Officials. (2008). 2007 State Public Health Workforce Survey results. Retrieved from http://www.astho.org/Programs/Workforce-and-LeadershipDevelopment/2007-State-Public-Health-Workforce-SurveyResults/ Bartholomew, L. K., Parcel, G. S., Kok, G., Gottlieb, N. H., & Fernandez, M. E. (2011). Planning health promotion programs: An intervention mapping approach. San Francisco, CA: Wiley. Bereiter, C. (2002). Design research for sustained innovation. Cognitive Studies, Bulletin of the Japanese Cognitive Science Society, 9, 321-327. Bloom, B., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals by a committee of college and university examiners: Handbook 1. Cognitive domain. London, England: Longmans. Bower, P., Kelsey, R., & Moretti, F. (2011). Brownfield action: An inquiry based multimedia simulation for teaching and learning environmental science. Science Education & Civic Engagement, 3, 7-14.
Cook, D. (2007). Web-based learning: Pros, cons and controversies. Clinical Medicine, 7(1), 37-42. Cook, D., & Dupras, D. (2004). A practical guide to developing effective web-based learning. Journal of General Internal Medicine, 19, 698-707. doi:10.1111/j.1525-1497.2004.30029.x The Council on Linkages Between Academia and Public Health Practice. (2010). Core competencies for public health professionals. Retrieved from http://www.phf.org/resourcestools/Documents/ Core_Public_Health_Competencies_III.pdf Cross, N. (2006). Designerly ways of knowing. London, England: Springer. Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11, 105-121. doi:10.2307/1466722 Elwyn, G., Kreuwel, I., Durand, M. A., Sivell, S., Joseph-Williams, N., Evans, R., & Edwards, A. (2011). How to develop web-based decision support interventions for patients: A process map. Patient Education and Counseling, 82, 260-265. doi:10.1016/j. pec.2010.04.034 Gilmore, G. (2012). Needs and capacity assessment strategies for health education and health promotion (4th ed.). Burlington, MA: Jones & Bartlett Learning. Green, L., & Kreuter, M. (2005). Health program planning: An educational and ecological approach (4th ed.). Boston, MA: McGraw-Hill. Health Resources and Services Administration. (2011). Funding opportunity announcement: Public Health Training Centers program (Catalog of Federal Domestic Assistance No. 93.516). Washington, DC: Author.
Institute of Medicine. (2003). Who will keep the public healthy: Educating public health professionals for the 21st century. Washington, DC: National Academies Press. Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler. Mayer, R. E. (2002). Multimedia learning. In H. R. Brian (Ed.), Psychology of learning and motivation (Vol.41, pp. 85-139). New York, NY: Academic Press. Merriam, S. B., Caffarella, R. S., & Baumgartner, L. M. (2007). Learning in adulthood: A comprehensive guide (3rd ed.). San Francisco, CA: Jossey-Bass. Norman, D. (1988). The design of everyday things. New York, NY: Basic Books. Robertson, S., & Robertson, J. (2006). Mastering the requirements process (2nd ed.). Upper Saddle River, NJ: Pearson Education. Schon, D. (1990). The design process. In V. A. Howard (Ed.), Varieties of thinking: Essays from Harvard’s Philosophy of Education Research Center (pp. 110-141). New York, NY: Routledge. Simon, H. (1969). The sciences of the artificial. Cambridge, MA: MIT Press. U.S. Department of Education. (2010). Evaluation of evidencebased practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Author. Wallas, G. (1926). The art of thought. New York, NY: Harcourt Brace Jovanovich. Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Upper Saddle River, NJ: Pearson.
Millery et al. / INNOVATIVE INSTRUCTIONAL TECHNOLOGY DESIGN PROCESS
47S
