Nurse Educator
Nurse Educator Vol. 39, No. 1, pp. 22-25 Copyright * 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins
Development, Evaluation, and Utility of a Peer Evaluation Form for Online Teaching Carol D. Gaskamp, PhD, RN, APHN-BC & Eileen Kintner, PhD, RN, FAAN Formative assessment of teaching by peers is an important component of quality improvement for educators. Teaching portfolios submitted for promotion and tenure are expected to include peer evaluations. Faculty resources designed for peer evaluation of classroom teaching are often inadequate for evaluating online teaching. The authors describe development, evaluation, and utility of a new peer evaluation form for formative assessment of online teaching deemed relevant, sound, feasible, and beneficial.
F
ormative assessments of teaching by faculty peers are important components of quality improvement for educators. In addition, teaching portfolios submitted for promotion and tenure are expected to include annual peer evaluations (PEs) of teaching. Faculty resources designed for PE of classroom teaching at our school of nursing did not reflect many aspects that were important for evaluating online teaching in Web-based courses. For example, the evaluation resources for traditional classroom observation focused on classroom performance and real-time interactions between faculty and students. The asynchronous nature of online teaching was not captured in the materials used for PE and did not reflect principles of design for online courses. A more appropriate PE form for online teaching was needed for the faculty whose teaching assignments included online courses. The process of developing a new PE form for online teaching began with a review of the literature. CINAHL, PubMed, and ERIC databases were searched using the following terms: evaluation, assessment, peer, and online teaching. The preponderance of published journal articles focused on student evaluation of online teaching and learning rather than PE, with the majority of articles published before 2005. The Google search engine was used to expand our pursuit of more recent information. The expanded search returned Internet sites of universities and organizations that sought to engage in peer review of online teaching. Based on the literature review, frameworks, guidelines, principles, and criteria used to inform (a) development of our PE of
Author Affiliations: Associate Professor of Clinical Nursing (Dr Gaskamp) and Associate Professor (Dr Kintner), School of Nursing, The University of Texas at Austin. The authors declare no conflicts of interest. Correspondence: Dr Gaskamp, School of Nursing, D0100, 1710 Red River St, Austin, TX 78701 ([email protected]). Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.nurseeducatoronline.com). DOI: 10.1097/NNE.0000000000000007
22
Volume 39 & Number 1 & January/February 2014
online teaching form and (b) design of our peer-review process are summarized in the following sections.
Choosing the Framework Origins of distance education are traced to 1840 when Sir Isaac Pitman offered a correspondence course to teach shorthand by mail.1 Completely online undergraduate courses were 1st offered in 1994.1,2 Initially, courses used technology available at the time, such as video conferencing, e-mail, listservs, and discussion boards.3 Online learning expanded as computer, information, and communication technology was advanced, and course delivery platforms and software improved.2,4 The quality of online teaching became of concern as more courses and programs were approved for distance learning.5,6 Historically, evaluation of distance education and best practices for online teaching relied on students’ perceptions of their learning experiences.7-12 In 2000, Billings10 introduced a framework for PE of online nursing courses with a focus on the use of technology, faculty and student support, educational practices, and a variety of learning and other outcomes. A year later, Cobb et al13 offered guidelines and processes for conducting peer reviews of teaching online nursing courses. Their guidelines contained Chickering and Gamson’s14 principles of good practice in undergraduate education as well as principles in instructional (eg, clear instructions for assignments) and graphic (eg, appearance of Web page) design.13 A 20-item rating scale based on the guidelines was developed for purposes of evaluating the course; however, some of the items were limiting. Their process of conducting peer reviews involved (a) delaying at least 1 year to give faculty the opportunity to test and revise the course design and teaching strategies, (b) waiting until well into the course so that the reviewers were able to observe how the guidelines were operationalized, and (c) obtaining student input for their reviews. We examined peer-review guidelines available from Kingsborough Community College15 and Pennsylvania State Nurse Educator
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
University16 and criteria for good online teaching practices obtained from a variety of other sources.17-21 The guidelines from these sources remained congruent with the guidelines and criteria offered by Cobb et al.13 In 2009, Wood and Friedel22 advanced the science by specifying a framework for peer review of online learning and teaching. The concepts contained in the expanded framework of Wood and Friedel22 reflected the major categories that we wanted to capture in our peer review, specifically instructional design, interface design, use of media, and technical aspects of the course. Therefore, we elected to use their framework to guide development of our online PE form. Two additional reports informed design of the process we used for our peer review.
Designing the Process The 1st report that informed design of the process used for our peer review was a pilot study that compared 2 peer reviews of online course rating scales available in the public domain23 from Quality Mattersi (QM)24 and the University of South Florida, College of Public Health (COPH).25 The QM scale contained 40 standards addressing course overview and introduction, learning objectives, assessment and measurement, resources and materials, learning interaction, course technology, learner support, and compliance with the Americans with Disabilities Act. The COPH scale contained 53 standards addressing (a) communication; (b) course delivery, organization, and design; and (c) instructional elements. Although the reviewers reported a preference for the QM because the tool was user-friendly and possessed content validity,23 for our purposes using the tool posed a barrier for implementation because of subscription cost and the need for specialized training to use the tool (Quality Mattersi, https://www.qualitymatters.org). We modeled the new form on the PE form currently used at our school for classroom observation to maintain a similar format, structure, and rating scale in order to standardize our school’s approach to PE. Little23 reported the use of an independent, 3rd-party peer-review team, composed of a nurse educator and an instruction designer, to evaluate the use both the QM and COPH rating scales in 2 online courses offered in an RN-BSN program. After reviewing the courses independently, the reviewers compared and discussed their individual ratings for each course using both tools. Rather than conducting an independent, 3rd-party review, we used our form to evaluate each other’s online courses, both of which were offered in
an RN-BSN program. We completed our reviews independently and then met to discuss our ratings. The 2nd report that informed design of our process described how an interdisciplinary team of health and education professionals used an action learning approach to PE of online courses.26 This approach entailed mutual development and support of each other’s courses across disciplines. Although their disciplines differed, team members shared the experience of teaching in a virtual online environment. They exchanged feedback on selected modules via e-mail and during face-to-face meetings. Similar to the interdisciplinary team, both of us had extensive experience in online and distance teaching. Rather than limiting ourselves to selected modules, we offered feedback via e-mail and during face-to-face meetings on the entire course. Despite the fact that the team members valued having an outside-the-discipline perspective of their courses,26 during initial development of the form we chose to evaluate faculty teaching and courses within our discipline. The newly developed form has been shared with other units in our university, and we anticipate future evaluations may be interdisciplinary.
Peer Evaluation Form The newly developed PE of online teaching form is a 40-item, 4-point Likert-type checklist with 1 overall rating item and 4 narrative response items (see Document, Supplemental Content 1, to view the tool, http://links.lww.com/NE/A114). Our review of literature was used to isolate potential prompts for our form. We revised statements and questions into prompts before grouping them under 4 major categories and 5 minor subcategories consistent with the Wood and Friedel22 framework (Table). Directions indicate that peer reviewers are to rate each prompt as truly exemplary, done well, needs improvement, or not applicable. Four open-ended items offer the opportunity for narrative responses by elaborating on strengths, highlighting areas for improvement, and providing additional comments. The form is not scored; rather, the prompts, ratings, summary item, and narrative comments are used to guide didactic discussion.
Validity Three faculty colleagues reviewed the new PE form for face and content validity. The reviewers used a standardized form
Table. Categories and Prompts in Peer Evaluation Form Category Instructional design
Interface design Use of media Technical aspects
Nurse Educator
Prompt Topics Clarity of expectations Building student knowledge Learning activities Learner assessment and evaluation Human interaction Ease of navigating and appearance of the site Appropriate, creative, and engaging use of technology with examples of videos, chats, podcasts, wikis, and discussion forums Technical support and accommodations for students with special needs
No. of Items 5 5 5 5 8 8 3
Prompts Prompts Prompts Prompts Prompts Prompts Prompts
3 Prompts
Volume 39 & Number 1 & January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
23
developed by Kintner27 for evaluating face and content validity to ensure consistent feedback. All 3 reviewers were doctorally prepared school of nursing administrators and recognized as expert teachers. Two reviewers were certified nurse educators who facilitated graduate level education/ teaching courses for nursing students and had experience teaching an online course. All of the reviewers reported that the form appeared sound and relevant. The purpose was deemed logically consistent with the aims. Although considered unnecessary for faculty in our school, the directions were judged to be clear and appropriate. The statements were found to be arranged in a logical order, grammatically correct, and appropriate for the response rating. Response ratings were judged to be appropriate for the instrument. Content was determined to be logically consistent and highly relevant with the instrument’s theoretical underpinnings. All 3 reviewers indicated that one of the human interaction prompts conveyed more than 1 thought. Two reviewers offered suggestions for revising another 1 of the human interaction prompts for clarity in meaning. Individual reviewers directed attention to 3 additional prompts considered to contain redundant or excess wording. The form was revised based on the internal reviewers’ comments and recommendations.
guidelines, and principles for good practice in online education. We found the form useful for formative assessment, to guide course revision and advance teaching, and to be a more objective assessment of online teaching compared with students’ subjective perceptions.
Conclusion The development of the new form and the review process exceeded the spirit of PE by generating lively dialogue about online teaching and learning and fostered collaborative relationships among faculty. We acknowledge the value of a national entity such as QM to continue to develop and expand the variety of options of standardized internal and external peer review, compared with the time and effort in developing a form that met our institutional standards without additional cost or specialized training. We shared this newly developed form with faculty teaching online courses in our School of Nursing, instructional design faculty in the College of Communication, and expert teachers supporting faculty development through the Center for Teaching and Learning at our university. We are offering the form to other schools and disciplines to also use the form and provide us with feedback and suggestions at peerreview. [email protected].
References Utility We adapted the action learning approach to conduct our PEs. Both of us granted each other instructor-level access to our respective online courses after the course ended to protect the privacy of students enrolled in both courses (eg, names and comments posted to a discussion forum) because some students were enrolled in both courses. The PE of online learning form was used as designed. Ratings and responses to the prompts naturally led to an overall summary and narrative comments that informed the basis for our face-to-face discussion. We provided written and verbal feedback via e-mail and in person. Face-to-face feedback allowed for sharing, validating, clarifying, and justifying perceptions of our teaching and learning activities. Although ratings for both of the peer reviews indicated the courses were well designed and facilitated, the form was valuable for guiding a thought-provoking discussion about teaching and learning in an online format. Teaching skills were advanced by exploring alternative methods for delivery of content and evaluation of student learning. Mutual synergy generated excitement about teaching and allowed for collaborative learning. By evaluating one another, we engaged in introspective reflection of our own methods and approaches to teaching and learning.
Feasibility and Benefits of the Form Faculty in our school initiate the peer-review process. We can request peer review from faculty within our school or from other units on campus to serve as our reviewer(s). Faculty teaching in online courses will now have a more applicable form that may be used by either faculty or instructional designers to provide feedback. The time investment required for completing the review was consistent with classroom observation reviews. Content reflected existing frameworks, 24
Volume 39 & Number 1 & January/February 2014
1. Phillips V. Virtual classrooms, real education. Nation’s Business. 1998;86(5):41. 2. Culatta R. Innovative Learning: Online Learning Timeline. 2011. Available at http://www.innovativelearning.com/online_learning/timeline.html. Accessed July 30, 2013. 3. Carlton KH, Ryan ME, Siktberg LL. Designing courses for the Internet: a conceptual approach. Nurse Educ. 1998;23(3):45-50. 4. Harris DA. Online distance education in the United States. IEEE Commun Mag. 1999;37(3):87-91. 5. Little BB. Quality assurance for online nursing courses. J Nurs Educ. 2009;48(7):381-387. 6. The Institute for Higher Education Policy. Quality on the line: benchmarks for success in Internet-based distance education. 2000. Available at http://www.ihep.org/assets/files/publications/ m-r/QualityOnTheLine.pdf. Accessed July 30, 2013. 7. Bailie JL. Effective online instructional competencies as perceived by online university faculty and students: a sequel study. MERLOT J Online Learn Teach. 2011;7(1):82-89. 8. Bangert AW. The seven principles of effective teaching: a framework for designing, delivering, and evaluating an Internet-based assessment course for nurse educators. Nurse Educ. 2005;30(5): 221-225. 9. Bangert AW. The development and validation of the Student Evaluation of Online Teaching Effectiveness. Comput Sch. 2008; 25(1-2):25-47. 10. Billings DM. A framework for assessing outcomes and practices in Web-based courses in nursing. J Nurs Educ. 2000;39(2):60-67. 11. Wagner RJ, Vanevenhoven JP, Bronson JW. A top ten list for successful online courses. MERLOT J Online Learn Teach. 2010; 6(2):542-545. 12. Billings DM, Connors HR, Skiba DJ. Benchmarking best practices in Web-based nursing courses. Adv Nurs Sci. 2001;23(3): 41-52. 13. Cobb KL, Billings DM, Mays RM, Canty-Mitchell J. Peer review of teaching in Web-based courses in nursing. Nurse Educ. 2001; 26(6):274-279. 14. Chickering AW, Gamson ZF. Seven principles for good practice in undergraduate education. AAHE Bull. 1987;39(7):3-7. Nurse Educator
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
15. Kingsborough Community College. Protocol for Peer Observation of an Online Course. March 2010. Available at https:// www.kbcc.cuny.edu/college_council/Documents/AttachmentA310 .pdf. Accessed July 30, 2013. 16. Taylor AH. A peer review guide for online courses at Penn State. 2010. Available at https://www.e-education.psu.edu/facdev/ peerreview. Accessed July 30, 2013. 17. Brinthaupt TM, Fisher LS, Garner JG, Raffo DM, Woodward JB. What the best online teachers should do. MERLOT J Online Learn Teach. 2011;7(4):515-524. 18. Brinthaupt TM, Pilati ML, King BR. Psychology teaching resources in the MERLOT Digital Learning Objects catalog. J Instr Psychol. 2008;35(3):240-245. 19. Multimedia Educational Resource for Learning and Online Teaching (MERLOT). Evaluation Criteria for Peer Reviews. 2010, 2013. Available at http://taste.merlot.org/evaluationcriteria .html. Accessed July 30, 2013. 20. Odom SE, Barnes K, Wicker M. Incorporating pedagogical principles in developing an online nursing research course. Comput Inform Nurs. 2005;23(3):146-152. 21. Zhu E, McKnight R, Edwards N. Faculty Development and
Nurse Educator
Support—Principles of Online Design: Evaluation of Online Course. 2009. Available at http://www.fgcu.edu/onlinedesign. Accessed July 30, 2013. 22. Wood D, Friedel M. Peer review of online learning and teaching: harnessing collective intelligence to address emerging challenges. Australas J Educ Technol. 2009;25(1):60-79. 23. Little BB. The use of standards for peer review of online nursing courses: a pilot study. J Nurs Educ. 2009;48(7):411-415. 24. Quality Mattersi. Peer course review rubric FY05/06. 2005. Available at https://www.qualitymatters.org/. Accessed March 17, 2009. 25. University of South Florida. COPH online course standards. 2006. Available at http://itt.usf.edu/publichealth/standards/standards/ std_012606/COPH_Online_Course_Standards_v5.doc. Accessed April 1, 2007. 26. Bennett S, Santy J. A window on our teaching practice: enhancing individual online teaching quality through online peer observation and support, A UK case study. Nurse Educ Pract. 2009;9:403-406. 27. Kintner EK. Testing of the School-Aged Child and Adolescent Acceptance of Asthma Model (PhD) [dissertation]. Tucson, AZ: The University of Arizona; 1996.
Volume 39 & Number 1 & January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
25
Nurse Educator Vol. 39, No. 1, pp. 22-25 Copyright * 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins
Development, Evaluation, and Utility of a Peer Evaluation Form for Online Teaching Carol D. Gaskamp, PhD, RN, APHN-BC & Eileen Kintner, PhD, RN, FAAN Formative assessment of teaching by peers is an important component of quality improvement for educators. Teaching portfolios submitted for promotion and tenure are expected to include peer evaluations. Faculty resources designed for peer evaluation of classroom teaching are often inadequate for evaluating online teaching. The authors describe development, evaluation, and utility of a new peer evaluation form for formative assessment of online teaching deemed relevant, sound, feasible, and beneficial.
F
ormative assessments of teaching by faculty peers are important components of quality improvement for educators. In addition, teaching portfolios submitted for promotion and tenure are expected to include annual peer evaluations (PEs) of teaching. Faculty resources designed for PE of classroom teaching at our school of nursing did not reflect many aspects that were important for evaluating online teaching in Web-based courses. For example, the evaluation resources for traditional classroom observation focused on classroom performance and real-time interactions between faculty and students. The asynchronous nature of online teaching was not captured in the materials used for PE and did not reflect principles of design for online courses. A more appropriate PE form for online teaching was needed for the faculty whose teaching assignments included online courses. The process of developing a new PE form for online teaching began with a review of the literature. CINAHL, PubMed, and ERIC databases were searched using the following terms: evaluation, assessment, peer, and online teaching. The preponderance of published journal articles focused on student evaluation of online teaching and learning rather than PE, with the majority of articles published before 2005. The Google search engine was used to expand our pursuit of more recent information. The expanded search returned Internet sites of universities and organizations that sought to engage in peer review of online teaching. Based on the literature review, frameworks, guidelines, principles, and criteria used to inform (a) development of our PE of
Author Affiliations: Associate Professor of Clinical Nursing (Dr Gaskamp) and Associate Professor (Dr Kintner), School of Nursing, The University of Texas at Austin. The authors declare no conflicts of interest. Correspondence: Dr Gaskamp, School of Nursing, D0100, 1710 Red River St, Austin, TX 78701 ([email protected]). Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.nurseeducatoronline.com). DOI: 10.1097/NNE.0000000000000007
22
Volume 39 & Number 1 & January/February 2014
online teaching form and (b) design of our peer-review process are summarized in the following sections.
Choosing the Framework Origins of distance education are traced to 1840 when Sir Isaac Pitman offered a correspondence course to teach shorthand by mail.1 Completely online undergraduate courses were 1st offered in 1994.1,2 Initially, courses used technology available at the time, such as video conferencing, e-mail, listservs, and discussion boards.3 Online learning expanded as computer, information, and communication technology was advanced, and course delivery platforms and software improved.2,4 The quality of online teaching became of concern as more courses and programs were approved for distance learning.5,6 Historically, evaluation of distance education and best practices for online teaching relied on students’ perceptions of their learning experiences.7-12 In 2000, Billings10 introduced a framework for PE of online nursing courses with a focus on the use of technology, faculty and student support, educational practices, and a variety of learning and other outcomes. A year later, Cobb et al13 offered guidelines and processes for conducting peer reviews of teaching online nursing courses. Their guidelines contained Chickering and Gamson’s14 principles of good practice in undergraduate education as well as principles in instructional (eg, clear instructions for assignments) and graphic (eg, appearance of Web page) design.13 A 20-item rating scale based on the guidelines was developed for purposes of evaluating the course; however, some of the items were limiting. Their process of conducting peer reviews involved (a) delaying at least 1 year to give faculty the opportunity to test and revise the course design and teaching strategies, (b) waiting until well into the course so that the reviewers were able to observe how the guidelines were operationalized, and (c) obtaining student input for their reviews. We examined peer-review guidelines available from Kingsborough Community College15 and Pennsylvania State Nurse Educator
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
University16 and criteria for good online teaching practices obtained from a variety of other sources.17-21 The guidelines from these sources remained congruent with the guidelines and criteria offered by Cobb et al.13 In 2009, Wood and Friedel22 advanced the science by specifying a framework for peer review of online learning and teaching. The concepts contained in the expanded framework of Wood and Friedel22 reflected the major categories that we wanted to capture in our peer review, specifically instructional design, interface design, use of media, and technical aspects of the course. Therefore, we elected to use their framework to guide development of our online PE form. Two additional reports informed design of the process we used for our peer review.
Designing the Process The 1st report that informed design of the process used for our peer review was a pilot study that compared 2 peer reviews of online course rating scales available in the public domain23 from Quality Mattersi (QM)24 and the University of South Florida, College of Public Health (COPH).25 The QM scale contained 40 standards addressing course overview and introduction, learning objectives, assessment and measurement, resources and materials, learning interaction, course technology, learner support, and compliance with the Americans with Disabilities Act. The COPH scale contained 53 standards addressing (a) communication; (b) course delivery, organization, and design; and (c) instructional elements. Although the reviewers reported a preference for the QM because the tool was user-friendly and possessed content validity,23 for our purposes using the tool posed a barrier for implementation because of subscription cost and the need for specialized training to use the tool (Quality Mattersi, https://www.qualitymatters.org). We modeled the new form on the PE form currently used at our school for classroom observation to maintain a similar format, structure, and rating scale in order to standardize our school’s approach to PE. Little23 reported the use of an independent, 3rd-party peer-review team, composed of a nurse educator and an instruction designer, to evaluate the use both the QM and COPH rating scales in 2 online courses offered in an RN-BSN program. After reviewing the courses independently, the reviewers compared and discussed their individual ratings for each course using both tools. Rather than conducting an independent, 3rd-party review, we used our form to evaluate each other’s online courses, both of which were offered in
an RN-BSN program. We completed our reviews independently and then met to discuss our ratings. The 2nd report that informed design of our process described how an interdisciplinary team of health and education professionals used an action learning approach to PE of online courses.26 This approach entailed mutual development and support of each other’s courses across disciplines. Although their disciplines differed, team members shared the experience of teaching in a virtual online environment. They exchanged feedback on selected modules via e-mail and during face-to-face meetings. Similar to the interdisciplinary team, both of us had extensive experience in online and distance teaching. Rather than limiting ourselves to selected modules, we offered feedback via e-mail and during face-to-face meetings on the entire course. Despite the fact that the team members valued having an outside-the-discipline perspective of their courses,26 during initial development of the form we chose to evaluate faculty teaching and courses within our discipline. The newly developed form has been shared with other units in our university, and we anticipate future evaluations may be interdisciplinary.
Peer Evaluation Form The newly developed PE of online teaching form is a 40-item, 4-point Likert-type checklist with 1 overall rating item and 4 narrative response items (see Document, Supplemental Content 1, to view the tool, http://links.lww.com/NE/A114). Our review of literature was used to isolate potential prompts for our form. We revised statements and questions into prompts before grouping them under 4 major categories and 5 minor subcategories consistent with the Wood and Friedel22 framework (Table). Directions indicate that peer reviewers are to rate each prompt as truly exemplary, done well, needs improvement, or not applicable. Four open-ended items offer the opportunity for narrative responses by elaborating on strengths, highlighting areas for improvement, and providing additional comments. The form is not scored; rather, the prompts, ratings, summary item, and narrative comments are used to guide didactic discussion.
Validity Three faculty colleagues reviewed the new PE form for face and content validity. The reviewers used a standardized form
Table. Categories and Prompts in Peer Evaluation Form Category Instructional design
Interface design Use of media Technical aspects
Nurse Educator
Prompt Topics Clarity of expectations Building student knowledge Learning activities Learner assessment and evaluation Human interaction Ease of navigating and appearance of the site Appropriate, creative, and engaging use of technology with examples of videos, chats, podcasts, wikis, and discussion forums Technical support and accommodations for students with special needs
No. of Items 5 5 5 5 8 8 3
Prompts Prompts Prompts Prompts Prompts Prompts Prompts
3 Prompts
Volume 39 & Number 1 & January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
23
developed by Kintner27 for evaluating face and content validity to ensure consistent feedback. All 3 reviewers were doctorally prepared school of nursing administrators and recognized as expert teachers. Two reviewers were certified nurse educators who facilitated graduate level education/ teaching courses for nursing students and had experience teaching an online course. All of the reviewers reported that the form appeared sound and relevant. The purpose was deemed logically consistent with the aims. Although considered unnecessary for faculty in our school, the directions were judged to be clear and appropriate. The statements were found to be arranged in a logical order, grammatically correct, and appropriate for the response rating. Response ratings were judged to be appropriate for the instrument. Content was determined to be logically consistent and highly relevant with the instrument’s theoretical underpinnings. All 3 reviewers indicated that one of the human interaction prompts conveyed more than 1 thought. Two reviewers offered suggestions for revising another 1 of the human interaction prompts for clarity in meaning. Individual reviewers directed attention to 3 additional prompts considered to contain redundant or excess wording. The form was revised based on the internal reviewers’ comments and recommendations.
guidelines, and principles for good practice in online education. We found the form useful for formative assessment, to guide course revision and advance teaching, and to be a more objective assessment of online teaching compared with students’ subjective perceptions.
Conclusion The development of the new form and the review process exceeded the spirit of PE by generating lively dialogue about online teaching and learning and fostered collaborative relationships among faculty. We acknowledge the value of a national entity such as QM to continue to develop and expand the variety of options of standardized internal and external peer review, compared with the time and effort in developing a form that met our institutional standards without additional cost or specialized training. We shared this newly developed form with faculty teaching online courses in our School of Nursing, instructional design faculty in the College of Communication, and expert teachers supporting faculty development through the Center for Teaching and Learning at our university. We are offering the form to other schools and disciplines to also use the form and provide us with feedback and suggestions at peerreview. [email protected].
References Utility We adapted the action learning approach to conduct our PEs. Both of us granted each other instructor-level access to our respective online courses after the course ended to protect the privacy of students enrolled in both courses (eg, names and comments posted to a discussion forum) because some students were enrolled in both courses. The PE of online learning form was used as designed. Ratings and responses to the prompts naturally led to an overall summary and narrative comments that informed the basis for our face-to-face discussion. We provided written and verbal feedback via e-mail and in person. Face-to-face feedback allowed for sharing, validating, clarifying, and justifying perceptions of our teaching and learning activities. Although ratings for both of the peer reviews indicated the courses were well designed and facilitated, the form was valuable for guiding a thought-provoking discussion about teaching and learning in an online format. Teaching skills were advanced by exploring alternative methods for delivery of content and evaluation of student learning. Mutual synergy generated excitement about teaching and allowed for collaborative learning. By evaluating one another, we engaged in introspective reflection of our own methods and approaches to teaching and learning.
Feasibility and Benefits of the Form Faculty in our school initiate the peer-review process. We can request peer review from faculty within our school or from other units on campus to serve as our reviewer(s). Faculty teaching in online courses will now have a more applicable form that may be used by either faculty or instructional designers to provide feedback. The time investment required for completing the review was consistent with classroom observation reviews. Content reflected existing frameworks, 24
Volume 39 & Number 1 & January/February 2014
1. Phillips V. Virtual classrooms, real education. Nation’s Business. 1998;86(5):41. 2. Culatta R. Innovative Learning: Online Learning Timeline. 2011. Available at http://www.innovativelearning.com/online_learning/timeline.html. Accessed July 30, 2013. 3. Carlton KH, Ryan ME, Siktberg LL. Designing courses for the Internet: a conceptual approach. Nurse Educ. 1998;23(3):45-50. 4. Harris DA. Online distance education in the United States. IEEE Commun Mag. 1999;37(3):87-91. 5. Little BB. Quality assurance for online nursing courses. J Nurs Educ. 2009;48(7):381-387. 6. The Institute for Higher Education Policy. Quality on the line: benchmarks for success in Internet-based distance education. 2000. Available at http://www.ihep.org/assets/files/publications/ m-r/QualityOnTheLine.pdf. Accessed July 30, 2013. 7. Bailie JL. Effective online instructional competencies as perceived by online university faculty and students: a sequel study. MERLOT J Online Learn Teach. 2011;7(1):82-89. 8. Bangert AW. The seven principles of effective teaching: a framework for designing, delivering, and evaluating an Internet-based assessment course for nurse educators. Nurse Educ. 2005;30(5): 221-225. 9. Bangert AW. The development and validation of the Student Evaluation of Online Teaching Effectiveness. Comput Sch. 2008; 25(1-2):25-47. 10. Billings DM. A framework for assessing outcomes and practices in Web-based courses in nursing. J Nurs Educ. 2000;39(2):60-67. 11. Wagner RJ, Vanevenhoven JP, Bronson JW. A top ten list for successful online courses. MERLOT J Online Learn Teach. 2010; 6(2):542-545. 12. Billings DM, Connors HR, Skiba DJ. Benchmarking best practices in Web-based nursing courses. Adv Nurs Sci. 2001;23(3): 41-52. 13. Cobb KL, Billings DM, Mays RM, Canty-Mitchell J. Peer review of teaching in Web-based courses in nursing. Nurse Educ. 2001; 26(6):274-279. 14. Chickering AW, Gamson ZF. Seven principles for good practice in undergraduate education. AAHE Bull. 1987;39(7):3-7. Nurse Educator
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
15. Kingsborough Community College. Protocol for Peer Observation of an Online Course. March 2010. Available at https:// www.kbcc.cuny.edu/college_council/Documents/AttachmentA310 .pdf. Accessed July 30, 2013. 16. Taylor AH. A peer review guide for online courses at Penn State. 2010. Available at https://www.e-education.psu.edu/facdev/ peerreview. Accessed July 30, 2013. 17. Brinthaupt TM, Fisher LS, Garner JG, Raffo DM, Woodward JB. What the best online teachers should do. MERLOT J Online Learn Teach. 2011;7(4):515-524. 18. Brinthaupt TM, Pilati ML, King BR. Psychology teaching resources in the MERLOT Digital Learning Objects catalog. J Instr Psychol. 2008;35(3):240-245. 19. Multimedia Educational Resource for Learning and Online Teaching (MERLOT). Evaluation Criteria for Peer Reviews. 2010, 2013. Available at http://taste.merlot.org/evaluationcriteria .html. Accessed July 30, 2013. 20. Odom SE, Barnes K, Wicker M. Incorporating pedagogical principles in developing an online nursing research course. Comput Inform Nurs. 2005;23(3):146-152. 21. Zhu E, McKnight R, Edwards N. Faculty Development and
Nurse Educator
Support—Principles of Online Design: Evaluation of Online Course. 2009. Available at http://www.fgcu.edu/onlinedesign. Accessed July 30, 2013. 22. Wood D, Friedel M. Peer review of online learning and teaching: harnessing collective intelligence to address emerging challenges. Australas J Educ Technol. 2009;25(1):60-79. 23. Little BB. The use of standards for peer review of online nursing courses: a pilot study. J Nurs Educ. 2009;48(7):411-415. 24. Quality Mattersi. Peer course review rubric FY05/06. 2005. Available at https://www.qualitymatters.org/. Accessed March 17, 2009. 25. University of South Florida. COPH online course standards. 2006. Available at http://itt.usf.edu/publichealth/standards/standards/ std_012606/COPH_Online_Course_Standards_v5.doc. Accessed April 1, 2007. 26. Bennett S, Santy J. A window on our teaching practice: enhancing individual online teaching quality through online peer observation and support, A UK case study. Nurse Educ Pract. 2009;9:403-406. 27. Kintner EK. Testing of the School-Aged Child and Adolescent Acceptance of Asthma Model (PhD) [dissertation]. Tucson, AZ: The University of Arizona; 1996.
Volume 39 & Number 1 & January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
25
