LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Nurs Admin Q Vol. 38, No. 4, pp. E11–E22 c 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Copyright
Video Analysis Tool System Implementation and Evaluation of Use With Clinical Nursing Assessments of Older Adults Marlene M. Rosenkoetter, PhD, RN, FAAN; Deborah Smith, DNP, RN; Max E. Stachura, MD; JoEllen McDonough, PhD, RN; Carol Hunter, RN, MSN; Darrell Thompson, RN, DNP; Sally Richter, MSN, RN; Gail Jones, DNP, RN The purpose of this project was to implement and evaluate the Video Analysis Tool (VAT) system, a tool for capturing and analyzing video evidence of students’ clinical performance. Through the VAT system, nursing student dyads from 4 universities used a video camera, a computer, and a tripod in the residences of older adults to record interactions and psychosocial assessments of older adult clients. Using their recordings to compare their clinical activities with predefined clinical objectives derived from gerontological nursing standards, they made video clips of their assessments to demonstrate the required outcomes. Use of the VAT system received positive evaluations from students, faculty, and residents in multiple clinical sites. The process has significant implications for assessing clients and health care providers in their interactions in a variety of settings, including on home visits. It has additional applications for documenting performance measures of nurses and team members as they provide client care. Key words: assessment in practice, assessment of older adults, video recordings
A
DVANCES in information technology have changed the entire pedagogy of nursing, including how students are taught,
Author Affiliations: Department of Biobehavioral Nursing, College of Nursing (Drs Rosenkoetter, Smith, and McDonough and Ms Hunter) and Medical College of Georgia (Dr Stachura), Georgia Regents University, Augusta; Department of Nursing, School of Health Sciences, Middle Georgia State College, Macon (Dr Thompson); and College of Education & Health Professions, School of Nursing, Columbus State University, Columbus, Georgia (Ms Richter and Dr Jones). This study is supported by a Health Resources and Services Administration award (#D11HP22191). The authors declare no conflict of interest. Correspondence: Marlene M. Rosenkoetter, PhD, RN, FAAN, College of Nursing, Georgia Health Sciences University, 987 St. Sebastian Way, Augusta, GA 30912 ([email protected]). DOI: 10.1097/NAQ.0000000000000047
how they learn, even how they expect to learn, and ultimately how nurses practice. From cyber activity to the nuances of social media, access to electronic information has transformed and reconstructed the educational platform, requiring educators to find new ways to manage the numerous factors that impact and even challenge the teaching role. These new responsibilities are complicated by the limitations of qualified faculty and budgetary constraints that schools of nursing are experiencing. Financial restrictions are placed on clinical travel, whereas faculty-to-student ratios are often imposed by accrediting agencies. New electronic innovations must be incorporated into didactic activities to prepare nurses for future practice, yet they cannot be so intrusive or eccentric that they divert attention from the fundamental purposes of E11
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E12
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
nursing education and the nurse-patient relationship. We are compelled to prepare nurses who can skillfully and artfully care for patients with the support of, and sometimes despite, technology changes. With this myriad of issues in mind, our university nurse educator team sought a new approach to clinical teaching and practice. We implemented a method that permits faculty to monitor effectively the activities of multiple undergraduate students practicing in community geriatric nursing clinical sites by reviewing computer-assisted video-recorded clips rather than by being physically present. Using the Video Analysis Tool (VAT), students video record their interactions with residents, transfer them to a laptop, upload them to a secure server, and complete a self-assessment form. Faculty directly access the online recordings, evaluate the interactions, make comments, and provide immediate, online feedback. While this is an approach to teaching, it has broad based implications for practice.
THEORETICAL FRAMEWORK The theoretical framework (Figure 1) of this project is based on Sociotechnical Systems Theory (STST) incorporating technical aspects of the VAT and the social subsystem: personal qualities of students, faculty, and residents reflected in the Life Patterns Model (LPM). The STST1,2 originated at the Tavistock Institute in London in the late 1950s,
based on open systems theory. It implies that effective learning occurs when there is joint optimization of the technological (STST) and sociocultural (LPM) systems. Learning becomes technology-facilitated and indicator-driven, with the purpose of enhancing quality nursing education and patient care while providing personal meaning to students.3 The STST and the LPM have been used effectively in nursing management research.4-7 Here, they are applied to nursing education. Sociotechnical systems “create the organizational context for knowledge sharing, learning and innovation,”8 based on collaborative decision making, and suggesting that each of the parties involved—students, faculty, and participant residents—should have collaborative input and involvement in evaluating the impact of the technological systems. “Information, awareness, and the ability to use information are key features of knowledge.”9 Technology used for learning should therefore meet both the needs and expectations of the user, the user’s expectations while facilitating ease of use,10 and mitigating any associated risks.11 A previous research application of the STST and the LPM assessed sociotechnical system changes in a study of the perceptions of nurses (n = 1056) regarding implementation of a new intravenous infusion pump. Changes occurred in each of the life patterns. Validity and reliability of the instrument were established and a Cronbach α of 0.756 was
Figure 1. Theoretical framework.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System obtained.12,13 The LPM was also blended into a study comparing the technological impact of insulin infusion pumps and multiple daily insulin injections on the psychosocial patterns of patients and significant others14 (α = .900.950). Technology in the current project focuses on video recordings, computer systems, and equipment. These comprise the devices, processes, skills, and knowledge that are used by students to accomplish the task of using the VAT to facilitate nursing assessments. The social system formed by the students, faculty, and residents in the community is influenced by the impact of technology, in this case the VAT, on their life patterns. Joint optimization suggests that sociocultural, personal, and technical dimensions are interdependent, and outcomes, such as performance and learning, can be optimized only by jointly considering the 2 dimensions. This model reveals the duality of technology—it both enables and limits human action, whereas social interaction develops and changes technology. LIFE PATTERNS MODEL The LPM includes roles, relationships, support groups, self-esteem, use of time, and life structure12,13-15 and frames the questions of the social or personal factors of the Geriatric Assessment Tool and the 3 survey instruments used in the study, while incorporating the interaction effect of the technical (VAT) system. The LPM stems from an open systems framework whereby changes in each life patterns cause changes in each of the others. The purpose of the study was to determine whether the VAT system was a feasible approach to the teaching of clinical nursing in an undergraduate program and prepare nurses for the technology of practice. The primary research questions were as follows: (1) What is the impact of the VAT system on the life patterns of faculty, students, and residents? and (2) What is the impact of the VAT system on the teaching of geriatric nursing?
E13
VIDEO ANALYSIS TOOL Development of the VAT began in 2002 in the Learning and Performance Support Laboratory at the University of Georgia during investigations of technology integration into teacher preparation programs. A clearly identified need to assess situated instructional strategies prompted intense development of ways to capture evidence of both teaching and learning. The new approach evolved into a technology that allowed event codification, both live and postevent.16 During the past 10 years, the VAT has been implemented in more than 16 school systems, 20 universities, 2 mental health clinics, and a hospital. More than 5000 users have uploaded more than 7000 videos and created more than 22 000 clips. The system has been put through many rounds of usability testing and rebuilt. Each iteration improved functionality and ease of use. Rich and Hannafin17 provide a comprehensive review of the literature on video annotation tools, how VAT fits into that space, and the advances VAT makes for studies of teacher preparation. Bryan and Recesso18 wrote about reflection and perturbation of beliefs as a result of seeing one’s own practices on video. Concrete evidence helps one see what really happened as opposed to what one thought or recollected during complex events such as patient care encounters. The VAT captures events as they happen, helps faculty codify meaningful information about what took place, and then repurposes the event as an object that helps students learn. Beginning students can use the VAT to see basic attributes that build over time into effective practices. Using the “lenses,” a metaphor to describe the assessment instruments embedded in the VAT, one is able to see finegrained attributes inside the complexity of teaching and learning and suppress the extraneous noise of competing interests (practices or events that catch one’s attention but are not the education or assessment focal points).16 Processes can focus on self-assessment, reflection, supervision, and evaluation. This affords
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E14
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
the user a more systematic way to select and interpret evidence to inform decisions in highstakes environments,19 such as clinical nursing practice. OVERALL ARCHITECTURE AND FUNCTIONALITY OF eAGER The enhanced Assessment of Geriatric Residents (eAGER) created for this project combines the VAT system with a newly created Geriatric Assessment Tool to evaluate student performance. Equipment used to interface students, faculty, and clinical subjects with the VAT system includes a highcapacity laptop, a high-resolution, lightweight video camera with zoom capability and a tripod, and a HIPAA-compliant, high-speed connection to the Internet. Live and recorded demonstrations with hands-on training on camera/tripod use, positioning, video download to the laptop, and upload to VAT Web site were conducted with all faculty members and students. Students and faculty received 30 to 40 minutes of step-by-step demonstrations and online tutorials on effective data use prior to participant-resident interactions. Students, faculty, and participating residents interact with eAGER through a Web interface in a common Web browser. Students visit a resident and capture their interaction with the resident using a video camera. They use downloaded free video compression software and upload their recordings directly from the laptop to the VAT server. In VAT, students then conduct a self-assessment using the school’s assessment frameworks that were preloaded into the system. The framework consists of 3 tenants: effective communication, application of didactic content in clinical settings, and integration of technology considerations. Specific learning outcomes for students include an assessment of how they engage clients, their nonverbal communication, attentiveness and their ability to demonstrate course objectives related to communication approaches, application of didactic content in clinical settings,
and integration of technology. The focus of the clinical experience is on psychosocial assessments of well elderly community-residing residents. Not only are students able to share every resident interaction with their faculty and receive constructive feedback but also faculty can view every student’s interaction and provide feedback frequently, including during conferences at the clinical site if faculty are actually present. Previously, faculty were limited to evaluating directly only the number of student dyads they could accompany for resident visits. Only retrospective feedback was provided for the remaining studentresident interactions. Now, faculty can access the VAT from any computer connected to the Internet, access the students’ video(s), and conduct an assessment. Using the clips created by students, faculty evaluate the quality of student-resident interactions (Figure 2) and provide precise and detailed feedback concerning observations associated with specific times (eg, 1:05.00 PM to 1:15.00 PM) within the total recorded video session. Such feedback is critical to guiding students’ communication and assessment skills with older adults. Students then enter the system to access faculty feedback and communicate further (in person or asynchronously) with the faculty about successes or areas that require improvement. Faculty evaluate the student’s practice performance and justify their assessment with online comments to students. The VAT Faculty Function includes all system capabilities available to students as well as the ability to upload and evaluate additional items of evidence (eg, clinical samples, videotapes) and create clips from the students’ initial videos. Faculty can recommend that selected students contribute their clips as evidence of high-quality practices so others can view examples that meet a particular standard. Faculty decided that the use of the VAT system was a course requirement but that during the first semester of the initial trial it would be a pilot project and would have no impact on any student’s grade. Subsequently, grades were awarded. Benchmark performance standards against which students ranked their performances
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System
E15
Figure 2. Sample online screenshot.
for practice were based on guidelines that included Evidence-Based Geriatric Nursing Protocols for Best Practice,20 Assessment of Function,21 Caring for an Aging America: A Guide for Nursing Faculty,22 The Essentials of Baccalaureate Education for Professional Nursing Practice,23 and Recommended Baccalaureate Competencies and Curricular Guidelines for the Nursing Care of Older Adults,24 which outline competencies, content, and teaching strategies for geriatric nurse education. The eAGER tool uses feedback and data entered by faculty to generate and display an assessment of the student’s level of performance. Faculty can browse this assessment and see clearly the student’s level of performance for each standard. Faculty and students host threaded discussions about the assessment and feedback. Once the assessment is completed, the Faculty Function of eAGER generates 2 draft reports in Microsoft Word format—the Assessment Report and the Faculty Report. AVAILABLE ENHANCEMENTS TO THE VAT The eAGER-enhanced version of the VAT can include additional support modules as
part of the final architecture: Evidence Uploader (upload of video, documents, assessment images, and clinical data); Assessment View (by self and faculty) providing functions for aligning evidence to standards, associating comments, and aligning student practices demonstrated with video examples of practice provided by the College of Nursing; Electronic Portfolio with which students transfer evidence and data (eg, feedback, assignments, and assessments) to demonstrate growth along their nursing preparation continuum; Knowledge Base, where faculty can either upload evidence themselves (eg, from existing video libraries) or recommend a nursing student’s practices be included in this digital library and be recognized for excellence in demonstrating a standard; and Report Generator, which provides functions for students to export their portfolio. Faculty can analyze and export data accessible in text or data (csv for Excel) file formats. eAGER TRAINING OF STUDENTS AND FACULTY In this project, all faculty members and students completed 2 training sessions under a contract with the VAT company. Representatives from the clinical sites were
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E16
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
invited to attend. Emphasis was on the technology integrated into the processes of supervision, teaching, and learning. Everyone studied evidence capture (eg, video cameras) and uploading into the eAGER through a Web interface. Students reviewed the standards and expectations of their respective programs, learned about the most appropriate forms of evidence to capture, and considered the best techniques for capturing their practices (eg, how best to position a camera to capture their practice without hindering resident’s support) as well as how to complete the selfassessment. Students discussed how to support peer collaboration and what they would experience when interacting with classmates, residents, and faculty. Laptops were password protected, and students were instructed on restricting access only to themselves or risk severe penalties. Only minimal software, including Firefox and compression software, was placed on the computers and use was restricted to the clinical project. Students and faculty completed a course on HIPAA compliance in advance of this course. Faculty attended an additional session on the process for evaluating student video performance segments using the assessment tool embedded in the VAT. Sessions for faculty provided extra time to become familiar with the system and how it integrated with the assessment, supervision, and teaching of students. They learned the various forms of evidence that embody practices and how they align with the standards and expected outcomes of the nursing programs. Both the nursing students and faculty received “refresher” training sessions throughout the implementation period as needed. All users had immediate online access to technical support by e-mail and the company Web site to resolve technical issues. Training sessions were recorded in a lecture capture program for students and faculty to review as needed. HUMAN SUBJECTS APPROVAL All facets of the project are HIPAA compliant, including the VAT system processes, servers, backup, and devices on which all data
and videos are stored. The project was approved by the institutional review board (IRB) of each participating university and the appropriate housing authorities with oversight responsibility for the geriatric residential facilities. While this project is not classified as research, it involves the collection of data from human subjects and thus falls under the auspices of the review boards of each university. Consent letters explained to candidate participants in detail the relevant policies and procedures, the purpose of the project, and thus their rights as human subjects, and then solicited their informed participation. Only residents who could read and speak English, had the potential of understanding the consent forms as well as the use and purpose of the video camera, and the survey instrument were included in the project. Residents were oriented to the use of the VAT system by faculty and given the opportunity to accept or decline participation. Residents were consented prior to the beginning of actual clinical activities by the principal investigator and/or designated faculty or staff. All faculty and staff members were eIRB certified. At the beginning of each clinical day, participating residents were given an additional opportunity to decline. Anyone who declined was exempted from participation. If residents wished to exclude a portion of their home setting from the video, the camera’s field of view was adjusted accordingly. Residents were asked to complete an anonymous survey concerning their experience with VAT-enhanced training and were provided a return process. PARTICIPANTS AND CLINICAL SITES One of the initial schools was deleted at the end of the first year because of teaching assignment conflicts, and a fourth school was added. As a result, the faculty also changed. Each of the participating schools of nursing offer courses focusing on geriatric adults in residential communities or homes. To date, there have been more than 200 students at 4 universities and multiple faculty members participating at 7 retirement community
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System residential sites during the 3 years of the project. Each residential site had independent living facilities for “well elderly” residents and ranged in size from approximately 60 to more than 350 residents. While the sites differed somewhat, the participants included lower-, middle-, and upper-income adults aged 60 years and older. Faculty were primarily doctorally prepared and most had more than 20 years of teaching experience, including considerable experience with computer technology and distance education. VIDEO CAMERA SETUP AND IMPLEMENTATION Students took the assigned, lightweight, battery-powered video camera and a small tripod with them for clinical visits with residents. The video cameras were set up quickly and began recording to an internal hard drive with the push of a button. The wide-angle lenses of the video camera enabled capture of most of the room. The lenses, very sensitive for low-light conditions, did not require additional lighting. The highly sensitive builtin microphones did not require other people or equipment because the video cameras could be preconfigured to accommodate soft-spoken residents. During the training, students learned the visual cues for knowing if the camera was on and recording properly and how to approach and depart from residents’ homes. Students learned to position the camera to capture the student-resident interaction to the greatest extent possible. A detailed handout covering the use of the video cameras, associated ethical considerations, and opting-out issues for residents was provided. DEVELOPMENT OF SURVEY TOOLS AND PRE-/POSTTEST The first section of each survey includes questions on demographic characteristics. The second section focuses on questions on a 5-point Likert scale (strongly disagree to strongly agree) based on melding STST and
E17
the LPM. Survey instruments assessed each participant’s perceptions of the impact of the VAT system on the following: (1) roles, (2) relationships with others, (3) self-esteem, (4) use of time (5) time structure, and (6) support groups. The survey of residents focuses on perceptions of the use of video cameras in residential settings. Because of the ages of residents, these statements were kept simple and to a minimum. Large print was available. Prior input was sought from the employed staff in the residential communities to identify potential participants. Content validity was established by expert faculty. Reliability of the survey instruments is being assessed over time using the responses of multiple nursing classes at all participating sites. A pre-/posttest based on the geriatric nursing content of the course was developed and administered to all students at the beginning and again at the end of the geriatric nursing courses. RESULTS Research question 1 On the basis of the results from faculty surveys and ongoing discussions among participating faculty throughout the project length, the faculty endorsed the use of the system and noted that it increased their abilities to monitor students and provide positive feedback while facilitating student learning and individual assessments. Because of the small number of faculty involved, discussions were found to be more useful than the survey instruments. Faculty were able to view the students and residents during interactions, observe the home environment, and make pertinent comments to students. At the outset, there were problems uploading video recordings, accessing the system, and time lags between uploads and completion. Company officials were receptive to requests for assistance and by the end of the pilot semester, these problems had been resolved. A 2-hour upload process was decreased to 10 minutes with the use of free, downloadable video compression software. An initial inability to locate
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E18
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
uploaded videos in the VAT system was resolved. Initial challenges with faculty usage included accessing the secure VAT system (resolved by better scheduling of routine company system maintenance) and the multiple steps required to select and apply the appropriate evaluation tool for student evaluation. Company administrators hid listings of some tools, and the geriatric assessment evaluation tool was automated. Data continue to be collected, but of the students reported here (Table 1), most were 21 to 30 years old (n = 66), 14 had a bachelor’s degree in another discipline, 58.8% (n = 47) had no experience with the VAT, and 81.3% (n = 65) were female. Responses on the student survey provided important insights into changes in life patterns and the use of this technology. The majority (n = 47; 58.8%) reported that the VAT system documented their knowledge, skills, and interactions and helped with discussions with faculty (n = 42; 52.6%). For 56.2% (n = 45), the VAT did not interfere with their assessments, and for 43.8% (n = 35), it facilitated learning. Yet, 49.3% (n = 37) felt that they could have used their time better. Comments from students, faculty, and residents were also elicited in informal debriefing sessions and incorporated into revisions of the use of the VAT system. Students suggested that even more emphasis needed to be placed on updating passwords from previous students to facilitate their time use. The issues with password failures were addressed and resolved with the support of technology staff. Using the variables of camera experience, education, race, VAT experience, gender, and 12 Likert scale items, the Cronbach α coefficient for the student survey is 0.780. An even newer version of the student survey has been now been developed. Completion of surveys by residents decreased considerably over the 3 semesters of the project, primarily because many were repeat participants and voiced no interest in completing the surveys again. Hence, only first-semester participants are reported here (Table 2), along with comments made during faculty-resident interactions. From the
surveys, more than 77% liked the idea of the videotapes and no residents reported that it wasted their time or interfered with their other plans for their day. Students made appointments in advance to facilitate timeliness with resident visits. Only a couple of residents are known to have refused to participate, and, in fact, there was a “waiting list” at each of the sites for residents who wanted to participate but could not be included with that semester’s cohort. This was surprising due to the mid80s ages of the geriatric residents. Concern was expressed by faculty that some residents seem to “dress up” for the video recording and perhaps gave responses that they thought students wanted to hear rather than being candid. These observations gave faculty an opportunity to work with students on communication techniques to move their interactions beyond the superficial level and elicit more meaningful information. One resident stated that she did not mind being video recorded but stated: “Just don’t make me watch it.” Research question 2 Analyses of the pre- and posttests of geriatric nursing questions showed no overall significant difference; however, there was a significant difference for 3 of the 10 questions, namely, for nutrition and medication knowledge, and types of stress. No significant differences were found across schools. Not all students completed the posttest and while they were encouraged to take the 2 tests seriously, some may not have done so because no grade was associated. Faculty reassessed the test content for the level of knowledge, and a new version is forthcoming. More data will be collected over the next year. Because all students enrolled in the course were required to use the videotaping system and the pre-/posttests had not been used in previous years, comparisons with other students were not made. A grade of satisfactory is required in the course for progression in the program, and unsatisfactory grades are extremely rare due to the competitiveness and high admission
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
3.8 5.0 10.0 17.5 50.0 11.3 22.4 2.6 6.6 7.5 12.5 10.0
6.3
3 4 8 14 40 9 17 2 5 6 10 8
5
20
13
24
32
18 8 40
32
25.0
16.3
30.0
40.0
23.7 10.6 52.6
40.0
38.8
45.0
40.0
51.2
36.3
25.0 37.5
%
13
11
17
19
26 28 12
15
3
20
16
16
11
20 19
n
U
16.3
13.8
21.3
23.8
34.2 37.3 15.8
18.8
3.8
25.0
20.0
20.0
13.8
25.0 23.8
%
33
35
23
17
11 24 13
18
2
6
17
16
29
30 21
n
A
41.3
43.8
28.7
21.3
14.5 32.0 17.1
22.5
2.5
7.5
21.3
20.0
36.3
37.5 26.3
%
9
12
5
6
4 13 6
4
4
3
5
2
8
5 7
n
SA
11.3
15.0
6.3
7.5
5.3 17.3 7.9
5.0
5.0
3.8
6.3
2.5
10.0
6.3 8.8
%
September 4, 2014
31
36
32
41
29
20 30
n
D
NAQ-D-14-00013
Video Analysis Tool System
Abbreviation: VAT, Video Analysis Tool. a Omitted responses were deleted from calculations. SD, strongly disagree; D, disagree; U, undecided; A, agree; SA, strongly agree.
5.0 1.3
4 1
1. The VAT facilitated my learning. 2. The VAT interfered with my interactions with the resident. 3. I felt less comfortable in my role as a student in the presence of the camera. 4. The VAT interfered with my assessment of the resident. 5. The VAT facilitated my relationship with my resident. 6. The VAT interfered with my relationship with my instructor. 7. The resident refused to permit me to use the VAT during our visit. 8. I feel less confident when knowing my instructor is watching me. 9. I saved time using the VAT. 10. I could have used my time better without the VAT. 11. The VAT system takes time away from my clinical visit experience. 12. The VAT system enhances my experience with my resident. 13. The VAT system helps in my discussions with other students during debriefing sessions. 14. The VAT system effectively documents my knowledge, skills, and interactions during my clinical visits. 15. The VAT system helps with my discussions with my instructor during debriefing sessions.
%
n
SD
Statement
Table 1. Sample Student Survey Responsesa (N = 80)
LWW/NAQ 3:57
E19
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
NAQ-D-14-00013
%
23.5
7.8 18.8
3.9
12
4 9
2
standards for the programs. More data are being collected on the new pre-/posttest, but findings indicate the VAT system is an effective approach to teaching clinical geriatric nursing with implications for practice.
15.7
41.2 37.5
53.9
8 33.3
21 18 21.6 12.5 12.2
27 15.7
17 39.2
11 6 6 23.5 22.9 40.8 54.0
8 5.8
20 7.8 4
12 11 20 27 5.9 8.3 46.9 46.0 3 4 23 23
3 1.9 1
1. I like the idea of being videotaped when the student was with me. (n = 51) 2. The videotape helped me talk to the student.(n = 51) 3. The videotape made me feel important. (n = 48) 4. The videotape wasted my time. (n = 49) 5. The videotape interfered with other things that I had planned to do when the students were there. (n = 50) 6. I would like the videotape to be shared with my doctor. (n = 51)
% n Statement
n
D
%
n
U
%
n
A
%
DISCUSSION
a SD,
strongly disagree; D, disagree; U, undecided; A, agree; SA, strongly agree.
SD
Table 2. Resident Survey Responsesa First Semester (N = 51)
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
SA
E20
September 4, 2014
n
LWW/NAQ
If a person changes roles, it is predictable that there will be changes in relationships, use of time, and life structure. With changes in the use of time, there will likely be changes in roles that could ultimately impact self-esteem. Use of the VAT system changes student and faculty roles by incorporating technology and does so in an environment that allows faculty access for direct observation, supervision, and input into the system. Students become selfevaluators of their resident interactions as well as peer mentors. Such new mentorship roles could have a positive as well as negative impact on self-esteem when students are viewed as experts by fellow students. Faculty move from single-student dyad evaluators to evaluators of multiple dyads from distant locations. Faculty no longer have to be on-site to evaluate students accurately. Residents provide the medium and are an integral part of the evaluative effectiveness of the technological system. Students can rely on video-recorded clips to provide accurate documentation of their activities rather than “after-the-fact” reports. The VAT reduces the travel budget. While recording anxiety might be a factor, students know that evaluations are based on observed phenomena, not recollections, and any discrepancies can be mediated by another reviewer. Because students will be faced with computer technologies during both clinical experiences and upon entering the workforce after graduation, incorporating technology into their educational experience is vital. Changing the way faculty teach is also important when reaching out to the current generation of techno-savvy students, as well as for reducing financial costs. Using direct video recording for assessment and evaluation purposes in community settings affords faculty
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System the opportunity to enhance the learning of students’ knowledge and skills as they prepare for practice in community health. Through more responsible use of faculty time, budgetary constraints can be addressed while concomitantly incorporating more effective educational strategies. Through the reduced need to visit students in home health settings, faculty can avoid needless travel. Costs for the system are complex to analyze and vary with the program and the university. For example, 150 students with 15 faculty full-time equivalents (10:1 ratio), for 6 visits at $0.55 per mile at an average of 30 miles r/t = $1485.00 for travel for 1 course compared with zero travel expense using video technology. More distant clinical sites that were previously eliminated because of cost factors can now be reinstated. In these settings, video clips could capture the community, home surroundings, the inside of the dwelling, and even household pets to provide a more thorough assessment. Students were engaged and involved in the resolution of problems. Data will now be collected to determine ongoing effectiveness. In addition, graduate-level nurse practitioner students visit homes of clients in wide geographic regions, so feasibility of VAT usage is likely. LIMITATIONS AND IMPLICATIONS FOR PRACTICE The project was conducted with small convenience samples of undergraduate nursing students in a limited number of schools of nursing and limited geriatric residents in one geographic region. No attempt was yet made to use acute care facilities where video recording would have additional HIPAA, IRB, and
E21
ethical considerations. Residents were well elderly residents with no known dementia or serious disabilities. Ethernet connections were available at each geriatric retirement community; however, for some students living in rural areas, they had to wait until access to a secure connection was available. This process has excellent potential for applications in practice settings. It would be useful for public health nurses to document the status of clients as well as share knowledge with newly assigned nurses prior to a visit. In clinic settings, video recordings could be used to document client status, to share with health team members, and for client-focused conferences. In making home visits, nurses could send information to physicians and other health care providers and seek input on interventions, treatment methods, medications, and the like. The recordings could be used to teach and evaluate specific family members regarding dressing changes, insulin administration, and even how to ambulate someone with a walker, or crutches, or in need of additional assistance. By zooming in the camera, details can be observed and recorded. Client status, progress, and lack of progress can be noted. Procedures and treatments can be explained. The performance measure of nurses could be documented and evaluations completed with real-time evidence. Whether uploaded to a system, as with this project, or simply recorded and transferred to a secure site on the facility’s server, objective information can be recorded and retained. The ramifications of this approach are noteworthy in augmenting client and family care and the interface between health care providers, clients, and their families.
REFERENCES 1. Fox W. Sociotechnical system principles and guidelines: past and present. J Appl Behav Sci. 1995;31(1):91-105. 2. Churchman C, Verhulst M, eds. Management Sciences, Models and Technique. London, England: Pergamon; 1960.
3. Harteloh P. Quality systems in health care: a sociotechnical approach. Health Policy. 2003; 64(3):391-398. 4. Li J. A Sociotechnical approach to evaluating the impact of ICT on clinical care environments. Open Med Inform J. 2010;4:202-205.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
E22
5. 6. 7.
8.
9.
10.
11.
12.
13.
14.
15.
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
doi:10.2174/1874431101004010202. http://www .ncbi.nlm.nih.gov/pmc/articles/PMC3096882. Accessed July 7, 2013. Carayon P. Human factors of complex sociotechnical systems. Appl Ergon. 2006;37(4):525-535. Brooks B, Anderson M. Defining quality of nursing work life. Nurs Econ. 2005;23(6):319-326. Happ M. Sociotechnical systems theory. Analysis and application for nursing administration. J Nurs Adm. 1993;23(6):47-54. Aldridge J. Sociotechnical systems theory. http:// www.argospress.com/Resources/team-building/ socitechnisistem.htm. Published 2004. Accessed July 7, 2013. Mlitwa N. Technology for teaching and learning in higher education contexts: activity theory and actor network theory analytical perspective. Int J Educ Dev Using ICT. 2007;3(4). http://ijedict.dec .uwi.edu/printarticle.php?id=420&layout=html. Accessed July 7, 2013. Bjoko A. Using eye tracking to compare Web page designs: a case study. J Usability Stud. 2006;3(1):112120. Anderson S, Felici M. Emerging Technological Risk: Underpinning the Risk of Technology Innovation. New York, NY: Springer; 2012. Bowcutt M, Rosenkoetter M, Chernecky C, Wall J, Wynn D, Serrano C. Implementation of an intravenous medication infusion pump system: implications for nursing. J Nurs Manag. 2008;16: 188-197. Rosenkoetter M, Bowcutt M, Khasanshina E, Chernecky C, Wall J. Perceptions of the impact of smart pumps on nurses and nursing care provided. J Assoc Vasc Access. 2008;13(2):60-69. Rosenkoetter M, Stachura M, Astapova E, Brown D, Dias J. Age-related impact of intensive insulin therapies. Paper presented at: American Academy of Nursing Annual Meeting; October 13-15, 2011; Washington, DC. Stachura M, Rosenkoetter M, Astapova E, Dias J, Brown D, Continuous subcutaneous insulin infusion and multiple daily injections: an impact study. Paper presented at: Diabetes Technology Society
16.
17.
18.
19.
20.
21.
22.
23.
24.
Annual Meeting; October 29, 2011; San Francisco, CA. Recesso A, Hannafin M, Wang F, Deaton B, Shepherd C, Rich P. Direct evidence and the continuous evolution of teacher practice. In: Adamy P, Milman N, eds. Evaluating Electronic Portfolios in Teacher Education. Greenwich, CT: Information Age Publishing Inc; 2009:38, chap 2. Rich P, Hannafin MJ. Video annotation tools: technologies for assessing and improving preservice teachers’ instructional decision making. J Teach Educ, 2009;60(1):52-67. Bryan L, Recesso A. Promoting reflection with a Webbased video analysis tool. J Comput Teach Educ. 2006;23(1):31-39. Recesso A, Zepeda SJ. Evidential reasoning and decision support in assessment of teacher practice. In: Kowalski TJ, Lasley TJ, eds. Handbook on Databased Decision Making in Education. Hillsdale, NJ: Lawrence-Erlbaum; 2008:363-381. Capezuiti E, Zwicker D, Mezey M, Fulmer T. eds. Evidence-Based Geriatric Nursing Protocols for Best Practice. New York, NY: Springer Publishing Co; 2008. Kresevic D. Assessment of function. In: Capezuti E, Zwicker D, Mezey M, Fulmer T, eds. Evidence-Based Geriatric Nursing Protocols for Best Practice. 3rd ed. New York, NY: Springer Publishing Co; 2008:2340. Thornlow D, Latimer D, Kingsborough J, Arietti L. Caring for an Aging America: A Guide for Nursing Faculty. Washington, DC: American Association of Colleges of Nursing; 2006. American Association of College of Nursing. The Essentials of Baccalaureate Education for Professional Nursing Practice. Washington, DC: American Association of College of Nursing; 2008. American Association of College of Nursing. Recommended Baccalaureate Competencies and Curricular Guidelines for the Nursing Care of Older Adults. Washington, DC: American Association of Colleges of Nursing, and New York: New York University College of Nursing, Hartford Institute for Geriatric Nursing; 2010.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
NAQ-D-14-00013
September 4, 2014
3:57
Nurs Admin Q Vol. 38, No. 4, pp. E11–E22 c 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Copyright
Video Analysis Tool System Implementation and Evaluation of Use With Clinical Nursing Assessments of Older Adults Marlene M. Rosenkoetter, PhD, RN, FAAN; Deborah Smith, DNP, RN; Max E. Stachura, MD; JoEllen McDonough, PhD, RN; Carol Hunter, RN, MSN; Darrell Thompson, RN, DNP; Sally Richter, MSN, RN; Gail Jones, DNP, RN The purpose of this project was to implement and evaluate the Video Analysis Tool (VAT) system, a tool for capturing and analyzing video evidence of students’ clinical performance. Through the VAT system, nursing student dyads from 4 universities used a video camera, a computer, and a tripod in the residences of older adults to record interactions and psychosocial assessments of older adult clients. Using their recordings to compare their clinical activities with predefined clinical objectives derived from gerontological nursing standards, they made video clips of their assessments to demonstrate the required outcomes. Use of the VAT system received positive evaluations from students, faculty, and residents in multiple clinical sites. The process has significant implications for assessing clients and health care providers in their interactions in a variety of settings, including on home visits. It has additional applications for documenting performance measures of nurses and team members as they provide client care. Key words: assessment in practice, assessment of older adults, video recordings
A
DVANCES in information technology have changed the entire pedagogy of nursing, including how students are taught,
Author Affiliations: Department of Biobehavioral Nursing, College of Nursing (Drs Rosenkoetter, Smith, and McDonough and Ms Hunter) and Medical College of Georgia (Dr Stachura), Georgia Regents University, Augusta; Department of Nursing, School of Health Sciences, Middle Georgia State College, Macon (Dr Thompson); and College of Education & Health Professions, School of Nursing, Columbus State University, Columbus, Georgia (Ms Richter and Dr Jones). This study is supported by a Health Resources and Services Administration award (#D11HP22191). The authors declare no conflict of interest. Correspondence: Marlene M. Rosenkoetter, PhD, RN, FAAN, College of Nursing, Georgia Health Sciences University, 987 St. Sebastian Way, Augusta, GA 30912 ([email protected]). DOI: 10.1097/NAQ.0000000000000047
how they learn, even how they expect to learn, and ultimately how nurses practice. From cyber activity to the nuances of social media, access to electronic information has transformed and reconstructed the educational platform, requiring educators to find new ways to manage the numerous factors that impact and even challenge the teaching role. These new responsibilities are complicated by the limitations of qualified faculty and budgetary constraints that schools of nursing are experiencing. Financial restrictions are placed on clinical travel, whereas faculty-to-student ratios are often imposed by accrediting agencies. New electronic innovations must be incorporated into didactic activities to prepare nurses for future practice, yet they cannot be so intrusive or eccentric that they divert attention from the fundamental purposes of E11
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E12
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
nursing education and the nurse-patient relationship. We are compelled to prepare nurses who can skillfully and artfully care for patients with the support of, and sometimes despite, technology changes. With this myriad of issues in mind, our university nurse educator team sought a new approach to clinical teaching and practice. We implemented a method that permits faculty to monitor effectively the activities of multiple undergraduate students practicing in community geriatric nursing clinical sites by reviewing computer-assisted video-recorded clips rather than by being physically present. Using the Video Analysis Tool (VAT), students video record their interactions with residents, transfer them to a laptop, upload them to a secure server, and complete a self-assessment form. Faculty directly access the online recordings, evaluate the interactions, make comments, and provide immediate, online feedback. While this is an approach to teaching, it has broad based implications for practice.
THEORETICAL FRAMEWORK The theoretical framework (Figure 1) of this project is based on Sociotechnical Systems Theory (STST) incorporating technical aspects of the VAT and the social subsystem: personal qualities of students, faculty, and residents reflected in the Life Patterns Model (LPM). The STST1,2 originated at the Tavistock Institute in London in the late 1950s,
based on open systems theory. It implies that effective learning occurs when there is joint optimization of the technological (STST) and sociocultural (LPM) systems. Learning becomes technology-facilitated and indicator-driven, with the purpose of enhancing quality nursing education and patient care while providing personal meaning to students.3 The STST and the LPM have been used effectively in nursing management research.4-7 Here, they are applied to nursing education. Sociotechnical systems “create the organizational context for knowledge sharing, learning and innovation,”8 based on collaborative decision making, and suggesting that each of the parties involved—students, faculty, and participant residents—should have collaborative input and involvement in evaluating the impact of the technological systems. “Information, awareness, and the ability to use information are key features of knowledge.”9 Technology used for learning should therefore meet both the needs and expectations of the user, the user’s expectations while facilitating ease of use,10 and mitigating any associated risks.11 A previous research application of the STST and the LPM assessed sociotechnical system changes in a study of the perceptions of nurses (n = 1056) regarding implementation of a new intravenous infusion pump. Changes occurred in each of the life patterns. Validity and reliability of the instrument were established and a Cronbach α of 0.756 was
Figure 1. Theoretical framework.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System obtained.12,13 The LPM was also blended into a study comparing the technological impact of insulin infusion pumps and multiple daily insulin injections on the psychosocial patterns of patients and significant others14 (α = .900.950). Technology in the current project focuses on video recordings, computer systems, and equipment. These comprise the devices, processes, skills, and knowledge that are used by students to accomplish the task of using the VAT to facilitate nursing assessments. The social system formed by the students, faculty, and residents in the community is influenced by the impact of technology, in this case the VAT, on their life patterns. Joint optimization suggests that sociocultural, personal, and technical dimensions are interdependent, and outcomes, such as performance and learning, can be optimized only by jointly considering the 2 dimensions. This model reveals the duality of technology—it both enables and limits human action, whereas social interaction develops and changes technology. LIFE PATTERNS MODEL The LPM includes roles, relationships, support groups, self-esteem, use of time, and life structure12,13-15 and frames the questions of the social or personal factors of the Geriatric Assessment Tool and the 3 survey instruments used in the study, while incorporating the interaction effect of the technical (VAT) system. The LPM stems from an open systems framework whereby changes in each life patterns cause changes in each of the others. The purpose of the study was to determine whether the VAT system was a feasible approach to the teaching of clinical nursing in an undergraduate program and prepare nurses for the technology of practice. The primary research questions were as follows: (1) What is the impact of the VAT system on the life patterns of faculty, students, and residents? and (2) What is the impact of the VAT system on the teaching of geriatric nursing?
E13
VIDEO ANALYSIS TOOL Development of the VAT began in 2002 in the Learning and Performance Support Laboratory at the University of Georgia during investigations of technology integration into teacher preparation programs. A clearly identified need to assess situated instructional strategies prompted intense development of ways to capture evidence of both teaching and learning. The new approach evolved into a technology that allowed event codification, both live and postevent.16 During the past 10 years, the VAT has been implemented in more than 16 school systems, 20 universities, 2 mental health clinics, and a hospital. More than 5000 users have uploaded more than 7000 videos and created more than 22 000 clips. The system has been put through many rounds of usability testing and rebuilt. Each iteration improved functionality and ease of use. Rich and Hannafin17 provide a comprehensive review of the literature on video annotation tools, how VAT fits into that space, and the advances VAT makes for studies of teacher preparation. Bryan and Recesso18 wrote about reflection and perturbation of beliefs as a result of seeing one’s own practices on video. Concrete evidence helps one see what really happened as opposed to what one thought or recollected during complex events such as patient care encounters. The VAT captures events as they happen, helps faculty codify meaningful information about what took place, and then repurposes the event as an object that helps students learn. Beginning students can use the VAT to see basic attributes that build over time into effective practices. Using the “lenses,” a metaphor to describe the assessment instruments embedded in the VAT, one is able to see finegrained attributes inside the complexity of teaching and learning and suppress the extraneous noise of competing interests (practices or events that catch one’s attention but are not the education or assessment focal points).16 Processes can focus on self-assessment, reflection, supervision, and evaluation. This affords
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E14
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
the user a more systematic way to select and interpret evidence to inform decisions in highstakes environments,19 such as clinical nursing practice. OVERALL ARCHITECTURE AND FUNCTIONALITY OF eAGER The enhanced Assessment of Geriatric Residents (eAGER) created for this project combines the VAT system with a newly created Geriatric Assessment Tool to evaluate student performance. Equipment used to interface students, faculty, and clinical subjects with the VAT system includes a highcapacity laptop, a high-resolution, lightweight video camera with zoom capability and a tripod, and a HIPAA-compliant, high-speed connection to the Internet. Live and recorded demonstrations with hands-on training on camera/tripod use, positioning, video download to the laptop, and upload to VAT Web site were conducted with all faculty members and students. Students and faculty received 30 to 40 minutes of step-by-step demonstrations and online tutorials on effective data use prior to participant-resident interactions. Students, faculty, and participating residents interact with eAGER through a Web interface in a common Web browser. Students visit a resident and capture their interaction with the resident using a video camera. They use downloaded free video compression software and upload their recordings directly from the laptop to the VAT server. In VAT, students then conduct a self-assessment using the school’s assessment frameworks that were preloaded into the system. The framework consists of 3 tenants: effective communication, application of didactic content in clinical settings, and integration of technology considerations. Specific learning outcomes for students include an assessment of how they engage clients, their nonverbal communication, attentiveness and their ability to demonstrate course objectives related to communication approaches, application of didactic content in clinical settings,
and integration of technology. The focus of the clinical experience is on psychosocial assessments of well elderly community-residing residents. Not only are students able to share every resident interaction with their faculty and receive constructive feedback but also faculty can view every student’s interaction and provide feedback frequently, including during conferences at the clinical site if faculty are actually present. Previously, faculty were limited to evaluating directly only the number of student dyads they could accompany for resident visits. Only retrospective feedback was provided for the remaining studentresident interactions. Now, faculty can access the VAT from any computer connected to the Internet, access the students’ video(s), and conduct an assessment. Using the clips created by students, faculty evaluate the quality of student-resident interactions (Figure 2) and provide precise and detailed feedback concerning observations associated with specific times (eg, 1:05.00 PM to 1:15.00 PM) within the total recorded video session. Such feedback is critical to guiding students’ communication and assessment skills with older adults. Students then enter the system to access faculty feedback and communicate further (in person or asynchronously) with the faculty about successes or areas that require improvement. Faculty evaluate the student’s practice performance and justify their assessment with online comments to students. The VAT Faculty Function includes all system capabilities available to students as well as the ability to upload and evaluate additional items of evidence (eg, clinical samples, videotapes) and create clips from the students’ initial videos. Faculty can recommend that selected students contribute their clips as evidence of high-quality practices so others can view examples that meet a particular standard. Faculty decided that the use of the VAT system was a course requirement but that during the first semester of the initial trial it would be a pilot project and would have no impact on any student’s grade. Subsequently, grades were awarded. Benchmark performance standards against which students ranked their performances
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System
E15
Figure 2. Sample online screenshot.
for practice were based on guidelines that included Evidence-Based Geriatric Nursing Protocols for Best Practice,20 Assessment of Function,21 Caring for an Aging America: A Guide for Nursing Faculty,22 The Essentials of Baccalaureate Education for Professional Nursing Practice,23 and Recommended Baccalaureate Competencies and Curricular Guidelines for the Nursing Care of Older Adults,24 which outline competencies, content, and teaching strategies for geriatric nurse education. The eAGER tool uses feedback and data entered by faculty to generate and display an assessment of the student’s level of performance. Faculty can browse this assessment and see clearly the student’s level of performance for each standard. Faculty and students host threaded discussions about the assessment and feedback. Once the assessment is completed, the Faculty Function of eAGER generates 2 draft reports in Microsoft Word format—the Assessment Report and the Faculty Report. AVAILABLE ENHANCEMENTS TO THE VAT The eAGER-enhanced version of the VAT can include additional support modules as
part of the final architecture: Evidence Uploader (upload of video, documents, assessment images, and clinical data); Assessment View (by self and faculty) providing functions for aligning evidence to standards, associating comments, and aligning student practices demonstrated with video examples of practice provided by the College of Nursing; Electronic Portfolio with which students transfer evidence and data (eg, feedback, assignments, and assessments) to demonstrate growth along their nursing preparation continuum; Knowledge Base, where faculty can either upload evidence themselves (eg, from existing video libraries) or recommend a nursing student’s practices be included in this digital library and be recognized for excellence in demonstrating a standard; and Report Generator, which provides functions for students to export their portfolio. Faculty can analyze and export data accessible in text or data (csv for Excel) file formats. eAGER TRAINING OF STUDENTS AND FACULTY In this project, all faculty members and students completed 2 training sessions under a contract with the VAT company. Representatives from the clinical sites were
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E16
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
invited to attend. Emphasis was on the technology integrated into the processes of supervision, teaching, and learning. Everyone studied evidence capture (eg, video cameras) and uploading into the eAGER through a Web interface. Students reviewed the standards and expectations of their respective programs, learned about the most appropriate forms of evidence to capture, and considered the best techniques for capturing their practices (eg, how best to position a camera to capture their practice without hindering resident’s support) as well as how to complete the selfassessment. Students discussed how to support peer collaboration and what they would experience when interacting with classmates, residents, and faculty. Laptops were password protected, and students were instructed on restricting access only to themselves or risk severe penalties. Only minimal software, including Firefox and compression software, was placed on the computers and use was restricted to the clinical project. Students and faculty completed a course on HIPAA compliance in advance of this course. Faculty attended an additional session on the process for evaluating student video performance segments using the assessment tool embedded in the VAT. Sessions for faculty provided extra time to become familiar with the system and how it integrated with the assessment, supervision, and teaching of students. They learned the various forms of evidence that embody practices and how they align with the standards and expected outcomes of the nursing programs. Both the nursing students and faculty received “refresher” training sessions throughout the implementation period as needed. All users had immediate online access to technical support by e-mail and the company Web site to resolve technical issues. Training sessions were recorded in a lecture capture program for students and faculty to review as needed. HUMAN SUBJECTS APPROVAL All facets of the project are HIPAA compliant, including the VAT system processes, servers, backup, and devices on which all data
and videos are stored. The project was approved by the institutional review board (IRB) of each participating university and the appropriate housing authorities with oversight responsibility for the geriatric residential facilities. While this project is not classified as research, it involves the collection of data from human subjects and thus falls under the auspices of the review boards of each university. Consent letters explained to candidate participants in detail the relevant policies and procedures, the purpose of the project, and thus their rights as human subjects, and then solicited their informed participation. Only residents who could read and speak English, had the potential of understanding the consent forms as well as the use and purpose of the video camera, and the survey instrument were included in the project. Residents were oriented to the use of the VAT system by faculty and given the opportunity to accept or decline participation. Residents were consented prior to the beginning of actual clinical activities by the principal investigator and/or designated faculty or staff. All faculty and staff members were eIRB certified. At the beginning of each clinical day, participating residents were given an additional opportunity to decline. Anyone who declined was exempted from participation. If residents wished to exclude a portion of their home setting from the video, the camera’s field of view was adjusted accordingly. Residents were asked to complete an anonymous survey concerning their experience with VAT-enhanced training and were provided a return process. PARTICIPANTS AND CLINICAL SITES One of the initial schools was deleted at the end of the first year because of teaching assignment conflicts, and a fourth school was added. As a result, the faculty also changed. Each of the participating schools of nursing offer courses focusing on geriatric adults in residential communities or homes. To date, there have been more than 200 students at 4 universities and multiple faculty members participating at 7 retirement community
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System residential sites during the 3 years of the project. Each residential site had independent living facilities for “well elderly” residents and ranged in size from approximately 60 to more than 350 residents. While the sites differed somewhat, the participants included lower-, middle-, and upper-income adults aged 60 years and older. Faculty were primarily doctorally prepared and most had more than 20 years of teaching experience, including considerable experience with computer technology and distance education. VIDEO CAMERA SETUP AND IMPLEMENTATION Students took the assigned, lightweight, battery-powered video camera and a small tripod with them for clinical visits with residents. The video cameras were set up quickly and began recording to an internal hard drive with the push of a button. The wide-angle lenses of the video camera enabled capture of most of the room. The lenses, very sensitive for low-light conditions, did not require additional lighting. The highly sensitive builtin microphones did not require other people or equipment because the video cameras could be preconfigured to accommodate soft-spoken residents. During the training, students learned the visual cues for knowing if the camera was on and recording properly and how to approach and depart from residents’ homes. Students learned to position the camera to capture the student-resident interaction to the greatest extent possible. A detailed handout covering the use of the video cameras, associated ethical considerations, and opting-out issues for residents was provided. DEVELOPMENT OF SURVEY TOOLS AND PRE-/POSTTEST The first section of each survey includes questions on demographic characteristics. The second section focuses on questions on a 5-point Likert scale (strongly disagree to strongly agree) based on melding STST and
E17
the LPM. Survey instruments assessed each participant’s perceptions of the impact of the VAT system on the following: (1) roles, (2) relationships with others, (3) self-esteem, (4) use of time (5) time structure, and (6) support groups. The survey of residents focuses on perceptions of the use of video cameras in residential settings. Because of the ages of residents, these statements were kept simple and to a minimum. Large print was available. Prior input was sought from the employed staff in the residential communities to identify potential participants. Content validity was established by expert faculty. Reliability of the survey instruments is being assessed over time using the responses of multiple nursing classes at all participating sites. A pre-/posttest based on the geriatric nursing content of the course was developed and administered to all students at the beginning and again at the end of the geriatric nursing courses. RESULTS Research question 1 On the basis of the results from faculty surveys and ongoing discussions among participating faculty throughout the project length, the faculty endorsed the use of the system and noted that it increased their abilities to monitor students and provide positive feedback while facilitating student learning and individual assessments. Because of the small number of faculty involved, discussions were found to be more useful than the survey instruments. Faculty were able to view the students and residents during interactions, observe the home environment, and make pertinent comments to students. At the outset, there were problems uploading video recordings, accessing the system, and time lags between uploads and completion. Company officials were receptive to requests for assistance and by the end of the pilot semester, these problems had been resolved. A 2-hour upload process was decreased to 10 minutes with the use of free, downloadable video compression software. An initial inability to locate
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
E18
NAQ-D-14-00013
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
uploaded videos in the VAT system was resolved. Initial challenges with faculty usage included accessing the secure VAT system (resolved by better scheduling of routine company system maintenance) and the multiple steps required to select and apply the appropriate evaluation tool for student evaluation. Company administrators hid listings of some tools, and the geriatric assessment evaluation tool was automated. Data continue to be collected, but of the students reported here (Table 1), most were 21 to 30 years old (n = 66), 14 had a bachelor’s degree in another discipline, 58.8% (n = 47) had no experience with the VAT, and 81.3% (n = 65) were female. Responses on the student survey provided important insights into changes in life patterns and the use of this technology. The majority (n = 47; 58.8%) reported that the VAT system documented their knowledge, skills, and interactions and helped with discussions with faculty (n = 42; 52.6%). For 56.2% (n = 45), the VAT did not interfere with their assessments, and for 43.8% (n = 35), it facilitated learning. Yet, 49.3% (n = 37) felt that they could have used their time better. Comments from students, faculty, and residents were also elicited in informal debriefing sessions and incorporated into revisions of the use of the VAT system. Students suggested that even more emphasis needed to be placed on updating passwords from previous students to facilitate their time use. The issues with password failures were addressed and resolved with the support of technology staff. Using the variables of camera experience, education, race, VAT experience, gender, and 12 Likert scale items, the Cronbach α coefficient for the student survey is 0.780. An even newer version of the student survey has been now been developed. Completion of surveys by residents decreased considerably over the 3 semesters of the project, primarily because many were repeat participants and voiced no interest in completing the surveys again. Hence, only first-semester participants are reported here (Table 2), along with comments made during faculty-resident interactions. From the
surveys, more than 77% liked the idea of the videotapes and no residents reported that it wasted their time or interfered with their other plans for their day. Students made appointments in advance to facilitate timeliness with resident visits. Only a couple of residents are known to have refused to participate, and, in fact, there was a “waiting list” at each of the sites for residents who wanted to participate but could not be included with that semester’s cohort. This was surprising due to the mid80s ages of the geriatric residents. Concern was expressed by faculty that some residents seem to “dress up” for the video recording and perhaps gave responses that they thought students wanted to hear rather than being candid. These observations gave faculty an opportunity to work with students on communication techniques to move their interactions beyond the superficial level and elicit more meaningful information. One resident stated that she did not mind being video recorded but stated: “Just don’t make me watch it.” Research question 2 Analyses of the pre- and posttests of geriatric nursing questions showed no overall significant difference; however, there was a significant difference for 3 of the 10 questions, namely, for nutrition and medication knowledge, and types of stress. No significant differences were found across schools. Not all students completed the posttest and while they were encouraged to take the 2 tests seriously, some may not have done so because no grade was associated. Faculty reassessed the test content for the level of knowledge, and a new version is forthcoming. More data will be collected over the next year. Because all students enrolled in the course were required to use the videotaping system and the pre-/posttests had not been used in previous years, comparisons with other students were not made. A grade of satisfactory is required in the course for progression in the program, and unsatisfactory grades are extremely rare due to the competitiveness and high admission
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
3.8 5.0 10.0 17.5 50.0 11.3 22.4 2.6 6.6 7.5 12.5 10.0
6.3
3 4 8 14 40 9 17 2 5 6 10 8
5
20
13
24
32
18 8 40
32
25.0
16.3
30.0
40.0
23.7 10.6 52.6
40.0
38.8
45.0
40.0
51.2
36.3
25.0 37.5
%
13
11
17
19
26 28 12
15
3
20
16
16
11
20 19
n
U
16.3
13.8
21.3
23.8
34.2 37.3 15.8
18.8
3.8
25.0
20.0
20.0
13.8
25.0 23.8
%
33
35
23
17
11 24 13
18
2
6
17
16
29
30 21
n
A
41.3
43.8
28.7
21.3
14.5 32.0 17.1
22.5
2.5
7.5
21.3
20.0
36.3
37.5 26.3
%
9
12
5
6
4 13 6
4
4
3
5
2
8
5 7
n
SA
11.3
15.0
6.3
7.5
5.3 17.3 7.9
5.0
5.0
3.8
6.3
2.5
10.0
6.3 8.8
%
September 4, 2014
31
36
32
41
29
20 30
n
D
NAQ-D-14-00013
Video Analysis Tool System
Abbreviation: VAT, Video Analysis Tool. a Omitted responses were deleted from calculations. SD, strongly disagree; D, disagree; U, undecided; A, agree; SA, strongly agree.
5.0 1.3
4 1
1. The VAT facilitated my learning. 2. The VAT interfered with my interactions with the resident. 3. I felt less comfortable in my role as a student in the presence of the camera. 4. The VAT interfered with my assessment of the resident. 5. The VAT facilitated my relationship with my resident. 6. The VAT interfered with my relationship with my instructor. 7. The resident refused to permit me to use the VAT during our visit. 8. I feel less confident when knowing my instructor is watching me. 9. I saved time using the VAT. 10. I could have used my time better without the VAT. 11. The VAT system takes time away from my clinical visit experience. 12. The VAT system enhances my experience with my resident. 13. The VAT system helps in my discussions with other students during debriefing sessions. 14. The VAT system effectively documents my knowledge, skills, and interactions during my clinical visits. 15. The VAT system helps with my discussions with my instructor during debriefing sessions.
%
n
SD
Statement
Table 1. Sample Student Survey Responsesa (N = 80)
LWW/NAQ 3:57
E19
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
NAQ-D-14-00013
%
23.5
7.8 18.8
3.9
12
4 9
2
standards for the programs. More data are being collected on the new pre-/posttest, but findings indicate the VAT system is an effective approach to teaching clinical geriatric nursing with implications for practice.
15.7
41.2 37.5
53.9
8 33.3
21 18 21.6 12.5 12.2
27 15.7
17 39.2
11 6 6 23.5 22.9 40.8 54.0
8 5.8
20 7.8 4
12 11 20 27 5.9 8.3 46.9 46.0 3 4 23 23
3 1.9 1
1. I like the idea of being videotaped when the student was with me. (n = 51) 2. The videotape helped me talk to the student.(n = 51) 3. The videotape made me feel important. (n = 48) 4. The videotape wasted my time. (n = 49) 5. The videotape interfered with other things that I had planned to do when the students were there. (n = 50) 6. I would like the videotape to be shared with my doctor. (n = 51)
% n Statement
n
D
%
n
U
%
n
A
%
DISCUSSION
a SD,
strongly disagree; D, disagree; U, undecided; A, agree; SA, strongly agree.
SD
Table 2. Resident Survey Responsesa First Semester (N = 51)
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
SA
E20
September 4, 2014
n
LWW/NAQ
If a person changes roles, it is predictable that there will be changes in relationships, use of time, and life structure. With changes in the use of time, there will likely be changes in roles that could ultimately impact self-esteem. Use of the VAT system changes student and faculty roles by incorporating technology and does so in an environment that allows faculty access for direct observation, supervision, and input into the system. Students become selfevaluators of their resident interactions as well as peer mentors. Such new mentorship roles could have a positive as well as negative impact on self-esteem when students are viewed as experts by fellow students. Faculty move from single-student dyad evaluators to evaluators of multiple dyads from distant locations. Faculty no longer have to be on-site to evaluate students accurately. Residents provide the medium and are an integral part of the evaluative effectiveness of the technological system. Students can rely on video-recorded clips to provide accurate documentation of their activities rather than “after-the-fact” reports. The VAT reduces the travel budget. While recording anxiety might be a factor, students know that evaluations are based on observed phenomena, not recollections, and any discrepancies can be mediated by another reviewer. Because students will be faced with computer technologies during both clinical experiences and upon entering the workforce after graduation, incorporating technology into their educational experience is vital. Changing the way faculty teach is also important when reaching out to the current generation of techno-savvy students, as well as for reducing financial costs. Using direct video recording for assessment and evaluation purposes in community settings affords faculty
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
September 4, 2014
3:57
Video Analysis Tool System the opportunity to enhance the learning of students’ knowledge and skills as they prepare for practice in community health. Through more responsible use of faculty time, budgetary constraints can be addressed while concomitantly incorporating more effective educational strategies. Through the reduced need to visit students in home health settings, faculty can avoid needless travel. Costs for the system are complex to analyze and vary with the program and the university. For example, 150 students with 15 faculty full-time equivalents (10:1 ratio), for 6 visits at $0.55 per mile at an average of 30 miles r/t = $1485.00 for travel for 1 course compared with zero travel expense using video technology. More distant clinical sites that were previously eliminated because of cost factors can now be reinstated. In these settings, video clips could capture the community, home surroundings, the inside of the dwelling, and even household pets to provide a more thorough assessment. Students were engaged and involved in the resolution of problems. Data will now be collected to determine ongoing effectiveness. In addition, graduate-level nurse practitioner students visit homes of clients in wide geographic regions, so feasibility of VAT usage is likely. LIMITATIONS AND IMPLICATIONS FOR PRACTICE The project was conducted with small convenience samples of undergraduate nursing students in a limited number of schools of nursing and limited geriatric residents in one geographic region. No attempt was yet made to use acute care facilities where video recording would have additional HIPAA, IRB, and
E21
ethical considerations. Residents were well elderly residents with no known dementia or serious disabilities. Ethernet connections were available at each geriatric retirement community; however, for some students living in rural areas, they had to wait until access to a secure connection was available. This process has excellent potential for applications in practice settings. It would be useful for public health nurses to document the status of clients as well as share knowledge with newly assigned nurses prior to a visit. In clinic settings, video recordings could be used to document client status, to share with health team members, and for client-focused conferences. In making home visits, nurses could send information to physicians and other health care providers and seek input on interventions, treatment methods, medications, and the like. The recordings could be used to teach and evaluate specific family members regarding dressing changes, insulin administration, and even how to ambulate someone with a walker, or crutches, or in need of additional assistance. By zooming in the camera, details can be observed and recorded. Client status, progress, and lack of progress can be noted. Procedures and treatments can be explained. The performance measure of nurses could be documented and evaluations completed with real-time evidence. Whether uploaded to a system, as with this project, or simply recorded and transferred to a secure site on the facility’s server, objective information can be recorded and retained. The ramifications of this approach are noteworthy in augmenting client and family care and the interface between health care providers, clients, and their families.
REFERENCES 1. Fox W. Sociotechnical system principles and guidelines: past and present. J Appl Behav Sci. 1995;31(1):91-105. 2. Churchman C, Verhulst M, eds. Management Sciences, Models and Technique. London, England: Pergamon; 1960.
3. Harteloh P. Quality systems in health care: a sociotechnical approach. Health Policy. 2003; 64(3):391-398. 4. Li J. A Sociotechnical approach to evaluating the impact of ICT on clinical care environments. Open Med Inform J. 2010;4:202-205.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
LWW/NAQ
NAQ-D-14-00013
E22
5. 6. 7.
8.
9.
10.
11.
12.
13.
14.
15.
September 4, 2014
3:57
NURSING ADMINISTRATION QUARTERLY/OCTOBER–DECEMBER 2014
doi:10.2174/1874431101004010202. http://www .ncbi.nlm.nih.gov/pmc/articles/PMC3096882. Accessed July 7, 2013. Carayon P. Human factors of complex sociotechnical systems. Appl Ergon. 2006;37(4):525-535. Brooks B, Anderson M. Defining quality of nursing work life. Nurs Econ. 2005;23(6):319-326. Happ M. Sociotechnical systems theory. Analysis and application for nursing administration. J Nurs Adm. 1993;23(6):47-54. Aldridge J. Sociotechnical systems theory. http:// www.argospress.com/Resources/team-building/ socitechnisistem.htm. Published 2004. Accessed July 7, 2013. Mlitwa N. Technology for teaching and learning in higher education contexts: activity theory and actor network theory analytical perspective. Int J Educ Dev Using ICT. 2007;3(4). http://ijedict.dec .uwi.edu/printarticle.php?id=420&layout=html. Accessed July 7, 2013. Bjoko A. Using eye tracking to compare Web page designs: a case study. J Usability Stud. 2006;3(1):112120. Anderson S, Felici M. Emerging Technological Risk: Underpinning the Risk of Technology Innovation. New York, NY: Springer; 2012. Bowcutt M, Rosenkoetter M, Chernecky C, Wall J, Wynn D, Serrano C. Implementation of an intravenous medication infusion pump system: implications for nursing. J Nurs Manag. 2008;16: 188-197. Rosenkoetter M, Bowcutt M, Khasanshina E, Chernecky C, Wall J. Perceptions of the impact of smart pumps on nurses and nursing care provided. J Assoc Vasc Access. 2008;13(2):60-69. Rosenkoetter M, Stachura M, Astapova E, Brown D, Dias J. Age-related impact of intensive insulin therapies. Paper presented at: American Academy of Nursing Annual Meeting; October 13-15, 2011; Washington, DC. Stachura M, Rosenkoetter M, Astapova E, Dias J, Brown D, Continuous subcutaneous insulin infusion and multiple daily injections: an impact study. Paper presented at: Diabetes Technology Society
16.
17.
18.
19.
20.
21.
22.
23.
24.
Annual Meeting; October 29, 2011; San Francisco, CA. Recesso A, Hannafin M, Wang F, Deaton B, Shepherd C, Rich P. Direct evidence and the continuous evolution of teacher practice. In: Adamy P, Milman N, eds. Evaluating Electronic Portfolios in Teacher Education. Greenwich, CT: Information Age Publishing Inc; 2009:38, chap 2. Rich P, Hannafin MJ. Video annotation tools: technologies for assessing and improving preservice teachers’ instructional decision making. J Teach Educ, 2009;60(1):52-67. Bryan L, Recesso A. Promoting reflection with a Webbased video analysis tool. J Comput Teach Educ. 2006;23(1):31-39. Recesso A, Zepeda SJ. Evidential reasoning and decision support in assessment of teacher practice. In: Kowalski TJ, Lasley TJ, eds. Handbook on Databased Decision Making in Education. Hillsdale, NJ: Lawrence-Erlbaum; 2008:363-381. Capezuiti E, Zwicker D, Mezey M, Fulmer T. eds. Evidence-Based Geriatric Nursing Protocols for Best Practice. New York, NY: Springer Publishing Co; 2008. Kresevic D. Assessment of function. In: Capezuti E, Zwicker D, Mezey M, Fulmer T, eds. Evidence-Based Geriatric Nursing Protocols for Best Practice. 3rd ed. New York, NY: Springer Publishing Co; 2008:2340. Thornlow D, Latimer D, Kingsborough J, Arietti L. Caring for an Aging America: A Guide for Nursing Faculty. Washington, DC: American Association of Colleges of Nursing; 2006. American Association of College of Nursing. The Essentials of Baccalaureate Education for Professional Nursing Practice. Washington, DC: American Association of College of Nursing; 2008. American Association of College of Nursing. Recommended Baccalaureate Competencies and Curricular Guidelines for the Nursing Care of Older Adults. Washington, DC: American Association of Colleges of Nursing, and New York: New York University College of Nursing, Hartford Institute for Geriatric Nursing; 2010.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
