Journal of Perinatology (2014) 34, 781–786 © 2014 Nature America, Inc. All rights reserved 0743-8346/14 www.nature.com/jp
ORIGINAL ARTICLE
Correlations between technical skills and behavioral skills in simulated neonatal resuscitations T Sawyer1, D Leonard2, A Sierocka-Castaneda3, D Chan4 and M Thompson4 OBJECTIVE: Neonatal resuscitation requires both technical and behavioral skills. Key behavioral skills in neonatal resuscitation have been identified by the Neonatal Resuscitation Program. Correlations and interactions between technical skills and behavioral skills in neonatal resuscitation were investigated. STUDY DESIGN: Behavioral skills were evaluated via blinded video review of 45 simulated neonatal resuscitations using a validated assessment tool. These were statistically correlated with previously obtained technical skill performance data. RESULT: Technical skills and behavioral skills were strongly correlated (ρ = 0.48; P = 0.001). The strongest correlations were seen in distribution of workload (ρ = 0.60; P = 0.01), utilization of information (ρ = 0.55; P = 0.03) and utilization of resources (ρ = 0.61; P = 0.01). Teams with superior behavioral skills also demonstrated superior technical skills, and vice versa. CONCLUSION: Technical and behavioral skills were highly correlated during simulated neonatal resuscitations. Individual behavioral skill correlations are likely dependent on both intrinsic and extrinsic factors. Journal of Perinatology (2014) 34, 781–786; doi:10.1038/jp.2014.93; published online 15 May 2014
INTRODUCTION Cognitive skills, technical skills and behavioral skills are the three intrinsic elements of human performance at play during neonatal resuscitation.1–3 Cognitive skills include ‘what is in our brains and how our brain works’.1 Technical skills involve ‘things that we do with our hands’ or ‘the psychomotor techniques of particular procedures that require (among other things) manual dexterity’.1,4 Behavioral skills have been defined as ‘decision-making and team interaction processes used during the team’s management of a situation’ or ‘activities that are not merely manual procedures requiring dexterity’.4,5 In neonatal resuscitation, cognitive skills involve knowing the Neonatal Resuscitation Program (NRP) algorithm, recalling drug doses, remembering the MR SOPA (mask, reposition, suction, open mouth, pressure, airway) mnemonic, and so on. Technical skills in neonatal resuscitation include procedures such as bag-valve mask ventilation, chest compressions, intubation, umbilical cannulation and medication administration. The NRP has identified 10 key behavioral skills important in neonatal resuscitation, which include: knowledge of environment, anticipation and planning, leadership, communication, distribution of workload, attention allocation, information utilization, resource utilization, calling for help early and professional behavior.6 Behavioral skills have been correlated with improved NRP compliance and overall quality of care in the delivery room.7 As a result, some neonatal care units have instituted team training to improve the behavioral skills of their neonatal resuscitation teams.8 However, the intricate relationships, correlations and interactions between cognitive skills, technical skills and behavioral skills during neonatal resuscitation have yet to be fully defined. Multiple reports have shown that improvements can be seen in cognitive, technical and behavioral skills during neonatal resuscitation as a result of simulation-based training.9–18 The retention of
these skills has also been investigated.19,20 However, few studies have described the correlations between these different skill domains. Thomas et al.7 used video recordings of neonatal resuscitations and found weak correlations between the overall quality of neonatal resuscitation and three behavioral skills, which included communication, management and leadership. In a study of anesthesiology residents managing a simulated adult intraoperative cardiac arrest, Riem et al.21 reported a strong correlation between technical skills and behavioral skills. The strongest correlations were seen with situational awareness, team working and decision making. To the author’s knowledge, no prior study has examined the correlations between the 10 key behavioral skills taught in the NRP course with technical skills during neonatal resuscitation. To evaluate the correlation between technical skills and the 10 key NRP behavioral skills we conducted the following study. We hypothesized that a positive correlation would exist between technical skills and behavioral skills, and that certain behavioral skills would be more strongly correlated with technical skills than others. METHODS Objectives and design To evaluate correlations and interactions between technical skills and behavioral skills in neonatal resuscitation, we performed a secondary analysis of a set of neonatal resuscitation simulation videos, which were obtained as part of prior investigation on the effects of deliberate practice on neonatal resuscitation performance.9 That study followed a repeatedmeasures design, in which pediatric and family medicine residents completed a series of three high-fidelity simulations, with a facilitated debriefing after each. Objective measurements of technical skills were obtained during each of the three simulation sessions by way of blinded video review using a validated technical skills assessment tool. For the current study, we performed a second blinded review of the same video
1 Department of Pediatrics, University of Washington School of Medicine, Seattle, WA, USA; 2Sacred Heart Medical Center, Eugene, OR, USA; 3Department of Pediatrics, National Naval Medical Center, Bethesda, MD, USA and 4Department of Pediatrics, Tripler Army Medical Center, Honolulu, HI, USA. Correspondence: Dr T Sawyer, Division of Neonatology, Department of Pediatrics, University of Washington School of Medicine, 1959 NE Pacific Street, RR 539, HSB, Seattle, 98195 WA, USA. E-mail: [email protected] Received 6 December 2013; revised 1 March 2014; accepted 9 April 2014; published online 15 May 2014
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
782 set, this time using a previously validated behavioral skill assessment tool that evaluated each of the 10 key NRP behavioral skills. Statistical analysis was then performed to examine correlations and interactions between the observed technical and behavioral skills.
Participants All pediatric and family medicine residents at Tripler Army Medical Center were invited to participate in the study. Tripler Army Medical Center is a 450 bed tertiary care center in Honolulu with approximately 3000 deliveries, and 300 neonatal intensive care unit admissions per year. Participating residents were paired into teams of two, consisting of residents from the same program and year group. Each team of residents remained the same throughout the study. Scoring was based on both team and team leader performance, as applicable. Informed consent was obtained from all participants. The study protocol was approved by the Institutional Review Board at Tripler Army Medical Center.
Simulation sessions All simulations were performed in a realistic simulated delivery room equipped with the standard materials found in a typical delivery room, including an infant warmer, oxygen, suction, resuscitation equipment and medications. The residents were assisted during each simulation by one of the researchers (DC) who acted as a confederate labor and delivery nurse. During the resuscitations, the ‘nurse’ was able to perform bag-valve mask ventilation, chest compressions and draw up medications, but only with instruction from the residents. The ‘nurse’ did not provide direction in NRP or make treatment decisions. Simulations followed a distributed practice learning model and were scheduled on separate days, separated in time, to take advantage of the ‘spacing effect’. The spacing effect is a welldescribed phenomenon in educational psychology; for a given amount of study time, spaced study yields better learning results than does massed study.22 Each of the three simulations sessions were scheduled between 2 and 4 months apart. The simulator used was a Laerdal SimBaby (Laerdal Medical, Stavanger, Norway), which had been specially modified to include an integrated umbilical catheterization task trainer.23 The three neonatal resuscitation scenarios were fetal hypoxia, volume loss from placental hemorrhage and fetal sepsis, which were completed in that order by each resuscitation team. The initial condition of the infant in all simulations was limp, apneic, cyanotic and bradycardic (heart rate o 60 b.p.m.). All simulations required intubation and intravenous epinephrine. The second and third simulations also required a normal saline bolus. For standardization all simulations were timed. A maximum of 10 min was allowed for each simulation session. Each simulation session was facilitated by a single investigator (TS) who also facilitated the debriefing session that immediately followed. Discussion during the debriefings covered both technical and behavioral skills. Debriefing discussions of behavioral skills focused specifically on three key NRP behavioral skills: communication, resource utilization and leadership, in relation to the team’s performance during the simulation. For standardization, debriefing sessions were also timed and limited to 20 min.
Behavioral skill scoring Behavioral skills during each of the three neonatal resuscitation simulations were scored by two blinded reviewers (AS, DL) by way of blinded video review. The reviewers were blinded to both the resident year group and the simulation session number. Behavioral skills were scored using the Behavioral Assessment Tool (BAT). The BAT is a 5-point (0 to 4) behaviorally anchored rating scale developed specifically to measure behavioral performance in each of the 10 key NRP behavioral skills. Performance in each behavioral skills was given a score from 0 to 4, with 0 indicating ‘poor’ performance, 2 indicating ‘acceptable’ performance and 4 indicating ‘excellent’ performance. Specific and observable behaviors that describe performance at each of the five levels are provided as behavioral anchors on the BAT. An overall behavioral skill score was calculated by averaging the values of the 10 key behavioral skills. Both the validity and reliability of the BAT has been reported previously.25–27 The inter-rater reliability of the BAT, via Cronbach’s α, has ranged from 0.95 to 0.97. Rater training on the use of the BAT for this study included a didactic session explaining the use of the tool and practice exercises grading video-taped simulated resuscitations.
Clinical exposure All subjects were asked to report their actual clinical experience in neonatal resuscitation at the time of the first and third simulations. These data were used to provide a measure of clinical experience gains during the course of the study. The correlation of clinical experience gains and behavioral skill changes during the study was analyzed to account for the confounding variable of natural clinical skill acquisition during the course of the study.
Statistical analysis One-way repeated-measures analysis of variance, or Friedman repeatedmeasures analysis of variance on ranks, with Holm–Sidak, or Student–Newman–Kuels corrections, as appropriate, were used to evaluate changes in technical skills and behavioral skills from the first to the third simulation sessions. Changes in technical skills and behavioral skills were calculated based on the raw dichotomous scores on the NRPE, and the 5-point scale of the BAT. Spearman’s rank-order correlation coefficient (ρ) was used to determine correlations between technical skills and behavioral skills. Pearson’s product–moment correlation coefficient was used to measure the correlation between natural clinical experience gains during study and behavioral skill changes. To investigate the interaction of technical skills and behavioral skills the technical performance scores of teams with behavioral performance scores in the top quartile were compared to teams with behavioral performance scores in the bottom quartile using a Mann–Whitney rank-sum test or Student’s t-test, as applicable. The same analysis was performed on the behavioral performance scores of teams with technical skill performance scores in the top quartile and bottom quartiles. The educational effect size was calculated using Eta squared (η2), based on the ratio of effect to total variance derived from the repeatedmeasures analysis of variances.28,29 By convention, an η2 effect size of ⩽ 0.01 was considered 'small', η2 around 0.06 'moderate' and η2 ⩾ 0.14 'large'.30 P o0.05 was considered statistically significant. Data were analyzed using SigmaPlot 11.0 (Systat Software, San Jose, CA, USA).
Technical skill scoring Technical skills were scored as part of a prior investigation using a validated scoring instrument called the Neonatal Resuscitation Performance Evaluation (NRPE).9 The NRPE provides a dichotomous (yes = 1, no = 0) score for 30 technical skill metrics in neonatal resuscitation, which are divided into seven subdomains: preparation and initial steps, communication of heart rate, bag/mask ventilation, chest compressions, intubation, medication administration and umbilical vessel catheterization. The NRPE also records times to achieve key steps during resuscitation, including time to start positive-pressure ventilation, time to start compressions, time to successful intubation, time to first epinephrine dose and time to intravenous access. Data derived from the NRPE has been found to have strong evidence of validity using several lines of evidence as described by Downing,24 which included: content, consequence, response process, relationship to other variables and internal structure.9 The NRPE reliably differentiates teams with more experience in neonatal resuscitation from teams with the less experience, and prior reliability testing showed a good inter-rater reliability for the various subdomains, with a mean Cohen’s κ of 0.63.9 Journal of Perinatology (2014), 781 – 786
RESULTS Thirty-two residents, divided into 16 teams of two, participated in the study. One team was unable to complete all three simulation sessions and their performance data was omitted from the statistical analysis. Thus, the data presented here is based on the 15 teams of two residents who completed all three simulations. Table 1 provides demographic information on the study participants. The 15 teams completed the series of three simulations over a period of 9 months. Time between the simulation sessions averaged 14 weeks from the first to the second simulation sessions, and 9 weeks from the second to the third simulation sessions. A total of 45 resuscitation videos, 15 from each of the three simulations sessions, were reviewed and scored by the two blinded reviewers using the BAT. These scores were compared against the previously obtained NRPE score data. Cronbach’s α on the BAT in this study was 0.92. © 2014 Nature America, Inc.
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
783 Table 1.
Demographic data (n = 30)
Residency program Pediatrics Family medicine
22 (73%) 8 (27%)
Sex Male Female
15 (50%) 15 (50%)
Postgraduate year 1 2 3
8 (27%) 12 (40%) 10 (33%)
Number of times NRP course taken 1 2 3
21 (70%) 8 (27%) 1 (3%)
Time since last NRP course (months) 0–3 3–6 6–12 >12
14 7 2 7
(47%) (23%) (7%) (23%)
Abbreviation: NRP, Neonatal Resuscitation Program.
Table 2.
Changes in key behavioral skills in neonatal resuscitation Simulation session number 1
Knowledge of environment, mean (s.d.) Anticipation and planning Assumption of leadership role Communication among team members Distribution of workload Attention allocation Utilization of information Utilization of resources Calling for help Professional behavior Overall
2
P-value
3
2.4 (0.8) 2.7 (0.7) 2.7 (0.9)
0.12
2.1 (0.9) 2.6 (0.9) 2.7 (0.9) 1.8 (0.8) 2.0 (0.8) 2.0 (0.9) 2.3 (0.9) 2.6 (0.9) 2.8 (1.0)
0.004 0.54 0.03
2.1 1.9 1.9 2.1 1.5 2.6 2.1
(0.7) (0.9) (0.9) (0.8) (0.9) (0.7) (0.8)
2.6 2.2 2.4 2.2 2.0 2.8 2.4
(0.7) (1.0) (1.0) (0.9) (1.1) (0.8) (0.9)
2.8 2.3 2.4 2.7 2.1 3.1 2.6
(0.7) o0.001 (1.0) 0.13 (1.0) 0.03 (0.9) 0.01 (0.9) 0.01 (0.8) 0.02 (0.9) o0.001
Scores ranged from 0 to 4, with 0 indicating ‘poor’ performance, 2 indicating ‘acceptable’ performance, and 4 indicating ‘excellent’ performance.
Overall scores for both technical and behavioral skills increased from the first simulation to the third simulation (technical: F (2,449) = 11.8; P o0.001; behavioral: F (2,149) = 27.2, P o0.001). Behavioral scores improved from the first to the third simulation in all three resident year groups (P o0.001). Third-year residents scored higher in behavioral skill performance than first-year residents on the first simulation (overall BAT scores: Sim 1, 3rd year = 2.6 ± 0.8 vs 1st year = 1.7 ± 0.7; P o0.001), but scores were not different by the third simulation (overall BAT scores: Sim 3, 3rd year = 2.8 ± 0.9 vs 1st year = 2.6 ± 0.9; P = 0.12). Table 2 shows the changes in each of the 10 key NRP behavioral skills from the first to the third simulation. There was no association between clinical experience gained during the course of the study and changes in overall behavioral skill performance from the first to third simulation (r = 0.09; P = 0.76). There was a strong positive correlation between overall technical skill and behavioral skill performance scores (ρ = 0.48; © 2014 Nature America, Inc.
Figure 1. Correlation between overall technical and behavioral skill scores in 45 simulated neonatal resuscitations. Behavioral Assessment Tool (BAT), Neonatal Resuscitation Performance Evaluation (NRPE).
Table 3. Correlations between overall technical skills and the 10 key behavioral skills in neonatal resuscitation
Knowledge of the environment Anticipation of and planning Assumption of leadership role Communication among team members Distribution of workload Attention allocation Utilization of information Utilization of resources Call for help early enough Professional behavior Overall
ρ
P-value
0.33 0.38 0.29 0.32 0.60 0.42 0.55 0.61 0.25 0.45 0.48
0.21 0.15 0.27 0.23 0.01 0.11 0.03 0.01 0.36 0.08 0.001
Abbreviation: NRP, Neonatal Resuscitation Program.
P = 0.001) (Figure 1). All 10 key NRP behavioral skills positively correlated with overall technical skill, with correlations ranging from 0.25 to 0.61. The strongest correlations were seen in distribution of workload, utilization of information and utilization of resources (Table 3). There was a positive correlation between resident year group and overall technical (ρ = 0.32; P = 0.03) and behavioral (ρ = 0.42; P = 0.004) skill scores. There was also a positive correlation between simulation session number and technical and behavioral skill scores (technical: ρ = 0.33; P = 0.03; behavioral: ρ = 0.29; P = 0.05). Correlations between times to achieve key steps in resuscitation and overall behavioral skills are provided in Table 4. There was a significant difference in the technical performance scores between teams with behavioral performance scores in the top and bottom quartile (95.5 ± 5.7 vs 83.3 ± 12.2; P = 0.005). There was also a significant difference in the behavioral performance scores between teams with technical performance scores in the top and bottom quartile (3.0 ± 0.7 vs 1.9 ± 0.4; P ⩽ 0.001). The educational effect size of participation in the three simulations on technical and behavioral skills was large (technical: η2 = 0.42; behavioral: η2 = 0.53). DISCUSSION Performance during neonatal resuscitation is dependent on both the intrinsic performance of the resuscitation team members and the influences of extrinsic, or environmental, factors.1 The three Journal of Perinatology (2014), 781 – 786
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
784 Table 4.
Correlations between overall behavioral skills and time to achieve key steps in neonatal resuscitation
Start of positive pressure ventilation Start of compressions Successful intubation First epinephrine dose Emergency UVC access
ρ
P-value
−0.51 −0.25 −0.49 −0.49 −0.44
o0.001 0.1 0.002 0.001 0.002
Abbreviation: UVC, umbilical vessel catheterization.
intrinsic elements of human performances are cognitive skills, technical skills and behavioral skills.1–3 Using this paradigm, intrinsic human performance can be described by the following formula: cognitive skills+technical skills+behavioral skills = intrinsic performance. Overall resuscitation performance is then derived from intrinsic performance plus or minus the effects of extrinsic factors. As seen in the current study, and in multiple prior investigations, intrinsic performance in neonatal resuscitation can be improved through simulation-based training. What remains to be elucidated are the intricacies of the interactions between the three elements of human performances and the influence that extrinsic or environmental characteristics play. In the current study, over the course of three simulation sessions, scores for both technical and behavioral skills improved from baseline. There was a strong positive correlation between overall technical skills and behavioral skills. The key NRP behavioral skills most strongly associated with overall technical skills were distribution of workload, utilization of information and utilization of resources. There were significant negative correlation between behavioral skills and time to achieve key steps in resuscitation, indicating that a higher level of behavioral skill was associated with a decreased time to reach that step in resuscitation. Improvements in behavioral skills were not associated with clinical experience gained during the course of the study, which suggests that the improvements seen were the result of the simulation training, rather than natural skills acquisition from clinical exposure. Teams with the highest level of technical skills also demonstrated the highest level of behavioral skills, and vice versa, which highlights the intricate relationship between those two elements of human performance. Our study adds to, and expands upon, the existing literature on the interactions between technical skills and behavioral skills during neonatal resuscitation. In the only other study on this topic, Thomas et al.7 investigated teamwork and quality of care during delivery room resuscitation, and found weak correlations between quality of care and three teamwork behaviors: communication, management and leadership. The three teamwork behaviors identified by Thomas et al. were based on factor loading of nine different variables of teamwork within the three areas. The quality of resuscitation was based on adherence to NRP guidelines during resuscitation, as determined by blinded video review of actual delivery room resuscitation videos. The majority of the 132 videos reviewed in that study were of residents and nurses taking care of ‘relatively healthy’ infants, with only 8% undergoing intubation, 1.5% receiving chest compressions and one (0.75%) receiving epinephrine. Correlations between the three teamwork behaviors and the quality of resuscitation in that study ranged from ρ = 0.201 to 0.288 (P = 0.02 and o 0.001, respectively). Unlike the relatively healthy infants in the Thomas et al. study, the simulated infants in the current study were extremely ill and required extensive resuscitation efforts including intubation, chest compressions, umbilical vessel catheterization and medication administration. The differences in patient acuity may provide some explanation Journal of Perinatology (2014), 781 – 786
for the differences in correlations between technical skills and behavioral skills between to the two studies. Differences in statistical significance may be a result of the much larger number of resuscitations videos scored in the Thomas et al. study versus the current study (132 vs 45). The strong correlation between behavioral skills and technical skills in our study is consistent with reports of adult resuscitation simulations. In a retrospective analysis of previously recorded videos of anesthesiology residents managing a simulated intraoperative cardiac arrest, Riem et al.21 found a very similar overall correlation between technical skills and behavioral skills (R = 0.45; Po 0.05). In that study, scores on a specially developed cardiac arrest technical skills checklist were correlated with scores on the Anesthesia Non-Technical Skills scale,31 a behavioral scoring tool that evaluates behavioral skills in four behavior domains: task management, teamwork, situational awareness and decision making. The strongest correlations were seen in situational awareness, team working and decision making. Riem et al. speculated that cognitive load theory may provide an explanation for the observed relationship between technical skills and behavioral skills. Cognitive load theory is a theoretical model of cognitive architecture in which human memory is composed of ‘working’ and ‘long-term’ memory. Working memory is limited in capacity, and can only process a fixed amount of novel information at a time, while long-term memory is essentially limitless. According to cognitive load theory, due to the limitations of working memory, the human mind has adopted a strategy by which mental schemas (developed through practice and expertise) are stored in long-term memory and are accessed during times of high mental activity to reduce cognitive load by freeing up ‘space’ in the working memory.32,33 Riem et al. theorized that preexisting automated schemas (or expertise) in one domain (e.g., technical skills) reduced cognitive load, thereby freeing up cognitive resources to concentrate on the other domain (e.g., behavioral skills). They used cognitive load theory to explain why participants in their study with high technical skills also demonstrated high behavioral skills, and vice versa. Our results may provide additional evidence to support cognitive load theory. In our study, teams with superior performance in one domain (e.g., technical skill) also demonstrated superior performance in the other domain (e.g., behavioral skill), and vice versa. Cognitive load theory would posit that the high-performing teams in our study, with wellestablished technical or behavioral skills schemas, benefitted from a decrease in cognitive load, thus allowing them to concentrate their mental efforts on the other skill domain. This resulted in improved performance in both skill domains, as compared with teams without well-developed long-term memory schemas in either domain who had to expend working memory capacity on both technical and behavioral skills simultaneously. Careful interpretation of the generalizability of our results is warranted as our results were likely significantly influenced by both the intrinsic performance of the team members as well as extrinsic characteristics of the simulation scenarios. The baseline intrinsic behavioral and technical skills of the residents in the study may have also influenced our results. As seen in Table 2, the baseline behavioral skills of the teams in the study were fairly low, with most scores rating around ‘acceptable’, to less than acceptable. Future studies investigating the correlations between technical skills and behavioral skills in teams with higher levels of experience and experience in neonatal resuscitation are required to determine if the correlations reported here hold true for teams with higher levels of competency. The personnel resources in the scenarios were very limited (two physicians and one nurse to complete the entire resuscitation) in this study. We speculate that this extrinsic limitation imposed unique performance challenges on the teams. In this context, optimal sharing of work and utilization of the available limited resources were required. The significant correlations between overall technical skills and © 2014 Nature America, Inc.
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
785 the behavioral skills of distribution of workload and utilization of resources indicate that optimizing these behavioral skills resulted in improved technical performance. It is possible that the use of larger teams would have modified the correlations between technical skills and behavioral skills, and further research with larger resuscitation teams is required. The lack of improvement in the behavioral skills of knowledge of environment, attention allocation and leadership shown in Table 2 deserve attention. Knowledge of environment, according to the BAT, is defined by knowing aspects of environment, checking equipment, confirming readiness of environment, and not hesitating to ask questions when the need arises. At baseline, the teams in our study demonstrated fairly high scores in this skill, as compared with other skills, which may have made it difficult for significant improvements to be seen in this small sample. Attention allocation skills, according to the BAT, include not becoming distracted, being cognizant of details yet adequately monitors patient’s overall condition, prioritizing demands for attention and avoiding fixation errors. There were no imbedded equipment malfunctions or distractors within the scenarios to ensure practice with this specific skill. This may have contributed to a lack of significant improvement. Team leadership during the simulations was not predefined and was left to the discretion of the teams at the time of the simulation. Teams included two residents, each from the same program and year group. We speculate that this may have made it difficult for one team member to clearly establish themselves as the leaders. During the simulations, we observed a high degree of shared leadership. This shared leadership likely influenced the leadership scores on the BAT, as a clearly identified leader is a marker for excellent leadership. Additionally, the limited number of personal in the simulations made it difficult for the leader to separate themselves from being actively involved with the procedures of resuscitation (e.g., airway management, umbilical catheterization, etc.). This may have made it difficult for the reviewers to clearly identify the leader among the team members. Finally, the facilitated debriefings provided to the teams in our study focused on both technical and behavioral skills. We chose this method of debriefing as we felt it more accurately reflects a real life facilitated debriefing session, wherein the facilitator is obligated to provide feedback on both technical and behavioral skills, as applicable to team performance. In a prior report by Yee et al.34 evaluating the utility of teaching behavioral skills in anesthesia, the investigators were able to show significant improvements in all 15 domains of the Anesthesia Non-Technical Skills after a single high-fidelity simulation session. The debriefings in that study focused solely on behavioral skills. It is possible that an increased focus on behavioral skills during the debriefings in our study could have resulted in larger improvements in behavioral skills. What light does this study shed on the way we should train teams in neonatal resuscitation using simulation? First, this study highlights the benefits of behavioral skills training. Based on our finding of a high correlation between behavioral and technical skills, which is supported by cognitive load theory, focused training on behavioral skills is likely to provide significant benefits to resuscitation teams, with potential gains in both behavioral and technical skills. As seen in Table 4, behavioral skills were highly correlated with the speed in which several key steps in resuscitation were reached, which is a critical metric of resuscitation performance. Second, the study highlights the importance of objective evaluation of behavioral skills, using tools such as the BAT. This objective assessment could be used to focus behavioral skill training on specific areas of deficiency. The training could be focused by modification of the simulation scenario or through modification of extrinsic factors, such as the number of team members, or other environmental factors or constraints. Finally, our results highlight the benefits of distributed practice on simulation-based learning. In this study we were able to achieve © 2014 Nature America, Inc.
significant improvements in both technical and behavioral skills despite only 90 min of total training (10 min of simulation+20 min of debriefing×3 simulation sessions) distributed over a 9-month period. Our study has some limitations. The sample size was relatively small, thus placing our nonsignificant results at risk for type II error. A post hoc power analysis revealed that a sample size of 15 teams, with an α of 0.05, provided an 80% power to detect a 0.9 point difference in behavior scores between simulations 1 and 3, and a correlation of 0.25. Another limitation is that the NRPE combined elements of cognitive skills (decisions when to perform a procedure) and technical skills (how well the procedure was performed). Thus, the correlations described here may not be a pure reflection of the correlations between technical skills and behavioral skills, and may rather reflect the combined correlation of cognitive and technical skills with behavioral skills. However, we find it difficult to imagine a technical performance scoring method that would allow assessment of technical skills in isolation from cognitive skills during resuscitation, given the intimate association between the two. Obviously, one must know that a procedure is indicated (cognitive skills) in order to perform the procedure (technical skills). Finally, the self-reported clinical exposure data is at risk for recall bias. The strengths of this study include the use of standardized scenarios and debriefing methods and the use of blinded reviewers using reliable and valid technical and behavioral skill assessment tools. CONCLUSION We found a high degree of correlation between technical skills and behavioral skills in simulated neonatal resuscitations. The key NRP behavioral skills most strongly associated with overall technical skills in this study were distribution of workload, utilization of information and utilization of resources. Behavioral skill correlated with a faster time to reach key steps in neonatal resuscitation. Teams with superior behavioral skills had superior technical skills, and vice versa. The degree of correlation between specific behavioral skills and technical skills is likely dependent on both intrinsic and extrinsic factors. Further research investigating the impact of intrinsic and extrinsic factors on the correlations between technical skills and behavioral skills are needed. CONFLICT OF INTEREST The authors declare no conflict of interest.
REFERENCES 1 Halamek LP. Rigorous human performance assessment in neonatal–perinatal medicine. NeoReviews 2013; 14(8): e379–e386. 2 Weinstock P, Halamek LP. Teamwork during resuscitation. Pediatr Clin N Am 2008; 55(4): 1011–1024. 3 Halamek LP. The simulated delivery-room environment as the future modality for acquiring and maintaining skills in fetal and neonatal resuscitation. Semin Fetal Neonat Med 2008; 13(6): 448–453. 4 Gaba DM. Training and nontechnical skills: the politics of terminology. Simul Healthc 2011; 6(1): 8–10. 5 Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89(1): 8–18. 6 Kattwinkel J (ed). Neonatal Resuscitation (NRP) Textbook -6th edn. American Academy of Pediatrics and American Heart 2010; p18. 7 Thomas EJ, Sexton JB, Lasky RE, Helmreich RL, Crandell DS, Tyson J. Teamwork and quality during neonatal care in the delivery room. J Perinatol 2006; 26(3): 163–169. 8 Finer N, Rich W. Neonatal resuscitation for the preterm infant: evidence versus practice. J Perinat 2010; 30: S57–S66. 9 Sawyer T, Sierocka-Casteneda A, Chan D, Berg B, Lustik M, Thompson M. Deliberate practice using simulation improves neonatal resuscitation performance. Simul Healthc 2011; 6(6): 327–336.
Journal of Perinatology (2014), 781 – 786
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
786 10 Thomas EJ, Taggart B, Crandell DS, Lasky RE, Williams AL, Love LJ et al. Teaching teamwork during the Neonatal Resuscitation Program: a randomized trial. J Perinatol 2007; 27(7): 409–414. 11 Thomas EJ, Williams AL, Reichman EF, Lasky RE, Crandell DS, Taggart B. Team training in Neonatal Resuscitation Program for interns: teamwork and quality of resuscitations. Pediatrics 2010; 125(3): 539–546. 12 Cordero L, Hart BJ, Hardin R, Mahan JD, Nankervis CA. Deliberate practice improves pediatric residents' skills and team behaviors during simulated neonatal resuscitation. Clin Pediatr (Phila) 2013; 52(8): 747–752. 13 Surcouf JW, Chauvin SW, Ferry J, Yang T, Barkemeyer B. Enhancing residents' neonatal resuscitation competency through unannounced simulation-based training. Med Educ Online 2013; 21(18): 1–7. 14 Barry JS, Gibbs MD, Rosenberg A. A delivery room-focused education and deliberate practice can improve pediatric resident resuscitation training. J Perinatol 2012; 32(12): 920–926. 15 Finan E, Bismilla Z, Campbell C, Leblanc V, Jefferies A, Whyte HE. Improved procedural performance following a simulation training session may not be transferable to the clinical environment. J Perinatol 2012; 32(7): 539–544. 16 Finan E, Bismilla Z, Whyte HE, Leblanc V, McNamara PJ. High-fidelity simulator technology may not be superior to traditional low-fidelity equipment for neonatal resuscitation training. J Perinatol 2012; 32(4): 287–292. 17 Kaczorowski J, Levitt C, Hammond M, Outerbridge E, Grad R, Rothman A et al. Retention of neonatal resuscitation skills and knowledge: a randomized controlled trial. Fam Med 1998; 30(10): 705–711. 18 Lai NM, Ngim CF, Fullerton PD. Teaching medical students neonatal resuscitation: knowledge gained and retained from a brief simulation-based training workshop. Educ Health (Abingdon) 2012; 25(2): 105–110. 19 Patel J, Posencheg M, Ades A. Proficiency and retention of neonatal resuscitation skills by pediatric residents. Pediatrics 2012; 130(3): 515–521. 20 Mosley CM, Shaw BN. A longitudinal cohort study to investigate the retention of knowledge and skills following attendance on the Newborn Life support course. Arch Dis Child 2013; 98(8): 582–586.
Journal of Perinatology (2014), 781 – 786
21 Riem N, Boet S, Bould MD, Tavares W, Naik VN.. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 2012; 109(5): 723–728. 22 Dempster FN. The Spacing Effect. Am Psych 1988; 43: 627–634. 23 Sawyer T, Hara K, Thompson MW, Chan DS, Berg B. Modification of the Laerdal SimBaby to include an integrated umbilical cannulation task trainer. Simul Healthc 2009; 4(3): 174–178. 24 Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003; 37: 830–837. 25 LeFlore J, Anderson M, Michael J, Engle W, Anderson J. Comparisons of selfdirected learning versus instructor-modeled learning during a simulated clinical experience. Simul Healthc 2007; 2(3): 170–177. 26 LeFlore J, Anderson M. Alternative educational models for interdisciplinary student teams. Simul Healthc 2009; 4(3): 135–142. 27 Anderson M, LeFlore J, Anderson J. Evaluating videotaped role-modeling to teach crisis resource management principles. Clin Sim Nurs 2013; 9(9): e343–e354. 28 Bakeman R. Recommended effect size statistics for repeated measures design. Behav Res Methods 2005; 37(3): 379–384. 29 Olejnik S, Algina J.. Generalized Eta and omega squared statistics: measures of effect size for some common research designs. Psych Methods 2003; 8(4): 434–447. 30 Cohen J.. Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Earlbaum Associates: Hillsdale, NJ, USA, 1988. 31 Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists' NonTechnical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003; 90(5): 580–588. 32 Sweller J. Cognitive load during problem solving: effects on learning. Cogn Sci 1988; 12: 257–285. 33 van Merriënboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ 2010; 44: 85–93. 34 Yee B, Naik VN, Joo HS, Savoldelli GL, Chung DY, Houston PL et al. Nontechnical Skills in Anesthesia Crisis Management with Repeated Exposure to Simulationbased Education. Anesthesiology 2005; 103(2): 241–248.
© 2014 Nature America, Inc.
ORIGINAL ARTICLE
Correlations between technical skills and behavioral skills in simulated neonatal resuscitations T Sawyer1, D Leonard2, A Sierocka-Castaneda3, D Chan4 and M Thompson4 OBJECTIVE: Neonatal resuscitation requires both technical and behavioral skills. Key behavioral skills in neonatal resuscitation have been identified by the Neonatal Resuscitation Program. Correlations and interactions between technical skills and behavioral skills in neonatal resuscitation were investigated. STUDY DESIGN: Behavioral skills were evaluated via blinded video review of 45 simulated neonatal resuscitations using a validated assessment tool. These were statistically correlated with previously obtained technical skill performance data. RESULT: Technical skills and behavioral skills were strongly correlated (ρ = 0.48; P = 0.001). The strongest correlations were seen in distribution of workload (ρ = 0.60; P = 0.01), utilization of information (ρ = 0.55; P = 0.03) and utilization of resources (ρ = 0.61; P = 0.01). Teams with superior behavioral skills also demonstrated superior technical skills, and vice versa. CONCLUSION: Technical and behavioral skills were highly correlated during simulated neonatal resuscitations. Individual behavioral skill correlations are likely dependent on both intrinsic and extrinsic factors. Journal of Perinatology (2014) 34, 781–786; doi:10.1038/jp.2014.93; published online 15 May 2014
INTRODUCTION Cognitive skills, technical skills and behavioral skills are the three intrinsic elements of human performance at play during neonatal resuscitation.1–3 Cognitive skills include ‘what is in our brains and how our brain works’.1 Technical skills involve ‘things that we do with our hands’ or ‘the psychomotor techniques of particular procedures that require (among other things) manual dexterity’.1,4 Behavioral skills have been defined as ‘decision-making and team interaction processes used during the team’s management of a situation’ or ‘activities that are not merely manual procedures requiring dexterity’.4,5 In neonatal resuscitation, cognitive skills involve knowing the Neonatal Resuscitation Program (NRP) algorithm, recalling drug doses, remembering the MR SOPA (mask, reposition, suction, open mouth, pressure, airway) mnemonic, and so on. Technical skills in neonatal resuscitation include procedures such as bag-valve mask ventilation, chest compressions, intubation, umbilical cannulation and medication administration. The NRP has identified 10 key behavioral skills important in neonatal resuscitation, which include: knowledge of environment, anticipation and planning, leadership, communication, distribution of workload, attention allocation, information utilization, resource utilization, calling for help early and professional behavior.6 Behavioral skills have been correlated with improved NRP compliance and overall quality of care in the delivery room.7 As a result, some neonatal care units have instituted team training to improve the behavioral skills of their neonatal resuscitation teams.8 However, the intricate relationships, correlations and interactions between cognitive skills, technical skills and behavioral skills during neonatal resuscitation have yet to be fully defined. Multiple reports have shown that improvements can be seen in cognitive, technical and behavioral skills during neonatal resuscitation as a result of simulation-based training.9–18 The retention of
these skills has also been investigated.19,20 However, few studies have described the correlations between these different skill domains. Thomas et al.7 used video recordings of neonatal resuscitations and found weak correlations between the overall quality of neonatal resuscitation and three behavioral skills, which included communication, management and leadership. In a study of anesthesiology residents managing a simulated adult intraoperative cardiac arrest, Riem et al.21 reported a strong correlation between technical skills and behavioral skills. The strongest correlations were seen with situational awareness, team working and decision making. To the author’s knowledge, no prior study has examined the correlations between the 10 key behavioral skills taught in the NRP course with technical skills during neonatal resuscitation. To evaluate the correlation between technical skills and the 10 key NRP behavioral skills we conducted the following study. We hypothesized that a positive correlation would exist between technical skills and behavioral skills, and that certain behavioral skills would be more strongly correlated with technical skills than others. METHODS Objectives and design To evaluate correlations and interactions between technical skills and behavioral skills in neonatal resuscitation, we performed a secondary analysis of a set of neonatal resuscitation simulation videos, which were obtained as part of prior investigation on the effects of deliberate practice on neonatal resuscitation performance.9 That study followed a repeatedmeasures design, in which pediatric and family medicine residents completed a series of three high-fidelity simulations, with a facilitated debriefing after each. Objective measurements of technical skills were obtained during each of the three simulation sessions by way of blinded video review using a validated technical skills assessment tool. For the current study, we performed a second blinded review of the same video
1 Department of Pediatrics, University of Washington School of Medicine, Seattle, WA, USA; 2Sacred Heart Medical Center, Eugene, OR, USA; 3Department of Pediatrics, National Naval Medical Center, Bethesda, MD, USA and 4Department of Pediatrics, Tripler Army Medical Center, Honolulu, HI, USA. Correspondence: Dr T Sawyer, Division of Neonatology, Department of Pediatrics, University of Washington School of Medicine, 1959 NE Pacific Street, RR 539, HSB, Seattle, 98195 WA, USA. E-mail: [email protected] Received 6 December 2013; revised 1 March 2014; accepted 9 April 2014; published online 15 May 2014
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
782 set, this time using a previously validated behavioral skill assessment tool that evaluated each of the 10 key NRP behavioral skills. Statistical analysis was then performed to examine correlations and interactions between the observed technical and behavioral skills.
Participants All pediatric and family medicine residents at Tripler Army Medical Center were invited to participate in the study. Tripler Army Medical Center is a 450 bed tertiary care center in Honolulu with approximately 3000 deliveries, and 300 neonatal intensive care unit admissions per year. Participating residents were paired into teams of two, consisting of residents from the same program and year group. Each team of residents remained the same throughout the study. Scoring was based on both team and team leader performance, as applicable. Informed consent was obtained from all participants. The study protocol was approved by the Institutional Review Board at Tripler Army Medical Center.
Simulation sessions All simulations were performed in a realistic simulated delivery room equipped with the standard materials found in a typical delivery room, including an infant warmer, oxygen, suction, resuscitation equipment and medications. The residents were assisted during each simulation by one of the researchers (DC) who acted as a confederate labor and delivery nurse. During the resuscitations, the ‘nurse’ was able to perform bag-valve mask ventilation, chest compressions and draw up medications, but only with instruction from the residents. The ‘nurse’ did not provide direction in NRP or make treatment decisions. Simulations followed a distributed practice learning model and were scheduled on separate days, separated in time, to take advantage of the ‘spacing effect’. The spacing effect is a welldescribed phenomenon in educational psychology; for a given amount of study time, spaced study yields better learning results than does massed study.22 Each of the three simulations sessions were scheduled between 2 and 4 months apart. The simulator used was a Laerdal SimBaby (Laerdal Medical, Stavanger, Norway), which had been specially modified to include an integrated umbilical catheterization task trainer.23 The three neonatal resuscitation scenarios were fetal hypoxia, volume loss from placental hemorrhage and fetal sepsis, which were completed in that order by each resuscitation team. The initial condition of the infant in all simulations was limp, apneic, cyanotic and bradycardic (heart rate o 60 b.p.m.). All simulations required intubation and intravenous epinephrine. The second and third simulations also required a normal saline bolus. For standardization all simulations were timed. A maximum of 10 min was allowed for each simulation session. Each simulation session was facilitated by a single investigator (TS) who also facilitated the debriefing session that immediately followed. Discussion during the debriefings covered both technical and behavioral skills. Debriefing discussions of behavioral skills focused specifically on three key NRP behavioral skills: communication, resource utilization and leadership, in relation to the team’s performance during the simulation. For standardization, debriefing sessions were also timed and limited to 20 min.
Behavioral skill scoring Behavioral skills during each of the three neonatal resuscitation simulations were scored by two blinded reviewers (AS, DL) by way of blinded video review. The reviewers were blinded to both the resident year group and the simulation session number. Behavioral skills were scored using the Behavioral Assessment Tool (BAT). The BAT is a 5-point (0 to 4) behaviorally anchored rating scale developed specifically to measure behavioral performance in each of the 10 key NRP behavioral skills. Performance in each behavioral skills was given a score from 0 to 4, with 0 indicating ‘poor’ performance, 2 indicating ‘acceptable’ performance and 4 indicating ‘excellent’ performance. Specific and observable behaviors that describe performance at each of the five levels are provided as behavioral anchors on the BAT. An overall behavioral skill score was calculated by averaging the values of the 10 key behavioral skills. Both the validity and reliability of the BAT has been reported previously.25–27 The inter-rater reliability of the BAT, via Cronbach’s α, has ranged from 0.95 to 0.97. Rater training on the use of the BAT for this study included a didactic session explaining the use of the tool and practice exercises grading video-taped simulated resuscitations.
Clinical exposure All subjects were asked to report their actual clinical experience in neonatal resuscitation at the time of the first and third simulations. These data were used to provide a measure of clinical experience gains during the course of the study. The correlation of clinical experience gains and behavioral skill changes during the study was analyzed to account for the confounding variable of natural clinical skill acquisition during the course of the study.
Statistical analysis One-way repeated-measures analysis of variance, or Friedman repeatedmeasures analysis of variance on ranks, with Holm–Sidak, or Student–Newman–Kuels corrections, as appropriate, were used to evaluate changes in technical skills and behavioral skills from the first to the third simulation sessions. Changes in technical skills and behavioral skills were calculated based on the raw dichotomous scores on the NRPE, and the 5-point scale of the BAT. Spearman’s rank-order correlation coefficient (ρ) was used to determine correlations between technical skills and behavioral skills. Pearson’s product–moment correlation coefficient was used to measure the correlation between natural clinical experience gains during study and behavioral skill changes. To investigate the interaction of technical skills and behavioral skills the technical performance scores of teams with behavioral performance scores in the top quartile were compared to teams with behavioral performance scores in the bottom quartile using a Mann–Whitney rank-sum test or Student’s t-test, as applicable. The same analysis was performed on the behavioral performance scores of teams with technical skill performance scores in the top quartile and bottom quartiles. The educational effect size was calculated using Eta squared (η2), based on the ratio of effect to total variance derived from the repeatedmeasures analysis of variances.28,29 By convention, an η2 effect size of ⩽ 0.01 was considered 'small', η2 around 0.06 'moderate' and η2 ⩾ 0.14 'large'.30 P o0.05 was considered statistically significant. Data were analyzed using SigmaPlot 11.0 (Systat Software, San Jose, CA, USA).
Technical skill scoring Technical skills were scored as part of a prior investigation using a validated scoring instrument called the Neonatal Resuscitation Performance Evaluation (NRPE).9 The NRPE provides a dichotomous (yes = 1, no = 0) score for 30 technical skill metrics in neonatal resuscitation, which are divided into seven subdomains: preparation and initial steps, communication of heart rate, bag/mask ventilation, chest compressions, intubation, medication administration and umbilical vessel catheterization. The NRPE also records times to achieve key steps during resuscitation, including time to start positive-pressure ventilation, time to start compressions, time to successful intubation, time to first epinephrine dose and time to intravenous access. Data derived from the NRPE has been found to have strong evidence of validity using several lines of evidence as described by Downing,24 which included: content, consequence, response process, relationship to other variables and internal structure.9 The NRPE reliably differentiates teams with more experience in neonatal resuscitation from teams with the less experience, and prior reliability testing showed a good inter-rater reliability for the various subdomains, with a mean Cohen’s κ of 0.63.9 Journal of Perinatology (2014), 781 – 786
RESULTS Thirty-two residents, divided into 16 teams of two, participated in the study. One team was unable to complete all three simulation sessions and their performance data was omitted from the statistical analysis. Thus, the data presented here is based on the 15 teams of two residents who completed all three simulations. Table 1 provides demographic information on the study participants. The 15 teams completed the series of three simulations over a period of 9 months. Time between the simulation sessions averaged 14 weeks from the first to the second simulation sessions, and 9 weeks from the second to the third simulation sessions. A total of 45 resuscitation videos, 15 from each of the three simulations sessions, were reviewed and scored by the two blinded reviewers using the BAT. These scores were compared against the previously obtained NRPE score data. Cronbach’s α on the BAT in this study was 0.92. © 2014 Nature America, Inc.
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
783 Table 1.
Demographic data (n = 30)
Residency program Pediatrics Family medicine
22 (73%) 8 (27%)
Sex Male Female
15 (50%) 15 (50%)
Postgraduate year 1 2 3
8 (27%) 12 (40%) 10 (33%)
Number of times NRP course taken 1 2 3
21 (70%) 8 (27%) 1 (3%)
Time since last NRP course (months) 0–3 3–6 6–12 >12
14 7 2 7
(47%) (23%) (7%) (23%)
Abbreviation: NRP, Neonatal Resuscitation Program.
Table 2.
Changes in key behavioral skills in neonatal resuscitation Simulation session number 1
Knowledge of environment, mean (s.d.) Anticipation and planning Assumption of leadership role Communication among team members Distribution of workload Attention allocation Utilization of information Utilization of resources Calling for help Professional behavior Overall
2
P-value
3
2.4 (0.8) 2.7 (0.7) 2.7 (0.9)
0.12
2.1 (0.9) 2.6 (0.9) 2.7 (0.9) 1.8 (0.8) 2.0 (0.8) 2.0 (0.9) 2.3 (0.9) 2.6 (0.9) 2.8 (1.0)
0.004 0.54 0.03
2.1 1.9 1.9 2.1 1.5 2.6 2.1
(0.7) (0.9) (0.9) (0.8) (0.9) (0.7) (0.8)
2.6 2.2 2.4 2.2 2.0 2.8 2.4
(0.7) (1.0) (1.0) (0.9) (1.1) (0.8) (0.9)
2.8 2.3 2.4 2.7 2.1 3.1 2.6
(0.7) o0.001 (1.0) 0.13 (1.0) 0.03 (0.9) 0.01 (0.9) 0.01 (0.8) 0.02 (0.9) o0.001
Scores ranged from 0 to 4, with 0 indicating ‘poor’ performance, 2 indicating ‘acceptable’ performance, and 4 indicating ‘excellent’ performance.
Overall scores for both technical and behavioral skills increased from the first simulation to the third simulation (technical: F (2,449) = 11.8; P o0.001; behavioral: F (2,149) = 27.2, P o0.001). Behavioral scores improved from the first to the third simulation in all three resident year groups (P o0.001). Third-year residents scored higher in behavioral skill performance than first-year residents on the first simulation (overall BAT scores: Sim 1, 3rd year = 2.6 ± 0.8 vs 1st year = 1.7 ± 0.7; P o0.001), but scores were not different by the third simulation (overall BAT scores: Sim 3, 3rd year = 2.8 ± 0.9 vs 1st year = 2.6 ± 0.9; P = 0.12). Table 2 shows the changes in each of the 10 key NRP behavioral skills from the first to the third simulation. There was no association between clinical experience gained during the course of the study and changes in overall behavioral skill performance from the first to third simulation (r = 0.09; P = 0.76). There was a strong positive correlation between overall technical skill and behavioral skill performance scores (ρ = 0.48; © 2014 Nature America, Inc.
Figure 1. Correlation between overall technical and behavioral skill scores in 45 simulated neonatal resuscitations. Behavioral Assessment Tool (BAT), Neonatal Resuscitation Performance Evaluation (NRPE).
Table 3. Correlations between overall technical skills and the 10 key behavioral skills in neonatal resuscitation
Knowledge of the environment Anticipation of and planning Assumption of leadership role Communication among team members Distribution of workload Attention allocation Utilization of information Utilization of resources Call for help early enough Professional behavior Overall
ρ
P-value
0.33 0.38 0.29 0.32 0.60 0.42 0.55 0.61 0.25 0.45 0.48
0.21 0.15 0.27 0.23 0.01 0.11 0.03 0.01 0.36 0.08 0.001
Abbreviation: NRP, Neonatal Resuscitation Program.
P = 0.001) (Figure 1). All 10 key NRP behavioral skills positively correlated with overall technical skill, with correlations ranging from 0.25 to 0.61. The strongest correlations were seen in distribution of workload, utilization of information and utilization of resources (Table 3). There was a positive correlation between resident year group and overall technical (ρ = 0.32; P = 0.03) and behavioral (ρ = 0.42; P = 0.004) skill scores. There was also a positive correlation between simulation session number and technical and behavioral skill scores (technical: ρ = 0.33; P = 0.03; behavioral: ρ = 0.29; P = 0.05). Correlations between times to achieve key steps in resuscitation and overall behavioral skills are provided in Table 4. There was a significant difference in the technical performance scores between teams with behavioral performance scores in the top and bottom quartile (95.5 ± 5.7 vs 83.3 ± 12.2; P = 0.005). There was also a significant difference in the behavioral performance scores between teams with technical performance scores in the top and bottom quartile (3.0 ± 0.7 vs 1.9 ± 0.4; P ⩽ 0.001). The educational effect size of participation in the three simulations on technical and behavioral skills was large (technical: η2 = 0.42; behavioral: η2 = 0.53). DISCUSSION Performance during neonatal resuscitation is dependent on both the intrinsic performance of the resuscitation team members and the influences of extrinsic, or environmental, factors.1 The three Journal of Perinatology (2014), 781 – 786
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
784 Table 4.
Correlations between overall behavioral skills and time to achieve key steps in neonatal resuscitation
Start of positive pressure ventilation Start of compressions Successful intubation First epinephrine dose Emergency UVC access
ρ
P-value
−0.51 −0.25 −0.49 −0.49 −0.44
o0.001 0.1 0.002 0.001 0.002
Abbreviation: UVC, umbilical vessel catheterization.
intrinsic elements of human performances are cognitive skills, technical skills and behavioral skills.1–3 Using this paradigm, intrinsic human performance can be described by the following formula: cognitive skills+technical skills+behavioral skills = intrinsic performance. Overall resuscitation performance is then derived from intrinsic performance plus or minus the effects of extrinsic factors. As seen in the current study, and in multiple prior investigations, intrinsic performance in neonatal resuscitation can be improved through simulation-based training. What remains to be elucidated are the intricacies of the interactions between the three elements of human performances and the influence that extrinsic or environmental characteristics play. In the current study, over the course of three simulation sessions, scores for both technical and behavioral skills improved from baseline. There was a strong positive correlation between overall technical skills and behavioral skills. The key NRP behavioral skills most strongly associated with overall technical skills were distribution of workload, utilization of information and utilization of resources. There were significant negative correlation between behavioral skills and time to achieve key steps in resuscitation, indicating that a higher level of behavioral skill was associated with a decreased time to reach that step in resuscitation. Improvements in behavioral skills were not associated with clinical experience gained during the course of the study, which suggests that the improvements seen were the result of the simulation training, rather than natural skills acquisition from clinical exposure. Teams with the highest level of technical skills also demonstrated the highest level of behavioral skills, and vice versa, which highlights the intricate relationship between those two elements of human performance. Our study adds to, and expands upon, the existing literature on the interactions between technical skills and behavioral skills during neonatal resuscitation. In the only other study on this topic, Thomas et al.7 investigated teamwork and quality of care during delivery room resuscitation, and found weak correlations between quality of care and three teamwork behaviors: communication, management and leadership. The three teamwork behaviors identified by Thomas et al. were based on factor loading of nine different variables of teamwork within the three areas. The quality of resuscitation was based on adherence to NRP guidelines during resuscitation, as determined by blinded video review of actual delivery room resuscitation videos. The majority of the 132 videos reviewed in that study were of residents and nurses taking care of ‘relatively healthy’ infants, with only 8% undergoing intubation, 1.5% receiving chest compressions and one (0.75%) receiving epinephrine. Correlations between the three teamwork behaviors and the quality of resuscitation in that study ranged from ρ = 0.201 to 0.288 (P = 0.02 and o 0.001, respectively). Unlike the relatively healthy infants in the Thomas et al. study, the simulated infants in the current study were extremely ill and required extensive resuscitation efforts including intubation, chest compressions, umbilical vessel catheterization and medication administration. The differences in patient acuity may provide some explanation Journal of Perinatology (2014), 781 – 786
for the differences in correlations between technical skills and behavioral skills between to the two studies. Differences in statistical significance may be a result of the much larger number of resuscitations videos scored in the Thomas et al. study versus the current study (132 vs 45). The strong correlation between behavioral skills and technical skills in our study is consistent with reports of adult resuscitation simulations. In a retrospective analysis of previously recorded videos of anesthesiology residents managing a simulated intraoperative cardiac arrest, Riem et al.21 found a very similar overall correlation between technical skills and behavioral skills (R = 0.45; Po 0.05). In that study, scores on a specially developed cardiac arrest technical skills checklist were correlated with scores on the Anesthesia Non-Technical Skills scale,31 a behavioral scoring tool that evaluates behavioral skills in four behavior domains: task management, teamwork, situational awareness and decision making. The strongest correlations were seen in situational awareness, team working and decision making. Riem et al. speculated that cognitive load theory may provide an explanation for the observed relationship between technical skills and behavioral skills. Cognitive load theory is a theoretical model of cognitive architecture in which human memory is composed of ‘working’ and ‘long-term’ memory. Working memory is limited in capacity, and can only process a fixed amount of novel information at a time, while long-term memory is essentially limitless. According to cognitive load theory, due to the limitations of working memory, the human mind has adopted a strategy by which mental schemas (developed through practice and expertise) are stored in long-term memory and are accessed during times of high mental activity to reduce cognitive load by freeing up ‘space’ in the working memory.32,33 Riem et al. theorized that preexisting automated schemas (or expertise) in one domain (e.g., technical skills) reduced cognitive load, thereby freeing up cognitive resources to concentrate on the other domain (e.g., behavioral skills). They used cognitive load theory to explain why participants in their study with high technical skills also demonstrated high behavioral skills, and vice versa. Our results may provide additional evidence to support cognitive load theory. In our study, teams with superior performance in one domain (e.g., technical skill) also demonstrated superior performance in the other domain (e.g., behavioral skill), and vice versa. Cognitive load theory would posit that the high-performing teams in our study, with wellestablished technical or behavioral skills schemas, benefitted from a decrease in cognitive load, thus allowing them to concentrate their mental efforts on the other skill domain. This resulted in improved performance in both skill domains, as compared with teams without well-developed long-term memory schemas in either domain who had to expend working memory capacity on both technical and behavioral skills simultaneously. Careful interpretation of the generalizability of our results is warranted as our results were likely significantly influenced by both the intrinsic performance of the team members as well as extrinsic characteristics of the simulation scenarios. The baseline intrinsic behavioral and technical skills of the residents in the study may have also influenced our results. As seen in Table 2, the baseline behavioral skills of the teams in the study were fairly low, with most scores rating around ‘acceptable’, to less than acceptable. Future studies investigating the correlations between technical skills and behavioral skills in teams with higher levels of experience and experience in neonatal resuscitation are required to determine if the correlations reported here hold true for teams with higher levels of competency. The personnel resources in the scenarios were very limited (two physicians and one nurse to complete the entire resuscitation) in this study. We speculate that this extrinsic limitation imposed unique performance challenges on the teams. In this context, optimal sharing of work and utilization of the available limited resources were required. The significant correlations between overall technical skills and © 2014 Nature America, Inc.
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
785 the behavioral skills of distribution of workload and utilization of resources indicate that optimizing these behavioral skills resulted in improved technical performance. It is possible that the use of larger teams would have modified the correlations between technical skills and behavioral skills, and further research with larger resuscitation teams is required. The lack of improvement in the behavioral skills of knowledge of environment, attention allocation and leadership shown in Table 2 deserve attention. Knowledge of environment, according to the BAT, is defined by knowing aspects of environment, checking equipment, confirming readiness of environment, and not hesitating to ask questions when the need arises. At baseline, the teams in our study demonstrated fairly high scores in this skill, as compared with other skills, which may have made it difficult for significant improvements to be seen in this small sample. Attention allocation skills, according to the BAT, include not becoming distracted, being cognizant of details yet adequately monitors patient’s overall condition, prioritizing demands for attention and avoiding fixation errors. There were no imbedded equipment malfunctions or distractors within the scenarios to ensure practice with this specific skill. This may have contributed to a lack of significant improvement. Team leadership during the simulations was not predefined and was left to the discretion of the teams at the time of the simulation. Teams included two residents, each from the same program and year group. We speculate that this may have made it difficult for one team member to clearly establish themselves as the leaders. During the simulations, we observed a high degree of shared leadership. This shared leadership likely influenced the leadership scores on the BAT, as a clearly identified leader is a marker for excellent leadership. Additionally, the limited number of personal in the simulations made it difficult for the leader to separate themselves from being actively involved with the procedures of resuscitation (e.g., airway management, umbilical catheterization, etc.). This may have made it difficult for the reviewers to clearly identify the leader among the team members. Finally, the facilitated debriefings provided to the teams in our study focused on both technical and behavioral skills. We chose this method of debriefing as we felt it more accurately reflects a real life facilitated debriefing session, wherein the facilitator is obligated to provide feedback on both technical and behavioral skills, as applicable to team performance. In a prior report by Yee et al.34 evaluating the utility of teaching behavioral skills in anesthesia, the investigators were able to show significant improvements in all 15 domains of the Anesthesia Non-Technical Skills after a single high-fidelity simulation session. The debriefings in that study focused solely on behavioral skills. It is possible that an increased focus on behavioral skills during the debriefings in our study could have resulted in larger improvements in behavioral skills. What light does this study shed on the way we should train teams in neonatal resuscitation using simulation? First, this study highlights the benefits of behavioral skills training. Based on our finding of a high correlation between behavioral and technical skills, which is supported by cognitive load theory, focused training on behavioral skills is likely to provide significant benefits to resuscitation teams, with potential gains in both behavioral and technical skills. As seen in Table 4, behavioral skills were highly correlated with the speed in which several key steps in resuscitation were reached, which is a critical metric of resuscitation performance. Second, the study highlights the importance of objective evaluation of behavioral skills, using tools such as the BAT. This objective assessment could be used to focus behavioral skill training on specific areas of deficiency. The training could be focused by modification of the simulation scenario or through modification of extrinsic factors, such as the number of team members, or other environmental factors or constraints. Finally, our results highlight the benefits of distributed practice on simulation-based learning. In this study we were able to achieve © 2014 Nature America, Inc.
significant improvements in both technical and behavioral skills despite only 90 min of total training (10 min of simulation+20 min of debriefing×3 simulation sessions) distributed over a 9-month period. Our study has some limitations. The sample size was relatively small, thus placing our nonsignificant results at risk for type II error. A post hoc power analysis revealed that a sample size of 15 teams, with an α of 0.05, provided an 80% power to detect a 0.9 point difference in behavior scores between simulations 1 and 3, and a correlation of 0.25. Another limitation is that the NRPE combined elements of cognitive skills (decisions when to perform a procedure) and technical skills (how well the procedure was performed). Thus, the correlations described here may not be a pure reflection of the correlations between technical skills and behavioral skills, and may rather reflect the combined correlation of cognitive and technical skills with behavioral skills. However, we find it difficult to imagine a technical performance scoring method that would allow assessment of technical skills in isolation from cognitive skills during resuscitation, given the intimate association between the two. Obviously, one must know that a procedure is indicated (cognitive skills) in order to perform the procedure (technical skills). Finally, the self-reported clinical exposure data is at risk for recall bias. The strengths of this study include the use of standardized scenarios and debriefing methods and the use of blinded reviewers using reliable and valid technical and behavioral skill assessment tools. CONCLUSION We found a high degree of correlation between technical skills and behavioral skills in simulated neonatal resuscitations. The key NRP behavioral skills most strongly associated with overall technical skills in this study were distribution of workload, utilization of information and utilization of resources. Behavioral skill correlated with a faster time to reach key steps in neonatal resuscitation. Teams with superior behavioral skills had superior technical skills, and vice versa. The degree of correlation between specific behavioral skills and technical skills is likely dependent on both intrinsic and extrinsic factors. Further research investigating the impact of intrinsic and extrinsic factors on the correlations between technical skills and behavioral skills are needed. CONFLICT OF INTEREST The authors declare no conflict of interest.
REFERENCES 1 Halamek LP. Rigorous human performance assessment in neonatal–perinatal medicine. NeoReviews 2013; 14(8): e379–e386. 2 Weinstock P, Halamek LP. Teamwork during resuscitation. Pediatr Clin N Am 2008; 55(4): 1011–1024. 3 Halamek LP. The simulated delivery-room environment as the future modality for acquiring and maintaining skills in fetal and neonatal resuscitation. Semin Fetal Neonat Med 2008; 13(6): 448–453. 4 Gaba DM. Training and nontechnical skills: the politics of terminology. Simul Healthc 2011; 6(1): 8–10. 5 Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89(1): 8–18. 6 Kattwinkel J (ed). Neonatal Resuscitation (NRP) Textbook -6th edn. American Academy of Pediatrics and American Heart 2010; p18. 7 Thomas EJ, Sexton JB, Lasky RE, Helmreich RL, Crandell DS, Tyson J. Teamwork and quality during neonatal care in the delivery room. J Perinatol 2006; 26(3): 163–169. 8 Finer N, Rich W. Neonatal resuscitation for the preterm infant: evidence versus practice. J Perinat 2010; 30: S57–S66. 9 Sawyer T, Sierocka-Casteneda A, Chan D, Berg B, Lustik M, Thompson M. Deliberate practice using simulation improves neonatal resuscitation performance. Simul Healthc 2011; 6(6): 327–336.
Journal of Perinatology (2014), 781 – 786
Technical skills and behavioral skills in neonatal resuscitation T Sawyer et al
786 10 Thomas EJ, Taggart B, Crandell DS, Lasky RE, Williams AL, Love LJ et al. Teaching teamwork during the Neonatal Resuscitation Program: a randomized trial. J Perinatol 2007; 27(7): 409–414. 11 Thomas EJ, Williams AL, Reichman EF, Lasky RE, Crandell DS, Taggart B. Team training in Neonatal Resuscitation Program for interns: teamwork and quality of resuscitations. Pediatrics 2010; 125(3): 539–546. 12 Cordero L, Hart BJ, Hardin R, Mahan JD, Nankervis CA. Deliberate practice improves pediatric residents' skills and team behaviors during simulated neonatal resuscitation. Clin Pediatr (Phila) 2013; 52(8): 747–752. 13 Surcouf JW, Chauvin SW, Ferry J, Yang T, Barkemeyer B. Enhancing residents' neonatal resuscitation competency through unannounced simulation-based training. Med Educ Online 2013; 21(18): 1–7. 14 Barry JS, Gibbs MD, Rosenberg A. A delivery room-focused education and deliberate practice can improve pediatric resident resuscitation training. J Perinatol 2012; 32(12): 920–926. 15 Finan E, Bismilla Z, Campbell C, Leblanc V, Jefferies A, Whyte HE. Improved procedural performance following a simulation training session may not be transferable to the clinical environment. J Perinatol 2012; 32(7): 539–544. 16 Finan E, Bismilla Z, Whyte HE, Leblanc V, McNamara PJ. High-fidelity simulator technology may not be superior to traditional low-fidelity equipment for neonatal resuscitation training. J Perinatol 2012; 32(4): 287–292. 17 Kaczorowski J, Levitt C, Hammond M, Outerbridge E, Grad R, Rothman A et al. Retention of neonatal resuscitation skills and knowledge: a randomized controlled trial. Fam Med 1998; 30(10): 705–711. 18 Lai NM, Ngim CF, Fullerton PD. Teaching medical students neonatal resuscitation: knowledge gained and retained from a brief simulation-based training workshop. Educ Health (Abingdon) 2012; 25(2): 105–110. 19 Patel J, Posencheg M, Ades A. Proficiency and retention of neonatal resuscitation skills by pediatric residents. Pediatrics 2012; 130(3): 515–521. 20 Mosley CM, Shaw BN. A longitudinal cohort study to investigate the retention of knowledge and skills following attendance on the Newborn Life support course. Arch Dis Child 2013; 98(8): 582–586.
Journal of Perinatology (2014), 781 – 786
21 Riem N, Boet S, Bould MD, Tavares W, Naik VN.. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 2012; 109(5): 723–728. 22 Dempster FN. The Spacing Effect. Am Psych 1988; 43: 627–634. 23 Sawyer T, Hara K, Thompson MW, Chan DS, Berg B. Modification of the Laerdal SimBaby to include an integrated umbilical cannulation task trainer. Simul Healthc 2009; 4(3): 174–178. 24 Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003; 37: 830–837. 25 LeFlore J, Anderson M, Michael J, Engle W, Anderson J. Comparisons of selfdirected learning versus instructor-modeled learning during a simulated clinical experience. Simul Healthc 2007; 2(3): 170–177. 26 LeFlore J, Anderson M. Alternative educational models for interdisciplinary student teams. Simul Healthc 2009; 4(3): 135–142. 27 Anderson M, LeFlore J, Anderson J. Evaluating videotaped role-modeling to teach crisis resource management principles. Clin Sim Nurs 2013; 9(9): e343–e354. 28 Bakeman R. Recommended effect size statistics for repeated measures design. Behav Res Methods 2005; 37(3): 379–384. 29 Olejnik S, Algina J.. Generalized Eta and omega squared statistics: measures of effect size for some common research designs. Psych Methods 2003; 8(4): 434–447. 30 Cohen J.. Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Earlbaum Associates: Hillsdale, NJ, USA, 1988. 31 Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists' NonTechnical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003; 90(5): 580–588. 32 Sweller J. Cognitive load during problem solving: effects on learning. Cogn Sci 1988; 12: 257–285. 33 van Merriënboer JJ, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ 2010; 44: 85–93. 34 Yee B, Naik VN, Joo HS, Savoldelli GL, Chung DY, Houston PL et al. Nontechnical Skills in Anesthesia Crisis Management with Repeated Exposure to Simulationbased Education. Anesthesiology 2005; 103(2): 241–248.
© 2014 Nature America, Inc.
