RESEARCH ARTICLE
Evaluating an Education/Training Module to Foster Knowledge of Cockpit Weather Technology Erin A. Cobbett, Elizabeth L. Blickensderfer, and John Lanicci COBBETT EA, BLICKENSDERFER EL, LANICCI J. Evaluating an educarepresentations of areas of precipitation, meteorological tion/training module to foster knowledge of cockpit weather technolstation reports, forecasts, and satellite images on either a ogy. Aviat Space Environ Med 2014; 85:1019–25. multifunction cockpit display, or a mobile device such Background: Previous research has indicated that general aviation as a tablet computer. Many of the data-link weather (GA) pilots may use the sophisticated meteorological information available to them via a variety of Next-Generation Weather Radar (NEXRAD) products are NEXRAD based. NEXRAD is a network of based weather products in a manner that actually decreases flight safety. 159 high-resolution Doppler weather radars operated Methods: The current study examined an education/training method deby the National Weather Service (NWS). The graphical signed to enable GA pilots to use NEXRAD-based products effectively in nature of the information combined with the relatively convective weather situations. The training method was lecture combined with paper-based scenario Delivered exercises. Results: A multivariate analby Publishing Technology to: Umea Library rapid updateUniversity rate yields a potentially powerful tool to ysis of variance revealed that subjects in the condition performed IP:training 130.239.20.174 On: Mon,keep 06 Apr 2015 11:51:30 pilots apprised of current and forecasted weather significantly better than did subjects in the control conditionAerospace on several Copyright: Medical Association conditions with respect to their flight path (22). knowledge and attitude measures. Subjects in the training condition imIf the user/pilot does not have enough education and proved from a mean score of 66% to 80% on the radar-knowledge test and from 62% to 75% on the scenario-knowledge test. Discussion: Although training to understand how to use the product properly additional research is needed, these results demonstrated that pilots (24), these data-link weather products may very well can benefit from a well-designed education/training program involving cause more harm than good. Research has indicated that specific areas of aviation weather-related knowledge. at least three issues exist regarding pilot misuse of the Keywords: radar, NEXRAD, convective weather, pilot training.
R
ECENT RESEARCH paints a grim picture regarding weather-related accidents in general aviation (GA). Per year, GA incurs 88% of weather-related accidents and 733 deaths (14), and demonstrates a 61.9% overall lethality rate—a rate that remained relatively unchanged from 2000-2009 (1). GA pilots now have access to a variety of weather products delivered in real time to the cockpit and this includes Next-Generation Weather Radar (NEXRAD) based products. While these tools can improve a pilot’s situational awareness and reduce pilot workload, using these tools incorrectly can hinder the pilot and may lead to hazardous or fatal situations. We examined a training method to enable GA pilots to use NEXRAD-based products effectively in convective weather situations (i.e., thunderstorms). GA weather products and platforms run the gamut from mobile devices and laptop computers to tools designed specifically for the flight deck. Pilots can receive weather information from a variety of sources such as the radio, telephone, or the internet (15). Systems such as the EnRoute Flight Advisory Service give pilots timely and pertinent weather advisories, while a continuous broadcast of in-flight weather advisories is available from the Hazardous In-flight Weather Advisory System (15). ‘Glass cockpit’ technology provides pilots with access to products including navigation information, Temporary Flight Restrictions, aviation charts, and weather information (5). Additionally, data-link weather products, uplinked to the cockpit in real-time, allow pilots to view graphical
data link weather products: pilots not accounting for data latencies (i.e., time delays in processing/delivery), pilots using the information to make tactical decisions (i.e., decisions concerned with the 0–2 h timeframe versus strategic flight decisions that focus on a 2–10 h time frame), and pilots misinterpreting displays. Regarding the first issue, Latorella and Chamberlain (24) found that many pilots neglected to account for data latencies. At a minimum, radar scans are available every 5 min and a typical single-cell thunderstorm can last between 30 and 60 min (15). Thus, the radar image is created at least 5 min prior to the time a pilot may be viewing that image. Pilots who do not account for data latencies in the radar pictures are flying under false assumptions about cell location, cell intensity, and if convective weather is dissipating or intensifying, which can all lead to hazardous weather encounters (25,28). With respect to the second issue, despite the Federal Aviation Administration (FAA) recommendation for pilots not to use data-linked products for tactical decisionmaking (11,12,15), certain pilots may use the NEXRAD
From Embry-Riddle Aeronautical University, Daytona Beach, FL. This manuscript was received for review in May 2013. It was accepted for publication in June 2014. Address correspondence and reprint requests to: Elizabeth Blickensderfer, Ph.D., Embry-Riddle Aeronautical University, 600 S. Clyde Morris Blvd., Daytona Beach, FL 32114; [email protected]. Reprint & Copyright © by the Aerospace Medical Association, Alexandria, VA. DOI: 10.3357/ASEM.3770.2014
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1019
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. 4 subjects were women. The mean age of subjects was data to navigate their way around storms or storm sys21.2 yr (SD 5 3.0). Mean total flight hours was 257.6 (SD 5 tems regardless of the warning, which can cause them to 454.2; median 5 140 h). The highest level of pilot certififly below the FAA recommended distance from a storm cation was an air transport pilot certification, with one (15). Pilots are either uninformed or believe that the data pilot who held this certificate. There were 38 pilots who is good enough to navigate through a very small gap in had private pilot certificates, 21 pilots held commercial a storm (24,25). pilot certificates, and 9 held flight instructor certificates. Finally, differences in display resolution and other In addition, 34 pilots reported having an instrument characteristics can yield differences in pilots’ interpretarating. This study was approved in advance by the tion of the information. While all of the data comes from Embry-Riddle Aeronautical University Internal Review the NWS NEXRAD units, vendors may employ their Board for the Protection of Human Subjects, and each own processing algorithms to ‘tweak’ items in the final person provided written, informed consent before product such as colors, scales, and resolution (2). These participating. differences affect interpretation of the information. The majority of subjects had taken one or two acaBeringer and Ball (5) reported that pilots who viewed a demic courses in meteorology outside of their flight higher resolution weather overlay (2 km vs. 4 km or 8 km), training, with an average of 6.6 (SD 5 3.1) semester tended to fly closer to convective cells than did pilots credit hours. Subjects spent a mean of 10.8 h (SD 5 14.9, viewing the lower resolution displays—closer than the median 5 5) training on weather radar only. Among the Aeronautical Information Manual recommends be subjects in the study, 77% had never used NEXRADflown (i.e., within 20 miles of the storm) (15). Furtherbased products before and 9% of these subjects had more, pilots using the 2-km resolution data also took the NEXRAD available, but chose not to use it. Those who longest amount of time to make a decision about their had used NEXRAD in the cockpit previously reported flight path. by Publishing Technology to: Umea Library using it in theUniversity cockpit on average 28% (SD 5 32.9%, meEducation and trainingDelivered could provide a solution to IP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 dian 5 10%) of the time. many of the aviation safety problems associated with Copyright: Aerospace Medical Association data-link weather. For example, Ball (3) examined pilot performance in weather-intensive flight visual flight Equipment rules (VFR) scenarios when using Flight Information The study equipment consisted of the education/ Systems Data Link (FISDL). Pilots who had received training module, the control activity materials, and sevtraining on how to use FISDL effectively made weathereral measures/assessments. The Education/Training related decisions faster and kept a greater distance bemodule focused on the basics of radar as well as the tween themselves and the thunderstorm compared to functions and limitations of NEXRAD. This included the pilots who did not receive the training. While this the differences between clear-air and precipitation study shows that training can be effective for FISDL use, modes, the different stages of thunderstorms, and how additional data-link weather training programs are these stages would appear using NEXRAD imagery. needed. The training also discussed several NEXRAD products The purpose of the current study was to examine the and how to interpret NEXRAD output. The authors efficacy of a NEXRAD-based product education/traindrew from numerous sources in developing the moding program for GA pilots. The training included lecture ule content. These included NWS online weather modcombined with paper-based scenario exercises to give ules (29), the Federal Meteorological Handbook 11 subjects the opportunity to practice applying meteoroon NEXRAD (31), the NWS Distance Learning and logical information in the context of a flight scenario. It Operations Course on NEXRAD (30), FAA Advisory was expected that subjects who received the training Circulars on weather and decision making (11,13), and would demonstrate gains in knowledge, application of other publications and books on aviation weather (26). that knowledge, and attitudes regarding effective use of The instruction consisted of a lecture-based segment NEXRAD-based products as compared to subjects who and a scenario-based segment. The training session bedid not receive the training. gan with a 2.5-h, instructor-led PowerPoint7 presentation broken up into sections. In each section, the instructor METHODS discussed the learning objective, presented the educaThe experiment was a 2 3 2 mixed (between- and tional material, gave the subjects quiz questions, and prowithin-subjects) design. The between-subjects factor vided feedback regarding the answers to the questions. was training condition (training vs. control) and the The module also included two paper-based flight scewithin-subjects factor was period (pre-test vs. post-test). narios. Scenario-based training provides learners the Dependent variables were radar-knowledge test scores, opportunity to practice using necessary knowledge and scenario-based test scores, self-efficacy, and trainee skills in the context of thoughtfully constructed, realistic reactions. flight situations (6). Commercial [e.g., airline-oriented flight training (7,24,32)] and military aviation have a Subjects long history using a scenario-based approach to aviation training (19). Recent work has also demonstrated A total of 60 pilots from a southeastern U.S. univerthe effectiveness of the scenario-based approach to teach sity participated in the study. Of these, 56 were men and 1020
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. various knowledge and skills needed in GA piloting Scenario-based tests were also used. Three paper(6,9,16). based scenarios that were of parallel construction to While the scenario-based approach is typically used in those presented during the training but which covered conjunction with flight simulation or actual flight itself, the distinctly different situations were used for the pre-test current study incorporated the principles of scenario(scenario I), post-test (scenario I and scenario II), and the based training using paper-based, VFR, preflight, and inpost post-test (scenario III), respectively. For each sceflight weather scenarios. Paper-based scenarios can achieve nario, subjects responded to 15 questions that assessed some degree of cognitive fidelity as the learner must anathe subjects’ understanding of the weather information lyze the available information and consider the implicafor that particular scenario (e.g., whether there were any tions of the information for the flight (27). Each scenario nonmeteorological echoes present in any of the radar consisted of a preflight and in-flight segment and included images), as well as the implications for the flight situainformation which explained the level of urgency and extion. A percent correct score was calculated for each subtenuating circumstances of the situation. In one scenario, ject for each of the three tests. the purpose of the hypothetical flight was to attend the fuThe first attitudinal measure assessed self-efficacy, an neral of a family member the next day. The other scenario individual’s belief that he or she can succeed at a particuwas a flight to the AirVenture airshow with a limited arlar task. Self-efficacy is positively correlated with perforrival time window. The scenarios included all pertinent mance on the respective task (4). Training programs can weather information: a surface analysis, infrared satellite foster learner self-efficacy via the learner acquiring sucpicture, 850 hPa and 700 hPa wind charts, national and recessful experiences from performing the task (17,18), and gional radar mosaics, base and composite radar reflectivity researchers argue that self-efficacy is important to concharts from both departure and landing airports, radar sider when examining training effectiveness (23). Modiecho tops, surface meteorological reports, and terminal fied from Riggs et al. (33), this 10-item, 7-point Likert-style Delivered by Publishing Technology to: Umea University aerodrome forecasts. Subjects were instructed to respond questionnaire assessed Library the degree to which subjects beIP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 to a series of questions about the flightCopyright: and weather situalieved that they could use NEXRAD successfully. Based Aerospace Medical Association tion, and respond according to VFR. Subjects were also on high Cronbach’s Alpha values (a pre-test 5 0.88, a given a flight-bag ready “radar checklist” which prompted post-test 5 0.89), the 10 items were averaged together to them on key points to remember when interpreting form the pre-test self-efficacy and the post-test self-effiNEXRAD-based information. The subjects were encourcacy scores. A lower score indicated higher self-efficacy. aged to think of the checklist while responding to the Another attitudinal measure was used to assess subscenario-based questions. jects’ reactions to the training. Subjects responded to The instructor talked the subjects through the first nine items regarding their opinions of the content and scenario. This included use of an active questioning apconduct of the training module. Questions were asked proach and feedback to the subjects on their responses. on a 7-point Likert scale (1 5 strongly disagree; 7 5 In the second scenario, subjects proceeded indepenstrongly agree). Cronbach’s Alpha (a 5 0.96) indicated dently through the scenario. The instructor gave feedhigh internal consistency and the items were combined back to the subjects on their decisions upon completion into an average reactions score for each subject. of the exercise. For the control activity, the control group subjects Procedure watched three weather and aviation films, and were Subjects were randomly assigned to either the experigiven breaks between each film. The films “Tornado mental group or the control group. Both groups began the Glory,” “Weather Hazards: Thunderstorms, Wind Shear, experiment by completing the consent form, the demoand Microbursts,” and “The Deadliest Plane Crash” graphics survey, the self-efficacy questionnaire, the radar (8,20,34, respectively) were shown. knowledge pretest, and the scenario pretest (45 min total). Several measurement tools were used during the exFollowing a 10-min break, the control group went into one periment. A demographics survey obtained the subjects’ classroom and the experimental group went into a sepapilot experience, certifications, flight ratings, meteororate classroom. The groups remained in separate rooms for logical education, and familiarity with NEXRAD. To asthe 2.5-h activity (either the training module or the control sess learning, the study used a radar knowledge test and activity). After finishing the activity and a 15-min break, three scenario-based tests. Two attitudinal measures the subjects completed the self-efficacy survey, reactions (self-efficacy and trainee reactions) were also used. survey, the radar knowledge post-test, and the scenario The radar knowledge test covered radar principles, post-tests (scenarios I and II). Subjects in both groups reNEXRAD principles, NEXRAD products, thunderstorms, turned to the experimental site 3 d later and completed the and the radar checklist given in the module. The test con30-min, final scenario-based test (scenario III). sisted of true/false and multiple-choice questions for a total of 28 items. Two parallel forms of the test were used Statistical Analysis for the pre-test and post-test, respectively. The questions To test the module effectiveness, a mixed design multiwere not identical from the pre-test to the post-test; howvariate analysis of variance (SPSS MANOVA, general linever, they were designed to address the same learning ear model with repeated measures) was performed on objectives. A percent correct score was calculated for each the three dependent variables: radar knowledge, scenario subject for both the pre- and post-tests. Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1021
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. and post-test were significant for both the experimental group [F(1, 58) 5 62.7, P , 0.0001 (post-test scores were higher than the pre-test scores)] and control group [F(1, 58) 5 29.4, P , 0.0001 (post-test scores were lower than the pre-test)]. Finally, a t-test with a Bonferroni corRESULTS rection determined that there was no significant difference between the experimental and control group’s There were no significant differences between the expre-test scores [t(58) 5 0.5, P 5 0.60]. An additional t-test perimental group and the control group in age, total indicated that the post-test means were significantly difnumber of flight hours, total years flying, hours spent in ferent [t(58) 5 10.7, P , 0.0001]. weather radar training, or academic weather course Results of the follow-up univariate test showed a sigcredits earned. The results are summarized in Table I. nifi cant interaction effect on the scenario-based test The inspection of the MANOVA using Wilk’s Lambda [F(1, 58) 5 18.8, P , 0.0001, h2 5 0.3]. Because the interacshowed that the main effect of training condition was tion was significant, simple effects tests were performed significant on the combined dependent measures to inspect the difference between the means further. As [F(3, 56) 5 26.0, P , 0.0001]. The follow-up univariate seen in Table I, the subjects in the experimental group tests showed that the experimental group scored signifiscored significantly higher on the scenario test in the cantly higher than the control group on the radar knowl2 post-test than in the pre-test [F(1, 58) 5 36.3, P , 0.0001]. edge test [F(1, 58) 5 46.3, P , 0.0001, h 5 0.4], as well as 2 The control group, on the other hand, did not show sigon the scenario I test [F(1, 58) 5 15.7, P , 0.0001, h 5 nificant differences between pre- and post-tests [F(1, 58) 0.2], and the self-efficacy measure [F(1, 58) 5 20.9, 5 0.01, P 5 0.91]. Finally, a series of t-tests with a BonferP , 0.0001, h2 5 0.3]. roni correction on the interaction means determined Next, the Wilk’s Lambda showed that the main effect Delivered by Publishing Technology to: Umea University Library that was no significant difference between the exof period (pre-test vs. post-test)IP: was signifi cant on the 130.239.20.174 On: Mon, 06there Apr 2015 11:51:30 perimental and control group’s scenario pre-test scores combined dependent measures [F(3, 56) 5 9.3, P , Copyright: Aerospace Medical Association [t(58) 5 1.3, P 5 0.20] and that the experimental group 0.0001]. A follow-up univariate of period found no sigscenario post-test mean was significantly higher than nificant difference between pre- and post-test scores on the control group [t(58) 5 10.7, P , 0.0001]. radar knowledge [F(1, 58) 5 3.1, P 5 0.08]. The univariResults of the univariate test showed a significant interate follow-up for the scenario I test found that the postaction effect on self-efficacy [F(1, 58) 5 9.6, P 5 0.003, h2 test scenario I test scores were significantly higher than 5 0.10]. The means are shown in Table I (note that lower the pre-test scenario I scores [F(1, 58) 5 17.5, P , 0.0001, scores indicate higher self-efficacy and greater confih2 5 0.2]. The follow-up univariate for self-efficacy dence). Simple effects tests were performed to examine found that the post-test self-efficacy scores were signifithe difference between the means further. For the self-efcantly lower than the pre-test self-efficacy scores, where ficacy questionnaire, the experimental group had signifiF(1, 58) 5 15.4, P , 0.0001, h2 5 0.2. The use of the Wilk’s cantly higher self-efficacy in the post-test than in the Lambda showed that the interaction effect on the compre-test [F(1, 58) 5 24.7, P , 0.0001]. The control group’s bined dependent measures was significant [F(3, 56) 5 scores did not differ significantly between pre- and post32.0, P , 0.0001]. tests [F(1, 58) 5 0.30, P 5 0.56]. Finally, a series of t-tests Results of the follow-up univariate test showed a sigwith a Bonferroni correction were used for two additional nificant interaction effect on radar knowledge [F(1, 58) 5 comparisons on the interaction means. The experimental 88.9, P , 0.0001, h2 5 0.60]. Because the interaction was group’s pre-test self-efficacy was significantly higher significant, simple effects tests were performed to inspect the difference between the means further. For the than the control group’s pre-test self-efficacy [t(58) 5 radar-knowledge test, differences between the pre-test 22.4, P 5 0.02]. Likewise, the experimental group had I test, and self-efficacy. The between factor was training condition (training vs. control) and the within factor was period (pre-test vs. post-test). All subsequent univariate results reported are Greenhouse-Geisser statistics.
TABLE I. SUMMARY OF STUDY RESULTS (N 5 30 IN EACH GROUP). Measure Radar Knowledge Scenario I Self-Efficacy Scenario II Scenario III Reaction to Training
1022
Pre-Test M (SD)
Post-Test M (SD)
P (Interaction Term)
Effect Size h2
Exp. 5 66.2 (8.1) Control 5 65.0 (8.8) Exp. 5 62.0 (13.8) Control 5 57.1 (14.7) Exp. 5 3.8 (0.9) Control 5 4.5 (1.2)
Exp. 5 79.8 (7.6) Control 5 55.7 (9.8) Exp. 5 75.8 (10.7) Control 5 56.9 (15.3) Exp. 5 2.9 (0.7) Control 5 4.4 (1.1)
ⱕ0.01
0.60
ⱕ0.01
0.30
ⱕ0.01
0.10
Exp. 5 70.1 (16.5) Control 5 57.1 (12.5) Exp. 5 74.4 (10.6) Control 5 59.1 (14.9) Exp. 5 6.5 (0.4) Control 5 3.2 (1.3)
P (t-test) ⱕ0.01 ⱕ0.01 ⱕ0.01
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. in data-link weather products. These scenarios consisted of authentic meteorological events with conditions and images pulled from meteorological databases. Thus, GA pilots may encounter weather events such as these in actual flight. The experimental subjects performed better than control subjects in responding to questions about characteristics of the weather in the scenario, weather changes from preflight to during flight, and the implications for the flight. This type of weather information review and subsequent decision making is highly similar to what GA pilots experience in actual flight. Thus, it is likely that the learning gains achieved from the course will transfer to flight situations. While knowledge and skill gain are typically the primary goals of training, it is also important to assess DISCUSSION change in attitudinal variables such as self-efficacy (10,23) and trainee reactions (21). In the current study, If used effectively, NEXRAD-based weather products although the subjects in the experimental group had could be a powerful tool to aid weather-related decision higher pretest self-efficacy than did the subjects in the making in the cockpit. Prior research has indicated, control group, the self-efficacy of the control group did however, that GA pilots do not understand the nuances not change pre- to post-test. In contrast, the experimenof the weather technology products and pilots are not tal group subjects’ self-efficacy regarding their weatherusing the technology effectively (3,5). Training can help Delivered by Publishing Technology to: Umea University Library This increase is important radar-related skills increased. pilots use data-link weather successfully (3). To ensure IP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 for twoAssociation reasons. First, Davis and colleagues argued that high-quality aviation weather training, it is crucial that Medical Copyright: Aerospace when a training program provides exposure to the training developers follow established protocols during transfer setting, this gives trainees a frame of reference training product development as well as assess the for understanding the training content and, in turn, the training effectiveness of the training products (21). The trainees may actually learn more (10). This increased current research developed and tested the efficacy of a learning, in turn, is reflected in their post-training selftraining course for GA pilots on the use of NEXRADefficacy—the degree to which they are confident in apbased products in convective weather situations. The plying the knowledge they acquired in training to actual subjects who received the data-link training course flight (10). Additionally, as considerable prior research scored higher on a radar-knowledge test, a weather scehas found a positive relationship between self-efficacy nario application test, a retention test, and attitudinal and performance (4,10), the subjects who received trainmeasures about weather technology, as compared to ing will likely have a better chance at successfully using their scores on the pre-test and also in comparison to the the NEXRAD technology during actual flight. control group who did not receive training. With respect to trainee reactions, although a positive A thorough training course validation includes collectlearner reaction to a particular training program is not esing and examining gains in knowledge, skills, and attisential for learning to occur, it is desirable for the learners tudes (21,23). In terms of knowledge test results, the radar to enjoy the learning process. This may be particularly true knowledge pre-test demonstrated that the subjects had in any GA supplemental training programs that go beyond only a moderate level of knowledge of convective weather pilot certification requirements. In the current study, the and NEXRAD going into the course. In the post-test, the experimental group reacted much more favorably toward experimental group showed a large increase in radar knowledge and the control group actually declined from their training session than did the control group. Since the the pre- to post-test—a likely indication of experimental current course would be optional training and not required fatigue. All subjects were from an FAA Part 142 environfor pilot certification, positive reviews from past subjects ment with a stringent curriculum and many subjects had could motivate additional GA pilots to take the course. taken additional academic courses in meteorological Many avenues of research involving weather educatraining. Despite their strong educational background, tion and training have yet to be explored. As the typical the pilots still learned from the course. Thus, we suspect GA pilot is older than the subjects in the current study that pilots with a lower degree of weather knowledge and has more flight experience, expanding the current and aeronautical experience may benefit to an even research to incorporate GA pilots from non-university greater degree from this type of weather training. settings would provide insight into additional weather While knowledge gain is the first step of training eftraining needs. Additionally, to determine how this trainfectiveness, it is key that trainees can also apply what ing impacts pilot flight behavior, use of this training prothey learned (21,23). The scenario application tests program in conjunction with flight scenarios in flight training vided the opportunity for subjects to apply their newly devices or computer-based simulations complete with a acquired knowledge to flight weather scenarios. The simulated data-link weather source are needed. Measures scenarios included the preflight weather as well as subin a simulation study should include how and when the sequent in-flight radar images, similar to that available pilots consulted the data-link information, as well as
significantly higher self-efficacy in the post-test than did the control group [t(58) 5 6.1, P , 0.0001]. Three of the dependent measures, Scenario II, Retention/Scenario III, and Reactions, were given only during the post-test and were tested separately from the MANOVA. The results of three separate independent samples t-tests showed that the experimental group performed significantly better than the control group on Scenario II [t(58) 5 3.4, P 5 0.001], as well as 3 d later on the retention test (Scenario III) [t(55) 5 4.6, P , 0.0001)]. The results of the reactions measures showed that the experimental group responded much more positively to the training than did the control group [t(58) 5 13.1, P , 0.0001].
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1023
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. 3. Ball JD. The impact of training on general aviation pilots’ ability cognitive measures that assess understanding of the into make strategic weather-related decisions. Oklahoma formation presented. Another area of future research City, OK: FAA Civil Aerospace Medical Institute; 2008. FAA pertains to self-efficacy. Since the experimental group had Report DOT/FAA/AM-08/3. Available from: http://www. a moderate level of aviation weather self-efficacy prior to dtic.mil/cgi-bin/GetTRDoc?Location5U2&doc5GetTRDoc. pdf&AD5ADA477162. the course (as shown by the pretest), the results do not tell 4. Bandura A. Cultivate self-efficacy for personal and organizational us how participants with a lower level of self-efficacy at effectiveness. In: Locke EA, ed. Handbook of principles of the start of the course may fare in the course. It may be organization behavior. Oxford: Blackwell; 2000:120–36. 5. Beringer DB, Ball JD. The effects of NEXRAD graphical data that subjects with higher aviation weather self-efficacy resolution and direct weather viewing on pilots’ judgments would learn more when they begin the course than would of weather severity and their willingness to continue a flight. subjects with lower aviation weather self-efficacy. AddiOklahoma City, OK: FAA Civil Aerospace Medical Institute; tional research is needed to answer this. Finally, research 2004. FAA Report DOT/FAA/AM-04/5. Available from http://www.dtic.mil/cgi-bin/GetTRDoc?AD5ADA423239& regarding the assessment of weather knowledge may Location5U2&doc5GetTRDoc.pdf. also be useful. In this study, the average post-test scores 6. Blickensderfer EL, Strally S, Doherty S. The effects of scenariofor the experimental group were only moderately high based training on pilots’ use of an emergency whole-plane (79.8% on the radar knowledge and 75.8% on the scenario parachute. Int J Aviat Psychol 2012; 22:184–202. 7. Butler RE. LOFT: full-motion simulation as crew resource management knowledge). It may be that some of the assessment questraining. In: Weiner EL, Kanki BG, Helmreich RL, eds. Cockpit tions were too difficult and/or an imperfect match beresource management. San Diego, CA: Academic Press; 1993. tween the training content and the questions (i.e., 8. Cole K, Director. Tornado glory. [Documentary Film; DVD]. Angry Sky Entertainment and PBS; 2006. imperfect content validity of the test). 9. Craig PA. Evaluating pilots using a scenario-based methodology: Regarding practical applications, the current training a guide for instructors and examiners. International Journal of method may lend itself well to an online, individual Applied Aviation Studies (Oklahoma City, OK: FAA Academy). training program that gives pilots the opportunity to go 2009; 9:155–70. Delivered by Publishing Technology to: Umea University Library 10. Davis WD, Fedor DB, Parsons CK, Herold DM. The development through scenarios using an animated navigation disIP: 130.239.20.174 On: Mon, 06 Apr 2015cacy 11:51:30 of self-effi during aviation training. J Organ Behav 2000; play. Another practical application ofCopyright: this training modAerospace Medical 21:Association 857–71. ule could be to provide the training materials to training 11. Federal Aviation Administration. Use of cockpit displays of digital providers across the nation, similar to what the FAA’s weather and operational information (Advisory Circular 0063). Washington, DC: FAA, Flight Technologies and Procedures WINGS program provides to pilots through seminars. Division; 2004. Available from: http://rgl.faa.gov/Regulatory_ In conclusion, the results of this study indicate that and_Guidance_Library/rgAdvisoryCircular.nsf/list/ pilots can benefit from a well-designed training program AC%2000-63/$FILE/AC%2000-63.pdf. involving specific areas of weather-related knowledge. 12. Federal Aviation Administration. General aviation pilot’s guide to preflight weather planning, weather self-briefings, and The subjects who received training in the current study weather decision making. Washington, DC: FAA; 2009. will have a better chance at successful interpretation of Available from: http://www.faa.gov/pilots/safety/media/ NEXRAD based weather products used before and durga_weather_decision_making.pdf. 13. Federal Aviation Administration. Aviation weather services ing actual flight than had they not received this training. (Advisory Circular 00-45G). FAA: Flight Technologies and Well-designed weather training such as this can teach GA Procedures Division. Washington, DC: FAA; 2010. Available pilots to understand and use weather technology and from: http://rgl.faa.gov/Regulatory_and_Guidance_Library/ products and, in turn, to have safe and enjoyable flights. rgAdvisoryCircular.nsf/0/6a07efaa9a68c1a8862576c70052e9e ACKNOWLEDGMENTS This research was conducted under sponsorship of the Federal Aviation Administration‘s (FAA) Center for Excellence in General Aviation Research (CGAR) under FAA Cooperative Agreement Number 07-C-GA-ERAU-014. The views expressed in this paper are those of the authors and do not represent those of the organization with which they are affiliated or the FAA. The authors wish to thank Jon Dziok for helping to develop the training program, Julian Archer for assisting in data collection, the research subjects, and Shannon Cummings and Michael Vincent for assistance in preparing the manuscript. Authors and affiliations: Erin A. Cobbett, M.S.A., Georgia Perimeter College, Tucker, GA (formerly from Applied Aviation Sciences, EmbryRiddle Aeronautical University, Daytona Beach, FL); Elizabeth L. Blickensderfer, Ph.D., Human-Factors and Systems, and John Lanicci, Ph.D., Graduate Studies, Embry-Riddle Aeronautical University, Daytona Beach, FL. REFERENCES 1. Aircraft Owners and Pilots Association. 2010 Nall Report: accident trends and factors for 2009. Frederick, MD: AOPA Air Safety Foundation; 2010. Available from: http://www.aopa.org/-/ media/Files/AOPA/Home/News/All%20News/2012/ April/VFR%20in%20to%20IMC%20Learn%20to%20escape%20 the%20odds/10nall.pdf. 2. ATSC. Capabilities report: weather technology in the cockpit program. Norman, OK: Atmospheric Technology Services Company, LLC; 2010.
1024
5/$FILE/AC-0045G.pdf. 14. Federal Aviation Administration, Aviation Safety Information Analysis and Sharing. Weather-related aviation accident study 2003-2007. Washington, DC: FAA; 2010. Available from: http://www.asias.faa.gov/pls/apex/f?p5100:8:0::NO::P8_ STDY_VAR:2. 15. Federal Aviation Administration. Aeronautical information manual, chapter 7, safety of flight, meteorology. Washington, DC: United States Department of Transportation; 2011. Available from: http://www.faa.gov/air_traffic/publications/ATpubs/ AIM/aim0701.html#aim0701.html.1. 16. French J, Summers M, Ayers F, Blickensderfer E. The effectiveness of scenario based training for technically advanced aircraft [Abstract #521]. Aviat Space Environ Med 2007; 78:307. 17. Gist ME, Mitchell TR. Self-efficacy: a theoretical analysis of its determinants and malleability. Acad Manage Rev 1992; 17: 183–221. 18. Gist ME, Schwoerer C, Rosen B. Effects of alternative training methods on self-efficacy and performance in computer software training. J Appl Psychol 1990; 74:884–91. 19. Jentsch F, Curtis M, Salas E, eds. Simulation in aviation training. Burlington, VT: Ashgate; 2011. 20. Jeppesen Sanderson, Inc., Producers. Weather hazards: thunderstorms, wind shear, and microbursts [Documentary film; VHS recording]. Englewood, CO: Jeppesen Sanderson, Inc.; 1990. 21. Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels, 3rd ed. San Francisco: Berrett-Koehler Store; 1998. 22. Knecht WR. Use of weather information by general aviation pilots, part 1, quantitative: reported use and value of providers and
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL.
23. 24.
25.
26. 27.
products. Oklahoma City, OK: FAA Civil Aerospace Medical Institute; 2008. FAA Report DOT/FAA/AM-08/6. Retrieved October 2013 from http://www.dtic.mil/cgi-bin/GetTRDoc? Location5U2&doc5GetTRDoc.pdf&AD5ADA481118. Kraiger K, Ford JK, Salas E. Integration of cognitive, behavioral, and affective theories of learning into new methods of training evaluation. J Appl Psychol 1993;78:311–28. Latorella KA, Chamberlain JP. Graphical weather information system evaluation: usability, perceived utility, and preferences from general aviation pilots. Hampton, VA: NASA Langley Research Center; 2002. Paper Number: 2002-01-1521. Available from: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/ 20030013637_2003014788.pdf. Latorella KA, Chamberlain JP. Tactical vs. strategic behavior: general aviation piloting in convective weather scenarios. In: Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting; Sept. 30-Oct. 4, 2002; Baltimore, MD. Santa Monica, CA: HFES, 2002:101–5. Available from http:// ntrs.nasa.gov/search.jsp?R=20030014193&hterms=Latorella& qs=Ntx%3Dmode%2520matchallpartial%26Ntk%3DAll%26N %3D0%26No%3D10%26Ntt%3DLatorella. Lester P. Aviation weather. Englewood, CO: Jeppesen Sanderson, Inc.; 2007. Liu D, Macchiarella ND, Vincenzi DA. Simulation fidelity. In: Vincenzi DA, Wise JA, Mouloua M, Hancock PA. Human factors in simulation and training. Boca Raton, FL: CRC Press/ Taylor & Francis; 2008:61–73.
28. National Transportation Safety Board. In-cockpit NEXRAD mosaic imagery. Washington, DC: NTSB; 2012. Available from: http://www.ntsb.gov/doclib/safetyalerts/SA_017.pdf. 29. Jetstream Online School for Weather [Internet]. 2010; Accessed October 2013 from: http://www.srh.noaa.gov/jetstream// doppler/doppler_intro.htm. 30. National Weather Service Warning Decision Training Branch. Principles of meteorological Doppler radar, distance learning and operations course [Internet]. 2010; Accessed October 2013 from: http://www.wdtb.noaa.gov/courses/dloc/topic3/lesson1/ index.html. 31. Office of the Federal Coordinator for Meteorological Services and Supporting Research (OFCM). Federal meteorological handbook No. 11(FCM-H11A-2009). Washington, DC: National Oceanic and Atmospheric Administration; 2009. Available from: http://www.ofcm.gov/fmh11/fmh11parta/pdf/FMH-11%20 Pt%20A%20Feb%202013%20All.pdf. 32. Orlady HW, Orlady LM. Human factors in multi-crew flight operations. Burlington, VT: Ashgate; 1999. 33. Riggs ML, Warka J, Babasa B, Betancourt R, Hooker S. Development and validation of self-efficacy and outcome expectancy scales for job-related applications. Educ Psychol Meas 1994; 54:793–802. Retrieved from http://epm.sagepub. com/content/54/3/793. 34. Hebert C, Director. The deadliest plane crash [Documentary film; DVD]. Arlington, VA: NOVA; 2006.
Delivered by Publishing Technology to: Umea University Library IP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 Copyright: Aerospace Medical Association
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1025
Evaluating an Education/Training Module to Foster Knowledge of Cockpit Weather Technology Erin A. Cobbett, Elizabeth L. Blickensderfer, and John Lanicci COBBETT EA, BLICKENSDERFER EL, LANICCI J. Evaluating an educarepresentations of areas of precipitation, meteorological tion/training module to foster knowledge of cockpit weather technolstation reports, forecasts, and satellite images on either a ogy. Aviat Space Environ Med 2014; 85:1019–25. multifunction cockpit display, or a mobile device such Background: Previous research has indicated that general aviation as a tablet computer. Many of the data-link weather (GA) pilots may use the sophisticated meteorological information available to them via a variety of Next-Generation Weather Radar (NEXRAD) products are NEXRAD based. NEXRAD is a network of based weather products in a manner that actually decreases flight safety. 159 high-resolution Doppler weather radars operated Methods: The current study examined an education/training method deby the National Weather Service (NWS). The graphical signed to enable GA pilots to use NEXRAD-based products effectively in nature of the information combined with the relatively convective weather situations. The training method was lecture combined with paper-based scenario Delivered exercises. Results: A multivariate analby Publishing Technology to: Umea Library rapid updateUniversity rate yields a potentially powerful tool to ysis of variance revealed that subjects in the condition performed IP:training 130.239.20.174 On: Mon,keep 06 Apr 2015 11:51:30 pilots apprised of current and forecasted weather significantly better than did subjects in the control conditionAerospace on several Copyright: Medical Association conditions with respect to their flight path (22). knowledge and attitude measures. Subjects in the training condition imIf the user/pilot does not have enough education and proved from a mean score of 66% to 80% on the radar-knowledge test and from 62% to 75% on the scenario-knowledge test. Discussion: Although training to understand how to use the product properly additional research is needed, these results demonstrated that pilots (24), these data-link weather products may very well can benefit from a well-designed education/training program involving cause more harm than good. Research has indicated that specific areas of aviation weather-related knowledge. at least three issues exist regarding pilot misuse of the Keywords: radar, NEXRAD, convective weather, pilot training.
R
ECENT RESEARCH paints a grim picture regarding weather-related accidents in general aviation (GA). Per year, GA incurs 88% of weather-related accidents and 733 deaths (14), and demonstrates a 61.9% overall lethality rate—a rate that remained relatively unchanged from 2000-2009 (1). GA pilots now have access to a variety of weather products delivered in real time to the cockpit and this includes Next-Generation Weather Radar (NEXRAD) based products. While these tools can improve a pilot’s situational awareness and reduce pilot workload, using these tools incorrectly can hinder the pilot and may lead to hazardous or fatal situations. We examined a training method to enable GA pilots to use NEXRAD-based products effectively in convective weather situations (i.e., thunderstorms). GA weather products and platforms run the gamut from mobile devices and laptop computers to tools designed specifically for the flight deck. Pilots can receive weather information from a variety of sources such as the radio, telephone, or the internet (15). Systems such as the EnRoute Flight Advisory Service give pilots timely and pertinent weather advisories, while a continuous broadcast of in-flight weather advisories is available from the Hazardous In-flight Weather Advisory System (15). ‘Glass cockpit’ technology provides pilots with access to products including navigation information, Temporary Flight Restrictions, aviation charts, and weather information (5). Additionally, data-link weather products, uplinked to the cockpit in real-time, allow pilots to view graphical
data link weather products: pilots not accounting for data latencies (i.e., time delays in processing/delivery), pilots using the information to make tactical decisions (i.e., decisions concerned with the 0–2 h timeframe versus strategic flight decisions that focus on a 2–10 h time frame), and pilots misinterpreting displays. Regarding the first issue, Latorella and Chamberlain (24) found that many pilots neglected to account for data latencies. At a minimum, radar scans are available every 5 min and a typical single-cell thunderstorm can last between 30 and 60 min (15). Thus, the radar image is created at least 5 min prior to the time a pilot may be viewing that image. Pilots who do not account for data latencies in the radar pictures are flying under false assumptions about cell location, cell intensity, and if convective weather is dissipating or intensifying, which can all lead to hazardous weather encounters (25,28). With respect to the second issue, despite the Federal Aviation Administration (FAA) recommendation for pilots not to use data-linked products for tactical decisionmaking (11,12,15), certain pilots may use the NEXRAD
From Embry-Riddle Aeronautical University, Daytona Beach, FL. This manuscript was received for review in May 2013. It was accepted for publication in June 2014. Address correspondence and reprint requests to: Elizabeth Blickensderfer, Ph.D., Embry-Riddle Aeronautical University, 600 S. Clyde Morris Blvd., Daytona Beach, FL 32114; [email protected]. Reprint & Copyright © by the Aerospace Medical Association, Alexandria, VA. DOI: 10.3357/ASEM.3770.2014
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1019
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. 4 subjects were women. The mean age of subjects was data to navigate their way around storms or storm sys21.2 yr (SD 5 3.0). Mean total flight hours was 257.6 (SD 5 tems regardless of the warning, which can cause them to 454.2; median 5 140 h). The highest level of pilot certififly below the FAA recommended distance from a storm cation was an air transport pilot certification, with one (15). Pilots are either uninformed or believe that the data pilot who held this certificate. There were 38 pilots who is good enough to navigate through a very small gap in had private pilot certificates, 21 pilots held commercial a storm (24,25). pilot certificates, and 9 held flight instructor certificates. Finally, differences in display resolution and other In addition, 34 pilots reported having an instrument characteristics can yield differences in pilots’ interpretarating. This study was approved in advance by the tion of the information. While all of the data comes from Embry-Riddle Aeronautical University Internal Review the NWS NEXRAD units, vendors may employ their Board for the Protection of Human Subjects, and each own processing algorithms to ‘tweak’ items in the final person provided written, informed consent before product such as colors, scales, and resolution (2). These participating. differences affect interpretation of the information. The majority of subjects had taken one or two acaBeringer and Ball (5) reported that pilots who viewed a demic courses in meteorology outside of their flight higher resolution weather overlay (2 km vs. 4 km or 8 km), training, with an average of 6.6 (SD 5 3.1) semester tended to fly closer to convective cells than did pilots credit hours. Subjects spent a mean of 10.8 h (SD 5 14.9, viewing the lower resolution displays—closer than the median 5 5) training on weather radar only. Among the Aeronautical Information Manual recommends be subjects in the study, 77% had never used NEXRADflown (i.e., within 20 miles of the storm) (15). Furtherbased products before and 9% of these subjects had more, pilots using the 2-km resolution data also took the NEXRAD available, but chose not to use it. Those who longest amount of time to make a decision about their had used NEXRAD in the cockpit previously reported flight path. by Publishing Technology to: Umea Library using it in theUniversity cockpit on average 28% (SD 5 32.9%, meEducation and trainingDelivered could provide a solution to IP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 dian 5 10%) of the time. many of the aviation safety problems associated with Copyright: Aerospace Medical Association data-link weather. For example, Ball (3) examined pilot performance in weather-intensive flight visual flight Equipment rules (VFR) scenarios when using Flight Information The study equipment consisted of the education/ Systems Data Link (FISDL). Pilots who had received training module, the control activity materials, and sevtraining on how to use FISDL effectively made weathereral measures/assessments. The Education/Training related decisions faster and kept a greater distance bemodule focused on the basics of radar as well as the tween themselves and the thunderstorm compared to functions and limitations of NEXRAD. This included the pilots who did not receive the training. While this the differences between clear-air and precipitation study shows that training can be effective for FISDL use, modes, the different stages of thunderstorms, and how additional data-link weather training programs are these stages would appear using NEXRAD imagery. needed. The training also discussed several NEXRAD products The purpose of the current study was to examine the and how to interpret NEXRAD output. The authors efficacy of a NEXRAD-based product education/traindrew from numerous sources in developing the moding program for GA pilots. The training included lecture ule content. These included NWS online weather modcombined with paper-based scenario exercises to give ules (29), the Federal Meteorological Handbook 11 subjects the opportunity to practice applying meteoroon NEXRAD (31), the NWS Distance Learning and logical information in the context of a flight scenario. It Operations Course on NEXRAD (30), FAA Advisory was expected that subjects who received the training Circulars on weather and decision making (11,13), and would demonstrate gains in knowledge, application of other publications and books on aviation weather (26). that knowledge, and attitudes regarding effective use of The instruction consisted of a lecture-based segment NEXRAD-based products as compared to subjects who and a scenario-based segment. The training session bedid not receive the training. gan with a 2.5-h, instructor-led PowerPoint7 presentation broken up into sections. In each section, the instructor METHODS discussed the learning objective, presented the educaThe experiment was a 2 3 2 mixed (between- and tional material, gave the subjects quiz questions, and prowithin-subjects) design. The between-subjects factor vided feedback regarding the answers to the questions. was training condition (training vs. control) and the The module also included two paper-based flight scewithin-subjects factor was period (pre-test vs. post-test). narios. Scenario-based training provides learners the Dependent variables were radar-knowledge test scores, opportunity to practice using necessary knowledge and scenario-based test scores, self-efficacy, and trainee skills in the context of thoughtfully constructed, realistic reactions. flight situations (6). Commercial [e.g., airline-oriented flight training (7,24,32)] and military aviation have a Subjects long history using a scenario-based approach to aviation training (19). Recent work has also demonstrated A total of 60 pilots from a southeastern U.S. univerthe effectiveness of the scenario-based approach to teach sity participated in the study. Of these, 56 were men and 1020
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. various knowledge and skills needed in GA piloting Scenario-based tests were also used. Three paper(6,9,16). based scenarios that were of parallel construction to While the scenario-based approach is typically used in those presented during the training but which covered conjunction with flight simulation or actual flight itself, the distinctly different situations were used for the pre-test current study incorporated the principles of scenario(scenario I), post-test (scenario I and scenario II), and the based training using paper-based, VFR, preflight, and inpost post-test (scenario III), respectively. For each sceflight weather scenarios. Paper-based scenarios can achieve nario, subjects responded to 15 questions that assessed some degree of cognitive fidelity as the learner must anathe subjects’ understanding of the weather information lyze the available information and consider the implicafor that particular scenario (e.g., whether there were any tions of the information for the flight (27). Each scenario nonmeteorological echoes present in any of the radar consisted of a preflight and in-flight segment and included images), as well as the implications for the flight situainformation which explained the level of urgency and extion. A percent correct score was calculated for each subtenuating circumstances of the situation. In one scenario, ject for each of the three tests. the purpose of the hypothetical flight was to attend the fuThe first attitudinal measure assessed self-efficacy, an neral of a family member the next day. The other scenario individual’s belief that he or she can succeed at a particuwas a flight to the AirVenture airshow with a limited arlar task. Self-efficacy is positively correlated with perforrival time window. The scenarios included all pertinent mance on the respective task (4). Training programs can weather information: a surface analysis, infrared satellite foster learner self-efficacy via the learner acquiring sucpicture, 850 hPa and 700 hPa wind charts, national and recessful experiences from performing the task (17,18), and gional radar mosaics, base and composite radar reflectivity researchers argue that self-efficacy is important to concharts from both departure and landing airports, radar sider when examining training effectiveness (23). Modiecho tops, surface meteorological reports, and terminal fied from Riggs et al. (33), this 10-item, 7-point Likert-style Delivered by Publishing Technology to: Umea University aerodrome forecasts. Subjects were instructed to respond questionnaire assessed Library the degree to which subjects beIP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 to a series of questions about the flightCopyright: and weather situalieved that they could use NEXRAD successfully. Based Aerospace Medical Association tion, and respond according to VFR. Subjects were also on high Cronbach’s Alpha values (a pre-test 5 0.88, a given a flight-bag ready “radar checklist” which prompted post-test 5 0.89), the 10 items were averaged together to them on key points to remember when interpreting form the pre-test self-efficacy and the post-test self-effiNEXRAD-based information. The subjects were encourcacy scores. A lower score indicated higher self-efficacy. aged to think of the checklist while responding to the Another attitudinal measure was used to assess subscenario-based questions. jects’ reactions to the training. Subjects responded to The instructor talked the subjects through the first nine items regarding their opinions of the content and scenario. This included use of an active questioning apconduct of the training module. Questions were asked proach and feedback to the subjects on their responses. on a 7-point Likert scale (1 5 strongly disagree; 7 5 In the second scenario, subjects proceeded indepenstrongly agree). Cronbach’s Alpha (a 5 0.96) indicated dently through the scenario. The instructor gave feedhigh internal consistency and the items were combined back to the subjects on their decisions upon completion into an average reactions score for each subject. of the exercise. For the control activity, the control group subjects Procedure watched three weather and aviation films, and were Subjects were randomly assigned to either the experigiven breaks between each film. The films “Tornado mental group or the control group. Both groups began the Glory,” “Weather Hazards: Thunderstorms, Wind Shear, experiment by completing the consent form, the demoand Microbursts,” and “The Deadliest Plane Crash” graphics survey, the self-efficacy questionnaire, the radar (8,20,34, respectively) were shown. knowledge pretest, and the scenario pretest (45 min total). Several measurement tools were used during the exFollowing a 10-min break, the control group went into one periment. A demographics survey obtained the subjects’ classroom and the experimental group went into a sepapilot experience, certifications, flight ratings, meteororate classroom. The groups remained in separate rooms for logical education, and familiarity with NEXRAD. To asthe 2.5-h activity (either the training module or the control sess learning, the study used a radar knowledge test and activity). After finishing the activity and a 15-min break, three scenario-based tests. Two attitudinal measures the subjects completed the self-efficacy survey, reactions (self-efficacy and trainee reactions) were also used. survey, the radar knowledge post-test, and the scenario The radar knowledge test covered radar principles, post-tests (scenarios I and II). Subjects in both groups reNEXRAD principles, NEXRAD products, thunderstorms, turned to the experimental site 3 d later and completed the and the radar checklist given in the module. The test con30-min, final scenario-based test (scenario III). sisted of true/false and multiple-choice questions for a total of 28 items. Two parallel forms of the test were used Statistical Analysis for the pre-test and post-test, respectively. The questions To test the module effectiveness, a mixed design multiwere not identical from the pre-test to the post-test; howvariate analysis of variance (SPSS MANOVA, general linever, they were designed to address the same learning ear model with repeated measures) was performed on objectives. A percent correct score was calculated for each the three dependent variables: radar knowledge, scenario subject for both the pre- and post-tests. Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1021
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. and post-test were significant for both the experimental group [F(1, 58) 5 62.7, P , 0.0001 (post-test scores were higher than the pre-test scores)] and control group [F(1, 58) 5 29.4, P , 0.0001 (post-test scores were lower than the pre-test)]. Finally, a t-test with a Bonferroni corRESULTS rection determined that there was no significant difference between the experimental and control group’s There were no significant differences between the expre-test scores [t(58) 5 0.5, P 5 0.60]. An additional t-test perimental group and the control group in age, total indicated that the post-test means were significantly difnumber of flight hours, total years flying, hours spent in ferent [t(58) 5 10.7, P , 0.0001]. weather radar training, or academic weather course Results of the follow-up univariate test showed a sigcredits earned. The results are summarized in Table I. nifi cant interaction effect on the scenario-based test The inspection of the MANOVA using Wilk’s Lambda [F(1, 58) 5 18.8, P , 0.0001, h2 5 0.3]. Because the interacshowed that the main effect of training condition was tion was significant, simple effects tests were performed significant on the combined dependent measures to inspect the difference between the means further. As [F(3, 56) 5 26.0, P , 0.0001]. The follow-up univariate seen in Table I, the subjects in the experimental group tests showed that the experimental group scored signifiscored significantly higher on the scenario test in the cantly higher than the control group on the radar knowl2 post-test than in the pre-test [F(1, 58) 5 36.3, P , 0.0001]. edge test [F(1, 58) 5 46.3, P , 0.0001, h 5 0.4], as well as 2 The control group, on the other hand, did not show sigon the scenario I test [F(1, 58) 5 15.7, P , 0.0001, h 5 nificant differences between pre- and post-tests [F(1, 58) 0.2], and the self-efficacy measure [F(1, 58) 5 20.9, 5 0.01, P 5 0.91]. Finally, a series of t-tests with a BonferP , 0.0001, h2 5 0.3]. roni correction on the interaction means determined Next, the Wilk’s Lambda showed that the main effect Delivered by Publishing Technology to: Umea University Library that was no significant difference between the exof period (pre-test vs. post-test)IP: was signifi cant on the 130.239.20.174 On: Mon, 06there Apr 2015 11:51:30 perimental and control group’s scenario pre-test scores combined dependent measures [F(3, 56) 5 9.3, P , Copyright: Aerospace Medical Association [t(58) 5 1.3, P 5 0.20] and that the experimental group 0.0001]. A follow-up univariate of period found no sigscenario post-test mean was significantly higher than nificant difference between pre- and post-test scores on the control group [t(58) 5 10.7, P , 0.0001]. radar knowledge [F(1, 58) 5 3.1, P 5 0.08]. The univariResults of the univariate test showed a significant interate follow-up for the scenario I test found that the postaction effect on self-efficacy [F(1, 58) 5 9.6, P 5 0.003, h2 test scenario I test scores were significantly higher than 5 0.10]. The means are shown in Table I (note that lower the pre-test scenario I scores [F(1, 58) 5 17.5, P , 0.0001, scores indicate higher self-efficacy and greater confih2 5 0.2]. The follow-up univariate for self-efficacy dence). Simple effects tests were performed to examine found that the post-test self-efficacy scores were signifithe difference between the means further. For the self-efcantly lower than the pre-test self-efficacy scores, where ficacy questionnaire, the experimental group had signifiF(1, 58) 5 15.4, P , 0.0001, h2 5 0.2. The use of the Wilk’s cantly higher self-efficacy in the post-test than in the Lambda showed that the interaction effect on the compre-test [F(1, 58) 5 24.7, P , 0.0001]. The control group’s bined dependent measures was significant [F(3, 56) 5 scores did not differ significantly between pre- and post32.0, P , 0.0001]. tests [F(1, 58) 5 0.30, P 5 0.56]. Finally, a series of t-tests Results of the follow-up univariate test showed a sigwith a Bonferroni correction were used for two additional nificant interaction effect on radar knowledge [F(1, 58) 5 comparisons on the interaction means. The experimental 88.9, P , 0.0001, h2 5 0.60]. Because the interaction was group’s pre-test self-efficacy was significantly higher significant, simple effects tests were performed to inspect the difference between the means further. For the than the control group’s pre-test self-efficacy [t(58) 5 radar-knowledge test, differences between the pre-test 22.4, P 5 0.02]. Likewise, the experimental group had I test, and self-efficacy. The between factor was training condition (training vs. control) and the within factor was period (pre-test vs. post-test). All subsequent univariate results reported are Greenhouse-Geisser statistics.
TABLE I. SUMMARY OF STUDY RESULTS (N 5 30 IN EACH GROUP). Measure Radar Knowledge Scenario I Self-Efficacy Scenario II Scenario III Reaction to Training
1022
Pre-Test M (SD)
Post-Test M (SD)
P (Interaction Term)
Effect Size h2
Exp. 5 66.2 (8.1) Control 5 65.0 (8.8) Exp. 5 62.0 (13.8) Control 5 57.1 (14.7) Exp. 5 3.8 (0.9) Control 5 4.5 (1.2)
Exp. 5 79.8 (7.6) Control 5 55.7 (9.8) Exp. 5 75.8 (10.7) Control 5 56.9 (15.3) Exp. 5 2.9 (0.7) Control 5 4.4 (1.1)
ⱕ0.01
0.60
ⱕ0.01
0.30
ⱕ0.01
0.10
Exp. 5 70.1 (16.5) Control 5 57.1 (12.5) Exp. 5 74.4 (10.6) Control 5 59.1 (14.9) Exp. 5 6.5 (0.4) Control 5 3.2 (1.3)
P (t-test) ⱕ0.01 ⱕ0.01 ⱕ0.01
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. in data-link weather products. These scenarios consisted of authentic meteorological events with conditions and images pulled from meteorological databases. Thus, GA pilots may encounter weather events such as these in actual flight. The experimental subjects performed better than control subjects in responding to questions about characteristics of the weather in the scenario, weather changes from preflight to during flight, and the implications for the flight. This type of weather information review and subsequent decision making is highly similar to what GA pilots experience in actual flight. Thus, it is likely that the learning gains achieved from the course will transfer to flight situations. While knowledge and skill gain are typically the primary goals of training, it is also important to assess DISCUSSION change in attitudinal variables such as self-efficacy (10,23) and trainee reactions (21). In the current study, If used effectively, NEXRAD-based weather products although the subjects in the experimental group had could be a powerful tool to aid weather-related decision higher pretest self-efficacy than did the subjects in the making in the cockpit. Prior research has indicated, control group, the self-efficacy of the control group did however, that GA pilots do not understand the nuances not change pre- to post-test. In contrast, the experimenof the weather technology products and pilots are not tal group subjects’ self-efficacy regarding their weatherusing the technology effectively (3,5). Training can help Delivered by Publishing Technology to: Umea University Library This increase is important radar-related skills increased. pilots use data-link weather successfully (3). To ensure IP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 for twoAssociation reasons. First, Davis and colleagues argued that high-quality aviation weather training, it is crucial that Medical Copyright: Aerospace when a training program provides exposure to the training developers follow established protocols during transfer setting, this gives trainees a frame of reference training product development as well as assess the for understanding the training content and, in turn, the training effectiveness of the training products (21). The trainees may actually learn more (10). This increased current research developed and tested the efficacy of a learning, in turn, is reflected in their post-training selftraining course for GA pilots on the use of NEXRADefficacy—the degree to which they are confident in apbased products in convective weather situations. The plying the knowledge they acquired in training to actual subjects who received the data-link training course flight (10). Additionally, as considerable prior research scored higher on a radar-knowledge test, a weather scehas found a positive relationship between self-efficacy nario application test, a retention test, and attitudinal and performance (4,10), the subjects who received trainmeasures about weather technology, as compared to ing will likely have a better chance at successfully using their scores on the pre-test and also in comparison to the the NEXRAD technology during actual flight. control group who did not receive training. With respect to trainee reactions, although a positive A thorough training course validation includes collectlearner reaction to a particular training program is not esing and examining gains in knowledge, skills, and attisential for learning to occur, it is desirable for the learners tudes (21,23). In terms of knowledge test results, the radar to enjoy the learning process. This may be particularly true knowledge pre-test demonstrated that the subjects had in any GA supplemental training programs that go beyond only a moderate level of knowledge of convective weather pilot certification requirements. In the current study, the and NEXRAD going into the course. In the post-test, the experimental group reacted much more favorably toward experimental group showed a large increase in radar knowledge and the control group actually declined from their training session than did the control group. Since the the pre- to post-test—a likely indication of experimental current course would be optional training and not required fatigue. All subjects were from an FAA Part 142 environfor pilot certification, positive reviews from past subjects ment with a stringent curriculum and many subjects had could motivate additional GA pilots to take the course. taken additional academic courses in meteorological Many avenues of research involving weather educatraining. Despite their strong educational background, tion and training have yet to be explored. As the typical the pilots still learned from the course. Thus, we suspect GA pilot is older than the subjects in the current study that pilots with a lower degree of weather knowledge and has more flight experience, expanding the current and aeronautical experience may benefit to an even research to incorporate GA pilots from non-university greater degree from this type of weather training. settings would provide insight into additional weather While knowledge gain is the first step of training eftraining needs. Additionally, to determine how this trainfectiveness, it is key that trainees can also apply what ing impacts pilot flight behavior, use of this training prothey learned (21,23). The scenario application tests program in conjunction with flight scenarios in flight training vided the opportunity for subjects to apply their newly devices or computer-based simulations complete with a acquired knowledge to flight weather scenarios. The simulated data-link weather source are needed. Measures scenarios included the preflight weather as well as subin a simulation study should include how and when the sequent in-flight radar images, similar to that available pilots consulted the data-link information, as well as
significantly higher self-efficacy in the post-test than did the control group [t(58) 5 6.1, P , 0.0001]. Three of the dependent measures, Scenario II, Retention/Scenario III, and Reactions, were given only during the post-test and were tested separately from the MANOVA. The results of three separate independent samples t-tests showed that the experimental group performed significantly better than the control group on Scenario II [t(58) 5 3.4, P 5 0.001], as well as 3 d later on the retention test (Scenario III) [t(55) 5 4.6, P , 0.0001)]. The results of the reactions measures showed that the experimental group responded much more positively to the training than did the control group [t(58) 5 13.1, P , 0.0001].
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1023
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL. 3. Ball JD. The impact of training on general aviation pilots’ ability cognitive measures that assess understanding of the into make strategic weather-related decisions. Oklahoma formation presented. Another area of future research City, OK: FAA Civil Aerospace Medical Institute; 2008. FAA pertains to self-efficacy. Since the experimental group had Report DOT/FAA/AM-08/3. Available from: http://www. a moderate level of aviation weather self-efficacy prior to dtic.mil/cgi-bin/GetTRDoc?Location5U2&doc5GetTRDoc. pdf&AD5ADA477162. the course (as shown by the pretest), the results do not tell 4. Bandura A. Cultivate self-efficacy for personal and organizational us how participants with a lower level of self-efficacy at effectiveness. In: Locke EA, ed. Handbook of principles of the start of the course may fare in the course. It may be organization behavior. Oxford: Blackwell; 2000:120–36. 5. Beringer DB, Ball JD. The effects of NEXRAD graphical data that subjects with higher aviation weather self-efficacy resolution and direct weather viewing on pilots’ judgments would learn more when they begin the course than would of weather severity and their willingness to continue a flight. subjects with lower aviation weather self-efficacy. AddiOklahoma City, OK: FAA Civil Aerospace Medical Institute; tional research is needed to answer this. Finally, research 2004. FAA Report DOT/FAA/AM-04/5. Available from http://www.dtic.mil/cgi-bin/GetTRDoc?AD5ADA423239& regarding the assessment of weather knowledge may Location5U2&doc5GetTRDoc.pdf. also be useful. In this study, the average post-test scores 6. Blickensderfer EL, Strally S, Doherty S. The effects of scenariofor the experimental group were only moderately high based training on pilots’ use of an emergency whole-plane (79.8% on the radar knowledge and 75.8% on the scenario parachute. Int J Aviat Psychol 2012; 22:184–202. 7. Butler RE. LOFT: full-motion simulation as crew resource management knowledge). It may be that some of the assessment questraining. In: Weiner EL, Kanki BG, Helmreich RL, eds. Cockpit tions were too difficult and/or an imperfect match beresource management. San Diego, CA: Academic Press; 1993. tween the training content and the questions (i.e., 8. Cole K, Director. Tornado glory. [Documentary Film; DVD]. Angry Sky Entertainment and PBS; 2006. imperfect content validity of the test). 9. Craig PA. Evaluating pilots using a scenario-based methodology: Regarding practical applications, the current training a guide for instructors and examiners. International Journal of method may lend itself well to an online, individual Applied Aviation Studies (Oklahoma City, OK: FAA Academy). training program that gives pilots the opportunity to go 2009; 9:155–70. Delivered by Publishing Technology to: Umea University Library 10. Davis WD, Fedor DB, Parsons CK, Herold DM. The development through scenarios using an animated navigation disIP: 130.239.20.174 On: Mon, 06 Apr 2015cacy 11:51:30 of self-effi during aviation training. J Organ Behav 2000; play. Another practical application ofCopyright: this training modAerospace Medical 21:Association 857–71. ule could be to provide the training materials to training 11. Federal Aviation Administration. Use of cockpit displays of digital providers across the nation, similar to what the FAA’s weather and operational information (Advisory Circular 0063). Washington, DC: FAA, Flight Technologies and Procedures WINGS program provides to pilots through seminars. Division; 2004. Available from: http://rgl.faa.gov/Regulatory_ In conclusion, the results of this study indicate that and_Guidance_Library/rgAdvisoryCircular.nsf/list/ pilots can benefit from a well-designed training program AC%2000-63/$FILE/AC%2000-63.pdf. involving specific areas of weather-related knowledge. 12. Federal Aviation Administration. General aviation pilot’s guide to preflight weather planning, weather self-briefings, and The subjects who received training in the current study weather decision making. Washington, DC: FAA; 2009. will have a better chance at successful interpretation of Available from: http://www.faa.gov/pilots/safety/media/ NEXRAD based weather products used before and durga_weather_decision_making.pdf. 13. Federal Aviation Administration. Aviation weather services ing actual flight than had they not received this training. (Advisory Circular 00-45G). FAA: Flight Technologies and Well-designed weather training such as this can teach GA Procedures Division. Washington, DC: FAA; 2010. Available pilots to understand and use weather technology and from: http://rgl.faa.gov/Regulatory_and_Guidance_Library/ products and, in turn, to have safe and enjoyable flights. rgAdvisoryCircular.nsf/0/6a07efaa9a68c1a8862576c70052e9e ACKNOWLEDGMENTS This research was conducted under sponsorship of the Federal Aviation Administration‘s (FAA) Center for Excellence in General Aviation Research (CGAR) under FAA Cooperative Agreement Number 07-C-GA-ERAU-014. The views expressed in this paper are those of the authors and do not represent those of the organization with which they are affiliated or the FAA. The authors wish to thank Jon Dziok for helping to develop the training program, Julian Archer for assisting in data collection, the research subjects, and Shannon Cummings and Michael Vincent for assistance in preparing the manuscript. Authors and affiliations: Erin A. Cobbett, M.S.A., Georgia Perimeter College, Tucker, GA (formerly from Applied Aviation Sciences, EmbryRiddle Aeronautical University, Daytona Beach, FL); Elizabeth L. Blickensderfer, Ph.D., Human-Factors and Systems, and John Lanicci, Ph.D., Graduate Studies, Embry-Riddle Aeronautical University, Daytona Beach, FL. REFERENCES 1. Aircraft Owners and Pilots Association. 2010 Nall Report: accident trends and factors for 2009. Frederick, MD: AOPA Air Safety Foundation; 2010. Available from: http://www.aopa.org/-/ media/Files/AOPA/Home/News/All%20News/2012/ April/VFR%20in%20to%20IMC%20Learn%20to%20escape%20 the%20odds/10nall.pdf. 2. ATSC. Capabilities report: weather technology in the cockpit program. Norman, OK: Atmospheric Technology Services Company, LLC; 2010.
1024
5/$FILE/AC-0045G.pdf. 14. Federal Aviation Administration, Aviation Safety Information Analysis and Sharing. Weather-related aviation accident study 2003-2007. Washington, DC: FAA; 2010. Available from: http://www.asias.faa.gov/pls/apex/f?p5100:8:0::NO::P8_ STDY_VAR:2. 15. Federal Aviation Administration. Aeronautical information manual, chapter 7, safety of flight, meteorology. Washington, DC: United States Department of Transportation; 2011. Available from: http://www.faa.gov/air_traffic/publications/ATpubs/ AIM/aim0701.html#aim0701.html.1. 16. French J, Summers M, Ayers F, Blickensderfer E. The effectiveness of scenario based training for technically advanced aircraft [Abstract #521]. Aviat Space Environ Med 2007; 78:307. 17. Gist ME, Mitchell TR. Self-efficacy: a theoretical analysis of its determinants and malleability. Acad Manage Rev 1992; 17: 183–221. 18. Gist ME, Schwoerer C, Rosen B. Effects of alternative training methods on self-efficacy and performance in computer software training. J Appl Psychol 1990; 74:884–91. 19. Jentsch F, Curtis M, Salas E, eds. Simulation in aviation training. Burlington, VT: Ashgate; 2011. 20. Jeppesen Sanderson, Inc., Producers. Weather hazards: thunderstorms, wind shear, and microbursts [Documentary film; VHS recording]. Englewood, CO: Jeppesen Sanderson, Inc.; 1990. 21. Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels, 3rd ed. San Francisco: Berrett-Koehler Store; 1998. 22. Knecht WR. Use of weather information by general aviation pilots, part 1, quantitative: reported use and value of providers and
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
NEXRAD TRAINING FOR GA PILOTS—COBBETT ET AL.
23. 24.
25.
26. 27.
products. Oklahoma City, OK: FAA Civil Aerospace Medical Institute; 2008. FAA Report DOT/FAA/AM-08/6. Retrieved October 2013 from http://www.dtic.mil/cgi-bin/GetTRDoc? Location5U2&doc5GetTRDoc.pdf&AD5ADA481118. Kraiger K, Ford JK, Salas E. Integration of cognitive, behavioral, and affective theories of learning into new methods of training evaluation. J Appl Psychol 1993;78:311–28. Latorella KA, Chamberlain JP. Graphical weather information system evaluation: usability, perceived utility, and preferences from general aviation pilots. Hampton, VA: NASA Langley Research Center; 2002. Paper Number: 2002-01-1521. Available from: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/ 20030013637_2003014788.pdf. Latorella KA, Chamberlain JP. Tactical vs. strategic behavior: general aviation piloting in convective weather scenarios. In: Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting; Sept. 30-Oct. 4, 2002; Baltimore, MD. Santa Monica, CA: HFES, 2002:101–5. Available from http:// ntrs.nasa.gov/search.jsp?R=20030014193&hterms=Latorella& qs=Ntx%3Dmode%2520matchallpartial%26Ntk%3DAll%26N %3D0%26No%3D10%26Ntt%3DLatorella. Lester P. Aviation weather. Englewood, CO: Jeppesen Sanderson, Inc.; 2007. Liu D, Macchiarella ND, Vincenzi DA. Simulation fidelity. In: Vincenzi DA, Wise JA, Mouloua M, Hancock PA. Human factors in simulation and training. Boca Raton, FL: CRC Press/ Taylor & Francis; 2008:61–73.
28. National Transportation Safety Board. In-cockpit NEXRAD mosaic imagery. Washington, DC: NTSB; 2012. Available from: http://www.ntsb.gov/doclib/safetyalerts/SA_017.pdf. 29. Jetstream Online School for Weather [Internet]. 2010; Accessed October 2013 from: http://www.srh.noaa.gov/jetstream// doppler/doppler_intro.htm. 30. National Weather Service Warning Decision Training Branch. Principles of meteorological Doppler radar, distance learning and operations course [Internet]. 2010; Accessed October 2013 from: http://www.wdtb.noaa.gov/courses/dloc/topic3/lesson1/ index.html. 31. Office of the Federal Coordinator for Meteorological Services and Supporting Research (OFCM). Federal meteorological handbook No. 11(FCM-H11A-2009). Washington, DC: National Oceanic and Atmospheric Administration; 2009. Available from: http://www.ofcm.gov/fmh11/fmh11parta/pdf/FMH-11%20 Pt%20A%20Feb%202013%20All.pdf. 32. Orlady HW, Orlady LM. Human factors in multi-crew flight operations. Burlington, VT: Ashgate; 1999. 33. Riggs ML, Warka J, Babasa B, Betancourt R, Hooker S. Development and validation of self-efficacy and outcome expectancy scales for job-related applications. Educ Psychol Meas 1994; 54:793–802. Retrieved from http://epm.sagepub. com/content/54/3/793. 34. Hebert C, Director. The deadliest plane crash [Documentary film; DVD]. Arlington, VA: NOVA; 2006.
Delivered by Publishing Technology to: Umea University Library IP: 130.239.20.174 On: Mon, 06 Apr 2015 11:51:30 Copyright: Aerospace Medical Association
Aviation, Space, and Environmental Medicine x Vol. 85, No. 10 x October 2014
1025
