Risk Analysis, Vol. 34, No. 10, 2014
DOI: 10.1111/risa.12200
Sense and Avoid Requirements for Unmanned Aircraft Systems Using a Target Level of Safety Approach Richard Melnyk,1,∗ Daniel Schrage,1 Vitali Volovoi,2 and Hernando Jimenez1
One of the most critical challenges to full integration of unmanned aircraft systems (UAS) into the National Airspace System (NAS) is the requirement to comply with CFR 14 Part 91.113 to “see and avoid” other aircraft. Various attempts have been made to develop systems to “sense and avoid” other aircraft so UAS can comply with the intent of the regulation. This article proposes a framework to develop effectiveness requirements for any SAA system by linking UAS characteristics and operating environments to midair collision risk quantified by a fatality rate. The framework consists of a target level of safety (TLS) approach using an event tree format. Safety has been identified as the most important consideration in the UAS integration process. While safety can be defined in many ways, the authors propose using a fatality rate metric that follows other statistics used in the industry. This metric allows for the use of a TLS approach to the development of SAA requirements for system certification. Failure to adequately link system requirements to safety could result in the implementation of SAA systems that either do not adequately mitigate the risk associated with UAS operations or are overdesigned, resulting in increased cost and complexity. This article demonstrates the use of the proposed framework to develop specific SAA effectiveness standards based on UAS weight and airspace class combinations. KEY WORDS: Event tree; fatality rate; midair collision; sense and avoid (SAA); target level of safety; unmanned aircraft systems (UAS)
1. INTRODUCTION
wants to unleash the potential of UAS to generate revenue.(2,3) However, the reality of the situation remains that UAS operations in the NAS have been limited to military operations in “restricted” airspace and limited public/civil usage under restrictive Certificates of Authorization or Waiver (COAs). One of the most often cited obstacles to integration is the inherent lack of a sense and avoid (SAA) capability on UAS to comply with Code of Federal Regulations (CFR) 14 Part 91.113.(4) There have been several efforts to develop and test an SAA capability for UAS, including a ground-based system tested by the U.S. Army and an air-based one tested by the U.S. Air Force. The purpose of this article is to propose a framework to develop effectiveness standards for SAA systems developed for UAS. Effectiveness in this
Full integration of unmanned aircraft systems (UAS) into the National Airspace System (NAS) is the goal of many stakeholders in the aerospace community, in addition to being mandated by law as of February 2012.(1) The Department of Defense seeks greater access to training sites, other public users would like to use UAS for operations such as law enforcement and disaster relief, and the civil sector
1 School
of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA. 2 Independent Consultant, Alpharetta, GA, USA. ∗ Address correspondence to Richard V. Melnyk, School of Aerospace Engineering, Georgia Institute of Technology, 270 Ferst Drive, Atlanta, GA 30332, USA; [email protected].
1894
C 2014 Society for Risk Analysis 0272-4332/14/0100-1894$22.00/1
Sense and Avoid Requirements for UAS Using a TLS Approach context is defined as the combination of reliability and efficacy that indicates to what extent a system performs its intended function and is not prevented from doing so due to either system failures or insufficient performance standards.(5) In order to develop minimum effectiveness standards for SAA, the authors propose utilizing a framework that includes a target level of safety (TLS) approach to the problem and an event tree (ET) format risk model to predict midair collision (MAC) fatality rates resulting from UAS operations. This article complements similar work by the authors predicting third-party fatality rates on the ground from UAS operations.(6) 2. BACKGROUND It is impossible to design an SAA subsystem for UAS, or for any aircraft for that matter, without specific performance and reliability standards. These standards should include parameters such as the minimum intruder detection distance based on closure rates and the ability to maneuver to avoid collisions. They should also include a reliability or availability standard to ensure that the system maintains functionality. The parameters would also likely include the number of potential intruders the system must track, the accuracy of any positioning system, and a maximum number of false alerts. It was the opinion of the Federal Aviation Administration’s (FFA’s) Sense and Avoid Workshop that the foundation for these performance standards be a TLS approach.(7) The TLS has been defined as a “top-down approach which focuses on safety critical issues which could affect achievement of the safety target” by Haddon and Whittaker.(8) The metric used to define the TLS in the framework for this article is a midair or second-party fatality rate, or fatalities per UAS flight hour (FH). This metric was chosen because it mirrors the statistics already in use by the FAA for the NAS, and it also matches the metrics used by the Range Commander’s Council (RCC),(9) the Nuclear Regulatory Commission (NRC),(10) and other agencies concerned with safety such as the National Transportation Safety Board.(11) The real purpose behind using the TLS approach to this problem is to link the effectiveness of any SAA subsystem to a metric that matters most to the FAA, stakeholders, and the public. That metric should be safety related, which is why the fatality rate is suggested. Without linking SAA standards to safety, any design choices and tradeoffs made on the subsystem will not necessarily lead to safer skies.
1895
ETs are established reliability tools first used by the NRC in the 1970s.(12) The authors chose to use an ET format to model UAS operations in the NAS for several reasons. First, they provide a means to visually trace both the probability and impact of events, which can be either failures or normal operating events.(5) Second, fatalities in the NAS are extremely rare events. To properly identify and analyze events that only occur sometimes on the order of 1 × 10−8 using a model that replicates the behavior of the NAS like an agent-based model would require 1 × 109 simulations, at a minimum.(13) ETs combined with Monte Carlo simulation, on the other hand, can capture the effects of improbable events because even unlikely events can be featured in a branch of the tree. 3. FRAMEWORK DETAILS The primary ET used to model the risk of fatalities from UAS operations appears in Fig. 1. The ET is a series of branches that detail different options for the air environment, mitigation, and event outcomes based on the probability of each of the branches occurring and the effects (fatalities) of progressing along a particular branch. The overarching equation that governed the model appears in Equation (1), where Ec is the “expected casualty” (fatality) rate in fatalities per FH. Ec = Encounter Rate ×
3
P (Collision)i
i=1
× Effectsi
(1)
The encounter rate is further explained in Equation (5), the probability of a collision occurring in Equation (2), and the effects in Equation (7). Using the yellow path in the ET as an example, the probability of collision or P(Collision)1 appears in Equation (2): P (IFR) × (1 − P (SEPARATION|IFR) ×(1 − P (AVOIDANCE|IFR) , (2) where IFR is instrument flight rules. The term effects refers to the number of casualties per collision and depends on a number of factors to include the types of intruder aircraft in the representative airspace and the likelihood that the UAS penetrates the intruder aircraft. The effects blocks in
1896
Melnyk et al.
Fig. 1. Event tree.
Fig. 1 actually have further branches, but in the interest of space the tree is terminated and a more detailed depiction on the “Effects” branch and the associated equations will be discussed in Section 3.3. The likelihood of progressing along the branches, and the effects that occur on branches that result in collisions, is dependent on many factors related to the airspace and traffic environment. To allow for a high degree of flexibility and capture the dependencies associated with the model, the software tool @RiskTM was used to represent particular parameters in the model as probability distributions. This spreadsheet-based tool allows the user to substitute deterministic cells in the spreadsheet for probability distributions for use in a Monte Carlo simulation. For example, some of the variables required to calculate the encounter rate were modeled as probability distributions to capture some of the uncertainty associated with the air environment, such as intruder aircraft speeds. The probability of separation and probability of avoidance, which will be explained in more detail later, were modeled for some supporting analysis as deterministic values and in other cases as uniform distributions to capture a range of values or, in the case of avoidance, as a discrete distribution. Several of the factors that determine the fatalities per collision could also be modeled as probability distributions, if desired. As a result of this approach, the analyst is able to propagate some of the uncertainty
Table I. Example Use of Event Tree Parameter Encounter rate Probability of IFR P (IFR) Probability of separation P (Separation) Probability of avoidance P (Avoid) Effects
Value
Units
0.00001 75 99.99
Encounters per FH % %
75
%
50
Fatalities per collision
inherent in the input variables to this model to the resulting probability distribution representing the output or fatality rate. However, to demonstrate the use of the model in a deterministic manner, an example is offered following the yellow path in Fig. 1 (color visible in online version) and using the values from Table I. In this example, the equation to determine the fatality rate would be: Ec = Encounter Rate × ProportionIFR × (1 − P (Separation)) × (1 − P (Avoid)) × Effects or Ec = 0.00001 × 0.75 × (1 − 0.9999) × (1 − 0.75) × 50 = 9.375 × 10−7 Fatalities/FH. (3)
Sense and Avoid Requirements for UAS Using a TLS Approach
1897
Table II. Airspace Volume and Density Data Used for Model Validation Information
Value
Average annual total U.S. flight hours for GA Average annual total U.S. flight hours for all others Average GA aircraft aloft Average aircraft aloft all others Total U.S. land area Location of all MACs in 1994–1999 data Total U.S. airspace volume for GA Total U.S. airspace volume for all others Average airspace density
Units
25,396 19,785 2899 2258 3,796,742.2 Below 6,000 2,830,685.0 16,512,329.4 1.16 × 10−3
Table III. Velocity Values Information Manned traffic speeds
Relative velocity with intruders
Knots Triangular distribution: Min: 50.36 Peak: 100.72 Max: 302.17 141.918 + 0.0127 × UAS weight
Source
1000s FH
RITA database(22–25)
Aircraft/hour
Calculated
Mi2 Ft Nm3
Census data(26) NTSB data(27) Calculated
Aircraft/Nm3
Calculated
Table IV. IFR Statistics in the NAS Source
Average of radar reports(28)
Regression based on relative velocity simulations
Information Proportion of VFR time in NAS P (VFR) Proportion of IFR time in NAS P (IFR)
Percent
Source
Normal distribution: Mean: 83.6 Std. dev.: 8.8 Aggregate NAS statistics paper(21) Normal distribution: Mean: 16.4 Std. dev.: 8.8
Table V. Probability of Avoidance by Traffic Combination(15)
However, as noted earlier, the values used to populate the ET determine the encounter rate and calculate the effects of collisions, which is much more complex and dependent on the type of airspace and the composition of the intruder aircraft in that airspace. The airspace environment helps determine the airspace density, the proportion of “visual” and “instrument” traffic, and the speeds of the intruder aircraft, which in turn determines the encounter rate and the likelihood of separation and avoidance. In addition, the composition of the types of intruder aircraft plays a role in determining the relative speeds of the aircraft pairs, the likelihood of fatalities in the event of a collision, and the number of fatalities per collision. 3.1. Encounter Rate Using a Gas Particle Model An important component of the model was the encounter rate. Values for the probability of an encounter were based on a gas particle model that treated aircraft in a specified volume of airspace as randomly moving gas particles.(14–18) The airspace density (ρ) represents the density of the airspace in question in aircraft per volume. Other aircraft in the
Type of Encounter
P (Avoid)
Military/military Military/general Military/domestic Domestic/domestic Domestic/general General/general
20 20 20 75 75 85
airspace were treated as motionless particles, while the UAS was treated as a cylinder sweeping out volume at a rate based on the vertical (V) and horizontal (H) dimensions of the encounter cylinder and the relative velocities (VREL ) between the UAS (V1 ) and manned aircraft (V2 ). An encounter is defined as any time that the swept cylinder overlaps a particle in the volume. The angle between particles (δ) was assumed to be random. UAS encounters or collisions with other UAS were not considered for this study because the focus was on determining fatality rates due to UAS operations and it was assumed that no unmanned flight operations with people on board would occur in the near future.(15) Encounter Rate = 4 × ρ × H × V × VREL
(4)
1898
Melnyk et al. Table VI. Pilot Visual Avoidance Probabilities
Information
Probability of pilot visually detecting other aircraft
P (Avoid)
The unstructured nature in which the gas particle model portrays aircraft in the airspace is not entirely indicative of the measures of control inherent in the NAS. However, for the purpose of this research, given its overall statistical nature, the random motion of particles replicated by this model was favored. While this model is less realistic around airports or along airways, it provides a good statistical representation of overall activity in the NAS. In fact, with the advent of trajectory-based operations (TBOs) in the near future under NextGEN, aircraft behavior will actually more closely resemble random trajectories and rely less on more ordered trajectories focused on ground-based navigational aids.(29)
Source
100 knot closure 72.3–100 speed 200 knot closure 30.2–90.7 Aviation speed medicine article(34) 300 knot closure 16.2–48.7 speed 400 knot closure 9.2–27.6 speed 500 knot closure 5.0–15.0 speed
2 VREL = V12 + V22 − 2 × V1 × V2 × cos (δ)
(5)
The dimensions of the cylinder were based on physical dimensions of the UAS and not a larger separation volume, as other applications have used. That decision was based on the focus and purpose of this study. Previous uses of gas particle models were primarily for air traffic or collision avoidance purposes so the goal of those studies was to determine the likelihood of two aircraft requiring additional separation measures either from a controller or collision avoidance system. However, given the safety-related focus in this study on fatality rates, the primary concern was actual collisions, not near MACs. Thus, in this study, an “encounter” is effectively a collision if no separation or avoidance measures are in place. Separation and avoidance will be described in Section 3.3. If the study was expanded to determine the impact of UAS operations on air traffic and separation, then the larger separation volume could be used. First, the authors developed characteristics of the air environment to include the density of aircraft in the airspace in question, the type of aircraft in the environment,(19,20) and the type of airspace control or architecture.(21) The average aircraft density across the NAS was estimated through statistics on aircraft flights in the NAS,(22–25) which appears in Table II. The velocities used for manned aircraft to determine the relative velocity profiles appear in Table III. The manned traffic speeds are based on radar reports and historical data. The relative velocity, using Equation (6), is based on conducting a simulation substituting the manned traffic speeds distribution for V2 , a uniform distribution for the approach angle that ranged between 0° and 360°, and a velocity distribution for UAS for V1 that appears in Table VII.
3.2. Risk Mitigation Through Separation and Avoidance The third feature of the ET model was risk mitigation in the form of separation or avoidance. To describe the difference between separation and avoidance, it is instructive to review the nature of the gas particle model. In this model, the encounter rate quantifies the rate at which randomly moving particles would collide in a specified volume. Separation refers to proactive measures taken by agents external to the aircraft pair to prevent the encounter from occurring in the first place. Avoidance, on the other hand, refers to reactive measures taken by one or both aircraft in the pair to prevent a collision in the event that separation measures failed. In automobile terms, a traffic light would be a separation measure while swerving to prevent an accident with a car that ran a red light is avoidance. 3.2.1. Separation The probability of preventing a collision through separation was based on a combination of factors to include whether or not the airspace was under positive control by air traffic management (ATM). The success rate in preventing collisions through ATM control was based on statistics of the NAS.(13,30) The degree to which separation services are available depends on whether the aircraft operate under visual or instrument flight rules (VFR or IFR). This branch probability was based on statistics of the NAS from Table IV. To predict how effective the controller is at maintaining aircraft separation depends on many complex factors, including the type and amount of traffic, human factors, and a number
Sense and Avoid Requirements for UAS Using a TLS Approach
1899
UAS Mass
UAS Velocity
Relative Velocity
Kinetic Energy Penetration Probability
Intruder Velocity
Intruder Type
Number of Casualties
Number of Seats Number of Passengers Load Factor
Primary Information
Results
Derived Information
Fig. 2. Effects.
Table VII. UAS Parameters for Model Information
Value
Units
Speed
88.18 + 0.0279 × Weight
Wingspan
12.015368 ft + 0.0140653 × Weight − 4.6313e − 7 × (Weight − 1807.6)2 0.5073 × Weight lbs
Fuel load
Endurance
6.0627732 + 0.0079073 × Weight
ft/s
Source Regression based on weight Regression based on weight
Regression based on weight hours Regression based on weight
Table VIII. Manned Aircraft Airspace Composition Citation Ref. 41
Information General aviation Percent of total hours Air carrier
Percent
Source
59
GAMA databook and RITA database(19,20) FAA factbook CY 10 data(39)
41
Ref. 42
Ref. 43
Refs. 44, 43
of unique situations. However, to model the separation probability for an ET-based model such as this required using known values in an educated manner. First, the FAA publishes system reliability standards for its architecture. A critical system is required to have a reliability of 0.99999 or one failure every 100,000 cycles.(31) This takes into account the air traffic control system, but not the possibil-
Breakdown of air carrier aircraft by type Load factor for air carriers
Jet
55.0
All others
45.0
Normal distribution: Mean: 76.765 Std. Dev.: 0.457
FAA statistics(40)
ity of human error or errors on the part of the aircraft instrumentation that would affect proper separation. International standards for midair accidents due to air traffic control are 1.55 × 10−8 per FH.(13) This represents an extremely high level of reliability and effectiveness. Statistics for a 10-year period in U.S. airspace indicate that only 10 MACs occurred as a result of air traffic control issues, which when taken against all of the flights that occurred
1900
Melnyk et al.
Table IX. Air Vehicle Parameters Used for Model Validation Information
Value
Traffic speeds
Triangular distribution: Min: 50.36 Peak: 100.72 Max: 302.17 173 General aviation velocity (speed) Vertical cylinder 36 height (V)
Units
Source
Knots
Average of radar reports(28)
Knots
Waggoner study(45)
Ft
Average of studies and information on GA aircraft
36
Horizontal radius of cylinder (H; wingspan)
Table X. Historical Fatality Data Used for Model Validation Information Midair collision (MAC) rate for GA MAC rate for all aircraft Fatality rate for MACs Casualties per MACs
Value 5.9 × 10−7
Units Midair per flight hour
3.74 × 10−7 6.82 × 10−7 1.76
Source
NTSB data(46) Fatality per flight hour Fatality per midair
in a 10-year period, represents an almost negligible amount.(30) As a result, for the purpose of this model, the authors assumed that the probability of separation would be 99.99% when under IFR circumstances or in VFR controlled airspace. 3.2.2. Avoidance Avoidance rates for the aircraft in the environment were based on a combination of studies that detailed avoidance rates by both visual and electronic means,(15,32–35) and were largely based on the closure rates between manned aircraft and UAS. Avoidance is a highly complex behavior that depends on the angle of approach, closing speed, the accuracy of position reporting devices, the skill and alertness of the pilot in the manned aircraft, and the size of the aircraft involved, among other things. Despite the complexity of this aspect of the air collision risk
model, it is important to reflect the likelihood of avoidance as accurately as possible because this area represents one of the most controversial integration topics. To model the probability of avoidance, several studies were examined. Visual avoidance of other aircraft is a topic that has been studied in detail. There are studies dating back over half a century that quantify the probability that a pilot would see another aircraft either to avoid that aircraft or, in the case of some military studies, to attempt to engage that aircraft in air to air combat. Regardless, there are data available to quantify the probability of aircraft detection. Because this model deals with only one piloted aircraft, the fact that only one of the two parties was using visual detection means had to be taken into account. The aforementioned article on MACs by May cites figures from the Swedish Air Traffic Control Commission that quantifies the probability of avoidance, given that an encounter has occurred. The values, which are featured in Table V, depend on whether the aircraft in question are military, domestic (commercial), or general (general aviation [GA]). The differences in values reflect the average cruising speed of the different categories. For the type of traffic used in this model, the bottom three values of either 75% or 85% would be most appropriate. Another study conducted for the FAA in the 1980s also looked at see and avoid effectiveness based on closure rates. The pertinent results of the study appear in Table VI. The probability of not only detecting but avoiding another aircraft, given an encounter, range from 97% for lower closing speeds to as low as 47% for the highest closing speeds. An earlier article that used NMAC data from 1968 and simulations to estimate the effectiveness of see and avoid rated the probability of avoiding a midair collision (MAC) at 95% if the closure rate was less than 100 knots and as low as 32% for speeds over 400 knots. Based on average values of closure rate used in the simulation validation, the probability of detection and avoidance is approximately 85%, which is in the middle of the range corresponding to the 100 knot closure speed from the study for the FAA, which appears in Table V. Studies have also been conducted on the effectiveness of electronic avoidance such as the traffic alert and collision avoidance system (TCAS). One study by the MITRE Corporation, completed for the Department of Transportation, lists the effectiveness percentages of TCAS avoiding collisions based on varying levels of equipment usage. The report states
Sense and Avoid Requirements for UAS Using a TLS Approach
1901
Table XI. Air Environment Data Used for Midair Collision Fatality Estimation Information General aviation
Percent of total hours
Air carrier Breakdown of air carrier aircraft by type
Jet
Load factor for air carriers
Passengers per air carrier Energy absorption for Part 25 jet aircraft
All others Normal distribution: Mean: 76.765 Std. dev.: 0.457 106.11 10,000
that TCAS is capable of an avoidance probability of 94.7%. However, that requires all aircraft in question to be equipped with both TCAS and Mode C altitude reporting equipment.(36) When all aircraft are not equipped with TCAS or if some systems are not capable of reporting altitude information this avoidance rate can drop to as low as 57.6%. Based on the statistics from the abovementioned MITRE study detailing what portion of the NAS was equipped with equipment such as transponders, the avoidance rate would be approximately 75%. Because it was impossible to determine exactly what proportion of the NAS had some form of electronic avoidance measure or position reporting equipment, it was assumed that the ability to avoid a collision by electronic means was 75%. In IFR conditions, electronic means were the only method available for avoidance so the values from the MITRE study were used. Under VFR conditions the MITRE avoidance values and visual avoidance values from Table VI based on closure rates were combined in the model. 3.3. Effects: Expected Fatalities Due to an MAC The final component of the air risk ET was the effects that would occur if a MAC did occur. A depiction of the factors related to the fatalities per collision, or effects, appears in Fig. 2. This figure represents the extension of the ET effects blocks from Fig. 1. This segment consisted of determining the probability that an MAC between a UAS and a manned aircraft would cause fatalities and, if so, how many. The probability that an MAC would result in fatalities was based on studies of both bird strikes on air-
Value
Units
59
%
41 55.0
Source GAMA databook and RITA database(19,20) FAA factbook CY 10 data(39)
45.0 FAA statistics(40)
People/aircraft Joules
FAA factbook(39) Bird strike report(37)
craft from CFR 14 Part 25.631 and the European Aviation Safety Agency (EASA) Part CS-25,(4,37) and the risk to aircraft from falling debris from the RCC.(38) The RCC discusses the size of fragments that have the potential to cause damage to aircraft, and there is the potential for catastrophic damage for objects weighing even less than one pound. The equation to determine effects was: Casualties = P (Penetration|Collision) Collision × Number of Seats × Load Factor,(6) where P (Penetration|Collision) ∼ f (Intruder Type, Collision Kinetic Energy). (7) The U.S. and European certification requirements only require sufficient bird strike protection for Part 25 aircraft and a review of the energy threshold requirements reveals that only jets operating under Part 25 are required to have protection that could prevent a catastrophe in collisions with all but the smallest of UAS. Therefore, the assumption was made that if any collision with a UAS occurred with any aircraft in the model other than a Part 25 jet, then all aboard that aircraft were considered fatalities. This is a conservative approach, but necessary based on the lack of available studies into the effects of aircraft collisions with UAS. For jets operating under Part 25, a collision was deemed not to cause casualties if the energy associated with the UAS impact was less than the conservative end of the protection threshold, or 10 kJ. If a catastrophic collision did occur, the number of casualties caused was
1902
Melnyk et al.
Air Casualties / Ec: Regression Coefficients Wingspan
0.46
Air Density
0.31
Controlled Airspace
-0.29
SAA Effectiveness Rate
-0.25
Other Pilot Avoid
-0.22
Speed
0.09
IFR Rate:
-0.03
ATM Effectiveness
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
-0.3
-0.02
Coefficient Value Fig. 3. Sensitivity analysis for air risk model.
based on the type of aircraft in the NAS, their passenger capacity,(39) and load factor statistics.(40) Information about the representative UAS in the model was obtained by examining the current characteristics of the most common UAS in use at the time of writing this effort. The values for the UAS characteristics appear in Table VII. It was possible to establish relationships between the parameters mentioned below and the UAS weight. Information regarding other aircraft in the airspace was obtained from several sources related to the NAS. These data were used to calculate the likelihood of encountering different types of aircraft to base the probability of penetration on. The authors also used these values to determine the number of passengers per intruder aircraft. The data in question appear in Table VIII. 3.4. Validation of the Model A shortcoming of previous efforts to examine this problem was the lack of a method to validate the prediction model. UAS have not been able to operate in the NAS alongside manned aircraft to date except under very strict control measures, and statistics on encounters between UAS and manned aircraft in places like Iraq and Afghanistan were unavailable. The authors validated the model in this study by using historical data for GA MACs and fatalities. In other words, the authors substituted a representative
Table XII. Recommended UAS Categories by Weight Group
Weight (lbs)
Source
Group 1 (micro) Group 2 (mini) Group 3 (small)
DOI: 10.1111/risa.12200
Sense and Avoid Requirements for Unmanned Aircraft Systems Using a Target Level of Safety Approach Richard Melnyk,1,∗ Daniel Schrage,1 Vitali Volovoi,2 and Hernando Jimenez1
One of the most critical challenges to full integration of unmanned aircraft systems (UAS) into the National Airspace System (NAS) is the requirement to comply with CFR 14 Part 91.113 to “see and avoid” other aircraft. Various attempts have been made to develop systems to “sense and avoid” other aircraft so UAS can comply with the intent of the regulation. This article proposes a framework to develop effectiveness requirements for any SAA system by linking UAS characteristics and operating environments to midair collision risk quantified by a fatality rate. The framework consists of a target level of safety (TLS) approach using an event tree format. Safety has been identified as the most important consideration in the UAS integration process. While safety can be defined in many ways, the authors propose using a fatality rate metric that follows other statistics used in the industry. This metric allows for the use of a TLS approach to the development of SAA requirements for system certification. Failure to adequately link system requirements to safety could result in the implementation of SAA systems that either do not adequately mitigate the risk associated with UAS operations or are overdesigned, resulting in increased cost and complexity. This article demonstrates the use of the proposed framework to develop specific SAA effectiveness standards based on UAS weight and airspace class combinations. KEY WORDS: Event tree; fatality rate; midair collision; sense and avoid (SAA); target level of safety; unmanned aircraft systems (UAS)
1. INTRODUCTION
wants to unleash the potential of UAS to generate revenue.(2,3) However, the reality of the situation remains that UAS operations in the NAS have been limited to military operations in “restricted” airspace and limited public/civil usage under restrictive Certificates of Authorization or Waiver (COAs). One of the most often cited obstacles to integration is the inherent lack of a sense and avoid (SAA) capability on UAS to comply with Code of Federal Regulations (CFR) 14 Part 91.113.(4) There have been several efforts to develop and test an SAA capability for UAS, including a ground-based system tested by the U.S. Army and an air-based one tested by the U.S. Air Force. The purpose of this article is to propose a framework to develop effectiveness standards for SAA systems developed for UAS. Effectiveness in this
Full integration of unmanned aircraft systems (UAS) into the National Airspace System (NAS) is the goal of many stakeholders in the aerospace community, in addition to being mandated by law as of February 2012.(1) The Department of Defense seeks greater access to training sites, other public users would like to use UAS for operations such as law enforcement and disaster relief, and the civil sector
1 School
of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA. 2 Independent Consultant, Alpharetta, GA, USA. ∗ Address correspondence to Richard V. Melnyk, School of Aerospace Engineering, Georgia Institute of Technology, 270 Ferst Drive, Atlanta, GA 30332, USA; [email protected].
1894
C 2014 Society for Risk Analysis 0272-4332/14/0100-1894$22.00/1
Sense and Avoid Requirements for UAS Using a TLS Approach context is defined as the combination of reliability and efficacy that indicates to what extent a system performs its intended function and is not prevented from doing so due to either system failures or insufficient performance standards.(5) In order to develop minimum effectiveness standards for SAA, the authors propose utilizing a framework that includes a target level of safety (TLS) approach to the problem and an event tree (ET) format risk model to predict midair collision (MAC) fatality rates resulting from UAS operations. This article complements similar work by the authors predicting third-party fatality rates on the ground from UAS operations.(6) 2. BACKGROUND It is impossible to design an SAA subsystem for UAS, or for any aircraft for that matter, without specific performance and reliability standards. These standards should include parameters such as the minimum intruder detection distance based on closure rates and the ability to maneuver to avoid collisions. They should also include a reliability or availability standard to ensure that the system maintains functionality. The parameters would also likely include the number of potential intruders the system must track, the accuracy of any positioning system, and a maximum number of false alerts. It was the opinion of the Federal Aviation Administration’s (FFA’s) Sense and Avoid Workshop that the foundation for these performance standards be a TLS approach.(7) The TLS has been defined as a “top-down approach which focuses on safety critical issues which could affect achievement of the safety target” by Haddon and Whittaker.(8) The metric used to define the TLS in the framework for this article is a midair or second-party fatality rate, or fatalities per UAS flight hour (FH). This metric was chosen because it mirrors the statistics already in use by the FAA for the NAS, and it also matches the metrics used by the Range Commander’s Council (RCC),(9) the Nuclear Regulatory Commission (NRC),(10) and other agencies concerned with safety such as the National Transportation Safety Board.(11) The real purpose behind using the TLS approach to this problem is to link the effectiveness of any SAA subsystem to a metric that matters most to the FAA, stakeholders, and the public. That metric should be safety related, which is why the fatality rate is suggested. Without linking SAA standards to safety, any design choices and tradeoffs made on the subsystem will not necessarily lead to safer skies.
1895
ETs are established reliability tools first used by the NRC in the 1970s.(12) The authors chose to use an ET format to model UAS operations in the NAS for several reasons. First, they provide a means to visually trace both the probability and impact of events, which can be either failures or normal operating events.(5) Second, fatalities in the NAS are extremely rare events. To properly identify and analyze events that only occur sometimes on the order of 1 × 10−8 using a model that replicates the behavior of the NAS like an agent-based model would require 1 × 109 simulations, at a minimum.(13) ETs combined with Monte Carlo simulation, on the other hand, can capture the effects of improbable events because even unlikely events can be featured in a branch of the tree. 3. FRAMEWORK DETAILS The primary ET used to model the risk of fatalities from UAS operations appears in Fig. 1. The ET is a series of branches that detail different options for the air environment, mitigation, and event outcomes based on the probability of each of the branches occurring and the effects (fatalities) of progressing along a particular branch. The overarching equation that governed the model appears in Equation (1), where Ec is the “expected casualty” (fatality) rate in fatalities per FH. Ec = Encounter Rate ×
3
P (Collision)i
i=1
× Effectsi
(1)
The encounter rate is further explained in Equation (5), the probability of a collision occurring in Equation (2), and the effects in Equation (7). Using the yellow path in the ET as an example, the probability of collision or P(Collision)1 appears in Equation (2): P (IFR) × (1 − P (SEPARATION|IFR) ×(1 − P (AVOIDANCE|IFR) , (2) where IFR is instrument flight rules. The term effects refers to the number of casualties per collision and depends on a number of factors to include the types of intruder aircraft in the representative airspace and the likelihood that the UAS penetrates the intruder aircraft. The effects blocks in
1896
Melnyk et al.
Fig. 1. Event tree.
Fig. 1 actually have further branches, but in the interest of space the tree is terminated and a more detailed depiction on the “Effects” branch and the associated equations will be discussed in Section 3.3. The likelihood of progressing along the branches, and the effects that occur on branches that result in collisions, is dependent on many factors related to the airspace and traffic environment. To allow for a high degree of flexibility and capture the dependencies associated with the model, the software tool @RiskTM was used to represent particular parameters in the model as probability distributions. This spreadsheet-based tool allows the user to substitute deterministic cells in the spreadsheet for probability distributions for use in a Monte Carlo simulation. For example, some of the variables required to calculate the encounter rate were modeled as probability distributions to capture some of the uncertainty associated with the air environment, such as intruder aircraft speeds. The probability of separation and probability of avoidance, which will be explained in more detail later, were modeled for some supporting analysis as deterministic values and in other cases as uniform distributions to capture a range of values or, in the case of avoidance, as a discrete distribution. Several of the factors that determine the fatalities per collision could also be modeled as probability distributions, if desired. As a result of this approach, the analyst is able to propagate some of the uncertainty
Table I. Example Use of Event Tree Parameter Encounter rate Probability of IFR P (IFR) Probability of separation P (Separation) Probability of avoidance P (Avoid) Effects
Value
Units
0.00001 75 99.99
Encounters per FH % %
75
%
50
Fatalities per collision
inherent in the input variables to this model to the resulting probability distribution representing the output or fatality rate. However, to demonstrate the use of the model in a deterministic manner, an example is offered following the yellow path in Fig. 1 (color visible in online version) and using the values from Table I. In this example, the equation to determine the fatality rate would be: Ec = Encounter Rate × ProportionIFR × (1 − P (Separation)) × (1 − P (Avoid)) × Effects or Ec = 0.00001 × 0.75 × (1 − 0.9999) × (1 − 0.75) × 50 = 9.375 × 10−7 Fatalities/FH. (3)
Sense and Avoid Requirements for UAS Using a TLS Approach
1897
Table II. Airspace Volume and Density Data Used for Model Validation Information
Value
Average annual total U.S. flight hours for GA Average annual total U.S. flight hours for all others Average GA aircraft aloft Average aircraft aloft all others Total U.S. land area Location of all MACs in 1994–1999 data Total U.S. airspace volume for GA Total U.S. airspace volume for all others Average airspace density
Units
25,396 19,785 2899 2258 3,796,742.2 Below 6,000 2,830,685.0 16,512,329.4 1.16 × 10−3
Table III. Velocity Values Information Manned traffic speeds
Relative velocity with intruders
Knots Triangular distribution: Min: 50.36 Peak: 100.72 Max: 302.17 141.918 + 0.0127 × UAS weight
Source
1000s FH
RITA database(22–25)
Aircraft/hour
Calculated
Mi2 Ft Nm3
Census data(26) NTSB data(27) Calculated
Aircraft/Nm3
Calculated
Table IV. IFR Statistics in the NAS Source
Average of radar reports(28)
Regression based on relative velocity simulations
Information Proportion of VFR time in NAS P (VFR) Proportion of IFR time in NAS P (IFR)
Percent
Source
Normal distribution: Mean: 83.6 Std. dev.: 8.8 Aggregate NAS statistics paper(21) Normal distribution: Mean: 16.4 Std. dev.: 8.8
Table V. Probability of Avoidance by Traffic Combination(15)
However, as noted earlier, the values used to populate the ET determine the encounter rate and calculate the effects of collisions, which is much more complex and dependent on the type of airspace and the composition of the intruder aircraft in that airspace. The airspace environment helps determine the airspace density, the proportion of “visual” and “instrument” traffic, and the speeds of the intruder aircraft, which in turn determines the encounter rate and the likelihood of separation and avoidance. In addition, the composition of the types of intruder aircraft plays a role in determining the relative speeds of the aircraft pairs, the likelihood of fatalities in the event of a collision, and the number of fatalities per collision. 3.1. Encounter Rate Using a Gas Particle Model An important component of the model was the encounter rate. Values for the probability of an encounter were based on a gas particle model that treated aircraft in a specified volume of airspace as randomly moving gas particles.(14–18) The airspace density (ρ) represents the density of the airspace in question in aircraft per volume. Other aircraft in the
Type of Encounter
P (Avoid)
Military/military Military/general Military/domestic Domestic/domestic Domestic/general General/general
20 20 20 75 75 85
airspace were treated as motionless particles, while the UAS was treated as a cylinder sweeping out volume at a rate based on the vertical (V) and horizontal (H) dimensions of the encounter cylinder and the relative velocities (VREL ) between the UAS (V1 ) and manned aircraft (V2 ). An encounter is defined as any time that the swept cylinder overlaps a particle in the volume. The angle between particles (δ) was assumed to be random. UAS encounters or collisions with other UAS were not considered for this study because the focus was on determining fatality rates due to UAS operations and it was assumed that no unmanned flight operations with people on board would occur in the near future.(15) Encounter Rate = 4 × ρ × H × V × VREL
(4)
1898
Melnyk et al. Table VI. Pilot Visual Avoidance Probabilities
Information
Probability of pilot visually detecting other aircraft
P (Avoid)
The unstructured nature in which the gas particle model portrays aircraft in the airspace is not entirely indicative of the measures of control inherent in the NAS. However, for the purpose of this research, given its overall statistical nature, the random motion of particles replicated by this model was favored. While this model is less realistic around airports or along airways, it provides a good statistical representation of overall activity in the NAS. In fact, with the advent of trajectory-based operations (TBOs) in the near future under NextGEN, aircraft behavior will actually more closely resemble random trajectories and rely less on more ordered trajectories focused on ground-based navigational aids.(29)
Source
100 knot closure 72.3–100 speed 200 knot closure 30.2–90.7 Aviation speed medicine article(34) 300 knot closure 16.2–48.7 speed 400 knot closure 9.2–27.6 speed 500 knot closure 5.0–15.0 speed
2 VREL = V12 + V22 − 2 × V1 × V2 × cos (δ)
(5)
The dimensions of the cylinder were based on physical dimensions of the UAS and not a larger separation volume, as other applications have used. That decision was based on the focus and purpose of this study. Previous uses of gas particle models were primarily for air traffic or collision avoidance purposes so the goal of those studies was to determine the likelihood of two aircraft requiring additional separation measures either from a controller or collision avoidance system. However, given the safety-related focus in this study on fatality rates, the primary concern was actual collisions, not near MACs. Thus, in this study, an “encounter” is effectively a collision if no separation or avoidance measures are in place. Separation and avoidance will be described in Section 3.3. If the study was expanded to determine the impact of UAS operations on air traffic and separation, then the larger separation volume could be used. First, the authors developed characteristics of the air environment to include the density of aircraft in the airspace in question, the type of aircraft in the environment,(19,20) and the type of airspace control or architecture.(21) The average aircraft density across the NAS was estimated through statistics on aircraft flights in the NAS,(22–25) which appears in Table II. The velocities used for manned aircraft to determine the relative velocity profiles appear in Table III. The manned traffic speeds are based on radar reports and historical data. The relative velocity, using Equation (6), is based on conducting a simulation substituting the manned traffic speeds distribution for V2 , a uniform distribution for the approach angle that ranged between 0° and 360°, and a velocity distribution for UAS for V1 that appears in Table VII.
3.2. Risk Mitigation Through Separation and Avoidance The third feature of the ET model was risk mitigation in the form of separation or avoidance. To describe the difference between separation and avoidance, it is instructive to review the nature of the gas particle model. In this model, the encounter rate quantifies the rate at which randomly moving particles would collide in a specified volume. Separation refers to proactive measures taken by agents external to the aircraft pair to prevent the encounter from occurring in the first place. Avoidance, on the other hand, refers to reactive measures taken by one or both aircraft in the pair to prevent a collision in the event that separation measures failed. In automobile terms, a traffic light would be a separation measure while swerving to prevent an accident with a car that ran a red light is avoidance. 3.2.1. Separation The probability of preventing a collision through separation was based on a combination of factors to include whether or not the airspace was under positive control by air traffic management (ATM). The success rate in preventing collisions through ATM control was based on statistics of the NAS.(13,30) The degree to which separation services are available depends on whether the aircraft operate under visual or instrument flight rules (VFR or IFR). This branch probability was based on statistics of the NAS from Table IV. To predict how effective the controller is at maintaining aircraft separation depends on many complex factors, including the type and amount of traffic, human factors, and a number
Sense and Avoid Requirements for UAS Using a TLS Approach
1899
UAS Mass
UAS Velocity
Relative Velocity
Kinetic Energy Penetration Probability
Intruder Velocity
Intruder Type
Number of Casualties
Number of Seats Number of Passengers Load Factor
Primary Information
Results
Derived Information
Fig. 2. Effects.
Table VII. UAS Parameters for Model Information
Value
Units
Speed
88.18 + 0.0279 × Weight
Wingspan
12.015368 ft + 0.0140653 × Weight − 4.6313e − 7 × (Weight − 1807.6)2 0.5073 × Weight lbs
Fuel load
Endurance
6.0627732 + 0.0079073 × Weight
ft/s
Source Regression based on weight Regression based on weight
Regression based on weight hours Regression based on weight
Table VIII. Manned Aircraft Airspace Composition Citation Ref. 41
Information General aviation Percent of total hours Air carrier
Percent
Source
59
GAMA databook and RITA database(19,20) FAA factbook CY 10 data(39)
41
Ref. 42
Ref. 43
Refs. 44, 43
of unique situations. However, to model the separation probability for an ET-based model such as this required using known values in an educated manner. First, the FAA publishes system reliability standards for its architecture. A critical system is required to have a reliability of 0.99999 or one failure every 100,000 cycles.(31) This takes into account the air traffic control system, but not the possibil-
Breakdown of air carrier aircraft by type Load factor for air carriers
Jet
55.0
All others
45.0
Normal distribution: Mean: 76.765 Std. Dev.: 0.457
FAA statistics(40)
ity of human error or errors on the part of the aircraft instrumentation that would affect proper separation. International standards for midair accidents due to air traffic control are 1.55 × 10−8 per FH.(13) This represents an extremely high level of reliability and effectiveness. Statistics for a 10-year period in U.S. airspace indicate that only 10 MACs occurred as a result of air traffic control issues, which when taken against all of the flights that occurred
1900
Melnyk et al.
Table IX. Air Vehicle Parameters Used for Model Validation Information
Value
Traffic speeds
Triangular distribution: Min: 50.36 Peak: 100.72 Max: 302.17 173 General aviation velocity (speed) Vertical cylinder 36 height (V)
Units
Source
Knots
Average of radar reports(28)
Knots
Waggoner study(45)
Ft
Average of studies and information on GA aircraft
36
Horizontal radius of cylinder (H; wingspan)
Table X. Historical Fatality Data Used for Model Validation Information Midair collision (MAC) rate for GA MAC rate for all aircraft Fatality rate for MACs Casualties per MACs
Value 5.9 × 10−7
Units Midair per flight hour
3.74 × 10−7 6.82 × 10−7 1.76
Source
NTSB data(46) Fatality per flight hour Fatality per midair
in a 10-year period, represents an almost negligible amount.(30) As a result, for the purpose of this model, the authors assumed that the probability of separation would be 99.99% when under IFR circumstances or in VFR controlled airspace. 3.2.2. Avoidance Avoidance rates for the aircraft in the environment were based on a combination of studies that detailed avoidance rates by both visual and electronic means,(15,32–35) and were largely based on the closure rates between manned aircraft and UAS. Avoidance is a highly complex behavior that depends on the angle of approach, closing speed, the accuracy of position reporting devices, the skill and alertness of the pilot in the manned aircraft, and the size of the aircraft involved, among other things. Despite the complexity of this aspect of the air collision risk
model, it is important to reflect the likelihood of avoidance as accurately as possible because this area represents one of the most controversial integration topics. To model the probability of avoidance, several studies were examined. Visual avoidance of other aircraft is a topic that has been studied in detail. There are studies dating back over half a century that quantify the probability that a pilot would see another aircraft either to avoid that aircraft or, in the case of some military studies, to attempt to engage that aircraft in air to air combat. Regardless, there are data available to quantify the probability of aircraft detection. Because this model deals with only one piloted aircraft, the fact that only one of the two parties was using visual detection means had to be taken into account. The aforementioned article on MACs by May cites figures from the Swedish Air Traffic Control Commission that quantifies the probability of avoidance, given that an encounter has occurred. The values, which are featured in Table V, depend on whether the aircraft in question are military, domestic (commercial), or general (general aviation [GA]). The differences in values reflect the average cruising speed of the different categories. For the type of traffic used in this model, the bottom three values of either 75% or 85% would be most appropriate. Another study conducted for the FAA in the 1980s also looked at see and avoid effectiveness based on closure rates. The pertinent results of the study appear in Table VI. The probability of not only detecting but avoiding another aircraft, given an encounter, range from 97% for lower closing speeds to as low as 47% for the highest closing speeds. An earlier article that used NMAC data from 1968 and simulations to estimate the effectiveness of see and avoid rated the probability of avoiding a midair collision (MAC) at 95% if the closure rate was less than 100 knots and as low as 32% for speeds over 400 knots. Based on average values of closure rate used in the simulation validation, the probability of detection and avoidance is approximately 85%, which is in the middle of the range corresponding to the 100 knot closure speed from the study for the FAA, which appears in Table V. Studies have also been conducted on the effectiveness of electronic avoidance such as the traffic alert and collision avoidance system (TCAS). One study by the MITRE Corporation, completed for the Department of Transportation, lists the effectiveness percentages of TCAS avoiding collisions based on varying levels of equipment usage. The report states
Sense and Avoid Requirements for UAS Using a TLS Approach
1901
Table XI. Air Environment Data Used for Midair Collision Fatality Estimation Information General aviation
Percent of total hours
Air carrier Breakdown of air carrier aircraft by type
Jet
Load factor for air carriers
Passengers per air carrier Energy absorption for Part 25 jet aircraft
All others Normal distribution: Mean: 76.765 Std. dev.: 0.457 106.11 10,000
that TCAS is capable of an avoidance probability of 94.7%. However, that requires all aircraft in question to be equipped with both TCAS and Mode C altitude reporting equipment.(36) When all aircraft are not equipped with TCAS or if some systems are not capable of reporting altitude information this avoidance rate can drop to as low as 57.6%. Based on the statistics from the abovementioned MITRE study detailing what portion of the NAS was equipped with equipment such as transponders, the avoidance rate would be approximately 75%. Because it was impossible to determine exactly what proportion of the NAS had some form of electronic avoidance measure or position reporting equipment, it was assumed that the ability to avoid a collision by electronic means was 75%. In IFR conditions, electronic means were the only method available for avoidance so the values from the MITRE study were used. Under VFR conditions the MITRE avoidance values and visual avoidance values from Table VI based on closure rates were combined in the model. 3.3. Effects: Expected Fatalities Due to an MAC The final component of the air risk ET was the effects that would occur if a MAC did occur. A depiction of the factors related to the fatalities per collision, or effects, appears in Fig. 2. This figure represents the extension of the ET effects blocks from Fig. 1. This segment consisted of determining the probability that an MAC between a UAS and a manned aircraft would cause fatalities and, if so, how many. The probability that an MAC would result in fatalities was based on studies of both bird strikes on air-
Value
Units
59
%
41 55.0
Source GAMA databook and RITA database(19,20) FAA factbook CY 10 data(39)
45.0 FAA statistics(40)
People/aircraft Joules
FAA factbook(39) Bird strike report(37)
craft from CFR 14 Part 25.631 and the European Aviation Safety Agency (EASA) Part CS-25,(4,37) and the risk to aircraft from falling debris from the RCC.(38) The RCC discusses the size of fragments that have the potential to cause damage to aircraft, and there is the potential for catastrophic damage for objects weighing even less than one pound. The equation to determine effects was: Casualties = P (Penetration|Collision) Collision × Number of Seats × Load Factor,(6) where P (Penetration|Collision) ∼ f (Intruder Type, Collision Kinetic Energy). (7) The U.S. and European certification requirements only require sufficient bird strike protection for Part 25 aircraft and a review of the energy threshold requirements reveals that only jets operating under Part 25 are required to have protection that could prevent a catastrophe in collisions with all but the smallest of UAS. Therefore, the assumption was made that if any collision with a UAS occurred with any aircraft in the model other than a Part 25 jet, then all aboard that aircraft were considered fatalities. This is a conservative approach, but necessary based on the lack of available studies into the effects of aircraft collisions with UAS. For jets operating under Part 25, a collision was deemed not to cause casualties if the energy associated with the UAS impact was less than the conservative end of the protection threshold, or 10 kJ. If a catastrophic collision did occur, the number of casualties caused was
1902
Melnyk et al.
Air Casualties / Ec: Regression Coefficients Wingspan
0.46
Air Density
0.31
Controlled Airspace
-0.29
SAA Effectiveness Rate
-0.25
Other Pilot Avoid
-0.22
Speed
0.09
IFR Rate:
-0.03
ATM Effectiveness
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
-0.3
-0.02
Coefficient Value Fig. 3. Sensitivity analysis for air risk model.
based on the type of aircraft in the NAS, their passenger capacity,(39) and load factor statistics.(40) Information about the representative UAS in the model was obtained by examining the current characteristics of the most common UAS in use at the time of writing this effort. The values for the UAS characteristics appear in Table VII. It was possible to establish relationships between the parameters mentioned below and the UAS weight. Information regarding other aircraft in the airspace was obtained from several sources related to the NAS. These data were used to calculate the likelihood of encountering different types of aircraft to base the probability of penetration on. The authors also used these values to determine the number of passengers per intruder aircraft. The data in question appear in Table VIII. 3.4. Validation of the Model A shortcoming of previous efforts to examine this problem was the lack of a method to validate the prediction model. UAS have not been able to operate in the NAS alongside manned aircraft to date except under very strict control measures, and statistics on encounters between UAS and manned aircraft in places like Iraq and Afghanistan were unavailable. The authors validated the model in this study by using historical data for GA MACs and fatalities. In other words, the authors substituted a representative
Table XII. Recommended UAS Categories by Weight Group
Weight (lbs)
Source
Group 1 (micro) Group 2 (mini) Group 3 (small)
