From this week's Air Safety Week

Single Index Proposed to Measure Safety of Aviation System

A new aviation safety index - analogous to the famous Dow Jones measure of stock performance - will be used to measure the risk of death and injury in the aviation industry. Developed by the Federal Aviation Administration (FAA), the index applies to all people exposed to risk from the civil aviation system.

"This new index will serve as a vital trend indicator," according to the FAA's draft Flight Plan 2004-2008. This document lays out the agency's strategic goals. The safety index is touted as a tool to "help provide a more robust indicator of the state of aviation safety."

The index builds upon the work of Professor Arnold Barnett of the Massachusetts Institute of Technology (MIT), who for a decade has advocated quantitative measures of safety for the nation's air transportation system. Barnett supported the approach: "I applaud the idea of a statistic that summarizes recent death and injury risks over various forms of aviation. Of course, no one indicator is perfect: Some up-and-down movements over time will reflect little more than sheer luck, while some growing menaces might elude the statistic because they have not yet caused any accidents. Moreover, the initial version of the FAA indicator has to cope with data limitations we should strive to circumvent. But, while the indicator isn't everything, is there any other place to start a serious appraisal of system safety besides a clear 'bottom line' statement about how well we're doing now? Fortunately, the FAA thinks not."

Other experts were more skeptical. ASW contributing editor Alex Richman (AlgoPlus Consulting Ltd.), a veteran massager of the aviation service difficulty report (SDR) database, noted that the FAA posed a question for which the answer is questionable. In an FAA internal memorandum explaining how the safety index is computed, the question was asked in the section titled "Next Steps" whether a "single index, regardless of its nature" should be pursued.

"Clearly," the memorandum said, "the FAA believes the answer is 'yes,' as the work outlined in this document would suggest."

Richman demurred: "This conclusion is hard to understand. The document ... demonstrates that it is possible to create [a single index], but not that it is valuable."

Richman pointed out that the Dow Jones index embodies ten sub-indices of various industry sectors "to supplement the single index." A similar approach could be taken for aviation safety, he argued, since the relative safety records vary considerably between the large scheduled carriers and, say, general aviation (GA).

Other experts offered guarded plaudits for the concept, while voicing reservations about the computational details.

"Establishing a statistical level of risk from the perspective of aircraft occupants is valid and worthwhile, and we like the definition of risk as 'frequency plus exposure,' " said Nick Lacey, a consultant with Morten Beyer & Agnew in Virginia. Lacey was formerly the associate FAA administrator for regulation and certification and is familiar with the strengths and pitfalls of various approaches to quantifying air safety.

"This attempt is a good start. It also shows how difficult it is to develop a taxonomy," he added. Lacey said "there is too much in the bucket" and that the index should be limited to U.S. airspace, U.S. airports and U.S. operators. This approach would provide a more meaningful index "from the passenger's perspective," he asserted. For example, the index in its present form includes events like the fatal 1995 crash of an American Airlines [NYSE: AMR] jet on a mountain ridgeline near Cali, Columbia. "The index should [only] cover those factors over which the FAA has control," Lacey argued. By including flights that progress out of U.S. airspace, the FAA is incorporating into its safety index situations over which it has no control, such as out-of-country traffic separation and terrain separation.

Paul Hayes of UK-based Airclaims said,"The FAA is to be applauded in seeking new ways of handling and presenting accident statistics, and introducing some way to weight accidents for severity is the way to go."

"However, I do not see the point to having a single index which covers all types of aviation," he added.

Details of Lacey's and Hayes' concerns about methodology suggest that they share the same goal for any measure of safety - relevance and credibility.

Gary Eiff, an associate professor in the department of aviation technology at Indiana's Purdue University, also questioned the value of a single index. "Why are we doing this? It appears like the FAA is trying to show that things are getting better for its efforts, but there is a such a hodge-podge out there," he said, referring to aviation activity that runs the gamut from scheduled air carriers to enthusiasts flying their homebuilt ultralights.

Eiff added that the absence of incidents in the computations is a shortcoming. Incidents are often harbingers of disasters. "The only difference between an accident and an incident is circumstance," Eiff said. The difference between a runway collision and an incursion, or a mid-air near miss or collision, is frequently "a matter of a few feet or a few seconds," Eiff pointed out.

The index was characterized in the FAA's explanatory memorandum as a "work in progress," with further refinement to be expected.

The index covers in a single measure of safety four major areas of operation: (1) scheduled passenger and cargo airlines (Part 121), (2) scheduled regional airline operations with airplanes of 10 or more passenger seats (Part 121), (3) scheduled flights in aircraft with nine or fewer passenger seats and nonscheduled aviation operations (Part 135), and (4) general aviation (Part 91). According to the FAA's explanatory document, the "Personal Safety Index is meant to complement the current use of accident and fatal accident rates by integrating outcomes for all four major segments of aviation and thereby providing a single, brief indicator of how the entire system is performing in safety."

As an example, the index plotted in the FAA's Flight Plan 2004-2008 shows more than a 40 percent decline in the risk of death or injury over the past two decades, as measured in three-year rolling averages ending 2000-2002, starting from 1983-1985.

While the FAA is presenting the risk index as a single trend indicator for the entire aviation industry, it clearly could be calculated separately for each of the four major segments. Indeed, it could be calculated for individual operators, providing the sort of numeric index of safety that some legislators in Congress called for early in the tenure of the previous FAA administrator, Jane Garvey. Since the detailed data are aggregated to produce the overall index, it also appears possible to drill down, disaggregate the data, and produce a safety score for individual operators - a course likely to be fiercely resisted by the airlines.

The current administrator, Marion Blakey, has expressed her commitment to "performance based" measures of FAA and system performance and the Personal Safety Index appears to be an outgrowth of that theme.

The new measure is both similar to and yet unlike the Dow Jones index of stock performance. It is similar in the sense that the Dow Jones is a broad measure, not intended to be predictive, or to tell about the performance of a particular company. The FAA's prototype index is unlike the Dow Jones, in that it does not change on a daily basis and does not respond to transitory or short-term changes in system performance.

"Instead, it is intended as a first-order tool by which the FAA can get a quick sense of the health of the overall system," the FAA's explanatory document said.

The FAA's new index represents an attempt to shift the focus away from the traditional emphasis on fatalities to a broader estimation of systemwide safety. The index is the result of deaths and injuries divided by hours of exposure. However, a number of steps go into the determination of the numerator and denominator that are worth mentioning.

The numerator
Four steps go into determining the numerator:

Step 1: An injury score for an accident is calculated, whereby a death is accorded a weight of "1," a serious injury is considered 5.75 percent of a fatality and 162.8 minor injuries equal one death. These relative values for deaths and injuries are borrowed from those used by the Office of Management and Budget (OMB) for cost-benefit purposes. The weights are applied thusly:

Deaths + (serious injuries x .0575) + (minor injures/162.8) = Accident injury score

Thus, for the 1995 crash in Cali, Columbia, involving 160 fatalities and four serious injuries, the formula yields a score of 160.23 fatal equivalents.

Step 2: Account for variations in load factor. The severity of an accident can vary based on whether it is fully or lightly loaded. Cargo airplanes, corporate and personal flying also tend to involve aircraft with fewer people on board. Hence, in this step, the outcome from Step 1 is adjusted based on aircraft loading:

Accident injury score (160.23) x (aircraft capacity/occupants)

Step 3: Deaths and injuries to persons on the ground are included, using the same weights applied to persons on board. This step incorporates people who just happen to be in the impact area, such as the five persons killed when American Airlines Flight 587 crashed Nov. 12, 2001, in Belle Harbor, New York. As the FAA document explained, this step also captures "accident-related injuries to surface workers who perform various functions directly related to flight, such as fueling, towing, wing guides, etc." From 1983 through 2002, 14 surface workers were killed and 33 suffered serious injuries. Although these numbers "constitute a small share" of the human toll, "they impose a real risk within the system," the FAA document explained.

Exemplar accidents show how the event scores are calculated through Step 3

Step 4: The annual score is the sum of all event scores for a given year.

The denominator
The deaths and injuries must be presented in context - in terms of the number of people exposed in the system. The rate per number of aircraft departures would be one approach, but data on departures are not available for nonscheduled Part 135 operators or for Part 91 general aviation. "Consequently, the index defaults to flight hours as the basic building block for the denominator," the FAA document explained. A five-step process is involved:

Step 1: Part 121 carriers. Again, differences in load factors are accounted for by factoring in the number of passenger and crew seating positions, as in:

Exposure = Annual flying hours per aircraft type x number of seating positions

Step 2: Part 135 small commuters. This category includes scheduled commuter airplanes with single pilot operations and nine or fewer passenger seats. The formula assumes that nine seats are filled on every flight, hence:

Exposure = Total commuter hours x 9 seating positions

Step 3: Part 135 nonscheduled operations. This segment of the industry includes helicopters, jets with two-pilot crews, and single-pilot cargo operations. According to the FAA, this portion of the calculation "assumes an average capacity of 5.4 positions per flight hour." (Cont'd on p. 6)

Thus:

Exposure = Total flying hours x 5.4

The accuracy of flying-hour data in this category is questionable. In an August 20 letter to the FAA, the National Transportation Safety Board (NTSB) said operators should be required to report flying hours. The use of a sample with only a 53.9 percent response rate "poorly represents actual activity for nonscheduled Part 135 operations," the NTSB noted. The NTSB noted that the FAA determined the preliminary activity data for 2001 had been underestimated (see http://www.ntsb.gov/Recs/letters/2003/A03_37_39.pdf).

Step 4: General aviation (Part 91). For these generally smaller aircraft, the index applies a constant average of 3.5 occupied seat positions:

Exposure = Total flying hours x 3.5

Step 5: Total exposure is the sum of the four calculations above.

With a total system-wide numerator and denominator in hand for each year, the index is calculated as a three-year rolling average. This stratagem is used "to minimize the possible distortions caused by annual variations," according to the FAA document. Barnett, e-mail abarnett@mit.edu; Richman, e- mail arichman@algoplusaviation.com

The Problem With Deaths
From the FAA's 'Personal Safety Index' explanatory paper (extracts):

"Major fatal accidents ... have become such rare events that a single accident is now the difference between a good year and a bad year.

"The weakness here is that the number of fatalities can be utterly random. For example, all 230 occupants died on TWA-800. Yet that flight could have had twice as many or only half as many people on board. If either had been the case, a simple count of fatalities would imply that the risk is twice as great or only half as great, though the nature of the event and the inherent risk it posed would not have changed. A random load factor for a given flight on a given day would largely determine the meaning of an accident.

"Simultaneously, government, the media and the general public, typically focus on Part 121 carriers, as that is the element in which common carriage affects the largest population. Yet, in a typical year, GA [general aviation] will account for 5 to 10 times as many fatalities.

"Given the fundamental differences in these aviation segments, the focus on fatalities across the system would indicate a need to shift resources away from the system in which most of the public travels in order to address a very different segment of aviation, which is dominated by owner-operators. In short, a simple count of fatalities or the use of four fatality rates may not allow for a quick sense of how things are going overall."

Source: FAA