How eye tracking technology could improve safety on the flight deck

Analysing eye scan patterns is one of the most powerful methods for assessing a pilot’s cognitive processes in the cockpit. The visual information captured by eye trackers provides the opportunity to investigate the relationship between visual scan patterns and attention shifts while performing tasks. The application of eye tracking technology in the study of head-up display (HUD) design in a military cockpit is promising, as it provides direct feedback on human-computer interactions (HCI), which could diagnose potential factors that impact human performance, including the pilot's attention distribution, situation awareness (SA), and aeronautical decision-making (figure 1).  

Figure 1: The pilot’s fixation shifts between the HUD and the dynamic target, reflecting their attention, situation awareness and aeronautical decision-making.

A pilot’s attention shifts are related to cognitive processes, in relation to specific information from areas of interest (AOIs). A pilot has to perceive the stimuli of signals (information) in the cockpit, understand the encountering situation, and predict the possible consequences in the near future. As visual searching in a flight deck is critical for collecting information, it is the main reason that 75 per cent of pilot errors within the cockpit are caused by perceptual failures. 

This phenomenon highlights the importance of interface design in impacting pilots’ attention distributions and SA performance. By utilising a combination of an eye-tracking device and flight simulators, pilots’ eye movement patterns, including fixation numbers, fixation durations, saccades, and pupil dilations, could be collected for further analysis, and the results could be served as a feedback for improving interface design on HCI to improve human performance (figure 2). 

Figure 2: The sequences of a pilot’s fixation distributions is the window to approach HCI issues in flight operations.

Applying eye tracking technology to flight deck design

Monitoring the flight mode annunciator (FMA) and calling out the mode transitions are thought to be important for obtaining situation awareness on automation systems on the flight deck. The accident involving Asiana Airlines Flight 214 revealed that pilots can be easily confused by unexpected changes on the FMA.

Figure 3 shows a conventional primary flight display (PFD), including airspeed (left), attitude (centre), altitude (right) and navigation information (i.e. ILS deviation scales). The FMA is located on top of the display and shows "active" modes in green, as well as "armed" modes in white. The FMA is divided into three columns for autothrottle, roll-mode and pitch-mode. The red boxes in figure 3 list all the possible modes for each channel. The high cognitive effort required for interpretation can be demonstrated as follows: assuming the flight crew desires to find out if the airspeed is automatically controlled, they need to do the following:

• Read the FMA autothrottle column text.
• Interpret the text, as there are modes that cause the autothrottle to be "engaged", but not controlling the airspeed (e.g. "HOLD" or "THR").
• If the text is anything else than "SPD", the scanning continues to the pitch-mode, as this channel can also be used to control the airspeed. 
• Interpret the pitch-mode (check if it is FLCH SPD or VNAV SPD).
• Check the AFDS status indication, to ensure that the autopilot is engaged in case of airspeed being controlled via the pitch-mode.

Figure 3: The layout of primary flight display, including different functions on the FMA.

By improving the content and presentation of information, it is possible to facilitate pilots’ cognitive processes in decision-making. To precisely investigate pilots’ visual behaviours and interface design, eye-tracking technology is the best approach for flight deck design. It has the potential to facilitate system designers’ understanding of pilots’ cognitive processes relevant to attention distribution and situational awareness, and HCI issues on the interfaces in the flight deck.

In figure 4, the active flight modes "SPD", "LOC" and "G/S" are augmented with the respective parameters being highlighted. It is therefore instantly clear whether or not a certain parameter is controlled by the automation, simply by looking for the presence or absence of green rectangles. This new PFD layout aims to apply only small changes to the general appearance, in order to keep transition training for pilots to a minimum. At the same time, there is a distinct change in philosophy: instead of just displaying the flight mode as in the legacy design, the new design also highlights the parameter that is controlled. In simple terms, this could be described as showing what is controlled, rather than just how it is controlled. The basic FMA box is retained, while green rectangles are added for the parameters that are actively controlled by the automation.

Figure 4: Proposed flight mode annunciation concept by the "green rectangles".

View the eye tracking technology demonstration

How to investigate accidents related to decision-making on the flight deck

A key challenge for accident investigators is to understand why pilots do what they do in accident scenarios (such as shut down the wrong engine), and why the human error is not detected and corrected by the other crew members. Due to the limited information and parameters recorded on the flight data recorder (FDR) and the cockpit voice recorder (CVR), it is not possible to analyse the pilots’ attention distributions and situation awareness that is inextricably linked with the aircraft and flight parameters. The availability of a tool to record pilots’ visual scan patterns in the flight deck would provide powerful evidence to facilitate scientific analysis of accident investigation. Since pilots’ visual behaviour plays a central role in cognitive processing, understanding when and where pilots distribute their attentions is critical to both situation awareness and safe operations (figure 5).

Left image: 5a; right image: 5b          

Figure 5: (5a) The "heat map" shows the pilot’s visual attention shifts among attitude, altitude, heading and airspeed on PFD; (5b) the pilot’s attention shifts among PFD, ND, MCP and EICAS. 

View the eye tracking technology demonstration.

Visual tracking technology has been proven, through empirical research, to have a significant contribution to advanced flight deck design and training in aviation for decades. With over 75 per cent of pilot errors caused by perceptual failures in the flight deck, the application of eye-tracking in the aviation domain has been offered as a possible solution. Eye tracking provides direct feedback of visual attention, which could help investigators to diagnose factors that could impact upon pilots’ attention, situation awareness and in-flight decisions on the flight deck. It is the best scientific technology to facilitate investigating the fundamental issues of human factors in aviation accidents.

Relevant research

Journal articles

1. Li, W-C., Yu, C-S., Braithwaite, G. & Greaves, M. (2016). Pilots’ Attention Distributions between Chasing a Moving Target and a Stationary Target. Aerospace Medicine and Human Performance, 87(12), 989-995.
2. Brenna, M. & Li, W-C. (2016). The Investigation of Automation Induced Active Failures in Flight Operations. Journal of Transport Information and Safety, 34(4), 63-69.
3. Yu, C-S., Wang, E. M., Li, W-C., Braithwaite, G. & Greaves, M. (2016). Pilots’ Visual Scan Pattern and Attention Distribution during the Pursuit of a Dynamic Target. Aerospace Medicine and Human Performance, 87(1), 40-47.
4. Carugati, C., Li, W. C., & Brennan, M. (2015). The Interaction between Pilots and the Automated Systems during the Landing Phase. Journal of Aviation Safety and Management, 2(3), 227-251.
5. Li, W. C., Li, L. W, Harris, D., & Hsu, Y. L. (2014). The Application of Aeronautical Decision-making Support Systems for Improving Pilots’ Performance in Flight Operations. Journal of Aeronautics, Astronautics and Aviation, 46(2), 114-123. doi:10.6125/14-0324-789
6. Yu, C. S., Wang, E. M., Li, W. C. & Braithwaite, G. (2014). Pilots’ Visual Scan Pattern and Situation Awareness in Flight Operations. Aerospace Medicine and Human Performance, 85(7), 708-714. doi: http://dx.doi.org/10.3357/ASEM.3847.2014

Conference papers

1. Li, W-C., Cao, J-Q., Lin, J., White, J., Braithwaite, G., & Greaves, M. (2017). The Evaluation of Pilot’s First Fixation and Response Time to Different Design of Alerting Message. In Proceedings of 2017 International Conference on Human Computer Interaction. Vancouver, Canada.
2. Wang, T, Li, W- C. & Lin, J. (2017). Lessons Learned from GE-235: Integrated Pilots’ Visual Parameters into Cockpit Recorders for Accident Investigation and Prevention. 2017 International Society of Air Safety Investigators Conference. San Diego, USA.
3. Li, W- C. & Wang, T. (2017). Beyond ATR-72 Accident Investigation: The Application of Eye Tracking Device in the Cockpit to Facilitate Training and Accident Investigation. European Society of Air Safety Investigators Seminar. Ljubljana, Slovenia.
4. Li, W-C., White, J., Braithwaite, G., Greaves, M. & Lin, J. (2016). The Evaluation of Pilot’s Situational Awareness during Mode Change on Flight Mode Annunciators. In Proceedings of 18th International Conference on Human Computer Interaction. Toronto, Canada.
5. Li, W-C., Hsu, C-K., Braithwaite, G. & Greaves, M. (2016). Pilots’ Scan-path and Attention Shifts during Tactical Manoeuvres. In Proceedings of the International Conference on Ergonomics & Human Factors 2016. Daventry, U.K.
6. Ho, H., Su, H., Li, W-C., Yu, C., & Braithwaite, G. (2016). Pilots’ Latency of First Fixation and Dwell among Regions of Interest on the Flight Deck. In Proceedings of 18th International Conference on Human Computer Interaction. Toronto, Canada.
7. Li, W-C., Braithwaite, G., Greaves, M., Hsu, C-K., & S-C. Lin (2016). The Evaluation of Pilot’s Attention Distributions on the Flight Deck. In Proceedings of International Conference on Human Computer Interaction in Aerospace. Paris, France.
8. Li, W-C., Lin, J. & Braithwaite, G. (2016). The Development of Eye Tracking in Aviation (ETA) Technique to Investigate Pilot’s Cognitive Processes of Attention and Decision-making in the Cockpit. 32nd Conference of the European Association for Aviation Psychology. Cascais, Portugal.
9. Li, W. C., Yu, C. S., Braithwaite, G. & Greaves, M. (2015). Interface Design and Pilot’s Attention Distribution during Pursuing a Dynamic Target. In Proceedings of 17th International Conference on Human Computer Interaction (pp. 408-415). Los Angeles, CA. doi:10.1007/978-3-319-20373-7_39
10. Li, W. C., Yu, C. S., Greaves, M., & Braithwaite, G. (2015). How Cockpit Design Impact on Pilot’s Attention Distribution during Aiming a Stationary Target. 6th International Conference on Applied Human Factors and Ergonomics (pp. 5663–5669). Las Vegas, NV. doi:10.1016/j.promfg.2015.07.781
11. Li, W. C., Yu, C. S., Braithwaite, G. & Greaves, M. (2015). The Analysis of Pilots Fixation Distribution for Performing Air-to-Air and Air-to-Surface Tasks. In Proceedings of the International Conference on Ergonomics & Human Factors 2015 (pp. 466-473). Daventry, UK.
12. Li, W. C., Yu, C. S., Li, L. W. & Greaves, M. (2014). Pilots’ Eye Movement Patterns during Performing Air-to-Air Mission. 31th Conference of the European Association for Aviation Psychology. Valletta, Malta.