J U L Y 2 0 2 0 • M A T E R I A L S E V A L U A T I O N 873 2.56 million flights (Oliver et al. 2017). While overall air safety has been improving, incidents of the loss of control have not. Loss of control occurs when pilots fail to recognize and correct a potentially dangerous situation, causing an aircraft to enter an unstable condition. One example of the possible catastrophic consequences of automation is the tragic crash of flight AF447 (Oliver et al. 2017). Another very recent example concerns the Lion Air and Ethiopian Airlines crashes of Boeing 737 Max aircraft (Rice and Winter 2019). Such incidents are typically triggered by unexpected, unusual events, often comprising multiple conditions that rarely occur together, that fall outside of the normal repertoire of the pilot’s experience. In the case of the Boeing 737 Max, the handoff between the human and autopilot systems appears to have not been designed well, and many pilots were not instructed on its operation (Rice and Winter 2019). This paradox of almost totally safe systems, where the same tech- nology that allows the systems to be efficient and largely error-free, can also create systemic vulnerabilities that result in occasional catastrophes. Lessons learned from these cases include: (1) avoiding the cycle of implementing more automation to correct for poor human performance with existing autopilot systems (2) encouraging more hand-flying to prevent the erosion of basic piloting skills (3) improving the management of handovers from machines to humans (4) increasing pilot training for rare events and (5) supple- menting training using simulation of various rare event scenarios. It is critical to avoid the natural tendency to blame the human in these situations when the human-machine inter- face and/or algorithm design is poor (Hao 2019). Alterna- tively, it is important to find ways to make such systems robust and ideally “anti-fragile” to randomness and disorder in the environment (Taleb 2012). The human operator must have some level of “skin in the game” (Taleb 2012) and not become reliant on automation over time. Designing human- machine interfaces and providing the necessary training to achieve this balance is far from trivial. Best Practices for Design of IA and Human-Machine Interfaces Building on this prior work and experience, a series of best practices for IA in NDT 4.0 is proposed, highlighting how the operator should best interface with NDT data and algorithms. Algorithms clearly have a great potential to help alleviate the burden of big data in NDT however, it is important that operators are appropriately involved in secondary indication review and the detection of rare event conditions. The following best practices are proposed: l Provide inspectors with a natural user interface for NDT workflow management. Usability of human-machine inter- faces is a critical aspect of workflow management for NDT techniques, from setup, standardization, data acquisition, and indication review. Ideally, inspectors need a way to report results and efficiently provide feedback on indications. Frequently, there are means in NDT software systems to annotate indication results however, making this metadata readily available to external systems is one of the challenges for NDT 4.0 going forward. Such information will be very useful for refining NDT algorithms and improving life-cycle management. l Implement data analysis algorithms to address frequent NDT calls and complex data interpretation. It is important to address the low-hanging fruit on implementing algorithms for NDT applications and to help alleviate the burden for inspec- tors of reviewing “mostly good” data. As well, some complex interpretation problems (especially in ultrasonic NDT) can benefit from algorithms and data guides. The design of these algorithms requires a focus on the base capability for making NDT indication calls to provide value and help ensure relia- bility. The algorithm design process should consider the necessary engineering development time, cost for acquiring necessary data, and the approach with the highest likelihood of success. There will be a payoff for some applications, but not all applications may benefit from automation. However, as NDT 4.0 systems mature, development costs for each new application should be reduced. l Ensure inspectors provide a secondary review of indications and review data for rare events. While there is often an initial desire to have NDT algorithms make all indication calls and present simple (good or bad) calls, based on prior experi- ence, additional information is always requested by engi- neering and management to understand the details on why an indication call was made. Inspectors need a natural user interface to review each call with supporting data and provide feedback on the call details in light of the technical require- ments. As well, because no algorithm will be perfect, inspec- tors need to have a straightforward means to review NDT data quickly. This entails identifying rare indications and determining when the acquisition of the NDT data is out of specification. l Develop an integrated NDT “simulator” to provide operator training and support complex indication review. There is a potential to leverage the same software interface for training purposes, by having the operators periodically train and test their skills with various conditions in NDT data. Specific rare events can be stored and introduced periodically as part of the regular re-training process. Thus, the interface could be used similar to how flight simulators are used for pilots to verify their performance under standard conditions and rare events. As well, integrated models within the user interface can also provide a means for the verification of indications and support sizing by the inspectors. l Implement open architecture for NDT data and reporting. Promising software tools exist to support NDT practitioners with data archiving, visualization, and special queries (Sharp et al. 2009), and continued improvements with usability and functionality are expected in the future. Ideally, to share data between NDT 4.0 components, leveraging open data

874 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 0 standards (such as DICONDE and HDF5) and incorpo- rating flexible software architectures, will greatly accelerate the evolution of these systems (Meier et al. 2017 Vrana et al. 2018). l Reliability must be demonstrated for NDT 4.0 systems. The capability of inspection procedures incorporating NDT 4.0 systems that depend on the performance of both algorithms and the NDT inspector must be evaluated jointly. Probability of detection (POD) evaluation procedures, such as MIL-HDBK-1823A (US DOD 2009), are designed to validate the reliability of NDT techniques, regardless of how the indication call is made. l Software and algorithms can also support NDT reliability as process controls. Simply demonstrating POD capability does not ensure reliability of the technique (Rummel 2010). FMEA should be performed for all NDT techniques incorpo- rating automation to understand the potential sources for poor reliability (Bertović 2016a). In practice, NDT reliability depends on a reproduceable calibration procedure and a repeatable inspection process (Rummel 2010). Process controls and algorithms can thus be used to ensure all cali- bration indications are verified and to track key metrics that show the NDT process is repeatable over time and under control. As an example, recent work on model-based inverse algorithms with eddy current inspections has shown the potential to reduce error due to variability in probes through calibration process controls (Aldrin et al. 2017). NDT 4.0 systems are also expected to improve the safety of inspections in dangerous environments. By collecting environmental conditions (using environmental sensors and/or weather monitoring) and test system state data from the site, one can ensure the reliability of the inspection task and reduce the level of risk for all involved. l Build trust over time and consider the cost-benefit for future algorithms and user interface enhancements. Managing costs and mitigating risk drive most decisions for NDT today. For organizations that depend on NDT, there are likely certain applications that will provide the greatest payoff in terms of cost and quality for their customers, transitioning from conventional NDT ME TECHNICAL PAPER w ia and human-machine interfaces Area being inspected Inboard Location of web Fasteners Exterior Interior Ultrasonic signal sent in at angle to propagate down vertical leg to fastener holes Transducer Wing skin Vertical leg Potential crack locations Figure 2. Inspection of beam cap holes in C-130 aircraft: (a) photo of area being inspected (looking forward) and (b) diagram of inspection problem (from Lindgren et al. 2005). (a) (b)