concerns. However, small changes in sensor config- uration, such as coil tilt within eddy current sensors or slight depolarization of well-used ultrasonic trans- ducers, can influence the flaw detection response. Flaw-to-flaw variability can have a much greater impact on the NDT response. Previous studies have illustrated that the same size flaw can vary in ampli- tude response from an eddy current inspection by over 20% of a full screen height reading (Forsyth et al. 2015). Similar results can occur in ultrasonic testing as well as other NDT techniques. For ultrasound, fatigue crack morphology and tortuosity can affect a response. Local stress considerations from a fit-up of assemblies and changes due to use can vary crack closure, which, in turn, affects the magnitude of the ultrasonic signal. The variability can be addressed in simulation provided all the attributes of the flaw that affect detection are included in the simulation studies. This includes their interaction, which can become a very large study, especially when consider- ing engineering level validation of the simulation. While flaw-to-flaw variability can be broadly categorized as a function of the type of flaw, struc- tural variability can become much more challeng- ing in the analysis of NDT data. This is largely due to the extensive range of structures evaluated by NDT, which includes power generation, infrastruc- ture, and transportation, the latter which can be segmented into ground, aviation, and space catego- ries. In addition, other considerations include the materials being used, including metals, polymers, ceramics, and composites the manufacturing process being used, for example, automation, partial automation, or hand assembly plus, the assembly process used to join components, such as welding, fastening, and bonding. With all these parameters, it becomes very clear why NDT is the ultimate multi- disciplinary engineering domain! A significant challenge is how to evaluate the effect all these parameters, both individually and through important interactions, have on the NDT response. Consider the simple fastened joint between two metal surfaces, where up to 22 factors addressing equipment, flaws, and structure need to be included in a sensitivity study (Lindgren et al. 2007). Structural considerations include such things as composition of each layer the possibility of shims and their compo- sition assembly quality, such as fastener hole tilt or skewed fasteners and fit-up stresses as a function of what type of fastener is used and how it is installed. In addition, how these factors change as a function of time due to maintenance, repairs, modifications, and even use need to be included. Using AI/ML techniques for these applications can become very daunting when considering all the parameters that need to be addressed to make diagnostic decisions using automated processes. This includes how the statistical processes adjust to account for changes that occur as a function of time. In addition, how these affect the diagnostic capability of the NDT data must be validated to enable their use in system risk and life manage- ment. Therefore, the proper capturing of these factors in statistically representative methods presents itself as a significant challenge, but also a significant research and development opportunity. DAF Approach to AI/ML for NDT Data AFRL has been leading the development of algo- rithms to assist in the diagnostics of NDT data, including one of the first implementations for an aviation NDT application (Lindgren et al. 2005). Attributes that have made this approach successful include the use of multiple approaches to develop algorithms for the diagnostic capability combined with the approach that the algorithms will not replace all human interpretation of NDT data. The algorithms are used as a capability to facilitate and guide the interpretation to make the workload on an inspector easier and focused on the critical elements of the diagnostic process that do not easily lend themselves for automation. AFRL has called this approach intelligence augmentation (IA), but an alternative term being used in the sci- entific community is collaborative intelligence (CI) (Epstein 2015). This reflects how software tools and capabilities can be used to assist in the analysis of NDT data, which AFRL has named assisted data analysis (ADA). ADA algorithms combine multiple approaches to provide an optimized method to facilitate NDT diagnostics. These algorithms can be grouped into three general categories. The first uses heuristic-based methods that incorporate “rules of the road” that closely mimic the procedures by which inspectors interpret data. The second is a model-based inversion algorithm that uses simu- lation to represent the measurement response and iteratively solve for the unknown flaw or material state in the presence of variability. The third uses AI/ML methods trained using NDT data and as much diagnostics information as possible from available datasets. Frequently, the amount of well understood NDT data is much smaller than what would be required for robust AI/ML analysis, and likely requires supplementation from simulated data or transfer learning. Successful application of ADA has frequently included at least two of these approaches into an integrated diagnostic algorithm for the specific FEATURE |AI/ML 38 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 3 2307 ME July dup.indd 38 6/19/23 3:41 PM

NDT application being addressed. This includes the use of test data to ensure the intent of the application is being met and that the available data meets the needs of the application before a com- prehensive validation study is accomplished. The output of the ADA diagnostic is not the final dis- position of an indication. Depending on the appli- cation, the output enables inspectors to focus their attention on portions of the inspection data that have features of possible indications by screen- ing data with no attributes of a possible flaw. Alternatively, the output can be used to provide guidance on the nature of an indication so the proper disposition process can be rapidly iden- tified and implemented, minimizing the time a system is in the inspection stage of a maintenance process. The key attribute of this approach is the human inspector remains in the loop. The inspec- tor functions to ensure data quality, data fidelity, and can review any ADA outputs to make the final determination regarding an indication. Representative DAF Successes The following represents several examples devel- oped by AFRL and transitioned to the DAF. The ADA capabilities are presented as a function of increasing complexity from the perspective of combining the three technical approaches outlined in the previous section. However, this order should not be considered a listing of increasing complex- ity as each application had its unique degrees of complexity and used different approaches to tailor to the need and to the desired outcome of the inspection. A representative application that emphasizes the use of heuristics occurs in the manufactur- ing of aerospace composite structures, especially primary load carrying structures such as wing and fuselage skins. These parts require 100% ultrasonic inspection to detect delaminations and porosity where common rejection criteria are for delamina- tions greater than 6.35 mm (0.25 in.) in diameter or porosity that exceeds 2%. When considering the large areas to be inspected at manufacturing (note: this is not a requirement once a system is fielded), a bottleneck in the production flow can occur with the large volume of data to be assessed by inspec- tors. To minimize this bottleneck, a heuristic-based algorithm was developed to closely mimic the steps taken by an inspector to review data collected from these inspections (Aldrin et al. 2016). The ADA algorithm leverages the available A-scan and B-scan data that accompanies the C-scan data. Multiple steps are taken in each of the three data representations to determine if an indication has features associated with delami- nations that exceed the reject criteria. The repre- sentative result is shown in Figure 4 where C-scan features are identified as suspected defects and others are identified as benign. Though both may appear similar in the C-scan, attributes of the front wall, back wall, and volumetric gating can be used to distinguish between acceptable and rejectable features. The rejectable features are highlighted to the trained inspector who makes the final determi- nation regarding the indication. With this approach, inspection processes have been greatly acceler- ated, though exact metrics are not available for publication. Another representative case study includes the use of both simulation and heuristics to identify defects and discriminate between types of defects. The specific application is for rotating turbine engine components evaluated by an automated inspection system that can provide highly regis- tered data. Using a combination of model-based assessments and heuristic analysis methods, the response from data with varying probe conditions can be evaluated and provide guidance on what features are from suspected indications and what are due to the probe variability (Aldrin et al. 2019b). A representative illustration of this approach is the experimental response from a subsurface nonme- tallic inclusion in the presence of probe variation Gap Thickness transition Pad up Scan edge Part edge Lap Wrinkle Embedded defects Figure 4. Ultrasonic C-scan of a composite test article indicating regions identified by the assisted data analysis algorithms as potential defects. J U L Y 2 0 2 3 • M A T E R I A L S E V A L U A T I O N 39 2307 ME July dup.indd 39 6/19/23 3:41 PM