flaw that could lead to an increased risk of a cata- strophic outcome. It is important to recall that it is not the smallest flaw that can be detected, but the largest flaw that could be missed that impacts the safety of a system. This is especially true in aviation where single-load path structures are expected to have an extraordinarily low risk of failure when risk is managed by damage tolerance (US DOD 2016). This data sensitivity study demonstrates two critical issues that need to be considered when applying AI/ML algorithms to NDT data. The first is the number of data points required to achieve improved performance of AI/ML methods. Large training sets of actual flaws are hard to generate due to the time and cost of preparing such samples. A common complaint of POD studies that follow the guidance of MIL-HDBK-1823A is the high cost to prepare samples with characterized flaws. The minimum number of flaws for a versus a-hat (i.e., flaw size versus magnitude of the signal response from the measurement system) assessments is 40 and for hit-miss assessments is 60. Large datasets of flaw responses in NDT data are difficult to find from service since the engineering response to the detection of a growing number of flaws is either to modify or replace the structural element of concern before a large population of flaws is present. An option that has been pursued includes the use of simulation to generate the required datasets for training. However, the challenge is to create simu- lations that are representative of the flaws found in actual structures. This approach would require a validation process with a good amount of empiri- cal data covering the wide range of test conditions expected from an engineering perspective. The second issue is the ability to address outliers and nuances in data that can be indicators of flaws. The concern is the tendency of statistical methods to ignore such features when using large datasets. Unless the attributes of the outlier and nuance change in data are included in sufficient large quan- tities in training, the approach would tend to dismiss such features in the data, which could result in missed flaws. Conversely, if the AI/ML is sensitive to outliers, then the concern becomes that a large number of false calls could decrease the value of implementing the AI/ML algorithm. Thus, the lessons learned from the analysis of representative data includes the need to have the right data for training, including multiple flaws that are independent from each other. It is extremely important to recall that resampling the same data is not acceptable unless proper statistical methods to address correlated data are included in the analysis. Similarly, it is not acceptable to test AI/ ML methods using the same data that was used for training. Another aspect is to ensure factors that can affect the statistical analysis of data (such as SNR) are included in the training datasets. In addition, if simulation data is used in training, it must be from validated models that capture all the anticipated variances found in the NDT data for the inspec- tion. Lastly, the desired precision and accuracy of the diagnostics to be performed by AI/ML must be defined to ensure the amount of available data is sufficient to meet these objectives. This last consid- eration is especially true if unsupervised methods are being considered. Challenges for AI/ML in NDT As indicated by the sensitivity studies in the previous section, a significant challenge for the use of AI/ML in NDT data is to capture the effect of all the factors that can influence the capability to detect the flaws of interest. Figure 3 is a repre- sentation of these factors that the author has used extensively to illustrate the additional challenges when migrating from a laboratory to an operational environment. The three general classes of chal- lenges can be summarized as equipment variability, structural complexity and variability, plus flaw com- plexity and variability. In addition, these parame- ters can change as a function of the life of a system, which increases the capability validation difficulty of the NDT system when integrated into system life management. Equipment variability is the easiest of the three sources of variability to address from a research and development perspective. The variability in equip- ment settings can be defined and managed, but the unknown that frequently needs to be quantified is sensor variability and its impact on the diagnostics of flaws. Common NDT procedures address this with calibration processes, which alleviate many of these Find damage here ≠ Sensors Notch Plate L Rear spar Figure 3. Representative increase in challenges when migrating from a laboratory environment (left) to an operational environment (right). J U L Y 2 0 2 3 • M A T E R I A L S E V A L U A T I O N 37 2307 ME July dup.indd 37 6/19/23 3:41 PM

concerns. However, small changes in sensor config- uration, such as coil tilt within eddy current sensors or slight depolarization of well-used ultrasonic trans- ducers, can influence the flaw detection response. Flaw-to-flaw variability can have a much greater impact on the NDT response. Previous studies have illustrated that the same size flaw can vary in ampli- tude response from an eddy current inspection by over 20% of a full screen height reading (Forsyth et al. 2015). Similar results can occur in ultrasonic testing as well as other NDT techniques. For ultrasound, fatigue crack morphology and tortuosity can affect a response. Local stress considerations from a fit-up of assemblies and changes due to use can vary crack closure, which, in turn, affects the magnitude of the ultrasonic signal. The variability can be addressed in simulation provided all the attributes of the flaw that affect detection are included in the simulation studies. This includes their interaction, which can become a very large study, especially when consider- ing engineering level validation of the simulation. While flaw-to-flaw variability can be broadly categorized as a function of the type of flaw, struc- tural variability can become much more challeng- ing in the analysis of NDT data. This is largely due to the extensive range of structures evaluated by NDT, which includes power generation, infrastruc- ture, and transportation, the latter which can be segmented into ground, aviation, and space catego- ries. In addition, other considerations include the materials being used, including metals, polymers, ceramics, and composites the manufacturing process being used, for example, automation, partial automation, or hand assembly plus, the assembly process used to join components, such as welding, fastening, and bonding. With all these parameters, it becomes very clear why NDT is the ultimate multi- disciplinary engineering domain! A significant challenge is how to evaluate the effect all these parameters, both individually and through important interactions, have on the NDT response. Consider the simple fastened joint between two metal surfaces, where up to 22 factors addressing equipment, flaws, and structure need to be included in a sensitivity study (Lindgren et al. 2007). Structural considerations include such things as composition of each layer the possibility of shims and their compo- sition assembly quality, such as fastener hole tilt or skewed fasteners and fit-up stresses as a function of what type of fastener is used and how it is installed. In addition, how these factors change as a function of time due to maintenance, repairs, modifications, and even use need to be included. Using AI/ML techniques for these applications can become very daunting when considering all the parameters that need to be addressed to make diagnostic decisions using automated processes. This includes how the statistical processes adjust to account for changes that occur as a function of time. In addition, how these affect the diagnostic capability of the NDT data must be validated to enable their use in system risk and life manage- ment. Therefore, the proper capturing of these factors in statistically representative methods presents itself as a significant challenge, but also a significant research and development opportunity. DAF Approach to AI/ML for NDT Data AFRL has been leading the development of algo- rithms to assist in the diagnostics of NDT data, including one of the first implementations for an aviation NDT application (Lindgren et al. 2005). Attributes that have made this approach successful include the use of multiple approaches to develop algorithms for the diagnostic capability combined with the approach that the algorithms will not replace all human interpretation of NDT data. The algorithms are used as a capability to facilitate and guide the interpretation to make the workload on an inspector easier and focused on the critical elements of the diagnostic process that do not easily lend themselves for automation. AFRL has called this approach intelligence augmentation (IA), but an alternative term being used in the sci- entific community is collaborative intelligence (CI) (Epstein 2015). This reflects how software tools and capabilities can be used to assist in the analysis of NDT data, which AFRL has named assisted data analysis (ADA). ADA algorithms combine multiple approaches to provide an optimized method to facilitate NDT diagnostics. These algorithms can be grouped into three general categories. The first uses heuristic-based methods that incorporate “rules of the road” that closely mimic the procedures by which inspectors interpret data. The second is a model-based inversion algorithm that uses simu- lation to represent the measurement response and iteratively solve for the unknown flaw or material state in the presence of variability. The third uses AI/ML methods trained using NDT data and as much diagnostics information as possible from available datasets. Frequently, the amount of well understood NDT data is much smaller than what would be required for robust AI/ML analysis, and likely requires supplementation from simulated data or transfer learning. Successful application of ADA has frequently included at least two of these approaches into an integrated diagnostic algorithm for the specific FEATURE |AI/ML 38 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 3 2307 ME July dup.indd 38 6/19/23 3:41 PM