from both harvested and mock-up test articles were used to refine the decision-making process for the ADA algorithms. The inspection process was fully validated by a comprehensive POD study before being deployed. The inspections were accomplished by contractor field teams that would collect the data and ensure it had sufficient quality to be evaluated by the ADA. Suspected indications identified by the ADA were sent to an NDT engineer to make a final determina- tion if the indication was confirmed and needed to be sent to engineering for disposition. The next generation of ADA will expand the capability of algorithms to facilitate the identification of defects to the capability to characterize the defects in ways that are not available today. While inspec- tors can use methods to approximate defect size, attributes like fatigue crack depth are especially chal- lenging. However, using a combination of heuristics, simulations, and data-driven analytical methods, the use of ADA to determine the depth of a fatigue crack from a bolt-hole eddy current inspection was shown to have an average accuracy of 8.5% for fastener holes with minimal variability (Aldrin et al. 2019a). The next steps in the development process are to use this inte- grated approach to address fastener hole variability, such as skew and out-of-round attributes, to provide a crack depth estimated with a statistical bounds on accuracy, to enable rapid disposition of these defects in aerospace structures. Summary There is a continued potential for AI/ML methods to enhance data analysis and diagnostics for NDT data. However, there needs to be a realis- tic approach that includes evaluation of the data quantity, quality, and fidelity. This ensures it has the desired attributes that enable the AI/ML tech- niques to provide outcomes with sufficient statis- tical metrics for the results to be used in engineer- ing decisions. In addition, these outcomes require rigorous validation of the diagnostic capability before they can be trusted to help ensure the integ- rity, or safety, of systems. A representative example illustrated the chal- lenges in using AI/ML techniques for smaller and noisy datasets, highlighting how this can lead to outliers that would imply potentially missed defects if this approach was used for NDT datasets. Additional challenges exist in data variability from equipment, defects, and structure that impact the amount of quality data required for AI/ML approaches. While data for defects can be aug- mented by simulations, these must contain all the anticipated variability and complexity of the NDT evaluation technique to represent nuances and outliers that are challenging for AI/ML, but critical for high-accuracy flaw detection. The challenges of AI/ML when used for NDT data has led AFRL to pursue a hybrid approach that integrates AI/ML with heuristic- and model-based diagnostic algorithms to facilitate and reduce the workload of inspectors while not taking them com- pletely out of the loop. Representative examples for several DAF-related applications have demon- strated the power of combining at least two of these methods to enable complex inspections and diagnostics of NDT data. The ADA algorithms are combined with human analysis to maximize the value of the algorithms by reducing the workload of inspectors so they can focus on the critical data that could be indications of defects being present. Future work includes plans to expand the capabili- ties of ADA algorithms to characterize defects with statistical metrics of accuracy. Initial development efforts have shown the potential of this capability, which would decrease the disposition time of indi- cations and increase availability of the system to the end user. ACKNOWLEDGMENTS The author expresses his deep appreciation for the pioneering contributions and collaboration with Dr. John Aldrin of Computational Tools. Mr. David Forsyth of Texas Research Institute – Austin is recognized for his contribution to implementing ADA algorithms for multiple NDT applica- tions. The AI/ML analysis here would not be possible without the work performed by Mr. Tushar Gautam and Drs. Kirby, Hochhalter, and Zhe of the University of Utah. AUTHOR Eric Lindgren: AFRL and ASNT Fellow eric.lindgren @us.af.mil CITATION Materials Evaluation 81 (7): 35–42 https://doi.org/10.32548/2023.me-04364 ©2023 American Society for Nondestructive Testing REFERENCES: Aldrin, J. C., D. S. Forsyth, and J. T. Welter. 2016. “Design and Demonstration of Automated Defect Analysis Algorithms for Ultrasonic Inspection of Complex Composite Panels with Bonds.” AIP Conference Proceedings 1706. https://doi. org/10.1063/1.4940591. Aldrin, J. C., E. A. Lindgren, and D. S. Forsyth. 2019a. “Intelligence augmentation in nondestructive evaluation.” AIP Conference Proceedings 2102 (1): 020028. https://doi. org/10.1063/1.5099732. Aldrin, J. C., E. K. Oneida, E. B. Shell, V. Sinha, K. Keller, J. K. Na, A. L. Hutson, H. A. Sabbagh, E. Sabbagh, R. K. Murphy, S. Mazdiyasni, M. R. Cherry, and D. M. Sparkman. 2019b. “Model-based inversion of eddy current data for classification and sizing of planar and volumetric discontinuities.” Vol. 44: Electromagnetic Nondestructive Evaluation XXII 44:80–85. J U L Y 2 0 2 3 • M A T E R I A L S E V A L U A T I O N 41 2307 ME July dup.indd 41 6/19/23 3:41 PM

Aldrin, J. C., J. D. Achenbach, G. Andrew, C. P’an, R. Grills, R. T. Mullis, F. W. Spencer, and M. Golis. 2001. “Case Study for the Implementation of an Automated Ultrasonic Technique to Detect Fatigue Cracks in Aircraft Weep Holes.” Materials Evaluation 59 (11): 1313–19. Epstein, S. L. 2015. “Wanted: Collaborative intelligence.” Artificial Intelligence 221:36–45. https://doi.org/10.1016/j. artint.2014.12.006. Forsyth, D.S., J. Ocampo, H. Millwater, and J. Montez. 2015. “Structural Health Monitoring, Risk, and Reliability.” Aircraft Structural Integrity Program Conference. available at: https:// www.researchgate.net/publication/344886205_Structural_ Health_Monitoring_Risk_and_Reliability. Lindgren, E. A., J. R. Mandeville, M. J. Concordia, T. J. MacInnis, J. J. Abel, J. C. Aldrin, F. Spencer, D. B. Fritz, P. Christiansen, R. T. Mullis, and R. Waldbusser. 2005. “Proba- bility of Detection Results and Deployment of the Inspection of the Vertical Leg of the C-130 Center Wing Beam/Spar Cap.” 8th Joint DoD/FAA/NASA Conference on Aging Aircraft. Lindgren, E. A., J. S. Knopp, J. C. Aldrin, G. J. Steffes, and C. F. Buynak. 2007. “Aging Aircraft NDE: Capabilities, Challenges, and Opportunities.” AIP Conference Proceedings 894:1731–38. https://doi.org/10.1063/1.2718173. Lindgren, E.A. 2022. “Intelligence Augmentation for Aviation-based NDE Data.” 2022 Aircraft Structural Integrity Program Conference, available at: http://www.arctos meetings.com/agenda/asip/2022/proceedings/presentations/ P23251.pdf and https://doi.org/10.32548/RS.2022.005. Nagy, P. B., M. Blodgett, and M. Golis. 1994. “Weep hole inspection by circumferential creeping waves.” NDT &E International 27 (3): 131–42. https://doi.org/10.1016/0963- 8695(94)90604-1. US DOD. 2008. MIL-STD-3024: Department of Defense Standard Practice, Propulsion System Integrity Program (PSIP). US Department of Defense. US DOD. 2009. MIL-HDBK-1823A: Department of Defense Handbook, Nondestructive Evaluation System Reliability Assessment. US Department of Defense. US DOD. 2016. MIL-STD-1530D: Department of Defense Standard Practice, Aircraft Structural Integrity Program (ASIP). US Department of Defense. USAF. n.d. “BLUE: THE AI Advantage.” US Air Force. https:// www.af.mil/News/Photos/igphoto/2002319445/. FEATURE |AI/ML 42 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 3 2307 ME July dup.indd 42 6/19/23 3:41 PM