Aldrin, J. C., J. D. Achenbach, G. Andrew, C. P’an, R. Grills, R. T. Mullis, F. W. Spencer, and M. Golis. 2001. “Case Study for the Implementation of an Automated Ultrasonic Technique to Detect Fatigue Cracks in Aircraft Weep Holes.” Materials Evaluation 59 (11): 1313–19. Epstein, S. L. 2015. “Wanted: Collaborative intelligence.” Artificial Intelligence 221:36–45. https://doi.org/10.1016/j. artint.2014.12.006. Forsyth, D.S., J. Ocampo, H. Millwater, and J. Montez. 2015. “Structural Health Monitoring, Risk, and Reliability.” Aircraft Structural Integrity Program Conference. available at: https:// www.researchgate.net/publication/344886205_Structural_ Health_Monitoring_Risk_and_Reliability. Lindgren, E. A., J. R. Mandeville, M. J. Concordia, T. J. MacInnis, J. J. Abel, J. C. Aldrin, F. Spencer, D. B. Fritz, P. Christiansen, R. T. Mullis, and R. Waldbusser. 2005. “Proba- bility of Detection Results and Deployment of the Inspection of the Vertical Leg of the C-130 Center Wing Beam/Spar Cap.” 8th Joint DoD/FAA/NASA Conference on Aging Aircraft. Lindgren, E. A., J. S. Knopp, J. C. Aldrin, G. J. Steffes, and C. F. Buynak. 2007. “Aging Aircraft NDE: Capabilities, Challenges, and Opportunities.” AIP Conference Proceedings 894:1731–38. https://doi.org/10.1063/1.2718173. Lindgren, E.A. 2022. “Intelligence Augmentation for Aviation-based NDE Data.” 2022 Aircraft Structural Integrity Program Conference, available at: http://www.arctos meetings.com/agenda/asip/2022/proceedings/presentations/ P23251.pdf and https://doi.org/10.32548/RS.2022.005. Nagy, P. B., M. Blodgett, and M. Golis. 1994. “Weep hole inspection by circumferential creeping waves.” NDT &E International 27 (3): 131–42. https://doi.org/10.1016/0963- 8695(94)90604-1. US DOD. 2008. MIL-STD-3024: Department of Defense Standard Practice, Propulsion System Integrity Program (PSIP). US Department of Defense. US DOD. 2009. MIL-HDBK-1823A: Department of Defense Handbook, Nondestructive Evaluation System Reliability Assessment. US Department of Defense. US DOD. 2016. MIL-STD-1530D: Department of Defense Standard Practice, Aircraft Structural Integrity Program (ASIP). US Department of Defense. USAF. n.d. “BLUE: THE AI Advantage.” US Air Force. https:// www.af.mil/News/Photos/igphoto/2002319445/. FEATURE |AI/ML 42 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 3 2307 ME July dup.indd 42 6/19/23 3:41 PM

TIPS FOR EFFECTIVE MACHINE LEARNING IN NDT/E BY JOEL B. HARLEY, SUHAIB ZAFAR, AND CHARLIE TRAN The proliferation of machine learning (ML) advances will have long- lasting effects on the nondestructive testing/evaluation (NDT/E) community. As these advances impact the field and as new datasets are created to support these methods, it is important for researchers and practitioners to understand the associated challenges. This article provides basic definitions from the ML literature and tips for nondestructive researchers and practitioners to choose an ML architecture and to understand its relationships with the associated data. By the conclusion of this article, the reader will be able to identify the type of ML architecture needed for a given problem, be aware of how characteristics of the data affect the architecture’s training, and understand how to evaluate the ML performance based on properties of the dataset. Introduction Advances in ML have consistently gen- erated headlines in the past few years. These developments can be attributed to sophisticated algorithms, faster hardware, and reduced costs for data storage. The natural consequence of such advancements is the deluge of datasets, often known as the age of big data. ML algorithms, especially deep learning, capitalize on these foundations, finding applications in speech recogni- tion and object detection while opening up new possibilities through innovations such as ChatGPT (OpenAI 2023). These applications vary considerably from one another, yet the main task in each case is to recognize patterns in datasets. Pattern recognition is arguably the primary driving force behind new sci- entific and engineering discoveries. For instance, Kepler utilized the observa- tions of Tycho Brahe in astronomy to derive the laws governing planetary motion, which formed the basis for classical mechanics (Bishop 2006). However, data was not a driving force behind scientific inquiry until recently (Brunton et al. 2020), and these trends have also impacted NDT/E (Taheri et al. 2022), with recent advances such as crack detection in concrete using neural networks (Saleem and Gutierrez 2021) or identifying damage modes in compos- ite structures via clustering algorithms (Xu et al. 2020). Neural networks are one of the most widely used algorithms today and can be understood as a class of mathematical models inspired by the structure of the human brain. However, utilizing neural networks, or ML in general, for tasks such as defect detection or aiding data interpretation is a familiar trend in NDT/E. Martín et al. (2007) published a study in 2007 to interpret ultrasonic oscillograms obtained via the pulse-echo method with the aid of neural networks. Even earlier, in the 1990s, Mann et al. (1992) presented the use of neural networks to classify ultrasonic signals obtained from microfiber cracking in a specimen built using a metal matrix composite. These examples demonstrate that the NDT/E community has long recognized the need to augment human judgment with pattern recognition algorithms. Despite these advances, limita- tions of ML in NDT/E have mitigated its impact on the field when compared with other disciplines. A widely acknowl- edged problem is the limited amounts of data available, which is the driving force behind the success of ML in many applications. Even if the lack of training data is not an issue with data-intensive applications, such as acoustic emission testing (Sikorska and Mba 2008), acquir- ing data with a high signal-to-noise ratio (SNR) is a significant hurdle. Finally, an adequate level of understanding and experience in ML techniques is required to ensure the accurate performance of algorithms, which currently needs improvement (Vejdannik et al. 2019). In this article, we address important challenges in applying ML to NDT/E by providing guidelines for practitioners and researchers on building high-quality datasets and using appropriate algo- rithms to ensure high performance from trained ML models. The desired outcome of this effort is to encourage progress in realizing the full poten- tial of ML in NDT/E, leading to more accurate and efficient testing methods in the future. Note that the focus of this article is on how to assess datasets and results. Detailed descriptions of the ML algorithms can be found in other papers (Taheri and Zafar 2023). Forms of Machine Learning ML can be divided into various learning paradigms, each with its characteristics and uses. Below are descriptions for two of these paradigms: supervised learning and unsupervised learning. Examples of supervised learning and unsupervised learning are illustrated in Figure 1. Supervised learning: An ML paradigm that trains the parameters (often numerical weights) of a model from input data (features) and known output data (labels). Supervised learning is the most popular ML paradigm due to the ease at which model training can be directly translated to the target task. The key element of supervised learning is the availability of labeled data. Yet in AI/ML |NDT TUTORIAL J U L Y 2 0 2 3 • M A T E R I A L S E V A L U A T I O N 43 2307 ME July dup.indd 43 6/19/23 3:41 PM