ABSTR ACT Adaptive resistance spot welding systems typically rely on real-time analysis of dynamic resistance curves and other indirect measurements to estimate weld progress and guide adaptive weld control algorithms. Though efficient, these approaches are not always reliable, and consequently there is a need for improved feedback systems to drive adaptive welding algorithms. As an alternative, an advanced in-line integrated ultrasonic monitoring system is proposed, with real-time weld process characterization driven by artificial intelligence (AI) to create actionable feedback for the weld controller. Such a system would require real-time ultrasonic data interpretation, and for this a solution using deep learning was investigated. The proposed solution monitors the ultrasonic data for key process events and estimates the vertical size of the weld nugget proportional to the stack size throughout the welding process. This study shows that adaptive welding using ultrasonic process monitoring backed by AI-based data interpretation has immense potential. This research highlights the importance of nondestructive evaluation (NDE) in the zero-defect manufacturing paradigm. KEYWORDS: resistance spot welding, ultrasound, artificial intelligence, deep learning, NDE 4.0 Introduction Zero-defect manufacturing (ZDM) has been a dream for decades (Psarommatis et al. 2022, 2023). With respect to many manufacturing processes, this dream is considered within reach given the novel technologies that should be ubiqui- tous in an idealized Industry 4.0. Unfortunately, however, Industry 4.0 is not yet fully realized and thus the realization of ZDM suffers as well (Psarommatis et al. 2022). Though some requirements of Industry 4.0 are increasingly fulfilled (e.g., big data production, storage, and analytics increased con- nectivity and Internet of Things industrial automation), its full realization requires NDE 4.0 (Meyendorf et al. 2017). For example, NDE 4.0 is a prerequisite for Industry 4.0’s widely unfulfilled key requirement of decentralized and autono- mous decision-making (Escobar et al. 2021). Fulfillment of these requirements with respect to manufactured products and joining processes is promised by NDE 4.0 through (a) the automation of nondestructive inspections (b) the automated, consistent, generalized, and accurate interpretation of inspec- tion data and (c) the resultant characterization of manufac- tured products, which would be used to inform downstream decision-making without human intervention. Resistance spot welding (RSW) is a manufacturing process for which the ZDM dream is potentially within reach. Many industries heavily rely on RSW joints including automo- tive, aerospace, rail, and military. RSW is a favorable joining method in many cases because it is inexpensive to perform, has a fast cycle time, maintains integrity of the joined sheets, has minimal added weight and volume, is highly adaptable, is robust, and is generally amenable to nondestructive evalu- ation (NDE) (El-Banna 2006). However, across all industries, novel materials are increasingly being developed and incor- porated into manufactured products (Perez-Regalado et al. 2013). For example, in the automotive industry—which uses RSW approximately 5000 to 7000 times per vehicle—increas- ing vehicle electrification imposes new engineering challenges with respect to safety, lightweighting, and weight distribu- tion (Dugmore 2021). Consequently, there is an increasing use of novel lightweight and high-strength materials (e.g., advanced high-strength steels and aluminum alloys), as well as dissimilar-material joints, which pose new challenges for RSW (Dugmore 2021). Thus, there is an increasing demand for solu- tions that enable ZDM of RSW. REAL-TIME AI-DRIVEN INTERPRETATION OF ULTRASONIC DATA FROM RESISTANCE SPOT WELD PROCESS MONITORING FOR ADAPTIVE WELDING RYAN SCOTT*†‡, DANILO STOCCO*†, ANDRIY CHERTOV*†, AND ROMAN GR. MAEV*† *The Institute for Diagnostic Imaging Research, University of Windsor, Canada † Tessonics Inc., Windsor, Canada ‡ Corresponding author: rscott@uwindsor.ca Materials Evaluation 81 (7): 61–70 https://doi.org/10.32548/2023.me-04344 ©2023 American Society for Nondestructive Testing NDTTECHPAPER |ME J U L Y 2 0 2 3 • M A T E R I A L S E V A L U A T I O N 61 2307 ME July dup.indd 61 6/19/23 3:41 PM

There have been several attempts to support the reali- zation of ZDM in RSW through the use of adaptive welding systems. Conceptually, modern adaptive welding systems monitor one or more indirect proxies of weld progress (e.g., dynamic resistance curves, current, voltage, force, tip dis- placement [El-Banna 2006 Neugebauer et al. 2013 Reis et al. 2016]), process these monitored features in real time to create feedback, and serve the feedback to an algorithm that adapts weld process parameters (e.g., weld time, force, and current) accordingly. In practice, these proxies do not produce suffi- ciently reliable and consistent feedback for adaptive weld con- trollers, so these systems generally fail to meet expectations and consequently many users revert to fixed schedules with adaptive capabilities disabled. RSW is well-positioned to simultaneously meet the require- ments of NDE 4.0 and achieve a breakthrough in ZDM, largely due to recent advancements in RSW NDE research. RSW NDE can be conducted either in-process (during the weld) or post-process (after the weld) using a variety of NDE modalities (Runnemalm and Appelgren 2012 Summerville et al. 2019). One of the most prevalent modalities is ultrasound (Chertov and Maev 2004 Denisov et al. 2004 Ouellette et al. 2013 Maev et al. 2014, 2016 Sung Hoon et al. 2020). Ultrasonic inspection has important advantages in inspection speed, insensitivity to sample thickness, adaptability, and the ability to directly inspect the internal geometric properties of the joint. The current state of the art in ultrasonic NDE for RSW consists of post-process offline inspection via portable ultrasonic systems with 2D matrix probes (e.g., Denisov et al. 2004 Maev et al. 2005), post-process robotized in-line systems with a similar ultrasonic configuration, and in-line real-time process moni- toring systems using single-element probes (e.g., Chertov and Maev 2004 Ouellette et al. 2013 Maev et al. 2013, 2014 Sung Hoon et al. 2020). In any case, many NDE 4.0 requirements are already being met for such inspection systems, but only the in-line approach can provide real-time process monitoring and NDE data with 100% joint coverage, which is actionable in the context of an adaptive welding system that facilitates ZDM. In its current form, the in-line inspection approach involves embedding a single-element ultrasonic transducer into a welding electrode (Chertov and Maev 2004 Ouellette et al. 2013 Maev et al. 2014, 2016 Sung Hoon et al. 2020). The transducer is immersed in flowing water, which both cools the transducer and provides coupling. The copper electrode caps focus the ultrasonic waves into the heat-affected zone of the workpiece and provide coupling against the stackup due to the application of intense force during welding (Maev and Chertov 2010). Throughout the welding process, A-scans are sampled every millisecond in pulse-echo mode, aiming through the center of the weld region between the electrodes. An M-scan—a 2D ultrasonic signature of the weld process—is then formed by horizontally stacking A-scans, and currently only post-process interpretation of the ultrasonic signature is con- ducted for quality control (Maev et al. 2021). Therefore, toward adaptive welding, a major missing piece in existing in-line ultrasonic systems is real-time interpretation of the sequence of A-scan signals as they are collected. Classically, ultrasonic NDE data interpretation may involve signal/image processing, statistical analyses, search algorithms, model fitting, and hand-coded rules for decision-making. In some cases, these classical approaches are sufficient. However, in many application domains, such as RSW inspection—due to the many potential geometries, material combinations, and weld parameterizations, which can be encountered in a production environment—these approaches fail to meet the required performance, inference speed, and generality. Recently, deep learning approaches have been increasingly applied, to great effect, to a variety of problems in ultrasonic NDE data interpretation spanning essentially all use cases and specific tasks (e.g., defect detection and characterization, measurement automation, and so on [Cantero-Chinchilla et al. 2022 Taheri et al. 2022]). For example, Virkkunen et al. (2021) used a convolutional neural network (CNN) for crack detec- tion in ultrasonic inspection data from butt-fused stainless steel pipes. Similarly, Shafiei Alavijeh et al. (2020) developed an ultrasonic inspection approach using a chord transducer for butt-fused plastic pipe joints. In this case, they used an autoen- coder to conduct outlier detection on A-scans. Subsequently, the group developed an approach that classified A-scans in terms of defect presence/absence and according to defect type when a defect is detected (Shafiei Alavijeh et al. 2021). They compared several classical machine learning algorithms to four deep neural network architectures and determined that a CNN generally achieved the best performance on this task. Guo et al. (2019) combined CNN with recurrent neural networks (gated recurrent unit [GRU] and long short-term memory [LSTM]) to achieve high-performance debonding defect detection in ultrasonic C-scans of braided composite materials. They subsequently refined the approach in later works by instead framing the problem as semantic segmen- tation (Guo et al. 2023). Huang et al. (2022) also combined CNN and LSTM to detect defects in copper pipes in data from laser ultrasonic scanning. Maev et al. (2021) used an object detection approach with YOLOv3 (the “you-only-look-once” v3 object detector) to conduct post-process characterization of ultrasonic weld process signatures by identifying expulsions (discharge of molten material from the stackup due to intense pressure and rapid heating), while also identifying discrete weld-process events and measuring the position of the nugget at its maximum vertical size within the welded stackup. A more recent study by Zamiela et al. (2023) combined infrared with ultrasonic imaging data and developed a two-branch U-Net, which conducts semantic segmentation on the aligned images simultaneously to identify and characterize pores in metal structures in a single, unified output map. Deep learning has been proven to outperform classical computational NDE data interpretation approaches in terms of performance, inference speed, and generality thus, it is a promising potential solution for time-sensitive contexts such as real-time inspection and adaptive welding. ME |AI/ML 62 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 3 2307 ME July dup.indd 62 6/19/23 3:41 PM