in the M-scan (Figure 8d). It was not expected, however, that they would be so rapidly detected with 90% of detections occurring within 4 ms of ground truth. Because they appear so clearly in M-scans, they are also the easiest event to label during dataset preparation. On the other hand, saturation is by far the most difficult event to label as the saturation point was defined as “the moment at which the molten nugget appeared to stop growing vertically,” which is highly subjective without perfect nugget and stack boundary annotations. Similarly, but less so, melting and SSID are not always as apparent as expulsions. Thus, from our experience in reading these images, and considering the relative difficulty for a human to interpret ultrasonic M-scans and identify these events and the relative consistency of event annotations, we found that the relative detection rates of the four events completely align with expectations. Relatedly, as the ground truth labels for event timing as well as the top and bottom labels for the nugget and stack were used to develop the curves for MNS, the subjectivity and consistency of labels affects the performance of the models on the regression task as well. In particular, stack boundaries are almost always reasonably visible aside from after expulsions, while nugget boundaries vary in visibility based on nugget pool size, stage of weld, and stack geometry. With the investigated ME |AI/ML Non-weld Nominal Expulsion Insufficient Weld time (ms) 1 0 0 200 0 285 0 300 0 220 1 0 Figure 8. M-scan samples with ground truth markup and model outputs superimposed. Ground truth event timestamps are shown as dark thick vertical lines reaching the halfway mark vertically in images ground truth MNS is darker blue curve. Model outputs (thin curves – unprocessed model outputs thin vertical lines – event probability outputs thresholded at 0.5) are from most performant model. For event colors, green – melting yellow – SSID red – saturation purple – expulsion. Blue indicates model output for MNS. All model outputs and MNS targets are superimposed on images with 0 =bottom of image, 1 =top of image. Images cover various stackups and weld outcomes. Timing error (ms) 0.12 0.1 0.08 0.06 0.04 0.02 0 –30 –27 –24 –21–18 –15 –12–9 –6 –3 0 3 6 9 12 15 18 21 24 27 30 Timing error (ms) 0.12 0.1 0.08 0.06 0.04 0.02 0 –30 –27 –24 –21–18 –15 –12–9 –6 –3 0 3 6 9 12 15 18 21 24 27 30 Timing error (ms) 0.12 0.1 0.08 0.06 0.04 0.02 0 –30 –27 –24 –21–18 –15 –12–9 –6 –3 0 3 6 9 12 15 18 21 24 27 30 Timing error (ms) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 –30 –27 –24 –21–18 –15 –12–9 –6 –3 0 3 6 9 12 15 18 21 24 27 30 Figure 7. Timing error distributions for detected (a) melting (b) SSID (c) saturation and (d) expulsion for the model with the best overall performance. 68 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 3 2307 ME July dup.indd 68 6/19/23 3:41 PM AI output level Proportion of events Proportion of events Proportion of events Proportion of events

approach, continuous feedback could be provided to an adaptive weld controller using the regression output of MNS, or other discrete events could be derived from it. In either case, optimal use of MNS output in a production implementation would likely require calibration welds for novel weld type (given, for example, sheet thicknesses, sheet materials, etc.) to deter- mine an appropriate threshold or target value for MNS output. In future work, this ultrasound-based approach with AI-driven feedback will be rigorously compared against alternative feedback approaches (e.g., resistance-based feedback, etc.). With respect to this AI-based approach, a more rigorous optimization of hyperparameters within the feasible space of architectures may yield better performance. In this study, the largest model possible (in terms of parameter count) was used based on feasibility study results however, it is possible that better performance may be achieved by using various popular modules in the network (if feasible), for example, skip connections (He et al. 2016 Ronneberger et al. 2015), atrous spatial pyramid pooling (Chen et al. 2018), batch normaliza- tion (Ioffe and Szegedy 2015), attention mechanisms such as convolution block attention module (Woo et al. 2018), and so forth. Alternatively, other novel architectures, such as vision transformer (Liu et al. 2021), could be explored in future work. In addition, providing known welding parameters to the model as inputs (such as sheet thicknesses, sheet material encodings, force, welding cap face diameter, etc.) is another potential opportunity for improvement, which can be investigated in the future. Precise and continuous annotations, for both the nugget and stack, at all times throughout the weld would be ideal in order to derive event timestamps and MNS curves. From the standpoint of dataset development, this would essentially be the same as labeling the M-scans for semantic segmentation of the nugget and stack boundaries, which is significantly more tedious and laborious than the proposed approach, and still subjective (though, perhaps less subjective as it is less abstract). One advantage of the proposed approach is that it required, at most, eight clicks per annotated M-scan (each of the four event timings, two nugget labels, two stack labels) during data annotation, whereas semantic segmentation would conservatively require 20 clicks per segmented region to delineate each polygon—40 clicks in total between nugget and stack regions—so the proposed approach yielded a five- fold reduction in data preparation time. That said, semantic segmentation of the ultrasonic data is still a natural next step for this work like in the case of Guo et al. (2023). Other works have demonstrated the potential for semantic segmentation in real-time ultrasonic inspection in both NDE and medical contexts (Fiorito et al. 2018 Hu et al. 2022 Shandiz and Tóth 2022). If it were found to be performant, generalizable, and still sufficiently fast for adaptive RSW (i.e., 1 ms per A-scan infer- ence time in a production environment), semantic segmenta- tion could yield more precise and continuous measurements, and consequently better feedback. This would be especially valuable if continuous feedback to an adaptive weld controller was preferred over discrete feedback, or perhaps necessitated for a particular adaptive welding algorithm. Conclusion The investigated approach is not limited to ultrasonic NDE nor resistance spot welding such an approach could be applied to the interpretation of NDE data from a variety of other modalities for a variety of other joining methodologies. In all, the investigated approach is an exciting first step toward real-time interpretation of ultrasonic NDE data from RSW. It demonstrates the enormous potential of ultrasound-based process monitoring backed by real-time interpretation using deep learning, for real-time adaptive feedback systems in modern manufacturing. Such NDE 4.0 systems are integral to Industry 4.0 and the ZDM paradigm, and this work brings zero-defect RSW closer to reality. ACKNOWLEDGMENTS This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Collaborative Research and Development (CRD) grant CRDPJ 508935-17. It was also supported by the National Research Council Canada (NRC) Industrial Research Assistance Program (IRAP). The authors would also like to thank the Institute for Diagnostic Imaging Research at the University of Windsor, Canada, as well as Tessonics Inc. in Windsor, Canada, for providing experimental facilities, equipment, and materials for this research. REFERENCES Abadi, M., A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, et al. 2015. “TensorFlow: Large-scale machine learning on heterogeneous systems.” https://www.tensorflow.org. Shafiei Alavijeh, M., R. Scott, F. Seviaryn, and R. Gr. Maev. 2020. “NDE 4.0 compatible ultrasound inspection of butt-fused joints of medium-density polyethylene gas pipes, using chord-type transducers supported by customized deep learning models.” Research in Nondestruc- tive Evaluation 31(5–6). https://doi.org/10.1080/09349847.2020.1841864. Shafiei Alavijeh, M., R. Scott, F. Seviaryn, and R. Gr. Maev. 2021. “Using machine learning to automate ultrasound-based classifica- tion of butt-fused joints in medium-density polyethylene gas pipes.” Journal of the Acoustical Society of America 150 (1): 561–72. https://doi. org/10.1121/10.0005656. Cantero-Chinchilla, S., P. D. Wilcox, and A. J. Croxford. 2022. “Deep learning in automated ultrasonic NDE – Developments, axioms and opportunities.” NDT &E International 131. https://doi.org/10.1016/j. ndteint.2022.102703. Chen, L.-C., Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. 2018. “Encoder-decoder with atrous separable convolution for semantic image segmentation.” In Computer Vision – ECCV 2018: 15th European Confer- ence: 833–851. https://doi.org/10.1007/978-3-030-01234-2_49. Chertov, A., and R. Gr. Maev. 2004. “Determination of resistance spot weld quality in real time using reflected acoustic waves. Comparison with through-transmission mode.” 16th World Conference on NDT. Montreal, Canada. Chollet, F., et al. 2015. Keras. Software available at keras.io. Denisov, A. A., C. M. Shakarji, B. B. Lawford, R. G. Maev, and J. M. Paille. 2004. “Spot weld analysis with 2D ultrasonic Arrays.” Journal of Research of the National Institute of Standards and Technology 109 (2): 233–44. https:// doi.org/10.6028/jres.109.015. Dugmore, A. 2021. “New composites target EV applications.” SAE International. https://www.sae.org/news/2021/08/new-composites -target-ev-applications. El-Banna, M. 2006. “Dynamic resistance based intelligent resistance welding.” Doctoral dissertation. Wayne State University. J U L Y 2 0 2 3 • M A T E R I A L S E V A L U A T I O N 69 2307 ME July dup.indd 69 6/19/23 3:41 PM