744 M A T E R I A L S E V A L U A T I O N • J U L Y 2 0 2 1 employed, which allows for surface and subsurface anomaly detection while meeting the optimization requirements. As shown in Figure 5, a compact NDE system with a 4 × 2 sensor array was fabricated. In order to validate the performance of the proposed sensor, a steel sample was fabricated with electrical discharge machined cracks of various depths. Three 0.2 mm width cracks with depths from 1 to 3 mm were introduced into sample 1. The cracks on sample 2 are of the same width but the depths are 4, 4.5, and 5 mm, respectively. Two steel plates were placed side by side and scanned using the designed eddy current coil array sensor with a well-maintained 1 mm liftoff distance. The line scan results for the two samples are shown in Figure 6. All cracks were successfully detected quantita- tively, where different shifts of the eddy current sensor’s resonating frequencies correlates to different depths of the narrow crack. Robotic Systems Integration Regarding mapping, position, and orientation estimation, sensory inputs are fed into the simultaneous localization and mapping (SLAM) subsection of the AI-enabled autonomy stack. There are four primary sensory inputs. These include an inertial measurement unit (IMU), wheel encoders, LiDAR sensor, and an RGB (red, green, blue) depth camera. Software is managed via a framework referred to as the robotic oper- ating system (ROS) (ROS 2021). ROS allows segments of code to be organized in nodes such that they are easily managed. Data from the robot’s onboard IMU and its wheel encoders are fused through odometry using a ROS robot localization package. These data are then passed to the Georgia Tech smoothing and mapping (GTSAM) backend SLAM factor graph, where they are fused with the robot’s LiDAR data and data from its RGB depth camera. A map is then constructed and maintained in the 3D mapping and information fusion ROS node. It is also necessary for this node to forward its data to the damage-tracking node to maintain an awareness of the locations of detected damage. The map is continuously updated and maintained in a loop as the robot navigates its environment. The robot’s NDE probe is in constant contact with the infrastructure wall’s surface, depicted as the material under test in Figure 1. The acquired probe signals are passed along the NDE space through the reasoning space until those signals reach the damage analysis subsection. Detection of damage on the boiler wall is handled by the damage analysis subsection. After NDE data are recon- structed as an image in the reasoning space, the damage recognition module classifies each image scan as containing or not containing any damage using a regularized optimization machine learning model (Schmidt et al. 2007). If damage is correctly classified, this leads to another unique challenge: the location of the damage must be accurately estimated and inte- grated into the robot’s global map. This is handled by the damage-tracking module using a unique method called semantic segmentation (Zhang et al. 2018). This method provides a means of classifying an object within an image in terms of its pixels. Given where in the image frame the damage is classified allows for calculation of where it exists within the robot’s global map using a perspective projection calculation. Once damage has been detected and fused with the robot’s global map, the data is passed to the damage-tracking module. This module will instruct the planning module to ME TECHNICAL PAPER w ai-enabled robotic nde for structural damage Mounting holes Data port 4 × 2 sensor array Figure 5. The compact NDE system to be mounted on the robot. The sensor array includes 4 × 2 sensors. Each of the sensors is an 18-turn coil with trace spacing of 0.4 mm and trace width of 0.15 mm. 0 5 3 3.05 3.1 3.15 3.2 3.25 3.3 3.35 3.4 10 Distance (cm) 15 20 25 Gap between two plates 100 mm 8 mm 0.2 mm 0.2 mm 0.2 mm 100 mm 0.2 mm 0.2 mm 0.2 mm 1 mm depth 2 mm depth 3 mm depth 4 mm depth 4.5 mm depth 5 mm depth Figure 6. The single-channel line scan result from the two steel plates with cracks at different depths. Frequency (MHz)

J U L Y 2 0 2 1 • M A T E R I A L S E V A L U A T I O N 745 position the robot such that the damage is accessible to the repair tool. The planning module will then instruct the tooling controller in execution of the repair. The tooling controller is a microcontroller that positions the repair actuator directly over the located damage. The repair actuator is an XYZ gantry system that uses three encoded linear motors for positioning. The encoder data are sent back to the tooling controller for accurate closed-loop control. The probe actuator keeps the NDE probe adhered to the material under test at the optimal distance, which results in more accurate NDE sensor readings. Lastly, the FSW tool that repairs the defect in the material under test is mounted at the end effector of the repairing actuator. When the NDE probe is in position, the tool is engaged and the repair is executed. After repair completion, the tool is retracted, and the robot picks up where it left off, navigating the boiler in search of more damage in need of repair. Defect Localization and Estimation Because certain infrastructure may have many defects, many of which are not critical to the boiler’s functionality, simple signal processing with a set threshold is not sufficient for damage classification. Thus, some form of machine learning approach is necessary. One unique challenge to address for the proposed CPS framework is that, in order to execute effective repairs, it is not only necessary to classify damage accurately but also feed damage locations into the global map of the robot’s environment through data fusion. To achieve this goal, simple classification approaches such as support vector machines (Noble 2006) or traditional convolutional neural networks (CNNs) (Albawi et al. 2017) are not adequate for this application because they are only able to classify whether an object is within an image, but have no awareness of where the object is within the image. To address this, an instance-based semantic segmentation method is utilized. This approach differs from traditional semantic segmentation in that it can detect the number of instances of a certain class instead of grouping them together as a single object. As this approach is concerned only with a single object class (the damaged area) inside the entire scan region and there may exist multiple cracks within the same scan, the proposed instance-based approach is ideal for this particular application. The authors’ approach utilizes a two-staged mask, region- based CNN (mask R-CNN) (He et al. 2017) as shown in Figure 7. First, data are fed into a traditional CNN for a cropped subsection, called an anchor, of each NDE scan image. To support training multiple images per batch, all images are resized such that the shortest edge is 800 pixels, but the aspect ratio of the original image is preserved. For each, or any anchor where crack instances are detected, a proposal of its bounding box is generated and stored in memory. The next stage merges each anchor containing the same instance of detected damage region as a single instance. Lastly, a pixel mask is generated for each predicted instance of the damage class. To train any neural network model, a sufficiently large training dataset is necessary, and for this application, NDE data of representative cracks for power plant boiler structures are also needed. A dataset with the number of required Input image Input image Exact region proposals Convert to grayscale Find bounding rectangle Feature extraction Classification Output Fully connected Convolution Pooling Crack? yes Scruff? no Scrape? no Generate pixel mask Figure 7. Mask R-CNN process, architecture, and pixel mask generation.