99.67 99.01 99.20 96.00
DEF Model (Images +Time-Series, Custom Loss)-Recall_Defect
DEF Model (Images +Time-Series, Custom Loss)-Accuracy
100.00 100.00 100.00 100.00
DEF Model (Images +Time-Series, Custom Loss)-Recall_NoDefect 99.01 98.00 98.01 92.01
DEF Model (Images +Time-Series, BCE Loss)-Accuracy 84.23 80.15 82.30 81.29
DEF Model (Images +Time-Series, BCE Loss)-Recall_Defect 82.63 79.58 78.56 78.68
DEF Model (Images +Time-Series, BCE Loss)-Recall_NoDefect 81.14 78.48 79.57 78.66
CNN (Best Processed Images Model)-Accuracy 85.33 82.36 84.96 83.94
CNN (Best Processed Images Model)-Recall_Defect 100 100.00 100.00 100.00
CNN (Best Processed Images Model)-Recall_NoDefect 0.00 0.00 0.00 0.00
CNN (Best Raw Images Model)-Accuracy 83.00 75.15 73.26 63.56
CNN (Best Raw Images Model)-Recall_Defect 100.00 100.00 100.00 100.00
CNN (Best Raw Images Model)-Recall_NoDefect 0.00 50.00 50.00 50.00
FFNN w/SMOTE (Best Time-Series Model)-Accuracy 76.75 75.33 74.04 70.20
FFNN w/SMOTE (Best Time-Series Model)-Recall_Defect 73.43 70.25 72.96 70.57
FFNN w/SMOTE (Best Time-Series Model)-Recall_NoDefect 73.00 77.88 75.66 70.81
FFNN (Best Time-Series Model)-Accuracy 92.30 86.68 82.70 83.50
FFNN (Best Time-Series Model)-Recall_Defect 67.10 31.78 29.46 40.54
FFNN (Best Time-Series Model)-Recall_NoDefect 98.27 95.70 94.89 93.37
100
80
60
40
20
0
Models accuracy &recall (%)performance heatmap
Training Test_U16 Test_U17 Test_U18
5
4
3
2
1
CNN (Best Processed Images Model) 4.94 3.49 3.70 5.63
CNN (Best Raw Images Model) 5.67 3.66 4.27 5.96
DEF Model (Images +Time-Series, BCE Loss) 1.36 1.63 1.48 1.57
DEF Model (Images +Time-Series, Custom Loss) 0.01 0.01 0.01 0.04
FFNN (Best Time-Series Model) 0.26 0.36 0.41 0.39
FFNN w/SMOTE (Best Time-Series Model) 0.56 0.59 0.58 0.59
Models loss performance heatmap
Training Test_U16 Test_U17 Test_U18
Figure 8. DEF model single-defect results compared with the best single-modality models: (a) accuracy and recall heatmap across one training and
three test datasets (b) loss heatmap across the same datasets.
J U LY 2 0 2 6 • M AT E R I A L S E V A L U AT I O N 45
Model
and
metric
Model
DEF Model (Images +Time-Series, Custom Loss)-Recall_Defect
DEF Model (Images +Time-Series, Custom Loss)-Accuracy
100.00 100.00 100.00 100.00
DEF Model (Images +Time-Series, Custom Loss)-Recall_NoDefect 99.01 98.00 98.01 92.01
DEF Model (Images +Time-Series, BCE Loss)-Accuracy 84.23 80.15 82.30 81.29
DEF Model (Images +Time-Series, BCE Loss)-Recall_Defect 82.63 79.58 78.56 78.68
DEF Model (Images +Time-Series, BCE Loss)-Recall_NoDefect 81.14 78.48 79.57 78.66
CNN (Best Processed Images Model)-Accuracy 85.33 82.36 84.96 83.94
CNN (Best Processed Images Model)-Recall_Defect 100 100.00 100.00 100.00
CNN (Best Processed Images Model)-Recall_NoDefect 0.00 0.00 0.00 0.00
CNN (Best Raw Images Model)-Accuracy 83.00 75.15 73.26 63.56
CNN (Best Raw Images Model)-Recall_Defect 100.00 100.00 100.00 100.00
CNN (Best Raw Images Model)-Recall_NoDefect 0.00 50.00 50.00 50.00
FFNN w/SMOTE (Best Time-Series Model)-Accuracy 76.75 75.33 74.04 70.20
FFNN w/SMOTE (Best Time-Series Model)-Recall_Defect 73.43 70.25 72.96 70.57
FFNN w/SMOTE (Best Time-Series Model)-Recall_NoDefect 73.00 77.88 75.66 70.81
FFNN (Best Time-Series Model)-Accuracy 92.30 86.68 82.70 83.50
FFNN (Best Time-Series Model)-Recall_Defect 67.10 31.78 29.46 40.54
FFNN (Best Time-Series Model)-Recall_NoDefect 98.27 95.70 94.89 93.37
100
80
60
40
20
0
Models accuracy &recall (%)performance heatmap
Training Test_U16 Test_U17 Test_U18
5
4
3
2
1
CNN (Best Processed Images Model) 4.94 3.49 3.70 5.63
CNN (Best Raw Images Model) 5.67 3.66 4.27 5.96
DEF Model (Images +Time-Series, BCE Loss) 1.36 1.63 1.48 1.57
DEF Model (Images +Time-Series, Custom Loss) 0.01 0.01 0.01 0.04
FFNN (Best Time-Series Model) 0.26 0.36 0.41 0.39
FFNN w/SMOTE (Best Time-Series Model) 0.56 0.59 0.58 0.59
Models loss performance heatmap
Training Test_U16 Test_U17 Test_U18
Figure 8. DEF model single-defect results compared with the best single-modality models: (a) accuracy and recall heatmap across one training and
three test datasets (b) loss heatmap across the same datasets.
J U LY 2 0 2 6 • M AT E R I A L S E V A L U AT I O N 45
Model
and
metric
Model
