2.3. Deep Ensemble Fusion (DEF) Model Architecture
This section presents the proposed deep ensemble fusion
(DEF) model, including its architecture, hyperparameters,
and outcomes. The DEF model developed in this investiga-
tion combined data fusion and ensemble learning (Figure 5a).
Ensemble models are particularly advantageous because they
can enhance prediction accuracy and robustness by aggregat-
ing multiple models. On the other hand, data fusion combines
information from multiple sources, leading to more accurate
and reliable results than relying on a single data source. By
integrating diverse data types (e.g., acoustic, visual), data fusion
provides a more complete understanding of the observed
phenomena.
When designing the model, there were several options
regarding the types of data fusion and of ensemble learning to
be used. After considering computational and storage factors,
feature-level fusion was chosen over decision-level fusion
for three reasons: (1) it requires only a single classifier head
and a single forward pass at inference, compatible with the
measured 0.058 s closed-loop reported in Section 3.3. (2) it
allows the network to learn cross-modal correlations at inter-
mediate layers (for example, an intermittent low-amplitude
AE burst paired with a small visual gap is jointly diagnostic for
underextrusion in a way that neither modality is alone) and
(3) decision-level fusion would have required a set of inde-
pendent fully-trained classifiers plus a separately tuned fusion
rule, increasing memory and inference cost on the fog node
[27, 28, 32]. Furthermore, the decision to implement a deep
ensemble approach in the proposed DEF model was driven by
its compatibility with feature-level data fusion and its ability to
handle raw data inputs.
By leveraging ensemble learning and data fusion, the
DEF model uses both images and time-series data as input.
Specifically, the input layer is composed of images and wave-
forms. The feature extraction layer involves the tensor repre-
sentation of the input based on the optimal model (discussed
in Section 2.3.2.). The data fusion layer involves concatenating
the features from both modalities and inputting them to the
classification layer, where the model can indicate the defect.
The DEF model was designed to enhance real-time processing
of diverse sensing modalities, leveraging the unique strengths
of each. Its flexibility allows for incorporating additional
sensing methods and modifying deep learning architectures
for each method without altering the overall DEF framework.
The DEF strategy also fosters modularity in system design,
enabling automatic feature extraction independent of DAQ
systems or user input. Notably, as new deep learning tech-
niques emerge, they can be seamlessly integrated into the
existing ensemble, ensuring continuous adaptation to the
latest advances.
2.3.1. DEF SHANNON ENTROPY CUSTOM LOSS FUNCTION
In previous work by the authors, the Shannon information
entropy (SIE) metric was shown to be useful for detecting
damage [13, 46, 47]. Integrating SIE into deep neural networks
may improve prediction and robustness in complex monitoring
applications. In deep learning, loss functions play a pivotal role
in guiding algorithms during training. They quantify the dif-
ference between the model’s predicted outputs and the actual
target values. Two fundamental types of loss functions are
binary cross-entropy (BCE) and categorical cross-entropy (CCE).
ME
|
DIGITALTHREAD
conv2d_input input: [(None, 480, 640, 3)]
InputLayer output: [(None, 480, 640, 3)]
conv2d input: (None, 480, 640, 3)
Conv2D output: (None, 478, 638, 32)
max_pooling2d input: (None, 478, 638, 32)
MaxPooling2D output: (None, 239, 319, 32)
conv2d_1 input: (None, 239, 319, 32)
Conv2D output: (None, 237, 317, 64)
max_pooling2d_1 input: (None, 237, 317, 64)
MaxPooling2D output: (None, 118, 158, 64)
conv2d_2 input: (None, 118, 158, 64) dense_input input: [(None, 10240)]
Conv2D output: (None, 116, 156, 128) InputLayer output: [(None, 10240)]
max_pooling2d_2 input: (None, 116, 156, 128) dense input: (None, 10240)
MaxPooling2D output: (None, 58, 78, 128) Dense output: (None, 32)
flatten input: (None, 58, 78, 128) dense_1 input: (None, 32)
Flatten output: (None, 579072) Dense output: (None, 8)
concantenate input: [(None, 579072), (None, 8)]
Concantenate output: (None, 579080)
output input: (None, 579080)
Dense output: (None, 1)
...
Input layer
Time-series data input
Images data input
Hidden layers Hidden layers Fully connected
layers
Type of
damage/
defects
Feature extraction layer Data
fusion
layer
Classification layer
...
Convolution
layers
Convolution
layers
Pooling
layers
...
Figure 5. Deep ensemble fusion (DEF) model: (a) conceptual
architecture combining feature-level data fusion with ensemble
learning for real-time multimodal defect detection (b) DEF model
architecture with input/output dimensionality.
40
M AT E R I A L S E V A L U AT I O N • J U LY 2 0 2 6
This section presents the proposed deep ensemble fusion
(DEF) model, including its architecture, hyperparameters,
and outcomes. The DEF model developed in this investiga-
tion combined data fusion and ensemble learning (Figure 5a).
Ensemble models are particularly advantageous because they
can enhance prediction accuracy and robustness by aggregat-
ing multiple models. On the other hand, data fusion combines
information from multiple sources, leading to more accurate
and reliable results than relying on a single data source. By
integrating diverse data types (e.g., acoustic, visual), data fusion
provides a more complete understanding of the observed
phenomena.
When designing the model, there were several options
regarding the types of data fusion and of ensemble learning to
be used. After considering computational and storage factors,
feature-level fusion was chosen over decision-level fusion
for three reasons: (1) it requires only a single classifier head
and a single forward pass at inference, compatible with the
measured 0.058 s closed-loop reported in Section 3.3. (2) it
allows the network to learn cross-modal correlations at inter-
mediate layers (for example, an intermittent low-amplitude
AE burst paired with a small visual gap is jointly diagnostic for
underextrusion in a way that neither modality is alone) and
(3) decision-level fusion would have required a set of inde-
pendent fully-trained classifiers plus a separately tuned fusion
rule, increasing memory and inference cost on the fog node
[27, 28, 32]. Furthermore, the decision to implement a deep
ensemble approach in the proposed DEF model was driven by
its compatibility with feature-level data fusion and its ability to
handle raw data inputs.
By leveraging ensemble learning and data fusion, the
DEF model uses both images and time-series data as input.
Specifically, the input layer is composed of images and wave-
forms. The feature extraction layer involves the tensor repre-
sentation of the input based on the optimal model (discussed
in Section 2.3.2.). The data fusion layer involves concatenating
the features from both modalities and inputting them to the
classification layer, where the model can indicate the defect.
The DEF model was designed to enhance real-time processing
of diverse sensing modalities, leveraging the unique strengths
of each. Its flexibility allows for incorporating additional
sensing methods and modifying deep learning architectures
for each method without altering the overall DEF framework.
The DEF strategy also fosters modularity in system design,
enabling automatic feature extraction independent of DAQ
systems or user input. Notably, as new deep learning tech-
niques emerge, they can be seamlessly integrated into the
existing ensemble, ensuring continuous adaptation to the
latest advances.
2.3.1. DEF SHANNON ENTROPY CUSTOM LOSS FUNCTION
In previous work by the authors, the Shannon information
entropy (SIE) metric was shown to be useful for detecting
damage [13, 46, 47]. Integrating SIE into deep neural networks
may improve prediction and robustness in complex monitoring
applications. In deep learning, loss functions play a pivotal role
in guiding algorithms during training. They quantify the dif-
ference between the model’s predicted outputs and the actual
target values. Two fundamental types of loss functions are
binary cross-entropy (BCE) and categorical cross-entropy (CCE).
ME
|
DIGITALTHREAD
conv2d_input input: [(None, 480, 640, 3)]
InputLayer output: [(None, 480, 640, 3)]
conv2d input: (None, 480, 640, 3)
Conv2D output: (None, 478, 638, 32)
max_pooling2d input: (None, 478, 638, 32)
MaxPooling2D output: (None, 239, 319, 32)
conv2d_1 input: (None, 239, 319, 32)
Conv2D output: (None, 237, 317, 64)
max_pooling2d_1 input: (None, 237, 317, 64)
MaxPooling2D output: (None, 118, 158, 64)
conv2d_2 input: (None, 118, 158, 64) dense_input input: [(None, 10240)]
Conv2D output: (None, 116, 156, 128) InputLayer output: [(None, 10240)]
max_pooling2d_2 input: (None, 116, 156, 128) dense input: (None, 10240)
MaxPooling2D output: (None, 58, 78, 128) Dense output: (None, 32)
flatten input: (None, 58, 78, 128) dense_1 input: (None, 32)
Flatten output: (None, 579072) Dense output: (None, 8)
concantenate input: [(None, 579072), (None, 8)]
Concantenate output: (None, 579080)
output input: (None, 579080)
Dense output: (None, 1)
...
Input layer
Time-series data input
Images data input
Hidden layers Hidden layers Fully connected
layers
Type of
damage/
defects
Feature extraction layer Data
fusion
layer
Classification layer
...
Convolution
layers
Convolution
layers
Pooling
layers
...
Figure 5. Deep ensemble fusion (DEF) model: (a) conceptual
architecture combining feature-level data fusion with ensemble
learning for real-time multimodal defect detection (b) DEF model
architecture with input/output dimensionality.
40
M AT E R I A L S E V A L U AT I O N • J U LY 2 0 2 6
