data was simulated from a quadratic function, so models with
cubic terms constitute overfitting. None of the 10 000 values for
Material A existed, and the 10 000 values for Material B were
negative. Specifically, 90 covered a range of –0.0508 to –0.0162,
and 90/95 covered a range of –0.0421 to –0.0021. Even if the
cubic term were to provide an explanation for the negative
values, the absolute values of these critical values are an
order of magnitude smaller than those produced by the other
models. Future work, which carefully handles multicollinearity,
may provide different results.
Conclusion
The current military handbook for probability of detection
studies (POD) (MIL-HDBK1823A) uses a simple linear model
relating the signal from an NDE system to the observed discon-
tinuity size [4]. However, other important variables may be nec-
essary to describe the NDE response. Currently, POD studies
involving important variables beyond discontinuity size are
handled by either (1) creating subsets of the data, building a
POD for each one, and choosing the most conservative (largest
a90/95 result or (2) naively assuming that discontinuity size is
the only important variable, essentially averaging over all other
variables. Subsetting the data may cause problems with sample
size, and taking the most conservative result from the set can
lead to an unnecessarily conservative (i.e., overconservative)
estimate, rather than a fleet-wide average, as demonstrated
by the simulation experiment. Ignoring important variables
inflates the variance, creating a larger confidence interval.
Furthermore, without the other variables included, the linear
model will be skewed toward the cases where the majority of
the data occurs. Data collection is often influenced by cost and
availability—for example, if one material is cheaper, it may be
overrepresented in the study, skewing the results toward that
material. Neither of these options is ideal. Therefore, a meth-
odology was built so that additional variables included in the
linear model are correctly handled in the transformation to
probability with respect to discontinuity size.
It is reasonable to expect that a better linear fit to the data
will provide a more accurate POD estimate. Using a linear
model that omits important variables in a POD study can
introduce bias into the POD curve estimates. MIL-HDBK-1823A
explains how to perform POD analysis for the relationship
between signal response (â vs. a) and discontinuity size this
paper extends those methods by showing how to include other
important variables and higher-order terms of discontinuity
size. These methods can be used to build more accurate linear
models for NDE applications. The categorical variable used in
our example demonstrated differences among materials, but
these methods could also be used to model other categorical
variables—such as cracks versus notches—potentially eliminat-
ing the need to separately estimate a knock-down factor.
This paper used a simulation size of 10 000 to illus-
trate how simple models may yield biased estimates. Future
work could apply these methodologies to a large dataset
of bootstrap-sampled NDE data to determine the relation-
ship between the quality of a linear model fit and the bias
in the POD estimate. For some NDE applications, a simple
model may be sufficiently accurate, while for others, a more
complex model may be necessary. This paper provides a prac-
tical demonstration that can be readily extended to include
N continuous and/or categorical variables, along with addi-
tional higher-order terms. Overfitting the model can lead
TA B L E 8
Hodges-Lehmann estimated differences in the critical values when comparing simpler models to the
complex model (y =x2 +x +m +mx +mx2)
Model y =Material a50 (95% confidence) a90 (95% confidence) a90/95 (95% confidence)
x A –0.0028 (–0.0032, –0.0024) –0.1349 (–0.1353, –0.1346) –0.1620 (–0.1623, –0.1616)
x2 +x A 0.0609 (0.0607, 0.0611) 0.0543 (0.0541, 0.0545) 0.0539 (0.0537, 0.0541)
x A subset –0.1023 (–0.1024, –0.1022) –0.0993 (–0.0995, –0.0992) –0.1024 (–0.1026, –0.1023)
x2 +x A subset –0.0028 (–0.0031, –0.0025) –0.0413 (–0.0417, –0.0409) –0.0615 (–0.0619, –0.0611)
x +m A –0.0026 (–0.0031, –0.0020)* –0.1217 (–0.1223, –0.1212)* –0.1550 (–0.1555, –0.1544)*
x B –0.0947 (–0.0950, –0.0945) –0.2616 (–0.2619, –0.2614) –0.2928 (–0.2931, –0.2925)
x2 +x B –0.0310 (–0.0310, –0.0309) –0.0723 (–0.0724, –0.0722) –0.0768 (–0.0770, –0.0767)
x B subset –0.0691 (–0.0691, –0.0690) –0.0679 (–0.0680, –0.0678) –0.0696 (–0.0696, –0.0695)
x2 +x B subset –0.0947 (–0.0950, –0.0945) –0.1681 (–0.1683, –0.1678) –0.1923 (–0.1926, –0.1920)
x +m B –0.0947 (–0.0950, –0.0945)* –0.1524 (–0.1527, –0.1522)* –0.1703 (–0.1706, –0.1700)*
Note: Negative values indicate that the estimates from the quadratic model with interactions were smaller than those from the simpler model listed.
Mann-Whitney two-sample matched paired two-tailed tests indicated that every comparison was significant (all P-values 0.0001).
The values in bold were closest to the quadratic model with interactions.
*The critical values corresponding to the model that best fit the data.
A U G U S T 2 0 2 5 • M AT E R I A L S E V A L U AT I O N 71
cubic terms constitute overfitting. None of the 10 000 values for
Material A existed, and the 10 000 values for Material B were
negative. Specifically, 90 covered a range of –0.0508 to –0.0162,
and 90/95 covered a range of –0.0421 to –0.0021. Even if the
cubic term were to provide an explanation for the negative
values, the absolute values of these critical values are an
order of magnitude smaller than those produced by the other
models. Future work, which carefully handles multicollinearity,
may provide different results.
Conclusion
The current military handbook for probability of detection
studies (POD) (MIL-HDBK1823A) uses a simple linear model
relating the signal from an NDE system to the observed discon-
tinuity size [4]. However, other important variables may be nec-
essary to describe the NDE response. Currently, POD studies
involving important variables beyond discontinuity size are
handled by either (1) creating subsets of the data, building a
POD for each one, and choosing the most conservative (largest
a90/95 result or (2) naively assuming that discontinuity size is
the only important variable, essentially averaging over all other
variables. Subsetting the data may cause problems with sample
size, and taking the most conservative result from the set can
lead to an unnecessarily conservative (i.e., overconservative)
estimate, rather than a fleet-wide average, as demonstrated
by the simulation experiment. Ignoring important variables
inflates the variance, creating a larger confidence interval.
Furthermore, without the other variables included, the linear
model will be skewed toward the cases where the majority of
the data occurs. Data collection is often influenced by cost and
availability—for example, if one material is cheaper, it may be
overrepresented in the study, skewing the results toward that
material. Neither of these options is ideal. Therefore, a meth-
odology was built so that additional variables included in the
linear model are correctly handled in the transformation to
probability with respect to discontinuity size.
It is reasonable to expect that a better linear fit to the data
will provide a more accurate POD estimate. Using a linear
model that omits important variables in a POD study can
introduce bias into the POD curve estimates. MIL-HDBK-1823A
explains how to perform POD analysis for the relationship
between signal response (â vs. a) and discontinuity size this
paper extends those methods by showing how to include other
important variables and higher-order terms of discontinuity
size. These methods can be used to build more accurate linear
models for NDE applications. The categorical variable used in
our example demonstrated differences among materials, but
these methods could also be used to model other categorical
variables—such as cracks versus notches—potentially eliminat-
ing the need to separately estimate a knock-down factor.
This paper used a simulation size of 10 000 to illus-
trate how simple models may yield biased estimates. Future
work could apply these methodologies to a large dataset
of bootstrap-sampled NDE data to determine the relation-
ship between the quality of a linear model fit and the bias
in the POD estimate. For some NDE applications, a simple
model may be sufficiently accurate, while for others, a more
complex model may be necessary. This paper provides a prac-
tical demonstration that can be readily extended to include
N continuous and/or categorical variables, along with addi-
tional higher-order terms. Overfitting the model can lead
TA B L E 8
Hodges-Lehmann estimated differences in the critical values when comparing simpler models to the
complex model (y =x2 +x +m +mx +mx2)
Model y =Material a50 (95% confidence) a90 (95% confidence) a90/95 (95% confidence)
x A –0.0028 (–0.0032, –0.0024) –0.1349 (–0.1353, –0.1346) –0.1620 (–0.1623, –0.1616)
x2 +x A 0.0609 (0.0607, 0.0611) 0.0543 (0.0541, 0.0545) 0.0539 (0.0537, 0.0541)
x A subset –0.1023 (–0.1024, –0.1022) –0.0993 (–0.0995, –0.0992) –0.1024 (–0.1026, –0.1023)
x2 +x A subset –0.0028 (–0.0031, –0.0025) –0.0413 (–0.0417, –0.0409) –0.0615 (–0.0619, –0.0611)
x +m A –0.0026 (–0.0031, –0.0020)* –0.1217 (–0.1223, –0.1212)* –0.1550 (–0.1555, –0.1544)*
x B –0.0947 (–0.0950, –0.0945) –0.2616 (–0.2619, –0.2614) –0.2928 (–0.2931, –0.2925)
x2 +x B –0.0310 (–0.0310, –0.0309) –0.0723 (–0.0724, –0.0722) –0.0768 (–0.0770, –0.0767)
x B subset –0.0691 (–0.0691, –0.0690) –0.0679 (–0.0680, –0.0678) –0.0696 (–0.0696, –0.0695)
x2 +x B subset –0.0947 (–0.0950, –0.0945) –0.1681 (–0.1683, –0.1678) –0.1923 (–0.1926, –0.1920)
x +m B –0.0947 (–0.0950, –0.0945)* –0.1524 (–0.1527, –0.1522)* –0.1703 (–0.1706, –0.1700)*
Note: Negative values indicate that the estimates from the quadratic model with interactions were smaller than those from the simpler model listed.
Mann-Whitney two-sample matched paired two-tailed tests indicated that every comparison was significant (all P-values 0.0001).
The values in bold were closest to the quadratic model with interactions.
*The critical values corresponding to the model that best fit the data.
A U G U S T 2 0 2 5 • M AT E R I A L S E V A L U AT I O N 71
