For Material A, ​​ ​ 50​​​ has a similar mean for many of the
models, but the ​ =​ x​​ 2​ +x​ model (collapsed over material)
has a smaller ​​ ​ 50​​​ while the ​ =​ x​​ 2​ +x​ model (fit to Material A
only) has a larger ​​ ​ 50​​​ Figures 5 and 6 show that the results
from ​​ ​ 90​​​ and ​​ ​ 90/95​​​ are similar. For Material A, the ​ =​ x​​ 2​ +x​
model (collapsed over material) yields the smallest estimates,
likely because it averages over material, so the smaller values
in Material B cause the estimates to be smaller. All the other
models produce estimates larger than the quadratic model
with interactions. For Material B, the quadratic model with
interactions (​ =​ x​​ 2​ +x +m +mx +m ​ x​​ 2​​ yields smaller critical
values than all the simpler models. Models that excluded the ​​
x​​ 2​​ term (e.g., ​ =x​ were more conservative than they needed
to be (i.e., overconservative) when compared with models that
included the ​​ ​ ​ 2​​ term.
According to the AIC, BIC, and LRT results, the best overall
model for all the simulated data is the quadratic model with
interactions: ​ =​ x​​ 2​ +x +m +mx +m ​ x​​ 2​​ Since this model
0.2 0.3 0.4
a50
y =x2 +x +material (Mat A)
y =x +material (Mat A)
y =x2 +x (Mat A only)
y =x (Mat A only)
y =x2 +x (Collapsed)
y =x (Collapsed)
0.2 0.3 0.4
a50
y =x2 +x +material (Mat B)
y =x +material (Mat B)
y =x2 +x (Mat B only)
y =x (Mat B only)
y =x2 +x (Collapsed)
y =x (Collapsed)
0.2 0.3 0.4
80
60
40
20
0
a50
0.2 0.3 0.4
75
50
25
0
a50
Figure 4. Resulting a50 values from 10 000 simulations by model. The dashed black line indicates the median for the best-fitting model, the
quadratic model with interactions (y =x2 +x +m +mx +mx2).
0.2 0.3 0.4 0.5
a90
y =x2 +x +material (Mat A)
y =x +material (Mat A)
y =x2 +x (Mat A only)
y =x (Mat A only)
y =x2 +x (Collapsed)
y =x (Collapsed)
0.2 0.3 0.4 0.5
a90
y =x2 +x +material (Mat B)
y =x +material (Mat B)
y =x2 +x (Mat B only)
y =x (Mat B only)
y =x2 +x (Collapsed)
y =x (Collapsed)
0.2 0.3 0.4 0.5
40
20
0
a90
0.2 0.3 0.4 0.5
60
40
20
0
a90
Figure 5. Resulting a90 values from 10 000 simulations by model. The dashed black line indicates the median for the best-fitting model, the
quadratic model with interactions (y =x2 +x +m +mx +mx2).
A U G U S T 2 0 2 5 • M AT E R I A L S E V A L U AT I O N 69
Density
Density
Density
Density

form was used to generate the data, this result confirms
the usefulness of the AIC and BIC metrics for identifying
the best-fitting model. Table 8 shows the mean differences
between the critical values of ​​ ​ 50​​​ ​​ ​ 90​​​ and ​​ ​ 90/95​​​ from the best
model versus the simpler models. All the simpler models in
Table 8 were statistically significantly different from the qua-
dratic model with interactions (with all P-values 0.001). The
Hodges–Lehmann estimators provide the median difference
between the quadratic model with interactions and the simple
model, with values closest to zero being the most similar.
In all cases but one, the simpler models yielded larger
values than the quadratic model with interactions. Thus, for
this simulation, simpler models generally overestimated the
critical values and were overly conservative. Furthermore, once
transformed to probability space, no single simpler model was
consistently similar to the best-fitting quadratic model with
interactions—meaning no simpler model could ensure consis-
tent overconservatism.
At the opposite extreme, overfitted models that included ​​
x​​ 3​​ terms produced nonsensical critical values. Recall that the
ME
|
PODMODELING
0.2 0.3 0.4 0.5
40
20
0
a90/95 0.2 0.3 0.4 0.5
a90/95
y =x2 +x +material (Mat A)
y =x +material (Mat A)
y =x2 +x (Mat A only)
y =x (Mat A only)
y =x2 +x (Collapsed)
y =x (Collapsed)
60
40
20
0
a90/95 0.2 0.3 0.4 0.5 0.2 0.3 0.4 0.5
a90/95
y =x2 +x +material (Mat B)
y =x +material (Mat B)
y =x2 +x (Mat B only)
y =x (Mat B only)
y =x2 +x (Collapsed)
y =x (Collapsed)
Figure 6. Resulting a90/95 values from 10 000 simulations by model. The dashed black line indicates the median for the best-fitting model, the
quadratic model with interactions (y =x2 +x +m +mx +mx2).
TA B L E 7
Critical values and 95% confidence intervals for the 10 000 simulations, using the Hodges-Lehmann
estimator
Model y =Material a50 (95% confidence) a90 (95% confidence) a90/95 (95% confidence)
x Combo 0.2714 (0.2712, 0.2715) 0.4766 (0.4764, 0.4768) 0.5123 (0.5121, 0.5125)
x2 +x Combo 0.2076 (0.2075, 0.2077) 0.2873 (0.2871, 0.2874) 0.2964 (0.2962, 0.2965)
x A subset 0.3708 (0.3705, 0.3711) 0.4409 (0.4405, 0.4413) 0.4526 (0.4522, 0.4530)
x2 +x A subset 0.2714 (0.2712, 0.2716) 0.3830 (0.3828, 0.3833) 0.4118 (0.4116, 0.4120)
x +m A 0.2711 (0.2707, 0.2715) 0.4633 (0.4630, 0.4637) 0.5052 (0.5048, 0.5056)
x2 +x +m* A 0.2685 (0.2712, 0.2688)* 0.3416 (0.3672, 0.3419)* 0.3502 (0.3897, 0.3035)*
x B subset 0.2457 (0.2456, 0.2459) 0.2828 (0.2827, 0.2830) 0.2891 (0.2889, 0.2893)
x2 +x B subset 0.2714 (0.2712, 0.2716) 0.3830 (0.3828, 0.3833) 0.4118 (0.4116, 0.4120)
x +m B 0.2714 (0.2712, 0.2716) 0.3674 (0.3672, 0.3676) 0.3898 (0.3897, 0.3901)
x2 +x +m* B 0.1767 (0.1765, 0.1768)* 0.2150 (0.2149, 0.2151)* 0.2195 (0.2194, 0.2196)*
*The critical values corresponding to the model that best fit the data.
70
M AT E R I A L S E V A L U AT I O N • A U G U S T 2 0 2 5
Density
Density