JOURNAL OF

COMPOSITE
Article M AT E R I A L S
Journal of Composite Materials
2020, Vol. 54(21) 2823–2836
! The Author(s) 2020
Reduced allowable strength of composite Article reuse guidelines:
sagepub.com/journals-permissions
laminate for unknown distribution due DOI: 10.1177/0021998320903781
journals.sagepub.com/home/jcm
to limited tests

Yiming Zhang1, Nam H Kim1 , Upul R Palliyaguru2,

Jaco F Schutte3 and Raphael T Haftka1

Abstract
In design under uncertainty, random distributions are often determined by expensive sampling tests. A key question is
whether to invest in more samples or live with a reduced performance by fewer samples due to large uncertainty. The
question is particularly difficult to answer when the type of distribution is unknown. This paper investigates the tradeoff
between performance and conservativeness in estimating B-basis allowables, using experiments on composite plates with
holes. Two approaches that do not require a distribution type are examined: (1) bootstrap confidence intervals and (2)
Hanson-Koopmans non-parametric method. Based on the study, it is found that the Hanson-Koopmans method was
more conservative than the bootstrap method because the latter penalized allowables for small-size samples. For a small
number of samples (less than 29), conservative estimations are preferred over accuracy to account for the large
uncertainty. Based on this observation, the bootstrap-assisted Hanson-Koopmans method is proposed to enhance the
conservativeness. For the tested cases, the performance penalty using the bootstrap-assisted Hanson-Koopmans method
for a small number of samples is found to be substantial.

Keywords
Tolerance intervals, material strength, composite laminates, bootstrapping, uncertainty quantification

preferred when uncertainty exists especially for safety-

Introduction critical applications such as helicopter rotor blades.5
In design under uncertainty, the distributions of However, the reduction in performance can be substan-
random variables are often determined by expensive tial because sampling uncertainty is epistemic uncer-
samples. The uncertainty related to the material proper- tainty, which is usually treated more conservatively
ties of a composite laminate is significant and has to be than aleatory uncertainty. Consequently, there has
modeled properly for reliable designs.1 The manufac- been substantial recent interest in the tradeoff between
turing and testing process could be expensive and reducing sampling uncertainty and improving perform-
time-consuming which makes the design process more ance. Picheny et al.6 studied the influence of sample
challenging. One remedy for limited tests is to introduce sizes and target probability of failure on the conserva-
analytical models/simulations.2,3 Vallmajó et al.3 tive estimate. Bae et al.7,8 showed a tradeoff between
proposed an uncertainty quantification and manage- making design conservative and using more samples to
ment (UQ&M) framework based on analytical models
to compute the B-basis design allowables of notched
configurations. However, the analytical models are 1
Department of Mechanical and Aerospace Engineering, University of
not always available, and this paper focuses on experi- Florida, USA
2
mental tests only. With limited tests, a key question is National Institute for Aviation Research, Wichita State University, USA
3
whether to invest in more samples or live with the Materials Sciences Corporation, USA
reduced performance by fewer samples due to large
Corresponding author:
uncertainty. Identification of the distribution/uncer- Nam H Kim, Department of Mechanical and Aerospace Engineering,
tainty forms for statistical analysis would be also chal- University of Florida, PO Box 116250, Gainesville, FL 32611-6250, USA.
lenging with few samples.4 Conservative designs are Email: [email protected]
2824 Journal of Composite Materials 54(21)

reduce sampling uncertainty. However, they assumed sample size is smaller than a critical value. Besides the
that the samples are normally distributed. effort to develop statistical tools for characterizing TI,
Establishing the tradeoff between the increased extensive experimental studies have been reported to
sample size and reduced performance is more challen- understand the variation of material properties. The
ging when the type of distribution is unknown. This world-wide failure exercises23,24 provided experimental
paper presents a lesson learned from the regulations data and benchmarks for failure criteria of composites.
governing the design of composite materials used for The Laminate Variability Method25 was proposed to
aircraft structures. Composite materials are widely used incorporate the material properties at a lamina level
for the design of various mechanical systems due to the to mitigate the adverse effects of limited numbers of
outstanding capability to be tailored to specific load test coupons while computing B-basis values.
paths and conditions, resulting in weight efficient Carlsson et al.26 and the Composite Materials
designs. For example, more than 50% of the Boeing Handbook-17 (CMH-17)10 provided an in-depth guide-
787 airframe and the Airbus A350XWB are made of line for systematic experimental analysis.
carbon fiber composites.9 The B-basis allowables are often estimated from
To determine the conservative strength of composite experimental test results on limited samples (e.g. less
plates, multiple coupons at the same configuration than 29 samples). This paper investigates the estimation
(design) are tested. Design allowables (e.g. A- or B- of B-basis allowables, a typical TI, from limited sam-
basis allowables10,11), which are also termed as toler- ples using test results obtained from the composite
ance intervals/limits (TI), are usually obtained from the laminates with a hole.27 The tests were performed at
confidence interval of a low percentile of measured eight configurations with 18 samples per configuration.
strengths. Various methods for computing the TI of The eight configurations are selected by changing two
composite material are well documented in the litera- design variables: the size of the hole and the fraction of
ture.12,13 The calculation of TI depends on the type of 45-degree plies in the laminates. Experimental data
distributions (e.g. normal, lognormal, and Weibull). were collected on open-hole-tension (OHT) strength
Young11 summarized the statistical approaches for cal- tests28,29 following ASTM standard27 for this study.
culation of TI for discrete and continuous distributions. Two issues are investigated in this paper: (1) Are the
When there is no clear indication of a specific type of B-basis allowables estimated from small-size samples as
distribution, MIL-HDBK-17-1F14 recommends the reliable as those from large-size samples? (2) What is the
non-parametric method such as the Hanson- weight penalty when using the design allowables from a
Koopmans (HK) method.15 The non-parametric limited number of samples? In order to address these two
method compensates for the lack of knowledge of the issues, two approaches are evaluated in this paper: (1)
distribution by increasing conservativeness. bootstrap confidence intervals and (2) HK method. The
Conservative allowables imply more weight, but former does not assume an underlying probability dis-
Bhachu16 found that the non-parametric approach is tribution, while the latter assumes log-concave CDF
efficient in achieving conservativeness without excessive class, which is good for the distribution of composite
increase in weight, compared to methods based on failure strength.
assumed distribution. Still, the HK method can fail One challenge for predicting B-basis allowables is
when the sample size (number of the replicates) is smal- balancing conservativeness (for a safe design) versus
ler than a critical value, which varies with distribu- performance (weight penalty). Conservative prediction
tions.15 Besides the HK method, bootstrap confidence is a necessary requirement for certification by regula-
bounds have been used to infer design allowables with- tory bodies, like the Federal Aviation Administration.
out specifying a statistical distribution. Cross et al.17 B-basis allowables are sensitive to distribution form
estimated the confidence intervals of the crack growth and sample size, which complicates the calculation.
model using bootstrap confidence bound. Bigerelle This paper utilizes a partial set of samples (out of 18
et al.18,19 quantified the uncertainty in Paris law mater- samples per configuration) to estimate B-basis allow-
ial constant using the bootstrap. Bhachu et al.16,20 com- ables, from which the conservativeness and weight pen-
pared several common approaches for fatigue crack alty are calculated. This paper also explores the
growth problems. Romero et al.21,22 tested the perform- usefulness of combining the non-parametric estimation
ance of the TI method, kernel density method, Johnson with bootstrapping to account for the unidentified
method, and non-parametric method. uncertainty and ensure better conservativeness.
The TI approaches are based on rigorous mathe- The paper is structured as follows: The following
matic assumptions and work well with high variability. section discusses the experiments of the benchmark
However, with too small number of samples, it is diffi- OHT tests and statistics of the experimental results.
cul to identify the distribution type to apply TI. The Then, the estimation of B-basis at a given configuration
non-parametric B-basis approach is invalid when the with samples is detailed. ‘Estimating B-basis allowables
Zhang et al. 2825

of the OHT tests’ section evaluates the B-basis estima- gage middle

tions using the experimental results. In the penultimate F F

section, the bootstrap-assisted HK is proposed to D w
enhance the conservativeness from limited samples. In
conclusion, we summarize the major outcomes and
future work on the estimation of design allowables. Figure 1. Open-hole tension test specimen configuration and
observed failure mode.

OHT tests
Table 1. The test configurations in two design variables.
Experiments
Points 1 2 3 4 5 6 7 8
OHT test30,31 is a benchmark test to investigate the effect
of an unfilled hole on the tensile failure strength. In this ðw=D,R45 Þ (3,0.2) (3,0.8) (4,0.2) (4,0.5) (6,0.2) (6,0.5) (6,0.8) (8,0.2)
paper, it is used to investigate the approaches to estimate
B-basis allowables. The test specimen geometry is shown
in Figure 1. The composite laminates are made from
MTM45-1 PWC2 3K PW G30-500 fabric prepregs, Table 2. w/D test matrix at a given R45.
and the tests were performed according to ASTM
Batches/ No. of
D5766.27 The width of the specimen and the diameter
w (in.) D (in.) w/D replicates specimens
of the hole were denoted by w and D, respectively.
Two design parameters varied in the tests: the ratio 0.75 0.250 3 36 18
w=D and layups measured by the fraction of 45 plies 1.00 0.250 4 36 18
(R45 ). Eight configurations are examined in the 2D vari- 1.50 0.250 6 36 18
able space according to Table 1. Table 2 details the test 2.00 0.250 8 36 18
matrix for different w=D. Table 3 lists the test matrix
for different layups quantified by the fraction of 45
plies. Each configuration was composed of three pre- samples at different configurations. Configuration 2
preg batches, with each batch containing six samples. showed a pattern of Weibull distribution with a heavy
tail, configurations 1 and 7 follows a bimodal, and con-
figuration 6 is close to a uniform distribution. The
Statistics of the experimental strengths Kolmogorov–Smirnov (KS) test12 was used to quantify
The test results are first examined using boxplot (see which continuous distribution is the best fit. The KS
Figure 2) and statistics (see Table 4). The means of test is a non-parametric test to quantify the goodness-
strengths vary between 37 and 100 ksi in different config- of-fit between a given probability distribution and the
urations. For a fixed R45 fraction, the strength gradually empirical distribution of samples. The OHT samples
increases with w=D ratio. For a fixed w=D ratio, the are tested for a normal, uniform, and Weibull distribu-
strength gradually decreases with R45 fraction. The R45 tions, but there is no single distribution that fits all
fraction has a more significant impact on the strength than samples the best. Details of the KS tests are provided
the w=D ratio. Configurations 2 and 7 have the smallest in Appendix A.2. The normal distribution fits best for
strength and variation (R45 ¼ 0:8). The standard devi- four configurations, uniform distribution fits best at a
ation (SD) varies substantially from 1.33 to 5.41 (5.41/ configuration, and Weibull distribution fits best at three
1.33 ¼ 4.06), whereas the coefficient of variation (CoV) configurations. Based on the histograms and KS tests,
varies between 0.033 and 0.059 (0.059/0.033 ¼ 1.78). the non-parametric approach was selected for calculat-
The variation of CoV reduces noticeably compared with ing B-basis allowables.
SD, which indicates that SD is highly correlated with the
mean. The strengths of some samples are far from the rest, Estimating B-basis allowables for unknown
such as the lowest sample at configuration 4 or the highest
sample at configuration 5. The outliers of experiments
distributions
have a significant impact on the statistics.32 The max- A design allowable is determined such that it is less
imum normed residual test33 was recommended by than a large portion of the population with a high
CMH-1710 to detect outliers. However, no outlier was level of confidence. The B-basis allowable is a bound
identified for the OHT tests. Details of the maximum that is less than 90% of the population with 95% con-
normed residual test are included in Appendix A.1. fidence, as shown in Figure 4. Two mainstream meth-
Estimation of B-basis heavily depends on the type of ods are examined for calculating B-basis allowables
distributions. Figure 3 shows the histograms of the from unknown distributions.
2826 Journal of Composite Materials 54(21)

Table 3. Layup test matrix at a given w/D.

R45 Layup % 0 /45 /90 Ply stacking sequence Batches/replicates No. of specimens

0.2 40/20/40 [0/90/0/90/45/-45/90/0/90/0]s 36 18

0.5 25/50/25 [(45/0/45/90)2]s 36 18
0.8 10/80/10 [45/45/90/45/45/45/45/0/45/45]s 36 18

Figure 2. Boxplots of tensile strength using 18 samples at eight different configurations.

Table 4. Statistics of tensile strength using 18 samples at eight different configurations.

Configuration 1 2 3 4 5 6 7 8
ðw=D,R45 Þ (3,0.2) (3,0.8) (4,0.2) (4,0.5) (6,0.2) (6,0.5) (6,0.8) (8,0.2)

Tensile strength (ksi) Mean 87.35 37.68 91.45 67.62 97.43 71.06 50.29 100.11
SD 3.54 1.33 5.41 2.70 5.19 2.84 1.67 4.90
CoV 0.040 0.035 0.059 0.039 0.053 0.040 0.033 0.048

where k is a factor depending on sample size. The par-

A non-parametric approach using order statistics ameters r and k are found in Table 5.10
CMH-1710 recommended a non-parametric
approach for calculating B-basis allowables when sam- B-basis allowables using bootstrap confidence
ples do not demonstrate a clear distribution pattern
(e.g. due to a limited number of samples). The non-
intervals
parametric approach is based on the order statistics Bootstrapping34,35 is a data-driven approach for statis-
and varies with sample size. When more than 28 sam- tical inference and commonly used for estimating bias
ples are available, the B-basis value is the rth lowest and variance of a critical statistic. Various methods
sample, where r varies with sample size. For example, have been established for finding confidence bounds
r equals to one when 30 samples are available. The HK from bootstrap sampling distributions. The percentile
method is suggested for non-parametric estimation of method determines the confidence interval for the stat-
B-basis allowables with less than 29 samples. We istic of interest (e.g. 10th percentile) from bootstrapped
adopt the HK method for the non-parametric estima- distributions. The bias correction approach modifies
tion ðBHK Þ. the estimated statistic of interest by a bias to account
First, strength samples are ordered by magnitude; for the small-size samples. The bias-corrected acceler-
xð1Þ is the lowest strength, and xðrÞ is the rth lowest ated method corrects for both bias and skewness in the
sample. Then, B-basis is determined by distribution of bootstrap estimates. Picheny et al.6
incorporated bootstrapping for reliability analysis of a
 k
xð1Þ system response. Edwards et al.36 proposed an
BHK ¼ xðrÞ ð1Þ approach to estimate the lower percentiles of material
xðrÞ
properties using bootstrapping. Lee et al.37,38 evaluated
Zhang et al. 2827

Figure 3. Histograms of samples at different configurations.(a) Configuration 1, (b) Configuration 2, (c) Configuration 3,
(d) Configuration 4, (e) Configuration 5, (f) Configuration 6, (g) Configuration 7 and (h) Configuration 8.

Table 5. Parameters for the non-parametric B-basis allowables

Samples with different number of samples.

Number of samples 6 8 10 18
Material
properties r 5 6 6 9
Estimated 10th percentile K 3.064 2.382 2.137 1.354

B-basis allowable: lower 95% C.I. selected. Then, a sample statistic (e.g. mean or variance)
is calculated from each bootstrapped sample. An
Specimen configuration
empirical distribution of the statistic is obtained to
Figure 4. Illustration of B-basis allowables based on samples. define the uncertainty. The percentile for small-size
samples is computed according to Langford.39
We adopted the bias-corrected accelerated method
and improved the accurate coverage of the bootstrap which corrects the B-basis estimation for both bias
confidence interval. and skewness ðBboot Þ. The one-sided lower confidence
A typical bootstrapping procedure resamples (with bound is determined by
replacement) from the available sample set without   
assuming any probability model. The distribution is z0 þ z
Bboot ¼ G^ 1  z0 þ ð2Þ
approximated by the population of existing samples. 1  Aðz0 þ z Þ
Samples with the same size are drawn (with replace-
ment) from the population multiple times. The existing where G^ 1 ðÞ is the inverse empirical CDF of the boot-
samples are resampled with an equal chance of being strap sampling distribution, z is the z-score from the
2828 Journal of Composite Materials 54(21)

standard normal distribution, z0 is the bias correction, where Hðx 5 0Þ ¼ 0 and Hðx  0Þ ¼ 1. CF ¼ 1 means
A is the acceleration parameter, and ðÞ is the standard that all NR sets are conservative. Note that a good
normal CDF. A detailed procedure for determining the method will have a high conservative fraction with a
Bboot is described in DiCiccio and Efron.40 The bias low weight penalty factor. CF and WPF are calculated
correction z0 is used to modify the confidence bounds using BðkÞ ðkÞ
HK and Bboot .
and to account for a limited number of samples. A is
the adjustment to correct for the accelerating standard
error. We used the ‘‘bootci’’ function in MATLAB to Estimation of B-basis allowables from experimental
predict Bboot .
replicates
We first compared the B-basis allowables estimated
Estimating B-basis allowables of the from BðkÞ ðkÞ
HK and Bboot . The baseline B-basis allowables
OHT tests from all 18 samples are summarized in Table 6. Bð18Þ
HK
Test plan to evaluate the estimations of B-basis was more conservative than the Bð18Þ
boot at all configur-
allowables ations. The differences between Bð18Þ ð18Þ
HK and Bboot varied
within [0.78, 2.19] ksi.
We resort to the experimental results to estimate
B-basis allowables with different sample sizes. The The B-basis allowables estimated from NR ¼ 1000
resampling scheme31,41 is adopted to make the most sets of partial resamples are shown as boxplots in
use of experiments and investigate the B-basis estima- Figure 5. The mean values of BðkÞ
HK were conservative
tion with varying sample size. The lower 10th percentile ðkÞ
compared to that of Bboot by 4%–20%. Another inter-
of Bð18Þ is used as the baseline for comparison. Out of
the full set of 18 samples at each configuration, partial esting observation was that the mean values of BðkÞ
boot
samples are selected (without replacement) to examine remained almost the same, whereas that of BðkÞ HK
the performance of estimated B-basis allowables. The increased gradually by a few percents as k increases.
B-basis allowables with k partial samples are denoted as The performance of BðkÞ ðkÞ
boot and BHK with different
BðkÞ , where k ¼ 6, 8, 10. Partial sampling is repeated sample sizes is further compared using the normalized
NR ¼ 1000 times to calculate their statistics. The empir- mean value of the B-basis estimations from NR sets of
ical distributions from NR sets of BðkÞ ðkÞ
HK and Bboot are the resampled dataset, as
used to evaluate the mean and variance of estimated
B-basis allowables. meanðSÞ  meanðBðkÞ Þ
NMðkÞ ¼ ð5Þ
Two metrics are used to evaluate the margin (weight stdðSÞ
penalty) and conservativeness. The weight penalty is where meanðSÞ and stdðSÞ stand for the mean and stand-
based on the assumption that the weight of the laminate ard deviation of 18 full-set strength samples at a given
is inversely proportional to the load it carries. configuration, and meanðBðkÞ Þ is the mean of either BðkÞ boot
Therefore, it is defined as the relative difference between or BðkÞ
HK from 1000 resample sets. NM
ðkÞ
is essentially a
the baseline and estimated B-basis. Since there are NR measure of B-basis allowables scaled by standard devi-
weight penalties, the weight penalty factor (WPF) is ations below the population mean values, which is sum-
defined as the mean of them, as ðkÞ
marized in Table 7. NMHK varied within [1.82, 5.36] and
!
1 XNR
B ð18Þ
 BðkÞ was more conservative than NMðkÞ boot , which was within
j
WPFðkÞ ¼ ð3Þ [0.91, 1.84]. With an increasing number of samples,
NR j¼1 Bð18Þ
NMðkÞ ðkÞ
boot increased gradually and Bboot became more con-
where j is the index of the resampled set. The positive servative. In contrast, NMðkÞHK decreased noticeably with
WPF indicates how much the estimate is conservative ðkÞ
increasing samples as BHK increased.
on average.
The estimated B-basis allowables BðkÞ from partial The tradeoff between WPF and CF for the estimated
samples is conservative if it is smaller than the base- B-basis allowables was examined next. The means of
line Bð18Þ . The conservativeness fraction (CF) is the WPF and CF from the 1000 sets of resamples are visua-
fraction of conservative BðkÞ from NR sets of partial lized in Figure 6. It is noticeable that for BðkÞ
boot , WPFs
resamples were non-positive at most configurations and CFs were
! less than 0.55. BðkÞ
boot became more conservative with an
1 X NR
Bð18Þ  BðkÞ increasing number of samples. For BðkÞ
ðkÞ
CF ¼ H
j
ð4Þ HK , in contrast,
NR j¼1 Bð18Þ WPFs were all positive between 0.07 and 0.18 for six
samples and decreased with an increasing number of
Zhang et al. 2829

Table 6. Estimated baseline B-basis allowables (unit: ksi).

Configuration 1 2 3 4 5 6 7 8

ðw=D,R45 Þ (3,0.2) (3,0.8) (4,0.2) (4,0.5) (6,0.2) (6,0.5) (6,0.8) (8,0.2)

ð18Þ
Bboot 82.17 34.53 84.12 60.81 91.38 66.26 46.71 95.25
ð18Þ
BHK 80.30 33.54 81.93 58.8 89.79 64.58 45.28 94.47

Figure 5. Estimated B-basis allowables from the 1000 sets of resamples. B-basis allowables are estimated from BHK and Bboot .
(a) k ¼ 6 samples, (b) k ¼ 8 samples and (c) k ¼ 10 samples.

samples. It means that BðkÞ

HK penalized the allowables for the potential price to pay because of using fewer sam-
small-size samples by increasing WPF. CFs were much ples to estimate a B-basis. In the case of configuration
better than that of BðkÞ boot . The overall trend of CF 2, for example, with k ¼ 6, it was very likely to have a
decreased slightly with an increasing number of samples. B-basis less than 30 ksi, but with k ¼ 10, it was nearly
BðkÞ
HK at configurations 2 and 4 were of interest, as impossible for a B-basis to be less than 30 ksi. For con-
they have the smallest CF. Empirical cumulative distri- figuration 4, a jump is observed around ECDF ¼ 0.55,
bution function (ECDF) of BðkÞ HK for the two configur- which is more pronounced when using 10 samples. This
ations are summarized in Figure 7, which is basically jump is due to the close-to-outlier sample at
2830 Journal of Composite Materials 54(21)

Table 7. Normalized mean values of the B-basis estimations from 1000 sets of the resampled dataset.

Configuration 1 2 3 4 5 6 7 8

ðw=D,R45 Þ (3,0.2) (3,0.8) (4,0.2) (4,0.5) (6,0.2) (6,0.5) (6,0.8) (8,0.2)

ðkÞ
Bboot k¼6 1.28 1.41 1.17 1.39 0.98 1.28 1.46 0.91
k¼8 1.38 1.64 1.25 1.61 1.04 1.4 1.68 0.95
k ¼ 10 1.42 1.84 1.29 1.83 1.09 1.5 1.83 0.97
ðkÞ
BHK k¼6 4.83 5.21 4.38 5.19 3.67 4.97 5.36 3.59
k¼8 3.74 4.41 3.33 4.33 2.68 3.92 4.56 2.52
k ¼ 10 3.14 3.93 2.69 3.91 2.11 3.28 4.21 1.82
Note: Mean values are subtracted from the mean strength of the 18 samples and then divided by the standard deviation of the 18 samples.

Figure 6. Weigh penalty factor versus conservativeness fraction from the 1000 sets of resamples. Each set of selected data has 6,8,
or 10 samples. BHK and Bboot are adopted for B-basis estimations. (a) Estimation from 6 samples, (b) Estimation from 8 samples and
(c) Estimation from 10 samples.
Zhang et al. 2831

Figure 7. Empirical cumulative distribution of the estimated B-basis allowables using Hanson-Koopmans method: (a) at configuration
2 and (b) at configuration 4.

configuration 4, which dominates the estimation of B-

basis allowables. In Figure 2, we can see a close-to-out-
lier sample far from the other samples, which results in
the jump. When compared to the close-to-outlier
sample at configurations 3 and 5, the sample at config-
uration 4 is at the lower end of the strength values and
leads to undesirable conservativeness.
A further study is performed to investigate the
trend of decreasing CF with an increasing number of
samples for BðkÞ ðkÞ
HK . BHK strongly depends on the param-
eters r and k as shown in equation (1). The parameters
for estimating BðkÞHK are given in Table 5 with different
numbers of samples. The magnitude of xðrÞ might
increase or decrease and does not show a clear trend.
If we assume the xðrÞ and xð1Þ remain the same, BðkÞ HK
increases with decreasing k. By assuming the ratio
xð1Þ =xðrÞ ¼ 0:9, an exponential function with a 0.9 base
is plotted in Figure 8. The estimations using 8 and 10 Figure 8. Plot of an exponential function with a 0.9 base.
samples are close to each other compared with that of six
samples. As expected, BðkÞ HK increased and became less
conservative with an increasing number of samples. desirable at configurations 2 and 4. The estimated
With increasing samples, xð1Þ , xðrÞ and k would be stabi- B-basis allowables could be improved further to
lized and BðkÞHK is expected to have a reduced variance. account for the unidentified uncertainties, such as too
small sample size, to meet the threshold. We proposed
the bootstrap-assisted HK method Bboot:HK to make the
Bootstrap-assisted HK method for estimated allowables with small-size samples compar-
small-size samples able to those with large-size samples.
A flowchart for calculating Bboot:HK is shown in
Bootstrap-assisted HK Method Table 8. Assuming rc test coupons are available from
The BHK was originally proposed for B-basis estima- tests, Nr sets of samples are bootstrapped from the rc
tions using a limited number of samples and proved coupons. rc samples are included for each bootstrapped
to be reliable when the sample size is larger than a dataset allowing replacement. The B-basis allowable is
critical value.10 However, the critical sample size calculated for each bootstrapped dataset and results in
varies with application and the BHK might not be con- Nr estimations, from which ECDF is developed.
servative for limited tests. As shown in ‘Estimating The modified B-basis allowables from Bboot:HK is
B-basis allowables of The OHT tests’ section, although defined using the lower 5th percentile of the ECDF.
BHK provided conservativeness in WPF, CF was less The number of bootstrapped estimations Nr is chosen
2832 Journal of Composite Materials 54(21)

Table 8. Flowchart for the Bootstrap-assisted Hanson-Koopmans method from replicates.

Step Procedure

1 Obtain failure strength samples from rc test coupons (data acquisition)

2 Bootstrap Nr sets of samples from rc coupons.
3 Calculate the B-basis allowables for each bootstrapped dataset using HK method and obtain Nr estimations of
B-basis allowables
4 Develop the ECDF of the Nr estimations of B-basis allowables using interpolation
5 Define the modified B-basis allowables using the lower 5th percentile of the ECDF

Figure 9. Weight penalty factor versus conservativeness penalty factor from the 1000 sets of replicates. Each set of selected data
has 6,8, or 10 replicates. BHK and Bboot:HK are adopted for B-basis estimations. (a) Estimation from 6, (b) Estimation from 8 and
(c) Estimation from 10.
Zhang et al. 2833

Figure 10. Empirical cumulative distribution of the estimated B-basis allowables at the configuration 2 (a) to (c) and configuration 4
(d) to (f). (a) Configuration 2 with k ¼ 6, (b) Configuration 2 with k ¼ 8, (c) Configuration 2 with k ¼ 10, (d) Configuration 4 with k ¼ 6,
(e) Configuration 4 with k ¼ 8 and (f) Configuration 4 with k ¼ 10.

to be large enough, so that it does not affect the quality 0.81. For the estimation of 10 samples, the low CF of
of the results (relatively stable estimation). Bð10Þ
HK was also observed at configurations 2 and 4. The
CF of Bð10Þ
boot:HK increased significantly at configuration 2
B-basis allowables of OHT tests using the but not at configuration 4. Based on the preliminary
check, we found that the Bð10Þboot:HK improved CF at the
bootstrap-assisted HK method configurations with least CF. The exception for the
Bboot:HK is compared with the BHK following the test Bð10Þ at configuration 4 is further discussed.
plan in ‘Test plan to evaluate the estimations of B-basis The Bð10Þ at configurations 2 and 4 are of critical
allowables’ section. The means of the WPF and the CF interest because they have the least CF. ECDFs of
from the 1000 sets of resamples are visualized in Bð10Þ for the two configurations are summarized in
Figure 9. For the estimation from six samples, CF of Figure 10. With six samples, ECDF of Bð6Þ boot:HK is con-
Bð6Þ ð6Þ
HK varies between 0.82 and 1.00. CF of Bboot:HK varies servative than that of Bð6Þ HK . In contrast, the ECDFs
between 0.86 and 1.00, which is slightly more conser- showed a clear jump with 10 samples. For configur-
vative. CF at configurations 2 and 4 increased notice- ation 4, the jump is around ECDF¼0.55. This jump
ably from 0.85 to 0.95 with increased WPF around is due to the close-to-outlier sample, which dominates
ðkÞ
0.05. For the estimation from eight samples, CF of the estimation of B-basis allowables. Bboot:HK seemed
Bð8Þ ð8Þ
HK varies between 0.59 and 1.00. The CF of Bboot:HK invalid when a close-to-outlier sample dominates the
at configuration 4 increased significantly from 0.59 to B-basis estimation in the lower end.
2834 Journal of Composite Materials 54(21)

Conclusions
ORCID iD
Composite materials have been routinely used in load- Nam H Kim https://orcid.org/0000-0002-0221-9749
bearing structures due to their outstanding capability to
be tailored to specific load paths and conditions, result-
ing in weight efficient designs. Reliable stress limits, References
called design allowables, are of critical interest to
1. Toft HS, Branner K, Mishnaevsky L Jr, et al.
designers to balance safety, performance, and economic
Uncertainty modelling and code calibration for compos-
value. The properties of composite laminates usually ite materials. J Compos Mater 2013; 47: 1729–1747.
suffer from significant variation due to the complexity 2. Zhang Y, Schutte J, Meeker J, et al. Predicting B-basis
and inherent variability of the manufacturing process. allowable at untested points from experiments and simu-
Estimating the design allowables is challenging because lations of plates with holes. In: Proceedings of the 12th
of complicated failure mechanisms and the limited world congress on structural and multidisciplinary opti-
number of samples. mization, Braunschweig, Germany, www.researchgate.
This paper examines the estimation of B-basis allow- net/publication/318909364. (accessed 20 January 2020).
ables from a limited number of samples demonstrated 3. Vallmajó O, Cózar I, Furtado C, et al. Virtual calculation
in OHT strength testing experiments. The conservative- of the B-value allowables of notched composite lamin-
ness and weight penalty (margin of B-basis allowables) ates. Compos Struct 2019; 212: 11–21.
are evaluated for the bootstrap confidence interval 4. Nilakantan G, Obaid AA, Keefe M, et al. Experimental
ðBboot Þ and HK ðBHK Þ methods. The experiments are evaluation and statistical characterization of the strength
and strain energy density distribution of Kevlar KM2
thoroughly investigated by examining the effects of out-
yarns: exploring length-scale and weaving effects.
liers, the goodness-of-fit on different assumed statistical
J Compos Mater 2011; 45: 1749–1769.
distributions, and data visualization. The B-basis 5. Satheesh R, Narayana Naik G and Ganguli R.
allowable estimation using 18 samples (large-size sam- Conservative design optimization of laminated composite
ples) is used as the baseline. Partial subsets of samples structures using genetic algorithms and multiple failure
(limited number of samples) are used for evaluating criteria. J Compos Mater 2010; 44: 369–387.
different ways of calculating the B-basis allowables 6. Picheny V, Kim NH and Haftka RT. Application of
and compared with the baseline. bootstrap method in conservative estimation of reliability
Based on the study, it was observed that BHK was more with limited samples. Struct Multidisc Optimiz 2010; 41:
conservative than Bboot . The former penalized B-basis 205–217.
allowables for small-size samples and incorporated the 7. Bae S, Kim NH and Jang S. Reliability-based design opti-
effect of sample size better than the latter. It was also mization under sampling uncertainty: shifting design
observed that BHK was sensitive to outliers which domi- versus shaping uncertainty. Struct Multidisc Optimiz
nated the estimations of B-basis allowables. In this paper, 2018; 57: 1845–1855.
8. Bae S, Kim NH and Jang S-G. System reliability-based
the bootstrap-assisted HK method ðBboot:HK Þ was pro-
design optimization using trade-off between uncertainty
posed to enhance the reliability of B-basis allowables for
reduction and design shift. J Mech Des 2019; 141: 041403.
small-size samples. The proposed method was especially 9. Mrazova M. Advanced composite materials of the future
beneficial when only a limited number of samples are in aerospace industry. Incas Bull 2013; 5: 139.
available, yielding the least amount of conservativeness 10. Defense US Do. Composite materials handbook, volume
among the methods evaluated. 1 – polymer matrix composites guidelines for character-
ization of structural materials, http://app.knovel.com/
hotlink/toc/id:kpMHMILH5M/military-handbook-mil-
Acknowledgements 3/military-handbook-mil-3 (2002, accessed 20 January
OHT test data were furnished by National Institute for 2020).
Aviation Research (NIAR) at the Wichita State University. 11. Young DS. Tolerance: an R package for estimating tol-
erance intervals. J Stat Softw 2010; 36: 1–39.
Declaration of Conflicting Interests 12. Chakravarti IM and Laha RG. Handbook of methods of
applied statistics. New Jersey: John Wiley & Sons, 1967.
The author(s) declared no potential conflicts of interest with 13. Rust SW, Todt FR, Harris B, et al. Statistical methods for
respect to the research, authorship, and/or publication of this
calculating material allowables for MIL-HDBK-17. Test
article.
methods and design allowables for fibrous composites
(2nd volume). West Conshohocken, PA: ASTM
Funding International, 1989.
The author(s) disclosed receipt of the following financial sup- 14. Handbook M. MIL-HDBK-17-1F: composite materials
port for the research, authorship, and/or publication of this handbook, volume 1-polymer matrix composites guidelines
article: This research was supported by DARPA contract for characterization of structural materials. Virginia: US
W911QX-13-C-0137. Department of Defense, 2002.
Zhang et al. 2835

15. Hanson DL and Koopmans L. Tolerance limits for the 31. Zhang Y, Haftka RT, Kim N-H, et al. Allocation of
class of distributions with increasing hazard rates. Ann samples between exploration and replication for open-
Math Stat 1964; 35: 1561–1570. hole-tension test. In: Proceedings of the American
16. Bhachu KS. Effects of measured data uncertainty on Society of Composites-30th Technical Conference, East
damage tolerant design of aircraft structures. Florida: Lansing, MI, 28–30 September 2015.
University of Florida, 2015. 32. Osborne JW and Overbay A. The power of outliers (and
17. Cross RJ, Makeev A and Armanios E. A comparison of why researchers should always check for them). Pract
predictions from probabilistic crack growth models Assess Res Eval 2004; 9: 1–12.
inferred from Virkler’s data. J ASTM Int 2006; 3: 1–11. 33. Stefansky W. Rejecting outliers by maximum normed
18. Bigerelle M and Iost A. Bootstrap analysis of FCGR, residual. Ann Math Stat 1971; 42: 35–45.
application to the Paris relationship and to lifetime pre- 34. Mooney CZ and Duval RD. Bootstrapping: a nonpara-
diction. Int J Fatigue 1999; 21: 299–307. metric approach to statistical inference. Thousand Oaks:
19. Bigerelle M, Najjar D, Fournier B, et al. Application of Sage, 1993.
lambda distributions and bootstrap analysis to the pre- 35. Efron B. The jackknife, the bootstrap and other resampling
diction of fatigue lifetime and confidence intervals. Int J plans. New Delhi: SIAM, 1982.
Fatigue 2006; 28: 223–236. 36. Edwards DJ, Guess FM and Young TM. Improved esti-
20. Bhachu KS, Haftka RT and Kim NH. Comparison of mation of the lower percentiles of material properties.
methods for calculating B-basis crack growth life using Wood Sci Technol 2011; 45: 533–546.
limited tests. AIAA J 2016; 54: 1287–1298. 37. Lee SM and Young GA. Asymptotic iterated bootstrap
21. Romero V, Swiler L and Urbina A. An initial comparison confidence intervals. Ann Stat 1995; 23: 1301–1330.
of methods for representing and aggregating experimen- 38. Lee SM and Young GA. The effect of Monte Carlo
tal uncertainties involving sparse data. In: Proceedings of approximation on coverage error of double-bootstrap
the 13th AIAA non-deterministic approaches conference, confidence intervals. J R Stat Soc: Series B (Statistical
Denver, CO, 4–7 April 2011. Methodology) 1999; 61: 353–366.
22. Romero V, Swiler L, Urbina A, et al. A comparison of 39. Langford E. Quartiles in elementary statistics. J Stat
methods for representing sparsely sampled random quan- Educ 2006; 14: 1–27.
tities. Sandia National Laboratories report SAND2013- 40. DiCiccio TJ and Efron B. Bootstrap confidence intervals.
4561, September, 2013. Stat Sci 1996; 11: 189–212.
23. Hinton M and Kaddour A. Triaxial test results for fibre- 41. Zhang Y, Schutte JF, Seneviratne WP, et al. Sampling by
reinforced composites: the second world-wide failure exer- exploration and replication for estimating experimental
cise benchmark data. J Compos Mater 2013; 47: 653–678. strength of composite structures. AIAA J 2017; 55:
24. Hinton MJ, Kaddour AS and Soden PD. Failure criteria 3594–3602.
in fibre reinforced polymer composites: the world-wide fail-
ure exercise. Amsterdam: Elsevier, 2004.
25. Tomblin J and Seneviratne W. Laminate statistical allowable
generation for fiber-reinforced composite materials: lamina Appendix 1. Outlier detection and
variability method. USA: Office of Aviation Research and identification of distributions in the
Development, Federal Aviation Administration, 2009. open-hole-tension tests
26. Carlsson LA, Adams DF and Pipes RB. Experimental
characterization of advanced composite materials. Boca A.1 Outlier detection
Raton: CRC Press, 2014.
The outliers of samples have a significant impact on the
27. ASTM International ASTM D5766/D5766M-11.
Standard test method for open-hole tensile strength of poly-
statistics.32 The maximum normed residual (MNR)
mer matrix composite laminates. West Conshohocken, test33 was recommended by CMH-1710 to detect out-
PA: ASTM International, 2011. liers. A sample is identified as an outlier if the absolute
28. Hallett SR, Green BG, Jiang W-G, et al. The open hole deviation from the sample mean is too large. This pro-
tensile test: a challenge for virtual testing of composites. cedure assumes that nominal samples follow a normal
Int J Fract 2009; 158: 169–181. population. The MNR is defined as
29. Tomblin J, Sherraden J, Seneviratne W, et al. Advanced
 
general aviation transport experiments. A–Basis and B– xi  meanðxÞ
Basis design allowables for epoxy–based prepreg Toray MNR ¼ max ð6Þ
i stdðxÞ
T700GC-12K-31E/# 2510 unidirectional tape. Wichita:
National Institute for Aviation Research Wichita State
where xi denotes the experimental strength of rc test
University, 2002.
30. Zhang Y, Meeker J, Schutte J, et al. On approaches to
coupons. meanðxÞ and stdðxÞ are the mean and stan-
combine experimental strength and simulation with dard deviation of samples, respectively. The MNR is
application to open-hole-tension configuration. In: compared to a critical value depending on the sample
Proceedings of the American Society for composites: size with a specific significance level. The critical MNR
thirty-first technical conference, Williamsburg, VA, 19– for 18 samples is 2.65 using a significance level of 0.05
22 September 2016. (95% confidence). The MNR for the OHT tests is
2836 Journal of Composite Materials 54(21)

Table 9. Value of the maximum normed residual (MNR) tests for the detection of an outlier.

Points 1 2 3 4 5 6 7 8

ðw=D, R45 Þ (3,0.2) (3,0.8) (4,0.2) (4,0.5) (6,0.2) (6,0.5) (6,0.8) (8,0.2)
MNR 1.79 2.35 1.86 2.52 2.52 1.69 2.13 2.27
Note: The critical value to detect the outlier is 2.65.

Table 10. The p-value of the KS tests against Normal, uniform and Weibull distributions.

Points 1 2 3 4 5 6 7 8

ðw=D, R45 Þ (3,0.2) (3,0.8) (4,0.2) (4,0.5) (6,0.2) (6,0.5) (6,0.8) (8,0.2)
Normal 0.79 0.94 0.96 0.91 0.60 0.97 0.21 0.39
Uniform 0.63 0.16 0.43 0.02 0.01 0.99 0.01 0.01
Weibull 0.91 0.97 0.83 0.71 0.48 0.87 0.41 0.36
Best fit Weibull Weibull Normal Normal Normal Uniform Weibull Normal

provided in Table 9. The largest two MNRs are 2.52 the probability that samples do not reject the hypothet-
from configurations 2 and 4, which are smaller than the ical distribution. A high p-value denotes the high prob-
critical value 2.65. Therefore, no outliers are detected ability that samples are from the hypothetical
for the OHT tests. distribution. The OHT tests are examined against
Normal, uniform and Weibull distributions. The
p-values are summarized in Table 10. The Normal dis-
A.2. Kolmogorov–Smirnov tests tribution fits best for 4 configurations, uniform distri-
Kolmogorov–Smirnov (KS) test was adopted to iden- bution fits best for 1 configuration and Weibull
tify the distribution of OHT tests. The KS test is a non- distribution fits best for 3 configurations. No significant
parametric test to quantify the goodness-of-fit between indication favors a single distribution while assuming
a given probability distribution and the empirical dis- the experiments follow the same type of distributions at
tribution of samples. The p-value of KS test indicates different configurations.