DM404 – Design Of Experiments

Dr Colin Andrews
Recommended Reading
 World Class Quality: Using Design Of Experiments To Make It Happen,
KeKi Bhote and Adi Bhote, AMACOM, 2000
 D. M. Grove and T. P. Davis, Engineering Quality and Experimental Design,
Longman Scientific and Technical, 1992
 Design and Analysis of Experiments, 5th Edition, Douglas C Montgomery,
John Wiley and Sons, 2001.
 Design of Experiments for Engineers and Scientists, Jiju Antony,
Butterworth Heinemann, 2003
 Experimental Quality, Jiju Antony and Mike Kaye, Kluwer Academic
Publishers, 2000
What is an Experiment?

 An operation carried out under controlled conditions to identify an unknown

effect or establish a hypothesis or to illustrate a known law
 In an engineering environment, an experiment is a series of trials which
produce quantifiable outcomes
 Experiments are generally performed to explore, estimate or confirm
 Exploration – data gathering
 Estimation – determining the effects of factors or process variables or
design parameters
 Confirmation – verification of results
 Objectives of an experiment

1. Determining which variables (factors) (x) are most

influential on the response (y)
2. Determining where to set the influential variable (x’s) so that
response is almost always near the desired value (accuracy).
3. Determining where to set the influential variables (x’s) so that
variability in response is small (precision).
4. Determining where to set the influential variables (x’s) so that
the effects of the uncontrollable variables (z’s) are minimized
 Factor Effects
Input factor A affects the average Input factor B affects the standard
(A1A2) deviation (B1B2)
Y1 Y2 Y1

Y2
3 5

-5 -4 -3 -2 -1 0 1 4
-4 -3 -2 -1 0 1 2 3 4 -5 -4 -3 -2 -1 0 1 2 3 4 5
2

Y1, Y2 are outputs

(responses)
Input factor C affects the Input factor D
average and standard has no effect
deviation (C1C2) Y2 Y1=Y2
(D1D2)

Y1

© University of Strathclyde 2019

-6 -5 -4 -3 -2 -1 0 1 2 3 4
-6 -5 -4 -3 -2 -1 0 1 2 3 4 -6 -5 -4 -3 -2 -1 0 1 2 3 4
 WHAT IS INVOLVED IN AN EXPERIMENTATION?

 HYPOTHESIS – An assumption that motivates the experiment

 EXPERIMENT – A series of tests conducted to investigate the hypothesis

 ANALYSIS – Involves the nature of data and performing statistical analysis of the
collected data from the experiment
 INTERPRETATION – To understand the results of the statistical analysis

 CONCLUSION – To determine whether or not the hypothesis set at the beginning

is true or false
ONE-FACTOR-AT-A-TIME (OFAT) APPROACH TO
EXPERIMENTATION

OFAT approach is still popular in many companies due to its simplicity

In this approach, the experimenter varies one process variable (factor) at a time, keeping all the other process
variables fixed at a specific set of conditions

Consider a simple example in which there are two factors; temperature (A) and time (B) which affects the yield of
a chemical reactor (assume we need to improve the yield!)

• Temperature is tested at 4 levels and time at 3 levels

A1 = 700 C, A2 = 750 C, A3 = 800 C, A4 = 850 C

B1 = 40 min., B2 = 44 min., B3 = 48 min.

ONE-FACTOR-AT-A-TIME (OFAT) APPROACH TO
EXPERIMENTATION

• Assume the current settings of temperature and time are A 1 and B1

• In the first experiment, both temperature (A) and time (B) are kept at levels 1.
• In the second experiment, the level of B is changed to level 2, keeping the level of A fixed at level 1.
• In the third experiment, the level of B is changed to level 3, keeping the level of A fixed at level 1. The
results are shown below:
TEMPERATURE TIME YIELD (%) DIFFERENCE
A1 B1 91.5 ----

A1 B2 92.2 +0.7

A1 B3 92.8 +1.3
ONE-FACTOR-AT-A-TIME (OFAT) APPROACH TO
EXPERIMENTATION

• We conclude that B3 is the optimal level of factor B, and the improvement in yield from the current
settings is +1.3%
• Similarly, the levels of A have changed from A 1 to A2 , A2 to A3 and finally from A3 to A4, keeping the level
of B fixed (B3). It is concluded that A3 is the optimal level of factor A. The improvement of yield from the
current settings is +0.3.

TEMPERATURE TIME YIELD (%) DIFFERENCE

A1 B3 92.8 ----

A2 B3 93.1 +0.3

A3 B3 92.6 -0.2

A4 B3 92.3 -0.5
FROM THE RESULTS OF THE ABOVE EXPERIMENT, CAN WE
CONCLUDE THAT THE OPTIMAL LEVELS OF A AND B ARE A2B3 ?
ONE-FACTOR-AT-A-TIME (OFAT) APPROACH
TO EXPERIMENTATION

If we vary all the factors simultaneously at their respective levels, we will come up
with the following results
A1 A2 A3 A4

B1 91.5 91.9 93.0 92.8

B2 92.2 92.9 93.8 93.6

B3 92.8 93.1 92.6 92.3

Optimum setting Real Optimum setting

identified by OVAT
ONE-FACTOR-AT-A-TIME (OFAT) APPROACH TO
EXPERIMENTATION

• In this case the optimal values for A and B is A3B2 and not A2B3
• This illustrates that One-Factor-At-A-Time experiments can lead to
wrong results when predicting the optimum values for variables,
and in estimating the response value at the optimum setting.
• Success in an OFAT approach to experimentation relies heavily on
experience, intuition, guesswork and luck.
ONE-FACTOR-AT-A-TIME (OFAT) APPROACH TO
EXPERIMENTATION

94

93
B1 The chart lines are not parallel. This
Yield(%)

92 B2 indicates that the effect of A is different

Time
at different levels of B. In other words
B3 there is an interaction between A and B
91 (represented as AxB)

90
A1 A2 A3 A4

Temperature
Independent, Dependent, Constants or Controlled
Variables in an Experiment

Constants: Factors that are kept the same and not allowed to change.
Independent Variables: changes in the experiment that are controlled by the
experimenter
Dependent variables: changes that occur as a result of independent variables

It is important to control all factors instead of one

factors at a time (OFAT) to be able to interpret data
What is Design of Experiments
(DoE)?

Design of Experiments is a series of

purposeful changes to the process inputs
(factors) in order to observe the
corresponding changes in the outputs
(responses).

The right tool

at the right cost!!!
© University of Strathclyde 2019
DoE Overview
(C) controlled (constant) variables,
designated to be held constant during an
experiment (e.g. SOP)

(X) key process (Y) output (response)

variables, varied variables, measured
PROCESS
during an to evaluate process
experiment performance

(N) uncontrolled (noise) variables,

cannot be held constantThe right
during antool
experiment, and/or later on;
at the
(we like to make the process right cost!!!
insensitive to those variables, if possible)

© University of Strathclyde 2019

DoE Rationale

• DoE is a very powerful technique to study the effect of multiple variables

simultaneously
• Different types of experiments can be performed depending upon the objective
of the experiment and the nature of the problem at hand
• The purpose of scientific experiments is to understand the process behaviour by
establishing the relationship between the CTQs and a set of process
parameters which may influence these

The right tool

at the right cost!!!
© University of Strathclyde 2019
DoE Objectives

• To build empirical models which approximate the true relationship between the
output(s) (responses) and inputs (factors) in order to:
– Determine which factors (inputs) have an effect on response average, variability; and
those which not.
– Perform a sensitivity analysis which can be used for tolerance evaluations
– Gain maximum amount of information with minimum resources
– Validate results
– Optimise the process (improve performance characteristics, reduce costs & time
associated with product development & production)

© University of Strathclyde 2019

DoE vs. the “One Factor at a Time”
(OFAT) Approach
• The main advantages of DOE are:
• Efficiency (the time and resources to conduct experiments)
• Evaluates interaction effects between the factors
• OFAT approach is unable to evaluate interaction effects
• OFAT can miss optimum settings
• OFAT is expensive & time consuming

Note: DoE should only be used after the application of

IPO / PF / CE / CNX / SOP / FMEA

© University of Strathclyde 2019

Full factorial Design of Experiments

• All possible combinations of Levels for all Factors lead to number of experiments
equal to Lk
– For 2 levels per factor and the number of factors being k, 2k runs of the experiment are needed.

• Examines all types of interactions between factors

• It is only practical for small numbers of factors to be considered in an experiment
– If we have 3 levels and 4 factors to be considered, how many experiments would be needed in full
factorial designs of experiment?

– 34 =81
Fractional factorial designs of
experiments
Visualising – number of experiments in a Full
Factorial (23) experiment (three factors at two levels)

(low) Height (high)

)
C igh
( h
th
id
) W B
w
l( o
(low) Length (high)

A
Fractional Factorial (23-1)
Run A B C

1 1(-) 1(-) 1(-) Run 3

2 1(-) 1(-) 2(+)

3 1(-) 2(+) 2(+) Run 5

4 1(-) 2(+) 1(-)

5 2(+) 1(-) 2(+)

6 2(+) 1(-) 1(-) C

Run 7
7 2(+) 2(+) 1(-)
B
8 2(+) 2(+) 2(+)

Run 1 A
 Fractional Factorial design

1. Minimising required number of experiments - often there is lack of

time, resources and budget to perform a full factorial experiment
(always?). E.g. if k = 11 (number of factors), and each factor is
tested at 2 levels:
– “One Factor at a Time” will have 12 runs with 17 replications,
which gives 204 tests!
– Screening Designed Experiment will have 12 runs with 4
replications, which gives 48 tests.
2. Most widely used in industry
3. Can provide information on main effects and second-order
interactions only (AB, AC, BC …)
4. Assumption that higher order interactions (>= third order) are not
important (e.g. ABC, ACDE …)
Fractional Factorial design

Representation of the fractional factorial design for two levels experiments: 2(k-p)
k is the number of factors to be considered in an experiment
p is reducing experimental coefficient; 1/2p gives the fraction of the full factorial experiment.

For example:
2(5-2) means ¼ of a 25 full factorial design: 25 full factorial design = 32 experiments; 2(5-2) fractional
factorial design = 32/4 = 8 experiments.

What does 2(6-3) mean?

How many experiments (runs) does it indicate ?
How many factors will be analysed?
Fractional Factorial design
A general guide for two-level experiments

III Estimate main effects, but these may be confounded with two-factor interactions
Estimate main effects unconfounded by two-factor interactions
IV
Estimate two-factor interaction effects, but these may be confounded with other two-factor interactions
Estimate main effects unconfounded by three-factor (or less) interactions
V Estimate two-factor interaction effects unconfounded by two-factor interactions
Estimate three-factor interaction effects, but these may be confounded with other two-factor interactions
Strategies for Experimentation

• Screening - to separate vital few and trivial many

• Modelling - to be able to predict the response (average and standard deviation)

for a given combination of factors
• E.g. Full factorial, Fractional factorials and Screening experiments

But FIRST – carry out process improvement work, SPC, Capability Study, PFMEA, SOPs etc.)

© University of Strathclyde 2019

DoE Matrix
• The main parameters of DoE are:
• Factor (input to a process which can be manipulated during exp.)
• Level (a setting or testing value of a factor)
• e.g. 2 levels (low, high) or 3 levels (low, medium, high)
• Pre-existing Experiment Designs exist e.g. Taguchi Matrices / Orthogonal Arrays (see Orthogonal Arrays (york.ac.uk) )
Factors Responses Statistics
Run # X1 X2 X3 X4 Y1 Y2 Y3 Y4 Y bar S
1
Average &
2 Design Matrix - Multiple replicates -
standard
3 Combinations of repeated observations
deviation of
4 factor settings for of the response
the
5 each experimental measurement for each
responses
* run run
for each run
*

© University of Strathclyde 2019

 Perfect Process Knowledge
Perfect Gap can be
Knowledge quantified
in terms of
statistical
Confirm Results
confidence
Process Knowledge

Modelling with DOE

Screening with DOE

SOP/FMEA/CNC/SPC

Noise Cloud represents the limit of current knowledge

© University of Strathclyde 2022

Example: Application of DoE in a Non-Manufacturing
Environment

A large company was having a problem with receivables. The average age of
receivables due was 200 days after delivery of material. The company had $130
million that was 30 days or older after receipt by the customer. The impact of this delay
was significant and was causing a cash flow problem in the company.
• What is the cause of aging of receivables?
• How can we optimise the process of billing?
• How can we reduce the billing time?

© University of Strathclyde 2019

Reducing the Billing Time Alternatives

There were several options available that may further reduce billing time.
• Bill directly on the invoice
• Automate the billing and invoicing system
• Provide follow-up to the customers by management at 30 to 45 days by telephone or in writing
• Contract out the billing department to a professional billing activity

These options lend themselves to evaluation using a designed

experiment.

© University of Strathclyde 2019

Billing Factors and Levels
• Factor A – Billing
• -1 (low level) – directly on the invoice with the shipment
• +1 (high level) – billing from the billing department mailed separately from the shipment

• Factor B – Automation
• -1 (low level) – Automate the complete billing process with all billing generated automatically on shipment
• +1 (high level) – Maintain the current system in which the generation of billing is automated but the bills
and invoices are transmitted and routed in hard copy

© University of Strathclyde 2019

Billing Factors and Levels
• Factor C – Follow up
• -1 (low level) – Follow up by letter at 45 and 60 days
• +1 (high level) – Follow up by telephone at 45 and 60 days

• Factor D – Contract
• -1 (low level) – contract out the billing and follow-up
• +1 (high level) – keep the billing and follow-up in house

© University of Strathclyde 2019

Coded Design Matrix (Taguchi Matrix L8)

Trial Billing Automation Follow-up Contract

(A) (B) (C) (D)
1 -1 -1 -1 -1
2 1 -1 -1 1
3 -1 1 -1 1
4 1 1 -1 -1
5 -1 -1 1 1
6 1 -1 1 -1
7 -1 1 1 -1
8 1 1 1 1

FYI: This is a 2(4-1) design

© University of Strathclyde 2019
Decoded Design Matrix

Trial Billing Automation Follow-up Contract

(A) (B) (C ) (D)
1 Invoice Complete Letter Contract
2 Separate Complete Letter In House
3 Invoice Partial Letter In House
4 Separate Partial Letter Contract
5 Invoice Complete Telephone In House
6 Separate Complete Telephone Contract
7 Invoice Partial Telephone Contract
8 Separate Partial Telephone In House

© University of Strathclyde 2019

Results of the Experiment

Run A B C D y1 y2 y3 y4 y5 y6 ͞y R

1 -1 -1 -1 -1 49 46 56 59 47 44 50 15
2 1 -1 -1 1 79 84 86 78 86 91 84 13
3 -1 1 -1 1 51 55 64 53 63 61 58 13
4 1 1 -1 -1 93 96 81 79 80 88 86 17
5 -1 -1 1 1 47 46 44 51 40 49 46 11
6 1 -1 1 -1 59 61 69 62 54 66 62 15
7 -1 1 1 -1 46 52 49 55 59 42 51 17
8 1 1 1 1 62 61 64 68 69 60 64 9

y = number of days since billing (age of receivables)

© University of Strathclyde 2019
Interpreting DoE Results in Minitab

Main Effects Plot (data means) for Age of receivables

A B

70

Mean of Age of receivables

60

50
Invoice Separate Complete Partial
C D

70

60

50
Letter Telephone Contract In House

© University of Strathclyde 2019

Interpreting DoE Results in Minitab
Pareto Chart of the Standardized Effects
(response is Age of receivables, Alpha = .05)
2.02
Factor Name
A A
A B B
C C
C D D

AC
Term

AB

AD

0 2 4 6 8 10 12 14 16
Standardized Effect

© University of Strathclyde 2019

Interpreting DoE Results in Minitab
Normal Probability Plot of the Standardized Effects
(response is Age of receivables, Alpha = .05)
99
Effect Type
Not Significant
95 Significant

90 A F actor Name
A A
80 B B
B C C
70 D D
Percent

60
50
40
30
AC
20

10 C
5

1
-10 -5 0 5 10 15
Standardized Effect

© University of Strathclyde 2019

 DoE Conclusions
• Factors A, B and C have some significant impact on age of receivables (see
Pareto and Normal plot) – 5% significance level
• Moreover, interaction between A and C is also significant
• How do we interpret an interaction effect?
• Non-parallel lines imply that there is an interaction between two factors
• Interaction occurs when the effect of one factor at different levels of the other
factor is not the same.

© University of Strathclyde 2019

Interpreting Interactions in Minitab
Interaction Plot (data means) for Age of receivables
90 A
I nvoice
Separate

Mean 80

70

60

50

Letter Telephone
C

© University of Strathclyde 2019

DoE Process

Identify the Define your Plan your Conduct your Analyse your Implementation
Opportunity experiment experiment experiment experiment Plan

© University of Strathclyde 2019