18-May-20

Design of Experiments
Dr. Rohit Vishal Kumar
Associate Professor,
International Management Institute Bhubaneswar

Introduction
 What is an Experiment?
 A process of manipulating one or more independent variables and
measuring their effect on one or more dependent variables.
 A test under controlled conditions that is made to demonstrate a known
truth, examine the validity of a hypothesis, or determine the efficacy of
something previously untried.

 What is experimentation?
 Testing of an idea
 A set of actions /observations. Performed to verify a hypothesis or to
research causal relationship between phenomena.

1
 18-May-20

Why Experiment?
 To infer about causal relationships

 Concept of Causality:
 Refers to the Cause and Effect Relationship between variables
 Helps in measuring the predictability between the two events
 Is probabilistic in nature
 Assume that other variables do not have a significant impact
 It can never be proved deterministically
 We can only infer cause and effect relationship

Some Concepts
 Independent Variables:
 Are variables manipulated by the researcher
 Are assumed to be the cause of some phenomenon.
 Dependent Variables:
 Are variables which change because of change in independent variables
 Are caused by independent variables
 Experimental Designs
 Is the design of all information-gathering exercises where variation is
present, whether under the full control of the experimenter or not.
 Often the experimenter is interested in the effect of some process or
intervention ("treatment") on some objects ("experimental units")

2
 18-May-20

Validity of an Experiment

What is Validity?
 The ability of an experiment to give correct results
 Two types of Validity
 Internal Validity
 External Validity
 Not always possible to control validity
 Researchers have to compromise on validity

3
 18-May-20

Internal Validity
 An experiment is said to be internally valid when the observed
effects (outcome) are solely due to experimental treatment
 Eight distinct Levels
 History
 Refers to wrong notions or beliefs
 Maturation
 Refers to the learning ability of animate test subjects
 Testing
 Refers to the impact of previously conducted experiments
 Instrumentation
 Refers to correct usage of recording and measuring instruments
 Selection
 Refers to correct selection of sample size
 Mortality
 Refers to inability / unwillingness to provide responses
 Regression
 Refers to the ability of giving “on-the-average” response
 Selection Interaction
 Refers to the effects that occur between investigator and investigated

External Validity
 Refers to the “Conclusion Drawing Ability” of the experiment
 Four different components:
 Reactive Effect of Testing
 Ability to learn and anticipate questions and responses and modify behaviour
accordingly
 Reactive Effect of Experimental Situations
 Ability to modify behaviour according to the surroundings
 Reactive Effect of History of treatment
 Inability to control the effect of exogenous variables
 Reactive Effect of Selection Treatment Interaction
 Ability to infer from the sample to the population

4
 18-May-20

Experimental Design Process

Steps in Detail (1/3)

 Determine the Objectives
 Objectives are too many to cover in a single study
 The same objective may be achieved in different ways
 Once objectives are decided we develop the experimentation
strategy
 Plan the action that needs to be taken on success or failure of the
objectives

 Define the Measure of Success

 How to know when we have achieved the objectives?
 Measures must be measurable, preferably quantitative in nature
 Avoid overcrowding of responses or respondents

10

5
 18-May-20

Steps in Detail (2/3)

 Verify Feasibility
 Identify the minimum sample size and trials required
 Cost feasibility also needs to be established
 Also depends on Sample Size and Trials

 Design the Experiment

 Identify the controls to be varied
 Identify the correct experimental design
 Determine whether the number of designs or trials is too large
 If need be shift through various designs to identify the most feasable

 Run the Experiment

Apply the KISS Principle

11

Steps in Detail (3/3)

 Collect and Analyse Data
 Identify suspect data – outlier and extreme values
 Analyse data as per the experimental design
 May wish to analyse critical data first

 Determine and Verify the Responses

 Identify the significance of the outcomes
 The outcomes – either singly or jointly – allows the researcher to
predict the behaviour of cause and effect relationship
 Allows for optimization of key variables

 Act on the Results

12

6
 18-May-20

Experimental Designs - Types

13

Types of Experimental Designs

 Pre-Experimental Designs:
 Do not employ randomization process to control for exogenous factors
 One-Shot Case study, One Group Pre-Test-Post-Test, Static Group Design
 True Experimental Designs:
 Employs full randomization techniques to control for exogenous factors
 Pretest Posttest Control Group, Posttest Only Control Group, Solomon Four
Group Design
 Quasi-Experimental Designs:
 Full randomization techniques are not applicable
 Time Series, Multiple Time Series
 Statistical Experimental Designs:
 Allows for statistical control and analysis of the impact of external variables
 Designed on the basis of their characteristics and usage
 Randomized Block Design, Latin Square Design, Factorial Designs

14

7
 18-May-20

Symbols and Terminology

 X : The exposure of a group to an independent variable,
treatment or event; the effect of which are to be
determined
 O : The process of observation or measurement of the
dependent variable on the test unit
 R : The random assignment of test units to separate
treatment
 EG : Experimental Group
 CG : Control Group [No Experimentation Performed]

 In addition:
 Movement from left to right indicate movement through time
 Horizontal alignment of symbols implies that all those symbols refer to a
specific treatment group
 Vertical alignment of symbols implies that the activities or events that occur
simultaneously

15

Pre-Experimental Design (1/3)

 One Shot Case Study:
X O1

 Basic Procedure:
 Measurement of an identified "outcome" after a treatment has been
implemented.
 Disadvantages:
 No measures available for comparison pre and post study
 Lack of control of extraneous variables

16

8
 18-May-20

Pre-Experimental Design (2/3)

 One Group Pretest Post Test:
O1 X O2

 Basic Procedure:
 Measurement of an outcome both pre and post exposure
 Disadvantages:
 No Control Group
 Maturation Effect
 Effect of the treatment on the study cannot be isolated

17

Pre-Experimental Design (3/3)

 Static Group Design
EG: X O1
CG: O2

 Basic Procedure:
 Two Groups: Experimental Group (EG) and Control Group (CG)
 EG exposed to the experiment, CG not exposed
 (O2 – O1) gives the impact of X
 Disadvantages:
 Lack of Randomization
 Treatment, Selection Bias may be present

18

9
 18-May-20

True Experimental Designs (1/3)

 Posttest Only Control Group
EG: R X O1
CG: R O2

 Basic Procedure:
 Random Allocation of Subjects to EG and CG
 Effect of Treatment is obtained by (O2 – O1)
 Simple to implement
 Disadvantages:
 No pre-test observations for comparison
 Sensitive to Mortality

19

True Experimental Designs (2/3)

 Pretest-Posttest Control Group
EG: R O1 X O2
CG: R O3 O4

 Basic Procedure:
 Random Allocation of Subjects to EG and CG
 Effect of Treatment is obtained by (O2 – O1) – (O4 – O3)
 Requires two groups and two measurement
 Disadvantages:
 Pretest measurement can impact subsequent responses
 More time and effort required

20

10
 18-May-20

True Experimental Designs (3/3)

 Solomon 4-Group-6 Design
EG: R O1 X O2
CG: R O3 O4
EG: R X O5
CG: R O6

 Basic Procedure:
 4 groups and 6 observations
 Combination of previous two designs
 Disadvantages:
 Difficult to implement and monitor

21

Quasi Experimental Designs (1/2)

 Simple Time Series
O1 O2 O3 O4 X O5 O6 O7 O8

 Basic Procedure
 A series of periodic measurement
 Treatment occurs naturally or is induced by experimenter
 Disadvantages:
 Less control on exposure to stimulus

22

11
 18-May-20

Quasi Experimental Designs (2/2)

 Multiple Time Series
EG: O 1 O2 O3 O4 X O5 O6 O7 O8
CG: O 1 O2 O3 O4 O5 O6 O7 O8
 Basic Procedure
 Similar to simple time series
 CG added for isolating effects of treatment
 Disadvantages:
 Less control on exposure to stimulus
 Non Random Exposure to treatment

23

Statistical Designs

24

12
 18-May-20

Introduction
 The most powerful set of experimental designs
 Allows for statistical control
 Based on the Principles of Randomisation, Replication and Local
Control
 Effect of multiple external variables can be measured
 Specific external variables can be statistically controlled
 Complex to implement and analyse
 Four broad Designs:
 Completely Randomized Design (CRD)
 Randomized Block Design (RBD)
 Latin Square Design (LSD)
 Factorial Designs

WARNING

Should not be used without proper knowledge and

without statistical consultation

25

Completely Randomized Design

 Simplest of All Statistical Designs
 Uses the principles of Randomization and Replication
 Applicable when the experimental plane is completely
homogenous

 Advantages:
 Extremely flexible
 Allows for “n” number of treatments either equally or unequally
 Robust, if data is lost information can be recovered
 Statistical Analysis Requires ANOVA 1 Way

26

13
 18-May-20

Completely Randomized Design

 Testing a new HR Policy
amongst employees
 Four Factories and Four
Grades of Employees.
 Employees are
considered
homogenous
Homogenous

 Factories are
considered
homogenous
 Results can be analyzed
using 1-Way ANOVA to
identify the most
effective HR Policy

Homogenous

27

Randomized Block Design

 Uses the principles of Randomization and Replication
 Applicable when the experimental plane is hetrogenous in one
direction

 Advantages:
 Extremely flexible
 Allows for “n” number of equal treatments
 Each treatment replicated in each block
 Robust, if data is lost information can be recovered
 Statistical Analysis Requires ANOVA 2 Way

28

14
 18-May-20

Randomized Block Design

 Testing a new HR Policy
amongst employees
A
 Employees
Heterogeneous – 4
different worker grades
 Each Color Represents
a Different HR Policy
B
 4 factories – A, B, C & D
 Results can be analyzed
using 2-Way ANOVA to
identify the most
C effective HR Policy

Shop Floor Clerical Frontline Middle

Workers staff Executives Managers

29

Latin Square Design

 Uses the principles of Randomization, Replication and Local
Control
 Applicable when the experimental plane is heterogeneous in both
directions

 Advantages:
 Not Flexible – Requires the same number of replications as the number of
treatments
 Always a Square Design
 Treatment occurs once and only once in each heterogeneous plane
 Robust, if data is lost information can be recovered
 Statistical Analysis Requires ANOVA 2 Way (With Replication)

30

15
 18-May-20

Latin Square Design

 Testing a new HR Policy
amongst employees
A
 Employees
Heterogeneous – 4
different worker grades
 Each Color Represents
a Different HR Policy
B
 4 factories – A, B, C & D
 Results can be analyzed
using 2-Way ANOVA to
identify the most
C effective HR Policy

Notice:

Each Policy is administered

D
once and only once in each
factory and in each
employee level
Shop Floor Clerical Frontline Middle
Workers staff Executives Managers

31

Factorial Designs
 Effect of various variables can be studied simultaneously
 Normally each factor is blocked at 2 levels – High and Low
 Assumed that Factor Interaction is additive in nature
 Most Complex of All Designs

32

16
 18-May-20

Limitations of Experimentation
 Time:
 Experiments are time consuming, specially in measuring long term effects
 Cost:
 Expensive in nature as cost has to be incurred for EG, CG and multiple
measurements
 Administration:
 Requires experienced and knowledgeable administrator
 Administration is generally difficult because of human nature
 Effects of exogenous variables are difficult to control

33

17