Experimental Design

Brian Mennecke College of Business Iowa State University

The Source
The source of much of this information comes from Campbell & Stanley
Campbell, D. T. and J.C. Stanley (1963). Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally College Publishing Company.

The Goal of Research

When one conducts science, the goal is to seek out the truth.
Question: How does one identify truth?

Experimentation is one mechanism for identifying causation, which is a step toward understanding how one set of factors influence another set of factors

Causation and Positivism

Positivism is a research perspective that has as its premise that inferences about cause can be made. David Hume espoused the conditions by which inference could be made; these include
Contiguity between the cause and effect Temporal precedence Constant conjunction (i.e., when the effect is seen, the cause is always present)

But how do we know something is true?

Some propositions are not true; how do we know when something is true or not? One approach is to test for validity.
Validity is a term used to describe whether the conclusions one draws about a proposition are true or false

Types of validity
Internal Validity: How sure are we that the cause leads to the expected results? In other words, is it appropriate for us to infer that the relationship between variables is causal External Validity: How sure are we that we can generalize the finding of causation to other populations, settings, or variables? Construct Validity: How sure are we that the variables we are using actually measure the concept (i.e., the construct) that we are seeking to measure? Statistical Conclusion Validity: Do the statistical tests that we perform accurately measure the relationships between the variables under study?

Threats to Internal Validity

(Campbell & Stanley)
History: events that occur between the first and second measurement that are unrelated to the experiment but that could affect the results. Maturation: Changes in the participants that occur as a function of the passage of time and not specific to the experiment. Testing: The effects of taking a first test on the scores of a second test. Instrumentation: Changes in the measurement instrument or changes or the observers make changes in the obtained measurements. Statistical regression (toward to mean): Groups having extreme scores on the pretest (or selected on the basis of extreme scores) will tend to have scores closer to the mean on the posttest. Selection: Biases resulting in differentials selection of respondents for the comparison groups. Experimental mortality: Differential loss of respondents from the comparison groups. Selection-maturation interaction, other interaction effects:

Threats to External Validity

Reactive or interaction effects of testing: The pretest itself might be a learning experience such that by taking the pretest students gain information that will affect posttest results Interaction of selection and the experimental variable: Different groups may respond differently to the experimental variable. Reactive effects of experimental arrangements: Subjects respond differently because they know they are in an experiment (i.e., the Hawthorne effect) Multiple treatment interference: Multiple treatments applied to the same respondents; the effects of prior treatments cannot be erased.

What is the basis for asking questions about causation?

The source for all questions pertaining to research experimentation is theory
Why is theory important?

Theory should always drive research because it defines expectations about the relationships that exist between variables.

Before we get started

Some definitions:
Construct: An idea or concept that you are attempting to measure
Latent Construct: A construct that cannot be measured directly (e.g., group cohesion)

Independent Variable: Variables that are presumed to be the cause of an effect being studied; independent variables are manipulated to examine their impact on results Dependent Variables: Variables that are observed to understand the result of causation. Hypothesis: A statement of a possible explanation for causation. An hypothesis is tested by drawing conclusions from an experimental examination of the variables that are expected to be related

Types of Experimental Designs

Pre-experimental designs: One group designs and designs that compare pre-existing groups Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental conditions but that do not use random selection and assignment to treatment conditions. True experimental designs: Experiments that have treatments, outcome measures, and experimental conditions and use random selection and assignment to treatment conditions. This is the strongest set of designs in terms of internal and external validity.

Pre-Experimental Designs
Design 1: One-Shot Case Study: A single group is studied once after some intervention/treatment that is presumed to cause change.
For example, a training program is implemented and participants are given a posttest at the conclusion of the training.

Pre-Experimental Designs
Design 2: One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group. Generally done with intact groups.
For example, a classroom teacher gives her students a pretest then implements an instructional strategy followed by a posttest.

O1

O2

Pre-Experimental Designs
Design 3: The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is compared to another group that has not had the intervention. The groups are not randomly selected nor randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest.
For example, comparison of GRE scores for students who attended a rural high school versus those who attended an urban high school.

X1 X2

O O

True Experimental Designs

Design 4: Pretest-Posttest Control Group Design: One group is administered a treatment while the other is not; all groups are observed before and after the treatment is administered.
For example, 50 freshman students are randomly selected to participate in a tutoring study. Half are randomly assigned to a tutor for their first semester and half are not. All students are given a pretest at the beginning of the term and a posttest at the end of the term.

R R

O1 O1

O2 O2

True Experimental Designs

Design 5: Solomon Four-Group Design: This design involves four experimental groups. Two of the groups parallel the structure of Design 4 while the remaining two groups include no pre-test (so that the effects of the pretest can be evaluated).
For example, 100 freshman students are randomly selected to participate in a tutoring study. 25 are randomly assigned to a tutor for their first semester and given a pretest. 25 are randomly assigned to a group where no tutor is assigned and they are given a pretest. Another 25 are randomly assigned to a tutor but not given a pretest. The remaining 25 are randomly assigned to a group where no tutor is assigned and they are not given a pretest. Whew!

R R R R

O1 O1

O2 O2 O2 O2

True Experimental Designs

Design 6: Posttest Only Control Group Design: One group is administered a treatment while the other is not; all groups are observed after the treatment is administered BUT not before the treatment.
For example, students are randomly assigned to two groups of 50 each. The experimental (treatment) group receives a new teaching method during a special class session. The second group (the control) receives a traditional teaching method during a special class session. No pretest is used for each group. Issues such as existing grades, SAT scores, and other factors are examined as covariates.

R R

O2 O2

Quasi-Experimental Designs
Design 7: The Time-Series Experiment: This design involves periodic measurements of some group or individuals and the introduction of a change into the conditions during the series.
For example, studying a group of workers over time and taking several measures of productivity during this period. At some point a new work process is introduced and measures of productivity are taken over several weeks following the intervention.

O1

O2

O3

O4

O5

O6

Quasi-Experimental Designs
Design 8: Equivalent Time-Samples Designs: This design involves periodic introduction of treatments followed by measurements with the treatments varied consistently over time.
For example, to study the effect on student discussions of having an observer appear in a classroom. At time period one, an observer is present and a measure of discussion level is made. At time two, no observer is present and a measure of discussion level is made. At time three an observer is present, a measure is taken. At time four an observer is not present, a measure is taken. Etc.

X1

X2

X1

X2

Quasi-Experimental Designs
Design 9: The Equivalent Materials Design: This design involves giving equivalent samples of materials to subjects, imparting interventions, and then making observations.
For example, subjects are asked to complete a survey instrument about their opinions related to current events. The students are then split into two groups and given two different sets of (falsified) survey results indicating how other students answered the survey. Both groups are then asked to complete the survey again to observe how they respond.

Experimental Materials A(O) X0 O Experimental Materials B(O) X0 O

Quasi-Experimental Designs
Design 10: Nonequivalent Control Group: This design involves an experimental and control group with both given pretests and posttest; however, these groups are not randomly selected because they constitute naturally assembled groups (e.g. classrooms). The assignment of X (the treatment) to one group or the other is randomly selected by the researcher.
For example, four sections of a course are chosen to participate in a study of teaching methods. Half are randomly assigned a new teaching method and half are not. All are given pretests at the beginning of the term and all are given posttests at the end of the semester.

O O

O O

Quasi-Experimental Designs
Design 11: Counterbalanced Designs: In this design all subjects receive all treatments but in a different order. Each treatment occurs once at each time period and once for each treatment group. A Latin-square design is a type of counterbalanced design in which four treatments are applied to four naturally assembled pools of subjects.
For example, consider a study of the effect of different training methods on learning. Subjects are placed into four groups (A,B,C, D) for different training methods, X1-X4.

Group A Group B Group C Group D

X1O X2O X3O X4O

X2O X4O X1O X3O

X3O X1O X4O X2O

X4O X3O X2O X1O

Quasi-Experimental Designs
Design 12: The Separate Sample Pretest-Posttest Design: Often used with large populations (i.e., in public opinion studies) where the researcher cannot randomize or segregate subgroups for different treatments. Two equivalent groups are identified, one sample is measured prior to the treatment and a different (but equivalent) sample is measured after the treatment. This design is also called the "simulated before and after" design.
For example, 100 community members are randomly surveyed concerning their opinions about local government policies. A PR campaign is then conducted for six weeks. A follow-up survey is then conducted with 100 different residents who are randomly selected.

R R

X X

Quasi-Experimental Designs
Design 13: The Separate Sample Pretest-Posttest Control Group Design: This design is similar to Design 12; however, a control group is added to the design.
For example, consider the PR campaign described in Design 12. In this case, the same design is used, but, in addition, the measurements are made in a similar nearby city where no PR campaign is run.

R R

X X

R R

O
O

Quasi-Experimental Designs
Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design): This is an approach used in field research. A researcher begins with an inadequate design and then adds features to control for one or more sources of invalidity. The result is an "inelegant accumulation of precautionary checks." The researcher is aware of rival interpretations (sources of internal invalidity) and incrementally identifies other data that would rule out rivals. The design exploits contextual features to refine the research as it progresses.
For example, this design would combine a longitudinal and cross sectional structure. One group will be exposed to X and measured at the same time as a second group that is just about to be exposed to X. A comparison of the two groups would be able to be made because it is equivalent to a static group comparison. The second group would be remeasured (posttest), which would make the design comparable to the one group pretest-posttest design.

Group A Group B

O1 O1 X O2

My Research Agenda
So, what type of research approach do you think I use?

General Research Themes

Geographic Information Systems (GIS) and Location Intelligence
Studies of the use of GIS as a decision support tool The use of GIS in Businesses and Organizations Location intelligence and the use of location in decision making Perceptions of space and geography

General Research Themes

Studies of Teams, Collaboration, and technology
Virtual Teams Team History Individual Characteristics

General Research Themes

Virtual Worlds
The application of VW to education and learning Perceptions of avatars, space and location in VWs Legal, tax, and social issues in VWs Communication and collaboration in VWs

General Research Themes

Mobile Commerce, Computing, and Virtual Teams
Mobile Device Interfaces Impressions of Mobile Device Users Applications of Mobile Devices

General Research Themes

Applications of Conjoint to IS Research
Human Resources Information Systems Analysis IT Planning

General Research Themes

IT Adoption and Implementation
User Acceptance of Mobile Devices The Use of Mobile Devices in Commerce

General Research Themes

The Application of IT for Training and Learning
The Application of Technology in Education The Role of Communication Technology in Learning

A Recent Study
Question: What is the impact of video conference technology and training methodology on student learning IV:
Training Mode:
Enactive Mastery Vicarious Experience

Communication Media
Face to Face Video Conferencing

Results
12 11

10

VE EM

6 FTF DVC

Results
Tests of Between-Subj ects Effects Dependent Variable: DQ - Total Type III Sum Source of Squares Corrected Model 2765.596 a Intercept 112.075 MIDGSETO 339.197 PFTSCORE 1506.172 MEDIA 198.404 TRAINING 75.519 MEDIA * TRAINING 232.381 Error 7060.445 Total 53232.000 Corrected Total 9826.041 df 5 1 1 1 1 1 1 141 147 146 Mean Square 553.119 112.075 339.197 1506.172 198.404 75.519 232.381 50.074 F 11.046 2.238 6.774 30.079 3.962 1.508 4.641 Sig. .000 .137 .010 .000 .048 .221 .033

a. R Squared = .281 (Adjusted R Squared = .256)