Experimental Designs

CHAPTER 9

Chapter Objectives

Distinguish between causal and correlational analysis Explain the difference between lab and field experiments. Explain the following terms: nuisance variables, manipulation, experimental and control groups, treatment effect, matching and randomization.
2

Chapter Objectives

Discuss internal and external validity in experimental designs. Discuss the seven possible threats to internal validity in experimental designs. Describe the different types of experimental designs. Apply what has been learned to class assignments and exams.
3

Experimental Designs

Experimental designs fall into two categories: Experiments done in an artificial or contrived environment, known as lab experiments, and Experiments done in the natural environment in which activities regularly take place, known as the field experiment.
4

Experimental Designs

Experimental designs are set up to examine possible cause and effect relationships among variables. Correlational studies examine the relationships among variables without necessarily trying to establish if one variable causes another.
5

Experimental Designs

1.

2.

3.

To establish that variable X causes variable Y, all three of the following conditions should be met: Both X and Y should covary (when one goes up, the other should also go up or down). X (the presumed causal factor) should precede Y. In other words, there must be a time sequence in which the two occur. No other factor should possibly cause the change in the dependent variable Y.
6

Experimental Designs

To establish causal relationships between two variables, several variables that might covary with the dependent variable have to be controlled. This control would allow us to say that variable X alone causes the dependent variable Y.
7

Experimental Designs

establishing cause-and-effect relationships is not easy because: Several other variables that covary with the dependent variable have to be controlled. It is not always possible to control all the variables while manipulating the causal factor (the independent variable that is causing the dependent variable) in organizational settings, where events flow naturally and normally.
8

Experimental Designs

It is possible to first isolate the effects of a variable in a tightly controlled artificial setting (the lab setting), and After testing and establishing the causeand-effect relationship under these tightly controlled conditions, see how generalizable such relationships are to the field setting.
9

Example 1

Suppose a manager believes that staffing the accounting department completely with personnel with M.Acc (Master of Accountancy) degrees will increase its productivity. Thus, he wants to examine the hypothesis that possession of a M.Acc degree would cause increases in productivity.

10

Example 1 (Cont.)

The hypothesis can be tested in an artificially created setting (not at the regular workplace) because if it was tested at the regular workplace, the manager should transfer all those without the M.Acc degree currently in the department to other departments and recruit fresh M.Acc degree holders to take their place. Such action will disrupt the work of the entire organization.
11

Example 1 (Cont.)

The work of the organization will be disrupted because of the following factors: The new people will have to be trained. Employees will get upset. Work will slow down.

12

Example 1 (Cont.)

The hypothesis that possession of a M.Acc degree would cause increases in productivity can be tested not at the regular workplace but in an artificial created setting in which an accounting job can be given to three groups of people.
13

Example 1 (Cont.)

An artificial created setting in which an accounting job can be given to three groups of people: The first group contains those with a M.Acc degree. The second group contains those without a M.Acc degree. The third group contains a mixed group of those with and without a M.Acc degree (as in the case in the present work setting)
14

Example 1 (Cont.)

If the first group performs exceedingly well, the second group poorly, and the third group falls somewhere in the middle, there will be evidence to indicate that the M.Acc degree qualification might indeed cause productivity to rise.
15

Example 1 (Cont.)

If such evidence is found, then planned and systematic efforts can be initiated to gradually transfer those without the M.Acc degree in the accounting department to other departments and recruit others with this degree to this department. It is then possible to see to what extent productivity does, in fact, go up in the department because all the staff members are M.Acc degree holders.
16

The Lab Experiment

When a cause-and-effect relationship between an independent and a dependent variable of interest is to be clearly established, then all other variables that might contaminate or confound the relationship have to be tightly controlled so that the actual causal effects of the investigated independent variable on the dependent variable can be determined. It is also necessary to manipulate the independent variable so that the extent of its causal effects can be established.
17

The Lab Experiment

The controls and manipulations are best done in an artificial setting (the laboratory), where the causal effects can be tested. When controls and manipulations are introduced to establish cause-and-effect relationships in an artificial setting, we have laboratory experimental designs or lab experiments.

18

Control of the Dependent Variable

When we assume cause-and-effect relationships between two variables X and Y, it is possible that some other factor, say A, might also influence the dependent variable Y. In such a case, it will not be possible to determine the extent to which Y occurred only because of X, since we do not know how much of the total variation of Y was caused by the presence of the other factor A. So we have to control the contaminating factor, A.

19

Example 2

A Human Resource Development manager might arrange for special training to a set of newly recruited secretaries in creating web pages to prove that such training would cause them to function more effectively. However, some of the secretaries might function more effectively than others because they have had previous experience with the web. In this case, the manager cannot prove that the special training alone caused greater effectiveness, since the previous experience of some secretaries with the web is a contaminating factor.
20

Example 2 (Cont.)

If the true effect of the training on learning is to be assessed, then the learners previous experience has to be controlled. This might be done by not including in the experiment those who already have had some experience with the web.
21

Manipulation of the Independent

In order to examine the causal effects of an independent variable on a dependent variable, certain manipulations need to be tried. Manipulation means that we create different levels of the independent variable to assess the impact on the dependent variable.
22

Example 3

If we want to test the theory that depth of knowledge of various manufacturing technologies is caused by rotating the employees on all the jobs on the production line and in the design department, over a 4-week period.
23

Example 3 (Cont.)

To test this theory, we can manipulate the independent variable, rotation the employees, by: - rotating one group of production workers and exposing them to all the systems during the 4-week period. - rotating the second group of workers and exposing them to only half of the manufacturing technologies during the 4-weeks. - leaving the third group to continue to do what they are currently doing, without any special rotation.

24

Example 3 (Cont.)

By measuring the depth of knowledge of these groups both before and after the manipulation (also known as treatment), it would be possible to assess the extent to which the treatment caused the effect, after controlling the contaminating factors.
25

Example 3 (Cont.)

If deep knowledge is indeed caused by rotation and exposure, the results would show that: - the third group had the lowest increase in depth of knowledge. - the second group had some significant increase, and - the first group had the greatest gains.
26

Example 4

We want to test the effects of lighting on worker production levels among sewing machine operators. To establish cause-and-effect relationship, we must follow the following steps:

27

Example 4 (Cont.)

1. 2.

3.

First measure the production levels of all the operators (60 operators) over a 15-day period with the usual amount of light they work with- say 60 watt lamps. Split the operators into three groups of 20 members each: Allowing one subgroup to continue to work under the same conditions as before (60-watt lambs) Manipulate the intensity of the light for the second subgroup by working with 75-watt lambs, and Manipulate the intensity of the light for the third subgroup by working with 100-watt lambs.
28

Example 4 (Cont.)

After the different groups have worked with these varying degrees of light exposure for 15 days, each groups total production for these 15 days may be analyzed to see if the difference between the pre-experimental and the post-experimental productions among the groups is directly related to the intensity of the light to which they have bees exposed.
29

Example 4 (Cont.)

If the hypothesis that better lighting increases the production levels is correct, then the subgroup that did not have any change in the lighting (called the control group), should have no increase in production and the other two groups should show increases: - the group with the 100-watt lambs showing the greatest increase, and - the group with the 75-watt lambs showing increase lower than the 100-watt group.
30

Example 4 (Cont.)

The independent variable (lighting) has been manipulated by exposing different groups to different degrees of changes in light. This manipulation of the independent variable is also known as the treatment, and the results of the treatment are called treatment effects.
31

Example 7.1: Using both Controlling and Manipulation in a Lab Setting

The owner of a toy shop is disappointed with the number of imitation Ninja turtles (which is greatly in demand) produced by his workers, who are paid wages at an hourly rate. He might wonder whether paying them piece rates would increase their production levels.
32

Example 7.1: Using both Controlling and Manipulation in a Lab Setting

Before implementing the piece-rate system, he would want to make sure that switching over to the new system would indeed achieve the objective.

The researcher might first want to test the causal relationships in a lab setting, and if the results are encouraging, conduct the experiment later in a field setting.
33

Example 7.1: Designing the Lab Experiment

1. 2. 3.

The researcher should first think of possible factors that would affect the production level of the workers, and then try to control these factors. The factors that would influence the production levels of the employees, other than piece rates, are: Previous job experience. Gender differences. Age The researcher needs to control these three variables.
34

Example 7.1: Designing the Lab Experiment

To control these three variables, the researcher intends to set up four groups of 15 people each, for the lab experiment. one to be used as the control group the other three subjected to three different pay manipulations.
35

Example 7.1: Designing the Lab Experiment

1. 2.

The variables that may impact on the cause-and-effect relationship can be controlled in two different ways: Either by matching the groups or Through randomization.

36

Controlling the Contaminating Exogenous or Nuisance Variables

Matching Groups Is done by matching the various groups by picking the confounding characteristics and deliberately spreading them across groups. In our example, if there are 20 women among the 60 members, then each group will be assigned 5 women. Likewise, age and experience factors can be matched across the four groups, such that each group has a similar mix of individuals in terms of gender, age , and experience.
37

Controlling the Contaminating Exogenous or Nuisance Variables

Because the suspected contaminating factors are matched across the groups, we may take comfort in saying that variable X alone causes variable Y. But here, we are not sure that we have controlled all the nuisance factors, since we may not be aware of them all.
38

Controlling the Contaminating Exogenous or Nuisance Variables

Randomization Another way of controlling the contaminating variables is to assign the 60 members randomly to the four groups. Every member would have a known and equal chance of being assigned to any of these four groups. We might throw the names of all the 60 members into a box, and draw their names. The first 15 names drawn may be assigned to the first group, the second 15 to the second group, and so on.

39

Controlling the Contaminating Exogenous or Nuisance Variables

In randomization the process by which individuals are drawn and their assignment to any particular group are both random. Thus, the confounding variables, age, sex, and previous experience (the controlled variables) will have an equal probability of being distributed among the groups.

40

Controlling the Contaminating Exogenous or Nuisance Variables

Randomization would ensure that all variables that have effects (known or unknown factors) on the dependent variable will be distributed equally among all groups. Any causal effects found would be over and above the effects of the confounding variables.
41

The Difference between Matching and Randomization

We expect that the process of randomization would distribute the inequalities among the groups, based on the laws of normal distribution. Thus, we need not be concerned about any known or unknown confounding factors. In matching groups, individuals are deliberately and consciously matched to control the differences among group members.
42

The Difference between Matching and Randomization

Matching might be less effective, since we may not know all the factors that could possibly contaminate the cause-and-effect relationship in any given situation, and hence fail to match some critical factors across all groups while conducting an experiment. Randomization will take care of this, since all the contaminating factors will be spread across all groups.
43

The Difference between Matching and Randomization

Even if we know the confounding variables, we may not be able to find a match for all such variables. For instance, if we have only 2 women in a four-group experimental design, we will not be able to match all the groups with respect to gender. Thus, lab experimental designs involve control of the contaminating variables through the process of either matching or randomization, and the manipulation of the treatment.
44

Table 7.1: Cause and Effect Relationship after Randomization

Treatment
$1.00 per piece $1.50 per piece $2.00 per piece

Treatment effect (%
increase in production over pre-piece rate

system)

10 15 20 0
45

Table 7.1 (Cont.)

Note that because the effects of experience, sex, and age have been controlled in all the four groups by randomly assigning the members to them, and the control group had no increase in productivity, it can be concluded from the result that the percentage increases in production are a result of the piece rate (treatment effect). Here we have a high internal validity or confidence in the cause-and-effect relationship.

46

Example 5

47

48

49

50

Internal Validity

Internal validity refers to the confidence we place in the cause-andeffect relationship. Internal validity addresses the question, To what extent does the research design permit us to say that the independent variable A causes a change in the dependent variable B?
51

Internal Validity

In research with high internal validity, we are relatively better able to argue that the relationship is causal, whereas in studies with low internal validity, causality can not be inferred at all. In lab experiments where cause-and-effect relationships are substantiated, internal validity can be said to be high.

52

External Validity or Generalizability or Lab Experiments

If we do find a cause-and-effect relationship after conducting a lab experiment, can we then confidently say that the same cause-and-effect relationship will also hold true in the organizational setting? The answer is NO.
53

External Validity or Generalizability of Lab Experiments

The tasks in organizational settings are far more complex, and there might be several confounding variables that cannot be controlled. Under such circumstances, we cannot be sure that the cause-and-effect relationship found in the lab experiment is necessarily likely to hold true in the field setting.
54

The Field Experiment

The field experiment is an experiment done in the natural environment in which work goes on as usual, but treatments are given to one or more groups. In the field experiment, even though it may not be possible to control all the nuisance variables because members cannot be either randomly assigned to groups, or matched, the treatment can still be manipulated.
55

The Field Experiment

If there are three different shifts in a production plant, and the effects of the piece-rate system are to be studied, one of the shifts can be used as the control group, and the two other shifts given two different treatments or the same treatment,that is, different piece rates or the same piece rate.
56

The Field Experiment

Any cause-and-effect relationship found under these conditions would have wider generalizability to other similar production settings, even though we may not be sure to what extent the piece rates alone were the cause of the increase in productivity, because some of the other confounding variables could not be controlled.
57

External Validity

External validity refers to the extent of generalizability of the results of a causal study to other settings, people, or events. Internal validity refers to the degree of our confidence in the causal effect (that variable X causes variable Y).
58

External Validity

Field experiments have more external validity (the results are more generalizable to other similar organizational settings), but less internal validity (we cannot be certain of the extent to which variable X alone causes variable Y). In the lab experiment, the reverse is true. The internal validity is high but the external validity is rather law.
59

Trade-Off Between Internal and External Validity

If we want high internal validity, we should be willing to settle for lower external validity and vice versa. To ensure both types of validity, researchers usually try first to test the causal relationships in a tightly controlled artificial or lab setting and once the relationship has been established, they try to test the causal relationship in a field experiment.
60

Factors Affecting Internal Validity

Lab experiments could be influenced by factors that might affect the internal validity. These possible confounding factors pose a threat to internal validity.

61

Factors Affecting Internal Validity

62

History Effects

Certain events or factors that would have an impact on the independent variable-dependent variable relationship might unexpectedly occur while the experiment is in progress, and this history of events would confound the cause-and-effect relationship between the two variables, thus affecting the internal validity.
63

Example 7

Let us say that the manager of a Dairy Products Division wants to test the effects of the buy one, get one free sales promotion on the sale of the company-owned brand of packaged cheese, for a week. The manager carefully records the sales of the packaged cheese during the previous 2 weeks to assess the effect of the promotion.

64

Example 7 (Cont.)

On the first day the sales promotion goes into effect, the Dairy Farmers Association unexpectedly launches a multimedia advertisement on the benefits of consuming dairy products, especially cheese. The sales of all dairy products, including cheese, go up in all the stores.
65

Example 7 (Cont.)

Here, because of unexpected advertisement, one cannot be sure how much of the increase in sales of the packaged cheese in question was due to the sales promotion and how much to the advertisement of the Dairy Farmers Association. The effects of history have reduced the internal validity or the faith that can be placed on the conclusion that sales promotion caused the increase in sales. The history effects in this case are illustrated in Figure 7.1
66

Figure 7.1

67

Maturation Effects

Other uncontrollable variable is the passage of time which called maturation effect. The maturation effects are a function of the processes operating within the respondents as a result of the passage of time. Examples of maturation processes could include growing older, getting tired, feeling hungry, and getting bored.
68

Example 8

Let us say that an R & D director contends that increases in the efficiency of workers would result within 3 months time if advanced technology is introduced in the work setting. If at the end of the 3 months increased efficiency is indeed found, it will be difficult to claim that the advanced technology (and it alone) increased the efficiency of workers, because with the passage of time, employees would also have gained experience, resulting in better job performance and therefore in improved efficiency.

69

Example 8 (Cont.)

Thus, the internal validity also gets reduced owing to the effects of maturation inasmuch as it is difficult to pinpoint how much of the increase is attributable to the introduction of the enhanced technology alone. Figure 7.2 illustrates the maturation effects in the example.
70

Figure 7.2

71

Testing Effects

The respondents were exposed to the pretest might influence their responses on the posttest, which would adversely impact on internal validity. For example, if a challenging job is expected to cause increases in job satisfaction, and a pretest on job satisfaction is administered asking for employees level of satisfaction with their current jobs.
72

Testing Effects

When a challenging job is introduced and a further job satisfaction questionnaire administered subsequently, the respondents might now react and respond to the posttest with a different frame of reference than if they had not originally been sensitized to the issue of job satisfaction through the pretest.. This kind of sensitization through previous testing is called the testing effect, which affects the internal validity of experimental designs.
73

Instrumentation Effects

The instrumentation effects might arise because of a change in the measuring instrument between pretest and posttest. In organizations, instrumentation effects in experimental designs are possible when the pretest is done by the experimenter, treatments are given to the experimental groups, and the posttest on measures such as performance is done by different managers.
74

Instrumentation Effects

One manager might measure performance by the final units of output, a second manager might take into account the number of rejects as well, and a third manager might also take into consideration the amount of resources expended in getting the job done.
75

Selection Bias Effects

It comes from improper or unmatched selection of subjects for the experimental and control groups. For example, if a lab experiment is set up to assess the impact of working environment on employees attitudes toward work, and if one of the experimental conditions is to have a group of subjects work for about 2 hours in a room with some high temperature.
76

Selection Bias Effects

If the researcher select a group of volunteers who are poor and unemployed, those will be quite different from the other workers whom are not poor or unemployed, and their responses to the treatment might be quite different. Such bias in the selection of the subjects might contaminate the cause-and-effect relationships and pose a threat to internal validity.
77

Statistical Regression

The effect of statistical regression are brought about when the members chosen for the experimental group have extreme scores on the dependent variable to begin with. For example, if a researcher wants to test the understanding of students for Research Methods classes, he should not choose those with extremely low or extremely high ability students for the experiment.
78

Statistical Regression

This is because we know from the laws of probability that those with very low scores on a variable have a greater probability of showing improvement and scoring closer to the mean on the posttest after being exposed to the treatment. Likewise, those with very high abilities would also have a greater tendency to regress toward the meanthey would score lower on the posttest than on the pretest. Thus, those who are at either end of the continuum with respect to a variable would not truly reflect the cause-and-effect relationship.
79

Mortality

The mortality of the members in the experimental or control group or both, is another confounding factor on the cause-andeffect relationship. When the group composition changes over time, comparison between the groups becomes difficult, because those who dropped out of the experiment may confound the results.
80

Factors Affecting External Validity

External validity raises issues about the generalizability of the findings to other settings. The reasons are: - the effects of the treatment in lab experiments are not the same in the field. - the selection of the subjects in lab setting could be very different from the types of subjects selected by the organizations.
81

Example 10

Students in a university might be given a task that could be manipulated to study the effects on their performance. The findings from this experiment cannot be generalized to the real world of work, where the employees and the nature of the jobs would both be quite different.
82

Factors Affecting External Validity

Maximum external validity can be obtained by ensuring that the lab experimental conditions are as close to and compatible with the real-world situation. Thus, field experiments have greater external validity than lab experiments.
83

Simulation

Simulation is an alternative to lab and field experimentation. Simulation uses a model-building technique to determine the effects of changes. Computer-based simulations are becoming popular in business research.
84

Simulation

A simulation is an experiment conducted in a specially created setting that very closely represents the natural environment in which activities are usually carried on. The simulation lies somewhere between a lab and a field experiment since the environment is artificially created but not far different from reality.

85

Simulation

Two types of simulations can be done: One in which the nature and timing of simulated events are totally determined by the researcher (called experimental simulation). The other where the course of activities is at least partly governed by the reaction of the participants to the various stimuli as they interact among themselves (called free simulation).
86

Simulation

Experimental and free simulations are both expensive, since creating realworld conditions in an artificial setting and collecting data over extended periods of time involve a high costs.

87

Simulation

Causal relationships can be tested since both manipulation and control are possible in simulations. Cause-and-effect relationships are better established in experimental simulations where the researcher exercises greater control.
88

Areas Where Simulation can be Used

The effectiveness of various analytic review procedures in detecting errors in account balances has been tested through simulations. In the finance area, risk management has been studied through simulations. Simulations have also been used to understand the complex relationships in the financing of pension plans and making important investment decisions.
89

Simulation

Simulation has also been used by many companies to test the robustness and efficacy of various products. It is quite likely that we will see simulation being used as a managerial tool to enhance motivation, leadership, and the like, in the future. Simulation can also be applied as a problem-solving managerial tool in other behavioral and administrative areas.
90

Ethical Issues in Experimental Design Research

The following practices are considered unethical: Putting pressure on individuals to participate in experiments through coercion, or applying social pressure. Giving menial tasks and asking demeaning questions that diminish their self respect. Deceiving subjects by deliberately misleading them as to the true purpose of the research.
91

Ethical Issues in Experimental Design Research

Exposing participants to physical or mental stress. Not allowing subjects to withdraw from the research when they want to. Using the research results to disadvantage the participants, or for purposes not to their liking. Not explaining the procedures to be followed in the experiment.
92

Ethical Issues in Experimental Design Research

Exposing respondents to hazardous and unsafe environments. Not debriefing participants fully and accurately after the experiment is over. Not preserving the privacy and confidentiality of the information given by the participants.
93