Chapter 3: Reliability and Validity

**This chapter corresponds to chapter 6 of your book (Just the Truth) What it is: Reliability and validity are terms that refer to the quality of the measures used in a research study. Reliability refers to the consistency and validity refers to the accuracy of the measure. There are several types of reliability (test-retest, parallel forms, internal consistency, interrater) as well as several types of validity (content, criterion, construct). These different types of reliability and validity are used for different types of measures but can also work together. The more types of reliability and validity a measure demonstrates, the more confident we can be in the quality of the measure. When to use it: Reliability and validity are important any time you measure anything (so basically in every research study). The exact nature of a study determines which type of reliability and/or validity you should assess (see your text for details). Using SPSS to calculate Internal Consistency Reliability (Cronbachs Alpha) (dataset: Chapter3Example1.sav) The rest of this chapter will focus on internal consistency reliability (accessed with Cronbachs Alpha). We are focusing on this topic specifically because it is the most commonly utilized type of reliability as well as the most straightforward to demonstrate in SPSS. Cronbachs alpha ranges from 0 to 1 and tells you how internally consistent a group of items are. In other words, Cronbachs alpha tells you the extent to which a group of items measure the same thing. The closer the value of Cronbachs alpha is to 1, the more consistent the items in a measure. In this example you will use SPSS to calculate Cronbachs alpha for The Meaning in Life Questionnaire. Here is what that measure looks like: The Meaning in Life Questionnaire (Steger, Frazier, Oishi, & Kaler, 2006) Please take a moment to think about what makes your life feel important to you. Please respond to the following statements as truthfully and accurately as you can, and also please remember that these are very subjective questions and that there are no right or wrong answers. Please answer according to the scale below: Absolutely Untrue 1 2 3 4 5 6 7 Absolutely True

1. I understand my lifes meaning. 2. My life has a clear sense of purpose. 3. I have a good sense of what makes my life meaningful. 4. I have discovered a satisfying life purpose. 5. My life has no clear purpose.

Open up the data set Chapter3Example1.sav. It should look like this:

Youll see this data set includes the responses of 100 people to the 5 questions from the meaning in life questionnaire. Each question is called an item and is labeled with a variable name that tells you which question number it refers to (i.e., MIL1 is I understand my lifes meaning.). Take a second and go back and look at the 5 items included in the meaning in life questionnaire. Do any of these items jump out at you? Does any item seem like it might not belong with the others? You may have noticed that item 5 is different than the other 4 items. A person who believes their life has meaning would respond with high numbers to items 1-4 BUT with a low number to item 5. This is called a reverse coded item meaning that it should be coded opposite of the rest of the measure. Whereas responses with high numbers on items 1-4 indicate high meaning in life, responses with low numbers on item 5 indicate high meaning in life. How to handle reverse coded items When you have a measure that includes reverse coded items, you must use the recode procedure in SPSS to reverse peoples responses to those items. This means that a 7 is turned into a 1, a 6 is turned into a 2 and so on In this example, well need to reverse score one item: MIL5. To recode an item, highlight the transform menu and then click on recode into different variables as shown below:

That will bring up the following window:

Highlight the item you want to recode (MIL5) and click the arrow to move it over to the window. Next type in a name for what the new recoded variable will be called. You can name it whatever you want, but a handy way to do this is to simply add an r (r stands for recoded) after the existing variable name (e.g. MIL5r). Youll have to click the change button in the output variable box to make the new variable name show up in the numerical variable -> output variable box. Your screen should look like this:

Next click on the old and new values button to bring up the following window:

Type the value of the lowest number in your scale (i.e., a 1) in the value box under the word Old Value. Then type the value of the highest number in your scale (i.e., a 7) in the value box under the word New Value. In this example, this tells SPSS to turn 1s into 7s. Now click the Add button to make the numbers show up in the Old -> New box. Continue this process for each number that was a response option. The 2s become 6s, the 3s become 5s and so on. Your screen should look like this:

Once you have done this for all the numbers click continue and then ok. Take a look at the data window to confirm that your new variable is there it will be located in the last column of the data set. Spot-check some of the values in your new recoded variable to make sure the recoding worked (e.g., does it look like the 1s were switched to 7s?). Calculating Cronbachs Alpha Once you have reverse scored any reverse coded items, youre ready to calculate Alpha. To begin, click Analyze and then highlight Scale. Next click on Reliability Analysis as shown below:

That will bring up the following window:

Next highlight the first item that is part of your scale and click the arrow to move it over to the Items box. Continue until you have done this for each of the items in your scale. Be careful with the reverse coded item, you only want to include the reverse scored version (not both). Your screen should look like this:

Whenever you compute Cronbachs alpha, its helpful to ask SPSS for an extra bit of output that tells you what the alpha would be if you dropped any of the items. This helps you identify bad items. If SPSS tells you that the alpha for a scale would be quite a bit higher if an item were dropped this is a potential bad item. It may be a reverse coded item you forgot to recode or an item that should potentially be dropped from the scale because it is poorly worded or it measures something different. To get this extra output click on the statistics button and then check the box in the Descriptives for box that is labeled Scale if item deleted. Your screen should look like the following:

Now click continue and then OK. You output will look like the following:

Case Processing Summary N Cases Valid Excluded Total

% 100 0 100 100.0 .0 100.0

This box tells you the number of participants in your sample that completed enough items to be included in the analysis.

a. Listwise deletion based on all variables in the procedure.

This is the value of Cronbachs Alpha for your scale.

Reliability Statistics Cronbach's Alpha .846 N of Items 5

This is the number of items in your scale.

Item-Total Statistics Scale Mean if Item Deleted MIL1 MIL2 MIL3 MIL4 MIL5r 19.4300 19.2300 18.6000 19.3000 18.4400 Scale Variance if Item Deleted 19.783 18.502 19.636 19.646 18.128 Corrected ItemTotal Correlation .685 .740 .667 .643 .573 Cronbach's Alpha if Item Deleted .808 .791 .811 .817 .846

This table is the extra output we requested.

This column is the only part we need to pay attention to. Remember that were looking for items where alpa would go UP if an item was deleted. You can see that all of our items are good items because alpha would stay the same or go down if the item was deleted. Interpreting the Output We have two goals when we look at our output for Cronbachs alpha. 1. Make sure that our alpha is good. 2. Make sure we dont have any bad items. Interpreting the value of alpha can be somewhat subjective because people may have different ideas about whats a good enough value for alpha. Just remember that the closer alpha is to 1, the better. In psychology, people generally think of an alpha that is higher than .80 as good

enough and an alpha between .70 and .80 as OK. Anything below .70 may be a reason to worry. If you find any potential bad items (items that the alpha would increase if they were deleted), go back to your scale and try to figure out what might be going on with that item. Should it be reverse coded if it wasnt? If it was reverse coded, was that a mistake? If theres no coding problem, does it seem like the item might be hard to understand or is measuring something different than the rest of the scale? Sometimes, researchers delete such bad items from the scale for the sake of higher reliability. This is a judgment call that has to be made by the researcher (often, after years of practice). Now what? If youve decided that your alpha is good enough for you, the next step is usually to compute an average score for each participant. This is because what we really want to know is a persons average meaning in life rather than their responses to 5 different items. Having an internally reliable set of items tells you that its ok to average peoples responses on the items because they each measure the same thing. You may be wondering why researchers use multiple items in the first place if you have to mess with all of this alpha stuff and then average the items into a single number anyway. The reason has to do with validity. We know that asking one single item is not always a very good way to accurately measure what were interested in. This is based on the same logic as taking multiple exams in a single class over the course of a semester. Imagine if you only took one test in this class at the end of the semester. That might not be the most accurate way to assess how much you know about the topic what if you are just having a bad day? In the same way that taking multiple tests makes our assessment of your knowledge of the topic more accurate, so does asking multiple items about one thing. To compute a mean, click on transform and then click on compute variable as seen below:

This will bring up the following window:

In the Target Variable box you should type a name for the average score. For example, avgMIL for average meaning in life.

Next, you will want to tell SPSS how to calculate the mean by putting a command in the Numeric Expression box. To compute a mean, the command is Mean(item1,item2,item3). So type mean( in the numeric expression box. Next highlight the first item and click the arrow to move it into the box. Continue this for each item, separating the items with a comma. Dont forget to ONLY include the reverse coded item for MIL5. When you have done this for each item, type in the closing parentheses ). Your screen should look like this:

Now click OK and then navigate to the data view window to make sure your new variable is there It can be found at the end of the data set. Spot-check your new average variable (e.g., is the new variable the correct average of the five items for Participant 1?). You would be able to use this variable in many of the analyses you will learn about the rest of this semester. For example, you might use a t-test to compare meaning in life between 2 groups or you might use a regression analysis to predict meaning in life from some predictor. Exciting times are ahead!

Practice Problem (answers in appendix) In this problem you will assess the reliability of the openness to experience subscale of The Big Five Inventory (John, Donahue, and Kentle, 1991). People who are high in openness to experience are more intellectually curious and willing to think about different ideas. This measure is designed to tap the extent to which somebody is open to experiences. Here is what the scale looks like: Please choose a number next to each statement to indicate the extent to which you agree or disagree with that statement. Disagree strongly 1 2 3 4 5 Agree Strongly

1. I see myself as someone who is original, comes up with new ideas 2. I see myself as someone who is curious about many different things 3. I see myself as someone who is ingenious, a deep thinker 4. I see myself as someone who has an active imagination 5. I see myself as someone who is inventive 6. I see myself as someone who values artistic, aesthetic experiences 7. I see myself as someone who prefers work that is routine 8. I see myself as someone who likes to reflect, play with ideas 9. I see myself as someone who has few artistic interests 10. I see myself as someone who is sophisticated in art, music, or literature. The dataset chapter3problem1.sav includes the responses of 100 people to these 10 questions. A. Read the items carefully and look for any reverse coded items (hint: there are two items). Use SPSS to reverse code the items you believe are reverse coded write below which items you reverse coded.

B. Use SPSS to calculate an alpha for this measure. What is the alpha?

C. Do you see any potential bad items? What do you think might explain these bad items?

D. Calculate a mean for the openness scale then use the descriptive procedure to find the range, mean, and standard deviation for the scale. Write these values below.