Measuring and Monitoring Program Outcomes Chapter Outline Program Outcomes Outcome Level, Outcome Change, and Net Effect Identifying Relevant Outcomes Stakeholder Perspectives Program Impact Theory Prior Research Unintended Outcomes Measuring Program Outcomes ‘Measurement Procedures and Properties Reliability Validity Sensitivity Choice of Outcome Measures Monitoring Program Outcomes Indicators for Outcome Monitoring Pitfalls in Outcome Monitoring Interpreting Outcome Data204 Evaluation The previous chapter discussed how a programs process and performance can be ‘monitored. The ultimate goal of all programs, Roweve is not merely to function well, but to bring about change—to affect some problem or social condition in beneficial ways. The changed conditions are the intended outcomes or products of the programs. Assessing the degree to which a programa produces these outcomes isa core function of evaluators A programs intended outcomes are ordinarily identified in she programs impact theory. Sensitive and valid measuremtent of those outcomes is technically challenging but essential to assessing a programs success. In addition, ongoing monitoring of outcomes can be critical to effective program management. Interpreting the results of outcome ‘measirement and monitoring, however, presents a challenge to stakeholders because a -given set of outcomes can be produced by factors other than program processes. This chapter describes how programs outcomes can be identified, how they can be measured and monitored, and how the results can be properly interpreted. aims to improve is the most critical evaluation task because it deals with the thottom line” issue for social programs. No matter how well a program addresses target needs, embodies « good plan of attack, reaches its target population and delivers apparently appropriate services, it cannot be judged successful unless it actually brings about some measure of beneficial change in its given social arena Measuring that beneficial change, therefor, is not only a core evaluation function but also a high-stakes activity for the program. For these reasons, itis @ function that eval- uuators most accomplish with great care to ensure that the findings are valid and prop- ely interpreted, For these same reasons, it is one of the mast difficult and, often, politically charged tasks the evaluator undertakes. Beginning in this chapter and continuing through Chapter 10, we consider how best to identify the changes a program should be expected to produce, how to devise ‘measures of these changes, and how to interpret such measures. Consideration of pro ‘gram elfects begins with the concept of a program oudcome, so we first discuss that pivotal concept A ssessing a programs effects on the cients it serves and the social conditions it Program Outcomes An outcome is the state ofthe target population othe social conditions that @ program expected to have changed. For example, the amount of smoking among teenagers afterChapter 7 / Measuring and Monitoring Program Outcomes 208 exposure to an antismoking campaign in their high school is an outcome. The attitudes ‘toward smoking of those who had not yet started to smoke is also an outcome. Similarly, the “school readiness” of children after attending a preschool program would be an out- come, as would the body weight of people who completed a weight-loss program, the ‘management skills of business personnel after a management training program, and the amount of pollutants in the local river after a crackdown by the local environmental protection agency. Notice two things about these examples. First, outcomes are observed characteris- tics of the target population or social conditions, not of the program, and the definition of an outcome makes no direct reference to program actions. Although the services livered to program participants are often described as program “outputs? outcom as defined here, must relate to the benefits those products or services might have for the participants, not simply thet receipt, Thus, “receiving supportive family therapy" is not a program outcome in our terms but, rather, the delivery of a program service. Similarly, providing meals to 100 housebound elderly persons is not a program out- come; it is service delivery, an aspect of program process, The nutritional benefits of those meals for the health of the elderly on the other hand, are outcomes, as are any improvements in their morale, perceived quality of fe, and risk of injury from attempt- ing to cook for themselves, Put another way, outcomes always refer to characteristics that, in principle, could be observed for individuals or situations that have not received program services. For instance, we could assess the amount of stoking, the school readiness, the body weight, the management skills, and the water pollution in relevant situations where there was no program intervention. Indeed, as we will discuss late, we might measure outcomes in these situations to compare with those where the program ‘was delivered, Second the concept of an outcome, as we define it, does not necessarily mean that the program targets have actually changed or that the program has caused them to change in any way. The amount of smoking by the high school teenagers may not have changed since the antismoking campaign began, and nobody may have lost any weight during their participation in the weight-loss program. Alternatively, there may be change but in the opposite of the expected direction—the teenagers may have increased their smoking, and program participants may have gained weight Furthermore, whatever happened may have resulted from something other than the influence of the program, Perhaps the weight-loss program ran during a holiday sea- son when people were prone to overindulge in sweets. Or perhaps the teenagers decreased their smoking in reaction to news of the smoking-related death of a popu- lar rock music celebrity, The challenge for evaluators, then, is to assess not only the outcomes that actually obtain but also the degree to which any change in outcomes is attributable to the program itself