See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/247531056

Performance Measurement and Program Effectiveness: A Structural Equation

Modeling Approach

Article in International Journal of Public Administration · May 2008

DOI: 10.1080/01900690701640929

CITATIONS READS

20 689

2 authors, including:

Christopher Mausolff
Nevada County
4 PUBLICATIONS 94 CITATIONS

SEE PROFILE

All content following this page was uploaded by Christopher Mausolff on 28 September 2018.

The user has requested enhancement of the downloaded file.

 Intl Journal of Public Administration

ISSN: 0190-0692 (Print) 1532-4265 (Online) Journal homepage: http://www.tandfonline.com/loi/lpad20

Performance Measurement and Program

Effectiveness: A Structural Equation Modeling
Approach

Christopher Mausolff & John Spence

To cite this article: Christopher Mausolff & John Spence (2008) Performance Measurement
and Program Effectiveness: A Structural Equation Modeling Approach, Intl Journal of Public
Administration, 31:6, 595-615, DOI: 10.1080/01900690701640929

To link to this article: https://doi.org/10.1080/01900690701640929

Published online: 08 May 2008.

Submit your article to this journal

Article views: 224

View related articles

Citing articles: 10 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=lpad20
Intl Journal of Public Administration, 31: 595–615, 2008
Copyright © Taylor & Francis Group, LLC
ISSN 0190-0692 print / 1532-4265 online
DOI: 10.1080/01900690701640929

Performance Measurement and Program

1532-4265
0190-0692
LPAD
Intl Journal of Public Administration,
Administration Vol. 31, No. 6, March 2007: pp. 1–38

Effectiveness: A Structural Equation Modeling

Approach

Christopher Mausolff
Performance
Mausolff andMeasurement
Spence and Program Effectiveness

Department of Family Studies, College of Health and Human Services,

University of New Hampshire, Durham, New Hampshire, USA

John Spence
John D. Spence Consulting, Inc., Louisville, Kentucky, USA

Abstract: Existing research on the effectiveness of performance measurement systems

has generated inconclusive findings. In this study, the performance measurement-
effectiveness relationship is tested in human services programs using the ratings of
volunteer program evaluation teams. A sample of programs was evaluated in 2000
(n = 195), and a similar sample was evaluated again in 2002 (n = 167). In both years, the
quality of performance measurement was significantly correlated (at the 0.01 level) with
program performance. The role of organizational learning from the measurement data, as
a factor contributing to program performance, was supported only in the 2002 sample.
One explanation for this result is that additional time was needed for the incremental
benefits of data-driven learning to add up to substantial program impacts.

Keywords: performance measurement, program effectiveness, structural equation

modeling, organizational learning

INTRODUCTION

There is considerable interest in the effectiveness of performance measure-

ment. The topic has generated lively debates, conceptual think-pieces, case
studies, and survey research.[1–8] All of this attention is justified. Performance
measurement is a major worldwide movement. It is a central part of New Pub-
lic Management and reform efforts in nonprofit organizations and government

Address correspondence to Christopher Mausolff, Ph.D., Department of Family

Studies, University of New Hampshire, Pettee Hall, Room 218, 55 College Road,
Durham, NH 03824, USA; E-mail: [email protected]
596 Mausolff and Spence

agencies.[9–11] As a widely mandated intervention, policymakers should have

a good understanding of its effectiveness before imposing it. Nonetheless,
despite extensive research, we still do not have a good understanding of the
performance measurement-effectiveness relationship.
The research has yielded conflicting findings. Specific cities and programs
have achieved cost savings, improved resource allocation, and better customer
service using performance measurement.[4],[12–14] However, the successes
recorded in these case studies could be isolated events, due to special circum-
stances, rather than inherent benefits of performance measurement. The existing
survey research has also not yet provided a clear answer. In a few studies of local
government officials, the majority of respondents report at least modest benefits
from performance measurement.[6,15] However, in a recent survey of U.S. coun-
ties, just 32.5 percent of respondents agreed or strongly agreed with the statement
that performance measurement enhanced their ability to achieve improvements,
despite resource constraints.[16] Further complicating to these inconsistent find-
ings is a fairly sophisticated literature on the dysfunctional side effects of perfor-
mance measurement.[17–21] Overall, the existing research does not provide
convincing evidence for the effectiveness of performance measurement.
One relatively straightforward way to explain these conflicting findings is
the hypothesis that the performance measurement-effectiveness relationship
depends on the quality of the performance measurement system. According to
this hypothesis, poor performance measurement systems generate negative or
insignificant results, and only well-designed systems generate positive results.
The structure of the present study provides a test of this hypothesis. It will test
the relationship between the rating of performance measurement systems and
program effectiveness.
In addition to the question of whether performance measurement contrib-
utes to performance, there is also the question of how it might have this impact.
When performance measurement works, what are the causal mechanisms?
A number of researchers have discussed enhanced organizational learning as a
potential benefit.[22–25] It is likely that over time, continued learning would
generate measurable gains in effectiveness. In view of this potential, a second
purpose of this study is to test the importance of organizational learning as a
pathway by which performance measurement enhances effectiveness. The
performance measurement-effectiveness relationship will be examined in this
study through the development and testing of a structural equation model. The
data for testing this model come from approximately 170 health and human
service programs assisted by Metro United Way in Louisville, Kentucky.
This research is presented in five sections. In the first section, we use the
existing literature to develop a model of hypothesized relationships between per-
formance measurement and effectiveness. The second section is a description of
the research methods, including how evaluation teams assigned scores to each
program. In the third section, we summarize the findings. The final discussion and
conclusion sections provide an interpretation of findings in terms of prior research.
Performance Measurement and Program Effectiveness 597

A MODEL OF HOW PERFORMANCE MEASUREMENT

CONTRIBUTES TO EFFECTIVENESS

The proposed model illustrates a series of hypothesized relationships between

performance measurement and effectiveness (See Figure 1). According to the
model, performance measurement has both indirect and direct impacts on
performance. The indirect impact occurs by way of enhancing organizational
learning. Performance measurement provides feedback, which can trigger
problem solving and organizational learning.[26] The direct pathway captures
any other residual means by which performance measurement influences
effectiveness. Underlying the performance measurement-effectiveness rela-
tionship is organizational competence. Organizations with greater competence
should be more effective at performance measurement, organizational
learning, and achieving program results. In order to control for this influence,
and thereby obtain accurate parameter estimates, the influence of organiza-
tional competence is incorporated into the model. Each of the model pathways
is described in greater depth below.

Performance Measurement-Organizational Learning

One possible way that performance measurement might contribute to effec-

tiveness is by stimulating organizational learning. Performance measurement
provides feedback, which as psychologists have shown, contributes to learn-
ing in individuals.[27–29] Much of the research by psychologists on feedback is
conducted in the following manner: the researchers give the subjects simple

Learning
Objectives

Performance Results
Indicators
Measurement

Data
Competence

Clients Resources

Figure 1. Model of Performance Measurement and Effectiveness.

598 Mausolff and Spence

tasks to perform, such as drawing a line or pulling a lever. The subjects in the
control group are not given feedback about their performance, while those in
the treatment group are given feedback for a short period. The superior perfor-
mance of the treatment group that persists, after feedback is withdrawn, is
considered evidence of a learning effect.
It is reasonable to hypothesize that feedback could also stimulate learning
at the organizational level. Feedback is portrayed as central to learning in a
number of conceptual, theoretical works on organizational learning.[30–32]
The role of feedback in organizational learning is also supported in case
studies,[33] organization simulations,[34,35] and in longitudinal research.[36]
In addition to the literature discussing feedback in general, there is also
research on the role of performance measurement, specifically, in stimulating
learning. Mausolff's research on organizational learning from performance
measurement in employment services agencies provides detailed descriptions
of this learning process.[26] In the studied agencies, performance measurement
triggered awareness of performance gaps. In response, organization members
initiated problem solving activities, including developing interpretations for
the causes of the performance gaps, sharing these interpretations, searching
for additional information to better understand the presenting problems,
selecting solutions, and implementing solutions. At times, the problem was cor-
rected with a solution that involved a new theory-of-action. In such instances,
organization members were learning from performance measurement.[37]
Overall, research at the individual and organizational levels supports the
hypothesis that feedback contributes to learning. Since performance measurement
is a form of feedback, the following hypothesis is plausible.

Hypothesis 1: As implementation of performance measurement improves,

there will be an increase in organizational learning.

Organizational Learning-Effectiveness

Assuming that organizations can learn from performance measurement, what

is the evidence that this learning results in greater effectiveness? For learning
to improve performance, the newly acquired practices must be superior to the
old ones. While this condition should not be insurmountable, according to
some observers “. . . most new ideas are bad ones.”[38]
Another potential obstacle is the competency trap.[39] This obstacle can
develop when an organization receives favorable feedback from an inferior
practice, leading it to retain the practice and accumulate more experience with
it. This experience can generate enough competency that the organization
does not adopt superior approaches with which it has less experience.[39]
Despite these potential obstacles, there is evidence to support the
learning – effectiveness relationship. Denton conducted research on the
Performance Measurement and Program Effectiveness 599

learning practices of five large organizations.[40] Those that were closest to the
ideal of being learning organizations, with respect to strategy, structure, and
culture, had the highest performance. Yeung, Ulrich, Nason, and Von Glinow
studied over 400 business firms and identified four learning styles: experimen-
tation, competency acquisition, benchmarking, and continuous improve-
ment.[41] Of these, the experimentation approach had the highest performance
(as measured by competitiveness, innovativeness, and new product innova-
tion). If the style of learning matters, then by extension, learning matters. These
findings all help to establish the plausibility of the following hypothesis.

Hypothesis 2: Greater organizational learning contributes to improved

performance.

Performance Measurement-Effectiveness

There are other ways, in addition to learning, by which performance measurement

can contribute to effectiveness. In case studies of agencies with well-designed
systems, the performance measures can help to focus employees on the crite-
ria by which they will be judged.[13,42] The potential for performance measure-
ment systems to focus organization efforts is also supported by survey
research. In Poister and Streib's survey of 674 U.S. municipalities, 28.0 percent
of respondents reported that performance measurement increased employee
focus on organizational goals to a substantive degree and 39.9 percent of them
reported a moderate impact on focus.[15] The impact appears to be smaller in
Berman and Wang's survey of U.S. counties, in which just 37.2 percent of the
respondents agree or strongly agree with the statement that performance mea-
surement clarified agency goals and objectives.[16]
Increased motivation is another potential means by which performance
measurement can enhance effectiveness. For the motivational benefits of per-
formance measurement to be realized, a performance measurement system
must include consequences.[42] A performance measurement system can help
to inform managers about the behavior of employees. Employees might,
therefore, work harder to obtain the rewards administered by managers or
work harder to avoid the negative consequences of low performance. Modest
motivational impacts are reported in surveys of government officials. In U.S.
municipalities, the motivational benefits of performance measurement are
reported to be substantial by 9.6% of respondents and moderate by 33.3% of
them.[15] These results are consistent with a survey of U.S. counties in which
31.5 percent of respondents agree or strongly agree with the statement that
performance measurement increased commitment to excellence.[16] The
reviewed studies indicate benefits in focus or motivation for between 32 and 68
percent of respondents. These findings suggest other mechanisms, in addition to
learning, by which performance measurement could enhance performance.
600 Mausolff and Spence

There is also the potential for negative impacts from performance mea-
surement. The literature on management control systems in business firms
suggests a number of distortions that can arise from performance measurement,
including tunnel vision, suboptimization, myopia, ossification, gaming, and
misrepresentation.[18–21] Many of these distorting effects have also been observed
in government agencies.[17,43,44] In Poister and Streib's survey research, 21.1
percent of municipalities report modest goal displacement from performance
measurement and 1.5 percent report substantial goal displacement.[15]
Despite the possible negative impacts of performance measurement, the
potential for improved focus and motivation suggest the possibility of
improved effectiveness.

Hypothesis 3: Improved implementation of performance measurement has a

direct positive impact on effectiveness (independent of the learning benefits
from performance measurement).

Organizational Competence-Performance Measurement

Mathis and Jackson define individual competencies as basic characteristics,

such as knowledge, skills, and abilities, “that can be linked to enhanced per-
formance by individuals or teams.”[45] Organizational competence can be
defined as the sum of the competencies of the individual members, plus or
minus an allowance for how well members work together to integrate their
knowledge, skills, and abilities. Organizational competence is added to the
model because it is likely to be an underlying factor correlated with program
effectiveness. In a model testing the relationship between performance mea-
surement, organizational learning, and program results, any observed correla-
tions could be because quality performance measurement has a causal impact
on learning and effectiveness, or such correlations could simply reflect an
underlying shared variable. The basic competence that allows an organization
to do well at performance measurement may also be what allows it to do better
at learning from performance measurement and to achieve better program
results. Hence, any observed correlations between performance measurement,
learning, and results, could simply reflect an underlying organizational compe-
tence and not an inherent, instrumental benefit of performance measurement.
It is assumed that the types of competencies needed to develop appropriate
program objectives, indicators, and data gathering systems, may be limited
among human service agencies and, therefore, may distinguish low and high
performers.
The importance of certain basic competencies for performance measurement
has been supported in survey research studies.[15,16] Respondents in municipal
government reported “commonly” having trouble with measuring the quality
of programs and services (26.5%); keeping performance measures current and
Performance Measurement and Program Effectiveness 601

up to date (12.1%); and compiling and distributing data from the performance
measurement system in a timely manner (10.8%).[15]
The competence-performance measurement relationship was directly
tested in Berman and Wang’s research on performance measurement in
counties.[16] The researchers identified a number of competencies that are
significantly correlated with the use of performance measurement: the ability
to develop outcome measures, collect performance measurement data in a
timely way, and assess the validity of performance measures. These findings
all support the following hypothesis.

Hypothesis 3: Organizational competence is positively correlated with effective

implementation of performance measurement.

Competence — Organizational Learning

If performance measurement demands certain competencies, it would be

expected that learning from performance measurement would be at least as
demanding. The role of competencies in organizational learning, in general,
was explored by Yeung and associates.[41] Based on research in a wide
range of firms, the researchers proposed three competencies as essential for
organizational learning:

1. ability to generate ideas,

2. ability to generalize ideas, and
3. ability to identify learning disabilities.

There is also research on the specific challenges of organizational learn-

ing from performance measurement. Berman and Wang's survey research
identifies two competencies that appear to be relevant for organizational
learning from performance measurement: ability to analyze performance data
and ability to compare performance with other departments and jurisdic-
tions.[16] Each of these competencies is significantly correlated with the use of
performance measurement in U.S. municipalities. Overall, the existing
research provides support for the following hypothesis.

Hypothesis 4: Organizational competence increases organizational learning

from performance measurement.

Competence — Results

According to the resource-based view of the firm, employee competencies are

one of the few sustainable sources of competitive advantage.[46] Empirical
602 Mausolff and Spence

research in private businesses supports the competence-performance relation-

ship.[47,48] There appears to be little research on this relationship in nonprofit
organizations, but it is natural to assume that it is also important there as well.
Given the low salaries paid in the nonprofit sector, it may be more difficult for
these organizations to obtain skilled employees. As a consequence, employee
competence may be an even more serious constraint in nonprofit organizations.

Hypothesis 5: Organizational competence generates greater program

effectiveness.

RESEARCH METHODS

The organizations used to test the model hypotheses are nonprofits that
receive financial support from Metro United Way in Louisville, Kentucky.
Metro United Way is a large United Way affiliate with donations of over $30
million annually. Its 80 person staff serves over 100 health and human services
agencies in a seven county region of Kentucky and Indiana. These agencies
manage approximately 170 programs in the following four sectors:

• Successful children and youth;

• Strong, self-sufficient families;
• Lifelong health and independence; and
• Vital neighborhoods.

In the late nineties, Metro United Way began implementing a perfor-

mance measurement system among its member agencies. Since its inception,
Metro United Way's performance measurement system has had a strong
emphasis on learning and continuous improvement. This emphasis is reflected
in a number of good practices. First, the training that Metro United Way orga-
nized for member agencies provided instruction on how to use performance
data for program improvement. Participants were instructed on questions to
ask of their performance data, such as the following: Does the program have
sufficient scope, duration, and intensity to achieve results? Are the right activ-
ities being performed to achieve outcomes? Are influencing factors accounted
for? Participants also learned how to use TQM tools such as flow charting and
cause-effect diagrams. A learning emphasis is also indicated by the instrument
used to evaluate programs for funding renewal. In this instrument, learning
from performance measurement is given just as much weight as good program
results. This heavy learning emphasis most likely distinguishes Metro United
Way's performance measurement system from others. It is therefore possible
that the results generated from this population of programs will not generalize
well to other performance measurement systems with less of a learning
emphasis.
Performance Measurement and Program Effectiveness 603

Data Collection

The data used to test the model was gathered by 13 volunteer evaluation
teams. Each team was led by a volunteer and a Metro United Way staff member
and was comprised of between 12 and 25 members. Beginning in 2000, Metro
United Way has had evaluation teams assess all of the programs in its portfolio
every two years. The data used in this study is based on assessments conducted
in 2000 and 2002.
The program scoring process involves just a few simple steps. Prior to the
evaluation, the focal agency develops a program funding proposal consisting of:

• a description of clients served and their strengths,

• program objectives,
• indicators for measuring these objectives,
• data collected using these indicators,
• next year's results targets, and
• a description of the program activities and outputs and how these relate to
program objectives.

After studying the funding proposals, the evaluation team meets with the man-
agement and the Board representatives. At this point, the evaluation team can
seek clarification on any issues or gaps in the funding proposal. The evaluation
team then scores the program using a standardized evaluation form. There are
variations in the way each team determines these scores. Some teams simply
average each evaluator's scores, while others attempt to reach a consensus
score for each category. The scores in each category are weighted and then
added together to yield a single program score. The validity and reliability of
the scoring is enhanced by training the evaluators.
The criteria Metro United Way uses to evaluate the programs is quite dif-
ferent from other United Way affiliates. Other affiliates that we have worked
with use evaluation criteria based on management best practices, e.g. use of
strategic planning, job descriptions for employees, and appropriate by-laws.
Metro United Way has taken a very different approach. Instead of adherence
to a wide range of specific practices, the evaluation criteria focus primarily on
performance measurement, learning, and achieving results. Thus, rather than
measuring numerous processes, Metro United Way is more focused on learning
and results.
The scores assigned by the evaluation teams in each category are used to
test the proposed structural equation model. These scores should have higher
validity and reliability than those ordinarily obtained in survey research. In
much survey research, the scores for an entire organization are filled out by
just one of its members. By comparison, in this research, each score reflects
the input of 12 to 25 trained evaluators and, therefore, benefits from the
consideration of multiple perspectives. These evaluators also have the benefit
604 Mausolff and Spence

of being able to compare the program being evaluated with the others they
have previously assessed. Finally, the data collection method does not require
heroic assumptions about the cognitive abilities of respondents. In existing
survey research, the respondents answer questions such as whether performance
measurement “improved service quality” or enhanced the organization’s “ability
to achieve improvements…” Such questions require respondents to make
complex attributions about the causes of observed improvements in their
programs. Is it reasonable to assume that respondents have the data and the
training for judging causality? The present study uses a different approach.
The external evaluators merely assess how well each program has performed
on each dimension of the evaluation instrument. They do not need to make
attributions about the causal factors contributing to the results. Instead, possible
relationships between better quality performance measurement and program
results are assessed statistically. If performance measurement is not important
for effectiveness, then doing it better should not make any difference for orga-
nizational learning and results. Conversely, if performance measurement is
important, then when it is performed well, there should be a corresponding
improvement in organizational learning and program results.
A weakness of the data collection method is that the resulting data do not
have normal distributions. The average score on a 0–4 scale ranged from 2.7
to 3.5 for year 2000, and from 2.9 to 3.7 for year 2002. (See Tables 1 and 2).

Table 1. Descriptive Statistics, Year 2000 Data

Variable Mean Std Dev Skewness Kurtosis

Objectives 3.309 0.789 −1.300 2.159

Indicators 3.069 0.835 −0.862 0.576
Data 2.813 0.971 −0.545 −0.242
Learning 3.058 0.896 −1.051 1.088
Results 2.718 0.948 −0.581 0.182
Clients 3.501 0.748 −1.781 3.030
Resources 3.143 0.870 −0.773 −0.016

Table 2. Descriptive Statistics, Year 2002 Data

Variable Mean Std Dev Skewness Kurtosis

Objectives 3.346 3.342 −0.859 −0.404

Indicators 3.223 3.219 −1.010 0.846
Data 3.050 3.044 −0.662 −0.431
Learning 3.104 3.098 −0.570 −0.552
Results 2.934 2.928 −0.700 0.209
Clients 3.737 3.735 −2.284 5.948
Resources 3.211 3.206 −0.614 −0.527
Performance Measurement and Program Effectiveness 605

As a result, the data are negatively skewed with statistically significant skewness
scores ranging from −0.55 to −1.78 (year 2000) and −0.61 to −2.28 (year 2002).
Although high, these deviations from normality are, with the exception of the
clients variable in year 2002, below the recommended maximum threshold for
maximum likelihood estimation techniques.[49] As the level of skewness
increases,

1. the chi-square goodness-of-fit test can become less accurate and reject too
many true models, and
2. the parameter estimates can become biased and provide too many significant
results.[49]

Operationalization of the Constructs

The constructs in the model consist of both observed and unobserved vari-
ables. The observed variables are operationalized based on their description in
the evaluation instrument. The evaluators use these descriptions when assigning
scores to each program. The unobserved (or latent) variables are constructed
from these observed variables (See Figure 1 above). A description of each
construct in the model is provided below.

Performance Measurement

Performance measurement is not measured directly by the evaluation teams.

In the proposed model, performance measurement is an unobserved variable
consisting of three observed variables: program objectives, indicators, and data
gathering system. In determining the score for “objectives,” the evaluation team
assessed the extent to which program objectives

1. demonstrate significant or meaningful improvement in the lives of clients, and

2. are logically connected to one another.

The score for “indicators” is based on the extent to which the chosen measures
match the objectives and signal that the objective has been achieved. The
“data collection” score depends on how well the data collection instrument
provides credible data for all indicators, is usable for measuring program
effectiveness, and is usable for program improvement.

Learning

Learning is directly assessed by the evaluation teams. In the evaluation instrument,

learning is defined in terms of new insights gained from performance measure-
ment and whether the staff are taking specific steps to improve performance.
606 Mausolff and Spence

Results

Results and effectiveness are treated as equivalent terms in this study. Results are
operationalized primarily by the extent of improvement in client conditions attrib-
utable to the program. The evaluation instrument also includes, as one consider-
ation in scoring, the quality of the explanation of the factors affecting the results.

Competence

Competence is an unobserved variable consisting of two observed variables:

understanding of program clients and maximization of community resources.
In scoring the program clients variable, the evaluators assess the extent to
which the agency demonstrates a full understanding of the strengths and
conditions of the client population. The community resources variable is
scored based on the diversity of program funding, coordination with other orga-
nizations, and the use of volunteers and/or in-kind support to stretch resources.
These two variables reflect demanding and essential tasks for nonprofit agencies.
Our intent with these two variables is to tap underlying competencies that are
relevant for other organization tasks, such as performance measurement, organi-
zational learning, and achieving program results. Such competencies could
include the ability to coordinate, interact constructively, and reason inductively.

Data Analysis

The model relationships were tested using the maximum likelihood estimation
method in Amos 4.0 software. Maximum likelihood estimation is recom-
mended when the variable “distributions are not substantially nonnormal.”[49]

FINDINGS

The findings consist of

1. how well the overall model fits the data, and

2. the significance of the individual relationships in the model.

The test of the model is especially stringent in that it involves two tests: one
with year 2000 data and a second with year 2002 data.

Overall Model Fit

No one fit index is sufficient for evaluating overall model fit.[50,51] Researchers
therefore encourage reporting a range of fit indexes.[52] Statistics from both
Performance Measurement and Program Effectiveness 607

absolute and incremental fit indexes are reported here. Most of the overall fit
indexes are normed so that scores close to 1 are ideal and scores above .90 are
considered reasonable.[53]

Absolute Fit Indexes

Absolute fit refers to the degree to which the covariances implied by the
model match the observed covariances.[53] The scores for two absolute fit
indexes, the relative chi-square and the GFI, are reported in Table 3. There is
no precise significance limit for the relative chi-square. Researchers have sug-
gested, as maximum acceptable limits, scores ranging from three to five.[54]
The relative chi-square scores of 3.226 (year 2000 data) and 2.451 (year 2002
data) are therefore on the borderline, indicating relatively poor fit due to
restrictions placed on the model.
The goodness-of-fit index (GFI) measures the amount of variances and
covariances explained by the model.[53] It is similar to the r-square widely
used in multiple regression analysis.[53] The GFI scores above .95 for both the
year 2000 and year 2002 data indicate excellent model fit.

Incremental Fit Indexes

Incremental fit indexes measure the adequacy of a model in relation to a baseline

“worst fitting” model. The high scores of the model, well above .90, for the
Tucker-Lewis Index (TLI), the incremental fit index (IFI), and the comparative
fit index (CFI), suggest excellent overall model fit (See Table 4).

Table 3. Absolute Fit Indices

Statistic 2000 2002

Chi-square 32.257 24.510

df 10 10
p .000 .006
Relative Chi-square (X2/df) 3.226 2.451
GFI 0.956 0.958

Table 4. Incremental Fit Indices

Statistic 2000 2002

Tucker-Lewis Index (TLI) 0.910 0.940

Incremental fit index (IFI) 0.958 0.972
Comparative fit index (CFI) 0.957 0.971
608 Mausolff and Spence

Parameter Estimates

For the individual parameter estimates, the results are mixed. The learning
pathway is supported only for the year 2002 data, while the direct, perfor-
mance measurement-results relationship is supported for both years (See
Tables 5 and 6 below).
We hypothesized that performance measurement would improve results
by enhancing organizational learning (See Figures 2 and 3 below). This
hypothesis was not supported with the year 2000 data. However, if the model
is analyzed without the latent variable for organizational competence, perfor-
mance measurement contributes to learning in both 2000 and 2002. Therefore,
the organizations that do better at performance measurement also do better at
learning. However, for year 2000, this correlation is due entirely to underlying
competence, rather than an independent benefit of performance measurement.
For 2002, even after controlling for competence, there is still an independent,
incremental learning benefit from performance measurement.
The performance measurement-results path tests explanations, other than
learning, by which performance measurement could contribute to results.
These other mechanisms could include improved focus and motivation.[15,16]

Table 5. Unstandardized Parameter Estimates, Year 2000, n = 195

Variables: Estimate Std. Error

Learning <------- Performance_Measurement 0.039 0.644

Results <------------------------------- Learning 0.172 0.156
Results <--------- Performance_Measurement 1.766** 0.687
Performance_Measurement <--- Competence 0.927** 0.163
Learning <------------------------- Competence 1.123 0.749
Results <--------------------------- Competence −0.892 0.887

*p < .05, **p < .01.

Table 6. Unstandardized Parameter Estimates, Year 2002, n = 167

Variables: Estimate Std. Error

Learning <------- Performance_Measurement 0.492* 0.214

Results <------------------------------- Learning 0.240* 0.118
Results <--------- Performance_Measurement 0.903** 0.163
Performance_Measurement <--- Competence 1.272** 0.367
Learning <------------------------- Competence 1.322* 0.587
Results <--------------------------- Competence 0.101 0.475

*p < .05, **p < .01.

Performance Measurement and Program Effectiveness 609

error 7

.31 1 .40
error 6 Objectives Learning
error 8
1 1.00
.04 .17
.29 1 .38
error 5 1.12 Performance 1.77 Results
Indicators
1 Measurement
1.26
.40 1.12
1 .08 .93 –.89
error 4 Data
1 error 3 Competence

1.00 .31 .82

Clients Resources

1 .28 1 .57
error 2 error 1

Figure 2. Unstandardized Parameter Estimates, Year 2000.

error 7

.26 1 .31
error 6 Objectives Learning error 8
1 1.00
.49 .24
.23 1 .38
error 5 1.16 Performance .90 Results
Indicators
1 Measurement
1.33
.20 1.32
1 .21 1.27 .10
error 4 Data
1 Competence
error 3

1.00 .09 1.23

Clients Resources

1 .17 1 .45
error 2 error 1

Figure 3. Unstandardized Parameter Estimates, Year 2002.

In the present sample, these other, residual, mechanisms are important. There
is a positive relationship between performance measurement and results for
both 2000 and 2002 (significant at the .01 level).
610 Mausolff and Spence

We hypothesized that competence is an important factor explaining an

organization's ability to perform performance measurement, to learn from per-
formance measurement, and to achieve program results. Of these relation-
ships, the correlation between competence and performance measurement
is the only one that is significant for both 2000 and 2002 (at the .01 level).
The competence-learning relationship is significant only in 2002, while the
competence-results relationship is insignificant both years.

DISCUSSION

The Learning Pathway (Hypotheses 1 and 2)

According to the literature on organizational learning, feedback would be

expected to contribute to learning.[55] In response to performance problems
surfaced by the performance measurement data, there should be problem solv-
ing activities leading to new insights and new practices, i.e., organizational
learning.[37] To the extent that these new practices are superior, we would also
expect to see improved performance as a result of this data-driven learning
process. However, this learning pathway was supported only in the 2002 data.
There are a few possible explanations for the inconsistency of findings
from 2000 to 2002. One possible explanation is expertise with implementa-
tion. It is possible that by 2002, the organizations were developing better
expertise at implementing corrective action plans; therefore, learning was
making a more direct contribution to results. Another possible explanation is
the relatively small size of each new incremental improvement. The improve-
ments generated by learning may be just a small percentage of the overall
good practices needed to achieve good results. In year 2000, the Metro United
Way's performance measurement system was still just in its beginning phases.
By 2002, the incremental benefits of learning may have accumulated suffi-
ciently to account for a significant portion of program results.

The Direct Performance Measurement-Results Pathway (Hypothesis 3)

In the present sample of programs, there is a significant, positive correlation

between performance measurement and effectiveness. This clear result stands
in contrast to the existing quantitative work on performance measurement,
which has found only weak or inconsistent correlations with performance.
One way to interpret the inconsistent results of prior research is with the
explanation that the impact of performance measurement on effectiveness
depends on how well the performance measurement system is designed. This
research tests this hypothesis by measuring the correlation between the quality
of performance measurement and program results. The findings support this
Performance Measurement and Program Effectiveness 611

hypothesis. In the studied programs, better performance measurement is

correlated with positive program results, even after controlling for the impact
of organizational competence and organizational learning.

The Control Variable: Organizational Competence

(Hypotheses 3, 4, and 5)

The results support the validity of organizational competence as an important

factor mediating the relationships between performance measurement, organi-
zational learning, and results. The competence-performance measurement
relationship is significant for both 2000 and 2002, supporting the existing survey
research findings.[15,16] The competence-learning relationship is supported
only in 2002. This mixed result is surprising, as it would be expected that
organizational learning would be even more demanding on the competence of
organizations than performance measurement.
The insignificant relationship between competence and results is also sur-
prising. However, if the pathways between competence and performance mea-
surement and competence and organizational learning are eliminated from the
model, the competence-results relationship is statistically significant. This
finding suggests that competence contributes to results indirectly, by improving
both performance measurement and organizational learning. The fact that the
competence-results correlation disappears when performance measurement
and organizational learning are added into the analysis is quite profound in
its implications. This result suggests that the sole benefit of organizational
competence in the studied programs is to establish good performance mea-
surement systems and to use them for learning.

CONCLUSION

A structural equation model was used to test the impact of performance mea-
surement on effectiveness and the importance of organizational learning in
this process. The relatively large sample size of approximately 170 health and
human service programs, the independent scoring of programs, and replica-
tion of the test with two datasets, provided a relatively rigorous test of the
hypotheses.
The results were mixed. The organizational learning hypothesis was only
partially supported. It is statistically significant in 2002, but not 2000. One
explanation for this result is that the organizations needed additional time for the
incremental benefits of data-driven learning to add up to significant program
impacts.
The direct performance measurement-effectiveness relationship is
strongly supported in this study. The significant and positive correlations for
612 Mausolff and Spence

the model results with both the 2000 and 2002 data sets, provides strong evi-
dence for the efficacy of performance measurement. These results support the
promotion of performance measurement by the United Way of America and
other nonprofit organizations. Since this study is based entirely on nonprofit
health and human service organizations, caution should be exercised in gener-
alizing these results to government and private business.
The results also support the hypothesis that factors other than learning can
be important in explaining the impact of performance measurement on results.
These other mechanisms could include greater motivation and improved
focus. It would therefore be useful to conduct additional research on these
other mechanisms, to better understand their impact.

REFERENCES

1. Ammons, D. N. Performance measurement and managerial thinking.

Public performance and Management Review 2002, 25 (4), 344–347.
2. Berman, E. How useful is performance measurement. Performance mea-
surement and managerial thinking. Public Performance and Management
Review 2002, 25 (4), 347–353.
3. Bernstein, D. J. Comments on Perrin's ‘Effective Use and Misuse of
Performance Measurement.’ The American Journal of Evaluation 1999,
20 (1), 85–94.
4. Poister, T. H. Performance Monitoring. Lexington Books: Lexington,
MA, 1983.
5. Frank, H. A., & D'Souza, J. Twelve years into the performance measurement
revolution: Where do we need to go in implementation research. International
Journal of Public Administration 2004, 27 (8/9), 701–718.
6. Mandell, L. M. Performance Measurement and Management Tools in
North Carolina Local Government: Revisited. Public Administration
Quarterly 1997, 21 (1), 96–127.
7. Perrin, B. Effective use and misuse of performance measurement. The
American Journal of Evaluation 1998, 19 (3), 367–379.
8. Perrin, B. Performance measurement: Does the reality match the rhetoric?
A rejoinder to Bernstein and Winston. American Journal of Evaluation
1999, 20 (1), 101–112.
9. See, for example, Mayne, J. & Zapico-Goni, E. Monitoring performance
in the public sector. New Brunswick, NJ: Transaction Publishers, 1997.
10. Martin, L. L., & Kettner, P. M. Measuring the performance of human ser-
vice programs. Thousand Oaks, CA: Sage, 1996.
11. Plantz, M. C., Greenway, M. T., & Hendricks, M. Outcome measure-
ment: Showing results in the nonprofit sector. In Using performance mea-
surement to improve public and nonprofit programs; Newcomer, K., ed.;
New Directions for Evaluation 1997, 75, 15–30.
Performance Measurement and Program Effectiveness 613

12. Ammons, D. N. Benchmarking as a performance management tool: Experi-

ences among municipalities in North Carolina. Journal of Public Budget-
ing, Accounting & Financial Management 2000, 12 (1), 106–124.
13. Wholey, J. S., & Newcomer, K. E. Clarifying goals, Reporting results. In
Using performance measurement to improve public and nonprofit
programs; Newcomer, K., ed.; In New Directions for Evaluation 1997,
75, 91–98.
14. U. S. General Accounting Office. Managing for Results: State Experi-
ences Provide Insights for Federal Management Reform. GAO/GGD-95–
22. U.S. General Accounting Office: Washington, DC, 1994.
15. Poister, T. H., Streib, G. Performance measurement in municipal govern-
ment: Assessing the state of the practice. Public Administration Review
1999, 59 (4), 325–335.
16. Berman, E., Wang, X. H. Performance measurement in U. S. counties:
Capacity for reform. Public Administration Review 2000, 60 (5), 409–420.
17. Blau, P. M. The dynamics of bureaucracy: A study of interpersonal relations
in two government agencies. Chicago: University of Chicago Press; 1963.
18. Briers, M. & Hirst, M. The role of budgetary information in performance
evaluation. Accounting, Organizations, and Society 1980, 15 (4), 373–398.
19. Grizzle, G. A. Performance measurement and dysfunction: The dark side
of quantifying work. Public Performance and Management Review 2002,
25 (4), 363–369.
20. Jaworski, B. J., & Young, S. M. Dysfunctional behavior and management
control: An empirical study of marketing managers. Accounting, Organi-
zations, and Society 1992, 17 (1), 17–35.
21. Merchant, K. A. The effects of financial controls on data manipulation
and management myopia. Accounting, Organizations, and Society 1990,
15 (4), 297–313.
22. Kanter, R. M. & Summers, D. V. Doing well while doing good: Dilemmas of
performance management in nonprofit organizations and the need for a
multiple-constituency approach. In The nonprofit sector: A research
handbook. New Haven, CT: Yale University Press, Powell, W., ed.; 1987,
pp. 154–166.
23. Kaplan, R. S., & Norton, D. P. Strategic learning and the balanced score-
card. Strategy and Leadership 1996, 24 (5), 18–29.
24. Kravchuk, R. S., & Schack, R. W. Designing effective performance man-
agement systems under the Government Performance and Results Act of
1993. Public Administration Review 1996, 56 (4), 348–358;
25. Zapico-Goni, E., & Mayne, J. Performance monitoring: Implications for the
future. In Mayne, J. & Zapico-Goni, E., eds. Monitoring performance in the
public sector. New Brunswick, NJ: Transaction Publishers, 1997, pp. 253–276.
26. Mausolff, C. Organizational Learning and Performance Monitoring: A
Social Action Approach. Proceedings of the 2001 Academy of Manage-
ment Conference, Washington, DC, August 3–8, 2001.
614 Mausolff and Spence

27. Annett, J. Feedback and human behavior: The effects of knowledge of

results, incentives, and reinforcement on learning and performance.
Baltimore: Penguin Books; 1969.
28. Salmoni, A. W., Schmidt, R. A., Walter, C. B. Knowledge of results and
motor learning: A review and critical appraisal. Psychological Bulletin
1984, 95 (3), 355–386.
29. Winstein, C. J., & Schmidt, R. A. Reduced frequency of knowledge of
results enhances motor skill learning. Journal of Experimental Psychol-
ogy 1990, 16 (4), 677–691.
30. Cyert, R. M., & March, J. G. A behavioral theory of the firm. Englewood
Cliffs, NJ: Prentice-Hall, 1963.
31. Ford, J. D., & Baucus, D. A. Organizational adaptation to performance
downturns: An interpretation-based perspective. Academy of Manage-
ment Review 1987, 12 (2), 366–380.
32. Friedlander, F. Patterns of individual and organizational learning. In
Srivastava, S., ed., The executive mind. San Francisco: Jossey-Bass, 1983,
pp. 192–220.
33. Argyris, C. Knowledge for Action: A guide to overcoming barriers to
organizational change. San Francisco: Jossey-Bass, 1993.
34. Cangelosi, V. E., Dill, & W. R. Organizational learning: Observations
toward a theory. Administrative Science Quarterly 1965, 10 (2), 175–203.
35. Miles, R. H., & Randolph, W. A. Influence of organizational learning
styles on early development. In Kimberly, J. R., Miles, R. H., eds.,
The organizational life cycle. San Francisco: Jossey-Bass: 1980;
pp. 44–82.
36. Van de Ven, A. H., & Polley, D. Learning while innovating. Organization
Science 1992, 3 (1), 92–116.
37. Mausolff, C. Learning from feedback in performance measurement sys-
tems. Public Performance and Management Review 2004, 28 (1), 9–29.
38. Levitt, B., & March, J. G. Organizational learning. American Review of
Sociology 1988, 14, 319–340.
39. March, J. G. Exploration and exploitation in organizational learning.
Organization Science 1991, 2 (1), 71–78.
40. Denton, J. Organizational learning and effectiveness. New York: Routledge,
1998.
41. Yeung, A. K, Ulrich, D., Nason, S. W., & Von Glinow, M. A. Organi-
zational learning capability. New York: Oxford University Press,
1999.
42. Heinrich, C. J. Do government bureaucrats make effective use of perfor-
mance management information? Journal of Public Administration
Research and Theory 1999, 9 (3), 363–393.
43. Barnow, B. S. The effects of performance standards on state and local
programs. In Manski, G. F., Garfinkel, I., eds. Evaluating welfare and
training programs. Harvard University Press: Boston. 1992, pp. 277–309.
Performance Measurement and Program Effectiveness 615

44. Smith, P. Outcome-Related performance indicators and organizational

control in the public sector. In Holloway, J., Lewis, J., & Mallory, G.,
eds. Performance measurement and evaluation. London: Sage, 1995,
pp. 192–216.
45. Mathis, R. L., & Jackson, J. H. Human resource management. Cincinnati,
OH: South-Western College Publishing, 2000, p. 217.
46. Conner, K. R., & Prahalad, C. K. A Resource-based theory of the firm:
Knowledge versus opportunism. Organization Science 1996, 7 (5)
477–501.
47. Heffernan, M. M., & Flood, P. C. An exploration of the relationship
between the adoption of managerial competencies, organisational charac-
teristics, human resource sophistication and performance in Irish organi-
zations. Journal of European Industrial Training 2000, 24 (2), 128–138.
48. Julien, P. A., & Ramangalahy, C. Competitive strategy and performance
of exporting SMEs: An empirical investigation of the impact of their
export information search and competencies. Entrepreneurship Theory
and Practice 2003, 27 (3), 227–245.
49. West, S. G, Finch, J. F., & Curran, P. J. Structural equation models with
nonnormal variables: Problems and remedies. In Hoyle, R. H., ed. Struc-
tural equation modeling: Concepts, issues, and applications. Thousand
Oaks, CA: Sage, 1995, pp. 56–75.
50. Joreskog, K. G. Testing structural equation models. In Bollen, K. A., &
Long, J. S., eds., Testing Structural Equation Models. Thousand Oaks,
CA: Sage, 1993, pp. 294–316.
51. Hoyle, R. H., Panter, A. T. Writing about structural equation models. In
Hoyle, R. H., ed. Structural equation modeling: Concepts, issues, and
applications. Thousand Oaks, CA: Sage, 1995, pp. 158–176.
52. Maruyama, G. M. Basics of structural equation modeling. Thousand
Oaks, CA: Sage, 1998.
53. Hoyle, R. H., & Panter, A. T. Writing about structural equation models.
In Structural equation modeling: Concepts, issues, and applications;
Hoyle, R. H., eds. Thousand Oaks, CA: Sage, 1995, pp. 158–176.
54. Arbuckle, J. L., & W. W. Appendix C: Measures of fit. Amos 4.0 user's
guide. Chicago: Smallwaters Corporation, 1995.
55. Van de Ven, A. H., & Polley, D. Learning while innovating. Organization
Science 1992, 3 (1), 92–116.

View publication stats