Download the full version of the ebook at ebookfinal.

com

Assessing Impact Evaluating Professional Learning

3rd Edition Joellen Killion

https://ebookfinal.com/download/assessing-impact-evaluating-
professional-learning-3rd-edition-joellen-killion/

OR CLICK BUTTON

DOWNLOAD EBOOK

Download more ebook instantly today at https://ebookfinal.com

Instant digital products (PDF, ePub, MOBI) available
Download now and explore formats that suit you...

Assessing Children s Needs and Circumstances The Impact of

the Assessment Framework 1st Edition Hedy Cleaver

https://ebookfinal.com/download/assessing-children-s-needs-and-
circumstances-the-impact-of-the-assessment-framework-1st-edition-hedy-
cleaver/
ebookfinal.com

MMPI A Assessing Adolescent Psychopathology 3rd Edition

Robert P. Archer

https://ebookfinal.com/download/mmpi-a-assessing-adolescent-
psychopathology-3rd-edition-robert-p-archer/

ebookfinal.com

Learning Flash Cs4 Professional 1st ed Edition Rich Shupe

https://ebookfinal.com/download/learning-flash-cs4-professional-1st-
ed-edition-rich-shupe/

ebookfinal.com

The Diversity Scorecard Evaluating the Impact of Diversity

on Organizational Performance Improving Human Performance
1st Edition Edward E. Hubbard
https://ebookfinal.com/download/the-diversity-scorecard-evaluating-
the-impact-of-diversity-on-organizational-performance-improving-human-
performance-1st-edition-edward-e-hubbard/
ebookfinal.com
Professional C 3rd ed Edition Simon Robinson

https://ebookfinal.com/download/professional-c-3rd-ed-edition-simon-
robinson/

ebookfinal.com

Defining and Assessing Adverse Environmental Impact from

Power Plant Impingement and Entrainment of Aquatic
Organisms 1st Edition D.A. Dixon
https://ebookfinal.com/download/defining-and-assessing-adverse-
environmental-impact-from-power-plant-impingement-and-entrainment-of-
aquatic-organisms-1st-edition-d-a-dixon/
ebookfinal.com

Assessing for Learning Building a Sustainable Commitment

Across the Institution 1st Edition Peggy L. Maki

https://ebookfinal.com/download/assessing-for-learning-building-a-
sustainable-commitment-across-the-institution-1st-edition-peggy-l-
maki/
ebookfinal.com

Learning to Coach For Personal and Professional

Development 2nd Edition Nicola Stevens

https://ebookfinal.com/download/learning-to-coach-for-personal-and-
professional-development-2nd-edition-nicola-stevens/

ebookfinal.com

Improving Professional Learning Twelve Strategies to

Enhance Performance 1st Edition Alan B. Knox

https://ebookfinal.com/download/improving-professional-learning-
twelve-strategies-to-enhance-performance-1st-edition-alan-b-knox/

ebookfinal.com
Assessing Impact
Third Edition
To educators worldwide who strive to improve their practice
each day so their students achieve their dreams.
 Assessing Impact
Evaluating Professional Learning
Third Edition

Joellen Killion
Foreword by Michael Fullan

A Joint Publication
FOR INFORMATION: Copyright © 2018 by Corwin

Corwin All rights reserved. When forms and sample documents are included, their use is
authorized only by educators, local school sites, and/or noncommercial or nonprofit
A SAGE Company
entities that have purchased the book. Except for that usage, no part of this book may
2455 Teller Road be reproduced or utilized in any form or by any means, electronic or mechanical,
Thousand Oaks, California 91320 including photocopying, recording, or by any information storage and retrieval system,
(800) 233-9936 without permission in writing from the publisher.
www.corwin.com
All trademarks depicted within this book, including trademarks appearing as part of a
screenshot, figure, or other image, are included solely for the purpose of illustration and
SAGE Publications Ltd. are the property of their respective holders. The use of the trademarks in no way
1 Oliver’s Yard indicates any relationship with, or endorsement by, the holders of said trademarks.
55 City Road
London EC1Y 1SP
United Kingdom
Printed in the United States of America
SAGE Publications India Pvt. Ltd.
B 1/I 1 Mohan Cooperative Industrial Area ISBN: 978-1-5063-9595-1
Mathura Road, New Delhi 110 044
India

SAGE Publications Asia-Pacific Pte. Ltd.

3 Church Street
#10-04 Samsung Hub
Singapore 049483

Program Director: Dan Alpert This book is printed on acid-free paper.

Associate Editor: Lucas Schleicher
Editorial Assistant: Mia Rodriguez
Production Editor: Amy Schroller
Copy Editor: Terri Lee Paulsen
Typesetter: C&M Digitals (P) Ltd.
Proofreader: Dennis W. Webb
Indexer: Sheila Bodell
Cover Designer: Scott Van Atta
Marketing Manager: Charline Maher 17 18 19 20 21 10 9 8 7 6 5 4 3 2 1

DISCLAIMER: This book may direct you to access third-party content via web links, QR codes, or other scannable technologies,
which are provided for your reference by the author(s). Corwin makes no guarantee that such third-party content will be available
for your use and encourages you to review the terms and conditions of such third-party content. Corwin takes no responsibility
and assumes no liability for your use of any third-party content, nor does Corwin approve, sponsor, endorse, verify, or certify
such third-party content.
 Contents

Foreword  vii
Michael Fullan
Acknowledgmentsix
About the Author xi

Introductionxiii
1. Evaluation as Normative Practice 1
2. Professional Learning Program Evaluation Overview 7
3. Evaluating Professional Learning 27
4. Assess Evaluability 41
5. Formulate Evaluation Questions 69
6. Construct the Evaluation Framework 85
7. Collect Data 127
8. Organize, Analyze, and Display Data 135
9. Interpret Data 159
10. Report, Disseminate, and Use Findings 177
11. Evaluate the Evaluation 197
12. Shifting Perspectives About Evaluating Professional Learning 203

Appendix A: ESSA Definition of Professional Development 211

Appendix B: Standards to Guide Evaluation of
Professional Learning Programs 215
Appendix C: Logic Model Templates 227
Appendix D: Sample Logic Models 231
Appendix E: Sample Evaluation Planning Tools 235
Appendix F: Sample Evaluation Planning Tool Templates 239
Appendix G: Sample Data-Collection Instruments 243
Appendix H: Step-by-Step Evaluation Guide 267
Glossary273
References285
Index291
 Foreword

I lost count of how many times a state commissioner of education, or a school

district superintendent said to me “ we have so much professional develop-
ment going that I can’t keep track of it, and I have no idea whatsoever if it is
having any impact.” Well, your troubles are over (at least on this dimension);
Joellen Killion has written the bible on assessing professional learning. Assessing
Impact is without a question the definitive book on understanding professional
learning (PL) and determining its worth.
The book is comprehensive and up to date to the minute—capturing ESSA
and the overall purpose of assessing PL in order ” to assist school, district, and
state leadership in examining ongoing evidence for improvement.”. In page
after page, Killion demonstrates how to make evaluation and improvement
seamless. The twelve chapters (and eight appendixes) are replete with insights,
frameworks, tools, case examples, and step-by-step guides for conducting and
using evaluations for understanding and improving PL in action.
In education change there is an age-old problem of how to get inside ”the
black box of implementation”—or as one observer called it ”on the risk of
appraising non-events.” A non-event is something that was supposed to have
happened, was carried out superficially, and then evaluated without getting
inside what really happened (or didn’t happen in actual practice). Such evalu-
ations are impossible to evaluate, but Killion has a solution. She identifies the
black box and then provides us the “‘glass box” of professional learning: what
was supposed to have happened, what actually took place, and how much
impact it had. This is a book that is crystal clear about all aspects of a very com-
plicated phenomenon: professional learning in action.
Killion is very clear about “‘the theory of change” in terms of three action
steps that need to take place and be assessed ranging from assumptions (e.g.,
collaborative learning of new practices), to implementation (support for imple-
menting certain practices), to actual practice change and its impact on student
performance. Every chapter has a valuable rubric to guide action for every step
of the way: how to design the evaluation, how to conduct it, and a data inter-
pretation rubric. No detail is missed: how to attribute data, seek permission,
manage costs, manage data collection and reporting, avoid pitfalls, organize
and display data, interpret data, report and disseminate, and even how to eval-
uate the evaluation. It is all there in readable detail with case illustrations and
displays, and in every place a rubric. You need to figure out how to interpret

vii
viii  Assessing Impact
what is found? The data collection rubric in Chapter 7 is a gem—as is the data
interpretation rubric, and the portrayal of the types of data analysis available.
Table 10.2 provides a frame for the possible components of evaluation reports,
and so on.
In addition to being comprehensive about a very complicated phenomenon—
evaluation of a murky and multifaceted topic like PL—Joellen Killion shows us
exactly how the field of PL needs to shift from what it was to what it needs to be
in modern times. I especially liked the Table 12.1 comparing the paradigm shift
on eleven dimensions of what evaluating used to be and what it needs to be. To
take only one of the dimensions: ”feared by stakeholders and participants” to
”embraced by stakeholders and participants.”
In all, Assessing Impact: Evaluating Professional Learning not only fills the gap
with respect to a critical topic that is much neglected, but it overflows the
chasm of poorly or non-conducted treatments of professional learning in
action. This is a book brimming with practical and comprehensive ideas on one
of the least understood topics in education change: how do we know that pro-
fessional learning is worth the investment? Read this book and find out.

—Michael Fullan
Professor Emeritus
OISE/University of Toronto
 Acknowledgments

M any people have contributed in large and small ways to this third-edition
book. The first edition was the result of the vision, passion, and commit-
ment of Hayes Mizell, Learning Forward’s Senior Distinguished Fellow and
formerly the director of programs for student achievement at the Edna
McConnell Clark Foundation. Mizell’s leadership and zealous focus on the only
prize that matters in professional learning—student achievement—have pro-
vided encouragement and resources to assist schools and districts to improve
dramatically the quality and impact of their professional learning. He recog-
nized that effective professional learning had a single purpose: to improve
student success. After a series of initiatives led by the National Staff Development
Council, now Learning Forward, that examined content-specific professional
learning funded by the Edna McConnell Clark Foundation, Mizell realized that
the efforts to link educator learning with student learning were shallow and
limited. He called for action to prepare and support educators with the knowl-
edge, skills, and tools to be proactive in evaluating both the effectiveness and
the impact of their professional learning investments and efforts.
I am indebted to the countless individuals who shared sample evaluation
tools with me, provided examples of evaluations they conducted, shared their
learning about evaluating professional learning, and continue to express frus-
tration with it. It is for those who want to use evidence more effectively to
improve both quality and results that I continue to investigate and invest in this
work. I am particularly grateful to a small team of districts engaged in a small
investigatory study funded by Frontline Research & Learning Institute for their
commitment to be my learning partners in this next phase of the work. I want
to acknowledge and appreciate my colleague, Linda Munger, who serves as a
thinking partner in program design and evaluation. She patiently read every
word and commented on many. I trust her expertise and extensive experience in
evaluation and have had the professional privilege of working alongside her on
multiple initiatives. She provided innumerable insights and suggestions that
helped shape this third edition. She shares my passion for facilitating program
design and evaluation in schools and districts so that professional learning can
achieve its intended outcomes.
I am most grateful to the educators with whom I have shared this book in
various professional learning experiences around the world. They have been
my wisdom partners in this work, applied the steps, and shared their successes

ix
x  Assessing Impact
and challenges with me. I learn more each time I engage with those who care
deeply about being able to evaluate their work objectively and rigorously. Thank
you for your commitment to be responsible, accountable, ethical, and practical
in your effort to strengthen educator and student success.
I am grateful to the leadership team at Corwin for their commitment to
advance this next edition. They are investing in their own study about the
impact of their work and model the importance of evidence-driven decision
making to increase their overall service and support to their clients.
 About the Author

Joellen Killion is senior advisor to

Learning Forward. Since her retirement
from Learning Forward as deputy execu-
tive director, Killion focuses on helping
school systems and states develop and
enact comprehensive professional learn-
ing systems that ensure all educators
have access to highly effective profes-
sional learning that results in student
success. She is particularly passionate
about the role of coaching as a vehicle
for personalizing and differentiating pro-
fessional learning for teachers, princi-
pals, and central office staff. She has led
a number of local, state, and national
initiatives to improve professional learn-
ing, its planning, implementation, and evaluation; served as a consultant to
hundreds of schools and school systems; and is the author and coauthor of
numerous books and articles.

xi
 Introduction

A ssessing Impact: Evaluating Professional Learning guides education practi-

tioners to measure the progress and effects of their efforts in professional
learning. The necessity for evaluation is urgent and straight-forward logic. The
logic for this is simple. Researchers and practitioners agree that the quality of
teaching and school leadership are the single greatest factors influencing stu-
dent success. Professional learning expands and strengthens educators’ capac-
ity for teaching and leadership. To increase capacity, education leaders at the
state, district, and school levels invest in professional learning. To produce pro-
fessional learning’s intended results for educators and students, education
agencies and educators themselves invest in professional learning. To know if
the investment in professional learning produces results, educators evaluate it.
To make improvements in their efforts to strengthen educator capacity, educa-
tion leaders engage in evaluation that is practical, ongoing, and meaningful
and provides the information they want to improve their efforts.
Evaluation is essential to measure the progress and effects of any profes-
sional learning to know if the intended outcomes are achieved and to make
adjustments based on evidence in an expedient, effective, and equitable way.
Emil Posavac notes the natural and essential nature of evaluation: “The prac-
tice of evaluating one’s own effort is as natural as breathing. Cooks taste their
gravy and sauce and basketball players watch to see whether their shots go in.
Indeed, it would be most unwise after turning on the hot water to neglect to
check the water temperature before stepping into a shower,” he states. “At a
basic level, program evaluation is nothing more than applying this common-
sense practice to settings in which organized efforts, called ‘programs,’ are
devoted to helping people who need education, medical treatment, job train-
ing, safe streets, welfare assistance, safety while traveling in airplanes, recre-
ational services, or any of the thousand services provided in a modern
society” (2016, p. 1).
In the nearly two decades since the first initiative began and the decade since
the publication of the second edition, education decision makers, policy makers,
practitioners, evaluators, researchers, and elected officials continue to search for
practical, useful, and feasible ways to measure the effects of professional learn-
ing. Several well-known randomized control trial studies (Garet
et al., 2008; Garet et al., 2010) have failed to show that professional learning

xiii
xiv  Assessing Impact
results in changes in educator practice and stu-
A note about terminology . . . dent achievement. Smaller-scale, non-empirical
studies and program evaluations find positive
For the most part, unless a specific source uses
the term professional development, this effects, yet policy makers at the federal and state
text uses professional learning. The distinc- levels and private and public funders continue to
tion is to emphasize the difference between question the efficacy of professional learning as
the process, experiences, or activities to a viable investment or pathway for substantive
achieve a set of outcomes and the outcomes improvement of schools. The stakes in terms of
themselves. Assessing Impact intends to focus accountability for results continue to grow more
on results, yet acknowledges that assessing significant and the need more profound for edu-
processes provide critical information about cators to assume responsibility for evaluating
how to increase the potential for outcomes. A professional learning.
common mistake in evaluating professional The third edition of this book incorporates
learning or development is to assume that the changes in federal and state policy related to pro-
activities are the outcomes.
fessional learning. It also reflects the increased
accountability for results for investments in pro-
fessional learning. Federal, state, and local policy
makers question the return on their investments in professional learning, partic-
ularly since many studies of the effectiveness of professional learning conducted
over the last several decades demonstrate the inconclusive results. Educators
must do a better job contributing to the body of evidence about professional
learning while researchers continue to examine empirically if and how profes-
sional learning affects educator practice and student learning. Educators who
know from their experience that professional learning is a significant factor in
improving educator effectiveness are called on to provide more valid evidence
about their efforts both to strengthen their own practice and to increase results.
The Every Student Succeeds Act (ESSA), signed into law in December
2015, defines professional development as
“PROFESSIONAL DEVELOPMENT.—The term ‘professional development’
means activities that — ‘‘‘‘(42) PROFESSIONAL DEVELOPMENT.—The term
‘professional development’ means activities that—“(A) are an integral part of
school and local educational agency strategies for providing educators
(including teachers, principals, other school leaders, specialized instructional
support personnel, paraprofessionals, and, as applicable, early childhood edu-
cators) with the knowledge and skills necessary to enable students to succeed
in a well-rounded education and to meet the challenging State academic
standards; and
“(B) are sustained (not stand-alone, 1-day, or short term workshops),
intensive, collaborative, job-embedded, data-driven, and classroom-focused”
(S. 1177-295).
The full definition appears in Appendix A. Following this core definition is
an extensive list of additional features of professional development activities
intended to guide those responsible for decision-making. At this writing, non-
binding federal guidance documents support the implementation of the Every
Student Succeeds Act and the development of state consolidated plans for the
implementation of Title I, II, and V programs.
The revised definition of professional learning in ESSA specifies that profes-
sional learning is intensive (not a stand-alone, 1-day, or short-term experience),
 xv
Introduction  

collaborative, job embedded, data driven, and classroom focused. ESSA expands
allowable uses of Title II funding to focus on equity and excellence. It clarifies
levels of evidence and calls for states to use evidence to select programs with
proven or promising results. In addition, ESSA requires a plan for continuous
improvement of both the plan and its implementation and consultation with a
wide variety of stakeholders in the process of plan development (Revised ESSA
Consolidated Plan Template, 2016).
This edition rests on five guiding principles that collectively provide a ratio-
nale for professional learning leaders to engage in rigorous, ongoing evaluation
of their efforts. The first is the Data standard of Standards for Professional
Learning: Professional learning that increases the effectiveness of every educator
and results for all students uses multiple forms and sources of data about stu-
dents, educators, and systems to plan, implement, and evaluate professional
learning (Learning Forward, 2011). The second is included in ESSA’s definition
of professional development. “Professional development . . . may include activi-
ties that . . . (xi) as a whole, are regularly evaluated for their impact on increased
teacher effectiveness and improved student academic achievement, with the find-
ings of the evaluations used to improve the quality of professional development”
(ESSA SEC. 8002 GENERAL PROVISIONS—DEFINITIONS). The third is Principle 6
from The Learning Educator: A New Vision of Professional Learning: “Impact:
Evaluation strengthens performance and impact” (Hirsh & Killion, 2007, p. 73).
The fourth principle, from the PD Redesign Principles, is “Measuring the impact
of professional learning provides data that are essential to decision-making and
better allocation of resources.” And lastly, the principle “What gets measured,
gets done.” This statement’s origin is unclear, yet its meaning is not. An intent to
measure progress and results signals the importance of a set of actions and may
contribute to increased urgency to act to achieve the outcomes.
This third edition of Assessing Impact: Evaluating Professional Learning empha-
sizes the role of evaluation in continuous improvement, strives to give practi-
tioners more specific and rigorous processes for measuring planning,
implementation, effectiveness, and impact of professional learning; acknowledges
the continued call for accountability of public funds for professional learning; and
encourages states, school systems, and schools to adopt evaluation as a normative
practice to enhance data-driven decision-making processes. While much has
changed since the reauthorization of the Elementary and Secondary School Act of
1965 in 2015 in terms of requirements for states regarding the implementation
and operation of federally supported, mandated programs in the nine Title areas,
what has not changed is the responsibility of educators to be accountable for the
success of every student, make informed, data-driven decisions regarding stu-
dents’ education, and engage in continuous improvement to increase equity and
excellence in education. The law’s purpose is “to ensure that every child achieves”
through grants and subgrants to state and local education agencies to

‘‘(1) increase student achievement consistent with the challenging State

academic standards;
‘‘(2) improve the quality and effectiveness of teachers, principals, and
other school leaders;
xvi  Assessing Impact
‘‘(3) increase the number of teachers, principals, and other school leaders
who are effective in improving student academic achievement in
schools; and
‘‘(4) provide low-income and minority students greater access to effective
teachers, principals, and other school leaders” (ESSA, Sec. 2001).

What has not changed in this edition is the clear message that evaluation
of professional learning without thoughtful and thorough planning for and
resourcing of professional learning is inexcusable and foolhardy, especially if
evaluators seek to measure impact. If done with intentionality and rigor, eval-
uation requires a level of effort and cost that practitioners cannot afford to
waste on insufficiently planned and weakly resourced professional learning
efforts. The process of evaluating begins then with deliberate and comprehen-
sive planning to ensure clarity of outcomes, evidence-based actions, sufficient
resources to sustain professional learning over time, and ongoing formative
assessment to adjust practices and meet unexpected challenges to achieve the
intended outcomes.
The book encourages education practitioners to engage in evaluation that
generates useful information for program improvement and for accountability
of effort and investments. Practitioner-based evaluation differs from high-
stakes, external evaluation by integrating frequent engagement and review of
processes and results to improve their efforts. As such, it will rarely answer the
question, How do we know it was the professional learning that caused these
results? This question is a question for researchers to answer. Evaluations seek
to involve stakeholders in the examination of their practice and its results
before, during, and at the end of their efforts. The process is designed to culti-
vate “evaluation think,” a phrase that Joy Frechtling, a member of a national
advisory board for the initiative that launched the first edition of Assessing
Impact, uses to describe how individuals and teams in schools and districts look
critically and analytically at their work to discover what is working and what is
not in order to redefine their work and to improve results.

PURPOSE OF THIS BOOK

The purpose of this book is to assist state, school system, and school leadership
teams to

• plan evaluations of professional learning that provide ongoing evidence

for program improvement;
• conduct practitioner-based evaluations that focus on results for educa-
tors and students;
• increase the frequency and usefulness of evaluation as a part of pro-
gram management;
• build the capacity of education leaders and program stakeholders to
adopt “evaluation think”; and
• increase the engagement of education stakeholders in decision-making
about professional learning.
 xvii
Introduction  

WHAT THIS BOOK IS AND ISN’T

This book is a guide to assist state, school system, and school leadership teams
in planning and implementing evaluations of professional learning. It is
designed to facilitate asking the fundamental evaluation questions:

• Does our plan for professional learning meet the criteria for success?
• Is our program ready for evaluation?
• How will we measure progress and results?
• Are we progressing as we intended?
• What is working? Why?
• What is not working? Why?
• What needs to change to increase likelihood of achieving the intended
results?
• Did our professional learning efforts produce the intended results?
• What other effects of our efforts are we seeing?

As attributes of effective professional learning are increasingly clear from

research (Darling-Hammond, Wei, Andree, Richardson, & Orphanos, 2009;
Desimone & Garet, 2015; Desimone, Porter, Garet, Yoon, & Birman, 2002;
Garet, Porter, Desimone, Birman, & Yoon, 2001; Learning Forward, 2011;
Supovitz, Mayer, & Kahle, 2000; Supovitz & Turner, 2000; Yoon, Duncan,
Scarloss, & Shapley, 2007), education practitioners apply these attributes to
design more promising professional learning. Yet the application of the attri-
butes will not necessarily produce results. Educators take the next step beyond
applying the attributes in designing professional learning to using rigorous
measures to evaluate implementation and results of their efforts.
This book is not a program evaluation textbook, a data collection and anal-
ysis guide, or a statistics manual. These are topics in which professional evalua-
tors are well-versed and for which there are more scientific sources. This book is
designed specifically for professional learning leaders, program or project coor-
dinators, curriculum directors, principals, district and school improvement and
leadership teams, or others who want to evaluate their professional learning
programs to improve them and to be accountable for their investments and
efforts in professional learning and to make informed, data-based decisions.

VALUE FOR EDUCATORS

This book assists practitioners to conduct evaluations of their work. Specifically,
it suggests ways to

• design and conduct regular and rigorous evaluations to measure the

effectiveness and impact of professional learning on educator practice
and student achievement;
• report to policy makers and stakeholders about professional learning;
• assess implementation of professional learning to strengthen its impact;
• design powerful professional learning programs by making their
program’s theory of change and logic model explicit;
xviii  Assessing Impact
• provide practical ways to evaluate professional learning programs; and
• construct an information base for making decisions about professional
learning programs.

Perhaps Michael Quinn Patton (1997) has described best what this work
also is intended to do:
“Our aim is modest: reasonable estimations of the likelihood that particular
activities have contributed in concrete ways to observed effects—emphasis on
the word reasonable. Not definitive conclusions. Not absolute proof. Evaluation
offers reasonable estimations of probabilities and likelihood, enough to provide
useful guidance to an uncertain world (Blalock, 1964). I find that policy
makers and program decision makers typically understand and appreciate this.
Hard-core academics and scientists don’t” (p. 217).
 Evaluation as
Normative Practice

1
E ducators in states, school systems, and schools work tirelessly to meet the
learning needs of every student. Yet, some students continue to struggle. To
achieve their vision of success for every student, educators are increasingly
using data to understand and pinpoint opportunities for increasing the effec-
tiveness of their efforts and to make savvy decisions about education programs
to implement. Rather than assigning blame elsewhere or shirking their respon-
sibility, they double down on their commitment to be accountable to their
stakeholders, especially students and their families. They cringe each time
they see results that disappoint them and acknowledge that the education
they are providing is leaving some students behind.
Yet what educators choose to do when they face this situation is perhaps the
most crucial decision of all. Rather than grasping at anything available or what is
easy and familiar, educators become more deliberate and engage in focused contin-
uous improvement. They use available data to understand where the needs are and
their root causes. They investigate evidence-based programs that provide results.
They analyze the context in which their schools exist to assess the resources, cul-
ture, facilities, equipment, and human and social capital that will influence their
efforts. Using all this information, they plan thoroughly for implementing profes-
sional learning programs with high levels of promise. In conjunction with their
planning, they simultaneously plan how they will monitor implementation and
evaluate their progress and results to address the gaps they have identified. As the
professional learning programs are implemented, they review progress, make
adjustments, and measure impact on educators and students.
This backmapping process (Killion, 1999; Killion & Roy, 2009) for the
design, implementation, and evaluation of professional learning appears
in a variety of forms and by various names. It describes the process for using
data to identify needs, understanding context, studying research and evi-
dence, planning a program to address needs, implementing the program, mon-
itoring progress, and evaluating effects (see Figure 1.1). This process
acknowledges that program selection or planning occurs only after deep analy-
sis of student, educator, and system data. Not only do educators identify where

1
2  Assessing Impact
Figure 1.1 Backmapping Model

Step 7 Step 1
Implement, Analyze student
Step 2
sustain, and learning needs.
Identify
evaluate the characteristics
professional of community,
development district, school,
intervention. department,
and staff.

Improved
Step 6 student
Plan Step 3
learning
intervention, Develop
implementation, improvement
and evaluation. goals and specific
student
Step 5 outcomes.
Study the research
for specific Step 4
professional Identify educator
learning programs, learning needs.
strategies, or
interventions.

Killion, J. (1999). What works in the middle: Results-based staff development. Oxford, OH: National Staff Development Council.

gaps exist in student learning, but they also identify which students are in
greatest need, what the most likely root causes are for the existing gaps, and
which learning outcomes will eliminate the gaps. They also understand what
educator factors and system factors are influencing the results and are in most
need of addressing. For example, if most teachers in schools with the highest
percentage of students who are underperforming are those with the lowest
level of experience, how will program leaders address this factor in designing
and implementing professional learning? If teachers have insufficient time for
collaboration, a core research-supported component of increasing the quality
of teaching within a school, how will the school’s leadership team and district
leadership team address this potential inhibitor before the program is imple-
mented? Other similar processes such as the cycle of continuous improvement
(Killion & Roy, 2009; Learning Forward, n.d.a), Plan-Do-Study-Act (PDSA)
(Deming, 1994), or improvement science (Bryk, Gomez, Grunow, & LeMahieu,
2015) focus on shorter cycles of learning, experimentation, and implementa-
tion to make ongoing improvements in routine work.
Changes in the Every Student Succeeds Act increase state, local education
agency, and school leaders’ responsibility to evaluate professional learning to
improve student academic success, assessment, accountability, school improve-
ment, teacher and leader effectiveness, and use of federal, state, and local
 3
CHAPTER 1 Evaluation as Normative Practice   

resources. While increasing states’ flexibility, the law holds tight on states’
accountability to ensure that every student succeeds. It calls for better use of
evidence, ongoing commitment to improvement, and engagement of stakehold-
ers. It requires states rather than the federal government to determine criteria
and interventions for schools in need of improvement and holds the expecta-
tion that districts develop evidence-based strategies to address their needs.
Relevant to this book, ESSA redefines professional learning to include
personalized, job-embedded, ongoing, available to all teachers of all content
areas (including administrators and other school staff), collaborative, informed
by educator input and data, integrated into school improvement plans, and reg-
ularly evaluated. It requires states to use evidence to select programs to address
identified needs and to submit a description annually on how their selected
activities “improved teacher, principal, or other school leader effectiveness”
(S. 1177, Sec. 2104, Ia). This description is a form of evaluation.
As the focus on educator effectiveness increases, the importance of effective
professional learning grows as does the need for its evaluation. No longer will
documentation about participation levels or satisfaction surveys serve to
substitute for learning and impact. Hayes Mizell (2003) makes this point clear
in his article, “Facilitator: 10; Refreshments 8; Evaluation 0.” He says,
“Workshop satisfaction misses the point. Evaluation means understanding
what participants learn, when and how they apply the learning, and when
and how it benefits students” (p. 10). He calls upon professional learning lead-
ers to invest in their own learning about effective evaluation and how to use it.
He notes that there are two overarching reasons for this investment. First, he
notes is the continued realignment of resources that often result in reduction of
funding for professional learning. Second, he adds, is the increasing pressure to
educate all students to high levels, and that this requires ensuring that all edu-
cators have the capacity to meet the needs of all students.
Overly simplistic, event-focused perception surveys may produce data, yet
they are not the types of data that will enable professional learning leaders to
answer their most pressing questions. Data are most useful when they are placed
within the context of a systematic investigation of programs and processes.
Evaluation—not just data—is increasingly important for reforming schools
because evaluation, when thorough, provides state, school system, and school
leaders answers to questions about the impact of their efforts. Evaluation, as a
critical part of an ongoing improvement process, provides leaders insights into
what is working and what is not, and information to make better decisions.
Leaders interested in evaluation sit in every chair in education. A
10th-grade student evaluates the pieces of work in his portfolio to select one
which best exhibits his effort to conduct a science lab to solve a problem. A
fourth-grade teacher, implementing a new mathematics instructional practice
to make student thinking visible, learned in a summer workshop, is evaluating
its effectiveness by watching how her weakest students respond when she uses
the practice so she can ask her coach for more specific support when they meet
next week. The English department at the high school uses evaluation to assess
its implementation of a series of lessons on argumentation to understand how
to adapt those lessons in the future to address student misconceptions and
ensure more students are successful in writing arguments. The middle school
4  Assessing Impact
leadership team has implemented a research-based social-emotional skills pro-
gram, one of the additional criteria beyond student achievement required now
in their district for schools in need of improvement, and wants to measure its
success with their students and to report to parents, central office, the school
board, the state education agency, and the local community foundation that
funded the program about their results. The district talent development chief
implemented a five-year evaluation of the teacher and principal mentorship
program to know if the program is achieving its intended results and is suffi-
ciently resourced. The regional education agency is initiating a program to
increase the capacity of paraprofessionals to support literacy instruction in
preK and wants to measure its effectiveness.
Evaluation uses data to answer specific questions to create potential for
transforming teaching, learning, leadership, and the systems that support them.
It is not data alone that transform. Consider this simple analogy. More people are
sporting wearables to measure many types of active and passive activities, such
as heart rate, distance walked, hours of movement, and sleep patterns, and are
logging more and more information such as caloric intake, emotional state, and
so on. The apps used even provide daily or weekly reports. Yet it is not these data
that will change a person’s health, well-being, or activity level. It is interpreting
and using the data to make changes where needed, comparing last week’s to
this week’s results to know if progress is evident, and to know if one’s goals are
met. Simply logging caloric intake will not reduce weight, yet logging it, review-
ing the data, and acting on the data will have a role in changing behavior. The
same is true for professional learning. Knowing that 92 percent of school princi-
pals appreciated the district conference day options available to them will not
provide information about whether they reflected on how to integrate the new
practices, applied their learning on a routine basis, and realized changes in
teacher or student learning as a result of their new leadership practices.
Districtwide data management systems make data more readily accessible
to educators. As a result, more data conversations are occurring in schools.
Data walls display color-coded levels of student performance in a variety of sub-
ject areas are frequently visible in schools. Yet, the presence of data alone, how-
ever, does little to improve educator practice or student learning. Two missing
elements limit the potential of data. Often missing from data conversations is a
decision about a planned, purposeful set of actions to address identified needs.
ESSA requires more careful selection of evidence-based interventions, pro-
grams, or practices. Non-regulatory guidance specifies four levels of evidence
that states and districts can use for selecting interventions to address school
improvement and student learning. The levels are presented in Table 1.1.
Also missing from data conversations is a plan for evaluation. There are sev-
eral types of evaluations needed to select, implement, and measure outcomes of
an intervention. Planning evaluation involves data analysis and interpretation
to identify the specific problem or needs to address and understand the context or
conditions in which an intervention will be implemented. It results in the selec-
tion of an evidence-based program to address the identified problem or need.
Designing the implementation and outcomes evaluation occurs simultaneously
with implementation planning for the selected intervention and results in an
evaluation framework; a clear and detailed plan to conduct rigorous, systematic,
 5
CHAPTER 1 Evaluation as Normative Practice   

Table 1.1 Definition of Levels of Evidence

Level Description

Strong evidence Supported by a minimum of one well-designed,

well-implemented experimental, randomized control trial
study that meets the What Works Clearinghouse standards
without reservations

Moderate evidence Supported by a minimum of one well-designed,

well-implemented quasi-experimental study that meets the
What Works Clearinghouse standards with reservations

Promising evidence Supported by a minimum of one correlational study with

statistical controls for selection bias that use analytic
methods to compare the intervention group with a
non-intervention group

Demonstrated rationale Supported by a well-specified logic model, based on

research or evaluation, that demonstrates that the
intervention is likely to improve relevant outcomes

Adapted from U.S. Department of Education. (2016, September 16). Non-regulatory guidance: Using
evidence to strengthen education investments. Washington, DC: Author. https://www2.ed.gov/policy/
elsec/leg/essa/guidanceuseseinvestment.pdf.

and purposeful data gathering; analysis and interpretation; report of findings to

stakeholders; and use of findings to make improvements. The evaluation frame-
work begins with posing the questions that stakeholders want to answer about
the intervention and its use. The framework clarifies what data are needed, from
whom, how often, and how much. It specifies how data will be analyzed and
interpreted to measure merit, worth, progress, and impact to improve their efforts.
Because many who work in professional learning are action-oriented, they
tend to focus on what to do rather than the results they want to achieve. Process
becomes more important than results. They tend to think about short-term
accomplishments rather than long-term results. They are comfortable report-
ing what they have done rather than what results they are producing. More
effort is allocated to selecting and launching interventions than to implement-
ing and sustaining them. Yet repeatedly professional learning leaders are being
held to high standards of accountability for their efforts and are asked for evi-
dence of results. The evaluation process described in this book supports these
leaders in all phases of evaluation to meet the accountability expectations,
specifically for practitioners who want to add evaluation to their routine work.
Implementing evaluation as a natural component of all professional learn-
ing encourages and supports systematic review, study, and analysis of profes-
sional learning to improve outcomes, accountability, equity in access and
quality, effectiveness, and efficiency. Evaluation, when it is normative practice,
shifts the focus of professional learning leaders from a service orientation
(What can we do to meet the unique needs of the education workforce? What is
available and when? Who participates? How accessible are the options? How
aligned are the learning options with the school system’s strategic priorities?)
to a results orientation (How are educator practices changing? What supports
are increasing changes in practice? What else is changing within the system to
6  Assessing Impact
support educator changes in professional practice? How are changes in educa-
tor practice influencing student success?)
Leaders of professional learning who integrate evaluation into their
normative practice make more informed decisions, respond more quickly to chal-
lenges that may interfere with results, use clearly defined outcomes as the pri-
mary measure of their success, and keep a laser-like focus on those results. When
results are the driver, expectations are clearer and efforts are more aligned.
Stephan Bauer, a management consultant, notes, “Results-oriented leaders
know how to create systems, build coalitions, motivate employees, monitor per-
formance for effectiveness, and be responsible for results. Organizations and lead-
ers can develop this capacity by constantly asking themselves questions such as:

• What does our organization truly value? How do we prove a consistent

commitment to these values in all our work?
• What is our organizational vision for the world? How do we measure
progress toward that vision?
• How are we engaging with others toward the realization of our vision in
a way that helps us achieve more together than we can individually?
• How do we seek learning, and how does that learning inform how we
continually improve our practice and organization?
• How do we engage our staff in conversations about the qualities it takes
to lead, and provide them safe space to struggle with the practice of
leadership?” (Bauer, 2014).

Evaluation is, as Posavac (2016) says, like breathing. Everyone does it all the
time. Yet evaluators approach this work with purpose and intentionality to mea-
sure and understand authentic issues that matter most. Donna Mertens and Amy
Wilson (2012) state, “Evaluators’ ways of thinking are different from ordinary
daily decision making, because they engage in a process of figuring out what is
needed to address challenges through the systematic collection and use of data”
(p. 3). Professional learning leaders who engage in ongoing evaluation as a nat-
ural part of their work are results-oriented leaders committed to increasing the
success of every member of the education workforce and each student they serve.

QUESTIONS FOR CONSIDERATION

1. How do we currently evaluate professional learning?
2. Who is primarily responsible for evaluating professional learning?
3. What data are we using regularly to examine professional learning?
4. How do we engage stakeholders in decisions about the development, imple-
mentation, and evaluation of professional learning?
5. How do we choose interventions for school improvement, professional
learning, student learning, or other identified needs?
6. What might be a goal we set for ourselves about improving our use of eval-
uation so that it is more normative practice?
 Professional
Learning Program
Evaluation Overview

2
T his chapter reviews fundamentals about practitioner-driven program eval-
uation as it relates to professional learning. It includes the definition of
evaluation, the difference between evaluation and research, the purposes
of evaluation, the difference between programs and events, the types of eval-
uation, and an overview of the evaluation process.

EVALUATION DEFINED
Definitions of evaluation abound. Esteemed scholars in the field offer many
definitions that provide a way of thinking about evaluation in general. The
following definitions of evaluation from evaluation scholars deserve attention.
Daniel Stufflebeam (2001) asserts that evaluation is “a study designed and
conducted to assist some audience to assess an object’s merit or worth” (p. 11).
Michael Scriven (1991) says that evaluation is “the process of determining
the merit, worth, and value of things, and evaluations are the product of that
process” (p. 1).
Michael Quinn Patton (2008) defines program evaluation as “the system-
atic collection of information about activities, characteristics, and results of
programs to make judgment about the program, improve or further develop the
program effectiveness, inform decisions about future programming, and/or
increase understanding.” He specifies that “utilization-focused program evalu-
ation [a specific form of evaluation] is evaluation done for and with specific,
intended primary users for specific, intended uses” (p. 39).
Fournier (2005), in the Encyclopedia of Evaluation, defines evaluation as “an
applied process for collecting and synthesizing evidence that culminates in con-
clusions about the state of affairs, value, merit, worth, significance, or quality

7
8  Assessing Impact
of a program, product, person, policy, proposal, or plan. Conclusions made in
evaluations encompass both an empirical aspect (that something is the case)
and a normative aspect (judgment about the value of something). It is the value
feature that distinguishes evaluation from other types of inquiry, such as basic
science research, clinical epidemiology, investigative journalism, or public poll-
ing” (pp. 139–140).
Carol Weiss (1998) suggests that program evaluation is “the systematic
assessment of the operation and/or outcomes of a program or policy, compared
to a set of explicit or implicit standards, as a means of contributing to the
improvement of the program or policy” (p. 4).
To cull across these definitions and for the purpose of this book, evaluation
of professional learning, then, is a systematic, purposeful standards-driven
process of studying, reviewing, and analyzing data about a profes-
sional learning program gathered from multiple sources to make judg-
ments and informed decisions about the program. This definition implies
that an evaluation results in an individual or team making a reasoned judg-
ment based on the analyzed data. The individual or team uses the judgment to
gain insights and deeper understanding and to make decisions about the pro-
fessional learning program being evaluated. The definition assumes that a rea-
son exists for the evaluation, and decisions will be made as a result of the
evaluation. Evaluation calls for rigor in all phases of the work and is guided by
a set of standards that, when met, ensure the quality of the process. These stan-
dards for program evaluation are set by the Joint Committee on Standards for
Educational Evaluation (Yarbrough, Shulha, Hopson, & Caruthers, 2011), the
American Evaluation Association (2004), and Learning Forward (2011) and
appear in Appendix B.
Key to the definition of program evaluation is the concept of program.
Kathryn Newcomer, Harry Hatry, and Joseph Wholey (2015) define program
as a “set of resources and activities directed toward one or more common
goals, typically under the direction of a single manager or management team”
(p. 7). A program may be larger in scale, such as a statewide effort to increase
graduation rates, and involve multiple agencies in a coordinated fashion.
They may be district focused as in programs designed to improve student per-
formance of mathematics and include curriculum design, formative assess-
ments, professional learning on instructional methods, parent engagement in
family math activities, and classroom resources. A preschool program may be
upgrading the capacity of paraprofessionals to engage students in literacy-
oriented learning experiences aligned to the new preschool curriculum.
Programs may also be small in scale such as a team of teachers working
together to improve student writing skills. A professional learning program,
for the purpose of this book, is a set of planned and implemented actions,
guided by research, evidence, and standards of effective professional
learning, accompanied by adequate resources, and directed toward
the achievement of defined outcomes related to educator practice
and its impact on student learning.
In each case, regardless of the size of the program, evaluation is a crucial
component to understand how the resources and activities implemented influ-
ence the outcomes achieved and to make decisions for improving, expanding,
 9
CHAPTER 2 Professional Learning Program Evaluation Overview   

adjusting, or eliminating the program. The scope of the program and its evalu-
ation may vary along several features such as number and frequency of activi-
ties, duration, amount and frequency of data collection, and number of
stakeholders. A team of seventh-grade teachers learning together how to inte-
grate critical thinking routines into their instruction uses data from students to
examine the impact of their work monthly and adjust their practice. When
school improvement teams engage in evaluation to measure the effects of a
schoolwide professional learning program to increase students’ use of aca-
demic vocabulary, they increase the scope of the activities to achieve results,
broaden the data collection over an extended time period, and report to more
stakeholders about the effects of their work. They may adjust their activities
two or three times a year rather than as frequently as a teaching team. A school
system evaluation expands each area even further. It may involve gathering
data twice a year and reporting annually on progress and results. In other
words, the closer the evaluation process is to its impact, the narrower the scope
of the evaluation. Whether the evaluation is large scale or small scale, the eval-
uation process described in this book is useful.

KEY COMPONENTS OF
EVALUATION DEFINITIONS
The following words or phrases are the key components of the definition of
evaluation. Each of these terms contributes to a deep understanding of what
an evaluation is and its purpose.

Systematic
The term systematic suggests that evaluations
are orderly, planned, formal, rigorous, objective, Evaluation of professional learning is the
and guided by standards for success and quality systematic, purposeful, standards-driven
process of studying, reviewing, analyzing,
such as those offered by the Joint Committee on
and interpreting data about a professional
Standards for Educational Evaluation, the
learning program gathered from multiple
American Evaluation Association, and Learning sources to make judgments and informed
Forward (see Appendix B). decisions about the program.

Standards
Most evaluations measure merit, worth, or impact against predetermined
criteria. In addition to the standards for the quality and integrity of evaluation
practice, evaluations use standards for professional learning and program out-
comes as criteria against which to measure the program’s success.

Data
Data take many forms and come from many sources. They may be collected
in various ways. “Data are discrete bits of information that becomes evidence
through a process of analysis and interpretation” (Killion, 2015, p. 80). Data
alone are insufficient to make evaluation useful. It is through the analysis and
10  Assessing Impact
interpretation of purposefully collected data that evaluations have usefulness
and can inform decision makers.

Intended Uses of the Evaluation

Knowing the intended purpose(s) of an evaluation influences many deci-
sions made during the evaluation’s planning and implementation. Evaluators
and program leaders, therefore, need to identify their audiences and make
sure they prioritize the most significant uses of the evaluation before they
design it. For example, if a policy maker wants to know if program participa-
tion contributes to teacher retention, the evaluator may examine data related
to employee retention and engagement in the induction program. If a district
HR department wants to improve the program, they might want to collect
additional data about which aspects of the program participants perceive to
be most beneficial and their recommendations for upgrading the program.
They may also want to know what aspects of the program are most costly to
examine if other alternatives exist. And, if the purpose of the evaluation is to
determine whether students of novice teachers who participate in the induc-
tion program receive a high-quality education, a far more complex evalua-
tion design is required.

Program
This book is about evaluating professional learning programs rather than
events. A program implies more than isolated incidents and aligns with the fed-
eral definition of professional learning in ESSA. A program is a set of purpose-
ful, planned, research- or evidence-based actions and the support system
necessary to achieve the identified outcomes. Effective professional learning
programs use data to determine the content; occur in collaborative communi-
ties; employ multiple, active learning designs; provide implementation support;
engage leadership to support learning and create systems of support; include
adequate resources; occur over time; and link to educator performance and stu-
dent learning standards (Learning Forward, 2011). Programs, however, vary
widely in scope.
A workshop on classroom management is not a program. Such a workshop
is an event or one action included in the set of actions to achieve outcomes for
educator learning and student success. Program planners would likely derive
the need for a classroom management workshop from identified student
achievement needs, such as poor performance, and investigate further to dis-
cover that insufficient engagement and lost instructional time may be root
causes. To turn the workshop into a professional learning program, it would be
incorporated into a comprehensive plan of actions designed to develop educa-
tor knowledge, attitudes, skills, aspirations, and behaviors that would
include classroom demonstrations, visits to peer classrooms, planned class-
room-based support with ongoing coaching, opportunities to review and reflect
on practice, and collaboration among peers to assess and evaluate the effects of
their practice on students using defined criteria. Since much of what has been
 11
CHAPTER 2 Professional Learning Program Evaluation Overview   

provided as professional learning has not been considered a program, its results
have been mixed. For those who want to evaluate professional learning, they
must first have a professional learning program to evaluate. It is only in the
context of a full professional learning program that data emerging from indi-
vidual events contribute to understanding how a program is progressing
toward its defined outcomes.

Evaluation Versus Research

Many educators confuse evaluation with research. Both are a part of a
larger class of social science inquiry. Both are guided by questions, data col-
lection and analysis, and findings reported to stakeholders. Many education
decision and policy makers completed courses in research methodology;
few took courses in evaluation. Because they may not fully understand the
distinction, decision and policy makers often assume that evaluation and
research are the same. A frequent example of this confusion is a local school
board member asking the question: How do you know it was professional
learning that caused the changes in student achievement and not something
else? This type of question is one a researcher can answer under certain con-
ditions and with an appropriate research methodology. Evaluators can answer
other questions: Did the professional learning contribute to educator practice
and student achievement? What changes occurred in educator practices?
What actions of the program had the greatest impact on changing educator
practice? Which educator practices were most significant in influencing
student results? What conditions contributed to and interfered with the
program’s success? While there are many similarities in processes, methods,
and tools used in both research and evaluation, the purposes are different
(see Table 2.1).
An examination of the purposes of evaluation and research may be helpful
in understanding the distinction so that educators can draw appropriate con-
clusions and appropriately use evaluation findings. Education practitioners will
benefit from applying the eight-step evaluation process outlined in this book
to guide their work as evaluators rather than trying to conduct research or
present their evaluation findings as if they were research.
Carol Weiss (1998, p. 47) notes that evaluations are political in nature:
“Evaluations are conducted on social programs—most important, on social
programs in the public domain,” she notes. “Social programs are manifest
responses to priority individual and community needs and are themselves ‘the
creatures of political decisions.’ They are proposed, defined, debated, enacted,
and funded through political processes, and in implementation they remain
subject to pressures—both supportive and hostile” (p. 47). Evaluation, she con-
tends, “makes implicit political statements about such issues as the problematic
nature of some programs and the unchangeability of others” (p. 48). Because
professional learning programs are situated within a context that influences
the choice of the intervention, its implementation, and most likely its results,
evaluators consider carefully how to increase objectivity in their work to
provide honest, reliable evaluations that will be useful.
12  Assessing Impact
Table 2.1 Difference Between Program Evaluation and Research

Program Evaluation Research

Purpose Determines whether a particular Explores the interaction among

program has merit, worth, variables, adds to the general or
value, and impact (e.g., did it field-specific knowledge base
achieve its outcomes? Was it about a particular phenomenon,
perceived to be worthwhile? Did tests a theory, and advances
the activities occur as planned?) that field.
The focus is usually on the
effectiveness, results, and
improvement related to the
program.

Grounded in Uses a theory of change and Uses theory and literature

logic model. review to design and apply
intervention for study.

Focus Focuses on program description, Looks at results produced by

clearly defined outcomes, and groups in which a particular
data about results of a particular program does and/or does not
program that occurs within its occur; program is implemented
natural contexts; variations are in carefully controlled contexts
studied and noted especially in to minimize variations beyond
terms of how those variations the existence of the program
impact results. between treatment and control
or comparison groups.

Use of Identifies how effective the Generalizes the results to

Findings program is and informs develop theory and inform
decisions about program practice in other contexts and
improvement, continuation, populations.
discontinuation, expansion, and
so on for the same context and
population.

Dissemination Reports findings and proposed Reports findings to field for

of Findings next actions to decision makers, critique of design, validity, and
including policy makers, reliability and shares
stakeholders, participants, and intervention for implementation
program managers. in other settings.

Evaluation Versus Assessment

Another term frequently confused with evaluation is assessment. While
assessment may be a part of evaluation, evaluation is not always a part of
assessment. Assessment is an objective measurement based on data for improv-
ing something, whereas evaluation is objective measurement based on data or
evidence, and standards or criteria for success for judging or valuing some-
thing. When planning professional learning programs, planners determine the
standards of success when they develop the program’s goals and outcomes.
These standards become the criteria against which a program is judged as suc-
cessful. In the case of evaluation, assessments may focus on the processes of a
 13
CHAPTER 2 Professional Learning Program Evaluation Overview   

program to improve them during the program’s operation. The program’s eval-
uation is done at the end of the program or at significant points in the pro-
gram’s history to judge the program’s success against established criteria.
Assessment is formative in nature while evaluation is summative. Assessment
is for diagnostic purposes while evaluation judges the degree to which specific
outcomes are achieved.
While conducting an evaluation of an aspiring administrator program
within a district, the district steering committee may conduct both an assess-
ment and a periodic evaluation. The annual assessment may be based on a pro-
gram completion survey or interview of participants about what they
appreciated about the program, the significant experiences they had in the pro-
gram, and their perceptions of the program’s impact on their desire to become
an administrator. These data gathered annually provide the steering committee
with an assessment so they can make annual adjustments to the program to
improve it before the next cohort begins.
Periodically the steering committee or the school board may call for an
evaluation of the program. The designated evaluator gathers all the assessment
data from the previous years to use in the evaluation, and adds other data to be
able to make a judgment about the program’s overall impact, whether it meets
the program’s goals, and whether it is valued by the district and the partici-
pants. Some additional data the evaluator collects might include the number of
participants who move into and remain in an administrative position; a com-
parison of performance appraisals of administrators, new to the role in the past
five years who completed the program and those who did not participate in the
program; a comparison of the school performance data with principals who
completed the program and those who did not complete the program; and a
comparison of scores on a 360-degree survey of supervisors, supervisees, and
peers of new administrators in the past five years who completed the program
with those who did not.
Assessment is appropriate for continuous
improvement. Multiple assessments of a pro- Key Terms
gram may occur prior to or as a part of an eval-
uation of a program. Evaluation, however, is Merit: The intrinsic value of a program; how it
essential before any significant decision about a is perceived by those it intends to help based on
established criteria
program is made. Assessments are useful for
continuous improvement if small adjustments Worth: The extrinsic value of a program to
are to be made along the way; however, if a those outside the program, such as the larger
decision about continuation, discontinuation, community or society
expansion, or reduction in the program, its Value: The political or social contextual criteria
operation, or funding is needed, evaluation is influencing decisions about merit or worth; what
essential to ensure that the program meets its one person, organization, or entity perceives as
outcomes and the standards it is intended to valuable may not be perceived as valuable to
meet. It would, for instance, be unfair to deter- another one
mine if a student is able to write and support an Impact: The effect of a program on its partic-
argument if the only evidence available is the ipants and their clients; for example, teacher
student’s ability to identify supporting evidence professional practice and student learning
in a text or even a draft argument. It is in the
14  Assessing Impact
student’s finished product, measured against the standards set in a rubric for
exemplary performance, that final evaluation is made.
Evaluation then serves the purpose of judging merit, worth, value, and
impact of a program based on established standards and sufficient data or evi-
dence. This book acknowledges that most educators have a strong interest in
continuous improvement and use it frequently, yet may not evaluate as often.
Frequency of use does not diminish the necessity of being able to conduct eval-
uations of professional learning to make informed decisions about it.

PURPOSES OF EVALUATION
Evaluation serves a variety of purposes, and an evaluation’s purpose influences
decisions about how the evaluation is designed and implemented. Several sce-
narios involving evaluation of professional learning are described below.

1. The high school science collaborative learning team is shifting to

blended instruction and wants to know if what they are learning and
applying in their classroom is positively influencing student science
achievement.
2. A school improvement team is implementing a professional learning
program as a part of the school improvement plan and wants to mea-
sure the impact of the program on teaching, school culture, and stu-
dent learning.
3. The mathematics curriculum team works with the district math curric-
ulum coordinator to revise the mathematics curriculum and select new
instructional materials to implement the revised curriculum. They plan
and implement extensive teacher professional learning to ensure that
teachers have the mathematics content and the content-specific instruc-
tional processes to implement the new curriculum. They want to con-
duct an impact evaluation of the curriculum-related professional
learning on teacher classroom practice and student achievement.
4. A district committee responsible for implementing the College, Career, and
Civics Social Studies Framework across the district is strengthening social
studies instruction. Teachers engage in extensive training and school-
based support in using new instructional strategies. Teachers, princi-
pals, parents, and the district social studies coordinator want to know
how to determine whether the approach improves student achievement
in social studies.
5. A superintendent proposes a significant increase in the professional
learning budget to provide small grants for individual schools to address
the district’s strategic priority on personalization and deeper learning.
He engages the members of the school board in conversation about
what they want to know about the return on their investment to
demonstrate his intent to hold schools accountable for the investment
in professional learning.
 15
CHAPTER 2 Professional Learning Program Evaluation Overview   

6. A school improvement team is examining several professional learning

programs aimed at increasing students’ active engagement in learn-
ing. They want to learn more about the types of evaluations that were
conducted on the three programs they are considering.

7. The district improvement team wants to know whether schools’ use of

professional learning resources is effective. The teacher leaders who
serve as chairs of schools’ professional learning committees ask clari-
fying questions about what they specifically want to know and what
evidence the schools are required to provide, and seek guidance in how
to go about collecting and providing the requested evidence.

8. A state professional learning supervisor is required to provide an

annual impact report on the use of Title I and Title II funds to the state
school board and to the U.S. Department of Education as a component
of the state’s report on its comprehensive plan.

9. A director of teaching and learning is facing a budget cut and, with the
district’s professional learning advisory team, is preparing a recom-
mendation to the superintendent about which professional learning
program to fully fund and which to reduce. She wants the recommen-
dation to be data informed.

10. A district leadership team wants to ask its school board and commu-
nity to support changing the teacher workday to include weekly
90-minute, student-free blocks for teachers to engage in professional
learning teams. To solidify the rationale, the district committee wants
to provide a plan for assessing the impact of the allocated time.

The overarching purpose of evaluation is to conduct qualitative inquiry,

a process of making meaning and understanding how things work to be able
to assess their value, merit, worth, and impact to improve them. Patton
(2015) summarizes seven contributions evaluation makes. It illuminates
meaning; studies how things work; captures stories to understand perspec-
tives and experiences; understands how systems work and how they influ-
ence people; understands context and its impact on experience; identifies
unanticipated consequences; and makes comparisons to find trends and
themes. For those in education, particularly public education, the purposes
for evaluating professional learning depend on the key stakeholder request-
ing the evaluation. Table 2.2 summaries common uses, users, and purposes
of evaluation of professional learning.
Clarity of purpose in evaluation facilitates decision making during the eval-
uation’s planning and implementation phases. Yet, occasionally in evaluation
planning, various stakeholders have different and sometimes competing pur-
poses for the evaluation. The varying interests and demands can be challenging
within a single evaluation and may necessitate multiple evaluations, some
occurring simultaneously or nested together in a more sophisticated design.
While evaluations can be designed to accommodate multiple purposes, they
become more complex. Even though an evaluation may have more than one
16  Assessing Impact
Table 2.2 Intended Uses, Users, and Purposes of Evaluation

Intended Uses
of Evaluation Intended Users Purposes of Evaluation

Determining Program directors, policy To determine merit, worth, and

merit, worth, makers, decision makers, impact of a program; to determine
value, and and stakeholders the overall effectiveness of a
impact program and make decisions about
its adjustment, expansion,
continuation, or discontinuation; to
compare a program with other
programs

Facilitating Program directors, such as To make improvements to an

improvements professional learning existing program; to determine a
leaders or curriculum program’s strengths and
coordinators, principals, weaknesses; to identify unexpected
school improvement benefits and barriers; to determine
teams, or professional progress toward the intended
learning communities outcomes; to identify and solve
unanticipated problems; to assess
participants’ progress toward
intended outcomes

Generating Program developers, such To add to the body of knowledge

knowledge as professional learning about the program’s effectiveness;
leaders and curriculum to share experiences with programs
coordinators with others; to identify patterns of
best practices or principles of
effectiveness across programs; to
understand how a program’s
components or actions interact

Providing Policy makers and To assess compliance with intended

accountability decision makers, such as purposes and mandated processes;
boards of education, to provide evidence of fiscal
superintendents, program responsibility; to provide evidence of
managers, principals, public accountability for resources;
program funders, state to submit annual descriptions on
education agency, and how selected interventions improved
local education agency effectiveness of teachers, principals,
staff other school leaders, and other
employees and affect their primary
clients

purpose, just one should be identified as the primary purpose. When an evalua-
tion is to serve more than one purpose, specific aspects of the evaluation will
need to be customized and appropriately sequenced to address various needs
and interests. Newcomer and colleagues note that the “evaluation’s purpose
and major subordinate purposes must mesh at least in part—and certainly not
directly conflict with—the interests of key stakeholders; otherwise the evalua-
tion process is unlikely to get off the ground, and even if it does the process and
its findings will be misused and ignored” (2015, p. 46).
 17
CHAPTER 2 Professional Learning Program Evaluation Overview   

For example, most school systems provide professional learning to staff

to improve performance. This may be the work of a single department or sev-
eral departments or even be outsourced to individual schools and divisions.
The superintendent and school board want to know if they are fiscally
responsible for investments in professional learning and are asking if the
professional learning available to staff is producing results. School systems
and even schools often have several professional learning programs operating
simultaneously, all under the umbrella of a comprehensive, cohesive system-
based professional learning program. Individual program managers may
have other purposes for evaluating their programs, such as measuring their
impact on educator practice and student success, and evaluating the per-
ceived value of the program rather than fiscal accountability. One compo-
nent of an overall professional learning program in a school system is an
induction program designed to address the specific developmental needs of
new teachers, teacher leaders, principals, and other employees. The school
system’s comprehensive professional learning program most likely includes
other programs, some of which may be required by federal, state, or local
regulation or law, to meet the needs of other employees or to address other
goals such as a customer service program for support staff who interact
frequently with the public or a new program to meet the literacy needs of
preschool, second language learners.
To answer the superintendent’s and school board’s question, profes-
sional learning leaders need to first determine what is being evaluated—the
comprehensive professional learning program, its individual programs,
some of the individual programs, the operation of the professional learning
system, the performance of the director of professional learning, or some-
thing else. The evaluator or evaluation team defines the purpose or purposes
of the evaluation by establishing the specific constructs to evaluate and
the criteria for measuring merit, worth, value, or impact. These decisions
are often political in nature because they are made based on the need for an
evaluation, its purpose, and stakeholder interests and needs. Occasionally
evaluators negotiate with stakeholders to either narrow or broaden the pur-
pose of an evaluation to increase the usability of an evaluation. Because the
specific outcomes of the collection of professional learning programs avail-
able differ, an impact evaluation will require various outcome measures and
multiple smaller impact evaluations to provide evidence of impact across
multiple programs. It is possible to conduct a merit, perceived value, effi-
ciency, or effectiveness evaluation across multiple programs, without an
impact evaluation, if there are common criteria that apply to all programs.
For instance, school system leaders might be interested in knowing the qual-
ity of its overall professional learning program. They might use an instru-
ment such as the Standards Assessment Inventory (Learning Forward, 2012)
to measure teacher perception of the effectiveness of professional learning
based on the Standards for Professional Learning. This measure, however,
is not an impact evaluation. An impact evaluation of professional learning
is one that measures changes in educator behavior and their effect on stu-
dents or other clients.
18  Assessing Impact
TYPES OF EVALUATION
Evaluations may occur at different phases of a program’s development and
implementation. The most common types of evaluations in terms of when they
occur are planning, formative or implementation, and summative or end of a
program.

Planning Evaluations
Planning evaluations, those conducted before a professional learning
program is designed, help identify the social conditions and needs that a pro-
gram should address. Using this information, those leading professional
learning can make better decisions about the types of interventions that are
more likely to achieve results. If the needs are not clear and aligned to student
learning outcomes and educator performance outcomes, and if the context
and conditions are unknown, professional learning is difficult to design. For
instance, a school that continues to experience low student achievement
results in reading, yet whose staff has had extensive training in literacy, most
likely does not need more training, assuming the training was aligned to the
student outcomes and curriculum. What might be more appropriate is exten-
sive classroom-based support for implementation of the learning and oppor-
tunities for extending or personalizing learning for staff who want more
specifics about literacy instruction and opportunities for staff to meet in col-
laborative learning teams to plan for integration of the new practices, to
examine their practice and its impact on students, to visit each other’s class-
rooms, and to identify collectively necessary refinements in instruction to
meet the needs of all students. The caution for all professional learning lead-
ers is to be sure they identify real, evidence-based needs and their root causes
rather than perceived needs or preferences.

An Example: Listening to Teachers’ Needs

The district professional learning leadership team analyzed student read-
ing performance on the statewide assessment. Tenth graders’ scores were
consistently well below the state average. Principals and the reading con-
sultant in the district discussed the need to provide training to teachers
on reading in the content areas. They envisioned providing eight total
hours of training in sessions scheduled throughout one school year. The
program would follow a “training of trainers” model. Under the proposed
plan, each school’s literacy coordinator would take the lead in training
fellow staff members at his or her schools and then be available to provide
follow-up support to interested teachers.
When all the school literacy coordinators met to plan the training,
several of them expressed doubt that teachers would implement the
 19
CHAPTER 2 Professional Learning Program Evaluation Overview   

strategies they planned to teach. One even said that teachers were unwill-
ing to incorporate reading strategies into their instructional lessons
because they believed that it would take time away from their course con-
tent. Together with the reading consultant, the literacy coordinators
expressed a concern that most teachers perceived that they did not share
responsibility for student reading performance, and thus were not open to
learn how to improve students’ reading performance. They also worried
that teachers might believe that if students had not learned to read by the
time they reached 10th grade they were unlikely to learn within their
content-area courses and would need interventions from a reading spe-
cialist. The group continued to guess about the perceived discontent
among teachers until one of the literacy coordinators suggested that they
ask some teachers. By conducting focus groups at each high school, the
reading coordinators discovered that teachers did expect students “to read
independently for comprehension” by the time they got to high school and
were finding that students’ inability to read was interfering with their
academic and dispositional success. They also learned that many teachers
were eager to tackle the challenge because they saw a tremendous benefit
for them as well as their students.
Literacy coordinators also learned that teachers needed less help with
reading comprehension strategies and more help with understanding and
assessing students’ reading deficiencies and how to intervene within the
context of their individual courses. They wanted guidance in making
appropriate instructional decisions that would lead to student success
regardless of their reading levels. They also wanted resources with differ-
ent levels of text complexity for student use. Coordinators realized that
their initial plan would likely have been met with resistance and even
failure. Taking time early on to understand teacher needs by engaging
teachers in informal focus groups to listen to their concerns, the district
team could remodel their plan for professional learning, shifting to a
coaching and consulting model rather than a training model, and supple-
ment it with resources for student use before it was implemented.

Formative Evaluations
While planning evaluations are done prior to designing a program, for-
mative evaluations are conducted during a program. They provide informa-
tion about how a professional learning program is working. This type of
evaluation information is essential to improving programs, preventing and
managing problems related to implementation, and ensuring that the pro-
gram is fully functioning, thereby increasing the likelihood of achieving the
intended outcomes. Evaluators can also use formative evaluations to help
explain how the program works and how it contributes to the results it
achieves. To replicate a program, strengthen it, and tell the story of the profes-
sional learning, program managers seek to understand and explain how the
program’s activities lead to results.
20  Assessing Impact

An Example: Making Adjustments to Improve Outcomes

The district science coordinator launched a professional learning program

to develop middle-grade teachers’ content knowledge to implement the
district’s new science curriculum based on the Next Generation Science
standards. The change had been approved by the board of education for
full implementation during the following school year. She wanted to help
teachers be prepared to use the curriculum when the school year started.
First, teachers completed three 2-hour workshops on the standards
and curriculum online at their own pace within a two-month period. The
coordinator visited schools several weeks after the teachers completed
the workshops for follow-up conversations, to answer questions, and to
explore challenges teachers anticipated with the curriculum. She discov-
ered that several teachers had tried parts of one of the sample units she
provided in the workshops. While the coordinator was pleased that teach-
ers had so quickly tried the units, she realized that they were not aligned
to the existing curriculum and that no teachers mentioned developing
their own units based on the new standards.
During the visits. she observed and interviewed teachers as a part of
her formative evaluation of the professional learning she started. She
gathered the information she needed to make some significant changes in
her initial plan, which included only the online workshops. Her visits
helped her realize that one of two things was likely to happen when the
new standards and curriculum were launched: Teachers would run out of
model units and revert to their old units, or teachers would demand that
she provide more units. She discovered she left out a crucial component of
her professional learning plan—helping teachers use the standards and
curriculum to design units to teach them. She realized that she could
address this gap by inviting a small group of teachers from multiple
schools to come together to examine how to use the new curriculum and
standards to design units. She could also engage a group of more expert
teachers in developing more units to share as models. Now that she real-
ized these missing elements, she could work to integrate additional actions
into her plan to help teachers understand how to use the standards and
curriculum for instruction, develop units, critique each other’s units,
revise existing units that somewhat aligned with the new standards, and
help teachers find appropriate instructional materials to integrate into the
new units. Her revised plan also provided an opportunity for some teach-
ers to field test and revise units before the next year’s full implementation.
Her informal, yet informative, formative evaluation helped her identify
several problems related to her plan. Fortunately she had time to revise
and adjust before implementation and without negatively impacting stu-
dent achievement in the next school year. She also could differentiate how
teachers experience the added professional learning.

Summative Evaluations
Summative evaluations are perhaps the most familiar. Typically, a sum-
mative evaluation provides information on achievement of the program’s
outcomes or overall impact, and it may include information that might be
 21
CHAPTER 2 Professional Learning Program Evaluation Overview   

useful for improving the program. Summative evaluations are done at the end
of a program or at a particularly important benchmark point, such as the end
of a year of implementation for a multiyear professional learning program. If
a program is three years long, a summative evaluation would obviously be
conducted at the end of the three years. If the program continues beyond its
three years, ongoing evaluation might be done every three to five years to
determine whether the program continues to impact both educator and stu-
dent learning. Sometimes programs culminate with the end of a funding
cycle. When conducted at that point, summative evaluations provide infor-
mation about the program’s success and may or may not serve as a basis for
decisions about continuation of the program or justification for future fund-
ing. Sometimes summative evaluations are conducted only to meet funders’
requirements and have little or no influence on a program’s future unless a
program continues beyond the external funding cycle.
Increasingly, programs with a specified duration, such as five years, incor-
porate annual formative evaluations to provide information to program man-
agers to make improvements and occasionally to determine continued
funding. In general, policy makers may be more interested in summative eval-
uations (“what are the results?”) and less interested in formative or planning
evaluations (“how are we getting the results?”). Summative evaluations may
have less value or usefulness to program stakeholders, who may be more
interested in understanding how what they have done influences the
outcomes.

An Example: Impact of Project-Based Learning

A team of high school teachers received a district innovation grant to

implement project-based learning in several science and social studies
courses. The district required that extensive professional learning for
staff be a component of the innovation. The district had been consid-
ering moving toward deeper learning in the high school and decided to
offer several grants to teachers willing to experiment with a variety of
related innovations. The district wants to understand what each innova-
tion would take to be successful, whether one approach was more effec-
tive than another, whether it was possible to have multiple approaches
within a single school, and whether a focus on deeper learning had a
positive or negative impact on student academic success, teacher grad-
ing and assessment practices, and students’ postsecondary experiences.
The teachers worked with the district’s research and evaluation, cur-
riculum, and professional learning departments to develop a specific
plan for implementing project-based learning and the evaluation that
would take place over the next two years. The evaluation team helped
the high school steering committee identify different types of data to col-
lect to answer as many of the district’s questions as possible, yet made it
clear that some questions, such as the impact on students’ postsecond-
ary experiences, would not be able to be answered in the short period of
funding for this innovation.
22  Assessing Impact
Planning, formative, and summative evaluations have distinct purposes.
Planning and formative evaluations can provide essential information to
program stakeholders and inform ongoing improvements. Summative evalu-
ations typically lead to decisions regarding program continuation, adapta-
tion, or discontinuation. Professional learning directors will sometimes want
to conduct all three types of evaluations for one program. In other cases,
they will select just one or two types of evaluations to conduct. Table 2.3 is
useful for determining the type or types of evaluation needed and how to
complete each.
Although the three types employ many of the same evaluation tools and
strategies, their purposes and timing are different. Effective uses of the three
types vary from case to case. Using evaluation to improve professional learning
before, during, and after its design and implementation can sometimes
strengthen a program, increase its chances of achieving its intended results,
and maintain stakeholders’ commitment to engaging in evaluation as they
collaborate to make the needed improvements.

Table 2.3 Types of Evaluation

Type of Evaluation Processes and Action Steps Needed

Planning Uses collaborative processes to

Evaluations
•• determine overall needs and problem to be solved;
•• determine which stakeholders to include in the program
planning and as its participants;
•• assess participant needs, characteristics, and working
conditions;
•• identify and address conditions and context that are likely
to support and potentially inhibit the program’s success;
•• clarify outcomes; and
•• assess stakeholders’ reaction to the intended program
plan.

Formative Uses collaborative processes to

Evaluations
•• assess using data whether the program is being
implemented as designed;
•• construct understanding of what is working, not working,
and why;
•• assess progress toward outcomes;
•• identify positive or negative unintended consequences;
and
•• study data in relationship to identified benchmarks
to inform revisions, improvements, or adjustments to
program design.

Summative Uses systematic evaluation processes to

Evaluations
•• collect data upon which to base judgments about the
program’s merit, value, worth, and impact; and
•• provide a summary judgment about the program’s
performance and impact.
 23
CHAPTER 2 Professional Learning Program Evaluation Overview   

STEPS IN THE EVALUATION PROCESS

Planning and conducting evaluations is a linear process in which the steps are
highly interrelated. The success of one step depends on the success of the pre-
vious step. While some recursiveness is involved, much of the work is sequen-
tial. For example, the evaluator would have difficulty in determining how to
collect data unless he has already decided what data to collect. The evaluation
process outlined in this book has eight steps (see Table 2.4). Each step will be
briefly described here. Subsequent chapters contain in-depth discussions of
each step.

Getting Ready to Start the Evaluation Process

Until a professional learning program is designed, it cannot be evaluated. If
professional learning leaders want to commission an evaluation of their pro-
gram, they first make sure that their professional learning program is ready to
be evaluated. “Designed” programs include outcomes, planned actions to
achieve those goals, and the resources necessary to implement the
plan. Designed professional learning programs meet the Standards for
Professional Learning (Learning Forward, 2011), district or state standards
for professional learning, or another set of rigorous standards for the quality of
professional learning.

Step One: Assess Evaluability

Now the evaluation process can begin. The evaluator or evaluation team
seeks to understand the professional learning program to be evaluated if she
has not been a part of the program planning process. If, for some reason, the
professional learning program is insufficiently designed, the evaluator helps
them do so. In some cases, the program exists but has not been clearly defined.
Based on their understanding of the program, the evaluator and program lead-
ers together determine whether the program is “evaluable” (i.e., ready to be
evaluated). This simply means asking, “Can this evaluation even take place?”
The evaluator examines the program’s design to determine whether it is
complete, sufficient, feasible, and logical. This involves examining the pro-
gram’s outcomes, standards, and indicators of success; theory of change; and
logic model—all critical to conducting a successful evaluation. Evaluation
planning depends on program planning, and in ideal situations, they are done
simultaneously. Chapter 4 covers assessing evaluability in depth.

Step Two: Formulate Evaluation Questions

Formulating good evaluation questions depends on first identifying all the
intended users of the evaluation, their needs and expectations, the purposes of
the evaluation, and the program’s outcomes. Clear purposes will help to shape
decisions about the overall design of the evaluation and guide the development
of evaluation questions and the selection of an evaluation methodology.
24  Assessing Impact
Once the users and purposes are clarified, the evaluator formulates the
questions the evaluation seeks to answer. These questions will direct the scope
and methodology of the evaluation (the framework). Creating good evalua-
tion questions requires thoughtful attention and may benefit from collabora-
tion among the evaluator, those who commissioned the evaluation, the
program manager or other professional learning leaders, and other stake-
holders. With mutually agreed-upon evaluation questions, the framework for
the evaluation can be developed. Chapter 5 focuses on formulating evalua-
tion questions.

Step Three: Construct the Evaluation Framework

Multiple methodologies exist for conducting evaluations. Evaluators of pro-
fessional learning programs tend to adopt either qualitative or descriptive meth-
odologies. Because of their sophistication, other methodologies might require
support from someone with deep expertise in evaluation. External evaluators
or technical assistance providers may be tapped for support in the design and
implementation of the evaluation, if necessary. The evaluation team selects an
evaluation methodology that aligns with the questions to be answered (i.e., rea-
sonable and feasible) and that will provide the evidence needed to answer the
questions. A thorough discussion of how to construct the framework is pre-
sented in Chapter 6.

Step Four: Collect Data

Once the evaluation framework is ready, the evaluator collects data using
the collection methods identified within that framework. The data-collection
methods align with the type of data required to answer the evaluation ques-
tions. Multiple sources of data can strengthen an evaluation study particularly
if they provide multiple perspectives on the same construct. Carefully designed
data-collection tools and well-trained data collectors will facilitate this often
labor-intensive process. In this crucial step, the evaluator manages the data-
collection process, making sure consistent and careful processes result in accu-
racy. Chapter 7 presents information about managing data collection and
selecting appropriate scoring processes.

Step Five: Organize, Analyze, and Display Data

Organizing, analyzing, and displaying data include summarizing, collat-
ing, synthesizing, displaying, and analyzing data to examine patterns, trends,
outliers, and anomalies. Creating visual displays or infographics to display
data facilitates the process of forming conclusions based on the data and sub-
sequent interpretation and reporting. Chapter 8 is a more detailed discussion
of these processes. Some of the processes may be technical and may require
the evaluator to tap the expertise of someone who is skilled in statistical anal-
ysis. Nevertheless, those practitioners skilled in using spreadsheets and com-
puting descriptive statistics will be able to perform most data analyses.
 25
CHAPTER 2 Professional Learning Program Evaluation Overview   

Step Six: Interpret Data

Interpreting data is the process of examining the analyzed data and
responding to the evaluation questions, based on the strength of the data. The
interpretive process, detailed in Chapter 9, requires the evaluator, program par-
ticipants, and others included in this phase of the work to form judgments that
are based on the analysis of the data and make recommendations about the
program. Because interpreting data is a process of making meaning, engaging
a wide range of stakeholders in this step enhances the meaning-making pro-
cess. This phase often uses preestablished criteria against which the analyzed
data are compared to determine the merit, worth, value, or impact of the pro-
fessional learning program.

Step Seven: Report, Disseminate,

and Use Evaluation Results
Reporting, disseminating, and using the evaluation results involves pre-
paring both written and oral reports of the evaluation results, sharing the
reports, and using the results to make informed decisions about the profes-
sional learning program. The evaluator may prepare multiple reports for
dissemination to different audiences to address their specific interests in the
evaluation. For some, a very comprehensive technical report is best, while
for others a brief executive summary will be appropriate. Using visual dis-
plays or infographics to share data and to allow the audience to see how the
data supported the findings increases readers’ ability to understand the
report. Evaluators support stakeholders in sharing the evaluation results
with the appropriate audiences, including stakeholders and professional
learning program participants. Various presentation formats for dissemina-
tion are discussed in Chapter 10. Once the evaluator prepares and dissemi-
nates reports, the evaluation process continues as stakeholders and other
decision makers use the evaluation results to make informed decisions about
the program’s future and about other professional learning programs, if
applicable.

Step Eight: Evaluate the Evaluation

A step not often included in evaluations is meta-evaluation of the evalua-
tion. Both evaluators and stakeholders benefit from assessing the quality of the
evaluation, procedures used to conduct the evaluation, and the perceived value
of the evaluation. This might be done informally or formally. Improving the
skills of the evaluator or evaluation team, design of the evaluation, dissemina-
tion of findings, or evidence collected are some reasons for evaluating the eval-
uation. This process benefits both the evaluator, by providing an opportunity to
reflect on his or her work, and the stakeholders, by ensuring an opportunity for
them to express their views about the usefulness of the evaluation. Both pur-
poses can lead to improved evaluation practices. Chapter 11 lends insight into
this last step of the evaluation process and will encourage practitioners to
“finish what they start.”
26  Assessing Impact
Table 2.4 Steps in the Evaluation Process

Planning phase

•• Assess evaluability Is the program sufficiently designed and ready

for evaluation?

•• Formulate evaluation What questions do we want to answer in this

questions evaluation?

•• Construct the framework What is the plan for the evaluation?

Conducting phase

•• Collect data What data are we collecting, how, and from

whom or what?

•• Organize, analyze, and How are the data organized, analyzed and
display data displayed in order to make meaning of them?

•• Interpret analyzed data What do these analyzed data mean?

Report phase

•• Disseminate and use the To whom are we disseminating the findings and
findings how? How will we use the findings?

•• Evaluate the evaluation How well did we conduct the evaluation? What
did we learn about the evaluation process to
improve future evaluations? What did we learn to
strengthen our evaluation skills?
 Evaluating
Professional Learning

3
EFFECTIVE PROFESSIONAL LEARNING
Three fundamental conditions are necessary for effective professional learning.
If the conditions are not in place, further planning and design work should
precede the evaluation. If the conditions are not completely addressed in the
planning phase of the program and stakeholders opt to continue with evalua-
tion, the evaluator may decide to conduct only a formative evaluation rather
than a full summative evaluation. The absence of any condition has the poten-
tial for a significant negative impact on the outcomes of the professional
learning.

Condition 1: The professional learning program is data driven,

research or evidence based, and well designed to meet
standards for effective professional learning.
What contributes to professional learning that influences educator practice
and student learning is no longer a mystery. Research over the last several
decades and attention to the development of standards for professional learn-
ing (Learning Forward, 2011) make it clear what constitutes effectiveness.
There is no longer a need for guesswork nor for inadequacy in the design and
implementation of professional learning. Professional learning for educators
must rest within a comprehensive system of professional learning that estab-
lishes the vision, standards, roles and responsibilities, resources, assessment,
and evaluation (Killion, 2013). Such a system should exist at the state and dis-
trict levels to ensure that every member of the education workforce has equita-
ble access to effective professional learning aligned directly to his performance
standards and to his clients’ performance standards. For teachers, clients
include peers and students. For administrators, clients include teachers, peers,
and students.

27
28  Assessing Impact
Professional learning leaders use data about students, educators, and
systems to design effective, standards-based, data-driven, and research- or
evidence-based professional learning programs. They use student data and
analyses of root causes to understand the needs or problems to address. They
establish outcomes for students based on the identified needs. Next, they
examine educators’ learning needs in relationship to student outcomes and
establish outcomes for educators that identify the necessary changes in
knowledge, attitudes, skills, aspirations, and behaviors to lead to student suc-
cess. They examine attributes of the system including the context, culture,
and structures in which the professional learning will occur because the eco-
system influences success and often must be improved for professional learn-
ing to have its full impact. They study research and evidence to determine
what types of professional learning have contributed to achieving similar
outcomes in similar contexts. With this information, they develop a theory of
change, a logic model, or both that define the program’s planned, sustained
actions to achieve the outcomes. They allocate appropriate resources to
implement the planned actions. With these elements in place, they may begin
to plan both the formative and summative evaluation. When professional
learning programs are grounded in data that define the needs and root
causes, they are likely to take advantage of critical opportunities for learning
and support to fill gaps.

Condition 2: The education agency seeking to evaluate

professional learning has the capacity, including fiscal and
human capital, to implement both the program and the
evaluation with fidelity to their designs.
A plan for a professional learning program or its evaluation is not equiva-
lent to a program’s implementation or evaluation of that program. Without
capacity to put the plans in action, neither the professional learning nor its
evaluation is likely to succeed. Capacity means human, fiscal, time, technology,
and material and leadership advocacy. Both professional learning and its evalu-
ation require these resources and advocacy.
Some worry that redirecting any resources to an evaluation drains resources
from the professional learning program and deprives participants of services. On
the contrary, evaluation adds value because it provides valuable information to
improve decision making and make better use of resources. Evaluation provides
program stakeholders with essential information about what works and what
does not work so they can strengthen their program operation. For example, if
teachers know that their application of new learning is missing a critical compo-
nent, they are better able to adjust their efforts early in the process. Principals
who learn that they are not reinforcing the use of the new practices in conversa-
tions with teachers can refine their practice of supporting teachers and focus on
integrating more attention to the topic in both learning walks, interactions with
teachers one-on-one and in meetings of facilitators of learning teams. District
leaders who find that recent requests to schools unrelated to the professional
learning priority are sending mixed messages to school staff about what is
 29
CHAPTER 3 Evaluating Professional Learning   

important and can modify their demands on school staff so they are able to focus
on the priority areas.
Typically, the more complex an evaluation design, the costlier it is in
terms of fiscal and human resources. Yet even low-cost evaluations are valu-
able because they can produce useful information. When decision and policy
makers want more rigorous evaluation, such as a program’s impact on stu-
dent learning, they must be ready to invest the resources and leadership
advocacy to support it so that the evaluation is reliable, valid, and valuable.
In an era of accountability, the cost of evaluation is a requisite part of any
fiscal planning.
Newcomer, Hatry, and Wholey (2015) identify some typical evaluation
costs that include (1) the lead evaluator who plans and conducts the evalua-
tion; (2) engagement of stakeholders who serve as a part of the evaluation
team; (3) participants in professional learning who contribute data to the eval-
uation; (4) staff and support staff who coordinate, collect, manage, and orga-
nize data, and prepare reports and disseminate the findings; (5) costs for
compensating or replacing services, if necessary, to support participation in the
evaluation; (6) materials and supplies; and (7) technology resources. They also
acknowledge that evaluation may cost potential loss of goodwill among partic-
ipants who feel burdened by the extra workload associated with engaging in the
evaluation. Another potential cost is the time the policy and decision makers
invest in reviewing the evaluation report and making decisions based on it.
Most of these costs can be considered business as usual in a learning organiza-
tion committed to continuous improvement.
Evaluation is a value-added process. When an evaluation is designed
thoughtfully, it can become a part of the intervention itself. For example, when
reviewing literature about professional learning in K–12 writing to design the
program and its evaluation, an evaluator noted the importance of teacher col-
laboration in learning communities for achieving improved results for students.
To apply this finding, and practice and increase the amount of collaboration
among program participants, the evaluator selected data-collection methods
that reinforced collaboration such as focused groups and collaborative work
products rather than those that were individually oriented such as a survey or
interview. The evaluator engaged teachers in conducting collaborative class-
room walk-throughs to observe student and teacher behaviors during writers’
workshop and used scoring conferences to assess several samples of student
work at different times during the evaluation.
Not only were these experiences rich opportunities for data collection, but
they also modeled structures for teacher collaboration with a focus on student
achievement. The evaluation design, in this case, extended the professional
learning intervention and might have contributed to the impact of the pro-
gram. The decisions the evaluator makes may also positively influence partici-
pant ownership and commitment to the program and the usefulness of the
evaluation. In addition, the evaluation results may be more understandable
and useful to all stakeholders. When focused on carefully crafted questions that
reflect stakeholder priorities, evaluations are perceived as valuable tools and
normative practice for continuous improvement.
30  Assessing Impact
There are many options for evaluation designs, some of which are costlier
than others, so it is important to weigh the need for a complex design and its
projected cost against the perceived value of an evaluation. When evaluations
are designed to address specific questions intended users want to answer, the
evaluation is an investment rather than a cost. Involving the users in decisions
about the evaluation is critical.

Condition 3: Key stakeholders in education

agencies intend to use the evaluation
results to make decisions about the program.
Justifying the costs of evaluation requires that the investment be for a
worthwhile, utilization-focused evaluation designed for a specific audience
(users) and for specific purposes (uses). When evaluations are constructed for
identified purposes and for identified stakeholders, the perceived value of the
evaluation increases, as do stakeholders’ understanding of and interest in the
evaluation, especially if they have been actively involved in planning the design
of the evaluation and the eventual use of the results (Patton, 2008). The value
of evaluations can be determined by three factors: the reliability and validity of
the evidence; the credibility of the evaluation to those who commissioned it
and who plan to use it; and the usefulness of the findings in decision making
(Patton, 2015). In describing utilization-focused evaluation methodology,
Patton (2005) comments that “intended users are more likely to use evalua-
tions if they understand and feel ownership of the evaluation process and find-
ings [and that] [t]hey are more likely to understand and feel ownership if
they’ve been actively involved. By actively involving primary intended users,
the evaluator is preparing the groundwork for use” (p. 423).
The intent to use evaluation findings demonstrates and sustains interest in
both the program and its evaluation. The evaluation itself sends a message
about the importance of the program and has the potential to increase partici-
pants’ perceived commitment to the program. It may also increase participants’
interest in and effort exerted toward the program. The expression, “What gets
measured, gets done,” is an apt adage of the interrelationship that exists among
evaluation and stakeholder and participant interest and effort.
Before an evaluation is conducted, the evaluator determines who the pri-
mary stakeholders of the evaluation are and who will be the primary users, and
what will be the primary uses of the evaluation. In addition, the evaluator and
professional learning leaders determine the degree to which their program
meets the conditions described and make necessary adjustments prior to
launching the program or its evaluation.

ANALYZING EVALUATION APPROACHES:

UNDERSTANDING “GAPS” AND “BOXES”
A major challenge in the evaluation of professional learning is the frequent use
of overly simplistic methodology for evaluation. This often occurs when inexpe-
rienced evaluators assume that available data are an adequate representation
 31
CHAPTER 3 Evaluating Professional Learning   

or substitute for other data—for example, assuming that a survey of partici-

pant reactions to a learning experience is a viable substitute for a measure of
participant learning or implementation of learning. Implementation or impact
evaluations of professional learning require examining, analyzing, and mea-
suring a series of interrelated changes that work together to produce the
intended outcomes. This series might be considered a pathway or sequence of
learning that begins with an educator’s decision to participate in the program
and continue through full, deep, and consistent application in the workplace
and eventual changes in clients. Providing information or documentation about
the duration of participation, the number of coaching visits, or a description of
the content will not suffice as a measure of acquisition or application of learn-
ing, nor the impact of changed practice on clients. These approximations of
outcomes are insufficient to make strong claims about the program’s impact.
Stephan Katz and Lisa Ain Dack (2013) define learning as “permanent
change in thinking or behavior” (p. 3). The pathway to learning that leads to a
change in professional practice, way of thinking, and results for clients (think
students for teachers or teachers for administrators) cannot be reduced to
merely providing information. Professional learning that changes practice,
thinking, and results for clients requires multiple, interconnected actions care-
fully designed to achieve the desired outcomes. To determine if professional
learning alters educator practice and results for their clients, evaluators must
gather evidence along the pathway of activities designed to achieve those ends.
Yet, in some professional learning, the pathway to the outcomes is under-
developed. It is riddled with holes or missing entire sections. Gaps in the path-
way negatively influence the potential for professional learning to succeed. The
gaps may occur anywhere from the initial planning phase to the expected com-
pletion. They may rest in the culture of the environment in which the learning
occurs or is implemented. They may be in the commitment of educators to
engage in continuous learning and improvement. They may be in the lack of
resources, leaders’ advocacy for professional learning, clarity of outcomes or
expectations, or sufficient implementation support to achieve high levels of
professional practice with integrity, fidelity, and consistency.
When gaps occur in professional learning, the evaluation is also likely to
fall short because evaluators will have fewer opportunities to gather data along
the pathway. The absence of data may result in weak or faulty findings or con-
clusions, insufficiently supported with evidence, about the relationship between
the program’s action and the outcomes. There is another problem that may
arise. The program may be well designed and include adequate actions
sequenced logically and sufficient resources to achieve the outcome, yet its
evaluation may be weak. Weak evaluations have poor designs, collect insuffi-
cient evidence to support claims, skip data collection about critical actions
along the pathway to demonstrate relationships, or use approximations rather
than authentic representations of the intended changes.

Black-Box and Glass-Box Approaches to Evaluation

When shortcomings occur either in the program design or in the evaluation,
the “black-box” appears. It is an opaque or empty space that fails to articulate
Exploring the Variety of Random
Documents with Different Content
to live in the land. But the boast merely means that it is not
permitted to add black slaves from abroad; they cannot live in
England; nor do I think they could. I do not comprehend the boast,
unless on the ground that it would be an expensive as well as
useless cruelty to land even blacks, merely to have the trouble and
cost of burying their carcases.
I have called these low-castes Serfs, disregarding the barbarian
fiction which calls them free. Not long since they were slaves under
precise law; now they are so by universal custom. When they were
legal slaves they had some care and protection; there was a tie
existing between master and servant; hearty service and
affectionate concern rendered the relation not merely supportable,
but positively advantageous. The tie is severed; there is neither
hearty service nor affectionate concern. The master possesses
everything as before, but he is no longer obliged to maintain his
labourers. These are numerous; they must work or starve. Whilst
they work they get enough perhaps to live; no longer able to work,
mere pauper-life in poorhouses and the pauper's grave await them.
Nor do the masters even pay for these; they have cunningly
contrived to have the expense borne by all who have anything to be
taxed. Thus the severance of the ancient tie has only enriched the
High-Castes and freed them from all obligation to care for the
labourer, and sunk him into a condition of hopeless and debasing
poverty. The freedom is all on the strong side; the slavery more
abject and less softened by humane sentiments.
Now there are a few, who have some dim perceptions of these so
obvious features to a disinterested spectator. They see that it is a
poor compensation to the wretched misery which holds thousands
hopelessly in its grasp, to point out an occasional accident of escape
—where some one, more gifted and more fortunate than his fellows,
happens to rise into comfort and slight esteem! These noble men,
the harbingers of light, who try to see and to act honestly, in spite of
early prejudice and habit, perceive that there is no hope for these
serfs, unless they can be moved with a higher interest. They think
they discover a chance to move them by offering them knowledge,
without, or nearly without, cost. But it is doubtful if they be not too
low, too brutal, to care for knowing anything. Then, "they must be
forced to send the children, to be taught—they must be whipped to
School." This is resented as an outrage on the freedom of the serf—
as an invasion of his family rights—as a positive, additional, tax and
burden. For he gets something from the petty work, or from the
begging or thieving of the children, and now the Master takes that!
Yet, probably, this is one needful thing—to take the children into the
hands of the State, in every case where the natural guardian is unfit
to properly care for them. But the State cannot half take them. It
cannot take anything of the present pittance, and claim to have
compensated by giving words instead. It is cruel to say to him who
starves in body, "Starve—I feed the mind!" A poor parent cannot
receive even knowledge in exchange for bread. And it cannot be
asked of him, in his low estate, to exchange the little added support
of the child's wage for the, to him, useless words of knowledge. In
the face of want one cares only for bread! Therefore, the State
which teaches must also feed the poor—or see to it that the honest
poor be first fed. If the parent can only feed by the help of the child,
the State must not arbitrarily assume to be Master and Judge—
saying, "Come to school—and starve, if must be."
The High-Caste, secretly, clog and obstruct all attempts to raise the
low. Learning belongs to the master—ignorance to the serf. It is
enough for him to obey and work. There will always be poor, and
vicious poor. It is better to merely watch and guard against an Evil,
for which there is no remedy. To give instruction to the low-orders, is
to arm demagogues with a dangerous weapon. "'A little knowledge
is a dangerous thing'—it only enables the multitude to see just what
it suits the purpose of the Agitator to show! There is nothing but evil
in these grand measures. All must be left to individual effort; and to
the Priests. These must work as comes in their way; instructing
those who wish, and encouraging those who dutifully obey, and
attend to the labours imposed upon them by divine Providence"
(Meaning, that Jah has ordained, from all time, that some must be
"Hewers of wood and drawers of water"—a quotation from the
Sacred Writings).
In this manner, the High-Caste, when it condescends to the subject
at all, dismisses it. Indeed, this Caste, the Master-Caste, really feels
no other concern in the low orders, but a concern for their peaceful
subjection. To this point they direct so much care, as to have always
trained bands of braves, and strong, picked, well-ordered men,
called Police, ready at hand. So, in case the wretched, degraded,
and despised serfs and thieves, should dare to raise any stir,
disturbing the ease and enjoyment of the luxurious High-Caste, they
may be shot down without mercy!
Necessarily, the elevation of the low-classes will be very gradual.
Many of the Priests, wishing to enlarge and maintain the influence of
the Superstition, actively exert themselves among the honest and
industrious poor. And some of these Bonzes are as benevolent as the
traditions of their Caste and of their Idolatry will permit.
It is doubtful if the present condition of the masses of the English
Barbarians be so manly and independent as ages ago—when they
were sufficiently intelligent to move in their own cause, and were
really of some importance in the State. Unfortunately, they did
remove from their necks the pressure of immediate, personal
service, only to accumulate upon them, as a Class, the whole weight
of the landed and trading interests. As a whole, therefore, they are
more servile, more abject, and more dependent; and the few
individuals who may raise themselves above the level of their class
cannot even flatter themselves in this to have gained. There never
was a time when these individuals did not exist—it is not clear that
their numbers have increased.

CHAPTER XI.
 OF THE HIGH CASTES: SOME PARTICULARS OF THEIR DOMESTIC
AND SOCIAL CUSTOMS.
In this chapter I shall try to show some of the peculiarities of the
opposite extreme of Barbarian life. From ignorant poverty, verging
upon crime, crime and vice; we are taken to luxury, also verging
upon crime, crime and vice—though under very different forms. The
All-wise and Sovereign Lord knows how to judge each class of
offenders!
The High-Caste is very exclusive—it will not, if it can avoid it, notice
one of a lower order; and never will do so unless it has some selfish
end in view. This cold-bloodedness characterizes all Castes. When
the Barbarians, therefore, chance to meet, and being of near Castes,
cannot be distinguished by dress, they never touch or address each
other—but stare rudely up and down the person, to see if it will be
safe to be civil, the one to the other.
In general, however, the two Higher-Castes present so many
features in common, that a spectator may regard them as one. Both
look upon all useful occupation as shameful; and whilst it is hard to
call up a blush for anything mean, detection in any honest work
covers with confusion!
The women of this Caste appear everywhere in public, with the
same boldness as men. They dress in laces, silks, satins, velvets;
richest furs, feathers, shawls, and scarfs. Are so addicted to these
things, and to costly jewels, and ornaments of gold, precious stones,
and the like, that a fortune is often carried upon and about a fine
Lady. (Lady is for the female like Lord for a male). In truth, a Lady
only lives for two purposes—to dress, and to marry. I ought to add
another, but whether it be subordinate or chief I know not; in fact, I
hardly know what it is. We have no very near word. It is a
something of which you hear constantly—to flirt. To dress, it is
necessary to shop [keat-hi]. This, is to buy the innumerable articles
which make up a fine Lady's wardrobe and personal appointments.
Heaven and earth, and all the lands beyond the great seas, are
ransacked to gratify the insatiate demands of Barbarian High-Caste
women. The finest paints for the cheeks and eyelids, the most
precious stones for the ears, the neck, the wrists, the fingers; the
most delicate perfumes, the pure gold, the richest furs and feathers,
spices, oils; the laces, scarfs, silks, embroideries;—an endless
variety. Shopping is, therefore, the serious occupation (subsidiary to
husband-catching and flirting) of ladies. Many ruin themselves, or
their fathers, their husbands, or relatives, in this expensive luxury of
idle vanity. High-Caste women show themselves in public, sometimes
on foot, but, more generally, lolling, with poodles in lap, within open,
grand carriages, drawn by great, high-stepping horses. (Poodles are
nasty dogs). They attend the Temples, waited upon by solemn
servants, clothed in showy colours, and bearing ostentatiously the
Sacred books. They are conspicuous, when at the Temple, for
audibly accompanying the Priest in the Invocations and Confessions:
"miserable offenders" seeming to be a phrase rolled like a sweet
morsel, and having a savour of repentance and humility, very
edifying!
The men do not appear very numerously with the women—leaving
them to do as they please. The men going off to their own exclusive
pleasures: gambling, betting, racing, boating, hunting, and other
things equally useful and improving.
All through the night, which is the time of High-Caste revelry, the
streets where the great live resound with the noise of the carriages,
constantly busy with the transporting of the High-Castes to and from
the Theatres, the Dances, the places of Amusements, the Dinners,
the Parties, Routs, and visits. To mark the difference of the Upper
from the Lower, time itself is reversed; night is taken for life and
sport, and the day for rest, gossip [Quen], and shopping. In nothing
could the difference be more striking. The luxuriousness of mere
self-indulgence, which takes no heed of the usual order of nature,
and does not suspect that day has any better use! When in the
country, there is the same round of busy nothings. Visits, feasting,
drinking—dancing, routs, and parties. Women taking the lead
everywhere and in everything. Here, as in town, the business of life
with women is to flirt, to marry, to dress—the last should be first.
The men add to the follies of women some things more robust, but
not more useful. Betting, horse-racing, riding over country with
dogs, pursuing timid creatures—or gambling, drinking, and feasting.
When I first arrived in England, I was amazed, and supposed all
women were shameless [ba-tsi] that I saw, whenever I went in
public. In our Flowery Land this class [ba-tsi], under the strictest
survey and care of the magistrates, are barely tolerated, and forced
to the most scrupulous decorum of dress and conduct. With us no
modest woman of any rank ever appears in public. Therefore my
surprise and astonishment may be imagined. Afterwards these were
moderated, and I could make allowances for the force of custom.
None the less the custom is remarkable, and will receive attention
elsewhere.
The mode of dress is simply wonderful. It is ever changing and ever
indelicate and monstrous—especially for women. When I first saw
one of these with a huge hunch on the top of her back, I thought
the person was afflicted with an enormous tumour; but when I
observed the same thing on all hands, I saw my mistake. The great
hunch was no more than a machine placed on top of the seat, under
the outer garments. The effect is something amazing. The women in
walking also wear the robe drawn as tightly as possible back and
over the hips, so as to display the whole form from head to foot in
front, and also in rear, excepting at the back-seat where the
protuberance is. Here the clothes are clustered, and hang down in a
trail upon the ground! The feet are thrust into very high-heeled
shoes, or boots; so, in walking, the woman stoops mincingly forward
with short, unsteady steps, as if pinched at the toes, rattling her
heels upon the pavement, and tossing her back-gear and headdress,
and showing off to an astonished observer (unused to the
apparition) something to be remembered! On every little occasion
taking up her trail, and discovering legs and ankles.
At home, when receiving male and female friends to dinner, the
women do as they please—also in dances, routs, and the like. I was
invited, soon after my arrival, to dine. I had looked at a Book of rites
and ceremonies for the great, and hoped to get on tolerably well. On
arriving, my first mistake was to address the servant as Illustrious,
taking him for the master. In many houses the servant, dressed like
the master (being much more of a man in appearance), may well be
taken for him; but in some houses the servants are made to wear
badges and colours of their station. Women are very choice about
these men-servants, and will not have one unless he have very
large, well-formed calves [fa-tze]. I have heard that the rogues
supply this requirement by adding so much fine hay to the leg as will
give due swell and figure!
Upon being shown up to the room, where I was to address myself
first to the Lady—the Illustrious wife—I made my next blunder. The
lady was large, full of flesh, rather red, with bright eyes. Another
lady, just moving away, trailed her long robe suddenly before me—
my foot caught and held her. She turned her white shoulders upon
me, frowned—at the moment I stumbled, and recovered myself
awkwardly, with open hands full upon the ample bosom of the
Illustrious! Ah, my confusion! I could not recover my composure. I
could see nothing but necks, shoulders, backs, bosoms of women,
and eyes flashing at me—heads, and feathers and jewels—lights,
noise, confusion! I got away—never knew how.
Women, when undressed in this indelicate way, are said to be in full
dress. I think this is a sly sarcasm of the men. The men, however,
dress in a manner not at all better. When in full dress, they put on a
ridiculous close garment, slit up behind, and very scant, with two
tails, which pretend to cover the hinder parts. The trowsers (an
"unmentionable" article for the legs), no more than the under
garment worn by us, is the only covering for the legs and lower part
of the body! Imagine the indelicacy! In this style of full dress, the
women and men of the High-Caste Barbarians meet and mingle
together everywhere, and at all feasts, revelries, and dances.
In the shows within-doors the same mode prevails. At the public
spectacles, in full view of thousands, ladies sit exposed to the gaze
of men, who often level at them the magnifying glasses taken for
the purpose! Critically examining the exhibition before them from a
distance of twenty feet [tu-fai].
The dress of women on horseback is as follows:—The head is
covered with a man's head-gear, round, hard, high, black in colour,
with a narrow rim. The bust and body are just as tightly fitted as
possible, the hips and figure exposed in exact shape (how much
made up no one can more than conjecture), and the legs covered by
the dress falling over them long and full. The woman sits on a side-
saddle, one leg well up over a horn of the saddle near the front top,
and the other supported with the foot in a steel rest. She is lifted by
a male servant, relative, or friend, into her perch. And when she,
with the little whip in hand, takes up the long strips of leather by
which she guides the horse, and starts off, there is a show the most
curious! Up and down, with every motion of the horse, she bobs
[Ko-bys], exposing, to any one looking after her, the most precise
model of herself! but in an attitude and costume so remarkable, that
I never saw even the accustomed Barbarians disregard an
opportunity to see this show, however indifferent they may usually
be. Nor do I think that the Barbarian women esteem any exhibition
of themselves superior to this.
In the country you will see several apparitions of this kind, urging
their flying horses after men and dogs, all chasing pell-mell some
poor hare, which, running for cover, is pursued by a crowd of men
and women on horseback, with dogs, yelping, barking, men blowing
horns and shouting; the women on the horses leaping over fences,
ditches, and urging their horses as wildly, boldly as the men—and
sometimes in all respects as skilfully and well! This Sport is
considered by the Barbarians to be very manly—nor do they
consider a broken back, or even neck, as any objection to it!
The Rout is a favourite amusement with the High-Castes. So named
from the confusion of armed men when routed—put to flight. It is to
get together just as many people of both sexes as possible. With no
sort of regard to the size of the house, but only to show how many
of the High-Caste will respond to the invitation.
In full undress the ladies and gentlemen (Barbarian style for any
High-Caste man) crowd into the house. Every stairway, every hall,
room, chamber is filled. Refreshments are provided, but the flux and
reflux of the people render all eating and drinking very difficult. The
women flash in jewels, pendants in the ears, sparkling brilliants on
arms, busts, ornaments of flowers and gems in the hair, jewelled
fans in hand, perfumed laces and scarfs, tinted, and flushed, and
adorned, exposed to bewilder and intoxicate the men—in fine, in the
pursuit of husbands, or bent upon flirtations! These entertainments
are designed for the very purpose of excitement and match-making.
"Society is kept alive—life is made endurable by these things," the
High-Caste women say. They have no other business but to attend
to such matters; and to them Society looks to save it from
dissolution and despair!
In the Rout all is confusion and opportunity. The young people, the
old people, the highest and the lowest (permissible), are thrown
promiscuously together. Women and men mingle, jostle, jamb,
crowd, wriggle, and writhe together as best they can. The young
lady suddenly finds herself quite in the arms of the young man who
has saved her from a fall; and he, in turn, "begs pardon" of some
woman, into whose lap he has almost been thrown by a sudden
press.
Acquaintances may be made, flirtations begun, ending in something
or nothing. But Society has had its excitement, and its members
their chances for mere idle display, gossip, sensual gratification, or
the more serious business of High-life—fortune-hunting by men and
husband-catching by women! The Waltz and Dance are, however,
the great game (for they are really one) of Barbarian life. Every
Caste, according to its ability, dances—the low imitating, to their
best, all the "airs and graces," dress and flirtations of their superiors.
In the Waltz, when the music strikes up, the man takes the woman
about the waist, standing with the other dancers in the middle of the
floor, and she leans upon his shoulder interlocking the fingers of her
disengaged hand in his. In this close position, they begin to wheel
around, around; one couple follows another about the clear space
left for them, till many couples are seen twirling, whirling about,
around to the sound of the music—ever in this wild, whirling sort of
a gallop, following one after another, rapidly! The long trails of the
woman are held up, the embroidered skirts fly out, the silken shoes
and hose flash; she is held close and more closely in the supporting
arm, her cheek almost touches, her bust, neck, and face glow with
excitement, the eyes and jewels sparkle, the man and woman whirl
about, till intoxicated, dazed, and nearly exhausted, she sinks upon
his arm and motions for rest, and he half supports and half leads her
to some soft bench or chair! Such briefly is the Waltz. The dance is
the same thing nearly, only more variety of movement is introduced.
The whole object is to bring the sexes together, and keep Society
alive, as before. Flirtation and match-making being main elements of
social life.
The manners of the High-Caste are not really more refined than
elsewhere; only there is a cool tone. Nothing must surprise, nothing
confuse, nothing abash. A blush must be as rare as a laugh. A young
woman seeing a young man gazing at her with bold admiration,
must coolly look him down—if she please. His is an action of mere
rudeness, or should be, when directed to a virtuous woman: but no,
"a man may gaze upon what is everywhere exhibited for his
admiration—may he not?" And yet, with strange inconsistency, a
woman has a right to complain if a man, captivated by the very
means designed, too rudely express his pleasure. And one man is
required to chastise another for the rudeness to his relative, though
he know that, in the nature of things, the female should expect what
she encounters—and more, the complexity is further involved, that
though one man must call another to account for this sort of
rudeness, yet every man indulges in it!
Young people, in public, of the two sexes, without shame appear in
close intimacy—and will look upon statues and paintings of naked
women and men, talking and criticizing, examining the works and
looking at them in company, without confusion, or appearance of
there being any indelicacy. As if, in fact, in the bosoms of the High-
Caste there did not exist any of the passions of ordinary mortals!
There are very numerous galleries of Art, where statues, paintings,
pictures, models, and the like, are shown, which are always crowded
by High-Caste women, children, and men. And shop-windows are
made attractive by displays of pictures of nude, or half-nude, women
and men, who act in the Plays, or who are notorious in Spectacles.
This sort of indecency prevails; and strikes one, not used to it, with
an unpleasant surprise. He knows not what to think of its
significance—have all his ideas of decency been indecent?
I am not able to say much of the interior life of the family. I was told
that a happy family was rare—quite an exception. It is only where
the wife rules that any peace is secured. The wife is allowed to do,
generally, in Society and at home, as she will. The husband goes off
to his pastimes and pursuits. Children whilst young are committed to
the care of servants, and when older sent away to be educated and
trained by hirelings.
The daughters, when grown, often move the jealousy of the mother
by attracting more attention from men—they are often snubbed and
made to dress unbecomingly, so that the mother may shine.
Marriage among the High-Caste is an arrangement for an
establishment; and to secure the succession of family name and
title. To these ends great care is given to the money question. The
man demands money for taking the wife. Domestic happiness is
hardly thought of; unless, occasionally, by very young people, and
they are laughed out of their ridiculous romance.
In the marriage ceremony, the wife, in the presence of the Idols,
and following the Invocations of the Priest, solemnly promises to
obey the husband. But this is regarded as a mere form. Any husband
who undertakes to enforce obedience, finds himself branded by
Society, as a "brute!" Much of the infelicity in marriage rests upon
this false basis. For, with the virile instinct, man naturally expects
obedience; yet has, in his unmarried days, fallen in with the false
notion of woman's superiority in delicacy and moral virtue. This
peculiar affectation colours all Barbarian intercourse with the sex. It
has its root in the Superstition, possibly; where an immaculate virgin
gives birth to a Son of god-Jah! who is the Christ-god. Thus, woman
came to be mother of God!
From this, very likely, followed all the false worship and gallantry of
the barbarians; who still, keeping up this mode of treating women as
superior in excellency, could scarcely deny to them a superior place
in the family. Assumed to be absolutely chaste and pure, they are to
be implicitly trusted—nor to them is there impropriety! Hence follows
the fine Art exhibitions—the undress dress; the waltz; the mixed
crowds—the everything, where women, according to the ordinary
feelings of cultivated men, should not be, or be in a very different
way. But the man before marriage, and afterwards, too, (excepting
to his own wife), pretends to look upon woman as a divinity—as
something far above him in moral goodness! After marriage, it is
difficult to dethrone this divinity—the man has not a divinity at the
head of his family; but all his friends (male friends) pretend to think
so; Society says so; and he is himself compelled to pretend to the
same thing. Under these circumstances he will never be likely to get
much obedience. None the less, a struggle commences; the man
persistent, strong; the woman unyielding, crafty; the family divided;
the children demoralized; a false and wretched farce of conjugal
Play, so badly acted as to deceive not even Society! and finally
ending in the Divorce Court.
This is the tribunal where Causes Matrimonial are settled; and, if one
may judge from its Reports in the Gazette, conjugal contention is
exceedingly common. For the public cases must be few, compared
with those where publicity is avoided by private arrangement.
Doubtless, a fine man and an excellent woman may unite, and live
happily together, in spite of the unfavourable conditions. But, more
commonly, the high-minded man, really believing in the superior
purity of the sex, and her greater moral delicacy, finds his Ideal to
be too high; and without absolute cause to quarrel; in fact, seeing
that his Ideal was itself only an error of the prevailing delusion; ever
after struggles to bring himself into harmony with the existing fact—
to love and respect a woman and only a woman, with a woman's
vanity, love of excitement, frivolity and caprice—a very weary work.
The woman, too, still flattered, and exacting the devotion which her
lover (now her husband) gave to her in his days of delusion, thinks
herself treated with coldness; and, gradually, by her unreasonable
complaints, estranges altogether the husband, whom she, too, tries
to forget, in the admiration, flatteries, and excitements of Society!
The affectation and falsity, therefore, respecting woman, tends to a
fundamental error in the relation of the sexes and the ordering of
the family. It is a strange and almost fatal error to give this
exaltation to woman. No doubt, a real trust and respect tend to
secure, in some degree, the virtues accorded; and this true respect
of an honest man, who places his wife, or his relative, before himself
in purity, challenges the best of nature in the female. But man has
reversed the true order, and run counter to the true instinct of the
race (quite as strong in the female as in himself), when he thus puts
woman before him, in anything. What authority is there for this
reversal of the natural order? Why is woman more moral, more
chaste? There is nothing in the nature of things, why the man, here,
as in all things, should not be, as he is, the superior—the master. In
morals he should be her guide, her teacher, her best support. That
Society is, indeed, unsound, wherein the man may be low and
sensual, and fancy, or pretend to fancy, that the woman is better
than himself—it is a delusion. Man gives the real character to any
Society—the woman will not be, cannot be better than the man. The
English Barbarians, in spite of the absurd falsity of their customs,
must have some tolerably happy families. The innate perception of
the eternal fitness of things will cause many couples to arrive at a
proper method. The wife, without exactly admitting it, even to
herself, submits to her husband; and the husband, without exactly
commanding (except in rare instances), feels that he is really the
head of the house—and the family gets on pretty smoothly, because
living in the natural order. But, in general, the struggle for mastery
destroys either the existence of the family, or all attempts at
affectionate ways of living. To avoid public scandal, the members do
not actually separate; but all harmony and true domestic life are lost
—and life is a dismal and disorderly rout.
The exaltation of the sex and the complete freedom allowed to them
belong to a state of society, if any such there be, where man is still
more excellent. There, indeed, a bright and beautiful ideal is made
real, and men and women know how to love and to obey; and love
is as true as the respect and the obedience. The Barbarians, full of
immorality, of rudeness, of strong passions, of selfishness, controlled
by a false conception founded in their Idolatry, act, in respect of
their women, as if purity, cultivation, generosity, and the highest
morality, everywhere existed! This, so false, is well-nigh fatal to
them. Yet, it is only an illustration of the uncultivated and confused
state of mind, even in the highest, that so simple a thing as the
natural order governing the relation of sex and family is not
comprehended; and that their Society is saved from absolute wreck
only by the strong and controlling instinct of nature, which, in spite
of obstacles, does bring the female into subjection to the male—at
least to an extent sufficient to make life possible!
None the less the disorder of households is dreadful. Sons and
daughters, as they grow strong, assert themselves [Quan-hang-ho].
They act and speak (and in this follow the wife and mother) as if the
sole business of the father was to give the means of selfish, idle
indulgence. This would not be so unjust among the High-Caste, but
it descends to all grades, and the middle orders are content to see
the father toil at his business till overworked, or ruined altogether, in
his efforts to supply these daily exactions. No doubt he himself is a
victim to the whole vicious falseness—yet the cold-bloodedness of
this conduct on the part of children and wives is remarkable.
"Obedience," or "gratitude!"—Words sneered at, laughed at!
The daughters, directed by Mamma [na-ni-go], are taught to dress,
to look modest, to practise all those arts by which they may attract
the male and secure husbands, and are exhibited in public places
and in Society accordingly.
The sons are sent off to be taught. In the Halls of Learning they
acquire but little of the knowledge paid for in the Lists, but a great
deal of that which does not appear there. A youth may have
entered, at least, honest, moral, and generous—he still leaves
unlearned, but dishonest, corrupt, selfish—he has acquired that
knowledge most sought for (even by his parents), a knowledge of
the World [Quang]! In truth, the youth instinctively feels that it is
better for his success in life to know the World than to know Letters.
He acts upon this feeling, which thrives in the demoralised
atmosphere which he breathes. Father is called Governor, and is
regarded as a sort of creature to be made the most of! The money
allowed (perhaps too ample for really useful purposes) is spent in
things foolish and hurtful. Money and time are wasted. The latter is
valueless, to be sure, to these youths anywhere—but the money
may be wrung from relatives, who put themselves on short diet to
enable the son or brother (who is defrauding them) to appear well in
Society! To perfect himself in the learning which he feels to be
effective, he devises new methods of wringing more money from the
Governor, who begins to protest. To drink, smoke, lounge about with
easy and cool impudence; to stare into the face of women; to bet,
gamble; to get in debt, and curse the creditors who presume to ask
for pay; to make, or pretend to make, love; and generally to lay
broad and deep that moral and cultivated elegance, to take on that
exquisite polish [gla-mshi], which shall dazzle society; shall attract
the silly butterflies (women) who have influence or money; shall, in
fine, shine in the Grand Council, or at the head of armed bands, or
to the illumination of the Courts of Law! Noble ambition, based upon
manly principles! With the Barbarians to be a moral and wise man is
to be a milksop [Kou-bab]; to be a polished man of the World—
admirable!
The English Barbarians who are fathers, generally consider it rather
a joke to have their sons trick them and poke fun at the "Governor,"
only it must be marked with some pretence of deference. If the
"young fellows" do not positively disgrace the family—that is, marry
some poor creature whom they have first debauched; or actually
forge, or rob, or descend to improper friendships with inferior Castes
—the parents esteem themselves to be fortunate. If he have
acquired no knowledge of letters, nor of anything but vices, yet he is
a "fine, manly fellow, who will make his mark in the world." That is,
he is a tall, strong, active Barbarian—just fit for the armed bands!
The infelicities and disorders of family life, which only prefigure the
inevitable confusion and evils of the whole Society, are more
intolerable among the Middle Castes. In the Highest, secured
revenues enable the wife and the husband each to see as little of
each other as they please; and so long as the husband is not stirred
up by Mrs. Grundy (who is not severe with this Caste) he cares but
little what his wife may do. He goes about his sports and his
pleasures as he pleases; and his wife, not wishing to be looked after,
does not look after him. On this free-and-easy footing, with no want
of money (Mrs. Grundy's decorum being observed), they get on well
enough, and may even form quite a friendship for each other. But it
is not possible to establish this condition in a family of small income
—and here it is that the wretchedness of false principles has full
scope. The husband and father, honest and good, finds himself
mated to a woman, weak and vain, with children moulded by her.
He, misled by false notions and ignorance, took to his heart one
whom he fully trusted as simply true and modest; he took her for
herself and without money, and flattered himself that she would be a
helper and solace. She and her children have made him a miserable
slave, who finds no quiet unless he satisfy all their clamorous
demands—to shine in Society! If a good man, he tries to obey and
live, even under exactions beyond his utmost efforts; for he has
learned to see that his wife, though weak, is no worse than the
Society which she loves, and which he also cannot escape; he is
merely in a false position, and must largely thank himself for having
heedlessly entered upon it!
But this kind of man is not universal, and one may judge what
follows, where there is a man who will not yield, or yields only
because he no longer cares for anything but his personal ease and
indulgence—seeking for pleasure, though unlawful, abroad, as the
only recompense attainable for the loss of happiness at home!
Such a man feels that life is insupportable, where he makes so
wretched an object—to be merely the mute beast of burden for the
family, without receiving so much tenderness and consideration as is
accorded to the dogs lolling in the lazy laps of the females of the
house! He seeks, therefore, abroad for some means of enjoyment,
though illicit!
This sort of picture is to be seen everywhere in the Barbarian
Literature, and is constantly shown in all its minute and miserable
exhibition at the Courts of Divorce.
Adultery, which in our Flowery Land is punished by death, is not so
much as a crime among the English Barbarians. And, as it is the
chief cause for which the bond of marriage may be wholly severed,
one may judge whether the Court do not encourage the immorality.
For when parties wish to live apart, here is a way to secure it, lying
directly in the path of desire and opportunity. Then, too, the
seduction of a maiden, which with us may be punished even to
death, receives no sort of reprobation in the Court, and scarcely in
Society. If the ruined girl be of low caste, her relatives feel no
disgrace if the seducer be a High-Caste—rather an honour; receive
from him some paltry sum (not so much as he lavishes upon some
favourite dogs), and buy with the money a husband for her from her
own Caste!
With us a guilty intrigue is almost unknown; with the Barbarians it is
almost a pursuit.
None the less, there is too much vigour in the organism; too much
moral, intellectual, and physical strength, to suffer total decay. As is
always the case, where the mind is active, even Idolatry itself has
intermixed a pure morality, and the Barbarian nature, still unformed,
untrained; still rude and stirred by passion and by force; wrestles
with the divine instinct, and, unconsciously, often moulds to its light.
Away from the glitter and sham (sometimes in it, but not of it), there
are quiet families which live lives of honour. The father works
honestly and cheerfully; the wife, in her house, finds the beginning
and end of her aims, of her love, and her duty. The husband-father
is head; on him rests all responsibility, and to him belong obedience.
This is not exacted; it is not questioned. It is founded in love and
respect; love and loving obedience spontaneously arising from
uncorrupted natures. His whole being responds with unmeasured
joy. Whatever is pure, high, tender; all are for these—his wife, his
family; so true, so trusting, so helpful, so delightful. He feels no
hardship; there can be no sacrifice, for these; all that is done is in
harmony with himself. Everywhere he is in accord. The very ills and
misfortunes of life touch him not, for he is living in the divine order.
And from such a man, the inside-life being serene, outer ills fall
away. He is so clear and simple; so whole that nature smiles for him,
even in pain and sorrow; he lives in the presence and calm of the
Sovereign Lord.
These families are the Salt which saves. Among the Barbarians they
are generally obscure, and as wholly unconscious of the service
which they render as are the glittering inanities which ignore them.
This should be reversed, and the Inanities sink into obscurity.
I will now say a word or two as to the personal appearance and
demeanour of the Barbarians. There is no standard of best-looking,
and each tribe will judge from its best type. In general the eyes are
too prominent and open; the nose large and irregular; the teeth bad
or false; the height indifferent; the figure either too lean or too fat.
The hair all colours; red and light most common. The women are so
made up, judging from the articles openly exposed for sale, that one
cannot speak of them with any certainty. The hair, teeth,
complexion, bust, outline of form, are all false or artistically got up.
The eyes are too bold and open. The feet long, and hands large. Too
tall, and either too meagre or too stout. The youth are sometimes
pretty. The women are often brilliant under gaslight (a bright,
artificial light). I have spoken of dress, but I may mention that the
women, not content with every sort of made-up thing to add to their
attractions, pile upon their heads an enormity of false curls, bands of
hair, laces, and high sort of head-ornaments; it is truly amazing.
Some of these gewgaws are hung upon big pig-tails of false hair,
and some are stuck high a-top. Nothing really can be more absurd,
unless the false, mincing steps, and protruding back. Some women
are beautiful; but to my unaccustomed looks, even the brilliant eyes
could not blind me to so immodest an exhibition—or, to me, not
modest—so instinctively do we demand that especial quality in the
sex, as the crowning grace of true beauty.
One thing of a personal kind in the habits of all, high and low, I
remarked, which would be intolerable to us. A lady or a gentleman,
whilst conversing with you, or at the table of feasting, will suddenly
apply a handkerchief [mün-shi] to nose, and blow that organ in the
most astounding manner; and this may be continued for some
minutes, even accompanied by hauks and spits, and closed by many
nice attentions to the orifices not worth while to describe. Surely this
strange thing disconcerted me very greatly at first, nor do I
understand how any people above savages could do it. A fine lady
will interrupt herself in the very midst of speech, or of eating, with
spasmodic effort, to clear her head; emptying into her fine pocket-
handkerchief the obnoxious matter, and then returning the article to
her silken pocket.
However, we should not expect refinement in a Society where the
women may boldly mount a horse-back, and follow men and dogs
over ditch and wall, urging her steed with the best, to come in to the
death of the poor hunted creature. And this, a noble sport, fit for a
lady! Nor this only, but will crowd to public spectacles, and be
hustled and crowded promiscuously, forgetful of all delicate reserve.
These habits are only to be criticised because of the boasted
prëeminence claimed in all such matters. But what would be thought
of our Literati piling into the mouth huge morsels of flesh, or of
guzzling [kun-ki] (with a gulping noise in the throat), great swallows
of a hot, greasy liquid, besmearing the lips and beard. The
Barbarians know nothing of our delicate mode of eating, where all is
silence and decorum whilst in the act. Another most unaccountable
thing to a stranger is the robbery allowed by the servants of the
High-Caste. If you accept of the hospitality of a great man, you must
submit to be plundered by his servants; and, as a stranger cannot
know the limits imposed upon this rapacity, it goes far to destroy all
the pretence of graciousness in one's reception. When you have
eaten at my Lord's table, to think you are to be fleeced [pe-ekd] by
my Lord's flunki!
I was once invited by a High-Caste to come to his house in the
country and shoot game. I accepted, and soon went into the copses
to hunt for birds for the table. A servant accompanied me by
command of his master, to show me the grounds and to wait upon
me. He was very civil. The next day, upon my leaving, this man,
decked in the livery [bung-shi] of his Lord, closely eyed and stuck to
me, till, at length, I perceived he wanted something. Only partially
aware of the Barbarian custom, and blushing at the idea of feeing
[tin-ti] or giving anything in return for hospitality, I awkwardly
fumbled in my purse and handed to him a half-crown. He
contemptuously looked at the silver piece, then at me; and remarked
that the "gentlemen of my Lord did not receive gratuities of that
colour." Meaning that gold was only fit for such an exalted minion.

CHAPTER XII.
OF THE APPEARANCE OF THE COUNTRY—AND OTHER THINGS.
The country is so small, that, riding in the swift steam-chariots, it is
traversed in an incredibly short time.
In those parts not disfigured by the smoke vomited out from the
huge fire-chimneys of factories, mines, and the like, nor by the
nearness of great towns, the country presents a green and
cultivated look; nearly as well tilled as our provinces, Quang-tun and
Chiang-su. The villages, Temples with lofty towers, great Houses of
the High-Castes, here and there; trees, gardens, smooth fields of
fine verdure, over which cattle and sheep are feeding; rising hills
and sheltered valleys, rich with copses, orchards, and groves—all
seen in moving views—give an aspect of peace, comfort, and
wealth. You do not see the poverty, nor, too closely, observe the
dwellings of the poor.
In winter it is cold, and the whole appearance changes. Far to the
North, the sun gives but little light—and, like the climate of our
provinces by the great Northern Wall, the cold is severe, and the
gloom deeper. Ice is formed upon the streams and canals, and snow
frequently covers the ground.
In approaching great towns, you often catch glimpses of the
crowded, wretched streets, where misery only thrives. In some
places, in the winter cold, smoke and darkness, life becomes
intolerable to many. Out of doors you can hardly find your way, and
thieves and beggars emerge from covert to ply their trades. In the
night, at such times, it is only possible to move by the glare of many
torches; and people are often robbed, or bewildered and lost. At this
season of darkness many go mad. There is a strong vein of horror in
the Barbarian imagination, derived from their ferocious ancestors,
from their old idolatries, and deepened by the new. In the gloom,
the misery, the wretchedness—sometimes in sheer disgust of life—
many rush upon self-destruction—throwing themselves under the
wheels of the steam-chariots, and from the bridges into the canals
and rivers. Many persons are thrown down, maimed or killed in the
highways, by horses or by vehicles moving along. Yet, in the grim
humour of these barbarians, this is the very time when the High-
Castes begin their revelries, and the Low-Castes most indulge in
drink and riot.
In travelling through the country, you will occasionally notice, seated
upon an eminence, some strong Castle, or Place, of hewn stone,
belonging to a High-Caste. It will be approached through long
avenues of lofty trees, and stand pre-eminent among fine groves,
surrounded by broad lands. These wide Parks contain many
thousands of acres [met-si], left untilled and unproductive; merely
with their green slopes and spaces, interspersed with trees, to give
grandeur to the Castle and its Lord. Still, if you look closely, you will
discover near by, the squalid huts where huddle the Serfs, who are
starving in the midst of this rich profusion—Serfs, who never have
an inch [toe] of land of their own, and to whose wornout carcases is
begrudged a pauper grave!
The inequality between Castes is quite as conspicuous in country as
in town. One is born to an abundance, the other to hunger; one to a
life of self-indulgence, the other to one of enforced and hard-worked
self-sacrifice. The one, at last, is covered by a tomb, emblazoned
with Honour; the other is cast into an obscure corner of despised
dead, to rot in forgetfulness—though, often, judged upon a true
measure of merit, the resting-places should be exchanged—and the
idle and vicious Lord [chiang-se] descend into ignominious neglect!
You will see deer, pheasants, partridges, hares, and the like, almost
tame, in the meadows and copses; but the tillers of the soil must not
touch them, though starving—they are carefully preserved for the
Lord [Tchou]. Not that he needs them, or cares for them for food—
sometimes he likes to shoot them for idle diversion!
You will notice sturdy tramps (beggars) resting, or lazily slouching
along by the ways, with heavy staves in their hands; and, if you
suddenly come upon these in a secluded place, very likely you will
be accosted—"Master, I be'se hungry—will ye give me tuppence?"
You do not like the bearing of the man—and would not notice him.
But you observe his face and the clutch of his thick stick—and you
hurry to hand him a sixpence, and get away! These scamps prowl
about, idle, ready for mischief, scornful of honest work—the terror of
women and children who meet them, unexpectedly, without
protection.
Sometimes the Iron-roads for Steam-chariots are carried over the
housetops, in entering towns; sometimes, through long tunnels
under the houses, or under hills—and the works in connection with
these roads are surprising. The Barbarians of the Low-Castes are
forced to incessant labours, to prevent starvation. These must be
greatly directed to mines of iron, coal, copper, and tin; and to
various things made from these, and from wool and cotton. For the
fruits of the land cannot feed the population. The amount of food
which must be brought from beyond seas is very great—and to pay
for this, the products of industry must be given. Now, other
Barbarian tribes make these things also, and; having them, do not
require the English; in fact, in more distant parts, undersell them.
From this cause, many are unemployed and turned adrift—they have
no land to till; they beg, steal, and starve. Should this inability of the
English Barbarians increase, there would be no sufficient
employment for the Low-Castes—there would not be the means of
paying for the food required—and depopulation must ensue! The
wealth of the High-Caste must shrink—the English tribe must decline
in strength!
Many of the High-Caste, already anticipating danger to themselves—
fearing not merely loss of revenue, but the savage ferocity of
starving multitudes—promote schemes by which large numbers of
the poor are shipped off far beyond the great Seas (so that they
never shall return)—to starve, or live, as may chance. "England is
well rid of them!" they say.
In the neighbouring island, Ireland, an actual starvation of the
people in vast numbers happened a short time since. As in England,
the poor serfs, tilling the soil and owning none; at the best, toiling
for the High-Castes for such pittance as would buy the cheapest
food—potatoes; when these failed, could buy nothing—all else too
dear. These failed, the serfs died by thousands and tens of
thousands. Not because Ireland was destitute of food; such was the
abundance that ample stores were actually sold for other and distant
tribes! but because, in the midst of plenty, the starving were
powerless to touch it; it was out of their reach—out of the reach of
paupers! The potatoes were not—and they must die. The annals of
no people record such a depopulation of a fertile land, in the midst
of peace and plenty—there is no parallel! A people dying, not from
idleness, nor unwillingness to work; not from want of food at hand;
not from the ravages of war, nor pestilence; but from sheer poverty!
Yet, the English Barbarians boast that no people are so rich, so
generous! In our own annals are recorded great sufferings from
floods, failures of crops, and natural causes; where our vast
populations have been for a time deficient in food; but we have
nothing to compare with this Barbarian horror!
The Thames is the only considerable river. This flows through the
greatest of all the cities of the West—London. It is an insignificant
stream—much less than even the Quang-tun, in our chief Southern
province.
As it flows through the great city it is, in some places, confined by
high hewn-stone terraces [kar-tra]. These are truly great works, and
useful, worthy of a strong people. On the river bank is the vast Hall
of the Grand Council; with its lofty towers, turrets, clocks, and many
bells. The architecture is not like anything known to us—it is the
Gothic, which I have mentioned elsewhere. Why this style, so
characteristic and fit in the Temples, is used in this grand Hall, I
know not; but probably because this barbarous form was that of the
old Hall, destroyed by fire some time since. And the barbaric stolidity
sticks to its habit, however inconvenient and unfit. Not far away,
may be seen the Dome and Towers of a fine Roman-Grecian Temple,
clear and defined, giving expression to an orderly and trained mind,
severe in dignity and beauty. But the Gothic, expressing, or trying to
express, something very different; and, rising in the Temples of a
gloomy, dark Superstition, to a horrible and unformed shape! With
that the disorderly brain burdened itself and the river bank—a pile at
once wonderful and abortive!
London is very large, perhaps equal to some of our greatest cities.
For the most part very dirty and grim, and badly built. The river
shows its great trade—not inland, but from abroad. You can discern,
rising above the buildings, the many tall masts of the ships like
forests dried up. And you will observe the numerous vessels with
high chimneys; these are the vessels moved by steam—and the
incredible number of small craft. At one point you will remark the tall
white towers and the high prison walls of stone, erected by the
Barbarian chief from the Main Land who subdued the English tribes
in our dynasty Song, and made this huge Castle a stronghold and
prison.
Lower down rises, close by the shore, one of the best in style of all
the Barbarian monuments. It is a fine Palace in carved stone, built,
after the Roman forms, to perpetuate the remembrance of Victories
gained over distant tribes. Within are great Paintings of these
Victories. Terrible scenes of devastation and cruelty; bloody fights
and dreadful conflagrations, by sea and land; rapine, massacre,
unbridled fury! These are the most admired of all things by the
Barbarians—by the Low-Castes, who are almost entirely the victims,
as much as by the High. The sight of these kindles their passion for
bloody force. They Hoorah! with an indescribable yell [zung]
whenever they wish to show their frantic delight at any exhibition of
brutal ferocity. This yell is greatly gloried in, and vaunted to be far
more terrible than that of any other tribe—that by it alone, when
raised upon the air by fierce bands, English Barbarians have routed
armed hosts!
When one is in the narrow seas of the English, very many vessels
may be seen, and near the coasts fleets of fishing craft. The
fishermen live in great poverty, in miserable villages by the seaside.
They use lines and snares, sometimes like ours, but are not so
ingenious in catching the sea-creatures as are our fishermen. They
have never trained birds to the work. Their huts are noisome, and
their habits and dress unclean. They wear a curious cover upon the
head, like a basin, with a long wide flap behind. This is all
besmeared with a thick, black oil—and their clothing is stiff and
nasty with the same unctuous stuff. The oil is to exclude the sea-
spray and wet. Their speech is nearly unintelligible to the Literati,
though comprehended by their own Caste; they are of the lowest—
serfs. Multitudes of these rude and unlettered Barbarians perish
amid the waves in the storms of winter—being forced to imperil their
lives that they may live at all. They are quite a feature in some
parts, with their awkward uncouthness. They are addicted to the
grossest superstitions of the Superstition. They have many legends
about the dark devil-god, and swear by him mostly. They seem to
think to cheat him—though they cautiously observe those things
which may entrap them, and nothing would tempt them to put to
sea on the devil's day—Friday. To do so, would be to go to the devil's
Locker (as they call it) at once! This class is similar to the sailor
[mat-le-si] known in our ports, and the character may therefore be
fairly judged. The fisherman, in fact, often changes into the ships
and goes upon distant voyages.
There are no mountains, only pretty high hills, in the English
provinces. The loftiest are in the far Northern parts, where are also
some small lakes. In the winter these loftier ridges of land are
sometimes white with snow. The inhabitants are savages, having
their legs naked and bodies wrapped about in loose robes and skins,
secured by a belt, into which a knife is stuck, and to which a long
leather pouch is hung. In this pouch they place some dry corn
[matze], which, with strong wine in a bottle suspended from the
neck, enables them to live for days. Thus equipped, they descend to
the valleys, and drive off to their haunts in the rocky hills the cattle
of the more civilised people of the plains.
The English Barbarians have never conquered these fierce tribes of
the Northern hills, but have contrived gradually to destroy and to
remove them. So that, at present, what few remain are quite tamed.
A great many, in times past, were cunningly betrayed to the English
and put to the sword; but, in latter days, the head-chiefs have been
bought by the English, and used to entice their ignorant but devoted
serfs to enter into the armed bands to be sent beyond seas. By
these methods, those distant Northern parts have been, in good
degree, depopulated and made quiet.
The Low-Castes furnish the fierce savages so well known in our
Celestial Waters as those who live in the great fire-ships.
Now, when the English tribe, being in need of many men for these
ships (just about to go away to plunder and to fight), determines to
have them, this follows:—Strong, brutal men, are paid to watch for
the poor of the Low-Caste, and seize them. These cruel wretches are
armed with clubs and swords and small firearms. They are sent into
the places where the poor and friendless abound, to seize any man
whom they think they can carry off without much fuss [pung]. The
poor cower and hide away; but these savage bands hunt them out,
and bear off from wife and children, it may be, or from any chance
of succour, some unfriended man to their dreadful dens. Here they
are beaten, or put in irons, or otherwise maltreated; or they may
have been brutally knocked down when captured. When gangs [twi-
sz] are collected, the victims are forced on board the fire-ships to
work in the dark, filthy holes, till, completely cowed, they are made
to fire the great cannons, and to learn the art of sailing and fighting!
Many of these slaves of selfish, cruel force, never see their own land
again, but are killed in fight, or by accident, or by disease.
Multitudes sometimes perish by a single disaster. These are,
however, fortunate. They have escaped the brutal whipping, the
loathsome diseases, the vile contagions, the inexpressible horrors of
a continued captivity!
By these press-gangs (so-called) the fire-ships are often supplied
with victims snatched from the unprotected Low-Castes; and the
Upper enjoy the idle and luxurious security which they rob from the
blood and limbs of the friendless and obscure.
This unjust custom, frightful in every aspect, receives the
approbation and applause of the Barbarians very generally, who say,
"Let the fellows thank their stars that they can receive the Queen's
money and fight for her! Then look at the chance for prize!" By
prize, they mean some pitiful fraction of the plunder taken. The stars
are referred to, because the Barbarians fancy that everybody is born
under the influence of some star!
I once noticed a painting, wherein a young man and maiden were
represented as just leaving a Temple, where they had been married.
Both were nicely dressed, young and handsome, with roses and
nosegays [bong-no]. They were walking arm-in-arm, happily
engrossed in each other, when, from an alley, out springs a black-
whiskered bully [kob-bo] with drawn cutlass, followed by a band of
half-drunken, armed wretches, wearing the sea-garb of the Queen;
he grasps the young man roughly by the collar—the picture attempts
to show the indignant surprise of the man, the clinging tenderness,
fear, and horror of the maid! But more striking to an observing
stranger than even these, is the merely passing curiosity of the
people moving about! The scene to them is not so novel. It is merely
a press-gang doing its lawful work—if, by chance, a wrong sort of
man be seized, it is none of the affair of these indifferent passers.
Probably, the picture means to excite some compassionate interest
by showing how very hard the press-gang system may work!
It would be vain to call the least attention to the matter, if the victim
were merely a common labourer; even the accessories of wife and
children would not raise the scene into one of compassion. Nor does
the representation, for one moment, cause any reflection upon a
system wherein bullies [kob-toe] are employed to waylay and carry
off unbefriended and unoffending men, at so much per head! For,
besides the regular pay, a reward is given for each victim captured!

CHAPTER XIII.
LONDON.
London is the capital city of the British Empire. This is the style
assumed by the English when they speak of their whole power. It is
a curiously constructed empire—in some respects like that of the old
Romans, who, however, obtained their domination more directly by
valour and wisdom—whereas the English rather by cunning,
accident, and fraud. I say accident, because the immense regions
possessed by virtue of discovery come under the term; and the
vastest of all their distant provinces, that of India, was obtained
chiefly by fraud, assisted by force. I say curiously constructed,
because these Christians are content to wring from Heathen subjects
their last bit of revenue utterly indifferent to the idolatries and to the
miseries of the people. If the Taxes come in and the wretched
Hindoos starve, the main thing is to make the money and support
'our magnificent Empire' (as the English have it). So the wildest
excesses may go on, and the native chiefs, who are mere creatures
of their distant masters, may oppress the poor inhabitants; still, now
and ever, the Master demands money; this secures the yoke upon
the neck of the subjugated, and enables the English to make the
vast Hindoo world a field where golden harvests are to be reaped.
Boasting of liberty at home, there, a tyranny most odious is practised
without pity. Then, the distant settlements where the poor English
Barbarians go, to cultivate the lands and to trade and plunder, are
held in subjection chiefly to give places, with large revenues
attached, to members of the Aristocracy, who must be provided for
in some way, as they can do nothing for themselves. So this
arrangement is very satisfactory, because the stupid Englishman
abroad is just as devoted to the Upper-Caste and to the Superstition
as at home, and feels honoured to have a "scion of nobility" foisted
upon him; and is amply repaid all the cost by the privilege of
"cooling his heels" in an ante-room of the great man, when he holds
his little Court.
The result is, that back upon London flows all the wealth which the
English Barbarians can contrive to get. Having these distant regions,
and a greater trade across sea, London has become the greatest
mart of all the Western tribes. It is, perhaps, as large and populous
as our Pekin. It is the centre of Authority and of business; not only
so, but is the Metropolis of all the Christ-worshipping Tribes—or, as
the Barbarians phrase it, of Christendom.
The population is 3,500,000, or thereabouts. The bulk of this
multitude is poor, and a large fraction paupers. Yet the English boast
that "it is the richest city in the world!"
Most of the streets, courts, and buildings are very mean. In the
winter, nothing can equal the repulsiveness of the place. To the
squalor of beggary, the meanness of abject poverty, add the
darkness and smoke; and the conditions seem unfit for human life.
The rich shut themselves within their houses, drop the heavy
draperies over windows, stir up the fires, light the flaring flames of
the curious gas-lights, eat, drink, and sleep—shutting out from sight
and sound that hideous outside. This is the time when the wretched
in mind and body find existence too great a burden, and cast it off
with a shriek and a rush—plunging into the river or canal, or dashing
beneath the wheels of the swift steam-chariots.
At all street-corners one notices the gin and beer shops. These are
the homes of the poor, who find in them the warmth and comfort
which are wanting in their domestic haunts. These shops are closed
at mid-night, when the half or wholly drunken loiterers must straggle
off into those holes and corners which are their homes. Probably
there is no feature in barbaric life so curious and so characteristic as
this—this Gin-house of the poor. The Government licenses these
places, and derives a great income. The Upper-Castes fatten upon
this very thing. What can be said of it—what done with it?
Another remarkable object in the London streets is the Street Arab.
This is the name given to it by the Barbarians. But the Arab of Asia
(if my reading be correct) is nothing like this creature. The London
Arab is of the degraded and thieving class—the very sediment—but
not yet fully weighted! In years a youth, but in feeling a ravening,
sharp, adroit animal, quickened by the exercise of every instinct, and
cool and expert from constant habit. He dodges in and out from
under the heads of horses and the wheels of vehicles; mounts a
lamp-post, or anything by which he may get a sight; seizes the
bundle which you may have in hand; touches his uncombed front
locks of hair, "Please, Sir, le' me carry it, Sir;" and trots before you,
happy if he get twopence. Nobody knows where he sleeps, or eats,
nor how he lives, at all. I have suddenly come upon two or more of
them, when resting upon an iron grating. Their naked feet and