Eric Ed465868

DOCUMENT RESUME
ED 465 868 CE 083 403
AUTHOR Bingman, Mary Beth

TITLE Documenting Outcomes for Learners and Their Communities: A
Report on a NCSALL Action Research Project. NCSALL Reports.
INSTITUTION National Center for the Study of Adult Learning and
Literacy, Boston, MA.
SPONS AGENCY National Inst. on Postsecondary Education, Libraries, and
Lifelong Learning (ED/OERI), Washington, DC.
REPORT NO NCSALL-R-20
PUB DATE 2002-03-00
NOTE 81p.; With Olga Ebert and Brenda Bell.
CONTRACT R309B60002
AVAILABLE FROM Web site: http://www.gse.harvard.edu/'"ncsall/order-all.html
($10). For full text:
http://www.gse.harvard.edu/'"ncsall/research/report2O.pdf.
PUB TYPE Reports - Research (143)
EDRS PRICE MFOl/PCO4 Plus Postage.
DESCRIPTORS Action Research; *Adult Basic Education; Adult Students;
*Data Collection; *Documentation; Educational Assessment;
Educational Research; Identification; *Outcomes of
Education; Professional Development; Program Development;
Program Effectiveness; Program Evaluation; *Self Evaluation
(Groups); Self Evaluation (Individuals); Self Management;
Student Development; Student Educational Objectives;
*Student Participation; Student Responsibility
IDENTIFIERS Kentucky; *Outcome Oriented Evaluation; Tennessee; Virginia
ABSTRACT
Action research focused on developing approaches local
programs can use to document outcomes of student participation in adult basic
education (ABE) programs. Teams of teachers and administrators from three ABE
programs examined current documentation practices, were introduced to
approaches to documentaion, and developed documentation processes using a
cycle of planning, implementation, and evaluation. Documentation efforts
focused on aspects of students' lives which the program or students
identified as areas that they hoped to change. The Virginia team established
a process to help students identify changes they hoped to make and document
goal and outcome achievement. The Tennessee team documented outputs and
outcomes as part of their focus on the Equipped for the Future framework
standard called Take Responsibility for Learning. In the Kentucky program,
students used calendars to document activities that supported their
children's education. Teams, programs, and students found these documentation
efforts useful tools for instructional planning and for learner and project
assessment. Researchers reported the following: (1) that local ABE programs
can develop documentation processes useful for planning and assessing their
work; such factors as limited paid time for teachers typical of adult
education make participation in action research difficult; and action
research is effective for professional and program development in ABE.
(Appendixes include 30 references; participant list and sample documents.)
(YLB)
Reproductions supplied by EDRS are the best that can be made

from the original document.
Documenting Outcomes for
Learners and Their Communities:
A Report on a NCSALL Action Research Project
Mary Beth Bingman

with Olga Ebert and Brenda Bell
Center for Literacy Studies
The University of Tennessee
NCSALL Reports #20

March 2002
National Center for the Study of Adult Learning and Literacy

Harvard University Graduate School of Education
101 Nichols House, Appian Way
Cambridge, MA 02138
3
NCSALL Reports #20 March 2002
TABLE OF CONTENTS
ACKNOWLEDGEMENTS ........................................................................................ i
EXECUTIVE SUMMARY........................................................................................ 11
..
CHAPTER 1 :INTRODUCTION............................................................................. 1
CHAPTER 2: METHODOLOGY............................................................................ 5
ACTIONRESEARCH ................................................................................................... 5
THEDOCUMENTING OUTCOMES PROJECT ................................................................. 6
THEPARTICIPATING TEAMS...................................................................................... 9
SUMMARY ............................................................................................................... 11
CHAPTER 3: IDENTIFYING THE QUESTION, PLANNING THE
WORK ....................................................................................................................... 13
STAGE1: THETENNESSEE TEAMEXPLORES DOCUMENTING OUTCOMES ...............13
BEGINNING TO DOCUMENT OUTCOMES ................................................................... 16
STAGE 2: VIRGINIA AND KENTUCKY JOININ A MORESTRUCTURED PROCESS ........17
DEFINING THE ISSUE:WHAT ARE THE OUTCOMES W E CARE ABOUT? ...................18
THEDOCUMENTATION MATRIX .............................................................................. 19
USING INSTRUCTIONAL ACTIVITIES TO DOCUMENT OUTCOMES ............................. 22
EFF ......................................................................................................................... 24
REVIEWING INPUTS-TO-IMPACTS ............................................................................ 25
Two REGIONALMEETINGS ..................................................................................... 26
SUMMARY ............................................................................................................... 27
CHAPTER 4: THE TEAMS’ DOCUMENTATION PROCESSES .......... 29
SUPPORTINGCHILDREN’S EDUCATION .................................................................... 29
TAKINGRESPONSIBILITY FOR LEARNING ................................................................ 31
DOSETMET ............................................................................................................. 34
SUMMARY OF TEAMACTIVITIES IN STAGE 2 ........................................................... 38
CHAPTER 5: FINDINGS: WHAT DID WE LEARN? ....................................... 39
DOCUMENTING OUTCOMES.OF PARTICIPATION IN ADULT BASIC EDUCATION
PROGRAMS .............................................................................................................. 39
THEREWARDS AND CHALLENGES OF ACTIONRESEARCH IN ADULT BASIC
EDUCATION PROGRAMS .......................................................................................... 41
WHAT ELSEHAVEW E LEARNED?ACTION RESEARCH AS A LEARNING
..................................................................................................................
PROCESS 44
SUMMARY: WHAT DIDW E LEARN? ........................................................................ 47
4
CHAPTER 6: IMPLICATIONS AND FINAL THOUGHTS .............................. 49
REFERENCES ......................................................................................................... 55
APPENDICES .......................................................................................................... 59
5
TABLES
OUTCOMES PROJECT ...............................

TABLE1: OVERVIEW OF THE DOCUMENTING 8
2: SUMMARY OF STAGE 2 TEAMACTIVITIES

TABLE DEVELOPING OUTCOME
........................................................................................................
DOCUMENTATION 38
FIGURES
1. EFF STANDARD
FIGURE “TAKERESPONSIBILITY .........................
FOR LEARNING” 32
ARTIFACTS
DESCRIPTION
ARTIFACT 1: ONE-PAGE OF ACTIONRESEARCH PROJECT USED IN
RECRUITMENT............................................................................................
PROGRAM 10
ARTIFACT 2: LOGIC MODELFRAMEWORK BY PROGRAMS ......................

DEVELOPED 15
ARTIFACT 3: NOTESFROM FIRSTVIRGINIA TEAMMEETING.2/1/99 .......................... 19
ARTIFACT 4: THEKENTUCKY DOCUMENTATION ..........................................

MATRIX 21
ARTIFACT LOG. 2/26/99 .................................................

5 : NOTESFROM TEACHER’S 23
ARTIFACT 6: REFLECTIONS FROM A FACILITATOR AFTER THE APRIL 1999

.....................................................................................................................
MEETING 26
ARTIFACT 7: PLANNING DONEON WHITEBOA RD ........................................................ 29
ARTIFACT 8: FROMTHE OVERHEAD USED BY THE KNOTT COUNTY TEAM AT THE

NOVEMBER 1999 REGIONAL MEETING....................................................................... 30
ARTIFACT 9: DO/SET/MET.......................................................................................... 37
ACKNOWLEDGEMENTS
The collaboration that resulted in this report involved many people over several
years. The heart and soul of this action research project were the teachers and
administrators of the three participating adult education programs: Knox County
Adult Literacy Program, Knoxville, Tennessee; Mount Rogers Regional Adult
Learning Program, Abingdon, Virginia; and Knott County Adult Learning
Center, Hindman, Kentucky. We would like to thank these individuals (listed
in Appendix 1) for their commitment to exploring ways to improve the delivery
of quality services to adults in their communities.
We would also like to thank Ernestine Scott, of the Virginia Department of

Education; Sandra Kestner, of the Kentucky Workforce Development Cabinet; and
Phil White, of the Tennessee Department of Labor and Workforce Development, for
assistance in identifying and supporting the participating programs from their
respective states.
Gary Kuhne, of Pennsylvania State University; Peter Waite, of Laubach

Literacy Action; and John Comings, of NCSALL, provided a thoughtful, useful,
and timely review of this report. Their suggestions strengthened this report, and
any oversights or confusions remain the responsibility of the authors.
Finally, our thanks to Dona1 Crosse and Rosemarie Mincey, colleagues at

the Center for Literacy Studies (CLS) who, along with the authors, provided the
participating programs field support at different stages of the project, and to CLS
staff members Aaron Kohring and Charlotte Duncan, who reviewed this paper.
7
EXECUTIVE SUMMARY
The action research described in this report focused on developing approaches that
local programs can use to document the outcomes of student participation in adult
basic education programs. This project has implications for professional develop-
ment as well as for outcomes documentation in adult basic education.
Over the course of two years, three teams of teachers and administrators from
three adult basic education programs in Tennessee, Kentucky, and Virginia, with a
team of NCSALL researchers from the Center for Literacy Studies at the University
of Tennessee in Knoxville serving as facilitators, addressed this issue. The teams
examined their current documentation practices, were introduced to a variety of
possible approaches to documentation, and developed their own documentation
processes using a cycle of planning, implementation, and evaluation. From the
action research, they developed new approaches to documentation and gained a new
understanding of their work as adult educators.
Documentation efforts focused on particular aspects of students’ lives that the

program or the students identified as areas in which they hoped to make change. The
Virginia team established a process that helped students identify the changes they
hoped to make (i.e., their goals for education) and to document achievement of those
goals and outcomes. The Tennessee team documented outputs and outcomes as part
of their focus on the Equipped for the Future framework standard Take Respon-
sibility for Learning. In the Kentucky program, students used calendars to document
their activities that supported their children’s education, such as reading to their
children and meeting with teachers. The teams, programs, and students found these
documentation efforts usehl tools for instructional planning and for learner and
project assessment. The programs have continued to use these documentation tools
and to develop new ones.
Although the main purpose of this research was to contribute to under-

standing of a systemic issue-how to document the outcomes of participation in
adult basic education-it also contributed to the participants’ professional, program,
and personal development. For the participating teams, this action research project
has led to increased understanding of how programs might identify and document
student outcomes in ways that meet local program needs and how that documen-
tation can support program improvement. Team members learned from working in
groups and having an opportunity to share experiences with other teachers in their
own teams and from other teams. The two aspects of this project that participants
seemed to particularly value were the focus on students and the opportunity to reflect
on the goals of their work.
ii
8
When the teams were asked to identify challenges, some named conceptual
issues, such as “the idea of documenting changes in learner lives versus test scores,”
but most named practical challenges common to the field of adult basic education.
These included student turnover; limited time for teachers to design, collect, and
implement documentation; and difficulty in finding ways to share results with
teachers in other programs.
The facilitators extensively reviewed project data. The findings from this
review with direct implications for practice and policy in adult basic education are:
Local adult basic education programs can develop documentation processes

useful in planning and assessing their work. However, more work is needed on
the local, state, and national level before these locally developed documentation
processes can be used for the performance accountability systems the Workforce
Investment Act requires.
Action research is an effective approach for professional and program

development in adult basic education, but if it is to be widely used, factors such
as the limited paid time typical of adult education in most states must be
addressed.
On the basis of these findings, the researchers make the following recom-
mendations to the field of adult basic education about outcomes documentation and
using action research as a tool for professional development, program improvement,
and performance accountability:
Those responsible for professional development in adult education should use

action research more extensively. To improve practice, teachers need to be paid
for the time they spend in action research, action research needs to be accepted
as professional development by the state system, and facilitation support needs
to be available.
Local and state adult basic education (ABE) administrators should encourage the
use of action research approaches to improve program quality. Systematic
processes of reflection to identify areas that need improvement, combined with
ongoing action and evaluation, help keep a program focused on continuous
improvement.
States should build consensus about the goals underlying their performance
accountability systems, using such participatory processes as action research.
Action research as professional development should include this local definition
iii
9
of goals as part of consensus building because this focus and measuring goal
achievement seem to build a program’s capacity to implement performance
accountability systems.
On the state and federal levels, resources should be committed to designing

outcome measurement and reporting systems flexible enough to include a
variety of goals and rigorous enough to measure performance.
iv
PO
CHAPTER 1 :INTRODUCTION
Teacher: They ’re working on their businesses in the classroom. So they were using
skills to try to buy this computer-pretend, you know. But, still, i f I had a folder and
I had a check-off list that said, “They negotiated, they listened actively, they spoke so
others could understand, they were respectful to the computer teacher, ’’ then I could
actually check that off: And I could have that in thefolder. And then I could go back
. . . and I could say, Okay, this helped me see that they did this well. I can see that
being a way into it, a way to do it.
Researcher: It just struck me that one of the things we ’re trying to do here is capture
little moments, when the process is very dynamic. We’re trying tofigure out how you
give evidence for moments that tell you that you’ve moved. It’s never going to be the
whole picture.
Administrator: Seeing what we do in documentation really ties together our

students ’progress.
Over the course of two years, teams of teachers and administrators from three adult
basic education programs addressed how local programs might document the
outcomes in students’ lives of their participation in adult basic education programs.
The teams examined their current documentation practices; were introduced to a
variety of possible approaches to documentation; and developed their own processes
using a cycle of planning, implementation, and evaluation. From the action research,
they developed new approaches to documentation and a new understanding of their
work. This report describes the work of the project and what can be learned from
this action research.
Two major national initiatives, the National Reporting System ( N R S ) and

Equipped for the Future (EFF), influenced this action research project. In 1998,
Congress passed the Workforce Investment Act (WIA) mandating a performance
accountability system for federally hnded adult basic education programs in the
United States (P.L. 105-220). States are required to set levels of performance for
three core indicators:
0 Demonstrated improvements in literacy skill levels in reading, writing, and

speaking the English language; numeracy; problem solving; English language
acquisition; and other literacy skills
Placement in, retention in, or completion of postsecondary education,

unsubsidized employment, or career advancement
Receipt of a secondary school diploma or its recognized equivalent
The N R S establishes the measures that states may use in their reports on
the WIA core indicators. It also provides optional secondary measures that a state
may report (for instance, registering to vote, increased involvement in children’s
education, or leaving public assistance), but these are not included in state per-
formance assessments. The N R S was being implemented as the action research
teams carried out their work.
The National Institute for Literacy’s Equipped for the Future (EFF)
standards-based system reform initiative has conducted a multi-year, field-based
research process to determine what adults need to know and be able to do in their
roles as workers, family members, and citizens (Stein, 2000). When completed, EFF
will provide a common framework for defining, tracking, and reporting results to
policymakers as well as to students and their local programs. The EFF framework
consists of:
Four Purposes for Learning, defined originally by adult learners and validated by
a wide range of adults. These purposes are access to information so adults can
orient themselves in the world; voice, or the ability to express ideas and opinions
with confidence; independent action, or the ability to solve problems and make
decisions independently; and a bridge to thefuture, or learning how to learn, to
keep up with a changing world.
Three role maps that define activities critical to carrying out the roles of worker,
citizen, and family member, such as Become and Stay Informed to be an effective
citizen, Promote Family Members ’ Growth and Development to be an effective
parent or family member, and Work Within the Big Picture to be an effective
worker.
Thirteen activities common across these three roles, such as Manage Resources,
Guide and Support Others, Create and Pursue Vision and Goals, and Keep Pace
with Change.
2
Sixteen Content Standards that provide specific and measurable statements of

what adults need to know and be able to do, clustered in four categories:
communication, interpersonal, decision-making, and lifelong learning skills.
Each Standard describes components of skills typically taught in adult basic
education, such as Read with Understanding and Resolve Conflict and Negotiate.
EFF has been developing an assessment fiamework to define performance

levels and identify progress measures for the skills identified as their standards for
what adults need to know and do to carry out the key activities in their lives. This
work began in 2001 and will be completed in 2003.
In the long term, EFF is addressing what adults should know and be able to
do and developing new approaches to assessing learner progress. The N R S has
established how to measure a narrow range of skill gains and limited outcomes, at
least in the short term. However, questions remain about documenting outcomes of
adult literacy education in students’ lives and about how local programs might
document outcomes in ways that meet student and practitioner needs. EFF expects
to eventually provide a way for programs to show “results that matter” for all
stakeholders, and the N R S establishes ways to document particular outcomes that
concern policymakers. In our study, we focused on how local programs might
document outcomes in ways immediately useful to students, teachers, and programs.
We determined that an action research project might be an effective way to explore
this issue.
This project was a part of the work that the Center for Literacy Studies (CLS)
at the University of Tennessee in Knoxville conducted as a partner in the National
Center for the Study of Adult Learning and Literacy (NCSALL). The CLS work for
NCSALL has addressed how to assess the impact of literacy learning in ways that
serve policymakers concerned about the results of their investment, practitioners
concerned about the efficacy of their work, and adult students concerned about how
their efforts to learn will benefit them in the rest of their lives (Merrifield, 1998).
This is particularly relevant in a time of increased emphasis on performance
accountability.
The CLS and NCSALL efforts to address assessment of the impact

of literacy learning have included:
A policy paper examining performance accountability in adult education

(Memfield, 1998)
3
A paper reviewing previous outcomes studies in adult literacy education

(Beder, 1999)
Two studies examining how learners assess the changes in their lives resulting
from participation in adult literacy programs (Bingman & Ebert, 2000; Bingman,
Ebert, & Smith, 2000)
A policy paper based on the findings (Bingman, 2000)
The remainder of this report includes a methodology chapter briefly dis-

cussing action research and giving an overview of the project processes, a chapter
describing the first and second project stages, a chapter describing the results of the
teams’ work in Stage 2, a chapter discussing the CLS researchers’ findings, and a
chapter presenting conclusions and recommendations. This report is based on our
field notes, project team reports, and project artifacts the facilitators and program
teams produced.
14
CHAPTER 2: METHODOLOGY
Action Research
Action research is an approach to research and development grounded in practice. In

a collection of articles edited for an adult education audience, Kuhne and Quigley
(1997) described action research as “a form of inductive, practical research that
focuses on gaining a better understanding of a practice problem or achieving a real
change or improvement in the practice context” (p. 23). They described a repeated
cycle of six steps in three phases:
Planning Phase
1. Pose problem
2. Define project and determine intervention
3. Determine measures of data collection
Action Phase
4. Implement action and observe results
Reflection Phase
5. Evaluate the results
6 . Reflect on the project, possibly posing another problem
In the same volume, Creating Practical Knowledge Through Action

Research: Posing Problems, Solving Problems, and Improving Daily Practice
(Quigley & Kuhne, 1997), Quigley discussed the development of action research,
from the work of Kurt Lewin to the reflective practice of Argyris and Schon to the
more radical participatory action research advocated by Kemmis and McTaggart.
Quigley situated action research in the practice of adult education in the
“institutional and staff development context” (p. 16) and as an instrument of
empowerment and social change.
In describing participatory action research, McTaggart (1997) emphasized

that action research involves “the intensive study of a situation and the production of
knowledge in some form or another, including important ideas like informed
practice” (p. 27). He discussed Lewin’s action research cycle as “a spiral of steps,
each of which is composed of planning, acting, observing, and evaluating the results
of the action” (p. 27) and described action research as a joint project of academics
and “workers” (p. 3 1). The guiding principles of participatory action research
McTaggart outlined include:
5
Concern on the part of academics and practitioners (workers) with under-

standing and improving both their own individual and their organizational
practice
Studying and changing the discourse and practice, including distribution of

power of the organizations involved
Getting started quickly and starting small, with a spiral of planning, acting,
observing, and reflecting
Using the project to both change practice and produce knowledge
Beginning with the subjective experiences of the participants
McTaggart described projects that began as small cycles of planning, action,

observation, and reflection but built over time to include multiple cycles and
extensive documentation of the changes in activities, discourse, social relationships,
and forms of organization and development of expertise (p. 39). These iterative
processes culminate in research in which evidence and critical reflection combine in
new critiques, knowledge, and theory, as well as in changed practice.
The Documenting Outcomes Project
Although this project included some of the elements of participatory action research
that McTaggart described, our work has been more limited-by 60th time and
resources. With the participating teams, we developed processes that enabled both
academics and practitioners to build understanding of their work and the contexts in
which they work. The teams started small and used expanding, iterative processes of
planning, action, and reflection. The project began with participants’ experiences,
has changed practice, and produced knowledge about that practice. The teams did
not-except incidentally-study practice discourse or power distribution. Although
the project did not include the level of documentation or length of process to build
new critiques, knowledge, and theory, it may contribute to a critique of limited,
highly bounded ways of measuring outcomes.
The methodology in action research is not necessarily predetermined. Its

description, therefore, is messier and more complex than that found in more tradi-
tional research. Our action research design included four not always linear steps:
6
I6
Understanding the current situation in local programs, what was and what was
not documented, and why
Examining various frameworks that integrate documentation into program work
Developing and implementing new approaches
Reflecting on the results
The project’s work occurred in two stages, each including cycles of planning,
action, and reflection. The first stage, in 1998, involved one program and focused on
clarifying the project’s issues and processes. The second stage, in 1999-2000,
involved three programs and was more structured. Both stages contributed to greater
understanding of the issues and led to new approaches to documenting outcomes. A
third stage is ongoing, as the three programs continue to build on their work in t h s
project. Each stage included:
Understanding the situation and clarifying the questions
Examining frameworks, particularly EFF, that could structure action to document

outcomes
Developing and implementing new approaches
Reflecting on results
CLS staff serving as facilitators and documenters and teams of practitioners

from three adult education programs carried out the project work. For the names and
positions of the staff and team members, see Appendix 1.
In the first stage, the methodology focused on understanding the situation and
posing the problem. The facilitators worked with one program to define terms and
clarify the meaning of “outcomes.” At the same time, the program explored the EFF
framework and the ways it might contribute to documenting outcomes. In the
second stage, the facilitators used varied activities to help the teams understand their
situation in terms of documenting outcomes. In both stages, the program teams did
the primary work in developing and implementing new approaches. The following
table gives an overview of the project and the three teams’ work.
Table 1: Overview of the Documenting Outcomes Proiect
Tennessee: Knox County Virginia: Mount Rogers Regional Kentucky: Knott County
Adult Literacy Program Adult Education Program Adult Learning Center
Fall Became EFF field site

1997 Began to meet with CLS
facilitators to plan the action
research
Winter Team attends EFF Institute
1998 Team experiments with using
EFF, e.g., students rewrite EFF
poster
Spring Inputs-to-Impacts model
1998 developed
Fall Teachers begin implementing
1998 outcomes documentation
processes
Start second round of EFF
field research
Stage 2
Winter Continuation of documenting Join action research project Join action research project
1999 outcomes work 0 Reflect on student outcomes Reflect on student outcomes
Completion of documentation Complete documentation 0 Complete documentation
matrix matrix matrix
Implement and evaluate Explore learning activities
learning activities EFF introduction
ReviewEFF Revise Inputs to Impacts
Revise Inputs to Impacts model
model
Spring Regional meeting Regional meeting Regional meeting
1999 EFF Debriefing Institute Revision of state reporting Work on Parent Role
forms Map with students
Summer Decision to focus on Developed and revised Summer Reading Project with
1999 documenting EFF standard DoSetMet form calendar documentation
Take Responsibility for
Learning (TRL)
Fall Design and implement TRL Pilot DoSetMet with lead Implement expanded calendar
1999 process teachers and Washington Regional meeting
Regional meeting project County
Evaluation meeting
Regional meeting
Winter Continued to use Continued to use Continued to use
2000 documentation processes documentation processes documentation processes
Spring Final reports Final reports Final reports
2000
8
The Participating Teams
In the first stage of the project, the facilitators worked with a team fiom the Knox
County Adult Literacy Program (KCALP). We chose KCALP as a research site
because of its interest in EFF, history of work with the CLS, and interest in
developing a system of outcomes documentation as part of a continuous improve-
ment process. KCALP served Level 1 students (testing below sixth-grade level in
reading or math) as part of the county school system’s adult basic education
program. Located in an urban area in downtown Knoxville, Tennessee, KCALP
works closely with the nonprofit Friends of Literacy to provide services. Both day
and evening classes were offered at the main center, and Friends of Literacy operated
family literacy classes in a Knoxville apartment development. At both locations, day
classes met for five hours on each of four days per week, and evening classes were
held six hours a week. The two programs had eight full-time teachers and 59 active
volunteers working with 220 students.
In consultation with state adult basic education staff in Kentucky and

Virginia, two teams were added in the second stage. State staff were given a project
description and asked to recommend programs for the project. (Artifact 1 is a flyer
sent to state and program staff.) In both Virginia and Kentucky, the programs the
states recommended were interested in participating. CLS staff met with program
administrators and developed work agreements. KCALP also continued to
participate in Stage 2, but on a slightly different schedule. At all three sites, team
members were paid an honorarium for the extra time the project required.
The Knott County Adult Learning Center is located in the coal-mining region
of the eastern Kentucky mountains. In 1999, the program enrolled 70 students in
adult basic education (AE3E) classes and 23 in literacy classes. Classes were offered
six hours a day in an adult learning center in the county seat. The majority of the
program’s students were young single mothers. All five staff members (three GED
instructors and two literacy instructors) were on the action research team. Two
were professional teachers and administered the program; the others were para-
professionals.
The Mount Rogers Regional Adult Education Program is located in rural

southwestern Virginia and serves five counties and two small towns. This program
serves 1,200-1,500 students per year in English as a second language (ESL), ABE,
and workplace classes. The program also works with community colleges to help
prepare students without high school diplomas and provides adult basic education in
several correctional facilities. The program has about 40 part-time teachers and two
full-time administrators. Class schedules vary, but most classes meet for a few hours
’9
4
once or twice a week in a variety of sites. The Mount Rogers action research team
included three instructors from one county. Their classes included a weekly GED
class held in a local library, classes in a public housing development, a class at the
county vocational school, community classes held in a high school, and classes in
the county jail. The lead teacher from that county also participated in the project’s
final stages.
Artifact 1: One-Pape Description of Action Research Proiect Used in

Program Recruitment
Documenting Outcomes for Learners and Their Communities:

Developing Performance Accountability at the Local Level
A Center for Literacy Studies NCSALL Project
What: Action research projects to develop ways to document the outcomes of participation in adult basic
education programs on the quality of life of adult learners and their communities.
When: January 1999 to December 1999
Who: Teams of 3-5 teachers and 2-3 administrators from three programs in Tennessee, Virginia, and Kentucky,
working with staff from the Center for Literacy Studies.
Why: To contribute to the development of knowledge about performance accountability systems by exploring
ways that local programs can document outcomes.
How:
Teams will develop processes for documenting outcomes of participation in adult education in learners’
lives and communities and will implement these documentation processes in their program on a trial basis.
The teams will explore using the Equipped for the Future (EFF) standards as a framework for their
documentation and will consider quality of life indicators and measures used in fields such as community
development.
The teams will consider connections to state and federal performance accountability systems as they develop
their processes.
Project Activities
Teams will identify possible/probable outcomes for learners’ lives.
Teams will explore connections with EFF framework.
Teams will develop methods to try out ways to document outcomes. This will not be a comprehensive
system but will focus on a few outcome areas.
Teams will evaluate documentation process tried by team members.
Teams will implement one or more effective processes programwide.
Teams will identify connections with state systems.
10
SO
Summary
To summarize the project methodology:
The project was framed as action research, defined by Kuhne and Quigley (1997)
as “inductive, practical research that focuses on gaining a better understanding of
a practice problem” (p. 23). The need to document outcomes of literacy edu-
cation that are possible and useful at the program level was the problem in this
instance.
Three adult education programs teams participated in the project. A team of CLS
researchers facilitated their work.
The research was conducted in two stages, the first with one program team, the
second with two additional teams.
The research process began with a series of activities that enabled teams to
examine their current practice and consider how outcomes documentation fit
with their program needs.
The facilitators introduced possible approaches, including EFF, that might prove
useful in this effort.
Each team built on these activities through a cycle of planning, implementation,

and evaluation to develop their own documentation processes.
11
CHAPTER 3: IDENTIFYING THE QUESTION, PLANNING THE WORK
Both Stage 1 and Stage 2 of this action research project included a planning cycle.
In Stage 1, planning and building understanding took the greater part of the year. In
Stage 2, the planning processes were more systematic, and the teams spent more time
developing and implementing their documentation processes. This chapter describes
the activities used to plan and build understanding in both stages. The CLS
facilitators designed and led these activities and provided feedback and support to
the teams.
Stage 1 :The Tennessee Team Explores Documenting Outcomes
KCALP carried out the first stage of the project. The Documenting Outcomes action
research project was one of KCALP’s three major projects in 1998, and it was
integrated with and supported the other two projects. In 1997, KCALP had moved
from an ambitious strategic planning process to the even more ambitious project of
continuous improvement as structured by the Malcolm Baldridge Educational
Criteria for Performance Excellence, which focus on improvements to such
organizational aspects as leadership and work systems. At the same time, KCALP
was a partner in the EFF field development work. KCALP has been very deliberate
in choosing to be involved in projects that support its long-term strategic goals and
chose to continue their EFF collaboration and participate in the action research
project because these efforts would advance its goals. (For more about KCALP’s
program improvement efforts, see Cody, Ford, & Haywood, 1998.)
KCALP’s participation in the action research project started as a team of four

teachers and two administrators (who also taught) began work on the EFF standards
field development project. For this, the team developed a program profile, studied
EFF, attended two training institutes, and conducted in-class research on the EFF
standards. Teachers used selected draft EFF content standards to plan and teach
lessons and document student performance. They reported on this process and
recommended changes in the standards.
While KCALP teachers experimented with using the EFF framework,

KCALP team leaders met regularly with the NCSALL facilitators from the CLS and
identified and clarified issues to.address in the Documenting Outcomes action
research. CLS staff reviewed the literature on quality of life measurement, which
informed the discussions. The action research team also reviewed logic models that
the United Way of America (1996) and others (Flora, Flora, & Wade, 1996) used to
move beyond a simple input-to-output evaluation model, in which only such
13
22
immediate activities as number of clients served are considered, to evaluations that

capture more meaningful changes or outcomes.
Based on the logic models they reviewed, the facilitators and the KCALP
team developed the Inputs-to-Impacts model to clarify the various aspects of adult
education program processes. The model lists student and program factors
separately and defines input as the factor available for performance (i.e., what the
student or program brought to the processes of the program). The processes include
the educational and organizational processes a program implements. Outputs are
defined as the immediate results of these processes, whereas outcomes are the
changes that occur in students’ lives through participation in the program or the
long-term results of program improvement. Impacts are the changes in the
community resulting from changes in students and programs. This “Inputs-to-
Impacts” model became an important tool in the action research process. The action
research teams found the model useful in both analyzing their broad program
structure and examining particular activities (e.g., disentangling program outcomes
from individual student outcomes). The version of the Inputs-to-Impacts model in
Artifact 2 is a revision that includes items from all three action research teams.
14
Artifact 2: Logic Model Framework DeveloDed bv Programs
Inputs-to-Impacts Model:
A tool for analyzing performance factors in ABE programs
Student Program ,
Previous educational experiences Building, equipment

Factors available for Life experiences Staff
performance Goals Curriculum
Abilities Materials
Commitment to learning needs Program goals
Temperament (e.g., shyness) Volunteers
Challenges (e.g., learning disability, Technology
childcare needs, transportation)
Intake interview Planning
Educational and Orientation Record keeping
organizational Reading, writing, math activities Assessment
processes contributing Social interactions Scheduling
to performance School governance activities Instruction
Testing and assessments Staff development
Discussiodanalysis Advice, guidance, support
Cultural expressions Referrals to human services
Computer use
~~~
outputs: Test scores Number of classes offered

Immediate results of Journals Number of hours of instruction
services provided More comfort in class Number of students
GED Number of staff development
Resume activities
Certificates Records kept
Documentation of improved
performances
Outcomes: New reading, writing, math Aggregation of student outcomes:
Longer-term results of practices test scores
, .
education for Changed self-concept GED
individuals and Opened a checkindsavings account student goals met
programs Computer skills Teacher changes
New goals Improvement in program quality
Changednew skills Changes in program philosophy
Driver’s license/commercial
driver’s license
Workforce skills
Citizenship
Job or job promotion
Impacts: Children more involved in school
Changes in community Increased use of public resources
brought about by More activity in civic life
changes in learners’ Pressure for improved neighborhood
lives Better educateddeveloped workforce
This draft was produced by the staff and action research teams of the CLS NCSALL Documenting Outcomes
Project, 1999.
15
24
NCSALE Reports #20 March 2002
As the KCALP teachers used the EFF framework, they explored ways to
integrate the broad EFF skill standards with the basic academic skills they continued
to teach. They began to focus more on helping students identify life goals and
connecting instruction and learning to those goals. EFF’s role maps and focus on
learners’ purposes helped make those connections. The KCALP team-teachers and
administrators-identified the need for assessment measures that went beyond the
standardized tests they used. They needed ways to assess learning gains as well as
outcomes of goal achievement. Although students often told them about outcomes,
they had no systematic way of recording those oral reports. They discussed evidence
of performance of EFF skill standards and what would serve as evidence of goal
achievement. In fall 1998, the KCALP team began to focus on ways to document
outcomes. Rather than collecting evidence on a broad range of possible outcomes,
they decided to focus on learner goals and ways to document achievement of
particular goals and the resulting outcomes.
Beginning to Document Outcomes
Each of the four KCALP teachers used a different approach to document outcomes,
but all based approaches on learner goals. One teacher worked closely with two
students who wanted to open their own business. She met with them twice a month
to discuss their progress and used taped interviews as a documentation method. The
students started a cleaning business and also identified changes or outcomes. These
included increased self-esteem, discovery of capabilities needed to start at business,
greater comfort speaking in front of others, and an ability to solve problems. A
second teacher documented students’ use of math-specifically measurement
skills-as they painted and decorated their new classroom. She classified the work
completed in class-learning to compute area and perimeter, measuring the room,
and drawing scale models-as outputs, whereas the newly decorated classroom and
student reports of using measurement skills at home were classified as outcomes.
The outputs and outcomes were documented with artifacts or reports in student
portfolios. A third teacher tried using student journals to document use of a math
skill (estimation) as an outcome in learners’ lives. A fourth teacher kept a collection
of products students had produced in the computer lab to meet needs in their
everyday lives, an invitation or a flyer for a home business, for example.
While the teachers developed documentation processes for students in their

classes, the team’s two administrators focused on student outcomes resulting from
participation in a student leadership team. The administrators also began
systematically reviewing test scores with students and discussing errors, helping
students identify better learning strategies. The KCALP team summarized their
16
work in the first stage of this action research on documenting outcomes in an

adaptation of the Inputs-to-Impacts model, found in Appendix 2.
When the KCALP team reviewed their work in Stage 1, they concluded they
had accomplished a great deal in terms of understanding ways to structure their
instruction and tie it to student goals. The team members were using EFF to both
frame instruction and support their continuous improvement work, building student
leadership and integrating a month-long “Learning Skills Class” orientation into the
broader program. But the teachers’ efforts to document learning outcomes in
students’ lives, particularly outcomes outside the classroom, had not proceeded as
they had hoped. Teachers were confused about the difference between outputs and
outcomes. The processes they had developed were time consuming and did not
enable teachers to readily document and report outcomes to other stakeholders. In
the second stage of the project, KCALP tried a new approach more directly grounded
in EFF. This is described in Chapter 4.
Stage 2: Virginia and Kentucky Join in a More Structured Process
The second stage of this action research project built on the work and experiences of
KCALP in Stage 1 and broadened the project to include the Virginia and Kentucky
teams. Although the EFF framework continued to inform the project, the two new
teams were not directly involved in EFF development work. Both new teams were
from rural programs and added the perspectives of two different states.
During Stage 1 of this project, the KCALP team members largely developed
their action research work themselves, with guidance from CLS facilitators. During
,
the Stage 2 year (1999), CLS facilitators structured each team’s initial work. CLS
staff held about a dozen meetings with each team, the earliest including specific
activities to introduce the project and begin developing documentation processes.
The CLS facilitators collected data on these activities, including agenda, minutes,
and field notes from each meeting. A sample facilitator’s agenda for a team meeting
can be found in Appendix 3.
Activities common to all three sites (with some variation for KCALP)
included:
An initial activity to define the issue
Completion of a documentation matrix of current documentation processes
0 Exploration of instructional activities to document learner change
17
0 An introduction to EFF
A review of the Inputs-to-Impacts model
These activities, as well as activities at two regional meetings all three teams
attended, provided opportunities and structures to explore the theory and practice of
documenting outcomes.
After the series of initial meetings, the program teams began to develop their
own documentation processes. The later meetings were times to report and reflect
on and sometimes revise the teams’ work. Teams reported on their ongoing work at
meetings with the CLS facilitators. Each team’s work developing outcomes
documentation processes particular to their programs is described in Chapter 4.
Defining the Issue: What Are the Outcomes We Care About?
The initial meeting with each team began with a process to examine what was meant
by “outcomes” and to begin to determine the outcomes documented in each program
(the KCALP process was somewhat different because of its earlier work). Each
team member was asked to think of two particular students and write about changes
in these students’ lives that may have resulted from their participation in adult
education programs. The changes or outcomes were listed on newsprint and
discussed in terms of types of outcomes and reasons to document. We noted that the
outcomes the team members named were often not those the program documented.
Artifact 3 is drawn from newsprint created in the initial meeting of the Virginia team.
18
Artifact 3. Notes from First Virginia Team Meeting, 2/1/99
Outcomes listed for an ESOL student:
Learned English
Learned to follow directions in English
Uses map
Much stronger confidence
Uses English to interact with other students
Built friendships
Uses phone to make inquiries
Uses Internet to find information
Uses computer for Internet and educational activities
Outcomes listed for a student in an ABE class held in a public housing project:
Uses Internet to locate parenting and health information

Able to critically read Internet material
Uses Internet to follow interests
Read a book
Opened checking account
Got a job
Read school reports
Tried new parenting skills
Improved use of phone book
Outcomes for a student in a jail class:
Reading level improved

Reading Bible
Passed all but math chapter on the GED practice test
Doesn’t curse anymore
Plans to drive truck
The Documentation Matrix
Developing a matrix helped the teams focus on the documentation processes that
they and their programs already used. The facilitators asked the teams to bring all
their documentation forms to the meeting in which they developed a matrix. The
meeting room tables were covered with tests, folders, forms, and printouts. Using
newsprint or a whiteboard, the teams listed their program’s various forms of
documentation and answered the following questions about each piece: Who does it?
For whom? How often? How is the information used? What are the key items
reported? Artifact 4 is the Kentucky team’s matrix.
19
28
After completing this process, the teams examined program goals: those
established by the state, those determined by the program, and their individual goals
for their work. The teams then looked at the matrix they had created and noted that
accomplishment of many of these goals, particularly their own goals for their work,
were not being documented. For example, in Kentucky, both the program and state
goals were primarily focused on inputs-that 6 percent of the population would have
its educational needs served-whereas the teachers’ goals included “ensuring that
every student feels like they have experienced success in the program,” “fostering
the process of students gaining self-esteem and taking on leadership roles,” and
“helping students see the importance of personal responsibility.” The documentation
identified in the matrix process measured attendance and achevement of a variety
of state-identified objectives on a checklist, for example, “earn a GED,” “learn
wellnesshealth.” The program did not have a way to document the other outcomes
they believed were important to students’ lives.
20
29
Artifact 4: The Kentuckv Documentation Matrix
Documentation
Who does it? For whom? How often? How used? Key items reported (themes)
method
Intake Interview Anyone available who For program record Daily with each new To track student goals Goals
has first contact with the For state record student TogetGED
student at the Human One initial intake To improve basic skills to
Services Center interview per student enter into other educational
Programs
Objectives Form DEAL/KVEC/SEP form For program record Filled out initially, then State uses for statewide Objectives
(Student Goal Sheet) For state record updates as student statistics to report to Academic goals
Interest Inventory Form reaches objectives federal government Life skills (voting,
Documentation to enter parenting, health)
into other ed. program Improve basic skills
Report Data Teachers enter Within KY Valley Ed. Monthly As a program evaluation Attendance
(for state) information on diskette to Cooperative For End-of-Year tool for needs assessment Objectives
send to state (Program To target areas for Performance Report Goal attainment
Evaluation & Planning improvement Test scores
Branch, Frankfort) State uses to evaluate
regional programs
Teacher Notes Notes kept on For teachers’ use within Can bedaily Information purposes Specific academic
Prescription Sheet the program Whenever needed To help make lesson skilldneeds
Assessment notations State evaluators also use plans Attendance follow-ups
(e.g. needs more work as a program assessment
with fractions) resource
Record of Student
Contracts Sheet
Interest Inventory Forms Students fill out Upon initial intake To help students Identifying services offered
Sheets Student Goal Sheet themselves Periodically determine and focus their Determining student goals
checkedhpdated for goals Identify career interest areas
assessment
Student Journals Students keep journal For student 2-3 times per week To document and monitor To record student feelings
Writing assignments For teacher docu- writing progress and reflections
mentation of student As a writing exercise
writing skills To prepare for GED essay
30 31
Using Instructional Activities to Document Outcomes
To begin exploring approaches to documenting outcomes, the facilitators suggested

several learning activities that involved students in reflecting on the outcomes of the
adult education experience in their lives. Each team member was asked to try one of
these and record the results. Suggested activities included:
Students write a sentence, paragraph, or essay (depending on skill level) starting

with the stem sentence, “Since I started adult ed classes, my life has changed . . .”
Teacher leads a story circle in which each person tells a story of a change in his or
her life. Teacher records on a flipchart.
Student identifies a goal (e.g., helping children with schoolwork) and keeps a
calendar to record evidence of meeting that goal.
Student keeps a portfolio of items that show outcomes of changes in their lives,
(e.g., a canceled check copy if they opened a checking account). Student and
teacher decide what is evidence.
Student and teacher identify a short-term goal and ways the student would know
he or she is meeting that goal (the evidence). Together, they make a checklist,
and, as the student does one of these things, it is dated on the list. For example, if
a student’s long-term goal is getting a GED but he or she needs numerous math
skills to get there, the checklist might be the various math skills.
Team members in Kentucky and Virginia tried these activities and recorded
the results. Some team members used an activity once, and others used the activity
in an ongoing process. For the most part, the teams did not find these activities
effective as outcomes documentation. With some of the stem sentences, reported
outcomes were very general, and students did not reflect on or provide evidence of
outcomes. One teacher used a four-page set of stem sentences requiring specific
outcomes and evidence, but she reported that the process took too long and students
resisted taking the time. Using calendars and lists helped focus on particular
learning objectives but not on outcomes in people’s lives. Three teachers tried a
“story circle.” One interpreted this as a group discussion to address a particular
topic-emergency phone numbers-and the outcomes she identified were her own
assessments of student change (e.g., increased self-confidence). She did not
document student responses. The other two teachers focused discussions on changes
in students’ lives, but only one framed the discussion in terms of how being part of
the class “had helped them [students] individually.” An excerpt from this teacher’s
22
32
log is reproduced here. This teacher reported that the activity had given her a
structured, helphl way to talk to students about their lives, and both she and her
students were excited about the changes they recognized. However, except i n her
log of the discussion, the outcomes were not recorded or integrated into a broader
system of outcomes documentation.
Artifact 5: Notes from Teacher’s Log, 2/26/99
What did I do?
I organized a group discussion with three students. The group consisted of two students from the School-to-
WorWGED program and an ESL student. I began our discussion with telling the group that, as instructors, we
were able to document their academic progress but that I was also interested in their personal progress, how being
a part of the GED program had helped them individually. We began our discussion with:
“What’can you do now that you couldn’t do before?”
I. Ican read!
2. I can use a ruler, and this has helped me with my vocational class and projects at home.
3. I am more committed to meeting the goals I have set. Before entering the program, I was not very
responsible, and I didn’t know how to work towards something, but now I do. I have also learned study
skills that have carried over into my other (vocational) class.
4. I am not afraid to talk to other people or to ask a question.
5. I can communicate in English. I can read and understand English.
“Have you noticed any changes at home with yourself or family members?”
1. I am reading books and the local newspaper.

2. I am able to communicate with my mother. I used to always be negative about myself and everything else.
and that caused a lot ofproblems between my mother and I. I try to be positive, and that has helped the two
of us get a long better. I can express myselfbetter.
3. My father is learning English now. I help him a lot. Before, we only talked to each other in Chinese, but
now we talk some in English. We now want my mother to learn.
4. I study English on my own every day at home. I read the newspaper, history books, and other books.
5. Sometimes we get things in the mail, and myfather does not understand it, but I can now read it and explain
it to him.
“What about work?’
I. I am better organized and work harder than before. I want to accomplish more than what is expected.
2. I am more committed and dependable.
3. I had never used a computer before entering this program. I can now use a computer in my vocational
class. I can also use a copy machine and a f a machine.
4. I am able to understand my customers better and can help my father, mother, and uncle understand them.,
What did I learn?
The students were excited to share their thoughts and seemed to be even more excited about the changes they had
noticed in themselves. I was also excited about our group discussion. It proved to me that we don’t just affect
3ur students academically, but we have a great impact in all aspects of their life. I really enjoyed being with them
md talking with them.
23
33
Team members reported that these instructional activities were useful ways to
know students better and helped students reflect on goals and accomplishments.
Trying the activities also seemed to help some of the team members experiment with
new instructional approaches. However, the team members did not believe these
activities would meet the program’s documentation needs. As the teams developed
their own documentation processes, they integrated some elements of the activities
but developed more structured approaches.
EFF
A review of EFF was included as an action research activity because it informed this
project from the beginning, and we wanted each team to consider the EFF frame-
work as a possible structure for documenting outcomes. KCALP was a partner in the
development of EFF and learned about it both by attending EFF-sponsored training
institutes and by integrating EFF into their program. Some Virginia programs were
involved in EFF development, and the Mount Rogers team had been introduced to
EFF through their state staff development system. However, they had not imple-
mented EFF in their program.
The Knott County team was not familiar with EFF. To introduce EFF to the
Knott County teachers, CLS facilitators used a process the EFF staff had developed.
Team members were asked to identify something they would like to learn to do and
to connect these to the EFF role maps and common activities. They then identified a
skill supporting that activity and brainstormed about ways they might practice that
skill. For example, one Kentucky team member identified a wish to speak more
comfortably in public. She located “Form and Express Opinions and Ideas” on the
Citizen Role Map and “Develop and Express Sense of Self’ from the Common
Activities, which could be supported by the generative skill “Speak So Others Can
Understand.” She talked with another team member about ways she could practice
this skill.
After the Knott County team reviewed the EFF framework, each teacher
planned to use EFF in an activity with students that would also help document
outcomes. The lead teacher reviewed the three role maps with a class of four
women. The group chose to focus on the citizen role. They looked at the key
activities and brainstormed about ways to do each. They then chose one activity
from their list: Write a letter to “Form and Express Opinions and Ideas.” This was
an activity that they could do in one class period and that also addressed a GED skill.
The teacher reviewed the format for a formal letter with them. Two wrote to their
Congressman about welfare reform, and two wrote to the state’s Department of
Transportation about bad roads. The letters served as documentation (of an output,
24
34
not an outcome). Other Knott County teachers tried different activities, some more
closely tied to the EFF framework than others. Team members were pleased with
the activities as ways to involve students in thinking about their goals and learning,
but the activities did not result in outcomes documentation. A similar process was
used to review EFF with the Mount Rogers team.
Reviewing Inputs-to-Impacts
In Stage 2, the Kentucky and Virginia teams reviewed the Inputs-to-Impacts model
developed with KCALP in Stage 1. Both teams added items to the original chart.
The version on page 14 includes their revisions.
The model had been developed as part of a process to define terms and
clarify the relationship of outcomes to program processes. It proved useful as a
p l k i n g and analysis tool as well. For example, as the Virginia team developed its
documentation processes, the facilitators used the model to distinguish outcomes
(e.g., help children with homework) from inputs (e.g., attend class regularly) in an
early list of “learner achievements” they planned to track. KCALP used the chart to
present its work to the other teams at the regional meeting and added categories
describing evidence of its outcomes. (See Appendix 2). The Kentucky team’s use
of the chart as part of the planning process is described in Chapter 4.
Using the Inputs-to-Impacts model as a tool to review the various

instructional activities the teams had tried led to clarification in thinking about
outcomes documentation. The teams looked at a list of the outcomes identified from
these activities and decided some were outcomes (e.g., helping children with their
homework), but others were outputs (e.g., learning computer skills or doing written
summaries). The facilitator noted at least three approaches to documentation of
outcomes. One was to take a broad look and document whatever outcomes were
found. The story circles and stem sentence activities did this, asking students to talk
or write about the changes they had seen in their lives. A second method was for the
program or teacher to identify a desired goal and document achievement. This was
done with the calendar activity in which the teacher’s goal was for students to keep
appointments, and she had students use a calendar to document this. A third
approach was for the student to identify a goal and to work with the teacher to
identify ways to determine and document when this outcome was reached. The
teams eventually developed this last approach.
25
Two Regional Meetings
In addition to meetings held at each program, the teams came together in two
regional meetings. The first, held in April 1999, gave people from each team the
opportunity to meet the other teams. Each program did a brief presentation on their
program and community, and the team from Knox County presented their Phase 1
work to develop outcomes documentation (see Appendix 2). Artifact 6 is one
facilitator’s summary of the themes of the first meeting.
Artifact 6: Reflections from a Facilitator after the April 1999 Meeting
Briefly, the broad topics that underpinned the meeting were:

Understanding the current situation
Looking at what is documented, how, why
Identij’jing outcomes that are not documented but are important
Understanding stateyedera1 systems
Understanding the theory

Introduction to Equippedfor the Future framework
Examining the Input-Impacts continuum
How do we collect evidence
Thinking about performance accountability
More experimentation with documentation processes, focused on those particular outcomes or goals.
Try promising processes programwide.
We are in the process of “acting” on the process of documentation while keeping all of the above in mind.
Discussion topics at the first meeting also included differences in reporting

requirements among the three states and differences between what the state found
important to document and report and what concerned students. The group also
spent some time considering how EFF could help frame instruction that led fiom
students’ goals to outcomes the students and others hoped for in their lives and
communities. At the end of the meeting, the teams expressed excitement about
meeting each other and about focusing on students’ needs and goals. At the same
time, they expressed frustration at the program funders’ demands, which some saw
as interfering with their work and requiring time to complete reports on results (e.g.,
attaining a GED) unrealistic for many students. They felt pushed to focus on things
that were not necessarily important to the students.
In November 1999, the groups met again and presented the documentation
processes they had developed and were now testing. Representatives from the
Tennessee and Kentucky offices of adult education who were involved in developing
26
reporting systems for their states also attended and commented on the presentations.
The state staff found the work exciting but were not optimistic that it could be
integrated into the state performance accountability reporting. At this meeting, the
team members also wrote evaluative comments on their project experience
(summarized in Appendix 4).
Summary
In the initial meetings of both stages of this project, the action research teams took
part in a variety of activities that increased their understanding of outcomes
documentation. They agreed on terminology and identified the documentation
processes already in place in their programs. They tried a variety of approaches to
documenting outcomes and learned about EFF and how it could be used to frame
their work. In the next cycle, the teams designed and tested their own approaches, as
described in the next chapter.
27
37
CHAPTER 4: THE TEAMS’ DOCUMENTATION PROCESSES
After the April 1999 meeting, the three teams began to focus on their individual
documentation projects. The plan was for each team to develop a documentation
process; pilot it with a few classes; and, if successful, move it into the entire
program. The facilitators originally thought the programs might develop processes
enabling them to document outcomes in ways acceptable to state systems. The
Virginia team focused on this. The Tennessee team’s process was designed to meet
needs identified in their program improvement process, and the Kentucky team built
their process around particular goals the teachers and students identified. We begin
with the Kentucky team’s results.
Supporting Children’s Education
The team at the Knott County Adult Learning Center was intrigued by the EFF
framework and saw it as a way to integrate some of their students’ life issues with
the program’s focus on academic skills. They discussed the EFF role maps with
students, and eventually the group (staff and students) determined that they all shared
the parenvfamily member role. The students listed areas of concern and parenting
issues of interest to them. At an action research team meeting, the team sorted
through this list and identified the overall goal of being a better parent and a subgoal
similar to the “Supports and Encourages Child’s Education” EFF Key Activity from
the Parent/Family Member Role Map. We used the Inputs-to-Impacts model to
analyze this Key Activity, and the program team decided to focus on reading to
children as a way to support children’s education. Artifact 7 is the team’s planning
grid the team created.
Artifact 7. Planninp Done on Whiteboard
(The plain text was created by the Knott County staff and students, and the items in
italics were added during the action research team meeting.)
1 Big Goal: Be a Better Parent: S U_ _R R Oand

~ ~ SEncouraees Children’s Education
Inputs Individual Goals Processes/Outputs Outcomes Impacts
0 Maintain 0 Pretest with TABE Demonstrate 0 Parenuchild
reading skills Develop book list improved reading transfer
0 Dobetter in 0 Fill in calendar test scores 0 Better reading in
Teachers school Folder for parent Continue to use classroom
Book- Parentkhild 0 Check calendars library 0 Increased library
‘L
mobile bonding weekly Closer to children. use
Visit library better relationship
0 Talk about reading Continue to read
to children to child
-
29
38
The team considered teacher observation or a checklist of age-appropriate

books as documentation but decided to have parents document their reading on a
weekly calendar. Over the next few months, a group of parents read to their children
and recorded what they read and the amount of time spent reading on calendar pages
the teachers provided (see Appendix 5). Some students participated in the county
library’s summer reading program with their children. The team used a computer
database to record the information from the calendars, including the title of the book
read, the source of the book (home or library), and the number of minutes read
(5-15, 15-30,30+). They also recorded the child’s name, age, and school, and the
parent’s pretest TABE (Test of Adult Basic Education) score administered in the
spring before this reading project began.
At the end of the summer, the CLS facilitators reviewed the data and
determined that the parents read to their children on an average (mean) of 8.8 days.
These days were all in June because a school vacation interrupted the program. (The
Knott County program had to move to a new center in July, causing additional
disruption.) The team administered the TABE again in the fall. The team reported
that 6 of the 10 participating adult students had advanced to another reading level
on this standardized test. At the fall regional meeting, the Kentucky team reported
that adult learners’ self-confidence had increased, their family relationships had
improved, and their desire for their children to be readers had increased. Artifact 8
is the summary.
Artifact 8. From the Overhead Used by the Knott Countv Team at the
November 1999 Regional Meeting
Inputs Processes outputs Outcomes

Negative past school Conduct orientation Library cards obtained Improved reading ability
experiences Administer pre-/post-test Calendars completed (parent and child)
Low self-confidence Visit library Pre-/post-test scores Continued use of library
Shyness Develop book list obtained Continued reading to
Lack of commitment to Create reading calendar Number who advanced a children
education Provide folders reading levellshowed Closer relationship with
Lack of reading Collect calendars weekly improvement children
material in the home Compile data Number of books read Greater appreciation of
Limited finances, Number of minutes spent education
inadequate childcare reading Increased self-confidence
and transportation Increased leadership role
in children’s education
30
After the summer reading program ended, the team met with the parents and
found they remained enthusiastic about focusing on their children’s education. They
wanted to encourage their children’s reading and had ideas about how to do so. The
team designed a new form on which parents were asked to record instances of
reading to their children, helping with homework, children’s school attendance,
children’s use of the public library, and meetings with their children’s teachers (see
Appendix 5). The team planned to collect these forms monthly and record the data
in a computer database.
This effort was not as successful as the summer reading program. Only a
few parents completed and returned the forms. The team decided the form was too
complicated and not relevant to everyone. They revised it, scheduled more parent
meetings, and loaned parents cameras to use as another way to document educational
activities at home. The team presented the pictures at the November regional
meeting. This was successful as a one-time effort but not something the team
continued.
In their final meeting with the CLS facilitators, the Knott County team said
the documentation work had helped them know their students better. They also said
both students and staff were more deliberate and purposeful in their work. They
described their changing student population and how this affected their docu-
mentation efforts. When the action research project began, their students were
mostly mothers in a welfare-to-work class. Several of the students in the parent
group had graduated or left the program. The program’s students now included
many more young male and female students who were not parents. The focus on
supporting children’s education was no longer as appropriate. After the Docu-
menting Outcomes project was “officially” over, the Knott County team continued
to experiment with EFF and with documenting their work. They have focused on
developing a process to help students set and document goals.
Taking Responsibility for Learning
In Tennessee, KCALP team teachers had developed processes to document outcomes

during Stage 1 but had not carried them beyond their individual classes (see pp.
15-1 6). After the April regional meeting, the Virginia and Kentucky teams focused
on developing their documentation processes, but KCALP went through a crisis
period brought on by conflicts between county and state agencies involved in
providing educational services to welfare-to-work clients. The program survived
this crisis, but KCALP no longer contracted to provide these services, and several
teachers left. By the end of the summer, the situation had stabilized, and KCALP
31
was ready to plan for a new documentation process. Three of the original team
members and four new teachers participated in the action research team in Stage 2.
Throughout the fall, the team met and developed a plan to document the
outcomes addressing the EFF Standard “Take Responsibility for Learning.” They
presented the plan at the November regional meeting (Appendix 6). Plans included
teaching about the standard, using a story that demonstrated someone taking
responsibility for learning, and collecting data on students taking responsibility for
learning.
Fipure 1. EFF Standard “Take Responsibility for Learning” (from Stein, 2000)
Take Responsibility
for Learning
Establish learninggoals that 3re
based on an understandungof one’s
own current and future learning
needs.
Identity own strengths and
weaknesses as a learner and seebt
out opportunitiesfor learning lhat
help build self-concept as a learner.
Become familiar wlth a range of
learning strategies to acquire or
retain knowledge.
ldentii and use strategies
appropriateto goals. task, context.
and the resourn available for
-Monitor proeress toward goal8

and modify strat-ies or other
leeturesoftheleamingsituation
Bsnecessarytoachieve~
32
41
Teachers gave students a presurvey on Take Responsibility for Learning

(TRL), asking them to write briefly:
What does it mean?
0 Why is it important?
0 How can I use it?
A postsurvey asked similar questions:
What does it mean to take responsibility for learning?
0 How does taking responsibility for learning work?
0 Is taking responsibility for learning important to you? Why?
How have you used responsibility for learning in your life?
0 How can you use responsibility for learning in your life in the future?
During the month between the surveys, teachers focused on TRL, introducing
the concept by asking students to read, write about, and discuss a story about a
famous person (e.g., Bill Cosby) who took responsibility for learning in his or her
life. Students were asked to keep a journal of events in their daily lives that indi-
cated taking responsibility for learning and to share these with the class. At first,
students tended to write the same things every day and did not seem to grasp the
TRL concept. The teachers tried brainstorming about TRL’s meaning in a staff team
meeting and then did a similar activity with students. These brainstorming activities
seemed to improve students’ understanding of and commitment to the project. The
teachers also kept logs of their observation of TRL in the classroom, though they
reported they did not really have the time to record everything they noticed.
.The KCALP team met with CLS facilitators and discussed their experiences
with using the TRL standard. They found the student journals were useful as a
writing activity, and the teacher log helped with planning. However, the pre- ahd
postsurveys were most useful in identifjmg and documenting instances of students
taking responsibility for learning both in class and in their everyday lives. Among
the changes they ,noted in students were:
33
42
Increased writing in journals and greater variety in examples of TRL that

students identified
More learning activities outside class
Some students were more aware how they best learned
Two students got e-mail accounts, and two got library cards
Students became better at setting short-term goals
Some students were more proactive about issues in their lives (e.g., getting a
landlord to make repairs or asking for a work schedule that fit with their
schooling)
The teachers also reported changes in themselves. As they worked on the

project, they adapted their own classes (e.g., using facilitated discussion in a
beginning literacy class to identify instances of students taking responsibility for
learning). One teacher said she became “less authoritarian” and “included more
independent and flexible student-centered learning time.” The project reinforced the
team’s commitment to involving students in decision-making. The project also
served as a team-building process for a group of new teachers. KCALP .plans to
continue to teach TRL, introducing this EFF standard as part of its month-long
Learning Skills orientation class. As the program implements EFF, TRL will be
applied in student goal-setting, planning, and assessment processes.
DoSetMet
From the beginning of the project, the Mount Rogers team focused on connecting
action research work with efforts to develop a process for reporting required data to
the state. They reviewed Virginia state documentation requirements as well as the
forms used locally. They then developed a new form to be used at student intake to
collect the information required by the state. This included demographic information
and the student’s source of information about the program, reasons for enrolling,
goals, test scores, and other information. They also added a list of “learner
achievements” based on a short checklist of personal, social, and academic learning
skills. These were chosen as a way to begin to document more than test scores.
Some were outcomes (e.g., “helped child with homework”) and some were class-
room activities (e.g., “worked on assigned tasks”). After discussion in action
research team meetings and several revisions, the team decided to move their
outcomes list to a separate document.
34
“ -4-3
The team had determined that using activity-based documentation, such as

story circles and stem sentences, was too time intensive. Instead, they developed
their outcomes document as a checklist, using some items fiom their learner
achievement list. They first conceived of the checklist as an exit instrument and a
supplement to the other reporting form. It was organized around the three EFF roles,
plus the category of “self.” The original draft had a space to check when something
was accomplished and a space for comments. The following were examples of
accomplishments: use library, ask for directions, read help-wanted ads, use a
computer, volunteer in child’s school, and pay bills. The team decided to add a goal-
setting column to use at intake. The learner could note which items she or he had
already accomplished and those that would indicate progress toward goals. Items
relevant to the student’s goals would be the focus, but other items accomplished
could be checked as well, Though these were decontextualized competencies or
activities, they could be contextualized by the learner’s goals. The team discussed
having a space for the goals on the form and giving the student a copy of the form,
both to facilitate documentation and as motivation.
The team tried using this new form with several students. They noted student
reactions, how long the students’ took to complete it, and whether students could
give evidence of their accomplishments. They found that although it took some time
to complete, most students liked the form and were able to describe their accom-
plishments. The students added items and suggested language changes. The team
found the list helped both the teacher and the students think about goals and out-
comes. The form was revised to include student suggestions. In its current iteration,
the form lists 43 items with three possible responses to each item: currently do (DO),
would like to do (SET), and now can do (MET). Artifact 9 reproduces the summary
version of this form.
In fall 1999, several Mount Rogers teachers outside the team tested the
DoSetMet form. These teachers met with the team in November to reflect on their
experience with the form. In this group meeting and in written evaluations, the
teachers were quite positive about the form’s usefulness both as a way to help them
know their students and as a goal-setting process for students. Several noted the
form helped build students’ self-esteem as they focused on what they could do. An
employee of the Virginia Department of Human Services who attended the meeting
requested a copy of the form to use as part of the department’s intake procedure for
new clients.
Virginia had introduced a statewide computerized data collection system

before it was required for the National Reporting System. One of the data items was
achievement of learner goals. Mount Rogers teachers used the DoSetMet form to
35
44
document learner goals as students set and achieved them. This information could
then be entered into the information system. Virginia recently revised its reporting
system, and identification and documentation of student goals is no longer a focus.
However, Mount Rogers teachers continue to have DoSetMet as an option to use for
goal setting and documentation; some teachers, including the action research team
members, continue to use it. But the Mount Rogers team has not been able to
integrate the form into their formal reporting system as they had hoped, and form
has not been introduced statewide.
36
45
Artifact 9: Do/Set/Met
DO/SET/MET GOAL SHEET
Instructor: Class:
Date:
Please fill in the total number of students in your class that have checked DO/SET/MET in each of the following
categories.
This form was developed by Rita Roper, Jeny Musick, and Shem Whitlock, teachers in the Mount Rogers Regional Adult
Education Program in Abingdon, Virginia.
37
NCSALL ReDorts #20 March 2002
Summary of Team Activities in Stage 2
Three diverse programs participated in the Documenting Outcomes action research

project. They all developed outcome documentation processes they found useful, at
least in the short term. Table 2 summarizes their efforts.
Table 2. Summary of Stage 2 Team Activities Developing Outcome

Documentation
KCALP, Tennessee Mount Rogers, Virginia Knott County, Kentucky

Documentation Five teachers spent a Three teachers developed Five staff members
Process month working with their a checklist of 43 possible worked with students to
students focusing on the goals/outcomes that . identify the goal to
EFF standard Take students use to evaluate support their children’s
Responsibility for current skills, set goals, education. Students
Learning (TRL) in and document documented activities
students’ lives. accomplishment. that did that, e.g., reading
to their children.
Documentation To determine how To help students set and To motivate students and
Purpose students were document goals for state document increased
implementing TRL in reports. activities supporting
their daily lives children’s education
Documentation Pre- and postsurveys DoSetMet checklist Calendar forms
Tools Student Journals
Teacher observations
recorded in journals
Where In program Countywide In program
Implemented
Used As a foundation skill for To report goal As a focus for
learning achievement to state instructional activities
To plan instruction
Continues In Learning Skills As an option, countywide No
orientation class Team teachers are using
38
47
CHAPTER 5: FINDINGS: WHAT DID WE LEARN?
Action research is a spiral of research, learning, action, and reflection. It is an

ongoing process without a clear point at which the researchers come to the end of the
project, analyze their data, and report their findings. As practitioners research their
own practice, analysis is ongoing, questions change, and the project doesn’t really
end. But at some point, the focused work stops, and researchers can summarize what
has been learned. We have reached that point with this report. In this section, we
begin to answer the following questions:
1. What have we learned about documenting outcomes of participation in adult

basic education programs?
2. What have we learned about conducting action research in adult basic education
programs?
3. What have we learned about how action research affected the participants?
The CLS team developed the findings in this section on the basis of an
extensive review of proj ect data, including team and regional meeting minutes,
artifacts from program teams, CLS staff field notes and personal reflections on
meetings, results from evaluations, interim and final team reports, and informal
interviews with team members.
Documenting Outcomes of Participation in Adult Basic Education Programs
In this project, we set out to work with adult basic education programs to develop
indicators and measures that could be used to document the outcomes of student
participation in adult education. We were interested in going beyond what could be
measured by intake/exit demographic data and standardized tests. We hoped
program teams would be able to document what one teacher called the “invisible
outcomes,” the changes teachers saw or heard about from their students but had no
way to report. We hoped that, with the project participants, we could create methods
to capture and report these outcomes as part of state performance accountability
systems. We imagined being able to document outcomes in a way that made a
compelling case for community impact.
The teams that took part in this action research developed ways to document
changes in students. Some of these changes fit our definition of outcomes and go
beyond classroom activities-they are changes that make a difference in students’
lives. The Virginia team’s DoSetMet form established a process that helped students
39
48
identify the changes they hoped to make (or their educational goals) and document
achievement of those goals, or outcomes. The Tennessee team documented
outcomes as well as outputs as part of their focus on EFF’s TRL Standard. The
Knott County Adult Learning Center students used calendars to document activities
that supported their children’s education, such as reading to their children and
meeting with teachers.
These documentation efforts focused on particular parts of studeqts’ lives that

the program or students identified as areas in which they hoped to make change. The
three teams developed ways to document changes the students reported and the
teachers observed. The teams, their programs, and their students have found these
documentation efforts useful tools for instructional planning and for learner and
project assessment. The programs have continued to use these documentation tools
and to develop others.
For the participating teams, this action research project has led to increased
understanding of how programs might identify and document the outcomes of adult
education participation in learners’ lives in ways that meet local program needs.
This project explored, through action research, the development of indicators and
measures of the impact of participation in adult literacy programs and used the EFF
framework to inform this work.
However, in the span of this project, we were not able to develop measures
acceptable to the state accountability systems with which we hoped to connect.
This was not caused by the involved states’ lack of interest. State adult basic
education staff from all three states followed their state team’s work and engaged in
at least some consideration of how it might be integrated into state adult education
reporting systems. Unfortunately, the federal performance accountability require-
ments limited use of locally identified and documented outcomes. The state staff
were more concerned with establishing the reliability and validity of locally devel-
oped documentation. Putting in place a new reporting system that would meet the
guidelines of the National Reporting System was their priority, and they did not feel
they had the resources or perhaps the latitude to experiment with other approaches.
For local documentation efforts to be part of a state performance account-

ability system, local programs need a commitment to rigorous, ongoing data
collection and analysis. Each process the action research teams developed included
collection of some baseline data (on goals, test scores, or assays of knowledge of a
particular EFF standard) and collection.of data on outcomes. However, analysis of
this data varied from project to project, most likely because of the facilitators’
limited focus on analysis and the teams’ limited time. Giving the teams more
40
49
extensive training in analysis as well as a longer commitment of time would have

been helpful.
The Rewards and Challenges of Action Research in

Adult Basic Education Programs
Participating in this action research was both a positive experience and a challenge
for the people involved. In this section, we examine what was most helpful and
valued by the participating teams, the challenges the teams and facilitators
encountered, and particular issues we faced as facilitators.
The Rewards
Throughout the project, we asked for the teams’ feedback on the action research
process through informal activities, such as having them write brief answers to
questions on cards, and through structured surveys completed at regional meetings
(see Appendix 3). In these activities and surveys, the team members identified a
variety of activities they found useful, including analyzing current documentation in
the Documentation Matrix and using the Inputs-to-Impacts chart. More of the
positive comments concerned the overall process. People liked working in groups
and having a chance to talk about their work and share experiences with teachers on
their own and other teams. As one teacher wrote:
I enjoy listening to the ideas/opinions/responses of my coworkers. It helps me

understand their focus.
Several people also mentioned appreciating having facilitators who brought

different ideas and different approaches to facilitation. One person said she found
“respect for everyone involved.”
The two aspects of this project the participants seemed to particularly value
were the focus on students and the opportunity to reflect on the goals of their work.
The focus on students and their goals influenced how team members approached
their work, as discussed in Chapter 5.
The Challenges
As the facilitation team reflected on what we learned from this project, one of us
summed up the challenges by saying, “This work is hard.” And it was, because of
the process and for reasons that are part of working in adult basic education.
41
We used an action research methodology because we wanted to work with

programs to develop outcomes documentation rather than field test a particular
approach. But this led to some initial frustration for team members, who noted,
“There were no predefined test instruments in place” and “We didn’t know what to
collect or what to do with it.” In the first stage, we worked with the Knox County
Adult Literacy Program to conceptualize what we were doing: What did we mean by
“outcomes”? Who was the documentation for? How did our documentation
processes fit with the program and state processes? In this first stage, we were able
to develop processes that helped the Stage 2 teams go through similar thinking in a
more structured way. But these teams also felt some initial frustration. Helping
everyone involved in the project understand and, if possible, appreciate the processes
of developing the action steps to be taken is one of the challenges of action research.
When the teams were asked to identify challenges, they named some
conceptual issues, such as “the idea of documenting changes in learner lives versus
test scores,” but most named practical challenges, including:
Getting students to provide data by completingjournals, calendars, and charts
Student turnover and a changing student population with more young students
Limited teacher time to design, collect, and use documentation
Difficulty in finding ways to share results with other teachers in other programs
Specific program issues, such as office moves or hnding changes
Many of the project’s challenges are those of our underfunded field. In

discussing the amount of personal (uncompensated) time her teachers devoted to the
project, one administrator commented, “This is not a new problem for adult edu-
cation-it is a continuing problem that is a barrier to providing excellent education.”
Most of the teachers were part-time. The marginal place of adult education in the
larger system is typified by the forced relocation of the Knott County program and
the loss of major hnding in Knox County.
The changing student population the action research project teams noted is
typical. In adult basic education, there is high student turnover (Young, Fleischman,
Fitzgerald, & Morgan, 1995) and an increasing number of younger students (Hayes,
2000). Documentation of outcomes assumes continuing contact with students or at
least an exit interview, but this is difficult to obtain, as students often stop attending
class without notice (Beder, 1999).
42
The requirements of the perforinance accountability structure developed in

response to Title I1 of the Workforce Investment Act (1998) also had implications for
local program teams. State reporting requirements were in flux. The pressure and
confusion of adopting a new reporting system added to the teams’ work. The
mandated performance accountability system seemed to idubit states’ willingness
to experiment with the processes that the action research teams were developing.
The Facilitation Process
As facilitators, we increased our knowledge of action research as a learning process.

We came to realize that coming to consensus was part of any action research. Our
growing understanding about the connections between documentation and other
program processes, as well as our understanding of the challenges programs and
teachers face in trying to integrate documentation with instruction, came with
reflection on the ongoing work.
One issue in action research in adult education is team members’

participation in all aspects of the research process (See Adelman, 1997; Peters,
1997). We did not begin this project thinking of it as participatory action research.
As facilitators, we had already defined the question we wanted to address, and we
established the structure of the process. But as the project progressed, we found it
was in many ways controlled by the teams. We had been clear from the beginning
that the documentation processes the programs would implement would meet their
programs’ needs. Although we, as the researchers, could have identified particular
measures of particular outcomes and asked the programs to pilot these, our approach
was to work with the program teams to develop measures-or processes-that fit
their contexts. The work sometimes proceeded in unexpected ways and was tied
more than we might have anticipated to particular instructional goals, but this was
what was needed and possible. The teams’ work also led to results beyond the
original intention of the project, as discussed in the next section. One facilitator
described our experience:
The hardest part and the thing that sometimes makes the process awkward is not
having a “playbook” because the process is organic, but the unknown, the twists and
t u m s of this kind of work, is where the yield that is the most informative and
ultimately the most gratifying comes into focus.
Another quandary we faced was how much we could reasonably request of

team members. Although they received compensation for time spent on the project,
they had too little time. In retrospect, we might have, for example, asked for more
written documentation of team meetings we did not attend. We might have
established a more extensive documentation process for the project, but we were
43
hesitant to push for too much. The teams worked hard and enthusiastically, and their
investment in the project may compensate for what we lost in data.
Looking back on the project, we have identified several factors we might

change in another cycle. First, as facilitators, we needed more time on the project.
Like the program teams, we were all part-time. The teams were in three different
states, which limited our ability to attend all team meetings. We also felt limited by
funding and did not extend the programs’ involvement more than a few months
beyond the original plan. In retrospect, it would have been useful to have had more
time to work with the program teams to help them refine and analyze data from their
documentation processes.
Although the facilitation process was not always smooth, we developed tools,
such as the Documentation Matrix, that we found quite useful in helping the teams
look more broadly at their programs and their practice. We came away from this
project with a renewed commitment to the value of action research in sharpening the
questions participants ask about their work and the work of adult basic education as a
system, and in giving participants tools to help answer their questions.
What Else Have We Learned? Action Research as a Learning Process
In his article “The Role of Research in the Practice on Adult Education,” Allan
Quigley (1997) places action research in the “practical” category of his “Research
Intentionality Framework” (p. 17), or research for “practitioner development and
institutional improvement.” Although the purpose of our research was to contribute
to understanding a systemic issue-how to document the outcomes of participation
in adult basic education-we found it also contributed to participants’ professional,
program, and personal development.
Professional development
“The result is improvement in what happens in the classroom and school, and a
better articulation and justification of the educational rationale for what goes on”
(Kemmis & McTaggart, 1984, p. 5 , quoted in Quigley, 1997). The teams reported
these kinds of results in their final reports:
The project built a team out of a group of newly hired teachers
The focus on TRL led the teachers to give students more responsibility
Teachers and students were better able to set goals
44
53
Teachers focused more on student goals
Teachers know students better and are better observers of students’ life situations
Teachers and students use different thinking processes and are more deliberate
and purposeful
Teachers are able to turn new ideas/approaches into teaching strategies
Effective professional development involves engagement in active thinking

about one’s practice, and this occurred during this project. As they participated in
this action research, team members changed their understanding about aspects of
their practice and reported ways their practice had changed. The process of
identifying desired outcomes led to increased instructional focus on meeting learner
goals and achieving desired outcomes. As one teacher said, “If you start with
student goals and you document reaching those goals, you better do something in the
classroom to help them learn what they need to get there.” In some instances, this
meant a classwide focus, as in the Knott County summer reading project; in others, it
meant extra focused help from a tutor as a student worked to meet a goal.
Developing processes to document changes in learners’ lives meant talking

to learners about topics at a depth new to many team members. They found they
gained understanding of their students’ lives and the barriers students faced. One
teacher who piloted the DoSetMet form spoke of how it led her to initiate dialogue
with students, and the team members from Virginia said that having a tool-the
DoSetMet form-facilitated their ability to talk to students about issues in the
students’ lives. A Tennessee teacher reported, “Based on the issues and concerns
revealed by this project, I revamped my classroom procedures to include more
independent and flexible student-centered leaming time.” Ten of 13 team
respondents to a survey late in the project said their instructional planning and
practice now focused more on student goals and needs.
As part of the project, team members also had opportunities to try new
instructional techniques and learn research skills of data collection and, in one site,
computer data entry. One Kentucky team member commented that she had
“stretched” herself. One final team report spoke of the staff being “able to turn
new ideas/approaches into teaching strategies.”
45
Student development
This project also led to learning opportunities for students. All three teams involved
students in designing documentation processes. The Virginia team tested each
version of what became the DoSetMet form with students and revised both the
content and the language on the basis of student suggestions. The Kentucky team
focused their project on parents’ reading to children after the team used the EFF
framework in a series of meetings with students to identify a goal (to support their
children’s education) and steps to meet that goal. The Tennessee team revised their
approach to documenting TRL on the basis of student feedback.
Students reflected on goals and accomplishments in each of the teams’,

projects. The Mount Rogers team reported that the process motivated students and
built their confidence. Students also set more specific goals when they used the
DoSetMet form. The Knox County team also reported that students were better at
setting short-term goals.
In addition to the goal-setting changes and an increased awareness of

accomplishments, project activities generated a variety of student outcomes and
outputs. The Knott County team reported improved reading, increased library use,
and better “organizational habits.” Some Knox County students developed an
interest in writing in ajournal and continued to do so after the TRL, project ended.
The team reported, “Almost all students seemed to have learned more about
themselves as learners.” The learning activities that led to these changes could, of
course, be carried out as part of regular classroom practice, but the action research
project encouraged the teams to undertake these activities.
Program development
The Knox County team has been intentional about program improvement for several
years. They are participating in the Baldridge National Quality Program award
process and have set clear priorities for change. Participation in the action research
contributed to this process, helping them “understand what they were doing and
why.” One KCALP team member felt affirmed in the program improvement effort
after observing the action research facilitators’ similar processes of reflection and
analysis. For all three teams, thinking through program processes led to increased
appreciation of how different program aspects-goal setting, instruction, outcomes
documentation-can be aligned.
46
Personal development
All team members reported that participation in the action research project changed
their classroom practice. For some, the changes went beyond the classroom. One
administrator told us the project had helped teachers think of themselves as actors
outside their own classroom. They saw they had knowledge and understanding to
contribute to solving program issues and, by extension, issues of concern to the field
of adult education. Several have presented at state conferences and written for
newsletters. Although we cannot attribute their activism to the action research
project, it seemed to support it.
One team member attributes a major change in her life-returning to

college-to her participation in the project. She speaks of the project as changing
her life because it convinced her that she could further her own education.
Summary: What Did We Learn?
From the Documenting Outcomes action research project, we have learned:
Local adult basic education programs can develop documentation processes

usehl for planning and assessing their work
More work is needed at the local, state, and national levels before locally
developed documentation processes can be used for performance accountability
systems, such as those required by the Workforce Investment Act
Although practitioners value participation in action research, such factors as the

limited paid time for teachers typical of adult education in most states make it
difficult
Action research is effective for professional and program development in adult

basic education
47
5 6.
CHAPTER 6: IMPLICATIONS AND FINAL THOUGHTS
Action research probably has most relevance for the programs in which it occurs.
But the action research described in this report also has wider implications for adult
basic education practice and policy. We have identified implications for professional
development, program improvement, and performance accountability.
Action research is a valuable professional development tool in adult education.
As they participated in this action research, the team members changed their
understanding about aspects of their practice. The work of understanding and
developing outcomes documentation created opportunities for reflections that led to
improved classroom practice. Developing processes to document changes in
learners’ lives meant talking to learners about topics and at a depth new to many
team members. They found they gained understanding of their students’ lives. The
process of identifying desired outcomes led to an increased instructional focus on
meeting learners’ goals and achieving the desired outcomes. Thinking through
program processes led to increased appreciation of how different program aspects-
goal setting, instruction, outcomes documentation-can be aligned. Team members
also gained a greater understanding of research and greater awareness of research as
a source of knowledge that might contribute to their work. Although some changes
the project teams reported might be specific to a focus on outcomes documentation,
the participants’ experiences indicate that action research in which the question and
methodologies are in part determined by others can serve as valuable professional
development.
The adult education literature has only begun to discuss action research as a
professional development tool, most extensively in Quigley and Kuhne’s 1997 New
Directions edition: Creating practical knowledge through action research: Posing
problems, solvingproblems, and improving daily practice. The adult education
literature supports using practitioner research (also referred to as practitioner inquiry)
as part of professional development (Drennon, 1994; Fingeret & Cockley, 1992;
Lytle, Belzer, & Reumann, 1992). Typically in practitioner inquiry, an individual
teacher identifies and investigates a question of concern to him or her. Teachers may
be part of a research group, but their question and investigation is usually their own.
Action research could extend these efforts by including group research on questions
identified by both programs and other entities (e.g., the EFF field research in which
the work of collecting data for EFF standards’ development and the accompanying
assessment framework is also changing teachers’ practice; Stein & Bell, 2001). In
K-12 educational literature, action research is recognized as an effective approach to
49
57
professional development (See, for example, Altricher, Posch, & Somekh, 1993;
Zeichner, 2001).
Recommendation
Those responsible for professional development in adult education should use action
research more extensively. By doing so, they can learn from the experiences
reported here, from the EFF work, and from the states where action research has
been used in professional development, particularly Pennsylvania and Tennessee
(see, for example, Quigley & Weirauch, 1997; Action Research Group on Learning
Disabilities of the .Center for Literacy Studies, 1994). To effectively improve
practice, teachers need to be paid for time spent in action research, the state system
needs to accept action research as professional development, and facilitation support
needs to be available.
Action research is a tool to help student set and monitor goals.
Students set particular goals and helped identify the particular outcomes that were
documented in this action research. In their research on learner persistence,
Comings, Parrella, and Soricone (2000) identified student goals and goal-setting as
important to supporting learner persistence in adult education. Involving students in
action research to identify goals and develop processes to document goal achieve-
ment may have a positive impact on student persistence.
Recommendation
Action research involving students in both setting and documenting achievement of

their goals should be pursued and evaluated as a tool to further learner persistence.
Action research can enhance processes for improving program quality.
Action research focusing on outcomes can contribute to building local program

quality by supporting systematic thinking about what the program does and why.
Establishing goals and examining outcomes focused the action research teams in this
project on the processes that lead from goals to outcomes. The cycle of observing,
planning, acting, and evaluating fits such approaches as the Baldridge National
Quality Program (National Institute of Standards and Technology, 1998) and Project
Equal, Pennsylvania's field-based program improvement initiative (Alamprese,
2000).
50
Recommendation
Local and state ABE administrators should encourage use of action research
approaches to improve program quality. Systematic processes of reflection to
identify areas that need improvement, combined with ongoing action and evaluation,
help keep a program focused on continuous improvement. Resources such as How
are we doing? (Bingman, 2001), a guide for local programs based on this action
research project, can be used to facilitate local inquiry into program improvement,
particularly when local programs have access to financial support for staff time.
Performance accountability could be supported and enhanced by action

research.
The Workforce Investment Act mandates “a comprehensive performance account-

ability system” (Section 212) that states are to use “to assess the effectiveness of
eligible agencies in achieving continuous improvement of adult education and
literacy activities.” In her review of performance accountability in adult basic
education, Juliet Merrifield (1998) recommended four principles that support
effective performance and accountability in adult literacy and basic education.
These may be summarized as:
0 Agree on performance by coming to consensus on goals and what is to be

measured as indicators of accomplishment of those goals
0 Develop mutual accountability relationships so communication and support

move in all directions within the system
0 Build performance capacity and be accountable so programs have needed

resources to meet goals and the capacity to measure achievement
Create new tools to measure performance so multiple measures are used and
instruction is not targeted at a few easily measured items
The action research conducted for this project addressed each of these
principles. The processes involved the teams in clarification of their goals and led to
consensus on at least some of the performances they hoped to measure. The action
research seemed to strengthen mutual accountability at the program level. Teachers
expressed a new understanding of students’ needs and a more focused effort to meet
particular needs. For students, identification of goals seems to deepen commitment
to their learning, which is also a finding of other studies (Comings, Parrella, &
Soricone, 2000). Documenting the activities in their lives that supported their goals
51
59
also built commitment. Research on ways to document outcomes built the capacity
of the teams to be accountable-that is, to document performance on at least some
goals-and new tools were created.
But local efforts such as those described in this report cannot by themselves
build a comprehensive performance accountability system. The work must extend to
the state and national level if the principles Memfield outlines are to be achieved.
For example, a more extensive and focused process of examination and consensus
building around goals involving all the programs in a region or state could undergird
a state performance accountability system. Mutual accountability should go beyond
teachers and students to include state and federal agencies. Systems of feedback and
accountability need to be put in place. For programs to have the capacity to truly be
accountable for both measuring and meeting goal achevement, the challenges
identified in this project, particularly limited staff time, will have to be addressed.
More extensive projects that involve more teams in a program or state have
the potential to build a system of accountability that integrates a variety of tools to
measure the performances that are recognized as most important. Local programs
can develop new ways to measure performance, but integrating these processes into
state reporting systems will require changes. The federal National Reporting System
and most state systems require standardized measures of only a few outcomes.
Increased flexibility on the part of state and federal policymakers is needed so that
locally developed processes for documenting a wider variety of outcomes can count
as measures for program accountability. And more complex and nuanced systems
must be developed and used to report the kinds of data collected by teams that take
part in projects such as this. Although national legislation focuses on economic
outcomes of adult education, learners have a wider variety of goals. Programs need
to have the ability to focus on these individual goals as well as nationally established
goals. As the EFF Assessment Framework takes shape, with a clearly identified
performance continuum for each of the 16 standards, local programs may be able to
use the EFF standards as the vehicle for identifying, assessing, and reporting student
goals and progress.
Recommendations
States should build consensus on the goals that are the basis of their performance
accountability systems by using participatory processes such as action research.
Action research should include some focus on locally defined goals as part of
consensus building because a focus on goals and measurement of goals’
52
achievement seem to build programs’ ability to implement performance

accountability systems.
State and federal resources should be committed to designing outcomes

measurements and reporting systems flexible enough to include a variety of goals
as well as rigorous enough to measure performance. These might include
performance-based assessment frameworks, such as that of EFF (Stein & Bell,
2001), and Web-based reporting systems that allow reporting of specific
evidence of goal achievement.
53
62
REFERENCES
Action Research Group on Learning Disabilities of the Center for Literacy Studies.
(1994). Ifonly I could ...read, write, spell: Identifiing and helping adults
who find learning dificult. Knoxville, T N : Center for Literacy Studies.
Adelman, C. (1997). Action research: The problem of participation. In R.

McTaggart (Ed.), Participatory action research: International contexts and
consequences (pp. 79-106). Albany: State University of New York Press.
Alamprese, J. (2000). Role of data in program management and accountability in

adult education. Harrisburg: Pennsylvania Department of Education.
Altricher, H., Posch, P., & Somekh, B. (1993). Teachers investigate their work: An
introduction to the methods of action research. London: Routledge.
Beder, H. (1999). The outcomes and impacts of adult literacy education in the
United States. Cambridge, MA: National Center for the Study of Adult
Learning and Literacy.
Bingman, M. B. (2000). Outcomes ofparticipation in adult basic education: The

importance of learners ’perspective. Cambridge, M A : National Center for
the Study of Adult Learning and Literacy.
Bingman, M. B. (2001). How are we doing? An inquiry guide for adult ‘education
programs. Cambridge, MA: National Center for the Study of Adult Learning
and Literacy.
Bingman, M. B., & Ebert, 0. (2000). “I’vecome a long way”: Learner-identified

outcomes of participation in adult literacy programs. Cambridge, MA:
National Center for the Study of Adult Learning and Literacy.
Bingman, M. B., Ebert, O., & Smith, M. (2000). Changes in learners ’ lives one
‘year after enrollment in literacy programs: An analysis from the longitudinal
study of adult literacy participants in Tennessee. Cambridge, MA: National
Center for the Study of Adult Learning and Literacy.
Cody, J., Ford, J., & Haywood, K. (1998). A story of improvement. Focus on
Basics, 2(C), 14-18.
55
62
Comings, J., Parrella, A., & Soricone, L. (2000). Persistence among adult basic
education students in pre-GED classes. Cambridge, MA: National Center for
the Study of Adult Learning and Literacy.
Drennon, C. (1994). Adult literacy practitioners as researchers. Adjunct ERIC

Clearinghouse for ESL Literacy Education, Washington, DC. (ERIC
Reproduction Service number EDO-LE-94-03)
Fingeret, H., & Cockley, S ; (1992). Teachers learning: An evaluation of ABE staff
development in Virginia. Dayton: Virginia Adult Educators Research
Network.
Flora, C., Flora, J., & Wade, K. (1996). Measuring success and empowerment. In
N. Walzer (Ed.), Community strategic visioningprocess (pp. 57-74). New
York: Praeger.
Hayes, E. (2000). Youth in adult literacy education programs. In J. Comings, B.

Garner, & C. Smith (Eds.), Annual review of adult learning and literacy, I
(pp. 74-1 10). San Francisco: Jossey-Bass.
Kuhne, G., & Quigley, A. (1997). Understanding and using action research in
practice settings. New Directions for Adult and Continuing Education, 73,
2340.
Lytle, S., Belzer, A., & Reumann, R. (1992). Invitations to inquiry: Rethinking staff
development in adult literacy education. Philadelphia: National Center on
Adult Literacy.
McTaggart, R. (1997) Guiding principles for participatory action research. In R.

McTaggart (Ed.), Participatory action research: International contexts and
consequences (pp. 25-44). Albany: State University of New York Press.
Merrifield, J. (1998). Contested ground: Performance accountability in adult basic

education. Cambridge, MA: National Center for the Study of Adult Learning
and Literacy.
National Institute of Standards and Technology. (1998). Malcolm Baldridge

National Quality Award: 1998 Education Criteria for Performance
Excellence. Gaithersburg, MD: Author.
56
63
Peters, J. (1997). Reflections on action research. New Directions in Adult and

Continuing Education, 73, 63-72.
Quigley, A. (1997). The role of research in the practice of adult education. New
Directions for Adult and Continuing Education, 73, 3-22.
Quigley, A., & Kuhne, G. (Eds.). (1997). Creating practical knowledge through
action research: Posing problems, solving problems, and improving daily
practice. New Directions for Adult and Continuing Education, 73.
Quigley, A., & Weirauch, D. (1997). Poking and prying with purpose: Action
research in Pennsylvania. MOSAIC: Research Notes on Literacy, 7( l),
1-2.
Stein, S . (2000). Equipped for the Future content standards: Nhat adults need to
know and be able to do in the 21”’ century. Washington, DC: National
Institute for Literacy.
Stein, S., & Bell, B. (2001). Teaching from strengths: A new research base. In L.
West, N. Miller, D. O’Reilly, & R. Allen (Eds.), Travelers tales: From adult
education to lifelong learning and beyond (pp. 375-378). Proceedings of the
3 1St annual conference of SCUTREA: The Standing Conference on
University Teaching and Research in the Education of Adults.
United Way of America. (1996). Measuring program outcomes: A practical

approach. Alexandria, VA.
Workforce Investment Act, Title 11. (1998). Public Law 105-220. August 7, 1998.
112 Stat. 936. Nashville, TN: Office of Adult and Community Education.
Young, M., Fleischman, H., Fitzgerald, N., & Morgan, M. (1995). National
evaluation of adult education programs. Washington, DC: U.S. Department
of Education.
Zeichner, K. (2001). Educational action research. In P. Reason & H. Bradbudry

(Eds.), Handbook of action research: Participative inquiry and practice
(pp. 273-283). London: Sage Publications.
APPENDICES
.......................................................................
APPENDIX 1: LISTOF PARTICIPANTS 61
REPORT.....................................
APPENDIX 2: KCALP STAGE1 DOCUMENTATION 63
APPENDIX 3: SAMPLE
FACILITATOR’S ......................................................
AGENDA 69
APPENDIX 4:SUMMARY OF RESPONSES FROM TEAMMEMBERS AT THE 11/19/99

REGIONAL
MEETING ................................................................................................... 71
APPENDIX 5: KNorr COUNTY
TEAMFAMILY
READING FORMS...............73
PROJECT
PLAN.........................................................................
APPENDIX 6: KNox COUNTY 75
59
65
APPENDIX 1:
List of Participants
b o x County Adult Literacy Program
Jane Knight
Jim Ford
Francis Hong
Ann Hudnall
Beth McConnell
Emily McDonald-Littleton
Janet Packard
Debbie Perrone
Renee Thompson
Marilyn Zaiko
Knott County Adult Learning Center
Shirley Asher
Betty Hall
Shelia Ann Jacobs
Jean Johnson
Glenna Short
Mount Rogers Regional Adult Education Program
Deborah Lowe Duty

Jerry Musick
Sue Rector
Rita Roper
Sherri Whitlock
Center for Literacy Studies
Beth Bingman
Brenda Bell
Dona1 Crosse
Olga Ebert
Rosemarie Mincey
61
66
APPENDIX 2:
KCALP Documentation Report
Evidence of How
Teacher Input Process output Outcomes How Collected Documented
Outcome
E. L. Where they’re at D Lessons in Learning Read paper to class Speaking to the Attendance Scheduled COPY
Whythey’re Skills Complete Learning group Student interviews and artifacts
there Writepaper Skills class Son came with the writing artifact collection Transcribe
How they feel - organize paper Complete what we parent to find out Observation Collect interviews
about being - discussions ask them to do about GED Interviews information/ Log-teacher
there D Interviewing Knowledgelunder- Used Plan, Do, Class observations on reflection
Student goals Before: what do you standing of Study, Act approach discussion occasion
Needs already know? skilldstepdgoal in CNA Training Other teacher Students
After: what do you Changes in student Showedother artifacts volunteer to share
know now? attitude: responsible, people how to apply
motivated, energized, tool she uses
confident Application outside
the classroom
B. M. Math and Graph the tests Frustration is less Used estimation in Explain Interviews Teacher
creative writing Math drills weekly Some are missing budgetinglgrocery process Weekly journal checklist on
Low math Writing about positive less but taking more shopping Workand writing students
scoreslwriting activities with math time/some less time checklist Informal (1.E.P)
skills - looking beyond (how but missing more Skills Bank discussion Reflection
to improve overall Taking noted list 1%
math skills) notebook - Notebook
Correcting work comprehension - graphs
themselves they explain it back - Tests
Organize a notebook putting work in to Journals
Subject areadchecklist prove the skill
Math rules in resource Skills Bank 80%
Interviewing/process/
goal - how can we
help?
68
APPENDIX 2 (continued)
Evidence of How
Teacher Input Process output Outcomes How Collected
Outcome Documented
D.P. Computer 0 Use beginning Increased independence Child’s Presidential 1 Skills Bank Students B Collection
knowledge of computer lesson Increased comfort level to List produced scores bring of
equipment plans computer Using the computer to B Teacher artifacts artifacts
Knowledge of Simulation of Increased sense of generate products for observations voluntarily filed in
computer secretarial duties responsibility everyday life 1 Keyboarding teacher
language and in relationship to Academic skills gained reports notebook
terms word processing Keyboarding ability/ skills P Student skills by student
Ability to use Students pursuing assessment
computer independent work sheets
0 Operating 0 Skills Bank B Artifacts
system (specific skills)
Ability to Typing tutor
navigate through 0 Teacher direction
software
0 Educational
objective
Individual needs
R. T. Hadn’t used Introduced 0 Students complete word Knowsthat Word Work Students’
math concepts to students to math problems by determining which measurement needs to problem produced work in
decorate a room formulas of “area” formula is needed to be used - be done so that she homework from lesson their
Hadn’t used a and “perimeter” Read Critically* doesn’t get the wrong Paper plans portfolio
tape measure to Had students use One student drew layout of size curtains Teacher New
measure a room tape measure to room and recorded the To be able to figure observation classroom
Hadn’t used measure room for measurements given by other out how much paint while
graph paper to area and perimeter students - Resolve Conflict and she will need to paint measuring
make a scale Students used Negotiate* her own house Students’
drawing of a graph paper and Scale drawings by each student May use the scale scale
room measurements to of new classroom, with ideas of drawing with her new drawings
make,scale model how the furniture should be apartment Student
room arranged - Work Together* 0 Use scale drawings to evaluations
(* EFF Standard) set up new classroom and opinions
69
Appendix 2 (continued)
Evidence of How
Teacher Input Process output Outcomes How Collected
Outcome Documented
J. K. Develop S.M.A.R.T. to develop Identified barriers to class Students: Observations Written work Collected
leadership skills goals listening Take more of business turned in samples of
Some students Readingactive listening Students wrote about control of meeting Learner self- work
had difficulty Discussed barriers to listening their panel and evaluation (written Teacher
understanding -listening process Two students participated education telecon- response to reflection
what teacher was -barriers to listening in teleconference and Take more ference questions) log of events
saying in Hold leadership business wrote about it control of Written
classroom meeting Students have to summarize children’s reflections
Some had Took Listening Inventory two things they heard at the education of students
problems with -introduced listening meeting Plan, problem Written
listening when process Participated in Volunteer solve, set homework
talking with -introduced barriers to Training workshop goals in Observations
children listening and strategies to (listened and made various adult of class
Things distracted compensate for them comments) roles using
them when they -introduced listening Student independent action strategies
tried to listen strategies in classroom and computer
Misunderstanding lab
came as a result Had students identify how
of poor listening active listening would help
skills them by using the process
writing about the
experience
J. F. Student releases Student and assessment Increased use of Skills Long-term Observations Teacher and student Written
Diagnostic coordinator review: Bank computer program progress in -classrooms review: reflections
approach to - educational objectives/ Classroom focus on educational -computer -educational SkillsBank
learning and diagnostic feedback educational objectives growth lab objectives giving assessments
development of - test and learning strategies Skills improvement Usedleamed Student diagnostic feedback TABE
better and newer Student informs teacher of Student reflections on strategies in comments -test and learning assessments
strategies for areas to work on testing weaknesses adult life and strategies Student
learning Teacher sets up class and New strategies on next areas reflections ABE class work and educational
Student’s poor computer lab work assessment computer lab work plan and
test and self- Teachers and students Progress with Skills Bank Teachersand goals
analysis analyzed their test taking levels students analyzed worksheet
strategies and skills their test-taking
strategies and skills
71 72
APPENDIX 3: Sample Facilitator’s Agenda
Documenting Outcomes for Learners and Their Communities

Action Research Team Meeting
Knott County Adult Learning Center
August 27,1999
Facilitator’s Agenda
Check In
Review project status

1. Rest of the fall developing and implementing documentation processes of
outcomes under broad goal of Support Child’s Education
2. CLS development of inquiry guide
3. November 19th meeting to share work and evaluate learning, processes, advise,
suggestions on using for continuous improvement
Report from August 16 meeting re: KY WIA documentation requirements-what do

they know now?
Report on my conversation with Sandy K.
Review work with parents
Report from team on meeting with parents. Is there still a commitment to focus on
parent role?
Have they identified additional outcomes?
If this hasn’t happened, see if it would be helpful to plan a parent session together.
We could assist or just help plan. It would be good for us to have this for our later
product.
If it has happened, what are the additional outcomes?
Documenting additional outcomes
Look at the outcomes (either determined with parents or some the team thinks are
likely). Eventually need to identify 4-5 to track.
What might be evidence of progress? Look at EFF standards for connections. How
could this evidence be collected? The Summer Reading Project (SRP) fits into this.
69
73
This “evidence of progress” are outputs. The reading to children was an output of
focusing on reading to children-and encouraged the reading.
What are the outcomes? For example, for the SRP outcomes might include:
Parents developing the habit of reading to children (evidence? documentation?)
Parents and children regularly using library (evidence? documentation?)
Improved reading skills for parents (evidence? documentation?)
Impacts of SRP and other work should include school success.
Summer Reading program data

Look at data display from June calendars. What do they note? Does it suggest any
changes in documentation or instructions?
Next steps
Either proceed with parent meetings or develop more documentation of progress and
outcomes processes/forms
Set implementation of process
Set evaluation
70
74
APPENDIX 4: Summary of responses from team members at the

11/19/99 regional meeting (N=13)
What has changed about 0 More focus on student goals and needs (10)
your instructional 0 More real life materials, situations (4)
practice and planning? Some changes in planning (3)
0 Integrating learning activities/processesaround EFFlTake
Responsibility for Learning (3)
Do you collect more or 0 More questions to individual students about their lives
different evidence/ (oral/written) (7)
itemdthings from Collecting evidence that students achieved specific
students? goals/outcomes (4)
What was the placehse 0 Basis/foundation/center of all our work (5)
of the EFF framework in 0 EFF slulls (7) and roles (4) are a part
what you did?
What were the things 0 No assessment in place, hard to document changes (7)
that made this difficult? 0 Student attendance (2)
71
75
APPENDIX 5: Knott County Team Family Reading Project Forms
Summer Reading Project

I Name:
ChildChildren’s Name Age Sex Attended School
Pre-Test Score: TABE Post-Test Score:
73
APPENDIX 5 (continued)
Knott Co. Adult Learning Center
EFF ParentIFamilv Member Role: Goal/Key Activities -

Foster informal education of chiIdredSupport children’s formal education
NAME: MONTH:
3. Good
Attendance
5. Meet with
Teacher
74
77
APPENDIX 6: Knox County Plan
Knox County Adult Literacy Program

Action Research Project
PLAN:
Question: How do programs capture changes in the lives of students? Do they

take responsibility for learning and how does it look in their daily lives?
Data to Collect:
Student writings (in a journal) or tape recordings of events that occur when they are
at home (not at school). They will answer the questions: What, When, Where, Why,
Who, How?
Students will use the EFF skill wheels to show skills they used. The journal and skill
wheels will be brought into class daily and discussed with the teacher.
Teachers will keep their own journals that reflect their observations of TRL in the
classroom.
Data will be collected daily for a four-week period.
Methodology:
Research will be conducted over a four-week period. All documentation will be
turned to Beth Bingman in January 2000.
We will use Bloom’s Taxonomy and the action words listed on the attached chart to
describe indicators of TRL. The process will be:
1. Each teacher will give students a pre-survey on TRL.
2. Teachers will select a model story that reflects the standard and use RWD to
present the story.
3. The teacher will teach a lesson on TFU and provide examples to students.
4. Teachers will work with students to get buy-in on this four-week project.
5 . Students will be ask to keep notes, write in a journal, or tape record events that
happen in their daily lives and report those back to the teacher each day. They
75
will be given 3-5 skill wheels to take home each day. When they report for
school in the mornings or evenings they will discuss events with their teacher,
turn in writings, and submit any evidence they choose and skill wheels that
indicate skills they used.
6 . Teachers will collect information daily. They will analyze and describe the
learning and link it back to TRL and/or other skills. This means teachers need to
listen and discuss events with students, document findings, and collect products.
Start out early recording observations and analysis in journals.
76
79
NCSALL’s Mission
The National Center for the Study of Adult Learning and Literacy (NCSALL) provides information
used to improve practice in programs that offer adult basic education, English for speakers of other
languages, and adult secondary education. In pursuit of this goal, NCSALL has undertaken research
in four areas: learner motivation, classroom practice and the teachingleaming interaction, staff
development, and assessment.
NCSALL conducts basic and applied research; builds partnerships between researchers and
practitioners; disseminates research and best practices to practitioners, scholars, and policymakers;
and works with the field of adult literacy education to develop a comprehensive research agenda.
NCSALL is a partnership of the Harvard Graduate School of Education, World Education,

Rutgers University, Portland State University in Oregon, and the Center for Literacy Studies at the
University of Tennessee in Knoxville. NCSALL receives funding from the U.S. Department of
Education’s Office of Educational Research and Improvement, National Institute for Postsecondary
Education, Libraries, and Lifelong Learning; the Wallace-Reader,s Digest Funds; the National
Institute for Literacy; and the Office of Vocational and Adult Education.
NCSALL’s Dissemination Initiative
NCSALL’s dissemination initiative focuses on ensuring that the research results reach practitioners,
administrators, policymakers, and scholars of adult education through print, electronic, and face-to-
face communication. NCSALL publishes research reports, occasional papers, research briefs, and
teaching and training materials; the quarterly journal Focus on Basics; and The Annual Review of
Adult Learning and Literacy, a scholarly review of major issues, current research, and best
practices.
For more information about NCSALL, to download free copies of NCSALL publications,
or to purchase bound copies, please visit:
http://ncsall.gse.harvard.edu
30
.d
NCSALL at World Education
44 Farnsworth Street
Boston, MA 02210
(617) 482-9485
http://ncsall.gse.harvard.edu
- National Educational Research Centers

t .
rz
w
0
I The Notiqnal Centerfor the Study of Adult Learning and Literocy (NCSALL) is o collaborative effort between the Horvord Graduate School of Educotion and World
Education. The University of Tennessee, Portlond State University, Rutgers University, and Brown University ore NCSALL's ortners. NCSALL is funded by the Educational Research
ond Development Centers Progrom,-Aword Number R309860002, 0 s administered by the 0ff;reofEducotionol Reseorcff and Improvement / Notional Institute on Postsecondary
Education, Libraries, and Lifelong learning, US. Deportment of Education
US. Department of Education
Ofice of Educational Research and Improvement (OERI)
National Libraly of Education (NLE)
Educational Resources Information Center (ERIC)
NOTICE
Reproduction Basis
This document is covered by a signed "Reproduction Release

0 (Blanket)" form (on file within the ERIC system), encompassing all
or classes of documents from its source organization and, therefore,
does not require a "Specific Document'' Release form.
This document is Federally-funded, or carries its own permission to

reproduce, or is otherwise in the public domain and, therefore, may
be reproduced by ERIC without a signed Reproduction Release form
(either "Specific Document'' or "Blanket").
EFF-089 (3/2000)

Eric Ed465868

Uploaded by

Copyright:

Available Formats

Eric Ed465868

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Eric Ed465868

Uploaded by

Copyright:

Available Formats

DOCUMENT RESUME

ED 465 868 CE 083 403

AUTHOR Bingman, Mary Beth

Reproductions supplied by EDRS are the best that can be made

Mary Beth Bingman

NCSALL Reports #20

National Center for the Study of Adult Learning and Literacy

CHAPTER 6: IMPLICATIONS AND FINAL THOUGHTS .............................. 49

OUTCOMES PROJECT ...............................

2: SUMMARY OF STAGE 2 TEAMACTIVITIES

ARTIFACT 2: LOGIC MODELFRAMEWORK BY PROGRAMS ......................

ARTIFACT 3: NOTESFROM FIRSTVIRGINIA TEAMMEETING.2/1/99 .......................... 19

ARTIFACT 4: THEKENTUCKY DOCUMENTATION ..........................................

ARTIFACT LOG. 2/26/99 .................................................

ARTIFACT 6: REFLECTIONS FROM A FACILITATOR AFTER THE APRIL 1999

ARTIFACT 7: PLANNING DONEON WHITEBOA RD ........................................................ 29

ARTIFACT 8: FROMTHE OVERHEAD USED BY THE KNOTT COUNTY TEAM AT THE

We would also like to thank Ernestine Scott, of the Virginia Department of

Gary Kuhne, of Pennsylvania State University; Peter Waite, of Laubach

Finally, our thanks to Dona1 Crosse and Rosemarie Mincey, colleagues at

Documentation efforts focused on particular aspects of students’ lives that the

Although the main purpose of this research was to contribute to under-

Local adult basic education programs can develop documentation processes

Action research is an effective approach for professional and program

Those responsible for professional development in adult education should use

On the state and federal levels, resources should be committed to designing

Administrator: Seeing what we do in documentation really ties together our

Two major national initiatives, the National Reporting System ( N R S ) and

0 Demonstrated improvements in literacy skill levels in reading, writing, and

Placement in, retention in, or completion of postsecondary education,

Receipt of a secondary school diploma or its recognized equivalent

Sixteen Content Standards that provide specific and measurable statements of

EFF has been developing an assessment fiamework to define performance

The CLS and NCSALL efforts to address assessment of the impact

A policy paper examining performance accountability in adult education

A paper reviewing previous outcomes studies in adult literacy education

A policy paper based on the findings (Bingman, 2000)

The remainder of this report includes a methodology chapter briefly dis-

Action research is an approach to research and development grounded in practice. In

In the same volume, Creating Practical Knowledge Through Action

In describing participatory action research, McTaggart (1997) emphasized

Concern on the part of academics and practitioners (workers) with under-

Studying and changing the discourse and practice, including distribution of

Using the project to both change practice and produce knowledge

Beginning with the subjective experiences of the participants

McTaggart described projects that began as small cycles of planning, action,

The Documenting Outcomes Project

The methodology in action research is not necessarily predetermined. Its

Examining various frameworks that integrate documentation into program work

Developing and implementing new approaches

Reflecting on the results

Understanding the situation and clarifying the questions

Examining frameworks, particularly EFF, that could structure action to document

Developing and implementing new approaches

CLS staff serving as facilitators and documenters and teams of practitioners

Table 1: Overview of the Documenting Outcomes Proiect

Fall Became EFF field site

The Participating Teams

In consultation with state adult basic education staff in Kentucky and