Lesson 4.

2: Designing and Evaluating Portfolio

Assessment in the Classroom

Learning Outcomes
At the end of this lesson, the learners are expected to:
1. Elaborately discuss portfolio assessment as an evaluation tool.
2. Discover the usefulness of portfolio assessment.
3. Improve students’ portfolio using the strengthening evaluation guide.
4. Constructively use portfolio assessment in analyzing and reporting data.

Motivation Question
How can portfolio assessment promote a student’s abilities?

Discussion
WHAT IS PORTFOLIO ASSESSMENT?
In program evaluation as in other areas, a picture can be worth a thousand words. As an evaluation
tool for community-based programs, we can think of a portfolio as a kind of scrapbook or photo album that
records the progress and activities of the program and its participants, and showcases them to interested
parties both within and outside of the program. While portfolio assessment has been predominantly used in
educational settings to document the progress and achievements of individual children and adolescents, it
has the potential to be a valuable tool for program assessment as well.
Many programs do keep such albums, or scrapbooks, and use them informally as a means of
conveying their pride in the program, but most do not consider using them in a systematic way as part of
their formal program evaluation. However, the concepts and philosophy behind portfolios can apply to
community evaluation, where portfolios can provide windows into community practices, procedures, and
outcomes, perhaps better than more traditional measures.
Portfolio assessment has become widely used in educational settings as a way to examine and
measure progress, by documenting the process of learning or change as it occurs. Portfolios extend beyond
test scores to include substantive descriptions or examples of what the student is doing and experiencing.
Fundamental to "authentic assessment" or "performance assessment" in educational theory is the principle
that children and adolescents should demonstrate, rather than tell about, what they know and can do (Cole,
Ryan, & Kick, 1995). Documenting progress toward higher order goals such as application of skills and
synthesis of experience requires obtaining information beyond what can be provided by standardized or
norm-based tests. In "authentic assessment", information or data is collected from various sources, through
multiple methods, and over multiple points in time (Shaklee, Barbour, Ambrose, & Hansford, 1997). Contents
of portfolios (sometimes called "artifacts" or "evidence") can include drawings, photos, video or audio tapes,
writing or other work samples, computer disks, and copies of standardized or program-specific tests. Data
sources can include parents, staff, and other community members who know the participants or program, as
well as the self-reflections of participants themselves. Portfolio assessment provides a practical strategy for
systematically collecting and organizing such data.

PORTFOLIO ASSESSMENT IS MOST USEFUL FOR:

1. Evaluating programs that have flexible or individualized goals or outcomes. For example, within a
program with the general purpose of enhancing children's social skills, some individual children may
need to become less aggressive while other shy children may need to become more assertive. Each
child's portfolio assessment would be geared to his or her individual needs and goals.
 2. Allowing individuals and programs in the community (those being evaluated) to be involved in their
own change and decisions to change.
3. Providing information that gives meaningful insight into behavior and related change. Because
portfolio assessment emphasizes the process of change or growth, at multiple points in time, it may
be easier to see patterns.
4. Providing a tool that can ensure communication and accountability to a range of audiences.
Participants, their families, funders, and members of the community at large who may not have much
sophistication in interpreting statistical data can often appreciate more visual or experiential
"evidence" of success.
5. Allowing for the possibility of assessing some of the more complex and important aspects of many
constructs (rather than just the ones that are easiest to measure).
PORTFOLIO ASSESSMENT IS NOT AS USEFUL FOR:
1. Evaluating programs that have very concrete, uniform goals or purposes. For example, it would be
unnecessary to compile a portfolio of individualized "evidence" in a program whose sole purpose is
full immunization of all children in a community by the age of five years. The required immunizations
are the same, and the evidence is generally clear and straightforward.
2. Allowing you to rank participants or programs in a quantitative or standardized way (although
evaluators or program staff may be able to make subjective judgements of relative merit).
3. Comparing participants or programs to standardized norms. While portfolios can (and often do)
include some standardized test scores along with other kinds of "evidence", this is not the main
purpose of the portfolio.

USING PORTFOLIO ASSESSMENT WITH THE STRENGTHENING EVALUATION GUIDE

Tier 1 - Program Definition

Using portfolios can help you to document the needs and assets of the community of interest.
Portfolios can also help you to clarify the identity of your program and allow you to document the "thinking"
behind the development of and throughout the program. Ideally, the process of deciding on criteria for the
portfolio will flow directly from the program objectives that have been established in designing the program.
However, in a new or existing program where the original objectives are not as clearly defined as they need
to be, program developers and staff may be able to clarify their own thinking by visualizing what successful
outcomes would look like, and what they would accept as "evidence". Thus, thinking about portfolio criteria
may contribute to clearer thinking and better definition of program objectives.
Tier 2 - Accountability
Critical to any form of assessment is accountability. In the educational arena for example, teachers
are accountable to themselves, their students, and the families, the schools and society. The portfolio is an
assessment practice that can inform all of these constituents. The process of selecting "evidence" for
inclusion in portfolios involves ongoing dialogue and feedback between participants and service providers.
Tier 3 - Understanding and Refining
Portfolio assessment of the program or participants provides a means of conducting assessments
throughout the life of the program, as the program addresses the evolving needs and assets of participants
and of the community involved. This helps to maintain focus on the outcomes of the program and the steps
necessary to meet them, while ensuring that the implementation is in line with the vision established in Tier
1.
Tier 4 - Progress Toward Outcomes
Items are selected for inclusion in the portfolio because they provide "evidence" of progress toward
selected outcomes. Whether the outcomes selected are specific to individual participants or apply to entire
communities, the portfolio documents steps toward achievement. Usually, it is most helpful for this
selection to take place at regular intervals, in the context of conferences or discussions among participants
and staff.
Tier 5 - Program Impact
One of the greatest strengths of portfolio assessment in program evaluation may be its power as a
tool to communicate program impact to those outside of the program. While this kind of data may not take
the place of statistics about numbers served, costs, or test scores, many policy makers, funders, and
community members find visual or descriptive evidence of successes of individuals or programs to be very
persuasive.

ADVANTAGES OF USING PORTFOLIO ASSESSMENT

❖ Allows the evaluators to see the student, group, or community as individual, each unique with its own
characteristics, needs, and strengths.
❖ Serves as a cross-section lens, providing a basis for future analysis and planning. By viewing the total
pattern of the community or of individual participants, one can identify areas of strengths and
weaknesses, and barriers to success.
❖ Serves as a concrete vehicle for communication, providing ongoing communication or exchanges of
information among those involved.
❖ Promotes a shift in ownership; communities and participants can take an active role in examining
where they have been and where they want to go.
❖ Portfolio assessment offers the possibility of addressing shortcomings of traditional assessment. It
offers the possibility of assessing the more complex and important aspects of an area or topic.
❖ Covers a broad scope of knowledge and information, from many different people who know the
program or person in different contexts (e.g., participants, parents, teachers or staff, peers, or
community leaders).

DISADVANTAGES OF USING PORTFOLIO ASSESSMENT

❖ May be seen as less reliable or fair than more quantitative evaluations such as test scores.
❖ Can be very time consuming for teachers or program staff to organize and evaluate the contents,
especially if portfolios have to be done in addition to traditional testing and grading.
❖ Having to develop your own individualized criteria can be difficult or unfamiliar at first.
❖ If goals and criteria are not clear, the portfolio can be just a miscellaneous collection of artifacts that
don't show patterns of growth or achievement.
❖ Like any other form of qualitative data, data from portfolio assessments can be difficult to analyze or
aggregate to show change.

HOW TO USE PORTFOLIO ASSESSMENT

I. Design and Development
Three main factors guide the design and development of a portfolio: 1) purpose, 2) assessment criteria, and
3) evidence (Barton & Collins, 1997).

1) Purpose
The primary concern in getting started is knowing the purpose that the portfolio will serve. This decision
defines the operational guidelines for collecting materials. For example, is the goal to use the portfolio as
data to inform program development? To report progress? To identify special needs? For program
accountability? For all of these?
2) Assessment Criteria
Once the purpose or goal of the portfolio is clear, decisions are made about what will be considered success
(criteria or standards), and what strategies are necessary to meet the goals. Items are then selected to
include in the portfolio because they provide evidence of meeting criteria, or making progress toward goals.
3) Evidence
In collecting data, many things need to be considered. What sources of evidence should be used? How much
evidence do we need to make good decisions and determinations? How often should we collect evidence?
How congruent should the sources of evidence be? How can we make sense of the evidence that is
collected? How should evidence be used to modify program and evaluation? According to Barton and Collins
(1997), evidence can include artifacts (items produced in the normal course of classroom or program
activities), reproductions (documentation of interviews or projects done outside of the classroom or
program), attestations (statements and observations by staff or others about the participant), and
productions (items prepared especially for the portfolio, such as participant reflections on their learning or
choices) . Each item is selected because it adds some new information related to attainment of the goals.
Steps of Portfolio Assessment
Although many variations of portfolio assessment are in use, most fall into two basic types: process
portfolios and product portfolios (Cole, Ryan, & Kick, 1995). These are not the only kinds of portfolios in use,
nor are they pure types clearly distinct from each other. It may be more helpful to think of these as two steps
in the portfolio assessment process, as the participant(s) and staff reflectively select items from their
process portfolios for inclusion in the product portfolio.
Step 1: The first step is to develop a process portfolio, which documents growth over time toward a goal.
Documentation includes statements of the end goals, criteria, and plans for the future. This should
include baseline information, or items describing the participant's performance or mastery level at
the beginning of the program. Other items are "works in progress", selected at many interim points to
demonstrate steps toward mastery. At this stage, the portfolio is a formative evaluation tool,
probably most useful for the internal information of the participant(s) and staff as they plan for the
future.
Step 2: The next step is to develop a product portfolio (also known as a "best pieces portfolio"), which
includes examples of the best efforts of a participant, community, or program. These also include "final
evidence", or items which demonstrate attainment of the end goals. Product or "best pieces"
portfolios encourage reflection about change or learning. The program participants, either individually or in
groups, are involved in selecting the content, the criteria for selection, and the criteria for judging
merits, and "evidence" that the criteria have been met (Winograd & Jones, 1992). For individuals and
communities alike, this provides opportunities for a sense of ownership and strength. It helps to
show-case or communicate the accomplishments of the person or program. At this stage, the
portfolio is an example of summative evaluation, and may be particularly useful as a public relations
tool.
Distinguishing Characteristics
Certain characteristics are essential to the development of any type of portfolio used for assessment.
According to Barton and Collins (1997), portfolios should be:
1) Multi-sourced (allowing for the opportunity to evaluate a variety of specific evidence)
Multiple data sources include both people (statements and observations of participants, teachers or program
staff, parents, and community members), and artifacts (anything from test scores to photos, drawings,
journals, & audio or videotapes of performances).
2) Authentic (context and evidence are directly linked)
The items selected or produced for evidence should be related to program activities, as well as the goals and
criteria. If the portfolio is assessing the effect of a program on participants or communities, then the
"evidence" should reflect the activities of the program rather than skills that were gained elsewhere. For
example, if a child's musical performance skills were gained through private piano lessons, not through 4-H
activities, an audio tape would be irrelevant in his 4-H portfolio. If a 4-H activity involved the same child in
teaching other children to play, a tape might be relevant.
3) Dynamic (capturing growth and change)
An important feature of portfolio assessment is that data or evidence is added at many points in time, not
just as "before and after" measures. Rather than including only the best work, the portfolio should include
examples of different stages of mastery. At least some of the items are self-selected. This allows a much
richer understanding of the process of change.
4) Explicit (purpose and goals are clearly defined)
The students or program participants should know in advance what is expected of them, so that they can
take responsibility for developing their evidence.
5) Integrated (evidence should establish a correspondence between program activities and life
experiences)
Participants should be asked to demonstrate how they can apply their skills or knowledge to real-life
situations.
6) Based on ownership (the participant helps determine evidence to include and goals to be met)
The portfolio assessment process should require that the participants engage in some reflection and self-
evaluation as they select the evidence to include and set or modify their goals. They are not simply being
evaluated or graded by others.
7) Multipurposed (allowing assessment of the effectiveness of the program while assessing performance
of the participant).
A well-designed portfolio assessment process evaluates the effectiveness of your intervention at the same
time that it evaluates the growth of individuals or communities. It also serves as a communication tool when
shared with family, other staff, or community members. In school settings, it can be passed on to other
teachers or staff as a child moves from one grade level to another.

Analyzing and Reporting Data

As with any qualitative assessment method, analysis of portfolio data can pose challenges. Methods
of analysis will vary depending on the purpose of the portfolio, and the types of data collected (Patton, 1990).
However, if goals and criteria have been clearly defined, the "evidence" in the portfolio makes it relatively easy
to demonstrate that the individual or population has moved from a baseline level of performance to
achievement of particular goals.
It should also be possible to report some aggregated or comparative results, even if participants have
individualized goals within a program. For example, in a teen peer tutoring program, you might report that
"X% of participants met or exceeded two or more of their personal goals within this time frame", even if one
teen's primary goal was to gain public speaking skills and another's main goal was to raise his grade point
average by mastering study skills. Comparing across programs, you might be able to say that the participants
in Town X on average mastered 4 new skills in the course of six months, while those in Town Y only mastered
2, and speculate that lower attendance rates in Town Y could account for the difference.
Subjectivity of judgements is often cited as a concern in this type of assessment (Bateson, 1994).
However, in educational settings, teachers or staff using portfolio assessment often choose to periodically
compare notes by independently rating the same portfolio to see if they are in agreement on scoring (Barton
& Collins, 1997). This provides a simple check on reliability, and can be very simply reported. For example, a
local programmer could say "To ensure some consistency in assessment standards, every 5th portfolio (or
20%) was assessed by more than one staff member. Agreement between raters, or inter-rater reliability, was
88%".
There are many books and articles that address the problems of analyzing and reporting on qualitative
data in more depth than can be covered here. The basic issues of reliability, validity and generalizability are
relevant even when using qualitative methods, and various strategies have been developed to address them.
Those who are considering using portfolio assessment in evaluation are encouraged to refer to some of the
sources listed below for more in-depth information.