!

9u7

'Geicl
to Selecting
Instructional Materials
for Mathematics Education

>The Association of State Supervisors of Mathematics

The National Council of Supervisors of Mathematics
Montana
rontana state
Slate LiDrary
Librai

IlllllillllWllllllill
Guide

to Selecting

Instructional Materials

for Mathematics
Education
 The Association of State Supervisors of
Mathematics

The National Council of Supervisors of

Mathematics

Permission to photocopy material from the Guide to Selecting

Instructional Materials for Mathematics Education is granted for
educational purposes. Permission must be sought for commercial use of
content from this publication, when the material is quoted in advertising,
when portions are used in other publications, or when charges for copies
are made. Fall 1993.
 Foreword

The is critical to the implementation of

selection of appropriate instructional materials
effective mathematics instruction. Available materials are in a state of evolution
influenced largely by expanding technology. Educators are no longer limited to the use of
textbooks alone. To keep abreast of the changing paradigm for mathematics instruction,
the Association of State Supervisors of Mathematics (ASSM) and the National Council
of Supervisors of Mathematics (NCSM) established a joint Instructional Materials
Committee to prepare this set of guidelines for the selection of instructional materials in
mathematics. Support for the committee was provided through a grant from the Carnegie
Foundation.

We would like to thank committee members for their efforts over the past year in
developing this document and the members of ASSM and NCSM who extensively
reviewed earlier drafts.

We hope that the Guide will help you improve mathematics education by providing
assistance with the selection of instructional materials.

Charles Allan L. Carey Bolster

President ASSM President NCSM

Committee Members

ASSM NCSM
Joan Akers (California) Ann Beyer (Michigan)
Bill Hopkins, Chair (Texas) Betty McDaniel (South Carolina)
Brenda Parnell (West Virginia) Richard Seitz (Montana)
Linda Pledger (Alabama)
Fernand Prevost (New Hampshire)
 Contents

Introduction 1

Use of this Document 1

Implications of the Standards for Instructional Materials 2

Determining the Criteria for Evaluating Instructional Materials 4

The Evaluation Instrument 4

Analytic and Holistic Evaluation 4

Sample Criteria for Evaluating Instructional Materials 6

Category 1 Mathematical Content
: 7
Category 2: Organization and Structure 8
Category 3: Student Experiences 8
Category 4: Teacher's Role 9
Category 5 Assessment
: 10

Establishing a Review and Selection Committee 11

Committee Membership 11

Professional Integrity 11

Training for the Committee 11

Evaluating and Selecting Instructional Materials 13

Identification of Instructional Materials 13

Publishers' Presentations 13

Committee Process 13

Consistency and Reliability Among Evaluators 14

Pilot Testing 15

Public Input and Challenge Process 15

Materials Selection 15

Other Issues 16

Next Steps 17

Appendix: Professional Standards for Selection and

Implementation of Instructional Materials 18

References 21

//
 Digitized by the Internet Archive
in 2012 with funding from
Montana State Library

http://archive.org/details/guidetoselecting63asso
 Introduction

The Curriculum and Evaluation Standards for School Mathematics and the Professional
Standards for Teaching Mathematics (hereafter referred to as the Standards) present a vision
for the reform of mathematics programs in our schools. They provide a broad framework
to guide reform during this decade. The development of the Standards is, however, only
the initial step in a long, complex process to change school mathematics. There are
many issues not specifically addressed in the Standards that will need solutions to
accomplish the changes needed in our school mathematics programs. To be addressed are
questions such as "How much and what kind of professional development will teachers
need?" and "What changes will be made in instructional materials so that they reflect the
envisioned program?" Also, "How will changes in technology impact the mathematics
program?"

This document addresses one critical issue - the selection of instructional materials for a
high-quality mathematics program. It is consistent with, and builds on, the NCTM
position statement, "Professional Standards for Selection and Implementation of
Instructional Materials." (See Appendix, page 18.) Many of the recommendations in this
document are taken directlyfrom the 1984 NCTM position statement, broadening the
concepts about possible formats and contents of instructional materials.

Traditionally, basal textbooks have been the primary component of mathematics

programs, with ancillary materials such as workbooks and manipulative materials being
somewhat separate. Since the textbook could stand alone, the student book could be the
basis for evaluation. This may no longer be true for new mathematics programs. For
example, a teacher's manual may be the primary component, manipulative materials and
technology may be fully integrated into the program, and a hard-bound student text may
not be included.

Use of this Document

This document is intended to provide guidance and support to school/district personnel
who are responsible for reviewing, evaluating, and selecting instructional materials in
mathematics. It is not meant to be comprehensive nor prescriptive. It may need to be
modified to meet local situations and needs.

This document is organized into the following four sections:

• Determining the Criteria for Evaluating Instructional Materials

• Establishing a Review and Selection Committee

• Evaluating and Selecting Instructional Materials

• Next Steps

Each section addresses a number of important aspects of the evaluation process. It is

those responsible for the process do careful planning. If possible, written
critical that
procedures, along with a time line, should guide the selection process so that all
personnel, as well as the public, can be well informed. Determining in advance how
decisions will be made or problems handled, can make the process run more smoothly.

The sample evaluation criteria and many of the procedural suggestions primarily were
written to be used for evaluation of materials for complete mathematics programs rather
than for supplemental materials. Evaluating materials for a year-long course or for one or
several grade levels is more complex than selecting materials for a portion of the program.
However, the criteria and suggestions can be adapted. For example,
if a school or district

is selecting a book of problem-solving school or district might decide to

activities, the
use the Student Experience category from the sample evaluation criteria and some of the
points from the Teacher's Role category and not use the Organization and Structure
category. Local decisions need to be made about what suggestions in this document are
appropriate for a particular situation.

Instructional for a complete mathematics program may have several

materials
components. This document suggests that all of the program's components such as
student written materials, teacher support materials, manipulative materials, technology
used in the program, and assessment materials be reviewed together. Evaluators should
consider how the various materials will be used by typical, effective teachers so that
students experience a high-quality mathematics program.

As and school educators use this document, they may find that many current
district
materials do not match the criteria presented herein. As a result, it may be tempting to
select materials that fall far short of what is desired. Rather than continuing to settle for
inadequate materials, mathematics educators should convey their desires and needs to
publishers. Publishers need to know that there is a market for innovative instructional
materials that reflect the Standards.

Implications of the Standards for Instructional Materials

The NCTM Standards are the basis for many of the recommendations in this document.
The Standards do not specifically describe instructional materials to be used in
mathematics classrooms. However, they make several fundamental implications about
instructional materials relevant to implementing programs based on the Standards.
that are
Some important points from the Standards are listed below. This followed by a list of
is

implications of the Standards for instructional materials. (See page 12 for a suggested
training activity for the evaluation committee.)

NCTM Standards: Important Points

• Educational goals for students must reflect the importance of mathematical

literacy.

• The NCTM Standards articulate five goals for all students:

•• learning to value mathematics

" becoming confident in one's own ability to do mathematics

•• becoming a mathematical problem solver
" learning to communicate mathematically
•• learning to reason mathematically

2
 The goal of teaching mathematics is to help all students develop mathematical
power.

What students learn is fundamentally connected with how they learn it.

All students can learn to think mathematically.

Teaching is a complex practice and hence not reducible to recipes or prescriptions.

Knowing mathematics is doing mathematics.

Some aspects of doing mathematics have changed in the last decade.

Changes in technology and the broadening of the areas in which mathematics is

applied have resulted in the growth and changes in the discipline of mathematics
itself.

The main purpose of evaluation is to help teachers better understand what their
students know and to make meaningful decisions.

Impl ications for Instructional Materials

Materials reflect current research in mathematics education.

Materials include, and consequently promote, an appreciation of the role of

mathematics as it relates to culture, history, science, and technology.

Materials provide numerous and varied experiences that encourage students to

develop trust in their own mathematical thinking.

Materials use an integrated problem-solving approach that reflects the spirit and
nature of the problem-solving process.

Materials provide ongoing opportunities for students to clarify, refine, and

consolidate their ideas, and to communicate through reading, writing, and
discussion.

Materials require students to make conjectures, gather evidence, and develop

arguments to support their reasoning.

Materials require students to use a variety of mathematical methods to solve

nonroutine problems.

Materials emphasize the development of conceptual understanding.

Materials emphasize how mathematics should be taught as well as what should be

taught.

Materials reflect high expectations for all students regardless of race, culture,
gender, religion, physical condition, and socioeconomic background.

Materials reflect an appropriate developmental sequence while allowing for teacher

flexibility.
Materials ensure active student participation in learning, creating, doing, and
evaluating mathematics.

Materials are current and reflect the diverse uses and applications of mathematics.

Materials reflect the changes in the nature of how mathematics is done.

Materials integrate technology and reflect the impact of technological advances.

Materials model multiple means of assessment that can be integrated with

instruction, assessing what students know, what students can do, and how students
think about mathematics.
 Determining the Criteria for
Evaluating Instructional Materials

The Evaluation Instrument

The used for evaluating and selecting instructional materials should be aligned
criteria
with the NCTM Standards and based on current research on the teaching and learning of
mathematics. The criteria might be similar to the example at the end of this section
(pages 6 - 10) or could be developed at the district or state level to also meet local needs
and requirements. Evaluation instruments developed locally should be done by a person
(or small group) with expertise in mathematics education, ideally by an individual who
also has experience teaching the grade levels or courses for which the instructional
materials will be selected.

A scoring scheme for the evaluation instrument will need to be established. This
document does not recommend a particular scoring scheme as there is no single correct or
best scheme. Questions that will need to be addressed in establishing a scoring scheme
include the following:

• Will holistic evaluation, analytic evaluation, or a combination of the two be used?

• What will be the rating scale? Will criterion be rated using a 4-point, a 5-point,
is between points
or other scale? Will evaluators be permitted to give a rating that
on the scale? Say a 3.5? What about a 3.9? (Allowing decimal ratings greatly
expands the number of points on a scale.)

• Will different criteria or categories be weighted differently? How should the

relative weights be established? Should the highest possible total of the weighted
scores be 100 or some arbitrary number?

• Should an "acceptable" score be set in advance and then other factors such as the
cost of various programs be considered before selecting a program? What if no
program is rated "acceptable?"

• Will individual evaluators be asked to score each program, or will the scores be
determined by the committee?

Analytic and Holistic Evaluation

All evaluations involve making judgments about what is being evaluated. In general,
there are two approaches to evaluation: analytic and holistic.

Analytic evaluation involves the use of many criteria, with a rating given for each
criterion. Then an overall score is determined based on a mathematical formula (which
may assign different weights to different criteria). A checklist is an example. Textbook
evaluation forms, in which a number of categories are identified with subpoints to be
independently rated, are analytical.

Holistic evaluation involves making a general judgment about a piece of material. The
assumption is that the whole is greater than the sum of the parts. With holistic scoring,
one is looking for the best fit, rather like determining the correct size when buying
clothes.
Holistic evaluation has been used to score writing samples, problem solving, and for
judging many things that are complex in nature.

Holistic evaluation does not have to be just one overall judgment or score. It can be done
in several categories. What distinguishes holistic from analytic scoring is whether the
evaluator is asked to make judgments on a whole or on the parts.

The sample criteria on pages 6-10 could be used as a holistic evaluation instrument by
asking evaluators to assign an overall score to a set of instructional materials or to assign
one score for each category and then weighting the categories for an overall score. The
same criteria could be used analytically by asking the evaluators to assign a score to each
of the items in the five categories and then weighting each item to provide a category
score and finally an overall score.

At first glance, analytic evaluation may appear to be more objective than holistic
evaluation since many more ratings are included in determining a total score. Yet the
question is, "Are many judgments, averaged together, about relatively smaller aspects of a
program, more indicative of the overall quality of a program than fewer judgments?" In
the sample criteria, each item is not a simple straight-forward statement that can be easily
checked off but requires sophisticated judgments based upon what is pervasive throughout
a set of materials. Even narrower statements require subtle judgments. For example,
suppose one criterion is, "The materials develop students' ability to use problem-solving
strategies." Judging this criterion is not cut and dried; it is not a matter of counting up
the number of lessons on "problem-solving strategies." In fact, an evaluator might rate
this criterion low even though a program asks students to frequently identify their
problem-solving strategies but expects students to learn and use the strategies in
procedural ways.

In holistic evaluation of instructional materials, an evaluator makes fewer judgments and

has the opportunity to look at how all the components work together. The example
criteriaon problem-solving strategies, stated above, is still important, but it is not
examined in isolation. It is used with other statements to judge the overall quality of a
program.

The following is a comparison of analytic and holistic evaluation:

Analytic Evaluation Holistic Evaluation

Many criteria are judged independently Judgment is made on the whole-the

and combined, using a mathematical whole is greater than the sum of its
formula to produce an overall score. parts. (Holistic ratings may be given
Ex: Looking at the trees. in several, but not numerous,
categories, rather than in a single
overall score.)
Ex: Looking at the forest.

Making many judgments can be tedious, An understanding of the overall picture

which can affect the judgments made. is required.The evaluator must
consider how the program as a whole
will function.
 .

The overall computed score may not Evaluators may overlook specific
match the holistic impression of the deficiencies in a program, or strengths
program, or ranks of competing in one area may inflate overall
programs may not line up with holistic judgment of the program,
impressions.

Sample Criteria for Evaluating Instructional Materials

Below is a sample set of criteria for evaluating instructional materials. The criteria are
based on the Curriculum and Evaluation Standards for School Mathematics and the
Professional Standards for Teaching Mathematics Much of the wording is directly from
.

these documents. The example below is only one model; it should be reviewed and
revised for use in a district's particular setting. To be used as an evaluation instrument, a
scoring procedure, including weightings for the different categories, should be added.

Since instructional materials for a program could be in a variety of formats, the sample
criteriahave not been organized into categories synonymous with the format. High-
quality instructional materials may be unintentionally penalized if the evaluation
instrument includes categories or criteria such as student textbook, teacher's manual, and
supplementary materials.

Rather, the categories offer the opportunity to examine the materials from different
perspectives. In this sample, the criteria are organized into the following categories.

1 Mathematical Content

2. Organization and Structure

3. Student Experiences

4. Teacher's Role

5. Assessment

Each category is used from a different perspective. The

to look at instructional materials
categories are not distinct; they overlap. For example, while the mathematical content
category describes what students study, content cannot be completely separated from how
students work. What should be considered is how all the aspects and the components of
the program work together so that students' experiences are of high quality.

When using the criteria, all the components of the instructional materials (that will be
purchased) should be considered as they typically will be used together in the classroom.
Many components of a program may be designed to be used together. The following are
examples of components that may be considered:
• teacher's guide • materials for parents (e.g., sample
letters to parents, suggestions for
• calculators/computers family projects)

• computer software • textbook

• manipulative materials • student activity sheets

• teaching materials,(e.g., overhead • laser discs

transparencies, posters, or bulletin
board materials) • video tapes

• distance learning materials • student assessment materials

• staff development materials • program assessment materials

Category 1: Mathematical Content

The mathematical content of the program reflects the curriculum Standards.

• Mathematics as problem solving is built into the program at all levels. The
mathe-matics is developed from problem situations. Situations are sufficiently
simple to be manageable but sufficiently complex to provide for diversity in
approach. They are amenable to individual, small-group, or large-group
instruction; involve a variety of mathematical domains; and are open and flexible
as to the methods to be used.

• Mathematics as communication is built into the program at all levels. Students

have many opportunities to use language to communicate their mathematical
ideas. The program asks students to explain, conjecture, and defend their ideas
orally and in writing. As students mature and develop, the program expects
students' mathematical communication to become more formal and symbolic.
Students are asked to form multiple representations of ideas, express relationships
within and among representation systems, and formulate generalizations.

• Mathematics as reasoning is built into the program at all levels. Throughout the
program, students are asked to explain and justify their thinking and to question
the statements of other students and the teacher. As students mature, the program
asks students to do both inductive and deductive reasoning. In Grades 9-12, the
program expects mathematically mature students to use informal and formal
arguments to support conclusions.

• Mathematical connections are clear in the program; the program approaches mathe-
matics as a whole. Concepts, procedures, and intellectual processes are interrelated
through specific instructional activities designed to connect ideas and procedures
among different mathematical topics, with other content areas, and to life
situations.

• The program is comprehensive and includes the mathematics content emphasized

in the Standards at each level.

Grades KA Grades 5-8 Grades 9- 12

Estimation Number and Number Algebra
Number Sense and Numeration Relationships Functions
Concepts of Whole Number Number Systems and Number Geometry from a Synthetic
Operations Theory Perspective
Whole Number Computation Computation and Estimation Geometry from an Algebraic
Geometry and Spatial Sense Patterns and Functions Perspective
Measurement Algebra Trigonometry
Statistics and Probability Statistics Statistics
Fractions and Decimals Probability Probability
Patterns and Relationships Geometry Discrete Mathematics
Measurement Conceptual Underpinnings of
Calculus
Mathematical Structure

10
Category 2: Organization and Structure

The program is organized into cohesive units, multiday lessons, and worthwhile tasks.

• The program is organized into units, modules, or other structure so that students

have sufficient time to explore and investigate in-depth major mathematical ideas.
The units or modules include lessons, activities, and projects that are multiday,
emphasize the connections between mathematical concepts, and promote the
attainment of several, rather than just one, instructional objectives.

The program asks students to work on worthwhile mathematical tasks. The tasks
do not separate mathematical thinking from mathematical concepts or skills; they
capture students' curiosity and invite them to speculate and to pursue their
hunches. Many tasks in the program can be approached in more than one
interesting and legitimate way; some have more than one reasonable solution.
The tasks require that students reason about different strategies and outcomes,
weigh the pros and cons of alternatives, and pursue particular paths.

• The instructional materials incorporate calculators and computers and other techno -
logy into the program as tools for students to use to do mathematics. The
program is designed with the expectation that calculators are available to all
students at all times and that all students have access to a computer for individual
and group work.

• The program is appropriate for all students. All students are expected to encounter
typical problem situations related to important mathematical topics. All students
are expected to experience mathematics in the context of the broad, rich curriculum
described in the K-8 Standards. However, the program recognizes that students
will differ in the vocabulary or notations used, the complexity of their arguments,
and so forth. For Grades 9-12, all students participate in the core program, with
explicit differentiation in terms of depth and breadth of treatment and the nature of
applications for mathematically mature students.

Category 3: Student Experiences

The program emphasizes students doing rather than memorizing mathematics. Students
are actively involved with mathematics.

• The program designed so that students are active learners. Students are encour-
is

aged to explore and investigate mathematical ideas. They are expected to read,
write, and discuss mathematics. The program asks students to conjecture, test, and
build arguments about a conjecture's validity. Students are asked to reason about
different strategies and outcomes, weigh the pros and cons of alternatives, and
pursue varied paths when working on tasks. Students are expected to work on
group and individual projects and assignments.

• Students are expected to construct their own understanding of mathematics. The

program recognizes that students approach a new task with prior knowledge and
encourages students to use natural language and informal procedures.

11
 • The program asks students to engage in mathematical discourse. The materials
ask students to talk with one another, as well as to respond to the teacher.
Students are expected to make public conjectures and reason with others about
mathematics. Students are asked to clarify and justify their ideas orally and in
writing.

• Students use manipulatives and technology to explore mathematical ideas, model

mathematical situations, analyze data, calculate numerical results, and solve
problems. Generally, students decide what tools are needed and when to use them.

• Students are expected to determine when they need to calculate in a problem and
whether they require an exact or approximate answer. Students are expected to
choose an appropriate procedure when calculating, whether it is using paper-and-
pencil, mental calculation, or a calculator.

• Students are expected to reflect on, make judgments about, and report on their own
behavior, performance, and feelings. Students are asked to do self-assessment on
selected aspects of their experiences as one method for evaluating student
performance and disposition.

Category 4: Teacher's Role

The instructional materials provide suggestions to teachers to assist them in shifting

toward the vision of teaching presented by the Standards.

• The instructional materials provide suggestions to teachers so that in tasks and

lessons teachers can help students to:

• work together to make sense of mathematics

- rely more on themselves to determine whether something is mathematically
correct

• reason mathematically

- learn to conjecture, invent, and solve problems

« connect mathematics, its ideas, and its applications to other topics within
mathematics and to other disciplines

• The instructional materials provide suggestions for teachers in initiating and

orches-trating mathematical discourse. The materials suggest questions that elicit,
engage, and challenge students' thinking. Teachers are encouraged to regularly
follow students' statements with, "Why?" and "What if?" Also, teachers should
ask students to explain their thinking and reasoning.

• The instructional materials provide assistance to teachers to facilitate learning by

all students. Suggestions are provided on how to use a variety of methods so that

allstudents can contribute to the thinking of the class. Students are expected to
express themselves in writing and pictorially, concretely and representationally, as
well as orally. The program encourages teachers to accept and respect the thinking
of all students by providing examples of how to probe students' thinking and
encourage students to follow and understand each others' approaches and ideas.

L2
 The instructional materials provide suggestions to teachers for establishing a class-
room learning environment focused on sense making. Teachers are provided
suggestions on how to:

•• structure the time so students can grapple with significant mathematical ideas
and problems
*• use physical space and material in ways that facilitates students' learning

•• assist students to work together collaboratively, as well as independently

• The instructional materials provide suggestions to teachers to help them reflect on

what happens in the classroom so that they can adjust or adapt their teaching
plans. Teachers are provided suggestions on how to observe, listen to, and gather
other information so they can assess and monitor student learning. Teachers also
are provided suggestions on how to examine the effect of the task, discourse, and
learning environment in promoting students' understanding of mathematics.

• The instructional materials provide suggestions for how parents can be involved
and kept informed about the program.

Category 5: Assessment

The student assessment in the instructional materials provides teachers with information
about what their students know and how they think about mathematics.

Student assessment is integrated into the instructional program. Assessment

activi-ties are similar to learning activities. Assessment activities examine the
extent to which students have integrated and made sense of information, whether
they can apply it to situations that require reasoning and creative thinking, and
whether they can use mathematics to communicate their ideas.

• Multiple means of assessment are used, informal as well as formal. Suggestions

are provided for assessing students, individually or in small groups, through
observations, oral and written work, student demonstrations or presentations, and
student self-assessment. The use of calculators, computers, and manipulatives are
built into assessment activities. Assessment is built into the instructional
materials as a continuous, dynamic, and often informal process.

• All aspects of mathematical knowledge and how they are interrelated are assessed
in the instructional materials.However, assessment is not of separate or isolated
competencies, although one aspect of mathematical knowledge might be
emphasized more than another in a particular assessment. Conceptual
understandings and procedural knowledge are frequently assessed through tasks that
ask students to apply information about a given concept in novel situations.

13
Establishing a Review and Selection Committee

Committee Membership
The school should establish a selection committee for mathematics instructional
district
materials. The committee shouldconsist of classroom teachers, mathematics specialists,
and administrators. Additional persons who may be considered to serve on the committee
include media specialists, computer consultants, special education representatives, Chapter
I teachers, and representatives of key interest groups such as parents, businesses,

partnerships, and the school board. Consideration should be given to balancing the
committee in terms of ethnic and socioeconomic background, gender, and grade level
representation. However, in an attempt to provide for this balance, districts should not
lose sight of the fact that members of the committee should be selected because of their
subject-matter and pedagogical knowledge and expertise.

Professional Integrity

The instructional materials selection process must be fair; the integrity of the process
must be ensured. Those involved in any way with instructional materials should not
benefit from their participation in the process. The district must ensure that there is no
conflict of interest or appearance of impropriety. Voting members of the selection
process should not have financial interest in any programs or materials being considered.
In addition, to ensure consistency and equity, guidelines for contact and interactions
between vendors and committee members should be established. For example, the
guidelines may address:

formal presentations by publishers to the committee

• the attendance at workshops seminars and other special events sponsored by

individual publishers

• contact between publishers' representative and individual members of the

committee

Training for the Committee

Prior to the actual review and evaluation of instructional materials, the committee needs
to participate in staff development activities in two areas:

• curricula and instructional practices that reflect current standards for mathematics
education

• the use of the criteria and the evaluation instrument

Staff instructional practices is important to provide

development on current curricular and
a common base for judging instructional materials. Some committee members such as
parents and administrators may not be familiar with the NCTM
Standards and other
mathematics education reform documents. Also, individuals may have quite different ideas
about what goes on in classrooms that are implementing recommended practices. A
presentation by an outside expert (if one is not available locally) on current research in

14
learning and teaching mathematics may be considered. Hands-on mathematics activities,
followed by discussions of the activities, can help committee members clarify how the
Standards might be reflected in instructional materials.

One possible training activity relates to the Implications of the Standards for Instructional
Materials on page 2 of this document. Have the committee list important ideas from the
Standards and then discuss and list the implications the Standards have for instructional
materials. The committee may want to go beyond the implications and brainstorm ways
the implications might be manifested in instructional materials. Encourage the
committee members not to limit their thinking to familiar kinds of materials but to
extend their thinking to alternative, but reasonable, possibilities.

The committee also needs to be trained on the use of the evaluation instrument so that all

will use it in a consistent The committee should have enough discussion to be

way.
reasonably comfortable that all members are reading and interpreting the criteria in the
same way and that a score means the same thing regardless of which committee member
determines it. A simulation of an evaluation, using the evaluation instrument and
materials that are not being considered for adoption, helps to establish consistency. (See
pages 14 - 15 for additional suggestions for promoting reliability and consistency among
evaluators.)

15
Evaluating and Selecting Instructional Materials {

The actual process of evaluating and selecting instructional materials can involve a
number of different aspects which are described below. Decisions about whether to
include all aspects and whether the suggestions are applicable to a particular situation
should be made locally.

Identification of Instructional Materials

A concerted effort should be made to identify all appropriate instructional materials to be

evaluated. Publishers and vendors frequently contact school systems to sell their
materials. Other publishers may be identified at national and regional NCTM (and
affiliate) meetings. Instructional materials developed by nonprofit organizations as well
as commercial publishers should be sought out and considered.

Publishers' Presentations

Providing an opportunity for publishers/authors/vendors to present their instructional

materials to the committee can be helpful. They can point out features of their programs
that may be overlooked by the reviewers. Guidelines about the presentations may help
ensure that all presentations address some of the same areas. For example, the publishers
may be asked to provide a brief overview of how their materials are organized and then
describe the content and instructional approaches. It may be desirable to have all

presenters address specific areas (e.g.. how their materials address mathematical
connections, how their programs develop student understandings of statistical concepts
through the grade levels). A question and answer period may be included at the end of the
presentation.

All presenters should have an equitable amount of allocated time. The time should be
sufficient so that the publishers can adequately present the materials, while considering the
overall amount of committee time for presentations. Fifteen minutes is too short to
adequately present a K-8 program (although it may be sufficient for supplementary
material). However, an hour may be too long for the attention span of committee
members, especially if several publishers have programs to present. To ensure equity, all
other access of publishers to the committee should be restricted during the review period.

Committee Process
made available for committee members
After the training period, sufficient time must be
to review and evaluate instructional materials, particularly if a number of different
programs are to be evaluated. Reviewers need adequate time, outside of committee
meetings, to become thoroughly familiar with each program prior to the time for
selection. Ideally, they should have some time before the publishers' presentations to
examine materials. Knowledge about the materials gained during the presentations may
be helpful as they review materials in more depth before meeting with other committee
members. The amount of time needed for individual review should be reasonable and |

based on the quantity of materials. In some instances, subcommittees might focus most

16
of their efforts on particular materials such as for a span of grade levels or a particular
course.

17
Some meetings might be structured so that the materials are examined in a variety of
ways. A few examples are:

• in-depth reviews of materials for a grade level (or particular course) by committee
members who have expertise at the grade level (or course)

• reviews across grade levels to see how concepts are developed over time

• comparisons of how different programs develop the same concept. For example,
at second grade, the committee might want to trace the development of place value
ideas. Different programs would be placed side-by-side so that the experiences
students will have in each may be compared.

Committee meetings should be Committee members should be encouraged

collaborative.
to share and discuss They should discuss the materials in relation to the
their findings.
evaluation criteria (and, therefore, indirectly, to the Standards). They should frequently
refer to illustrative examples in the materials to support their observations and
conclusions.

Consistency and Reliability Among Evaluators

When several evaluators individually judge instructional materials, holistically or

analytically, no assurance exists that evaluators will judge and rate materials using similar
standards. Also, no assurance can be given that an individual will apply the evaluation
criteria in the same way when rating materials from different publishers (or when rating
the same materials at different times). Of critical importance is the fairness of the
evaluation process. The ratings of different materials must not be affected by which
committee members review the materials or whether they reviewed the materials at a time
when they were tired. In this section, consistency refers to the extent evaluators apply the
criteria in the same way, and reliability refers to the extent each evaluator applies the
criteria in the same way to different materials or to the same materials at different times.

To promote consistency and reliability among the evaluators, the district can:

• use professionally active mathematics teachers and coordinators as a majority of

the evaluators

• establish acommon base of agreement through inservice education and discussion

on what constitutes a quality mathematics program. The NCTM Standards and
other recent documents should be used as resources. Inservice education is
particularly important for nonexpert evaluators. In-depth discussions help
evaluators clarify quality in their own minds. For example, a discussion could be
held around the question: "What might you see in instructional materials (or in a
classroom) that would be evidence that students will develop the ability to solve
problems?"

• use committee members with specific expertise or interests (e.g., businessmen and
women, parents, multicultural representatives, technology specialists, educators of
students with special needs) to examine specific aspects of the program

18
 • train the evaluators in the use of the evaluation instrument. Practice using the
instrument, if possible, with materials that will not be formally evaluated.
For
example, a simulated evaluation using materials from another country, old
textbooks, or alternative materials can help develop greater reliability among
evaluators if they discuss the reasons for their ratings.

• provide time for individuals to thoroughly review all instructional materials to be

evaluated and to share and discuss their reviews and finding in the committee.
Consider the possibility of evaluators working in pairs or teams.

• encourage evaluators to come to consensus, rather than voting individually. This

provides the opportunity for each evaluator to consider aspects that may have been
overlooked when a program was individually reviewed. The strategy paves the
way for a program to be successfully implemented.

• encourage sharing of and respect for all absence of consensus on

opinions. In the
materials, realize conscientious people will frequently disagree.Decisions about
the materials may be able to accommodate differing views. For example, schools
or teachers may be provided a number of choices of materials.

• encourage evaluators to justify their ratings, orally and in writing, and to include
examples to support their evaluation

Pilot Testing

Piloting instructional materials with students can provide useful information about how a
program will work. However, if not carefully thought out, piloting can result in teachers
developing a bias for (or against) products they pilot. Some suggested guidelines to
consider include the following:

• Conduct a pilot test only after a thorough evaluation of materials.

Conduct a pilot if several programs receive a top evaluation and only if the pilot
will provide information than cannot be obtained through the evaluation process.

Pilot similar components of competing programs. Pilot only those components

that will be considered for purchase.

• Distribute pilots evenly among different student populations and among teachers.
Perhaps, permit only committee members to pilot.

• Rotate programs that are piloted after a set period of time so that teachers have an
opportunity to use each one and to reduce the potential for bias.

Public Input and Challenge Process

Teachers and administrators not on the committee, parents, and the general public should
have an opportunity to review materials being considered for selection and to provide
input to the committee. Instructional materials being reviewed can be displayed in a
central location so that all interested parties have access to them.

19
The selection of instructional materials may produce complaints about the process or the
materials finally selected. A procedure for handling concerns or complaints should be
defined prior to the selection process.

Materials Selection

The committee should have the primary responsibility for selecting instructional
materials. Those who have not participated in the extensive review and evaluation process
should not have a determining influence. For example, voting by all teachers in a district
may be counterproductive to the selection process.

In the committee, ratings of the instructional materials using the evaluation instrument
can be done individually, in subgroups, or by the full committee. Ratings should be done
only after members of the committee have thoroughly discussed the materials and shared
observations. If no agreement is reached on the program to select, the committee might
recommend conducting a pilot or allowing schools to select from the top programs. If
committee members determine that no program meets the criteria, they might recommend
delaying the purchase of instructional materials.

The goal should be to identify the best possible program, not the program that is the
easiest to implement. Difficulties in implementation should be considered only if they
seem insurmountable since any adoption of new instructional materials will create some
problems of implementation.

The instructional materials recommended by the committee should be adopted unless

irregularities in the selection process are documented. Since the cost of instructional
materials is only a small fraction of any school's budget, the cost of these materials
should not be a factor. Potential financial savings do not justify the selection of less
desirable materials.

Other Issues

Districts and states may have specific requirements that will impact the selection process.
These need to be incorporated into the selection process from the beginning. If any
policies are inconsistent with the selection of quality instructional materials in
mathematics, every effort should be made so the potential conflicts can be resolved.

The committee may want to address some

issues, not specifically part of mathematics, as
they review instructional materials. (These issues may, or may not, be addressed in
district or state requirements.) One important issue is equity. Beyond the obvious
examination of materials for gender and ethnic diversity in illustrations and in problem
situations, the committee may want to consider how the materials provide support so that
all students can be successful in the program. For example, are suggestions made as to

how teachers can work with students whose primary language is not English? Are student
materials translated into other languages? Are options given so students so who wish to
delve more deeply into a topic can be challenged? Are students encouraged to work with
other students and to listen to and to value diverse points of view?

20
 Next Steps

The work does not end with the selection of a particular series or collection of materials.
The task of assisting teachers to implement a qualitymathematics program with new
instructional materials lies ahead and is the most important part of the process. An
inservice component should be part of a long-term commitment to the professional
development of all teachers of mathematics.

All change is even for those who desire it. Mathematics staff development for
difficult,
teachers who implement new materials should have begun long before the materials
will
are selected. Understanding current recommendations and their implications for classroom
practices is critical. In schools that are trying to implement the Standards, teachers will
welcome new instructional materials that are aligned with their efforts. Even so, it will
take additional time and effort for these teachers to become familiar and comfortable with
using new For teachers who are unaware of the national recommendations for
materials.
changes mathematics education, an ongoing inservice program that focuses on issues in
in
the Standards and other documents will be necessary for successful implementation of new
instructional materials.

Prior to implementation, teachers need specific information on new materials.

Information may address the following: the various components of a program (e.g.,
student materials, technology, manipulatives) and how they are integrated, how the
program is organized, the program's assessment philosophy, recommendations for
program management, etc.

At the same time, inservice education for new instructional materials often can be
integrated with the broader issues of mathematics reform. For example, some of the
information about the new materials might be delivered through hands-on activities that
model classroom activities. Following the activities, teachers can discuss the rationale for
the teaching approaches and how they reflect the Standards.

Teachers need ongoing support especially throughout the first year of implementation.
Grade-level meetings can provide teachers opportunities to share information on how they
use the materials to improve their mathematics program and to identify further areas that
need to be addressed.

21
 Appendix
Professional Standards for Selection and Implementation
of Instructional Materials
(NCTM Position Statement, approved by the Board of Directors, April 1984)

The content, organization, and style of instructional materials, particularly basal text-
books, are among the most important factors influencing outcomes of school
mathematics programs. Thus, the process of selecting new textbooks and ancillary
materials is a critical step in planning curricula and preparing for instructions. This
process includes setting criteria, screening materials, making choices, and implementing
the new program. Although teachers and students are vitally concerned with the quality of
the instructional materials they must use, the selection and preparation for use of those
materials are often constrained by inappropriate selection and implementation processes.

In order to improve curriculum and instruction in elementary and secondary mathematics

programs, the National Council of Teachers of Mathematics has adopted the following
standards for selecting instructional materials and implementing programs using those
materials. The Council urges compliance with these standards by all school districts.

Standard 1: The Review and Selection Committee

The entire process of instructional materials selection should be led by teachers and super-
visors with expertise and responsibility in mathematics education. Except in most
unusual circumstances the recommendations of the committee must be followed.

Membership on the Review and Selection Committee should include, but not necessarily
be limited classroom teachers representing the grades or courses for which instructional
to,

materials are being selected as well as teachers in grades or courses that precede or follow
the target courses; mathematicians or mathematics educators with knowledge of content,
curriculum, and pedagogical trends covering the full spectrum of K-12 mathematics and
beyond; and school administrators who have overall responsibility for supporting and
guiding the implementation of new programs.

Selection of committee members must be made with a person having

in consultation
expertise and responsibility in mathematics education. Whoever chosen should be a
is

full working member. Where the talents of a school or district mathematics staff cannot
fulfill the roles described above, appropriate outside consultants should be obtained.

Standard 2: Criteria for Selection of Instructional Materials

Criteria for selection must be defined and made public prior to the review process.

The criteria should identify the desired instructional approach as well as the mathematics
content. Criteria for selection should be based on program needs, which can be identified
by:

• evaluation of the present program, including analysis of student achievement,

attitudes and learning characteristics, and a survey of teacher opinions;

22
 • current research on the teaching and learning of mathematics;

• recommendations of professional organizations; and

• curriculum guidelines of local, state or provincial, and national governments.

Difficulties in implementation should be considered only if theyseem insurmountable.

Any adoption of new materials will create some problems of
instructional
implementation. However, the committee's goal should be the identification of the best
possible program, not the easiest to implement.

Copyright dates and formula-determined reading levels are generally inappropriate as

selection criteria. Although fashions in content and pedagogy change, a newer (or older)
copyright does not assure higher quality. Because of the large technical vocabulary asso-
ciated with mathematics, results of standard readability formulas should be used with
extreme caution.

Standard 3: The Review of Instructional Materials

The Review and Selection Committee should actively seek materials from a variety of
sources and evaluate the materials on the basis of the previously defined criteria.

The committee should consult the following sources to identify appropriate instructional
materials:

• critical review in professional journals;

recommendations of teachers and super visors in the district and in nearby districts
with similar characteristics and goals;

recommendations or authorizations of state or provincial committees where they

exist;

• independent information bases such as the National Diffusion Network; and

• representatives of all publishers of appropriate material.

Instructional materials developed by not-for-profit organizations as well as commercial

publishers should be considered. Whenever possible, committee representatives should
examine mate rials displayed at a national or regional NCTM meeting. Interviews with
members of the author team or a publisher's representative may also be helpful.

Once the materials for review have been collected, each should be evaluated for:

• content coverage and development of each major curricular strand;

• development of the full range of cognitive abilities, especially problem solving

and analysis;

• correlation of the content with local state or provincial, or national scope-and

sequence guidelines;

23
 • correlation of the content and instructional approach with the standards of
mathematics professional organizations;

• correlation between textual features and characteristics of special students;

• evidence of success based on publishers' field validation studies, implementation in

other districts, critical review in professional journals and reports, or local field
testing where possible; and

• representation of the diversity of national social, political, and economic life.

Since instructional materials comprise a small fraction of any school's budget, the cost of
the materials should not be a factor. Potential financial savings do not justify the
selection of less desirable materials. Where textbooks are accompanied by ancillary
materials, however, it is important to assess the extent to which successful use of the text
depends on such extra materials

Standard 4: Implementation of New Instructional Materials

The selection of new instructional materials must be only part of a careful curriculum
development process that includes appropriate in-service preparation for teachers and
supervisors and a formal evaluation of the new program's effectiveness.

In-service preparation should acquaint all teachers with the general program goals and the
emphasis for each topic every course. This emphasis should, where possible, be identi-
in
fied in terms of the appropriate number on instructional days to be devoted to each topic,
and the topics should be correlated with speci fie portions of the new instructional mate-
rials. In-service should provide the additional mathematics background and preferred
instructional approaches to the mathematics content.

Implementation should include periodic evaluation surveys of teachers' and students'

reactions to the materials as well as an analysis of student achievement. Results of these
evaluations should be used to evaluate the achievement of the program goals and plan for
ongoing modification of the program. Implementation is typically more successful when
such in-service support continues throughout the academic year.

Standard 5: Professional Integrity

The instructional materials selection process must not allow influences that compromise
the professional integrity of the process.

Any person connected with the process of selection and implementation of instructional
materials who has a financial interest in any program being considered must make that
interest known publicly at the outset. For the duration of the selection process those
involved must not accept gifts of travel, entertainment or materials from a vendor.

The instructional materials recommended by the Selection Committee must be adopted

unless irregularities in the selection process are documented.

24
 .

References

Blankenship, Glen. "How to Test a Textbook for Sexism." Social Education, April
1984, pp. 282-283.

Glatthorn, Allan A.Curriculum Renewal. Alexandria, VA: Association of Supervision

and Curriculum Development, 1987.

Mathematical Sciences Education Board, National Research Council. Counting on You:

Actions Supporting Mathematics Teaching Standards. Washington, DC:
National Academy Press, 1991.

Muther, Connie. "Textbook Adoption: A Process for Decision Making." A Workshop

Manual. Manchester, CT: Association for Supervision and Curriculum
Development, 1988.

. "ThePitfalls of Textbook Adoption-And How to Avoid Them." Alexandria,

VA: Association for Supervision and Curriculum Development, 1985.

. "What Every Textbook Evaluator Should Know." Educational Leadership

April 1985, pp. 4-8.

National Council of Teachers of Mathematics. The Addenda Series (A series of "teacher-

friendly books to translate the Standards into classroom practice). National
Council of Teachers of Mathematics, 1991, 1992.

. Curriculum and Evaluation Standards for School Mathematics. Reston, VA:

National Council of Teachers of Mathematics, 1989.

Position Statement on "Selecting Instructional Materials."

. Reston, VA:
National Council of Teachers of Mathematics, 1984.

. Professional Standards for Teaching Mathematics. Reston, VA: National

Council of Teachers of Mathematics, 1991.

National Research Council. Everybody Counts: A Report to the Nation on the Future of
Mathematics Education Washington, DC: National Academy Press, 1985.
.

Reshaping School Mathematics: A Philosophy and Framework for

Curriculum Washington, DC: National Academy Press, 1990.

Stenmark, Jean, ed. Mathematics Assessment: Myths, Models, Good Questions, and
Practical Suggestions. Reston, VA: National Council of Teachers of
Mathematics, 1991.

is