Program Assessment

Tool Kit:
A Guide to Conducting Interviews and Surveys

Created for University of Wisconsin-Madison Faculty and Staff

by LEAD Center Researchers
Sarah K.A. Pfatteicher, Dianne Bowcock, and Jennifer Kushner

Pilot Edition, Summer 1998

This work was supported by a grant from the University of Wisconsin-Madison Assessment Council to the
College of Engineering. Additional resources were provided by the Office of the Dean of the College of
Engineering and a Hilldale Foundation Grant from the UW-Madison Chancellor to the LEAD Center.

© 1998 LEAD Center. All Rights Reserved.

 FOREWORD

As UW-Madison moves forward with strategic planning and outcomes assessment activities
across campus, we are fortunate to have this Tool Kit from the LEAD Center. The Tool Kit
provides guidance in the use of interviews and surveys, two of the many important assessment
methods that are in wide use across campus. We encourage you to make use of this Tool Kit as
your program explores the use of interviews and surveys as a part of its assessment activities.

Please note that the authors of the Tool Kit invite you to provide feedback so they can improve it
in future editions.

For general information on other student outcomes assessment techniques, you are invited to the
UW-Madison Outcomes Assessment web site located at:
http://www.wisc.edu/provost/assess.html

Eden Inoway-Ronnie
Chair, University Assessment Council
 Section 1
INTRODUCTION

Introduction

This tool kit was developed for University of Wisconsin-Madison faculty and staff who are
involved in program assessment activities within their departments or majors. It is a complement
to a document prepared by the University Assessment Council1 which gives a broad overview of
assessment directions and accreditation requirements as well as an introduction to a variety of
assessment techniques. The Tool Kit has the goal of being a useful and user-friendly resource
framed around pertinent questions that might be asked by faculty or staff involved in program
assessment. It is an evolving document developed through the LEAD Center’s experience with
majors, departments, schools, and colleges at UW-Madison. In particular, the document
emphasizes the use of interviews and surveys because these are the primary tools selected by the
academic units with whom we have worked and these are the tools for which the LEAD Center
offers some expertise. We offer special thanks to the departments with whom we've worked for
providing us with the opportunity to learn more about program assessment.

Because the issues and assessment needs that each program faces are unique, we provide
checklists and open-ended worksheets primarily as ways to get started. We do not believe that
“one tool fits all.” We expect that you will select and modify items to fit your own needs. The
value of this tool kit will be measured by its usefulness to you and your colleagues involved in
program assessment.

The audience of the tool kit

This tool kit is primarily for UW-Madison faculty and staff who want to assess the quality and
effectiveness of their programs of study, curricula, degrees, or majors. Persons outside of UW-
Madison may also find the kit useful, but must recognize the assessment process outlined is placed
into the assessment environment at UW-Madison and designed with UW-Madison guidelines and
parameters in mind. We recognize that users of this kit will have varying degrees of experience
and types of expertise. This kit is intended to serve as a very basic introduction to program
assessment. As such, some of the information in it may not be useful to those with more
experience. The resources section at the end of the binder includes more sophisticated sources of
information for those who have prior experience with program assessment. It is our hope that this
kit will also “grow” with your program as you conduct and refine your own assessments.

1
“Using Assessment for Program Improvement: A Resource Manual for Academic Departments,”
January 1997, by the University Assessment Council and the Office of the Provost. University of
Wisconsin-Madison. The reader is referred to their website at
http://publications/provost/assess/manual.

1-1
How to use this “Tool Kit”

This tool kit was developed to guide you through a program evaluation by presenting key aspects
of the entire process. It is important to note that many sections are interrelated and need to be
understood in the context of the whole. As such, we do not advise taking individual sections out
of context but recommend that you review the Tool Kit from start to finish and then select those
sections or items that fulfill your program needs. Throughout this document we use the terms
assessment and evaluation interchangeably, although we recognize they are sometimes used as
separate and distinct terms.

The remainder of this kit is organized into nine sections. With the exclusion of Section 10 on
resources, each section provides a brief description of the topic, an overview to the section, and
worksheets or checklists that engage you in the process of implementing a program evaluation.
The worksheets provided in the kit are designed so that they can be copied as needed. They
appear on colored paper. In some of the sections we also include a “sample." We hope that over
time this kit will serve as a central location in which a program can store all of its evaluation-
related materials, including items such as program goals, evaluation records, and results, thus
keeping assessment information readily available for accreditation-related site visits and for future
evaluators.

The Tool Kit is organized by steps in the assessment process. These are:
• Section 2 suggests some "Questions for Assessment Planners" to ask as they begin the process
of program assessment.
• Section 3, “Setting Objectives,” includes a brief discussion of the value of setting educational
goals for a department or program and offers tips on defining objectives so as to make them
measurable as well as acceptable to a diverse group of faculty members.
• Section 4, “Selecting Assessment Tools,” discusses the pros and cons of various types of
assessment tools, from surveys and interviews to standardized exams and portfolios.
• Sections 5 and 6 discuss “Surveys” and “Interviews” in more detail, and cover issues such as
wording questions, gathering data, and analyzing responses. These sections also include
worksheets for tracking surveys and interviews as they are being conducted.
• Section 7 presents some of the “Ethical Dimensions” of conducting program assessment.
• Sections 8 and 9 describe various methods for “Reporting Results” and “Acting on Results;”
both emphasize making program assessment a practical, valued, and above all, useful
undertaking.
• Clearly, no tool kit can provide all of the information needed for conducting a program
assessment. Therefore, the final section of this tool kit lists a number of valuable resources
where you can find additional information on various stages of the assessment process. It
includes web sites, articles, books, and resources at UW-Madison.

We understand the process of evaluation to be cyclical, and as such it can become part of a plan
for continuous improvement. The diagram on the following page illustrates components of this
process and identifies the stages in the cycle where this tool kit provides resources.

1-2
 Developing and Implementing a Departmental Assessment Plan
For Programmatic Improvement

STEP 1
Define educational / programmatic goals &
objectives

Sections 2 & 3 of Tool Kit

STEP 6 STEP 2
Implement assessment plans & Determine instruments & methods
revise as needed needed for assessing student
achievement

Sections 5 - 9 of Tool Kit Section 4 of Tool Kit

STEP 5 STEP 3
Submit assessment objectives, Determine how results will be
methods, & timetable to school / disseminated & used for program
college academic Planning Councils improvement

Sections 2 & 8 of Tool Kit

STEP 4
Develop a timetable for assessment plan
implementation

Section 2 of Tool Kit

1-3
We value your feedback on this tool kit

Please let us know how this kit works for you! You can send us your feedback via e-mail or
campus mail to the address listed below. Specifically we are interested in what aspects of the kit
you find most useful, what seems unimportant, and what you would like to see included in future
editions. Since we see this as an evolving document that can be improved and expanded over
time, we appreciate any comments you can share that will help us improve this took kit for you
and others involved in program assessment.

To order additional copies or to comment, please contact:

Dianne Bowcock
LEAD Center
1402 University Avenue, 4th Floor
[email protected]

1-4
 Section 2
QUESTIONS FOR ASSESSMENT PLANNERS

There are two steps that you should take before you begin an assessment project:
• determine the parameters of the project (who’s doing it?, why?, for whom?, what resources
are available?)
• take stock of what information the department, college, and university already have available

The following pages present questions that the department members responsible for assessment
should consider. Keeping a written record of the answers to these questions will ease reporting to
administrators and accreditors, and will also facilitate handing off the assessment project to your
successors.

2-1
 WORKSHEET FOR PLANNING YOUR ASSESSMENT

1. Why are you conducting this assessment? Who or what initiated the assessment?
m Dean m Department Committee m Whole Department
m Department Chair m Accreditation Criteria m Other ____________

Briefly describe the impetus for this assessment: ___________________________________

__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________

2. Who will conduct the assessment? Internal or external evaluators? Whole department or a
committee? A combination of the above?
m External Evaluators m Whole Department m Department Committee
m Department Chair m Other ____________________

Name(s) and Title(s) or Affiliation(s) of Evaluator(s):

1. ________________________________________________________________________
2. ________________________________________________________________________
3. ________________________________________________________________________
4. ________________________________________________________________________
5. ________________________________________________________________________

3. Who will own the resulting data and reports?_______________________________________

4. How often will assessment be undertaken?

m One Month m One Semester m One Year
m One-Time Assessmentm On-Going Assessment (Annual, for instance)
m Other __________________________________________________________________

5. How much money and what resources (staff time, photocopying costs, etc.) are available?
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________

2-2
6. Given that a major goal of the assessment is determining if the department's goals relative to
earning degrees in the major are being met, what do you hope or expect to learn from the
assessment?
m To assess students’/alumni’s content knowledge.
m To assess students’/alumni’s general preparedness.
m To assess students’/alumni’s perceptions of program quality.
m To assess employers’ perceptions of program quality.
m Other
___________________________________________________________________

7. From whom do you plan to gather data?

m Undergrads m Graduating Seniors m Graduate Students m Alumni
m Undergrads Who Left the Major m Employers m Other _____________

8. For each of these individuals do you have:

m Accurate and Complete Address List m State/Country of Residence
m Gender m Age m Ethnicity
m GPA m Time and Credits to Graduation m # of Graduates Per Year
m Post-Graduation Plans (Graduate School or Employment) and Success Rate

9. Is there information that you are required to gather? ______________________________

________________________________________________________________________

10. At this present time, what kinds of information would be most beneficial? ____________
________________________________________________________________________
________________________________________________________________________

11. What types of information are you already gathering (through tests, exams, student records,
certifications, portfolios, existing questionnaires, standardized tests, etc.)? ____________
________________________________________________________________________
________________________________________________________________________

12. Will any of the information you currently gather be helpful to answer your present questions
and concerns?
_________________________________________________________________________
_________________________________________________________________________

13. Is there information that the university gathers that might be helpful in answering your
present questions and concerns? (E.g., through ISIS, student record data.)
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

2-3
14. Do you want information on the following categories (many of which may be out of the
control of your department)?
m Courses offered by other departments? m Specific courses and/or professors?
m Services offered at the college or university level such as advising or career services?
m Other
___________________________________________________________________

15. For your students or alumni as a group, do you have:

m approximate number of graduates of your program per year?
m typical post-graduation plans (graduate school, employment, military service, etc.)?
m success rate of graduates at obtaining employment or being accepted to graduate school?

16. For students and graduates in your field as a whole, do you have:
m approximate number of graduates in the U.S. per year?
m typical post-graduation plans of U.S. graduates?
m success rates of U.S. graduates?
m average starting salaries of U.S. graduates?
m typical career paths of U.S. graduates?

Of the data that is NOT readily available, which will be needed for the assessment? Where
can it be found? Who will find it? (See bibliography for a list of information resources.)
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________

17. Who is the audience for what the assessment finds?

m Department Chair m Department Committee m Whole Department
m Dean/Administration m Accrediting Agency m Employers
m Other ____________

18. What can and will you do with the results of the assessment?
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________

Given your answers above, become familiar with the types of evaluation tools (discussed in
Section 4) in order to choose a tool that best suits your present needs.

2-4
 Section 3
SETTING OBJECTIVES

Setting objectives for the program or major being assessed is a crucial, but often neglected,
starting point. If a department has well-written and well-thought out objectives, or goals,
measuring them becomes relatively easy. In essence, program objectives tell you, your fellow
faculty members and your students what your program values and what it expects students will
achieve by the time they graduate. For those departments who don't have agreed upon written
objectives already in place, this section provides some questions designed to stimulate ideas about
what your objectives might be, a checklist of what to do with those objectives, and some sample
program objectives from UW units. For departments who do have written objectives in place, this
section provides a stimulus to revisit and refine them. A bibliography in Section 10 refers to
sources of additional information on the usefulness of objectives and on how to go about
preparing them.

Tips about setting objectives

• Start brainstorming the department’s objectives before focusing on the accrediting agency’s
objectives – it’s important that the objectives be goals that your faculty can support.
• It may be useful to include the full faculty in brainstorming ideas for the objectives list, but
especially in large departments, it’s inefficient to have the full faculty involved in
“wordsmithing” the list – refer this duty to a small committee and return to the full faculty for
final approval.
• If a committee has already prioritized the objectives and refined the language, explain that to
the department, and don’t ask the full faculty for “comments” – ask them whether they can
live with the list. If they can’t, ask them to suggest alternatives.

3-1
 WORKSHEET FOR SETTING PROGRAM OBJECTIVES

m With fellow faculty, brainstorm a list of goals for graduates of your program.
• What makes your program distinct or different from its peers/competition?
• What makes your graduates marketable, valuable?
• How would you define/describe your graduates’ abilities?
• What will your graduates know or be able to do once they graduate that they couldn’t do
when they began your program?
• What would you tell a prospective student that your program can/will give him/her?

m Sort and prioritize the list of goals you produced.

• Which of the goals can the majority of the department faculty support?
• Which of the goals are the highest priorities?
#1 ______________________________________________________________________
#2 ______________________________________________________________________

m Compare the list with the objectives suggested or required by your accrediting
agency(ies) and with your college or school.
• Are any of the college’s or accreditors' objectives missing from your list? If so, can you
either add them or justify why you cannot?
• Are any of your objectives in conflict with those of the accrediting agency or your school?
If so, can you find a way to bring them in line with each other?

m Refine the language of each objective.

• Is each objective clear to someone who is not a specialist in your area, but is interested?
Would it make sense to a college sophomore or to a colleague in another department in
your college or school?
• Is each objective written in such a way as to be measurable? In other words, can you
think of at least one way to tell whether a student has reached that objective?

m Get departmental approval for the objectives.

• Can each faculty member in your department approve of the wording and ordering of the
objectives?

m Distribute the final list of objectives to both faculty and students on a regular basis.
• Are faculty encouraged, expected, or required to link their course objectives with these
department objectives?
• Are students encouraged, expected, or required to know what the program objectives are?

m Make a plan for updating the objectives on a regular basis.

3-2
SAMPLE PROGRAM OBJECTIVES
Department of Industrial Engineering

3-3
 SAMPLE PROGRAM OBJECTIVES

Goals for Botany Major

Department of Botany
University of Wisconsin-Madison

Broad education in plant biology

Graduates should be proficient with the knowledge covered by courses at all levels of
organization including molecular, cellular, organismal, and ecological.

Rigorous training as a plant scientist

Graduates should also be familiar with science related content outside the Botany Major.

Research Capability
Graduates should have the ability to formulate relevant biological questions, generate
hypothesis, devise experiments and interpret results.

Literacy
Graduates should be capable of communicating in clear scientific prose and of reading and
critically evaluating scientific literature.

Numeracy
Graduates should be capable of using quantitative methods of analysis and modeling.

Computer Literacy
Graduates should be capable of using computers in such areas as data processing, database
searches, and word processing.

Documentation
Graduates should be capable of establishing and maintaining a laboratory and/or field
notebook.

3-4
 Section 4
SELECTING ASSESSMENT TOOLS

There are a variety of ways to collect information to answer specific questions about your
program or major. We refer to these different data-gathering techniques as “assessment tools.”
The tools you select should be consistent with the type of information you seek. In some cases,
several tools may be needed to address a breadth of issues. Using a variety of tools is often
beneficial because the information gathered by some tools can be compared with information from
other tools (this allows one to confirm findings from one source with another source), or can
deepen your understanding about issues raised in one method. Some researchers purposely use a
variety of different tools to see if they reveal similar findings and to supplement and deepen
understanding about the information – this is one form of “triangulation.” One example of a
mixed-methods design is described below:
1. Use a focus group with subset of the sample to identify issues; then develop questionnaire
based on issues identified.
2. Distribute questionnaire to the entire sample.
3. After analysis of responses to questionnaire, conduct personal interviews with a subgroup
of the sample to explore issues in greater depth.

A primary UW-Madison Assessment Council report2 presents types of assessment divided into
“Direct” and “Indirect” Indicators of Learning. We highly recommend you review this report
which provides short summaries for each of these types of indicators.
• Direct Indicators of Learning are assessments used directly with students for the primary
purpose of assessing student learning, student achievement, and performance such as:
1. Capstone Course Evaluation
2. Course-Embedded Assessment
3. Tests and Examinations
4. Portfolio Evaluation
5. Pre-test/Post-test Evaluation
6. Thesis Evaluation
7. Videotape and Audiotape Evaluation of Performance
• Indirect Indicators of Learning are tools that do not assess student achievement per se, but
which provide important information about the program. Indirect indicators commonly
involve self-reports through surveys or interviews of students, alumni, or employers on their
experiences and perspectives about a program. These indicators include:
8. External Reviewers
9. Student Surveying and Exit Interviewing
10. Alumni Surveying
11. Employer Surveying
12. Curriculum and Syllabus Analysis

2
“Using Assessment for Program Improvement A Resource Manual for Academic Departments,”
(January 1997), The Assessment Council and the Office of the Provost, UW-Madison.

4-1
This tool kit focuses on the use of surveys and interviews as these have been the primary indirect
indicators used by departments at UW-Madison with whom we have worked. Some of the
advantages and disadvantages of a few selected types of tools are presented in the following table.
Your decision about the tools you select depends in part on your answers to questions in Section
2. the purpose of and available resources for conducting the assessment.

Assessment Tools - Advantages & Disadvantages

Tool Advantages Disadvantages
Survey Inexpensive. Can be quickly No control for misunderstood questions,
In-class administered to group. Best suited for missing data, untruthful responses.
Questionnaire simple and short questions. Can reach Not suited for exploration of complex issues.
every participant.
Survey Comparatively inexpensive. Can No control for misunderstood questions,
Mail Questionnaire attempt to reach participants over a missing data, untruthful responses.
wide geographical area. Not suited for exploration of complex issues.
Sometimes low response rate.
Takes times to receive responses.
Interview Relatively inexpensive. Best suited for Sometimes difficult to reach audience
Telephone relatively short and non-sensitive (especially alumni). Usually requires a sample
questionnaire or topics. Can attempt to reach of respondents. Not suitable for lengthy
interview participants over a wide geographical questionnaires and sensitive topics.
area. Respondents lack privacy and anonymity.
Interviewer requires skills.
Interview Interviewer can probe evasive answers Same as above. Often time-consuming to
Structured in-person and explore topics in depth, and clarify analyze due to volume and qualitative nature
Interviews questions. With good rapport, may of data.
obtain useful open-ended comments. Interviewee may distort information through
Usually yields richest data, details and recall error, selective perceptions, desire to
insights. Best if in-depth information please interviewer.
is wanted. Permit fact-to-face contact
with respondents.
Interview Useful to gather ideas, different Must take a sample. Limits time that each
Focus group viewpoints, new insights, improving individual has to contribute. Lacks individual
Interviews questions design. privacy, thus could reduce accuracy of results.
Tests/Standardized Provide comparative information Available instrument may be unsuitable for
certifications across populations. May be relatively population. Tests may contain unfairness or
easy to administer. bias. Developing and validating new project-
specific tests may be expensive and time
consuming. Test may be narrow in scope and
thus not allow student to demonstrate
complexity of understanding.
Portfolios Gathers information from variety of Without clear criteria, judgment is subjective.
sources over time. Usually respondent
self-reflects and rates own work.
Requires development of clear criteria
for judgment.
*This table is modified from the User-Friendly Handbook for Project Evaluation: Science, Mathematics,
Engineering and Technology Education, Directorate for Education and Human Resources, Division of Research,
Evaluation and Communication, National Science Foundation. Exhibit 5.

4-2
 WORKSHEET 1 FOR INDICATORS CURRENTLY IN USE

Assessment Efforts in __________________________

Date: ______________

Undergraduate Programs - Direct Indicators

A indicates “Active use”
P indicates “Plans to use”
National Local Capstone Embedded Student Review Performance Pre & Post
Exams Exams Course Testing Portfolios Theses Evaluation Tests
Year 1
Year 2
Year 3

Notes:

Why did you choose these tools?

Assessment Efforts in the ___________________________

Date: _______________

Undergraduate Programs - Indirect Indicators

A indicates “Active use”
P indicates “Plans to use”
Student Exit Alumni Employer External
Surveys Interviews Surveys Surveys Reviews
Year 1
Year 2
Year 3

Notes:

Why did you choose these tools?

4-3
 WORKSHEET 2 FOR INDICATORS CURRENTLY IN USE

Assessment Efforts in the ____________________

Date: ______________

Graduate Programs - Direct Indicators

A indicates “Active use”
P indicates “Plans to use”
National Local Capstone Embedded Student Review Performance Pre & Post
Exams Exams Course Testing Portfolios Theses Evaluation Tests
Year 1
Year 2
Year 3

Notes:

Why did you choose these tools?

Assessment Efforts in the ___________________________

Date: _______________

Graduate Programs - Indirect Indicators

A indicates “Active use”
P indicates “Plans to use”
Student Exit Alumni Employer External
Surveys Interviews Surveys Surveys Reviews
Year 1
Year 2
Year 3

Notes:

Why did you choose these tools?

4-4
 Section 5
SURVEYS

Surveys are an excellent way to gather information that describes, compares, or explains
knowledge, attitudes and behaviors. This section includes a worksheet on survey creation and a
worksheet on survey analysis.

We have found that THE SURVEY KIT by series editor Arlene Fink is an excellent resource for
conducting surveys. We highly recommend that you acquire and study this resource and therefore
we do not attempt to supply information in much detail regarding surveys. THE SURVEY KIT
contains 10 volumes, each focusing on a specific aspect of conducting a survey. This kit includes
volumes on asking survey questions, conducting self-administered and mail surveys, interviews by
phone and in person, designing surveys, how to select a sample for surveys, how to measure
survey reliability and validity, how to analyze survey data, and how to report on surveys. We
think it provides some of the more sophisticated information about using surveys that can guide
your survey development. THE SURVEY KIT is available through SAGE publications at the
address below:

SAGE Publications Inc.

2455 Teller Rd.
Thousand Oaks, CA 91320
email: [email protected]

Tips on surveys
• Allow enough time – plan for time needed to conduct first mailing, receive the first set of
responses, conduct follow-up mailing, receive follow-up responses, input data, conduct
analysis, and create report.
• Before using a survey, pilot it on a small sample of a similar audience. Then modify unclear or
poorly worded questions or delete those questions that do not seem to elicit worthwhile
information.
• Plan to do two mailings. The follow-up mailing can add up to 1/3 more responses.
• Keep the survey as short as possible.
• Sending surveys on colored paper can increase your return rate.
• Make sure the return date and sender address is bold, big and clear on the survey.
• Use a short cover letter on the survey from the department chair. Personalize the letter so
that the students, alumni or employers recognize they have valuable information to contribute.
• Use incentives to increase return rate - Response rates can be improved if you give a pizza
coupon when the survey is returned or enclose a card that becomes part of a drawing to win a
prize. When planning these techniques, plan them in a way that keeps the anonymity of the
respondent. Anticipate an average return rate of about 20% unless you do something
exceptional as an incentive.
• Identify one person to keep records and do follow-up mailings.

5-1
 SAMPLE QUESTIONS TO CONSIDER FOR USE IN SURVEYS OR INTERVIEWS

Do not attempt to use all of the questions below on a single assessment. These questions are
intended to provide help in brainstorming questions. This list is by no means complete, nor is
the wording of each question set in stone. To use this form, select which categories of questions
your assessment will include, then decide which questions or types of questions will best elicit the
type of information you need. Modify and tailor questions to fit your program needs.

QUESTIONS ON “DEMOGRAPHICS:”
m Name
m What is your specific area of interest within the overall field?
m Are you completing a minor, double major, or specialization option?
m What types of companies are you interviewing with and why did you choose those types?
(e.g., service, manufacturing, consulting, design, other)
m Have you been working while in school? How many hours per week? Summer/school year?
In what field?
m How many hours per week do you typically spend on coursework outside of class?
m How many semesters has it taken you to graduate? How many credits have you taken?
m Has working during school affected the quality of your classwork or the time it’s taken you to
graduate? For better or for worse? What recommendations do you have for other students
who are considering working while in school?

QUESTIONS ON THE PROGRAM OVERALL:

m How do you feel about your choice of major? What encouraged you to select your major?
m What, if any, changes in your undergraduate program would have helped better prepare you
for work as a ______________?
m Are there areas of your undergraduate program that you feel will not be useful to you? What
are they?
m What topics do you feel will be most useful to you and why?
m What did you expect of the program when you began your major?
m Do you feel that the curriculum has met your expectations (in preparing you for your
professional future)?
m Do you think the program has equipped you for a typical day on the job? What do you think a
typical day would be like?
m What do you think you learned in your undergraduate program that will assist you in your first
job?
m Please give an overall rating of how well your undergraduate education prepared you for your
professional career. [A scale of possible answers would follow this question.]
m Who or what drew you to this major? (Prompt: national reputation of department,
scholarships, etc.)
m How do you feel about your choice of major? Was it what you expected?
m Think of a fellow classmate who didn’t stay in the program. What kept you here (in the
department or at the university) when others left?
m If a cousin of yours were to consider this program, what kind of advice would you give
him/her?

5-2
m What are the strengths of the program?
m What aspects of the program could be improved?

QUESTIONS ON COURSES AND PROGRAM REQUIREMENTS:

m During your undergraduate study, what subject areas, if any, would you have liked to study
more? Why?
m Which courses were most valuable to you and why?
m Which courses were least valuable to you and why?
m Which courses were most difficult for you and why?
m Which courses were most interesting to you and why?
m What is your opinion of the number of credits required for a bachelor’s degree in this
program?
m If a change was made in the number of credits or in the required courses, which courses
would you eliminate or reduce?
m What do you think of the balance of fundamentals courses, core courses, elective courses,
liberal studies courses, etc.? If you would shift the balance, what changes would you make
and why?
m Has your opinion of the program requirements changed since you’ve been on the job?
m Do you have a sense of how the various course requirements fit together to make a “program”
as opposed to a series of individual courses?
m How could the department improve the course requirements or balance of courses?

QUESTIONS ON CO-OPS AND OTHER SPECIAL COURSEWORK:

m Did you participate in a co-op (or internship or other special course)? If so, where?
m If you completed a co-op, did you find it valuable? Why?
m How do you rate the value of co-op experience? Why do you rate it this way?
m How, if at all, could your co-op have been improved? Be as specific as you can.
m In what ways, if any, did your co-op experience influence your choice of career?
m If you went to graduate school, in what ways, was your graduate school experience influenced
by your co-op experience?
m Given your experience in the capstone design course, which aspects of the program best
prepared you for work as a __________________?
m Given your experience in the design course, are there areas in which you feel unprepared for
work as a ________________?
m Do you know how to learn about those areas on your own?
m Did you take an independent study course?
m If you completed an independent study, how do you rate it? Why do you rate it this way?
m How, if at all, could your independent study have been improved? Be specific as you can.
m In what ways, if any, did your independent study experience influence your choice of career?
m If you went to graduate school, in what ways, if any, was your graduate school experience
influenced by the independent study?
m Did you participate in a summer lab? (Or other course unique to your department)
m If yes, please rate the value of the summer lab in particular and comment on why you rated it
this way.
m Do you have suggestions for improving the program’s special coursework?

5-3
QUESTIONS ON NON-TECHNICAL SKILLS:
m Do you think you’ve gained the skills necessary to be an effective team member on the job?
What are those skills? Which parts of the program helped you gain those skills?
m Do you think you’ve gained the skills necessary to communicate effectively (orally and in
writing) on the job? Which parts of the program helped you gain those skills?
m What are the most important qualities or skills that a person working in this field should have?
Why? (For example: working independently, creative thinking, problem solving, time
management, communication, working in a team, intellectual curiosity, confidence in field,
ethical responsibility).
m Which skills would you like to see the program encourage or improve on?
m What skills would you like to develop that you haven’t yet?
m How, if at all, would you change the way the department helps students develop their non-
technical skills?

QUESTIONS ON LEARNING STYLES:

m How do you learn best? (Lecture, lab, homework, teamwork, computers, visual aids, reading,
etc.)
m What types of courses have helped you learn best?
m First I want you to think about how you learn. (Lecture, homework, in a group, alone, by
applying what you learned to real life) Then, think about your last two years in the program.
Overall, what were the best ways for you to learn? What helped you make connections with
the material?
m In which courses did you learn the most? What was it about those courses that was so
effective?
m Do you think teamwork is an effective way for you to learn? Why or why not?
m Can you think of ways the department could better address your style of learning?

QUESTIONS ON ADVISING AND MENTORING:

m How do you rate the quality of career advising you received?
m How could career advising be improved? Be as specific as you can.
m Did you have someone or some group who “took you under their wing” or who acted as a
mentor? ( Prompt: Formally or informally? Was it someone within your department?)
m What advice would you give to incoming students? To other students in the department?
m Do you have suggestions for strengthening the advising or mentoring process?

QUESTIONS ON POST-GRADUATION PLANS OR EXPERIENCE:

m What are your plans for the future? What are your long-range career goals?
m Do you plan to go to graduate school? When? Why or why not? In what field?
m Do you plan to begin working immediately after graduation? Why or why not? In what
field?
m If you plan to switch fields after graduation (or if you switched before graduation), what
influenced your decision? Could the department have done anything to convince you to stay
in the field?

5-4
m Do you plan to remain in the state (or country)? Why or why not?
m Since graduation, have you taken continuing education or industrial short courses? What
subjects have you studied? Why?
m Please write a short description of the type of work you do in your present position.
m Do you supervise the work of other engineers?
m (Especially for supervisors:) In your view, what deficiencies in preparation do people
beginning work in this field have?
m If you have attended or completed graduate school or are currently in graduate school, please
rate how well your undergraduate education prepared you for graduate study.
m How well did your training prepare you to compete within your field or current area of
employment?
m How does your educational preparation compare with that of peers in your field from other
schools?
m Rate how well your education prepared you in the following areas and also rate how useful
these areas have been in your career. [A chart listing the program courses would follow here.]
m If you rated any of the above topics as “very prepared” or “ poorly prepared,” please
comment below on why.
m If you rated any of the above topics as “very useful” or “not useful” please comment below
on why.
m How could the department have better prepared you for your job or for graduate school?

WRAP-UP QUESTIONS:
m Any other comments on the program or “messages” you’d like the faculty to hear?
m What suggestions do you have for addressing the limitations you’ve described?
m From your answers above, are there particular issues you’d like to emphasize or be sure we
include in our report?
m Is there anything we haven’t covered in this interview/survey that you’d like to mention?

5-5
 CHECKLIST FOR CONDUCTING SURVEYS

m Is your survey easy to read? (on lightly colored paper, large print, legible font and photocopy
quality)

m Does the survey include information at beginning and end about when and where survey
should be returned?

m Have you enclosed a stamped and addressed return envelope?

m Does each question ask only one question at a time?

m Do you have a system for recording the names and both old and new addresses of surveys
which were returned-to-sender?

m Do you have a system for recording response rates?

m Have you kept a copy of the original mailing list?

m If you are doing a second mailing, has it been at least two weeks since the first was sent?
(Ideally, you should wait two weeks before sending a second mailing)

m Have you changed the return date on the second mailing of the survey?

m Have you used another color paper on the second mailing of the survey?

m Have you updated your records with return-to-sender information?

m Have you established a way to enter the data?

m Have you established a way to conduct the analysis?

5-6
 WORKSHEET FOR CREATING A SURVEY

1. What information do you want to tell the participant about returning the survey? What can
you provide the participant to aid this process?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

2. What background or demographic information do you need to collect? (Examples: Name

[current and past], address, phone)
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

3. What information do you want to collect about participants' academic career or work
experience? (Examples: Years attended, locations, degrees earned, numbers of jobs, job
descriptions)
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

4. What are the broad topics you are interested in learning about? (Examples: Integration of
curriculum, learning preferences, preparation for work)
A.___________________________ B._________________________
C.___________________________ D._________________________
E.___________________________ F._________________________
G.___________________________ H._________________________

5. What are the particular dimensions of the topics you are interested in? (Examples: Parts of
the curriculum that seem extraneous? Particular projects or classes that help students learn
best? Ways alumni feel they were not adequately prepared for the world of work?)

Topic A Topic B
___________________________ _________________________
___________________________ _________________________
___________________________ _________________________
___________________________ _________________________

5-7
Topic C Topic D
___________________________ _________________________
___________________________ _________________________
___________________________ _________________________

Topic E Topic F
__________________________ _________________________
___________________________ _________________________
___________________________ _________________________
___________________________ _________________________

Topic G Topic H
__________________________ _________________________
___________________________ _________________________
___________________________ _________________________
___________________________ _________________________

7. Are there relationships between topics you would like to learn about? (Example: Is there a
relationship between advising and time-to-degree?)
___________________________ + _________________________
___________________________ + _________________________
___________________________ + _________________________

8. Are there topics you would like to know the value of in participants' views? (Example:
Strengths and weaknesses of internships? Value of co-op?) What specific dimensions of the
topic are you interested in?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

5-8
 WORKSHEET FOR SURVEY ANALYSIS

1. What is the demographic profile of your sample? For each category, list the sub-categories
below the heading. Add or change categories to fit your needs.

Number in total Percent in total Number in Percent in

population population sample sample
Age

Ethnicity

Gender

GPA

Year-in-School

Other

5-9
2. If your survey has open-ended questions, can the responses be grouped into like categories? If
so, what are they?
A.___________________________________________________________________
B.___________________________________________________________________
C.___________________________________________________________________
D.___________________________________________________________________
E.___________________________________________________________________

3. For each of the categories listed above, what are the varying comments/perspectives/issues?
A.____________________________________________________________________________
B.____________________________________________________________________________
C.____________________________________________________________________________
D.____________________________________________________________________________
E.____________________________________________________________________________

4. Are there responses that relate to each other? (Do not look at grouping of questions on survey,
but at responses)
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

5. Are there dimensions to the program that respondents feel are strengths? Why?
______________________________________________________________________________
______________________________________________________________________________

Weaknesses? Why?
______________________________________________________________________________
______________________________________________________________________________

5-10
 SAMPLE COVER LETTER FOR SURVEYS

[Department’s Return Address]

[Date]

Dear [Dept. Name] [Senior or Alumnus/a]:

The UW-Madison [Department Name] is conducting a self-assessment of its

[undergraduate/graduate] degree program in an effort to improve the education that our students
receive. Your opinions and comments about our program are important to us. We hope you will
take about fifteen (15) minutes to complete and return the enclosed survey.

All responses will be held confidential. The surveys will be read and analyzed by [name of office
or individual], and any reports produced as a result will not contain any identifying characteristics
of individual respondents. The original surveys will be available only to [name of office or
individual]. [Name of office or individual] will prepare a report for the department based on these
surveys, and a [copy or summary] of that report will be available to you [via a department
mailing, newsletter, etc.]. [Insert description of department’s plans for responding to the survey
findings.]

Please complete and return the survey to [name & address in bold] by [date in bold]. A stamped,
addressed return envelope is provided for your convenience. If you have any questions about the
survey or the department’s self-assessment, please feel free to contact [name of office or
individual] at [address, phone number, or e-mail].

We recognize how busy you are and appreciate your time and feedback. Thank you for assisting
us in our efforts to improve the [undergraduate or graduate] program.

Sincerely,

[Department Chairperson or Representative]

Encl.: Survey & Stamped, Addressed Envelope

5-11
 SAMPLE THANK-YOU POSTCARD FOR SURVEYS

We have received your completed

survey – Thank You!

We appreciate your assistance in

our efforts to improve the
[undergraduate/graduate] [Student or Alumnus/a Name and Address]
program and welcome your
additional comments at any time.

A [copy or summary] of the

survey results will be available
[when and how].

[Department Name]

5-12
 WORKSHEET 1 FOR SURVEY TRACKING

Department: _________________________________
Date: _______________________________________

Last Name, First Name

In Sample or Not?
# of Semesters to
Month & Year of

# of Credits to

GPA in Major

Male/Female
GPA Overall
Graduation

Graduation

Graduation

Ethnicity
ID #

123 Sample, Chris May-93 11 132 3.245 3.125 F Native Amer. yes

Lowest Value
in Range

Highest
Value in
Range

Averages
(total # of

(total # of
students)

years)

Totals &
Percentages

5-13
 WORKSHEET 2 FOR SURVEY TRACKING

Department: _________________________________
Date: _______________________________________

Included in 2nd Mailing?

Updated Address, if any

Included in 1st Mailing?

Data Entry Completed?

Last Name, First Name

2nd Mailing Address

Survey "Returned to
1st Mailing Address

Completed Survey
Returned (Date)
Sender"
(Date)

(Date)
ID #

321 Bradford Ln. Apt. A

321 Bradford Ln. 456 Marshall Rd. Eau Madison, WI 53706-
123 Sample, Chris Madison, WI 53706 Claire, WI 55555 0321 3/4/98 3/20/98 4/1/98 yes
(total # of

received)
returned)
names)

mailed)

mailed)
(total #

(total #

(total #

(total #
TOTALS

RETURN RATE = _____ Received out of _____ Mailed = _____%

5-14
 Section 6
INTERVIEWS

This section summarizes many of the important points you should consider as you use interviews
to evaluate your major, program, or department. It presents some broad guidelines which can be
applied to a variety of interviewees whether they are current students, graduating students,
alumni, or employers. The section also gives general background information about conducting
interviews in various formats such as individually or in groups, over the phone or face-to-face,
and tape recorded or not. The section discusses these formats, poses questions to consider when
planning interviews, provides checklists and worksheets for conducting interviews, and discusses
methods for analyzing interviews.

Thirteen questions to consider prior to interviewing

These questions are inter-related and should be considered as a whole instead of in a linear way.
(see worksheet at the end of this section that is guided by these questions.)

1. What is the purpose of the interviews?

Evaluators/Personnel who will be conducting the interviews need to communicate clearly with all
stakeholders (faculty, deans, staff, and students) that the purpose of the interviews is program
assessment and not to evaluate teaching or a specific course per se. The use of interviews as a
data collection method begins with the assumption that the participants’ perspectives are
meaningful, knowable, and able to be made explicit, and that their perspectives are important to
the success of the program.

2. Who will do the interviews?

a. Who, within your department or unit, will conduct interviews? The interviewers you select
should be interested in the project, have time to participate, and be willing to cooperate with
others to follow an interview plan over a specific timeframe. In each case, the interviewer
should be a person with no past, present, or likely future authority over the student/alumni and
one who is capable of facilitating an open discussion without passing judgment on student
responses or leading the student to make specific responses.
b. If possible, a small group of interviewers (two to four) should collaborate on the interview
process. Using multiple interviewers allows interviewers to be assigned to interviewees they
do not know. It also provides a range of interviewer backgrounds which can broaden the
interpretation and understanding of the findings (this is one form of triangulation).
c. It is important that one person act as a “point person” for the entire interviewing process.
This person will coordinate all of the activities and serve as a conduit for information and
communication.

3. Should we conduct individual or group interviews?

Some factors to consider when contrasting individual and group interviews are shown later in
this section in a table titled: “Which to use: Focus groups or In-depth interviews?” To be
useful, individual interviews generally need to be at least 30 minutes, preferably 45 minutes to

6-1
 one hour. Group interviews usually run from one hour to one and one-half hours.

4. How many interviews should be conducted? Should we select a sample?

You need to conduct enough interviews so that similar responses or patterns begin to occur. This
can usually be accomplished by fifteen to twenty individual interviews. Because some people may
choose not be participate in the interviews and others who do agree may not actually show up for
the interview, we suggest that you may need to ask about 25 people to participate in order to
accomplish 20 interviews.

Since most programs do not have resources to interview every student, alumni, or employer, you
will likely need to ask a sample to participate in the interviews. Some suggested ways to select a
sample to interview are presented below.
• Invite a percentage: If you want to select 25 students out of 50, for example, you can choose
every other name on the student roster.
• Invite everyone in the targeted group: You can ask everyone (through posters, e-mails, or
phone calls) in the program to participate in an interview. Then wait to see how many
respond and confirm interviews with the first 25 who respond to your request. This is an
efficient way but may not allow for a very representative sample.
• Invite a representative sample: For a sample to be representative of the entire population,
your sample should reflect proportions of the entire population based on criteria you identify
as important. These criteria will vary for different groups, however, some of the most
commonly used criteria to select a sample are: gender, ethnicity, GPA, length of time in
program. For example: An entering class in the School of Pharmacy had 100 students who
were 60% female, had an average GPA of 3.2, and half had completed their first two years at
UW-Madison. In their sample they purposely selected a sample of twenty students who were
60% female, had a range of GPA’s above and below 3.2, and half of whom had attended UW-
Madison for 2 years.

5. How should we invite students/people to interview?

Many departments have been successful at inviting students by e-mail and by telephone. Now that
most students have e-mail access, this seems to be a good medium because it is easy for the
respondent to respond. Allow several days for people to check e-mail messages. Telephone
contact is tedious and often requires back-and-forth messages. If you plan to do phone
interviews, call the student and ask if they will agree to an interview, and then set a convenient
date and time that you can call them back.

6. Should we use a written pre-survey?

Most programs we have worked with report that it is beneficial to use a short pre-survey that
students either complete quickly at the beginning of the interview or bring with them completed to
the interview. It provides a good beginning point for the interview discussion and conserves
interview time that might otherwise be spent collecting this background information. Pre-surveys
can be mailed out, faxed, or students can pick them up at a central location prior to the interview.
(See example of a pre-survey at the end of this section).
7. Should we provide questions to interviewees ahead of time?
Interviewees report that it is helpful to have the questions ahead of time. Providing questions

6-2
ahead of time may take away any anxiety about interviewing that some interviewees may feel. It
also allows interviewees time to prepare thoughtful answers which can improve the quality of the
responses you receive. Questions can be sent to the interviewees over e-mail at the same time you
confirm the interview time or made available at the main departmental office.

8. When should we interview?

An agreed-upon timetable that details the time frames for the interview process should be created.
If possible try to conduct all interviews within a concentrated two-week long to one-month long
period. Student interviews should be done early to mid-semester, and be completed by about the
11th week of the semester. Toward the end of the semester students are less likely to participate.
Try to avoid interviewing during holiday seasons such as Christmas and Easter when people are
extremely busy, or during summer when many people take vacations. If you are conducting
phone interviews, it may be necessary to make calls in the evenings. We suggest that you schedule
one-hour blocks for interviews with half hour between interviews (e.g. interview from 11:00-
12:00, schedule next interview to begin at 12:30). Also try to schedule no more than three
interviews in one day for the same interviewer as the focused nature of interviewing can be
tedious and tiresome.

9. How can we practice interviewing?

It is useful for people who are not experienced at interviewing to practice one or two interviews
prior to conducting real interviews. Practice allows the interviewer to become familiar with
recording equipment, to practice setting a person at ease, to practice pacing so the interview is
completed in the allotted time, and to try out questions that may be sensitive or difficult to ask.

10. Where should interviews be conducted?

Ideally you should choose an infrequently used private location and minimize interruptions. The
location should be private and quiet enough to hear respondents, and one that is comfortable,
easily accessible, and can be equipped with audio recording equipment. If a group of interviewers
are collaborating, it is useful to choose a location where you can leave equipment in a box.
Seating arrangements should encourage involvement and interaction and break down status and
power relationships that may often exist when faculty interview students. If possible place seats
side-by-side instead of having the interviewer sit behind a desk. Group interviews are best done
around a table so that everyone can see and hear each other.

6-3
11. Should we tape record or not? Should we transcribe or not?
There are three approaches to recording interview data, each with advantages and disadvantages
as described below.

Approaches to Recording Interview Data: Advantages and Disadvantages

Approach Advantages Disadvantages

Tape record and transcribe Completeness and the opportunity it Amount of time and resources
verbatim affords the interviewer to remain needed; inhibitory impact tape
Word-for-word transcription is attentive and focused during recording has on some participants.
completed. This requires interview. Allows other It is essential that interviewees are
resources and time, but is interviewers to read the transcript assured of confidentiality and
valuable when respondent’s and collaborate together on analysis. permission to tape is obtained.
own words and phrasing are
needed.
Take notes, tape record, but do Recommended when resources are Interviewer may be more selective
not transcribe scarce, and when the results must be or biased in what he or she writes.
This approach draws on the provided in a short period of time.
notes taken by interviewer. As Note expansion saves time and
soon as possible after the retains all the essential points of the
interview, the interviewer discussion.
listens to the tape to clarify
certain issues and to confirm
that all main points have been
included in the notes.
Take notes but do not tape record Useful if time and resources are The interviewer must frequently
Interviewer takes detailed notes short, results are needed quickly, listen, talk, and write at the same
during the interview and draws and evaluation questions are simple. time, a skill that is hard for some to
on memory to expand and achieve. Where more complex
clarify the notes immediately questions are involved, note taking
after the interview. alone does not allow one to
document all of the intricate
relationships and descriptions.

12. How can we assure confidentiality and anonymity?

Please see Section 7 which summarizes issues related to ethics and the use of a consent form.
A goal of interviewing is to gain adequate information about your program. Most interviewees
are more inclined to participate in interviews if they know that what they say will remain
anonymous, if they believe that there will be no negative repercussions to their opinions, and if
they are assured that their interview tape will be used for analysis purposes only. Some ways to
address these issues are to use a consent form, assign ID numbers instead of student names to the
audiotapes, remove the tapes from the interview area promptly after the interview, and design the
analysis phase so that only the transcriptionist or interviewer listen to the tape and only one or
two people actually read transcripts. After analysis, completed audiotapes and typed transcripts
should be stored in a safe location. Although the use of a consent form is optional when doing
program assessment activities, it is considered good practice to ensure that procedures were
communicated in an open manner and to notify interviewees about process and confidentiality.

6-4
13. What problems might we anticipate?
Some problems you might anticipate are summarized below.

Problem Description/Solution
Interviewee does not show up Some interviewees will not show up, or if you are doing phone interviews,
will not be available at the time you have arranged to call them. It is
appropriate to try to re-schedule the interview, but discontinue efforts after
one more contact.
Interviewee is very uncomfortable Depending on the types of questions, it is possible that interviewees may be
talking about issues or becomes uncomfortable discussing certain topics or some may become emotional and
emotional cry or withdraw. If this occurs, tell the interviewee it is OK. If you are
taping, offer to turn off the tape recorder, and offer to take a few minutes
break. Ask if the interviewee would like water. Take a break from
questioning or ask if you should proceed.
You run out of materials Have extra materials: copies of questions, paper and pencil, consent forms,
batteries, blank tapes.

Interviewee confides important It is possible that the interviewee may reveal information related to issues
information such as harassment, unethical, or illegal activity. In the event this occurs,
refer the student to the appropriate office or counselor. A list of these
references is provided in the resource section at the end of this kit.
Equipment Fails Equipment failure happens. Have extra equipment on hand. Encourage
interviewers to take some written notes and check the tape recorder during
the session. After the interview is completed, the interviewer should also
check the tape. If the machine did not work, the interviewer can create a
written summary based on notes while the discussion is still fresh in the
interviewer’s mind.

Developing interview questions

Section 5 includes a worksheet on developing questions which may also be valuable in developing
interview questions. However, interview questions differ from survey questions in that they
should be open-ended rather than limited “yes-no” type questions. Their purpose is to elicit
perspectives and opinions from the interviewee. Ideally, interview questions will be developed
with input from a cross section of stakeholders such as: faculty, deans, staff, and students. For
example, The Department of Botany used a committee that represented a cross-section of the
department: two faculty, two academic staff, and one graduate student. This group brainstormed
questions related to issues that were of particular concern within the department. Some of these
questions related to the senior thesis requirement of the department and others related to the
newly-created departmental student learning goals. Drafts of interview questions and the short
survey were distributed to faculty, discussed at a faculty meeting, and then modified based on
faculty input.

6-5
Tips on Interviewing

An interview, rather than a paper and pencil survey, is selected when interpersonal contact is
important and when opportunities for follow-up of comments are desired. In-depth interviews are
characterized by extensive probing and open-ended questions. In-depth interviews are particularly
appropriate when seeking an understanding of complex or highly sensitive subject matter.
Typically the interviewer works from a list of questions or issues that are to be explored and
probes areas of particular interest. The guide helps the interviewer pace the interview and makes
the process more systematic and comprehensive.

Interviewers should seek to encourage open responses. It is important for an interviewer to

capture respondents’ perspectives in their own words. An interview is like a guided conversation
in which the interviewer becomes an attentive listener. In contrast to a normal conversation, an
in-depth interview is not intended to be a two-way form of communication and sharing. The key
to being a good interviewer is being a good listener and questioner. It is not the role of the
interviewer to put forth his or her opinions, perspectives, or feelings.

Tips related to personal interaction and environment

1. Put person at ease through small-talk and connect to them as a person, use lay-person’s
speech (e.g. Gee it’s cold, did you get here OK? Did you get a chance to eat lunch? Did you
have trouble finding the place?)
2. Introduce yourself, de-emphasize rank and status and get close to the student and at their eye-
level.
3. Describe the purpose of the interview - for program improvement, not faculty evaluation;
explain who will see report. State that their viewpoint and experience is unique and very
important, and there is not right or wrong answer to the questions since each student’s
experience is different.
4. Explain confidentiality, no names will be used and identity will be safeguarded.
5. Explain the nature of report which will be created - e.g., we will interview 25 students and a
committee of five will merge together their ideas to identify common issues and themes
6. Ask for permission to tape-record. Whether they agree or not, they should sign consent form.
If the interviewee does not agree to record, continue the interview, but take detailed notes.
Tape-recorder should be near person and unobstructed. To achieve a good quality tape
recording try to:
• avoid talking over people
• pause and wait while a train or ambulance is passing
• place tape recorder an equal distance between the interviewer and respondent – sit as
close to respondent as comfortable for both persons.
• if respondent has a thick accent or tends to mumble, try to repeat important points they
make for clarification
• include a note to the transcriptionist that spells out any frequently used names,
abbreviations, acronyms, or professional and technical terms.

6-6
7. Keep materials accessible. A box of materials containing the following items can be kept in
the interview room if the room can be locked when not in use.
• master lists of the interview questions
• blank surveys for students to complete prior to interviews, if applicable
• blank human subject consent forms
• tape recorders plus batteries or outlet
• paper and pens
• blank audio tapes
• a list of student names and assigned ID numbers
• a sign saying “Interview in Progress” and adhesive tape to post the sign on the door
• general instructions that summarize the interview process

Tips on Asking Questions

1. The interviewee should do most of the talking. You want to get their ideas and perspectives,
not expound on your own. If students ask you questions about steps or requirements, answer
these questions only after you have explored their understanding about these.
2. Be receptive to all student response. Be aware of your own body language (frowns, smiles,
laughter, gestures). Avoid showing surprise at answers and avoid negative or positive
gestures or responses to student comments.
3. See interviewing as a spiraling conversation. View the questioning process as a structured,
but not linear, flowing conversation. The written questions are a starting point only -- use
follow-up questions to focus and probe deeper. Examples of deeper probes: “Tell me more.”
“Tell me why you feel that way.” “Why do you think that happened?” “Why was that
important?” “Can you give me another example of what you mean?”
4. Paraphrase by repeating ideas using interviewee’s words (if possible) to check out what you
hear. Examples: “It seems like you are saying ____. Is that what you mean?” “When you
described that event you used the word ‘frustrating.’ Tell me why you use that word?”
5. Avoid leading questions or giving your own opinion: Examples: “Oh yeah, that was my
favorite class. What about you?” “That was a stupid requirement, wasn’t it?” “I can’t
believe they made students do that.”

Tips about how to analyze the interviews

Although interviews can provide a wealth of information, the analysis of interviews is challenging
because responses are usually quite varied, interconnected, complex, and flow throughout the
entire interview. Your task throughout the analysis is to look for common and recurring ideas,
issues and themes across the set of interviews, as well as try to recognize the range of the
perspectives that interviewees raise. When doing analysis, one must continually attempt to put
aside your own perspective and be guided by what the interviewees said.

Whether or not you work alone or with a small committee to conduct the analysis, and whether
you work from verbatim transcriptions or written notes, the process of analysis is generally similar
to that described below:

1. If interviewers used notes, have each one collect their notes together. If transcripts were

6-7
 completed, distribute these to the people who will complete the analysis. If you used several
interviewers and had the interviews transcribed, a way to distribute these to people identified
as “first” and “second” readers is described below:
Divide transcripts equally among the analysis committee to ensure that each transcript will
be read by two readers who did not conduct the particular interview. Identify a primary
reader and a secondary reader as shown in the example grid below. Both first and second
readers read the transcripts in the same way, but the first reader is the main one
responsible for understanding the perspective of the student who was interviewed and
synthesizing that information, although the second reader should expand on the thoughts
of the first.

EXAMPLE -- Distribution list for analysis of transcripts in Department X

Student ID # Interviewer First Reader Second Reader

1 0032 A B C

2 4127 B A D

3 6219 C D A

4 0002 D C B

2. Read through transcripts/notes for each interview and do the following:

Write a key word or phrase in the margin to capture the topic (such as advising, labs) and also
write a few words to summarize the perspective. Note both positive and negative factors.
Highlight or underline student comments or phrases when these provide a good example of
the perspective or provide a clear illustration of the interviewee’s perspectives. If you have
time after reading the interview, make a short summary of the issues and perspectives at the
very end of the transcript. Re-read as needed.

3. Identify common issues and themes, cluster, and re-group. The committee should hold
lengthy meetings to review and summarize the perspectives presented in each transcript. The two readers for
in a blackboard or flip-charts full of comments. Patterns and common issues and perspectives
will begin to be noticeable. The
committee should cluster or group these issues or try to capture the range of opinions if there
is no common opinion expressed. A continual process of re-grouping and synthesis should
occur. Quotations from interviewees that represent the prevalent perspectives can be
identified.

4. Create a summary report. See Section 8 for a discussion of types of information that should
be included in the report.

6-8
 WORKSHEET FOR SCHEDULING INTERVIEWS WITH STUDENTS
in Department of ______________

Week of ______________________

Scheduled Interviews
Date Interviewer Times available to
Time, Location, Student Name
conduct interview
A 10:00-1:00 or 12:00-12:45, Room 512, Lisa Jones
9/3-Monday
3:00-5:00
B

A
9/4- Tuesday
B

A
9/5-Wednesday
B

6-9
 Which to use: Focus groups or In-depth interviews?

Factors to consider Use focus groups when. . . Use in-depth interview when. . .
Group Interaction Interaction of respondents may Group interaction is likely to be
stimulate a richer response or new and limited or nonproductive.
valuable thought.
Group/peer pressure Group/peer pressure will be valuable Group/peer pressure would inhibit
in challenging the thinking of responses and cloud the meaning of
respondents and illuminating results.
conflicting opinions.
Sensitivity of subject matter Subject matter is not so sensitive that Subject matter is so sensitive that
respondents will temper responses or respondents would be unwilling to
withhold information. talk openly in a group.
Depth of individual responses The topic is such that most The topic is such that a greater depth
respondents can say all that is relevant of response per individuals is
or all that they know in less than 10 desirable, as with complex subject
minutes. matter and very knowledgeable
respondents.
Data collector fatigue It is desirable to have one individual It is possible to use numerous
conduct the data collection; a few individuals on the project; one
groups will not create fatigue or interviewer would become fatigued or
boredom for one person. bored conducting all interviews.
Extent of issues to be covered The volume of issues to cover is not A greater volume of issues must be
extensive. covered.
Continuity of information A single subject area is being It is necessary to understand how
examined in depth and strings of attitudes and behaviors link together
behaviors are less relevant. on an individual basis.
Experimentation with interview Enough is known to establish a It may be necessary to develop the
guide. meaningful topic guide. interview guide by altering it after
each of the initial interviews.
Observation by stakeholders It is desirable for stakeholders to hear Stakeholders do not need to hear
what participants have to say. firsthand the opinions of participants.
Logistics Geographically An acceptable number of target Respondents are dispersed or not
respondents can be assembled in one easily assembled for other reasons.
location
Cost and Training Quick turnarounds is critical, and Quick turnaround is not critical, and
funds are limited. budget will permit higher cost.
Availability of qualified staff Focus group facilitators need to be able Interviewers need to be supportive
to control and manage groups. and skilled listeners.

Source: User-friendly Handbook for Mixed Method Evaluations, Directorate for Education and Human Resources,
Division of Research, Evaluation and Communication, National Science Foundation. August 1997, p. 3-11.

6-10
 WORKSHEET FOR PLANNING INTERVIEWS

1. Who will be interviewed?

mCurrent students mGraduating students m other:______________________
m Alumni m Employers

Summarize the purpose of your interviews:

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

2. Who will conduct your interviews?

Who will conduct the analysis of your interviews? (show “A” next to name)
1) _____________________________________________________________________
2) _____________________________________________________________________
3) _____________________________________________________________________
4) _____________________________________________________________________

3. Who will conduct individual, group or phone interviews and how many will be conducted?
m Individual interviews: How many?______
m Group interviews: How many groups?______ How many in a group? ______
m Phone interviews: How many?______

4. Describe how you will select the sample to interview.

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

5. Describe how you will invite people to participate in interviews.

me-mail mphone call mother:_______________________________________

Describe how interviews will be scheduled (Who will send e-mail or call?):_____________
_________________________________________________________________________
_________________________________________________________________________

6. Describe the time-frame during which the interviews will be conducted:________________

_________________________________________________________________________

6-11
7. Where will interviews be conducted?___________________________________________
(Consider: Privacy. Can site be locked? Can materials be stored at site? Who has key?)

8. Which of the following processes will be followed:

m tape record and transcribe verbatim – Who will do the transcription?______________
m tape record and make expanded notes, but do not transcribe
m do not tape record, but take detailed notes

9. How will confidentiality and anonymity be assured?

Will a consent form be used? m yes m no If so, attach form.

How will tapes be marked to keep anonymity?_________________________________

______________________________________________________________________
______________________________________________________________________

How will tapes and transcripts be stored after use? _____________________________

______________________________________________________________________

10. Will there be a written pre-survey? m yes m no

If yes, develop this survey and attach.

If yes, how will the survey be distributed to interviewees? _______________________

______________________________________________________________________
______________________________________________________________________

11. Will interviewees be provided with the questions ahead of time? m yes m no
If yes, how will these be distributed to interviewees? ___________________________
______________________________________________________________________

12. If appropriate, what plans have been made to practice interviewing?__________________

_______________________________________________________________________
_______________________________________________________________________

13. What problems may arise while administering the survey? Are there plans to address these
problems? If so, what are they?_____________________________________________
_______________________________________________________________________
_______________________________________________________________________
_______________________________________________________________________

6-12
 CHECKLIST FOR PREPARING AND CONDUCTING INTERVIEWS

Preparing for Interviews

m Develop questions
m Develop consent form
m Develop pre-survey if applicable
m Practice interviewing
m Identify interview site
m Schedule interviews
m Plan environment
m Prepare equipment/materials box
m Make plans for transcription and identification on tapes

Conducting interviews
m Put “Interview in Progress” sign on door
m Review consent form with interviewee, ask permission to tape record, get student signature
m If a written survey is completed, review completed written survey for take-off into questions
OR Ask student to complete written survey quickly before asking interview questions.
m If permission has been granted, begin audio tape
m Conduct interview
m Near beginning of interview, check to see that tape is recording
m At end of interview, thank interviewee
m Turn off audio tape
m Remove “Interview in Progress” sign.
m Tape recorder, blank consent forms, extra surveys and lists of questions remain in box.
m If using ID numbers write number on audio-tape and record on master
list that the person showed up.
m Put tape and written survey in identified location for delivery to transcriptionist or data entry.

6-13
 SAMPLE GUIDE FOR INDIVIDUAL OR GROUP INTERVIEWS

Introduction and Purpose

“Good morning. I am _______ (introduce self). The purpose of this interview is to get your
perspectives about MAJOR/DEPARTMENT. We are especially interested in your experience
and your suggestions. This is not an evaluation of specific faculty, so we hope you will focus on
the program/major/department. I am not here to share information nor to give you my opinion.
Your perspectives are what matter. There is no right or wrong or desirable or undesirable
answers. I would like you to feel comfortable saying what you really think and how you really
feel about the topic.”

Permission to tape
FOR INDIVIDUAL: “If it is OK with you, I would like to tape record our conversation. The
purpose of this is so that I can get all the details but at the same time be attentive to the
conversation with you. All of your comments will remain anonymous. No one but the
transcriptionist will listen to the tape, and others involved in summarizing the report will not
know your name. I will be interviewing approximately [number] people, and compiling a
report, but there will be no reference to individuals. If you agree to this interview, please sign
this consent form.”

FOR GROUP: “My colleague [name of colleague] will be taking notes and tape recording the
discussion so that I do not miss anything you have to say. I have already explained these
procedures to you when we set up this meeting. As you know everything is anonysmous. No
one will know who said what. I want this to be a group discussion, so feel free to respond to
me and to other members in the group without waiting to be called on. I will try to give
everyone a chance to give their opinions about each topic. However, I would appreciate it if
only one person talks at a time. The discussion will last approximately one hour. There is a
lot to discuss, so at times I may move us along a bit. Now let’s start by everyone sharing their
name and a sentence of background about yourself.”

Ask Questions (see your list of questions)

Closure
“Thank you for coming and giving your precious time to help us improve our programs. If you
think of further information you would like to share, I can be reached at ____. We hope to have
this report completed by ______. I wish you success in your program/summer.”

6-14
 SAMPLE QUESTIONS FOR PRE-INTERVIEW SURVEY

This form is intended to serve two functions. First, it provides background and a taking-off point
for the respondent’s comments during the interview. For example, it may be important to know
that a student who clearly understands the purpose of the various program requirements has
been working in the field throughout school. Second, the form can be used to identify areas to
pursue in an interview. For instance, an interviewer may want to question why a full-time
student took seven years to complete an undergraduate degree.

Student Name__________________________________ Date:_______________

1. When did you enter the ______ major? Semester 1 ____ Semester 2____ Summer____
Year Year Year

2. What type of bachelor degree as you working toward? (Circle one) BA BS

3. How many semesters has it taken you to reach graduation? _______________________

4. How many credits has it taken you to reach graduation? _________________________

5. Have you been a full-time student?

m Yes, throughout school.
m No, I’ve always been part-time.
m Sometimes full-time, sometimes part-time.

6. Check the statement below which best represents your path in college:
____ I have studied or plan to study all of my coursework at UW-Madison.
____ I transferred into UW-Madison from another institution. If so, how many semesters
have you been at UW-Madison? ________________________
____ Other: Please comment: _____________________________________________

7. Students in the ______ major can take several foundation courses or sequences before they
take upper level courses (e.g., in biology students can take Biocore or 151-152 or a different
choice). What foundations sequence did you take? (circle one below)
Biocore 151-152 Other:__________________________________

8. When do you plan to graduate? (Circle semester and year)

Spring Summer Fall 1998 1999 2000

6-15
WORK EXPERIENCE
9. Did you complete a co-op or internship?
m Yes. With what company/school? ______________________________________
m No.

10. Have you been working (other than co-op) while attending school?
m Yes, approximately _______ hours per week during the semester.
m Yes, approximately _______ hours per week during the summer.
m No.

11. If you have been working, has your work been related to your field of study?
m Yes, I’ve been working in (which field?) ________________________________
m No.
m Some yes, some no (which field?) ______________________________________

12. How many hours a week do you typically spend on coursework outside of class? _________

POST-GRADUATION PLANS
13. What are your post-graduation plans? (If you are planning on both employment and graduate
school, please indicate which you plan to do first.)
m Employment in (what field?) _______________________________________
m Graduate school in (what field?) ____________________________________
m Undecided

If Graduate School: How well prepared do you feel you are for graduate school?
_____Not Prepared ______Prepared _____Well Prepared

If Employment:
A. Do you have a job lined up after graduation? ____Yes ____No
If yes, what type of job is it?___________________________________________

B. How well prepared do you feel you are for employment?

_____Not Prepared _____Prepared _____Well Prepared

C. What type of employers are you interviewing with? Check all that apply.
m Consultingm Design m Manufacturing
m Service m Other _______________________________________

14. Do you plan to remain in the state/country after graduation?

m Yes m No m Undecided

6-16
 ADDITIONAL SAMPLES OF PRE-INTERVIEW SURVEY QUESTIONS

SAMPLE #1

As a result of your major in [list program name] at UW-Madison, do you think the program
adequately provides opportunity for the development of the program goals shown below?

RATING
Superior Good Fair Poor
PRIMARY PROGRAM GOALS Provides Provides Provides Provides
numerous adequate limited insufficient
opportunitie opportunitie opportunities opportunitie
s s s
Goal 1

Goal 2

Goal 3

Goal 4

Goal 5

Goal 6

Goal 7

SAMPLE #2

I FEEL WELL I HAVE NO STRONG I FEEL UNDER-

PREPARED IN THIS FEELINGS EITHER PREPARED IN
AREA. WAY. THIS AREA.
TECHNICAL KNOWLEDGE
DESIGN SKILLS
COMMUNICATION SKILLS
oral
written
TEAM WORK SKILLS
CONFIDENCE IN MY ABILITIES
AREAS OF THE CURRICULUM:
math & natural science
core courses
electives
computer science
technical communication
liberal studies
OTHER (PLEASE SPECIFY)

6-17
SAMPLE #3

I LIKE THIS PART I HAVE NO I SEE ROOM

OF THE STRONG FOR
PROGRAM; DON’T FEELINGS IMPROVEMENT
CHANGE IT. EITHER WAY. IN THIS AREA.
PROGRAM STRUCTURE OVERALL
# of credits required
time to graduation
formal advising
informal mentoring
TEACHING AND INSTRUCTION OVERALL
lectures/discussion
lab work
group work
homework
SPECIALIZATION OPTIONS OVERALL
construction eng & mgmt
environmental eng
surveying
AREAS OF THE CURRICULUM OVERALL
math & natural science
core courses
electives
computer science
technical communication
liberal studies
co-ops
OTHER (PLEASE SPECIFY)

6-18
 SAMPLE INTERVIEW QUESTIONS

Department of Botany
University of Wisconsin-Madison

Exit interviews with seniors (undertaken February 1997)

INTRODUCTION

1. When did you declare botany as your major?

2. Did you come to the UW knowing that you wanted to be a botany major, or is that something
you decided once you were here on campus?
3. Did you consider any other majors? If so, why did you decide to major in Botany?
(Prompt: advisor? Specific interest? reputation of department? scholarships?)
4. What introductory biology course or course sequence did you take? How was it that you
came to choose that sequence?
5. What is your specific area of interest within botany?

CAREER PLANS

6. What do you plan to do after receiving your bachelors degree?

7. What are your long-term career plans?

(Prompt: continuing education - what major, why, where)
(Prompt: employment - what field, why, where)

8. Do you think your undergraduate program has equipped you with the skills and knowledge
you are likely to need to pursue your career plans?

The following four questions all start with the following phrase:
Of the courses you have taken, are there any courses required for the botany major which you
consider to be: 9. Most worthwhile? Most valuable? Why?
10. Worthless or not relevant? Why?
11. Most difficult? Why?
12. Most interesting? Why?

13. The botany major requirements specify that students take courses in five out of six areas
within botany. Have you found that to be an advantage or disadvantage and why? Do you
feel there should be more flexibility in choices or do you agree that the design is adequate?

14. We strongly encourage/require our majors to get some sort of research or field experience
either as a 699 project or a thesis. Did you (or do you plan to) do some sort of research or
field project?
If answer is NO, then: Why did you decide not to?
If YES: How valuable do you think it was?

6-19
ADVISING

15. Do you feel that you received adequate advising prior to your decision to declare botany as
your major? Why? Can you suggest ways to improve the advising system?

16. Do you feel that you received adequate advising after declaring botany as your major? Can
you suggest ways to improve our departmental advising system?

17. Do you feel a sense of belonging to the botany department? Why or why not? Do you have
any suggestions related to this?

GENERAL

18. Taken as a whole, how would you assess our undergraduate program?

19. What do you consider to be some of the strengths of the botany major?

20. What do you consider to be some of the weaknesses of our program?

21. Can you suggest specific ways in which our undergraduate program might be strengthened or
improved?

22. Are there any concerns suggestions, or comments you would like to make about the program
that haven’t already been discussed?

6-20
 SAMPLE INVITATION TO STUDENTS TO INTERVIEW

Note: Can be e-mailed or summarized over the phone

********

Dear graduating senior in _____________,

The Department of ____________ is involved in activities to evaluate and improve its curriculum
and majors. As part of this, I would like to hold individual or group interviews with as many
graduating seniors as possible sometime between [dates]. The interview can be arranged at a time
convenient for you (Mondays through Fridays between 8:30 and 5:00). Because you are a senior
who is about to graduate, your opinions and suggestions are very important to help improve your
program. The interview will last about 45 minutes and participation is voluntary. There are four
people who will conduct the interviews which will take place at [identify location]. With your
permission we would like to tape-record the interview. Only I and the transcriptionist will listen to
the tape. If you do not want your interview taped, that is also OK. All student names and
identities will remain confidential. The discussion questions will be provided in advance so that
you can think about them ahead of time. I hope you will assist us by giving your feedback and
sharing your opinions about your program.

PLEASE RESPOND BY [date].

IF YOU CAN PARTICIPATE in an interview, please e-mail me with your first, second and third
choices of dates and one-hour time frames that will work for you between the dates [enter dates].
I will quickly respond to you with the specific time and a room location.

IF YOU DO NOT WANT TO INTERVIEW, please e-mail me with “No thanks.”

Thank you in advance for a quick response and for your participation in this important
improvement process.

6-21
 SAMPLE OF INDIVIDUAL INTERVIEW RECORD

Department: ___________________________
Year: ______________

INTERVIEWER NAME: _________________________________________________

INTERVIEWEE NAME/CODE #: _________________________________________

DATE OF INTERVIEW: _________________________________________________

CONSENT FORM SIGNED? Yes No

DATE TRANSCRIPTION ORDERED: _____________________________________

NOTES:

6-22
 SAMPLE OF THANK-YOU POSTCARD FOR INTERVIEWS

Thank you for taking the time

recently to be interviewed as part
of the department’s self-
assessment study.

We appreciate your assistance in [Student or Alumnus/a Name and Address]

our efforts to improve the
[undergraduate/graduate]
program and welcome your
additional comments at any time.

A [copy or summary] of the

interview results will be available
[when and how].

[Department Name]

6-23
 TOTALS
ID #

123 Chris
Sample,
(# of names) Last Name, First Name

608/555-5555
[email protected]
Address/Phone

3/4/98 left
phone msg.
1st Attempted Contact
e-mail
3/7/98
reached via

2nd Attempted Contact

Unreachable but contact

(# unreachable) info OK

Disconnected or
(# incorrect info) Incorrect Contact Info

Declined Interview
(# declined) (Reason?)
yes

(# agreed) Agreed to Interview

Department: ______________________________________
Date: ____________________________________________

Date/Time Scheduled
####### yes

(# interviewed) Interview Completed

WORKSHEET FOR CONDUCTING INTERVIEWS

(# no shows) No Show
yes

SAMPLE SIZE = _____ Interviewed out of _____ possible = _____ %

Consent Form Signed

yes

Transcript Completed
715 555-5555

Updated Address Info

6-24
 Section 7
ETHICAL CONSIDERATIONS

No matter which type of assessment tool you use, there are several ethical dimensions to
conducting a program assessment, and a little forethought can prevent some awkward situations
later.

Confidentiality and Anonymity

Both students and alumni are more likely to contribute openly and honestly if they can be assured
that their responses will not harm them in any way. Surveys can be returned anonymously
(though it will be necessary to collect enough demographic data to determine the
representativeness of the sample). Interviews can be conducted confidentially by non-department
members, or can be conducted by a single department member or group who will not reveal the
identities of the interviewees.

Although it is not required that a department contact the campus human subjects research
committees for approval of program assessment research, departments may still wish to provide a
consent form for participants, especially in interviews, where the identity of the interviewee may
be harder to conceal. A sample form is included at the end of this section. For additional
guidance on obtaining consent, departments can contact their local human subjects research
committee.

When Students Confide In You

Occasionally, when conducting surveys or interviews, a student or graduate will reveal sensitive
information such as an instance of sexual harassment, unethical behavior, or even illegal activity.
Although these revelations may be rare, anticipating and planning for them can make them easier
to deal with if they do arise.

How broadly will unfavorable findings be reported?

If students make disparaging comments about an individual professor, will the information be
circulated to the rest of the department? If alumni report on a shortcoming of the department,
will this information be forwarded to deans or other administrators?
• Make the purpose of the assessment clear up front. Program assessments request information
on the program rather than on individual faculty. (However, students often fail to make this
distinction, and often cannot make the distinction when discussing the individual courses that
make up the program.)
• The best assessments are those in which the informants can be completely honest. But such
openness can have limits; i.e., the full assessment report need not be made broadly available.
A small (but trusted) committee can determine how much and which information should be
distributed to the whole department, and the (trusted) department can determine what to
report to administrators. (Clearly, this system will work only when faculty and administrators
trust that findings are being addressed even when they are not reported.)

7-1
What to tell the student?
When a student has revealed sensitive information, the first concern should be to determine
whether the student wants or needs help, and in what form.
• Do they want to report the incident officially? (Do they think they are doing so by telling
you?)
• Do they want or need to talk about the incident further, perhaps with a counselor?

Whom to tell?
Section 10 of this Tool Kit lists some campus resources where students can be referred.

When to break confidentiality?

If a current or former student reveals information about a situation that is not only sensitive, but
potentially unsafe or illegal, when can or should you break confidentiality?
• If you are promising confidentiality, you may want to stipulate conditions under which you
will be obliged or willing to break that confidentiality.

Responding to the Findings

Finally, when students or alumni respond to your request for information by taking the time and
effort to contribute their opinions, you have an obligation to consider their responses seriously.
For more on how to do so, see Sections 8 and 9.

7-2
 PROGRAM ASSESSMENT CONSENT FORM

Department: ______________________________
Year: ___________

The ___________________ Department is conducting an assessment of its

undergraduate/graduate degree program. You are being asked to take part in an interview
concerning your experiences as a student in/graduate of that program. In all, we expect your
participation to take no more than one (1) hour to complete.

With your permission interviews will be audiotaped. Your participation is voluntary. The
audiotapes made during telephone interviews and transcripts of both telephone and e-mail
interviews will be available only to __________________, who will use them to obtain accurate
accounts of the interviews. All of your responses will be anonymous. Any reports produced as
a result of this study will not contain identifying material.

There are no reasonably foreseeable risks, discomforts, or benefits associated with participation in
this interview. Your participation is completely voluntary; you may stop participating at any time
prior to the completion of the project.

If you have any questions about your interview or the program assessment process, you may
contact __________________________ at the following address or phone number:
_________________________________. Copies of this form are available from this address as
well.

I have read the above and give my consent to participate in the study.

_____________________________________________ _________________
Signature Date

________________________________________________
Print name

7-3
 Section 8
REPORTING RESULTS

In order for evaluation efforts to inform the process of change, results from evaluation activities
will likely need to be made available to numerous players in the educational system. Findings can
be presented in a variety of ways although written reports are the most commonly used format for
disseminating evaluation findings. It is important that the evaluator know the potential audiences
of the report and can tailor the findings in a way that is accessible to each interested party. It
may be the case that the evaluator does not know the answer to many of the questions listed
below. If this is the case, ASK! Most often people welcome the opportunity to explain their
needs and interests. This section includes worksheets to help guide you in creating a report.

Some guiding questions about reporting results

Before you begin writing a report, you may want to revisit some of the planning work you have
done. In addition, you may want to step back from the evaluation and consider the big picture of
what it reveals. Some of the important questions to revisit and new questions to ask include the
following:

What were the original goals for conducting the evaluation?

Most programs conduct assessments for improving their program as well as meeting requirements
for accreditation or certification. It is important to keep these goals in mind while writing a report
of the findings so that the data is presented in a way that is aligned with the purpose of the
evaluation.

What issues/topics received many responses?

One of the ways an evaluator gains understanding of a phenomenon is by considering the quantity
and nature of the comments relating to various dimensions of an educational program. In addition
to noting the frequency with which participants address a topic, it is equally important that
evaluators note the descriptions of topics and issues. It may be the case that two out of twenty-
five participants experience discrimination. At first glance that may not seem like a major issue
since only two had the experience. However, if those two are the only two men in a group of
women, and if their discrimination related to their gender, then it is a significant finding.

Who are the audiences that would be interested in evaluation results?

Different audiences will be interested in different aspects of the evaluation. Policy makers and
administrators may be interested in program-level issues, such as whether the number of
credits/course required is adequate, or whether a course should be added or deleted. Faculty,
department leaders and program planners may be interested in whether the methods of instruction
in a program or course are facilitating learning. Students and alumni may be interested in whether
their voices were appropriately represented.

How is the audience going to use the data?

8-1
In deciding how to report the results of an evaluation, it is important to consider how the findings
will be used. If the results will be distributed to a funding source, a more detailed description of
the findings may be required. In contrast, if a department would like to present findings to its
faculty at a retreat or planning session, a summary of the findings on overhead slides may be more
appropriate.

What are your resources to do reporting?

While it may be that a formal report would most benefit the audience it may also be that the
available resources are not adequate for such a task. Or it may be that an interested party would
like to present some of the results at a conference. It is important to budget adequate funding for
reporting results early on in the planning process. Note that a common pitfall of new evaluators is
to leave inadequate time to report on results.

Who owns the product?

Clarifications about who owns the results of the evaluation should be made prior to the time
results are being disseminated. It may be the case that someone other than the author/evaluator
wants to reproduce some of the findings.

What do you do with "sensitive" information?

It may be the case that some of the results suggest dissatisfaction or difficulty with some aspect of
an educational program. While reporting on "sensitive" findings may be challenging, it can be
done in a way that opens the door for improvement. Names and places should be made non-
identifiable by using XXX's to fill the space and other information that reveals identity should be
eliminated. Comments that express dissatisfaction can be phrased in terms of "limitations" or
"areas for improvement."

Key components in the report

Consider the following major parts of a final report.

Introduction
In many cases if there is not an abstract or executive summary, people interested in the results of a
program evaluation will read only the introduction and the conclusion of a report. The primary
purpose of an introduction is to set the stage for the remainder of the report by highlighting the
purpose of the evaluation and the intended audiences for the findings. The introduction is also a
place to clarify contextual issues surrounding the study and discuss its limitations. It also should
provide a brief "reader's guide" to the report by listing what will discussed in each section.

Methods
This section is where methods of data collection are described. Included in discussion of methods
are the types of instruments or protocol used (such as survey), the procedure by which data was
collected, the number of respondents and/or issues with responding, and representativeness of
sample. This section also provides a brief discussion of the analysis process.

Findings/Themes

8-2
This section is where the findings are described and analyzed and usually comprises the bulk of
the report. The interpretation of the findings is as important as the presentation, and is perhaps
the most sophisticated aspect of doing an assessment. In writing a report on the findings, it is
important that the writer provide a guide for the reader in terms of relating the findings to the
questions. One way to organize the report is around the survey or interview questions. Another
way to organize a report is based on the major issues addressed through responses. In either
format, both quantitative and qualitative data can be used and interwoven to create a thorough
picture of the findings. If the primary methodology is interview or open-ended survey, it is
important that the report represents as authentically as possible the voice of the participants.
Tables and charts can be used to present quantitative data in an organized and readable fashion.

Conclusion
This is the section where the evaluator can synthesize different findings from the evaluation. In
summarizing the overarching findings, it is important to not introduce new material, but rather
focus on what has been presented before. Depending on your situation, it may be appropriate to
make recommendations based on the summary. Most often it is not appropriate to make
recommendations unless otherwise advised to do so by the coordinator of the evaluation.

Appendices
This is the place to attach copies of instruments, protocol, and other reference materials that will
help the reader understand the questions asked and other relevant content.

WORKSHEET FOR REPORTING RESULTS

8-3
Before you begin consider the following questions
1. What are your resource constraints for writing a report?
________________________________________________________________________
________________________________________________________________________

2. To whom should the results of the evaluation be given?

________________________________________________________________________
________________________________________________________________________

3. How will the results be used? (Examples: Internal feedback, documentation for
accreditation)
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

4. Who will have rights to reproducing the product?

________________________________________________________________________
________________________________________________________________________

Introduction
1. What was the purpose of the evaluation? What issues/ topics were evaluation planners
interested in?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

2. Are there contextual issues surrounding the evaluation that the reader should know about?
(Examples: Accreditation review coming up in one year; major changes in the program)
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

Methods/sample
1. What instruments were used? (Attach copy in Appendix)
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

2. What steps were taken to ensure confidentiality?

8-4
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

3. What was the sequence of events for data collection?

______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

4. Were there difficulties in garnering participation? If so, what was done to encourage
participation?
______________________________________________________________________________
______________________________________________________________________________

5. Compared with the whole, how representative was the responding sample?

Number in total Percent in total Number in Percent in

population population sample sample
Age

Ethnicity

Gender

GPA

8-5
Findings
1. What were the four biggest issues/topics that arose from the evaluation? (Example: Career
advising is limited)
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

2. Were there any "sensitive" issues that arose during the assessment?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

3. If you used multiple instruments, how does the information compare between instruments?
______________________________________________________________________________
______________________________________________________________________________

Conclusion
1. What are the key points you want the reader to leave thinking about?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

2. Based on the findings, do you have any recommendations?

______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

8-6
SAMPLE ASSESSMENT REPORT TABLE OF CONTENTS

8-7
 Section 9
ACTING ON RESULTS

Evaluation results are most likely to influence decision making when evaluators and top
administrators agree on the goals of the assessment and perceive information about outcomes as
an important source of feedback about organizational effectiveness (Weiss and Bucuvalas 1977).
The degree to which evaluators involve others in the process may affect the degree to which
interested parties act on results. Generally, the higher the level of participation, the greater the
involvement in acting on results. It is important to involve all the potential stakeholders
(administration, faculty, staff, students) in deciding how to act on results. The evaluator can play
a crucial role in facilitating action from findings.

In creating a plan to act on evaluation findings there are a number of potential challenges to be
aware of because they can affect participation.

1. Gaps between evaluators and faculty/staff/administrators. These gaps can be in

understanding of assessment objectives, communication about the process, and language that can
be alienating.

2. Faculty resistance. Faculty may resist if asked to devote effort to evaluation when it is not
rewarded with release from other responsibilities or if they do not understand the purpose of the
evaluation. Some faculty are hesitant to participate in evaluation activities because they assume
the evaluation will be a personal critique of their teaching rather than an evaluation of the entire
program. The evaluator should clarify that the purposes of program evaluation are to look at the
program as a whole.

3. Costs. Administrators may be wary of evaluation costs. One way to gain support is to use as
much existing data as possible. Find out what is available through campus records, etc.

4. Timing and follow through. Evaluation activities scheduled during busy times are likely to
be met with resistance by participants who are already busy. In addition, if there is no follow-
through with the findings, participants may feel that their time was wasted and be less inclined to
participate in the future.

5. Consider the work culture. On a final note, it is important to consider the culture in which
you are working. People will be more inclined to participate and act on evaluation activities that
are intended to improve educational programs if the social environment in which they are working
is one that supports and rewards high quality teaching and effective learning.

9-1
 WORKSHEET 1 FOR ACTING ON RESULTS

This worksheet is intended to begin the process of acting on results. Ideally, it should be
completed by those who will be involved in the change process. Once the changes are underway,
or at a designated time (such as the time identified on the timeline), faculty, staff and
administrators should revisit this plan and make necessary revisions. Worksheet 2 is intended as a
follow-up guide.

Topic or issue to Action Time-line Responsible Indicators of

address Statement Party Success
Example: Provide faculty Will complete round Faculty chairperson Each incoming
Inadequate training in one by end of fall and dean of college student will be
curricular requirements semester assigned an advisor,
advising will meet for ½ hour
each semester to
discuss courses.

9-2
 WORKSHEET 2 FOR ACTING ON RESULTS

Topic or Issue Actions Barriers to Revised Revised

to Address Achieved Completion Timeline Indicators
Example: Designed training Department chair Same Faculty trained by
Inadequate curriculum on sabbatical associate chair
curricular advising

9-3
 Section 10
RESOURCES

This section lists a variety of resources. It contains the following: campus-based assessment
resources; campus-based resources for student/alumni data; campus-based resources to which
students can be referred; names of faculty and staff experienced with assessment processes;
Internet resources on assessment; national sources of statistics and indicators; and literature
focused on assessment.

CAMPUS-BASED ASSESSMENT RESOURCES

Assessment Council - Office of the Provost
Contact: Eden Inoway-Ronnie
Rm 150 Bascom Hall
ph# 265-5975; email: [email protected]

Creating a Collaborative Learning Environment (CCLE)

Contact: Kathy Sanders or Chris Carlson-Dakes
Rm 271 Bascom
ph# 263-4259; email: [email protected]

LEAD Center
Contact: Susan Millar
1402 University Avenue
4th Floor
ph# 265-5943; email: [email protected]

The LEAD Center maintains a library of assessment and evaluation literature and resources
that can be searched by interested faculty and staff of the UW-Madison. LEAD Center
researchers can assist with identifying relevant resources and can be hired to help programs
conduct assessments.

Office of Quality Improvement (OQI)

Contact: Kathleen Paris
Rm 195 Bascom Hall
ph# 263-6856; email: [email protected]

Testing and Evaluation (T & E)

Contact: Allen Cohen
Rm 366 Educational Sciences Building
ph# 262-5863; email: [email protected]

10-1
University of Wisconsin Survey Center (UWSC)
Contact: James A. Sweet
Rm 2412 Social Sciences Building
ph# 262-2182; email: [email protected]

CAMPUS-BASED RESOURCES FOR STUDENT/ALUMNI DATA

Office of the Registrar
Contact: Dorothy Weidner
60 A. W. Peterson Building
750 University Avenue
Madison, WI 53706

Office of the Dean of Students

75 Bascom Hall
500 Lincoln Drive
Madison, WI 53706
608-5700

Office of Budget, Planning and Analysis/Data Digest

100 Bascom Hall
Madison, WI 53706
http://www.wisc.edu/obpa

Graduate School
http://www.wisc.edu/grad/

CAMPUS-BASED RESOURCES/ADVOCATES FOR STUDENTS

Campus Assistance Center
263-2400
420 N. Lake Street
Madison, WI 53706
www.wisc.edu/cac

University Health Services

Clinical
262-3016
1552 University Avenue
Madison, WI 53706

Counseling
262-1744
905 University Avenue
Madison, WI 53706
www.uhs.wisc.edu

10-2
FACULTY RESOURCES
Wayne Becker, Professor
Department of Botany
Rm B115 Birge Hall
262-5833
[email protected]

Sarah K. A. Pfatteicher, Director of Assessment

College of Engineering
1402 University Avenue
265-5925
[email protected]

Barbara Duerst, Clinical and Experiential Instructor

School of Pharmacy
Rm 5241 Chamberlain Hall
262-5703
[email protected]

Wendy Way, Professor

School of Human Ecology
Rm 120 Human Ecology Bldg.
262-1963
[email protected]

If you would be interested in having your name included here in future editions please
contact Dianne Bowcock (see Section 1, pg. 4)

NATIONAL SOURCES OF STATISTICS AND INDICATORS

Chronicle of Higher Education, annual Almanac issue (appears in late summer)

Digest of Education Statistics and The Condition of Education, each produced annually by the
National Center for Education Statistics (U.S. Department of Education)

Science & Engineering Indicators, produced annually by the National Science Board (National
Science Foundation)

10-3
INTERNET RESOURCES ON ASSESSMENT
UW-Madison, Assessment Council: http://publications/provost/assess/manual

Alverno College: http://www.alverno.edu

This is the home page of Alverno College, Milwaukee. Alverno’s ability-based learning program
is internationally recognized and is mentioned in almost all of the web sites on assessment.
Provides information on Alverno’s eight abilities, which include communication, analysis,
problem solving, valuing, social interaction, global perspective, effective citizenship, and aesthetic
response. NOTE: To find assessment information in this site, follow the easy-to-find links.

American Association for Higher Education: http://www.aahe.org

An excellent higher education assessment resource site. Provides information about current
publications; the AAHE Assessment Forum (the primary national network connecting and
supporting higher education stakeholders involved in assessment); and articles about assessment
from the AAHE Bulletin.

Assessment and Evaluation on the Web: http://ericac.net/INTBOD.STM

This site covers everything from pedagogy in educational measurement to assessment
organizations.

LITERATURE FOCUSED ON ASSESSMENT

Apps, J. W. (1985). Improving practice in continuing education. San Francisco : Jossey-
Bass.
Provides a philosophical bent and engages the reader in a process of reflection about the
teaching of adults. It provides a guide for conducting a critical analysis of programs.

Banta, T.W., Lund, J.P., Black, K.E., Oberlander, F.W. (1996). Assessment in practice.
San Francisco: Jossey-Bass.
A very thorough all-purpose guide to assessment. Uses case studies to illustrate concepts.

Caffarella, R. S. (1995). Program planning for adults: a comprehensive guide for adult
educators, trainers, and staff developers. San Francisco: Jossey-Bass.
Focuses on planning educational programs for adults. It is designed to lead the reader
through the process of planning a program and includes a chapter on evaluation relating to
program planning.

LeCompte, J.P., & Preissle, J. (1993). Ethnography and qualitative design in educational
research. (2nd. ed.) San Diego, Academic Press.
An excellent introductory book. It is a scholarly book well-grounded in philosophical,
theoretical and empirical literature (and has a 30-page bibliography). It is a
methodological book and chapters are structured around primary procedural topics
(researcher role, sampling, data collection, analysis).

10-4
Marshall, C., & Rossman, G.B. (1989). Designing qualitative research. Beverly Hills:
Sage.
Easy to read. It uses examples of actual research studies as examples of pragmatic
challenges that arise in designing and conducting qualitative inquiry.

Merriam, S.B., & Simpson, E. L. (1995). A guide to research for educators and trainers of
adults. Malabar, FL: Krieger.
Guides the reader through all the various steps of conducting an evaluation or research
study and provides information about the various philosophical frameworks which underpin
choices about methods and analysis. Also addresses ethical issues

Miles, M.B., & Huberman, A.M. (1984). Qualitative data analysis: A source book of new
methods. Newbury Park, CA: Sage.
Provides a format for analyzing and reporting qualitative data.

Patton, M.Q. (1997). Utilization-focused evaluation: the new century text. Thousand
Oaks, CA: Sage.
Both practical and theoretical. It focuses on how to conduct program evaluations with each
chapter containing a review of relevant literature and actual case examples.

Patton, M.Q. (1990). Qualitative evaluation and research methods. Beverly Hills: Sage.
Provides an overview of all major aspects of qualitative design and analysis, with an
evaluation focus.

Struass, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory
principles and techniques. Newbury Park, CA: Sage.
Describes analysis strategies in grounded theory studies. Easy to read.

Wolcott, H. F. (1990). Writing up qualitative research. Newbury Park, CA: Sage.

Easy-to-read practical advice for writing up findings from qualitative studies.

Worthen, B. R., Sanders, J. R., and Fitzpatrick, J.L. (1997). Program evaluation:
alternative approaches and practical guidelines. NY: Longman.
Presents a variety of approaches to doing program evaluation with practical guidelines for
planning and implementing program evaluations. Also discusses analyzing and reporting on
findings.

10-5