Measuring Principal Effectiveness in a Performance-Based System

Xiu Cravens Patrick Schuermann

NCCTQ Conference November 6, 2007

Agenda for Todays Session

Present the results of a comprehensive review of leadership evaluation instruments being used by districts and states Introduce a new evaluation system that is being pilot-tested

The Case for Leadership Assessment

Most school leadership improvement focuses on professional development, mentoring, licensing policies, and standards. Minimal attention has been paid to assessment, feedback, and subsequent action. Leadership assessment and feedback is an important missing link to improving and strengthening school leadership.

What and How do States and Districts Assess Leadership?

What is the state of principal leadership assessment for K-12 school principals? What is assessed? How are evaluations conducted?

The Sample Frame

A total of 74 districts in 43 states and the District of Columbia

Member Districts of the Council of the Great City Schools Districts participating in the Wallace Foundation Leadership for Educational Achievement in Districts (LEAD) project States participating in the Wallace Foundation State Action for Education Leadership Project (SAELP)

66 evaluations were obtained and 65 were used in the analysis

Content Analysis

An iterative and deductive process to develop a coding scheme based on actual evaluations Four general categories, each with multiple subcategories for the coding matrix:

Management (i.e., manages school facilities, follows fiscal policies, follows rules and regulations) External Environment (i.e., promotes the school, engages with parents,) School and Instruction (i.e., creates learning climate, implements vision, monitors instruction) Personal Characteristics (i.e., uses ethical behavior, uses listening skills, applies conflict resolution),

Usage Analysis

Propriety Standards

Access to evaluation results Conflict-of-interest statements Defined purpose of the assessment Evaluator credibility Source of the assessment Burden of the evaluation (length and frequency) Who is assessed Availability of assessor training Role of national/state standards Psychometric properties Instrument format (item scale)

Utility Standards

Feasibility Standards

Accuracy Standards

Number of Evaluation Items Per Instrument

Number of Items Per Instrument
35

Number of Instruments

30 25 20 15 10 5 0
0-19 20-39 40-59 60-79 80-99 100-119 120-139 140-159 160-179 180-199

General Content Focus

(1) School and Instruction
Frequency (Number of Instruments) Frequency (Number of Instruments)

(2) Personal Characteristics

30 25 20 15 10 5 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category

30 25 20 15 10 5 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category

(3) Management
35 30 25 20 15 10 5 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category
Frequency (Number of Instruments)

(4) External Environment

Frequency (Number of Instruments)

60 50 40 30 20 10 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category

A Closer Look at the Sub-Categories

M-General Management I-Implement Vision E-Parents and Community I-Data-based Decision P-Communication Skills I-Teacher Prof. Development P-Ethical Behavior M-Human Resource M-Finance P-Decision Making I-Instruction P-Diversity I-Climate of Order I-Climate of Learning I-Shared Vision I-Knowledge of Learning I-Professional Learning I-Curriculum I-Focus on Learning M-School Facility P-Understand "Politics" I-Teacher Evaluation P-Problem Solving I-Student Support Services I-Expectation (Staff) E-Resources I-Expectation (students) I-Technology Use I-Measures and Standards I-Staff Team E-Promote School I-Focus on Achiev. Gap P-Manage Change P-Risk Taking I-Time on Task I-System Alignment

0.08

0.07

7% of the instrument
0.06

0.05

0.04

0.03

0.02

0.01

0.00

Most assesses sub-categories:

general management implementing vision relationship with parents and communities data-based decisions communication skills alignment of curriculum maximizing time on task encourages risk taking and creativity managing change focus on achievement gap

Least assessed sub-categories:

Each subcategory, even the top-ranked ones, represent only a small fraction of each instrument (most less than 5%)

How are evaluations conducted?

(Based on 44 instruments with accompanying documentation)

Utility Mostly formative, for professional growth and development Usually assessed by direct supervisors only Assessment instruments are designed at the district level, with some using state templates Accuracy Little mentioning of assessor training ISLLC standards are used by 25% of the districts/states as a reference framework No documentation of psychometric properties

From the sampled principal leadership assessment instruments, we find:

Various levels of specificity Wide spread of assessed areas Limited depth Locally designed procedures for growth and development More on knowledge and skills, less on behavior Lack of consistent focus on school performance as measured by student achievement

Assessing Learning-Centered Leadership

The Research Team:

Andrew C. Porter (University of Pennsylvania) Joseph Murphy, Ellen Goldring, and Stephen N. Elliott (Vanderbilt University)

A three-year project funded by the Wallace Foundation (2005-2008)

Learning-Centered Leadership

Leaders should be assessed on leadership behaviors associated with student learning. Learning-centered leadership is leadership for student performance. Learning-centered leadership is the framework for our leadership assessment system.

2007

Assessing Learning-Centered Leadership: The VAL-ED vision. and circumstances, Works well in a variety of settings

Is construct valid, Is reliable, If feasible for widespread use, Provides accurate and useful reporting of results. Is unbiased, Yields a diagnostic profile for summative and formative purposes. Can be used to measure progress over time in the development of leadership, and Predicts important outcomes.

Purpose & Uses

The VAL-ED can be used as part of a comprehensive assessment of the effectiveness of a leader's behaviors. The VAL-ED reports principal performance through

Norm-referenced scores and Criterion-reference scores.

VAL-ED can be used annually or more frequently to:

Facilitate a data-based performance evaluation, Measure performance growth, and Guide professional development.

The Need for a Valid and Reliable Principal Leadership Assessment Tool

Virtually every school district in the United States (N = 14,000 school districts and over 90,000 schools) requires some form of evaluation of its principals. Many states and districts have developed their own leadership assessment tools. Our analysis of the assessments with large urban school districts indicates that few have a conceptual framework based on how leaders improve student learning, nor have they been validated for their intended uses.

Our Conceptual Model

Leadership Behavior Framework

Core Components
High Standards for Student Performance Rigorous Curriculum (content) Quality Instruction (pedagogy) Culture of Learning & Professional Behavior Connections to External Communities Systemic Performance Accountability

Key Processes
Planning Implementing Supporting Advocating Communicating Monitoring

Definitions of Core ComponentsStudent Learning There are individual, team, High Standards for

and school goals for rigorous student academic and social learning.

Rigorous Curriculum (content) There is ambitious academic content provided to all students in core academic subjects. Quality Instruction (pedagogy) There are effective instructional practices that maximize student academic and social learning. Culture of Learning & Professional Behavior There are integrated communities of professional practice in the service of student academic and social learning. There is a healthy school environment in which student learning is the central focus. Connections to External Communities There are linkages to family and/or other people and institutions in the community that advance academic and social learning. Performance Accountability Leadership holds itself and others responsible for realizing high standards of performance for student academic and social learning. There is individual and collective responsibility among the professional staff and students.

Definitions of Key Processes shared direction and coherent policies, practices, and PlanningArticulate

procedures for realizing high standards of student performance.

ImplementingEngage people, ideas, and resources to put into practice the activities necessary to realize high standards for student performance. SupportingCreate enabling conditions; secure and use the financial, political, technological, and human resources necessary to promote academic and social learning. AdvocatingPromotes the diverse needs of students within and beyond the school. CommunicatingDevelop, utilize, and maintain systems of exchange among members of the school and with its external communities. MonitoringSystematically collect and analyze data to make judgments that guide decisions and actions for continuous improvement.

Assessing Learning-Centered Leadership: Overview of the Instrument

The instrument consists of 72 items defining six core component subscales and six key process subscales. Principal, Teachers, & Supervisor provide a 360 degree, evidenced-based assessment of leadership behaviors. Respondents rate effectiveness (1= Ineffective to 5 = Outstanding) of 72 behaviors. Each respondent rates the principals effectiveness after indicating the sources of evidence on which the effectiveness is rated. Two parallel forms of the assessment facilitate measuring growth over time.

Directions for Completing Rating Scale

Teacher Response Form
Name: _____________________ School District: _____________________ Years as Principal of this School: _________ Date: ________________________________ School: ________________________________ School ID#: ________________________________

Directions: The Vanderbilt Assessment of Leadership in Education (VAL-ED) measures the effectiveness of a principals key leadership behaviors that influence teacher performance and student learning. You will be asked to make effectiveness ratings for each of 72 leadership behaviors based on evidence from the current school year. 1. Read each item describing a leadership behavior. In some cases, the principal may not have actually performed the behavior, but he or she has ensured that it was done by others in the school. Either way the behavior should be rated. Check () the key Sources of Evidence you use for the basis of your assessment. Note, at least one source of evidence must be checked for an item before you make an Effectiveness rating. If you check No Evidence, then Ineffective must be marked in the Effectiveness column. If you check any sources of evidence other than No Evidence, always make an effectiveness rating. The number of Sources of Evidence checked is not necessarily indicative of the effectiveness rating. Circle the 1 to 5 Effectiveness Rating to indicate how effectively the behavior was performed.

2.

3.

4.

An Example Set of Responses

Sources of Evidence
Check Key Sources of Evidence

Effectiveness Rating
Circle One Number to Indicate How Effective or Check DK

No Evidence

Personal Observations

School Documents

School Projects or Activities

Other Sources

Ineffective

Minimally Effective

Satisfactorily Effective

Highly Effective

Dont Know

Outstandingly Effective

Reports from Others

Leadership Behaviors

How effective is the principal at ensuring the school

1. plans for a culture of learning that serves all students. 2. evaluates the rigor of the curriculum. 1 2 3 4 5

For Item #1, which states How effective is the principal at ensuring the school plans for a culture of learning that serves all students, the respondent checked two sources of evidence for the basis of her evaluation of effectiveness and then circled a rating of 1 to indicate she perceived the principal as being ineffective regarding this leadership behavior. For Item #2, which states How effective is the principal at ensuring the school evaluates the rigor of the curriculum, the respondent checked one source of evidence for the basis of her evaluation and then circled a rating of 3 to indicate she perceived the principal as being satisfactorily effective regarding this leadership behavior.

The Development of VAL-ED

Three phases:

Phase 1 Leadership conceptualization and assessment system development Phase 2 Field testing the behavior rating scale and exploring its properties Phase 3 Dissemination of results and products

Interpretation of Rating Scale Results Analysis Descriptive

Total Score Core Components Subscale Scores Key Process Subscale Scores

Norm-Referenced Profiles Principal Teacher Supervisor Criterion-Referenced Profiles Advanced Proficient Novice

Results: Principals Self-Ratings of Core Components

2007

Results: Principals Self-Ratings of Key Processes

2007

Results: Teachers Median Ratings of Core

Components

2007

Results: Supervisors Ratings of Core Components

2007

Integrated Results Across Respondents

2007

Investigating the Psychometric Properties of the VAL-ED

Item and response scale development (completed) Based on review of learning-centered leadership literature and alignment to ISLLC standards Critiqued by education leaders and leadership researchers Item sorting study (completed) Established content validity by asking education leaders to sort the items into 36 cells Pilot/cognitive interviews (completed) Two rounds of cognitive interviews in three districts each Three respondents evaluated the format and items Bias review (fall 2007) Submit to urban districts to evaluate language

Investigating the Psychometric Properties of the VAL-ED

Cognitive Interviews of Online Instrument (fall 2007) Nine-school pilot test (completed) Estimated reliability of each of 12 scales Established construct validity through factor analysis Established face validity through questions to respondents 300-school field test (spring 2008) Replicating reliability and construct validity tests from pilot Conducting differential item functioning to determine biases Collecting concurrent information on leadership effectiveness and correlating with VAL-ED Establishing norms Proficiency standard setting (2008) Using body-of-work approach with 24 education leaders

Comparing Sampled Evaluations with the Learning-Centered Core Components

Rigorous Curriculum
Frequency (Number of Instruments)

60 50 40 30 20 10 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100%

Per Instrument Content Weight on This Category
Frequency (Number of Instruments)

Culture of Learning and Professional Behavior

60 50 40 30 20 10 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100%
Per Instrument Content Weight on This Category

Quality Instruction
Frequency (Number of Instruments)

60 50 40 30 20 10 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100%

Per Instrument Content Weight on This Category

Comparing Sampled Evaluations with the Learning-Centered Core Components

High Standards for Student Learning
40 35 30 25 20 15 10 5 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category
Frequency (Number of Instruments)

Connections to External Communities

40 35 30 25 20 15 10 5 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category
Frequency (Number of Instruments)

Performance Accountability
40 35 30 25 20 15 10 5 0 0-15% 16-30% 31-45% 46-60% 61-75% 76-90% 91-100% Per Instrument Content Weight on This Category
Frequency (Number of Instruments)

Effective leadership assessment can be an integral part of a standards-based accountability system. When designed accurately, executed in a proactive manner, and properly implemented, it has the power to improve school performance and to identify leaders of the future.

Supporting Research & Publications

Goldring, E., Porter, A.C., Murphy, J., Elliott, S.N., & Cravens, X. (2007, March). Assessment learning-centered leadership: Connections to research, professional standards, and current practice. New York, N.Y.: Wallace Foundation. Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (2007). Leadership for learning: A research-based model and taxonomy of behaviors. School Leadership & Management, 27 (2), 179-201. Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (in press). Leaders for productive schools. In M. Brundrett & M. Crawford (Eds.), Developing school leaders: An international perspective, London: Routledge. Murphy, J., Elliott, S.N., Goldring, E.B., & Porter, A.C. (2006). Learning-centered leadership: A conceptual foundation. New York, NY: Wallace Foundation. www.wallacefoundation.org/ELAN/TR/KnowledgeCategories/DevelopingLeader s/PerfMeasurement/PerfMeasurementResource.ConceptualFoundation.htm Porter, A.C., Goldring, E.B., Murphy, J., Elliott, S.N., & Cravens, X. (2006). A framework for the assessment of learning-centered leadership. New York, NY: Wallace Foundation. www.wallacefoundation.org/ELAN/TR/KnowledgeCategories/DevelopingLeader s/PerfMeasurement/PerfMeasurementResource.ConceptualFoundation.htm

2007

To learn more Visit: http://www.val-ed.com Contact : Joseph Murphy Vanderbilt University 615-322-8038 (office)
[email protected]