Evaluating the Impact of Charter Schools on Student Achievement: A Longitudinal Look at the Great Lakes States

Appendix G

June 2007

EPRU |

EDUCATION POLICY RESEARCH UNIT

Education and the Public Interest Center
School of Education, University of Colorado Boulder, CO 80309-0249 Telephone: (303) 492-8370 Fax: (303) 492-7090 Email: [email protected] http://education.colorado.edu/epic

Division of Educational Leadership and Policy Studies College of Education, Arizona State University P.O. Box 872411, Tempe, AZ 852872411 Telephone: (480) 965-1886 Fax: (480) 965-0303 E-mail: [email protected] http://edpolicylab.org

Education Policy Research Unit

The policy brief is available online at: http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU.pdf

This research was made possible by funding from the Great Lakes Center for Education Research and Practice

Appendix G: Student Achievement in Wisconsin Charter Schools

Appendix G Student Achievement in Wisconsin Charter Schools

The Wisconsin Charter Program was established in 1993 with authorization for 10 school districts to establish up to 2 charter schools for a total of 20 statewide. Thirteen charter schools were created under this law during that time. The charter school law underwent many revisions. In 1995, chartering authority was extended to all school boards statewide and the cap on the total number of charter schools was removed. In 1997, the state legislature extended chartering authority in Milwaukee to the chancellor of the University of Wisconsin-Milwaukee, the Milwaukee Area Technical College, and the Common Council of the city of Milwaukee (Wisconsin Charter School Association, 2007).1 Although there are a number of studies on charter schools in Wisconsin, there are few empirical studies or evaluations that look at the academic performance of its charter schools. An early report by Stuiber, Swenson, Normandin, and Varana (1998) provided a performance evaluation of Wisconsins charter schools open during the 1997-98 school year.2 The report compared charter-school programs and policies to traditional district schools. On average, charter school students scored higher on standardized tests than other district students, but test results were only available for a half dozen schools. A study by Witte, Weimer, Schlomer, and Shober (2004) found that charter schools in operation for more than one year generally were more likely to meet state standards than noncharter students.3 This study used in innovative approach for comparing charter school and district results, but only two years of data were considered (2000-01 and 2001-02) and no analyses of high school grades was pursued because most of these were deemed to be at-risk schools. Loveless (2003) analyses of charter school performance in 10 states using the states own criteria for rating schools found that 11 percent of charter schools were rated as failing in Wisconsin compared with 3.5 percent of traditional public schools that were rated as failing.4

Data Sources, Outcome Measures, and Methods for Analysis

In Wisconsin, students demonstrate their progress toward achieving academic standards in reading, language arts, mathematics, science, and social studies through their performance on the Wisconsin Student Assessment System (WSAS). The WSAS includes the Wisconsin Knowledge and Concepts ExaminationsCriterion-Referenced Test (WKCE-CRT), taken by nearly all students in grades 3-8 and 10. The WKCE-CRT is a large-scale, standardized achievement test administered using carefully defined directions, time limits, materials, and scoring procedures for all test takers to ensure uniform test-taking conditions. Many of the demographic variables used to create the predicted values were obtained from the Common Core of Data at the National Center for Education Statistics (NCES).5 From this data source were obtained variables covering school enrollment, ethnicity, free and reduced lunch, and urbanicity or locale. A variable designating charter school status also was used from this data set to distinguish the charter schools in the Page 1 of 8

http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Appendix G: Student Achievement in Wisconsin Charter Schools state. From the DPI and Office of Educational Accountability Web site, the mean schoollevel scaled scores for reading and math were obtained for the period 2001 through 2005 for grades 4, 8, and 10, including the number of test takers within each school that reported test results.6 A scale score is a score on a numeric scale with intervals of equal size. The scale is applied to all students taking the WKCE-CRT in a particular subject at a particular grade level. The scale score makes possible the comparison of scores from different groups of students or individualsor schoolsfrom year to year. Each content area is scaled separately. Therefore, the scale scores for one content area cannot be compared with the scale scores from another. Table 1. Test Data Used in Analyses, by Year, Grade, and Subject
2000-01 Grade 4 Grade 8 Grade 10 Reading & Math Reading & Math Reading & Math 2001-02 Reading & Math Reading & Math Reading & Math 2002-03 Reading & Math Reading & Math Reading & Math 2003-04 Reading & Math Reading & Math Reading & Math 2004-05 Reading & Math Reading & Math Reading & Math

The data set created for Wisconsin was complete in terms of test data and the demographic variables required for the regression analysis. No imputation of missing values was conducted, although there were a number of instances where schools were dropped from specific analyses because they had too few test takers (test results in Wisconsin are not reported if there are fewer than 5 test takers in a specific group). Variables Used to Create the Predicted Values for Each School Table 2 displays the variables used in developing the residual gain score analysis for Wisconsins charter schools. The predicted values and residual scores were created using an ordinary least squares multiple regression procedure, in the form of the linear equation given in Equation 1 Yi =a + b1 MINORITYi +b 2LOWINCOMEi +b3SPEDi +b4URBANICITYi +i Where is the predicted value for a schools mean scaled score i, expressed in terms of the constant a of the intercept term, and where MINORITYi is the percentage of minority students (minus Asians) for given school i, where LOWINCOMEi is the percentage of students receiving free or reduced lunch for given school i, and where URBANICITYi is the value (i.e., from 1-8) indicating population density for school i. In this equation, the regression coefficients (bs), also referred to as the partial regression coefficients, represent the independent contributions of each independent variable to the prediction of the dependent variable . Essentially, the predicted values indicate how the

http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 2 of 8

Appendix G: Student Achievement in Wisconsin Charter Schools school is expected to score based on how other schools in the state with similar demographics have performed on the same test. Table 2. Variables Included in Residual Gain Score Analysis for Wisconsin Variable Description

Mean Scale Score School-level mean scale score on the WKCE-CRT (dependent variable) Percent Minority Percent Low Income Special Education Urbanicity (locale) Percentage of nonwhite and non-Asian-American students enrolled Percentage of students receiving free or reduced lunch Percentage of students with disabilities Rating from 1-8, indicating population density

The residual is the difference between the actual score and the predicted score. If the residual score is negative, then the school is doing worse than expected. If the residual score is positive, the school is performing better than expected. In the data tables, there are rows that contain the average annual change scores, which indicated the relative direction the school is moving. For example, the school may have all negative residual scores; but if they are becoming less negative over time, the average annual change score will be a positive number. The average annual change score is computed for patterns of actual, predicted, and residual scores across time by subtracting the first score from the most recent and dividing by the number of observations (e.g., years) minus 1 (i.e., N-1). It is important to note that the results presented in Tables 3 and 4 are aggregate findings across all charter schools with available data. When calculating the aggregate results, the data were weighted by the relative number of test takers per school. For example, if a large school has extremely positive results, they will carry more weight than a small school with less positive results. Some schools were excluded because they either had too few test takers or else they did not report demographic data. The data for Wisconsin was surprisingly complete. In 2001, the trends captured 84 percent of the schools, and for the last 2 years of the trends we typically were able to include 100 percent of the schools that had students taking specific tests.

Actual Performance and Residual Gains for All Charter Schools

The data presented in Table 3 and Figure 1 illustrate the overall results aggregated for all Wisconsin charter schools from 2001 through 2005 for grades 4, 8, and 10 on WKCE-CRT math and reading assessments for which data were available. The dashed line in Figure 1 illustrates the actual or observed scale scores for charter schools. Based

http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 3 of 8

Appendix G: Student Achievement in Wisconsin Charter Schools

Table 3. Wisconsin Aggregate Results by Grade, Subject, and Year

School Name Year
2001 2002 2003 2004 2005 Average annual change

Math Schools
25 27 31 34 36

Reading Predicted
627.03 631.49 628.16 627.07 629.49 0.61 679.10 675.91 683.81 675.88 684.74 1.41 718.65 711.51 714.30 716.83 711.31 -1.83

Students
923 1,067 1,039 1,089 1,121

Actual
631.31 626.62 624.86 623.57 623.32 -2.00

Residual
4.28 -4.86 -3.30 -3.51 -6.16 -2.61 3.26 6.17 3.75 4.07 2.53 -0.18 -22.61 -16.09 1.18 -3.06 -7.55 3.77

Schools
25 27 31 34 36

Students
911 1,066 1,029 1,089 1,121

Actual
650.37 641.98 638.22 637.37 635.41 -3.74

Predicted
645.50 647.66 642.41 642.23 642.90 -0.65 670.24 668.82 671.22 665.61 673.42 0.80 702.84 690.35 701.06 695.89 698.03 -1.20

Residual
4.87 -5.68 -4.19 -4.86 -7.49 -3.09 2.64 2.45 0.20 -0.36 1.88 -0.19 -17.01 -8.07 1.30 -4.22 -2.87 3.54

Grade 4

Grade 8

2001 2002 2003 2004 2005

23 28 28 34 37

1,634 1,791 1,812 1,916 2,025

682.36 682.08 687.56 679.95 687.27 1.23

23 28 28 34 37

1,637 1,799 1,808 1,918 2,022

672.88 671.27 671.42 665.26 675.30 0.61

Average annual change

Grade 10

2001 2002 2003 2004 2005

15 21 17 16 19

333 511 382 404 490

696.03 695.43 715.48 713.77 703.75 1.93

15 21 17 16 19

329 515 383 404 493

685.82 682.28 702.36 691.67 695.17 2.34

Average annual change

800

4th Grade Math

40 20

800 750

4th Grade Reading

40 20

Grade 4

750 Scale Score 700 650 600 -20 550 500 2001 2002 2003 2004 2005 -40 0 Residual

Scale Score

700 650 600 0

-20 550 500 2001 2002 2003 2004 2005 -40

Residual

800

8th Grade Math

40 20

800 750

8th Grade Reading

40 20

Grade 8

750

Scale Score

Scale Score

700 650 600 0

700 650 600 0

Residual

-20 550 500 2001 2002 2003 2004 2005 -40

Residual

-20 550 500 2001 2002 2003 2004 2005 -40

800

10th Grade Math

40 20

800 750

10th Grade Reading

40 20

Grade 10
Mean Scaled Score

750 Scale Score 700 650 600 -20 550 500 2001 2002 2003 2004 2005 -40 0 Residual

Scale Score

700 650 600 0

Residual

Residual

-20 550 500 2001 2002 2003 2004 2005 -40

Figure 1. Wisconsin Aggregate Results: Residual Scores and Mean Scale Scores
http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 4 of 8

Appendix G: Student Achievement in Wisconsin Charter Schools on these trend lines, performance for students in Wisconsins charter schools is relatively flat indicating that the mean scale score for charter schools is not improving over time. The residual lines are also rather flat, except at grade 10 where noticeable improvements are evident. The average annual change in residual scores is negative for 4th grade, close to zero for 8th grade and positive at 10th grade where the charter school aggregate results improve dramatically in 2002. This increase in 2002 can largely be explained by the addition/inclusion of 6 more charter schools serving high school students. The aggregate results across all charter schools is still lower than state averages. Nonetheless, state figures should not be used to evaluate charter schools since the state results include a large portion of schools that are not similar in terms of student demographics to charter schools. Our approach that uses residual gains analysis, however, creates demographically similar comparison groups for each charter school.

Actual Performance and Residual Gains for Same Cohort of Schools

As noted earlier, the changes over time depicted in the results from Figure 1 may be due to the addition of new charter schools. The number of schools and the number of students included in each set of results is also indicated in Table 3. Between the first and last test dates, between 7 and 19 charter schools were added to the aggregate results. Therefore, changes in aggregate results may be due to the inclusion of new schools. To control for this we tracked a subset of the same charter schools that had test data available for all years. The results from these aggregate results for cohorts of the same schools over time are illustrated in Table 4 and Figure 2. For grades 4 and 8, the results for the cohorts of schools are rather similar to the results for all schools. The data for grade 10 had more gaps in it, so the best cohort we could construct of same schools started in 2002 instead of 2001. At grade 10, we were able to track 8 charter schools over time. Many of the small high schools or presumably many of those schools serving at-risk students did not have test data available for each year of the cohort and thus were not included. The cohort results for grades 4 and 8 are more robust and include a relatively large proportion of the all schools with valid test data. The residual scores for grades 4 and 8 are all positive, but over time become slightly less positive for grade 4. For grade 10, the residual scores are initially negative and improve slightly over time.

http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 5 of 8

Appendix G: Student Achievement in Wisconsin Charter Schools

Table 4. Wisconsin Results from Cohorts of Same Schools Tracked Over Time
Year Math Schools
21 21 21 21 21

Reading Predicted
626.89 632.66 628.94 629.20 631.35 1.12 678.15 674.88 680.71 674.43 682.05 0.97

Students
884 839 765 728 658

Actual
631.98 635.49 633.18 634.91 634.72 0.68

Residual
5.09 2.83 4.24 5.71 3.36 -0.43 4.66 7.79 6.58 7.46 5.57 0.23

Schools
21 21 21 21 21

Students
873 836 756 728 658

Actual
650.66 651.15 646.11 647.54 645.21 -1.36

Predicted
645.36 648.26 643.18 643.41 644.42 -0.24 669.46 668.06 668.85 664.51 671.51 0.51

Residual
5.30 2.89 2.93 4.13 0.79 -1.13 3.70 3.54 1.67 1.05 4.33 0.16

Grade 4

2001 2002 2003 2004 2005

Average annual change

Grade 8

2001 2002 2003 2004 2005

16 16 16 16 16

1,577 1,601 1,536 1,566 1,541

682.80 682.67 687.29 681.89 687.62 1.20

16 16 16 16 16

1,583 1,609 1,534 1,567 1,539

673.16 671.60 670.52 665.56 675.84 0.67

Average annual change

Grade 10

2001 2002 2003 2004 2005 8 8 8 8 320 265 320 303 693.78 708.98 710.38 704.01 3.41 697.80 706.42 711.85 709.46 3.88 -4.02 2.56 -1.47 -5.45 -0.48 8 8 8 8 324 265 320 300 681.70 700.43 690.11 696.95 5.08 682.75 697.66 692.50 696.84 4.70 -1.05 2.78 -2.39 0.11 0.39

Average annual change

800

4th Grade Math

40 20

800 750

4th Grade Reading

40 20

Grade 4

750 Scale Score 700 650 600 -20 550 500 2001 2002 2003 2004 2005 -40 0 Residual

Scale Score

700 650 600 0

-20 550 500 2001 2002 2003 2004 2005 -40

Residual

800

8th Grade Math

40 20

800 750

8th Grade Reading

40 20

Grade 8

750 Scale Score 700 650 600 -20 550 500 2001 2002 2003 2004 2005 -40 0 Residual

Scale Score

700 650 600 0

-20 550 500 2001 2002 2003 2004 2005 -40

Residual

800

10th Grade Math

40 20

800 750

10th Grade Reading

40 20

Grade 10

750
Scale Score

Scale Score

700 650 600 0

700 650 600 0

Residual

Residual

Mean Scaled Score

Residual

-20 550 500 2001 2002 2003 2004 2005 -40

550 500 2001 2002 2003 2004 2005

-20 -40

Figure 2. Wisconsin School Cohort Results: Residual Scores and Mean Scale Scores
http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 6 of 8

Appendix G: Student Achievement in Wisconsin Charter Schools

Summary of Findings
The evaluation questions in this study were: (1) How does student achievement in charter schools compare to demographically similar public schools? (2) Are charter schools an effective strategy for improving student achievement over time? Results for these two questions are summarized in Tables 5 and 6. Table 5 presents a cross-sectional comparison of six mean test residuals by grade for Wisconsin charter schools using the most recent year of available data (2005). Results revealed 77 instances in which charter school residuals are positive (i.e., student achievement is higher than expected) and 107 instances in which they are negative (i.e., student achievement is lower than predicted). In other words, only 40 percent of the comparisons favored charter schools. Therefore, one can conclude the charter schools in Wisconsin are currently performing at levels that are slightly lower than predicted and lower than demographically similar traditional public schools. Table 5. Cross-Sectional Comparison Test Residuals by Grade for Charter Schools Using the Most Recent Year of Available Data Grade 5 Math
# Schools with Positive Residuals # Schools with Negative Residuals

Grade 5 Reading 14 22

Grade 8 Math 18 19

Grade 8 Reading 20 17

Grade 11 Math 4 15

Grade 11 Reading 5 14

Totals 77 107

16 20

Table 6 presents a comparison of the average annual change in test residuals by grade for Wisconsin charter and cohort charter schools over five years. Results revealed that the residuals for charter schools overall are relatively unchanged overtime. The average annual change in residuals across all schools was +0.20. On the other hand, when we look at a cohort of same schools over time, the average annual change score in residuals is -0.20. This means that over a five year period, the trend in student achievement is relatively unchanged.

http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 7 of 8

Appendix G: Student Achievement in Wisconsin Charter Schools Table 6. Comparison of Average Annual Change (AAC) in Test Residuals by Grade for Charter Schools and Charter School Cohorts Over Five Years (2001 to 2005)
Grade 5 Math Grade 5 Reading Grade 8 Math Grade 8 Reading Grade 11 Math Grade 11 Reading Mean AAC across all tests

Average Annual Change in Residual Scores for All Schools with Available Data Average Annual Change in Residual Scores for Cohort of Same Schools

-2.61

-3.09

-0.18

-0.19

3.77

3.54

0.20

-0.43

-1.13

0.23

0.16

-0.48

0.39

-0.21

As the results in this section reveal, Wisconsins charter schools are not performing better than demographically similar public schools. Rather, the charter schools in Wisconsin are performing at levels that are rather similar to what would be predicted given the performance levels of demographically similar public schools. When looking at change over time for either the aggregate of schools or cohorts of same schools we found little or no change on average. The findings from Wisconsin are somewhat unique in the Great Lakes states. In the other states, the charter schools tend to be further behind their comparisons groups, but are making more improvements over time.

Notes and References

1

Wisconsin Charter School Association. (2007). Charter school law basics. Retrieved April 23, 2007, from http://www.wicharterschools.org/charter.law.cfm 2 Stuiber, P., Swenson, D., Normandin, H., & Varana, D. (1998). Charter school program: An evaluation. Madison, WI. Legislative Audit Bureau. 3 Witte, J., Weimer, D., Schlomer, P., & Shober, A. (2004). The performance of charter schools in Wisconsin. Madison: Robert La Follette School of Public Affairs, University of Wisconsin. 4 Loveless, T. (2003). Charter schools: Achievement, accountability, and the role of expertise. Washington, DC: The Brookings Institution. 5 Retrieved [February 27, 2007] from the Web site for the Common Core of Data: http://nces.ed.gov/ccd/. 6 Retrieved [February 23, 2007] from http://dpi.state.wi.us/oea/hist/profic.html. The scale scores for years before 2002-03 were retrieved [February 23, 2007] from http://dpi.state.wi.us/oea/hist/summaries.html.

http://epsl.asu.edu/epru/documents/EPSL-0706-236-EPRU-appg.pdf

Page 8 of 8