Do Vouchers Lead to Greater Learning? Depends Where (and When) You Look

The direction of America’s education system is seemingly at an inflection point. At the center of the current policy debate is the use of school vouchers to directly pay for children to attend whichever school they want, including private and parochial schools. As Brown and Douglas-Gabriel reported in the Washington Post, “the president is proposing a $168 million increase for charter schools…and a new $250 million private-school choice program, which would probably provide vouchers for families to use at private or parochial schools. Vouchers are one of the most polarizing issues in education, drawing fierce resistance from Democrats and some Republicans, particularly those in rural states.”

School vouchers are neither good nor bad in themselves. Advocates on both sides of the debate will point to evidence that vouchers are good or bad at improving student test scores. In this debate, both sides are sometimes right, but frequently wrong, employing cherry-picked results, using nonstandard levels of statistical significance, and misrepresenting the body of evidence.

What is the Evidence on Vouchers?

As shown in the figure, it is hard to point to any pattern that predicts where a voucher system improves on public schools and where it doesn't. Ten sets of results are shown in figure 1, where some show modest positive results, but most differences are statistically indistinguishable from zero. Two state programs show substantial negative impacts of private school vouchers (Ohio and Louisiana). Studies in two cities show modest positive impacts of vouchers for one subject (Milwaukee in math, and Washington, D.C. in English), without adjustments for the 19 comparisons in the figure, which would make the positive impacts no longer statistically distinguishable from zero.

Figure 1. Evaluation results for 11 voucher programs

Note: Size of marker indicates the inverse variance of the estimator; larger markers indicate more precise estimates, which is also reflected in the width of the confidence intervals. Results are drawn from Shakeel, Anderson, and Wolf (2016) and studies cited therein, Mills and Wolf (2017), Figlio and Karbownik (2016), and Dynarski, Rui, Webber, and Gutmann (2017).

One takeaway is that any claim that “the evidence points clearly in one direction” (Forster 2016; pg. 1) should be taken with a grain of salt. The older studies tended to find more positive impacts, and the newer studies for Ohio and Louisiana show more negative impacts, often surprisingly so. As Carey (2017) reported, “Martin West, a professor at the Harvard Graduate School of Education, calls the negative effects in Louisiana ‘as large as any I’ve seen in the literature’ — not just compared with other voucher studies, but in the history of American education research.” Results vary not only by time, but by place as well.

What Drives Variation?

The effects of a voucher program are relative to the schools which students would otherwise be attending. Students who exit a public school that produces minimal gains in learning for a private school that produces large gains will tend to fare better in the private school. When the public school is generating large gains in learning, students who exit to attend the private school will tend to do worse. So voucher programs will look good on average in some places, and produce negative impacts on learning in others, relative to the public alternative.

To increase the odds that voucher programs will further student learning relative to traditional public schools, we can look to lessons from another school choice model, charter schools. Like voucher-funded private schools, charter schools serve to promote school choice, and may produce greater or lesser gains than a traditional public school (Gleason et al. 2010). Two standout systems are in D.C. (Nichols and Özek 2011) and Massachusetts (Angrist et al. 2016). The D.C. and Massachusetts systems enforce strict licensing and quality standards. The D.C. Public Charter School Board applies aggressive monitoring, which may result in closures among low-performing schools. Massachusetts uses “a robust system of accountability” (Cohodes and Dynarski 2016) and tends to serve relatively disadvantaged students. Leonhardt (2016) quotes Sue Dynarski writing, “the gains to children in Massachusetts charters are enormous. They are larger than any I have seen in my career.”

The same cannot be said for private schools. Private schools are among the best and worst schools in the country, but there is no guarantee that a particular private school accepting public funds is even in the top half, unless some kind of monitoring and evaluation scheme ensures a minimum level of quality. As Mark Dynarski (2016) points out, “our historical understanding of the superior performance of private schools is no longer accurate. Since the nineties, public schools have been under heavy pressure to improve test scores.” What seems clear is that the same type of innovation that produces positive effects in one setting can produce negative impacts in another time or place.

Monitoring and Evaluation Are Critical

The overarching result is that a district can implement a voucher system and get lucky enough to see modest gains in some subgroups as did Milwaukee (Rouse 1998; Greene, Peterson, and Du 1999; Wolf 2012) or New York (Mayer et al. 2002). They can also get very unlucky and see lower scores (negative gains) as did Louisiana, Ohio, Indiana (Dynarski 2016), and DC (Dynarski, Rui, Webber, and Gutmann 2017) more recently. These comparisons need not be stable over time, either. If a public school system improves, as it has in DC (Stein and Brown 2016), a previously null or marginally positive advantage of private schools over a public school system can degrade over time until the advantage disappears or private schools begin to appear worse at improving test scores than the public system.

The only way to ensure a voucher (or charter, or any other) system continues to provide value is to provide for ongoing monitoring and evaluation as part of the funding stream. In considering the President’s voucher proposal, Congress would be wise to require monitoring of how the funds are used and rigorous evaluation of the impact of funded voucher programs on student learning. As Friedman (1955), who first introduced the idea of vouchers asserted, the “role of the government would be limited to assuring that the schools met certain minimum standards.” This is presumably not the only role of government, but it is an eminently sensible one.
John Thacker
"If a public school system improves, as it has in DC - a previously null or marginally positive advantage of private schools over a public school system can degrade over time until the advantage disappears or private schools begin to appear worse at improving test scores than the public system."

Yet, this is a sign of success, even if it makes things more difficult to judge. One of the hypothesized effects of a voucher, charter, or other choice system is to cause individual public schools and the system to improve due to competition. Indeed, under equilibrium we *should* expect the effect to tend to zero, as the worst public schools reform or close, and marginal new private schools are not as good as the ones when the supply is restricted. The lack of a gap is no way proves that the competition has no effect; indeed that a school system with a long standing voucher and charter program improves while the gap shrinks is evidence for the program and for continuing it.

That brings up another point: the Massachusetts system has in some ways "too good" results. The state has a sharp cap on the number of students who may be enrolled in charter schools. Under equilibrium we should not expect this. That strongly implies that MA has too few charter schools, as the state would be better off with more of them. The new charter schools on the margin should be expected to be worse than the average current charter school, but would be expected to replace the worst public schools.

The end result should be no gap in quality. If there is a gap in quality, that is a sign of a lack of equilibrium, a lack of access for a student from average or poor SES, or an artificial cap on the number of charter or private schools. The aim is to raise the quality of all schools.

(Another difficult point is that, particularly in areas where school test scores are very high (as is the pressure to achieve such scores), parents may deliberately seek out schools that do not "teach to the test." I realize that the consensus is not to give much weight to such things as parental satisfaction as opposed to test scores, but there is a robust group of people who are skeptical of the focus on tests, regardless of their views on vouchers or charters.)
6/3/2017 10:39:44 AM

Leave comment

 Security code