Working Paper Offers Initial Data on Academic Impact of N.C. School Vouchers

North Carolina State University researchers have taken the first steps toward measuring the academic impact of the state’s publicly funded school voucher program. A new working paper that compares scores of a small volunteer group of public and private school students on the same standardized test found positive effects for voucher recipients, particularly first-year participants in the North Carolina Opportunity Scholarship Program, but authors caution against reading too much into the results.

“This is a quasi-experimental study, so we are not able to say that these are causal effects because we were not working with a representative, random sample of students,” says Anna Egalite, an NC State assistant professor of education and lead author of a working paper on the results. “From a policy perspective, the biggest takeaway from this paper is just how many limitations there are to conducting a high-quality evaluation of the program’s academic impact, given current statutes.”

The Opportunity Scholarship Program has distributed more than $53 million in vouchers since the 2014-15 school year. Qualifying families receive up to $4,200 per year to attend a nonpublic school. Funding is set to increase by at least $10 million annually for 10 years.

Although legislation establishing the voucher program calls for evaluation, that’s been difficult because of a lack of funding and of common accountability measures. Private foundations with varying positions on the role of private school vouchers and other school choice programs have funded NC State’s research on the Opportunity Scholarship Program, and schools, parents and students volunteered to participate in the research, says Egalite, who co-authored the paper with NC State colleagues Trip Stallings and Stephen Porter.

To gather comparison data for the study, in spring 2017 the research team administered the nationally normed Iowa Test of Basic Skills to 698 student volunteers from public and private schools. Currently, public school students are required to take state tests, while private schools may administer any nationally normed test of their choosing to their Opportunity Scholarship students, Egalite says.

“North Carolina nonpublic schools are using a diverse set of tests from different types of publishers, so it’s difficult to compare student performance within the nonpublic school sector, let alone trying to draw comparisons between the public and private schools.”

Even with nearly 700 test scores, in the end, the additional data necessary about each student to contextualize those scores were only available for about 250 students—another example of the challenges faced by the research team to generate meaningful results. The final analysis sample included student demographic information and past test results from the Department of Public Instruction database.

Statistical analysis showed positive effects in math, reading and language for voucher recipients in their first year of private school.

“The effects were large – 36 percent of a standard deviation for math and 44 percent of a standard deviation for language,” Egalite says. Results for second-year participants, with a reduced sample size of 170, showed statistically significant effects only for language scores.

“Although we did our best to recruit a diverse group of students to take the same test for the first time, it’s not a representative, random sample so we can’t make any inference about the average Opportunity Scholarship user,” Egalite says. “We can only talk about our sample group, which could be the highest-performing voucher students in the state.”

A similar statistical matching design was used for research on Milwaukee’s voucher program and a 2001 study of North Carolina’s privately funded Charlotte Scholarship Fund, according to Egalite.

Last year NC State researchers published a series of three reports examining the reasons private schools and North Carolina families chose to participate in the voucher program, drawing on survey data, focus groups and interviews. The latest North Carolina results will provide fodder for the ongoing national discussion of voucher program effectiveness. To date, 15 studies using random samples have examined the programs’ effectiveness, with eight showing positive results for students in math or reading or both.

“The weight of the evidence overall points toward positive academic impact associated with voucher usage,” Egalite says. “However, the three most recent studies before our North Carolina study have shown negative impact.”

North Carolina is among a wave of newer voucher programs with broader eligibility guidelines and statewide enrollment, Egalite says. With about 9,500 applicants for the coming year, North Carolina’s program is in the “middle of the pack” nationally. In comparison Indiana has 34,000 participants and Florida has 98,000.

Egalite and co-author Stallings served on a statewide task force that made recommendations this year for future evaluation of the Opportunity Scholarship Program. Recommendations included allocating state funds for evaluation and limiting the number of nationally normed tests on the private school list.