Everyone remember that Harvard study which showed that vouchers – among black children, compared to a control group – lead to improved test scores? It was talked about quite a bit during the 2000 election.
It turns out, according to an article in today’s Times, the results don’t hold up. Fattypatties discusses the matter extensively (permalink is currently bloggered, so look for the April 7th entry entitled “Statistics and the Mystique of Science”). From the Times article:
A Princeton economist, Alan B. Krueger, took the offer, and after two years recently concluded that Professor Peterson had it all wrong — that not even the black students using vouchers had made any test gains. And Mr. Myers, Professor Peterson’s former research partner, agrees, calling Professor Krueger’s work “a fine interpretation of the results.”
What makes this a cautionary tale for political leaders seeking to draft public policy from supposedly scientific research is the mundane nature of the apparent miscalculations. Professor Krueger concluded that the original study had failed to count 292 black students whose test scores should have been included. And once they are added — making the sample larger and statistically more reliable — vouchers appear to have made no difference for any group.[…]
It is scary how many prominent thinkers in this nation of 290 million were ready to make new policy from a single study that appears to have gone from meaningful to meaningless based on whether 292 children’s test scores are discounted or included. “It’s not a study I’d want to use to make public policy,” Mr. Myers said. “I see this and go `whoa.’ “
Professor Krueger of Princeton (who also writes a monthly business column in The Times) said, “This appeared to be high-quality work, but it teaches you not to believe anything until the data are made available.”
Fattypatties is using this article as a jump-off for a series on science and fat issues. I’m looking forward to that.
I know this is just an old post that’s been dragged up by a spambot, but I’d like to know: What were the vouchers for? What were they supposed to do to improve grades?
The specifics of voucher programs can vary from state to state. However in general, parents are given vouchers that they can use to pay tuition at a school of their choice.
There are several issues to consider when deciding when deciding if you support vouchers or not:
1) Is giving parents a choice a good thing per se.
2) When the parents are given the choice, do the individual students actually do better, worse, or the same. (On any metric. For example, their academic performance could conceivably stay the same, but maybe they are somehow happier and fit in better — which is hard to measure. Or maybe the parents are just happier for some reason that is important to them.)
3) Does taking money out of the public schools and giving it to parents to pay tuition– often a private schools harm the public schools.
4) Is the program implemented in a way that give any *real* choices? Or is it really implemented to aggravate segregation etc.
5) Probably other factors! (Oh.. the religious one. Parents often pick religious schools, and one accusation is that people support vouchers to subsidize parents who want to send their kids to a religious school anyway..)
Regardless of what the answers are, it’s clear that any statistics provided to support someones claim should be based on an honest assessment of the data. The computational methods and choices should be clear.
To this end, It is always best if data are made publically available, (even in non-controversial research areas). It is even more important to make the data avaiable when the topic is a political hot potato.
BTW, I tend to think that it is a good idea to give parents choices. So, in my opinion, if vouchers make “no difference” to academic outcome, don’t harm public schools, and don’t have any identifyable harmful consequences for individual students we should have them. I am also btw an atheist, but I don’t even mind that parents might pick religious schools.
I am however, tremedously bothered by people who cherry pick data or seem to do so. Hiding the data, or doing mysterious or odd things with the data– and then not telling anyone you did that particular odd thing bothers me a lot!