A Wall Street insider tells the charter school sector ‘you’re not all that’…
By Mazinger Z
Charter school proponents have been all atwitter since CREDO released an update to its landmark 2009 study comparing performance to their public schools “competition.” The press release trumpeted that in 2013, charter school students were a whole “8 days of learning in reading beyond their local peers in public schools,” and were running just about even in math. Now, an average of 4 learning days doesn’t sound like much, but if true, it would certainly be an improvement over charter schools’ below-mediocre performance in the previous 2009 study, where charter students were shown to be an average of almost 15 days behind their public school peers. Problem is, it isn’t true…
If you keep reading past the headline, you will unearth this gem:
“charters […] have made modest progress in raising student performance in both reading and mathematics, caused in part by the closure of 8 percent of the charters[…].
So, charter schools improved “in part” by the simple expedient of removing 8% largely terrible charter schools from the comparison!*
In statistics, this is a textbook case of what is known technically as “survivorship bias”—otherwise known as “ignoring data that makes you look bad.” This statistical fallacy opens up enormous opportunities for people with flexible ethics and an entrepreneurial bent. It is a mainstay of the hedge fund industry, for instance. Hedge fund indices routinely appear to outperform simple, non-fee generating investment strategies like index funds by neglecting to include funds that closed down, and take funds out of the index whenever they stop reporting performance (hint: no fund fails to report good performance!).
R u hot or not?
Some funds, which shall remain nameless on the advice of counsel, improve quite a bit on these crude methods. They recruit new traders continuously, and give them a bit of capital. After a few months, the ones that make money are trumpeted to investors as the next hot thing, while the ones that don’t are quietly fired and never mentioned again. In this business model, what the traders actually do is irrelevant, as long as they are cheap and willing to sign leonine contracts. In fact, buying monkeys and letting them flip coins to predict market direction would work even better, though this might make the game a bit too obvious.
Repeat as necessary
The parallels to the emerging charter school paradigm are obvious. You start a lot of charters. Some do better than public schools, some do worse, but overall they underperform. You shut down the bad ones. Now repeat the analysis with the non-terrible ones. Improvement! Except that the students unfortunate enough to attend these terrible school don’t just disappear (we hope – let’s not give charter schools any ideas!). However, they do disappear from the CREDO study, and that seems to be good enough for the charter sector and its advocates to proclaim this a success story.
By the way, the CREDO people not only appear to be unaware that “survivorship bias” is problem, they seem to believe it is a feature, rather than a bug. In the final pages of the study, they run a bunch of scenarios detailing the dramatic improvement that you get in results by eliminating different kinds of underperforming charters (in addition to the ones that were closed down). They conclude:
“[…]the different scenarios make obvious the fact that the impact on quality that accompanies closure is more dramatic and enduring than efforts to improve the current stock of schools.[…]
A nation of geniuses
Why not go further? Close the bottom 90% of schools in the country, and we’ll leap ahead of our Singaporean or Finnish rivals! But why stop there? If we close down all of the schools except for Stuyvesant High, before long, every American kid will be doing calculus on polar coordinates by their 15th birthday!
To anyone with a modicum of statistical training, this is not at all a subtle point. The logical fallacy of survivorship bias is well known. The CREDO people could have run any number of simple simulations to attempt to correct for it. They could have repeated the study while keeping the 8% of shut down charter schools. Or, they could have eliminated the bottom 8% of public schools from the analysis. Either they didn’t, or they chose not to share the results with the rest of the world. Why?
* How big a “part”? The CREDO people don’t tell us, but they do provide, unwittingly, serious evidence that all of the improvement is caused by the removal of underperforming charters from the study. As a group, new charter schools started since 2009 are not subject to survivorship bias since (hopefully) nearly all of them should have managed to stay open four years. According to the study, these charter schools showed no improvement at all relative to the 2009 study.
Mazinger Z is the pen name of a Wall Street trader who has worked for hedge funds and investment banks for more than a decade. Send comments, questions and investment ideas to firstname.lastname@example.org.