By Christine Bricker, Ph.D., Iowa City
Imagine if a weatherman said there was a 50 percent chance of rain a month from today. You might believe him. Now imagine his prediction went up to 80 percent. Would you still buy it, or would it seem a tad extreme? Now imagine he told you that he was 99.7 percent certain it would rain one month from today. Might his extreme confidence for a prediction so far in the future make you a little skeptical?
Right now, the Iowa City Community School District is in the process of making decisions about boundaries and closing elementary schools for the next academic year based on predictions from the equivalent of a very confident weatherman. The District has hired a consultant to provide enrollment projections for all of its 20 elementary schools, three middle schools, and four high schools through academic year 2030/31. On three separate occasions in the consultant’s report, it is asserted that “projected 2026/27 [enrollment] has been validated at 99.7 percent accuracy.”
In a typical elementary school grade of 100 students, 99.7 percent accuracy means the prediction could be off by less than a single student (0.3 students to be exact). In an elementary school of 500 students, there would be a little more breathing room — 1.5 students. Perhaps the consultants were looking at a broader scale when asserting 99.7 percent accuracy: at the district level based on approximately 14,500 students, the prediction could be off by around 44 students. That sounds more plausible, but given that the District is interested in projecting enrollment for individual schools, it’s not really the number we care about.
In any event, 99.7 percent accuracy is an impressive number, which makes it all the more surprising when the report states that “enrollment prediction is a social science — not an exact science” and heavily qualifies its predictions. The reader is left to choose between two opposing claims: near-perfect certainty and a complete lack of a certainty, with no ambiguity in between.
How can we resolve this contradiction? How are we supposed to make important decisions about possible school closures or boundary changes when we are simultaneously perfectly confident about the future and yet totally in the dark?
Scientists — even social scientists, like me — navigate this quandary by using statistical models that account for uncertainty and by giving margins of error around their predictions. Every political poll has a little note saying “+/- 2% points” or something similar, telling the reader how confident to be in the result. If a model produces enrollment estimates for each of the 27 schools, then it should be able to give us 27 margins of error. That might not look as good as 99.7 percent accurate, but it would answer the questions the District needs to know and it would tell us how confident we can be about consultant’s enrollment projections.
It could be that the projections are more or less right. Unfortunately, the report doesn’t provide enough information to allow parents or the District to check. It is impossible to know what exactly their model is doing based on their documentation: variables are not clearly defined, the type of model is not described, and it is unclear whether it is possible for their model to produce a margin of error at all. These are not huge requests or ones that any quantitatively trained parent or school board member would have a problem processing. And this is not a problem that requires a data center — it’s just 27 schools with 13 grades across five years. This amounts to less data than contained in a picture on your phone, and it is all publicly available: school enrollment, birth rates, and building permits. Why not let the public behind the curtain when it comes to such important and irrevocable decisions?
There is a self-fulfilling nature to some predictions: if they are accepted as true and action is taken based on them to, for example, close schools, the predictions can never be tested, because the schools no longer exist, and the public might be left to think that they must have been true. The impression of perfect certainty given by these predictions is leading the District inexorably in one direction, but this certainty is an illusion. If we had more time, more access to the data, more insight into the models underlying these results, we could verify for ourselves just how certain we should be. As it is, all we can say is that we are in the realm of unknown unknowns, that we are certainly uncertain. Or at least 99.7 percent.
