Survivorship Bias in Journalism and History
I was talking to a friend and reader this past weekend over the phone and he brought up a point that I do not spend enough time on in the paper, and that is the problem of survivorship bias.
My friend had pointed out to me that one of my recent stories on the proposed sales tax increases to pay for a new justice center has come across as unbalanced. His point was that in small communities there are less people to pay for needed infrastructure and so taxes tend to be higher. By my stating it would be the third highest in the state, I was telling a fact, but perhaps it needed more context and history to be truly an honest and objective inquirery into the subject.
He was absolutely right of course; decisions are attached to the environment that an agent lives in. In simpler terms, our choices depend on the reality that we face here locally. Adding more context to a story like the sales tax issue will rarely be misplaced effort.
But this brings up a bigger problem in journalism when providing context on a story, a problem that I have never really seen mentioned in the journalism business; that is, explaining the “what ifs” that make up the context of an ongoing news event.
Known as more technically as Survivorship Bias, we humans tend to notice and pay attention to what happened instead of what did not happen. It is natural to pay attention to the things that happened and respond accordingly. But the modern world has forced humans to have to deal with situations where the valuable information is found in what did not happen. The problem is that we are not wired to automatically understand this point despite its importance in a complex world.
The most famous example of Survivorship Bias was documented during World War II when allied bomber planes suffered heavy losses during bombing raids across Germany. A group was assembled to try and figure out ways to modify the planes so that more survived. The group documented in detail the damage done to surviving planes that returned home. After they had enough data they suggested that the areas with the most documented damage be reenforced with armor. The statistician Abraham Wald went against the military’s suggestion and ordered the parts of the plane with the least amount of damage be reenforced with armor. His reasoning was simple yet counterintuitive, the planes that returned survived despite the damage documented. The planes that not return must have been hit in the areas least documented in the surviving planes.
The survivorship bias blinds us equally in many other parts of life were we only see those things, people, events, technologies, that have survived and fail to take into all those that did not.
This happens in history and journalism frequently. We place far to much credit and importance on what ended up happening and not what did not happen. When events take place, the results are rarely a forgone conclusion as they take place. Yet afterwards, the events seem obvious. The old saying hindsight is 20/20 comes in here. We do not work through the “what ifs” often enough when writing history or newspapers.
So back to the original point that started this look into Survivorship Bias. My friend was right in saying that I needed to compare tax rates in other small communities to see if they were higher than average due to a lack of population density to pay for need projects. The answer in Colorado is that sales and property taxes are generally higher in mountain towns, but overall population is not a good indicator of tax burdens. Eastern Colorado has low taxes and low populations. The difference seems to be the amount of tourism. The two highest sales tax cities are Silverton and Winter Park with other areas like Aspen, Crested Butte, Summit County, Vail, and Telluride all coming in high in both property and sales taxes.
But even here, it is important to not engage in naïve empiricism. Tourists play a factor as governments see it as a way for outsiders to help pay for the costs of infrastructure. But it is the mountainous terrain itself that also increases some costs of construction keep populations limit due to water and road access.
There are even further problems with the story I wrote using the logic above, there could have been more context on why so much money needs to be raised for the Justice Center. While the Tribune has tracked the Justice Center project for over five years, it is hard to get all that context into a new update on the subject. While the Justice Center Committee selected a large court and jail building to prepare for future growth in the county. The Tribune did not go into details on the other options that could be possible such as additions to both the jail and courthouse. I committed survivorship bias by only reporting on what ended up happening.
And this is the catch, my story was factual on everything that I reported, I even did some future consequences to the effects on the poor or problems from a recession with sales taxes. But I did not go backwards in time to fully bring into context the “what ifs.” The problem with the historian’s and journalist’s craft is that we focus on facts and take those facts at face value. It is considered to be objective and fair. But unlike physics where facts can lead to future predications (me dropping a pen will cause the pen to fall, a fact of gravity that does not require context) facts in journalism are tricky things. Context is everything and the process of exploring what ifs is often called opinions and biased journalism.
Journalism deals with both an uncertain past and often an uncertain future and trying to condense the complexity of the world into 800 words or less is problematic. But when faced with uncertainty, the solution to that uncertainty is obvious; precaution.
In journalism this means looking at a series of facts from many angles, to work hard on context, and to resist the urge to create a narrative that takes over the reporting. Just relying on the facts as they come does a disservice to both the readers and the subjects of a story. Survivorship bias is not easy to handle, but knowing the problem is a big part of the battle of what to do in the future.
-Jordan Hedberg
