“I have to chuckle whenever I read yet another description of American frontier log cabins as having been well crafted or sturdily or beautifully built. The much more likely truth is that 99.9% of frontier log cabins were horribly built—it’s just that all of those fell down. The few that have survived intact were the ones that were well made. That doesn’t mean all of them were.”
– Mike Johnston, The Online Photographer, “The Trough of No Value”
Survivorship bias is the mistake of focusing on the winners, instead of characteristics of those who failed. It’s attempting to learn the secrets to success, instead of learning why a venture failed. Wharton professor Katy Milkman defines it like this, “Survivorship bias is an error that arises because we look at the data we have but ignore the selection process that led us to have those data.” When we only look at one side of the problem, the failures become all but invisible.
Examples of Survivorship Bias
We encounter survivorship bias frequently in news reports. While journalism is concerned with the new and novel, only giving our attention to the rare success story can warp our perceptions. Survivorship bias leads us to ignore the elements that didn’t make it past a selection of some sort.
Bill Gates, Steve Jobs, and Mark Zuckerberg all dropped out of college – does this mean only college dropouts can create billion-dollar companies? Or that it is necessary to first be accepted into a top-tier institution, then drop out to build a successful software company? Outliers are anomalies by definition. These rare success stories alone shouldn’t influence us to change our ideas about what brings success. In fact, understanding what caused the failures of other contenders often sheds more light on a problem or path.
We’ve all heard the stories of centenarians, crediting their longevity to drinking soda every day, or arguably worse, smoking every day. Obviously, the path to a long and healthy life doesn’t involve either of these habits. These claims leave out the millions of people who die every year of complications from lung damage, cancer, diabetes, and heart disease. If we stopped at the claims, though, we’d be duped into thinking these destructive habits are some mythical spring of youth. These individuals are simply exceptions to the rule, thus proving the rule.
One of the most widely cited examples of survivorship bias involved Abraham Wald, a statistician during World War II. Wald was part of Columbia University’s elite Statistical Research Group, assigned to help the military improve their weaponry and decrease casualties. As planes returned from combat, this group assessed the locations of the bullet holes. The military determined it would be best to reinforce the armor on the wings, tail, and body, where the bullet holes were concentrated. Wald, seeing the unseen, responded that this was the opposite of what they should do. Instead, they should concentrate on reinforcing the areas which did not have damage. The planes that returned, he reasoned, were able to fly back; they had managed to survive the damage. The planes that didn’t come back, though, had encountered catastrophic failures. Instead of reinforcing places which were already tolerating damage, the planes should be reinforced where the returned planes had no damage, like the engines and cockpit.
We do this with less innocuous things, too. Many rock and roll fans will tell you the best songs of the genre were written during the Vietnam Era. Amazingly, over half of Rolling Stone Magazine’s 100 Greatest Songs of All Time – fifty-one, to be exact – are from those war torn years between 1965 and 1975. Yet, here too, we are only seeing part of the story.
These anthems are revered because they were and are so entwined with American culture at the time. These were the songs people listened to wondering whether they’d be drafted, and whether their loved ones would return home. This was the soundtrack of the civil rights era. These songs were birthed from a deep distrust and objection to the American government. There hasn’t been anything quite like that era in the music or the culture in the past fifty years. Those emotions are inextricably tied to these songs; you can’t think of an anti-war tune and not think of the cultural upheaval.
We then come to the question of what is missing? We’ve defined the era with the very best songs from the very best songwriters and musicians. It’s not as though every song written in that ten year span was symphonic gold. There were certainly terrible songs written during this time, but we don’t hear about them because they’ve faded into obscurity. For every Marvin Gaye or Bob Dylan there were a dozen kids struggling to string chords together, a hundred would-be poets attempting to put words to their emotions. For every Hendrix, there were a dozen lesser and forgotten would-be virtuosos.
We might ask, “What is missing?” and see what clues the bigger picture reveals. Just like with such trivial things as ranking songs, we make decisions with only a piece of the available information all the time. Now that we’ve seen examples of this bias in the wild, we’ll look at ways to correct it.
How we can Correct This
As helpful as reading stories of people and companies who found success might be, those success stories don’t depict the full picture. Each of those stories, by design or by default, leave out previous failures, influences, the role of luck or outliers or the impact of decisions made by other people. Plus, as we’ve seen today, we each have our own set of biases cast over our view like an Instagram filter, altering the final product we see.
One key way to correct this filter is to understand where your data is coming from. What is not present? Like Abraham Wald, look for the missing damage. One particularly novel implementation of this is satisfaction surveys, aimed not at customers, but at those who chose not to buy from a particular company. Simply asking people questions like, “Why didn’t they make a purchase today?” and,“What would it take for them to earn your business?” is a creative way to overcome the survivorship bias and gain valuable information into “failures.”
Another factor to keep in mind is the role of luck. While we live in an increasingly predictable word, luck and random happenstance continue to influence choices from the foods we eat to the employment opportunities we take. Luck often plays a role which we diminish looking back; things like finding the right information at just the right time or meeting the right person do impact the paths our lives take.
Finally, be aware of your own tendency to look back over your career or life and create a narrative. Don’t fall for the illusion that you followed a grand plan step by step, and knowingly arrived at greatness. We tend to make “sense” of our decisions and paths in hindsight, elevating successes and glossing over failures, until it our path appears as bright as the yellow brick road. We all know that progress is messy and is far from a steady forward movement. Beware of other people who construct a smooth narrative of their own stories, too.
Just like those log cabins, don’t mistake the few high quality standouts as examples of the average. Look for what is missing.
