Sometimes I’m tempted to start a whole blog about bad and misleading uses of data. There’s a classic book (as in very old and very good) called How to Lie With Statistics on this. It shows how the facts stated can be perfectly true, yet also completely misleading. Now that we’re allegedly living in the age of big data and it’s very easy for all and sundry to put out impressive looking charts, the problems it describes seem to be worse than ever and there are plenty of new ones to add to the list.
The strangest thing to my mind is that many of the examples even come from reputable people who certainly know how to do better, and who often bemoan stupid uses of data themselves.
One example I saw today…
FiveThirtyEight – of Nate Silver fame – proclaims We Don’t Know How Often Pilots Commit Suicide.
Remember I’m not talking about saying things that are false, I’m talking about misleading people with statements that are true. So what’s wrong with this article?
To most people the factual statement “We don’t know how often pilots commit suicide” translates into the implication: “Be afraid, pilot suicide could be a real danger”. If you just read the headline, if you just read the opening paragraph of the story, or even if you read the whole story but didn’t stop to ponder the numbers, that would be your take-away from the article: Be afraid.
There are a couple of reasons why “Be afraid” is totally the wrong conclusion to draw from the data. Basically there is a critical difference between “We don’t know X” and “We don’t know anything about the possible values of X”.
- Although we don’t know how often pilots commit suicide, we have a strict upper limit on how often it could be because plane crashes are incredibly rare. So at most pilot suicides while flying are incredibly rare.
We can reduce that upper limit even further by removing crashes whose causes are known to be something else. In the end the thing we don’t know (pilot suicides while flying) remains “unknown”, yet constrained into a range of possible values such that the rational response to it should be “it’s so rare it’s not worth losing sleep over”.
Does it matter?
Human beings aren’t so very rational though. They’re more responsive to stories, how the stories are framed, how many stories they saw than they are to the actual data mentioned. After being exposed to all the coverage that’s there’s been about pilot suicide lately, I’d find the possibility preying on mind if I were about to fly. I’d be able to dismiss the worry based on what I know of the stats, but I wouldn’t be able to avoid having the thought that it might happen. That’s how human minds work.
After the average human being reads that FiveThirtyEight article, they’re going to be left pretty uneasy about the whole question of pilot suicide. All the talk in the article about how it’s unknown, instances when it may have happened, listing risk factors for depression, the quotes from a pilot saying “If I had depression I wouldn’t admit it” will leave the reader anxious. Maybe they were already nervous about flying, and some of them will avoid it a bit more now.
That doesn’t sound too bad maybe?
Well on 9/11, about 3,000 people died. In the year after that many people got scared of flying, and switched to driving long distances instead of taking the plane. The problem: driving is much much more dangerous than flying. It’s estimated that about 1,500 extra people died on the roads that year because people were afraid to fly and drove instead.
How to improve the situation?
Of course it is entirely legitimate and responsible to study and write about topics like pilot suicide, or airline safety in general. Part of why flying has got to be so safe is that people have been poring over every incident for decades and learning the lessons from them.
However it’s easy and tempting to write in a way that’s irresponsible. Tempting because scary-sounding stories are going to get more attention, shares, pageviews etc. Easy because there is less work to do – just pick out some interesting facts, don’t worry too much about the context or putting them into perspective for the readers.
Given that, it’s maybe asking a lot that journalists and bloggers should strive for a higher standard. But one thing we can do to nudge them in that direction is call them out on it once in a while.