5 reasons why you shouldn’t trust “5 reasons why” lists

How reliable are the scientific findings that reach us? Despite being a defender and follower of scientific research I will challenge the validity of these findings through the following 5 points.

• The in/dependence of research and information channels from corporations
• Misuse of statistics
• The samples used in research papers
• The impact of cultural believes and cherry picking among them
• The oversimplification and popularity of the “5 reasons why” lists

1. Corporate influence in media and research #

Links between the information presented in the media and corporate or political interests have always existed. Yet the gap between our perception of their influence and their actual involvement might be becoming larger. Today, corporations are sponsoring news articles in respectable media like Time magazine or the New York Times. These would be ok except for the fact that they camouflage their ads as news. It’s a practice known as native advertising which has always been a big no-no but today these companies openly accept it.

The consequences are examples like Chevron writing in the New York Times about future energy trends, or corporations writing about studies concerning what makes you happy and directing you to their products.

This takes me to the influence of corporations on research. Now, you can clearly see why they would be willing to fund a research study with an interest to find a particular result, and only publish it if the result answers their needs. This is why peer reviewed papers and replications of studies should be a common practice, however it doesn’t seem to be the case. According to Gigerenzer, “an analysis of over 50 ‘landmark’ publications in top journals on cancer drugs revealed that the positive effects of most studies could not be replicated.”

2. Misuse of statistics #

Statistics can be a little devious, enabling a misuse of communication on statistical findings. Please let me take you through a real example.

Let’s say you read the following statement: “By taking this test you are reducing by 20% your chances of dying from condition X, and false positives only have a rate of 0.07%.”

It seems quite reasonable to take it! Except that in reality, by taking the test you go from having 5 chances out of 10,000 to 4 in 10,000. In fact you are improving by 1 in 10,000, but talking in relative terms (20% as it’s from 5 to 4) it looks a lot more powerful.

In some cases the benefits are communicated in relative terms, and the harms and risks in absolute risks. In this case, the false positive (7 out of 10,000–or 0.07%) is actually more probable than the chances of you dying from the condition.

Gerd Gigerenzer does a great job at showing us how, by combining absolute and relative percentages, harmful choices can be marketed as smart ones, as is the case for a common prostate cancer test and mammography screenings.

3. The samples used in research papers #

We commonly see broad claims published about human psychology and behavior, based on scientific studies. However, a great paper from the University of British Columbia warns us about their validity by introducing the term WEIRD (Western, Educated, Industrialized, Rich, and Democratic individuals).

The researchers found that samples used for behavioral science research were 96% weird, when the weird population represents 12% of the world. And we derive from these homogenous samples universal findings.

This bias is stronger in psychology research than in any other science. More so, mostly American undergraduates are used as subjects, when Americans are the most extreme in many cultural factors and undergraduates even more so among Americans.

To be able to declare that a finding is universal for the species, it’s important to take a diverse sample, given our diversity. Which takes us to point 4.

4. The impact of culture #

So what if we only study a small sample and extrapolate? Are we really that different between different cultures? Well, no. But in some psychological and behavioral traits, we are different. Variations among cultures are found in behavioral aspects like motivation, fairness, cooperation, or risk aversion. But also in very fundamental aspects of psychology like visual, and spacial cognition. Some optical illusions like the following, that we considered to be universal to our species, are not.

A WEIRD sample sees the upper line shorter than the lower line, unlike individuals from other cultures. Therefore, for universal findings on human psychology, a much more diverse sample is necessary.

5. The oversimplification and popularity of “5 reasons why” lists #

So overall, we have to be careful about intentional and unintentional cherry picking of the information. It’s not easy, as so much information is pushed towards us through classical media, looking for catchy headlines based on single event stories. And the info we pull via internet and social media is mostly “likeminded” and biased towards the same direction.

These lists often combine the defects of both worlds. The result is a catchy headline, a oversimplified message and explanation, and a high level of assertiveness and certainty in their conclusions.

As a general countermeasure I would humbly advice to be cautious of too much certainty. Try to read about the opposite point of view, take a look at the sources of the information and if possible, the samples and statistical results. But most of all, avoid the “5 reasons why” lists if you are looking for any seriously valid and trustworthy information!

That’s the end of my 5 reasons why list. ;-)

Author: C. Criado-Perez #

References: #

• Joseph Henrich, Steven J. Heine, Ara Norenzayan, The weirdest people in the world?
• G. Gigerenzer, Risk savvy: How to make good decisions
• N.N. Taleb, Antifragile: Things that gain from disorder