Seductive Science

Science is sexy, we all know that. There is a proliferation of pro-science Facebook pages, Instagram accounts, and way too many memes, both for jokes and for serious discussion, and that’s great. (My personal favourite sciency Facebook page is Research Wahlberg).


From Research Wahlberg

There is a bit of a downside to this though, as has been bothering me this week. What do you get when science and original research becomes more and more common among a largely scientifically illiterate population? Broscience. It’s so easy to read the abstract of an article, not fully understand what you’ve read, read the results and think you understand what is going on.

I’ll give an example of poor interpretation of research: recently, a paper comparing benching with bands vs benching with straight weights was shared on a Facebook page I’m a member of (you may have seen it.) At a quick glance, the paper concluded that the group benching with bands improved more than the group benching without (9 kg vs 7 kg.) There was actually some really good stuff done in the experiment; learning periods, randomised crossover design, the statistics were sound… But there were a few things that really bothered me.

  1. There were no p-values reported (“p < 0.05” doesn’t count, the difference between 0.05 and 0.01 is huge)
  2. n = 11
  3. Each intervention was only three weeks
  4. The subjects were untrained

When you add all those factors up, it leaves me with “Interesting, but not useful.”

The bit that really bothered me was not the research itself. The paper, while not perfect, provided an interesting data point. Like any other (low-power) study though, it can’t tell the whole story by itself. You take six studies like this, or eight, and then it begins to tell you something interesting. What really bothered me about this was that so many of the people commenting on it were almost immediately accepting the paper as gospel (I should really stay out the comment section on social media, it does bad things to my blood pressure).

There’s nothing wrong with reading original research, but you need to understand what you’re reading. Given that, I’ve written out a few broad guidelines of what you should be looking for in publications and some alternatives to having to slog through dozens of boring physiology articles!

Guidelines to Interpreting Research

P-values. A p-value tells you something about the strength of evidence. If you want to get technical, it tells you the probability of finding results at least as unusual as those found, under the assumption of the null hypothesis. Statistics is weird and full of double and triple negatives, so it is sufficient to think of it as the weight of evidence for something happening. Here’s a brief summary of one way different p-values could be interpreted.


A couple of notes:

  1. p = 0.05 is often taken for significance in literature. Be very careful when you see this, it’s mostly for historical reasons and really should only mean that you might be on the right track, not that you’ve figured it out!
  2. If p-values are reported as ” < 0.05″ or “<0.1” you basically need to assume it is only just under the actual number, 0.049 and 0.099 in this case. It’s also a warning sign for other things: why might someone want to hide the exact strength of their results?

Sample Size – no matter what your girlfriend said, bigger is better. You saw an effect in a sample of five people? Cool story. You saw the same effect is a sample of 500 people? Now that’s a different story.

It’s pretty intuitive, but here’s an example. Say you want to know the average maximum deadlift for 83 kg powerlifters in the NZPF, so you randomly draw a sample of one person from a list of all of the 83 kg powerlifters in the NZPF: there is an equal probability of drawing Brett Gibbs as there is of drawing anyone else, and then you’d conclude than an average deadlift is 320 kg. You would have made a sampling error: your sample was not representative of the population as a whole. There are all sorts of nifty tricks to help minimise this, but they all require you to take a somewhat decent sample size.

There’s still some critical thinking required here. Some populations are much smaller than others, some are harder to get hold of, and importantly, some experiments are just really hard for people to do! If you had an experiment that took 30 minutes, and your population was untrained males between the ages of 18 and 30, you could get a huge sample size. If an experiment took six weeks and you needed to be an elite powerlifter, it would be much harder to get a good sample size.

The Population Being Studied – This is easy to overlook. Who is being studied? If you’re a member of the population being studied, cool, you might be able to directly apply the findings to yourself. If not, how close are you? The further from the population being studied, the less applicable the results are to you. The study from before was on untrained men of “university age.” The difference between a trained person and an untrained person is pretty phenomenal: can results  be transferred directly into a trained population? No, not really.

Applicability of the Protocol – Is the protocol being used something that you could apply? What was the time frame? In the study from before, the length of the intervention was only three weeks. I don’t care what happens to my bench over the next three weeks (neither should you) – I care what happens over the next three months, 12 months, three years etc.

Weight of Other Evidence – You can never look at data in isolation. One paper doesn’t tell the whole story: a combination of Type-I Errors and publication bias means that sometimes you only see the odd paper where some spurious correlation has been made. If you see one journal article that says something, it might be worth filing it away for future reference. When you see three or four on suitably similar topics, then you can start drawing conclusions in an informed way.

Alternatives to Original Research

Lets be honest, journal articles are (usually) really boring. Fortunately, meta-analyses and textbooks exist. A meta-analysis is where some poor soul reads as much literature on a single topic as they can and rolls all of the research into one coherent and digestible chunk.  They’re usually really good because someone else gets to do all the boring reading for you, and tell you what you should think of some body of literature. They’re often easier to read than the original articles too. Textbooks are another good alternative: they’re usually a step more readable again, and fortunately for us they usually start at quite a basic level (journals usually assume quite a high baseline knowledge of a topic.) Unfortunately, writing a textbook is an arduous affair, and so textbooks are often a little out of date, even those that were just published. (Fortunately for us, the basics of sport science don’t change much over time, only the minutiae.)

In summary, science is seductive, but you need to make sure you know what you’re looking at. To round off my huge rant, I’m going to leave you with a quote I quite like (emphasis my own.)

Some people hate the very name of statistics, but I find them full of beauty and interest. Whenever they are not brutalised, but delicately handled by the higher methods, and are warily interpreted, their power of dealing with complicated phenomena is extraordinary. They are the only tools by which an opening can be cut through the formidable thicket of difficulties that bars the path of those who pursue the Science of man.

Francis Galton

I apologise for such a dry blog post this week, I promise to talk about something more interesting next week! Anyway, I hope your training is going well, and I’ll see you on the platform sometime soon.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s