This morning I saw an article in the NZ Herald on the “paleo diet” that rather frustrated me. It seems like a great example of poor science reporting, trying its hardest to turn a study into a story instead of doing any actual science reporting. The role of a science reporter is not to sensationalise, it’s to accurately report on science, and that includes making the drawbacks of a study clear and not exaggerating the conclusions.

In this case though, it looks like the author chose to omit half the results of the study, presumably so as not to pollute the narrative they had chosen. The take-home message of the article can be found in the first paragraph:

the best way to lose weight is by copying our ancient ancestors, a study suggests.

I’m not even going to get into the problems with characterising the so-called “paleo diet” as “copying our ancient ancestors”, that’s been adequately covered elsewhere. The information used to support this weight loss conclusion is that the study in question found that:

Women who adopted the so-called Palaeolithic diet lost twice as much weight within six months as those who followed a modern programme based on official health guidelines.

Wow, that sounds impressive. Case closed, right? Except, if you look at the actual study (not open access, unfortunately), which of course is not linked to from the online article, you’ll find another result that is curiously omitted from the Herald article:

Both groups significantly decreased total fat mass at 6 months (−6.5 and−2.6 kg) and 24 months (−4.6 and−2.9 kg), with a more pronounced fat loss in the PD [Paleolithic-type diet] group at 6 months (P<0.001) but not at 24 months (P=0.095).

So there was a statistically significant difference in fat loss after 6 months, as mentioned in the article, but after 24 months there was no statistically significant difference in fat loss between the groups. That is a negative result.

Although there was still an observed difference in fat loss between the groups at 24 months, it wasn’t big enough for the researchers to be reasonably confident that it wasn’t just due to random variation. That’s partly due to the size of the difference observed, and also because the study was so small. 70 people split into 2 groups is very small for this kind of study, whereas a good sample size would be hundreds or even thousands of participants, not just a few dozen. Of course, such large studies are much more difficult and expensive to undertake, so a lot of smaller studies like this do happen. Sample size is very important though – small studies like this are not nearly as reliable as the much larger ones – so it’s important to remember to take the sample size into account when evaluating a study’s conclusions.

The Herald article does mention, way down near the bottom, that all of the participants in the study were obese postmenopausal women. Everywhere else, however, it avoids that caveat and seems to imply that the conclusions should be applicable to everyone, or at least to all women.

It’s also rather frustrating that the article says that the study “found [the “caveman diet”] more effective than some modern diets”, and that this study suggests it is “the best way to lose weight”, even though the study didn’t compare it with “some modern diets”. It compared it with a single other diet, one based on the Nordic Nutritional Recommendations.

If the Herald wants some tips on how to report on science, a great place to start would be to take another look at the science itself. The conclusion in the abstract of the study they’re writing about seems much more appropriate, even if it does seem a bit dismissive of the negative 24 month results:

A PD [Paleolithic-type diet] has greater beneficial effects vs an NNR [Nordic Nutritional Recommendations] diet regarding fat mass, abdominal obesity and triglyceride levels in obese postmenopausal women; effects not sustained for anthropometric measurements at 24 months. Adherence to protein intake was poor in the PD group. The long-term consequences of these changes remain to be studied.

Then again, perhaps I should be glad the Herald didn’t reprint the original headline from the Daily Telegraph:

Caveman diet twice as effective as modern diets

I’m not sure I could come up with a more misleading headline if I tried.

Despite the horrific headline, the original article does have a bit more information in its second half from the study’s primary author that was truncated from the Herald’s reprint.