Don’t Take Their Word for It!

About a month ago, I ran into an excellent example of the importance of scepticism. A business card for a local gym was dropped off at my workplace, and they had a “Vibration Training” section on their website. Of course, my interests being as they are, I find myself entirely unable to resist anything involving the word “vibration”, so I took a look.

This page made 2 referenced claims:

the benefits have been well proven to match and surpass conventional forms of training – especially when time is factored in.*

*Roelants, Delecluse, Goris, Veshueren, 2004

A German skin clinic noted that the appearance of cellulite reduced on average 25.68%** with vibration training alone. When combined with conventional cardio exercise this percentage rose to 32.3 % (with a considerable increase in the total time spent exercising).

** Sanderm, 2003

The second claim, in particular, seemed strange to me. I had no idea how cellulite could be measured in a quantifiable way, but even allowing for my lack of knowledge here the figure 25.68% seemed far too specific. Something smelled fishy.


The next step, of course, was to look for the references themselves. Unfortunately, no links were provided on that page. Frustratingly, both references were also misspelled. The name of the first reference’s last author is actually Verschueren, not Veshueren, and the German skin clinic is called Sanaderm, not Sanderm (as far as I can tell, its fully qualified English name is the SANADERM Professional Clinic for Skin Disease and Allergology).

The first study wasn’t too hard to find, being available on PubMed and having been published in the International Journal of Sports Medicine. Here’s a link to the paper – Roelants et al.

The Sanaderm study, however, didn’t seem to be published anywhere. Eventually, however, I managed to find what appeared to be the study in its original German, on the website of another gym offering vibration training – Anti Cellulite Untersuchung.

Unfortunately, I don’t speak a lick of German, so the only way in which I’ve been able to read the study is via Google Translate (here’s a link to the translated study).

The first thing I looked for was how cellulite was measured. To my disbelief, I found that it was not measured in any quantifiable way. Instead, subjects were sorted into 1 of 3 qualitative categories. Here is the relevant passage from the machine translated study:

Defined for this study stages of cellulite

Stage 1
Cellulite is not normally visible. But if the skin Thigh, buttocks or belly with hands is pressed together, dents appear honeycomb.

Stage 2
Cellulite is not pushing together of the skin standing visible.

Stage 3
Cellulite is not pushing together of the skin lying visible.

The word order of the machine translation is unfortunately less than perfect, but I think the meaning is clear enough.


So now my question was how on Earth could they possibly get from a qualitative measurement of cellulite to so precise a measurement as a 25.68% decrease?

Unfortunately, although their results section seems to include all other measurements for each individual subject, the “Cellulite Degree” categorisation is only given for 3 individuals selected (the method of selection is not given) from each of the non-control groups.

Interestingly, one of these example subjects is given a “Cellulite Degree” categorisation of 2.5, despite this being an undefined value. I have not been able to find a justification for this datum anywhere in the study.

The appallingly bad method these researchers have used to compare their groups is to assign arbitrary values (1, 2, and 3) to each of the “Cellulite Degree” categories. It seems these values are associated with the category names of “Stage 1”, “Stage 2”, and “Stage 3”. To illustrate just how inappropriate these values are, I’d like to point out that the categories could easily have been named “A”, “B”, and “C”. There seems to be no justification anywhere in the study for the particular values that have been assigned to each category.

The researchers then added up these values for an entire group, and compared the totals of the first and final tests (omitting the results of the intermediate test that took place halfway through the study).

The sum of these values for the first test of the group that only took part in exercise on the Power Plate vibration machine (Group 1) was 46.50 (why they felt the need to record the result to 2 d.p. is entirely beyond me, and it is not due to the machine translation) and the final result was 37.00. Using these values, which are both almost entirely arbitrary, they seem to have performed the following calculations:

 46.50
-37.00
= 9.50

  9.50
 37.00
=25.68%

Note that this is absolutely not the calculation that should have been done. By this same calculation, if I start with 1 apple and eat half of it, it has decreased by 100%.

The other calculation, done for Group 2 (the group that participated in a cardio workout routine after the same vibration exercise as Group 1) appears to be of the same form:

 43.00
-32.50
=10.50

 10.50
 32.50
=32.30%

The values that they would have gained from the correct calculations would not have been 25.68% and 32.30%, but 20.43% and 24.42%, each of which is massively less. Also recall, of course, that these percentages were still gained using the unjustified values assigned to each qualitative category.

Every single calculation resulting in a percentage change seems to have been performed in the same way, resulting in every case in a result that is wrong and overstated. They also appear to have truncated results instead of employing rounding, although the effect of this mistake is to slightly decrease results (at most by 0.01%, e.g. 1.22% instead of 1.23%).


Aside from all this, it’s also worth noting that a control group of 5 is abysmally small. On top of that, the average initial cellulite categorisation (which I realise is arbitrary, but it’s the only indication available to me of each group’s cellulite) for the control group is lower than for the other 2 groups.

The average initial value for the control group is 1.5. My first calculations for the average initial values for Groups 1 and 2 resulted in 1.94 and 1.39, respectively, but then I realised that the results tables listed “Number of volunteers” as though no participants had dropped out.

From earlier in the results section I could find that 1 participant from Group 1 and 6 participants from Group 2 dropped out. Taking these data into account, the actual average initial values for Groups 1 and 2 were 2.02 and 1.72, respectively.

It’s hard to compare these values due to their arbitrary nature, but I feel it’s worth noting that they are both significantly above the average value of 1.5 for the control group.


This study, in particular its value of 25.68%, seems to be quoted practically everywhere the product used (Power Plate) is advertised. Here are some examples I found (this list is far from exhaustive, you’ll be able to find many more by searching for such things as “25.68% vibration”):

I’d also like to mention that the participants in this study were all females aged 25-45, yet none of the places where I saw it referenced recognised that its conclusions should not be applied to males or to females outside of that age range.

Also, as far as I can tell from the machine translation, the control group underwent no training, so the difference attributable to the Power Plate vibration equipment cannot be assessed from this study. Instead of comparing exercise to exercise+vibration, they have compared no exercise to exercise+vibration.

And, of course, I have not “debunked” or “disproven” the type of training done in this study. All I have done is criticise a particularly bad study.


Hopefully the title and contents of this post are enough to make the conclusions that you should draw quite clear. Just in case they’re not, though, let me state them explicitly:

Don’t Take Their Word for It!

You should not trust others to tell good science from bad. You should especially not trust people who are trying to sell you something.

Remember the importance of scepticism, and remember what it means to be sceptical. Ask questions. Investigate. Criticise. Don’t take their word for it.

Advertisements

One thought on “Don’t Take Their Word for It!

  1. Mark Hanna …

    A few points you may be interested in.

    (a) Most of the studies are BS or so inaccurate they can not be taken seriously. 99% of machines are fake.

    (b) Gyms can not buy real machines. They are too powerful and PTs are too stupid. Very bad combination.

    All they get to do is jump around on light Physio machines and try to look like they know what they are doing.

    (c) The company Power Plate you mentioned is run by marketers. They could not lie in bed straight.( I know because I used to be their boss ) . It truly astounded me what they thought was acceptable practice.

    They have turned a promising idea into a joke. But the real industry is still here.

    The only way you will think we are not like that is for you to come in and see me. I promise it will not be a waste of your time. It may help you piece

    Please note: This is not a marketing ploy. It can just be between you, me and your skepticism. But sorry, cant fix the dodgy studies.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s