When Marketing Trumps Science

Recently I’ve run across a couple of New Zealand companies that sell therapeutic products – one a weight loss pill, the other a jet lag drink – that seem to put marketing first and let science take the back seat. This is by no means new behaviour, but I want to use them as examples to illustrate this widespread problem and suggest what can be done to combat it.

Before promoting a therapeutic product, you should first have good reason to believe that it works. This, I hope, is common sense, but it’s also enshrined in the Fair Trading Act and the Therapeutic Products Advertising Code as they prohibit unsubstantiated claims. This means you have to test the product, and do so rigorously. Rigorous clinical trials are expensive to undertake though, so they’re quite a prohibitive first step.

Instead of jumping straight into the deep end, a useful first step can be to undertake a smaller and less rigorous (and therefore less expensive) experiment. In order to answer the question of whether or not a product actually works you need to conduct a more rigorous trial. There’s a great cost involved in doing this, but if the results of a preliminary trial are optimistic then you have reason to expect a more rigorous trial might give similar results, so the expense might be worth it. You may even be able to get some funding to help with a more rigorous trial on the basis that the preliminary results were positive.


This is what Tuatara Natural Products has been doing with their weight loss pill “Satisfax”. They have completed a low quality preliminary trial on their product, which has been colloquially dubbed the “Fat Mates” trial. Although I don’t believe it has yet been published in a journal at the time I’m writing this article, it was registered retrospectively in the Australian New Zealand Clinical Trials Registry: Effect of the dietary supplement Satisfax (Registered Trademark) on weight loss in overweight volunteers. Some information on the data in the trial can be found on their website as part of an analysis by Dr Chris Frampton: Analysis of Satisfax® Clinical trial

As you can see on the trial registration page, this was an uncontrolled trial on overweight adults. The original plan was to recruit 100 volunteers with the hope that at least 60 will complete the trial. I have to say I’m a bit confused about how many people were in the trial, as apparently the recruitment was increased to 200 applicants after applications opened (a change that has not been reflected in the trial’s registration) yet apparently about 400 people applied. One article claims there were 200 participants, a later media release from Tuatara Natural Products seems to imply 100 were recruited, and the analysis of the trial says there were only 81 participants. Either way, 81 participants completed the full 8 weeks, and 52 of them took the recommended dose for the whole duration of the trial.

If there were 19 or 119 participants who didn’t complete the trial, the statistical analysis seems to ignore them with no justification given. This is unusual – a 19% drop out rate is significant and shouldn’t be swept under the rug. A lot of the time Tuatara also seems to ignore the 29 participants who didn’t drop out but also didn’t take the full dose for the whole duration.

The Science Media Centre posted the responses of 2 experts, Associate Professor Andrew Jull and Professor Thomas Lumley, to a press release from Tuatara Natural Products in February. It’s a good analysis of some of the weaknesses with the study, and I recommend you read it: Kiwi diet pill claims – experts respond

This trial was uncontrolled, and therefore also unblinded and unrandomised. As Professor Lumley explains, this is a problem if you want to draw strong conclusions from its results. It is of low methodological quality, but that’s okay. There is no problem with doing less rigorous trials first if they’re done in order to determine if more rigorous trials are necessary. Dr Glenn Vile, Chief Technical Officer of Tuatara Natural Products and the principal investigator for this study, wrote the following in a comment on a post on the “Fat Mates” trial by Dr John Pickering:

The Fat Mates trial was designed by clinical trial specialists to generate information about the Satisfax® capsules that would help Tuatara Natural Products plan a larger and longer double blind, cross over, placebo controlled trial.

We will use this information to proceed with the next clinical trial, but in the meantime we were so excited the weight loss achieved by most of our Fat Mates was much greater than the placebo effect seen in other weight loss clinical trials that we decided to launch the product so that anybody who is overweight can try Satisfax® for themselves.

Dr Glenn Vile

I think the first part of what I’ve quoted above describes exactly what Tuatara Natural Products should be doing. They’ve conducted their low quality trial, and intend to use its results to proceed with a larger, longer, and more rigorous clinical trial. This is the right way to proceed – they now have an indication that their product might be effective, so they should do the research to find out.

The problem is that that’s not all they’re doing. After performing only a small low-quality trial, they’ve released their product for sale online and have been making a lot of noise about it. In my opinion, they’ve been significantly overstepping the results of their clinical trial. For example, in his comment Dr Vile also said:

our initial trial has shown [Satisfax] to be extremely effective in some overweight people.

Dr Glenn Vile

In their media release on the 20th of February, they reported the average weight lost only by the 52 participants who took the full dose to completion (rounded up from 2.9 kg to “close to 3kg”) but not the average weight lost by all participants. They then reported in bold that the top 26 participants lost more weight, and the top two participants lost even more weight than that!

This cherry picking of the best results appears to have been part of Tuatara Natural Products’ marketing strategy for at least a few months now. In January, Stuff published an article on the trial highlighting the single person who lost the most while participating in it: Blenheim ‘fat mate’ loses 13.5kg in 8 weeks.

That article particularly highlights the person who lost the most weight out of all those in the trial, at 13.5 kg (confusingly, the maximum weight loss reported in the analysis of the trial’s results is 13.3 kg). However, she was one of only 2 participants who lost over 10 kg, and on average the 52 participants who took the recommended dose for the full eight weeks lost 2.9 kg. Losing 13.5 kg is very far from a representative example. I’m not surprised that they didn’t choose instead to focus on the participant who gained 1.2 kg despite taking the recommended dose for the whole duration, but that is actually much closer to the mean change in weight.

The article is, for all intents and purposes, one big testimonial in favour of Satisfax. It was an article, not an advertisement, which is important because in New Zealand it’s illegal to publish any medical advertisement that:

directly or by implication claims, indicates, or suggests that a medicine of the description, or a medical device of the kind, or the method of treatment, advertised… has beneficially affected the health of a particular person or class of persons, whether named or unnamed, and whether real or fictitious, referred to in the advertisement

Medicines Act 1981 Section 58(1)(c)(iii)

This effectively bans all health testimonials from advertisements. I think this is a good part of the law, as testimonials can be both very convincing and completely misleading; a quack’s dream. Banning them should force businesses to instead focus on the results of research on their products, but this hasn’t stopped Tuatara Natural Products from getting stories written about the most extreme testimonials they could find from people who have lost weight at the same time as they were taking Satisfax.

More recently, Tuatara Natural Products has put out a press release multiple times (at least on the 20th of February and again on the 4th of March) that I think rather oversteps the results of their small preliminary trial:

A NEW ZEALAND SOLUTION TO A GLOBAL PROBLEM A little pill is providing an exciting answer to one of the worlds greatest and fastest growing problems: Obesity.A NEW ZEALAND SOLUTION TO A GLOBAL PROBLEM

A little pill is providing an exciting answer to one of the world’s greatest and fastest growing problems: Obesity.

Press Release – Tuatara Natural Products

I simply don’t think they are at all justified in saying that their new product is “providing an exciting answer to… Obesity”. They are putting marketing ahead of science, and that’s not okay.


Another company that seems to put marketing before research is 1Above. They make a drink which they claim can help you recover faster from jet lag, and have recently been in the news for signing a sponsorship deal with the fantastically successful golfer Lydia Ko.

At the end of that article about their sponsorship deal the reporter, Richard Meadows, made some comments regarding the science behind jet lag relief products and asked some good questions of 1Above’s CEO, Stephen Smith (emphasis mine):

[1Above’s] product contains a mixture of vitamins B and C, electrolytes, and Pycnogenol, a pine bark extract.

The efficacy of flight drinks to combat the effects of jetlag is unproven.

Late last year pharmacists were warned after the Advertising Standards Authority upheld a complaint against an ad saying a homeopathic anti-jet lag pill really worked.

[1Above CEO Stephen] Smith said 1Above would not be doing clinical trials, which were highly expensive and not necessary.

“What we tend to use is testimonials from people who have used the product and swear by it.”

Smith said the key ingredient, Pycnogenol, had itself had been tested in dozens of trials, including its effects on reducing jetlag.

Kiwi startup 1Above signs golf No 1 Lydia Ko

Yes, you read that correctly. The CEO of 1Above literally said that they won’t be doing clinical trials because they are “not necessary” and that they use testimonials instead.

As I said before, using testimonials to promote a therapeutic product, like a drink to help you recover faster from jet lag, can be both very convincing and completely misleading. There’s a reason why testimonials implying health benefits are illegal in New Zealand, and I hope that 1Above’s marketing will not violate this regulation.

Not all testimonials are prohibited, of course. It’s entirely acceptably to provide a testimonial from someone who thinks their drink tastes great, or that they provide great service. Basically anything for which a single person’s experience can provide a useful insight. Therapeutic effects, almost without exception, do not fall into this category, which is a big part of why we need to do clinical trials in the first place. If they quote someone in saying that their product helped them recover faster from jet lag, they may be in danger of breaching the Medicines Act.

For example, I’d expect they probably shouldn’t use a testimonial that says this:

Testimonial on the 1Above website, collected 2015/03/05
Testimonial on the 1Above website, collected 2015/03/05

On their website, 1Above currently does refer to research on one of the ingredients in their product, “pycnogenol”. Professor Lumley recently wrote a post about this on his other blog, Biased and Inefficient, regarding these studies and how they are used by 1Above: Clinically Proven Ingredients

I recently contacted 1Above to ask about some discrepancies I found between the abstract of the study they cited for showing pycnogenol reduced the duration of jet lag and their description of it on their website:

I was interested to see the claim your company made that Pycnogenol® has been shown to support circulation and reduce the length and severity of jet lag.

I have found the study “Jet-lag: prevention with Pycnogenol. Preliminary report: evaluation in healthy individuals and in hypertensive patients” that is mentioned on your website as the source for this claim, but I am only able to access the abstract of this preliminary report. Unfortunately, as far as I can tell, the study protocol didn’t involve blinding of participants or researchers.

The participants in the study took 50 mg Pycnogenol 3 times per day, but I haven’t been able to find out how much is contained in your products. Is this information available anywhere on your website? I notice the study also says the participants took this regimen for 7 days, starting 2 days prior to departure. Is this comparable with how your product is intended to be used?

I also noticed some differences between the description of the study and its results between the abstract and your website, I would be grateful if you could explain to me the source of these differences.

The abstract states the control group took, on average, 39.3 hours to recover and the experimental group took, on average, 18.2 hours to recover. However your website reports these as 40 and 17 hours respectively.

Also, your website states that the study involved 133 passengers (it’s not clear from the description on your website if they all took Pycnogenol or if some of them were in the control group) who reported the time it took them to recover from jetlag. However, the study’s abstract states that in the first experiment, which is the only one that involved the reporting of the time taken to recover from jetlag, only involved 68 participants – 30 in the control group and 38 in the experimental group.

I would be grateful if you could explain these differences to me, and if you could send me any other relevant scientific information that supports this claim.

To their credit, since receiving my message they did update their website to fix the discrepancies in the reported number of participants and times taken to recover from jet lag, and their CEO replied to thank me for pointing these discrepancies out.

However, they didn’t respond to my other questions about the amount of pycnogenol in their products or the study involving the participants taking pycnogenol for 7 consecutive days, starting 2 days before their flight, which is inconsistent with how 1Above recommends their products be used.

This is just one more company basing their marketing on preliminary trials instead of using them as the basis for research that could actually answer the question of whether or not a product is useful. Worse than Tuatara Natural Products, they even go so far as to consider clinical trials “not necessary” and apparently intend to rely on testimonials instead. It would be much more appropriate for them to spend some of their $2.4 million annualised income on researching their product rather than paying for a sporting celebrity to endorse them.


I try to make my rants constructive, so I want to end this article with the question “What can we do about this?”. If you have any suggestions, I’d love to hear them in the comments section.

I think the most important thing that anyone can do to address this problem is to ask for evidence. If you see a claim made about a product that you think you might buy, then get in touch with the company selling it to let them know you’re considering buying it and to ask for evidence. If they don’t have a good enough answer, then let them know that’s why you won’t be buying their product. If they give you evidence to back up their claim, then great!

The UK organisation Sense about Science has created a website for just this purpose: www.askforevidence.org

Asking for evidence doesn’t have to be a big deal, involving a formal letter or anything like that. When you see a weight loss product advertised on a one day deal site, a copper bracelet that apparently offers pain relief advertised on a store counter, or a jet lag cure promoted on Twitter, make your first response be to politely ask for evidence.

This isn’t a problem that’s going away any time soon. As consumers, we deserve to be able to make informed decisions about the products we buy, and when companies put marketing before research it becomes harder to make these informed choices. But if we work together then we can encourage companies like Tuatara Natural Products and 1Above to improve their behaviour and attitudes toward marketing and research.

Let’s turn “what’s the evidence?” into a frequently asked question for all companies that sell therapeutic products.

Advertisements

ACC and Acupuncture

Over the past year or so, various Official Information Act (OIA) requests have been made via the website fyi.org.nz, which allows for these requests and their responses to be made public. I have written previously about the results of some of these requests regarding ACC and Alternative Medicine. I’ve also shared some more information on Twitter, which Thomas Lumley has documented on the StatsChat blog: Sticking it to ACC

Following my previous post on this matter I also wrote to the ACC Minister, Judith Collins, in May to outline my concerns. Here’s what I said in my email:

Dear Ms Collins,

My name is Mark Hanna. I’m a consumer advocate from Auckland with a particular interest in alternative medicine. Over the past year and a half I’ve been actively trying to reduce the amount of misinformation regarding alternative medicine so often seen in advertisements, mainly by lodging complaints with the Advertising Standards Authority.

Last year, prompted by an earlier Official Information Act request, two friends of mine submitted OIA requests of their own to the Accident Compensation Corporation inquiring as to how much funding it had provided to “alternative therapies”:

The latter request has attracted some media attention: Big bill for alternative health

In looking further into this issue, I found that in 2004 Dr Paul Hutchison submitted a number of very interesting questions to the Minister of the ACC regarding their funding of acupuncture, such as how much had been paid, how much they expected to pay in the future, and whether the treatments covered were supported by scientific evidence.

I’m also aware, largely due to its mention in the recent article, and also due to this article from 2009, that in 2009 then-ACC Minister Nick Smith said the ACC would review the effectiveness of these treatments. Unfortunately, it seems no such review has been conducted.

Piecing together the information released by the ACC regarding their funding of acupuncture, I believe it paints a rather disturbing picture. I have attached a chart constructed from the data released by the ACC in response to your 2004 questions, and their updated expenditure data released earlier this month.

Now, of course, this wouldn’t be a problem if the ACC’s funding of acupuncture were due to a strong evidence base in favour of its effectiveness and safety. Unfortunately, this does not seem to be the case.

The earlier OIA request I mentioned asked the ACC how they measure the effectiveness of treatments that they fund. Their answer included the statement that:

With respect to processes and requirements for measuring effectiveness, Treatment Profiles have been developed for 150 common injury types. Treatment Profiles are guides to the treatment and rehabilitation services ACC expects a practitioner to provide to a client for a particular injury. They describe current ‘good practice’ and what outcomes should be achieved.

However, viewing the ACC’s Acupuncture Treatment Profiles document, I find instead that it is based on statements such as these:

  • “Acupuncture… Relieves pain by treating stagnation of Qi and Blood in the affected areas and channels”
  • “Qi is the vital force of life which… Is the material substrate of the Universe”
  • “Brain… Is considered to be the same in substance as marrow”
  • “the Kidney produces marrow”

That is only a small selection of the many statements in that document that are clearly – even blatantly – no more than pseudoscience at best. They certainly do not give the impression of science-based decision making.

In light of all this, I believe it is high time the ACC conducted a rigorous review of the evidence regarding acupuncture so as to cease funding treatment options that do not appear to be effective or science-based.

As current ACC Minister, I hope you will be able to help facilitate what seems to me like a long overdue review of the effectiveness of acupuncture and related practices as they are funded by the ACC.

Sincerely,
Mark Hanna

Here’s the chart I attached to that email:

ACC Acupuncture Spending 1994-2013

I heard back from Ms Collins on the 18th of June, you can view her response here:

Response from Judith Collins 18 June 2014

In her response, she described review ACC undertook of some of the evidence regarding acupuncture in 2011:

In 2011, ACC’s research team also conducted a literature review of the efficacy of acupuncture in the management of musculoskeletal pain. It found the most convincing evidence for the effectiveness of acupuncture related to the treatment of chronic neck pain and the improvement of pain and mobility in chronic shoulder pain. Evidence on the effectiveness of acupuncture is not dissimilar to other physical therapies such as physiotherapy, chiropractic and osteopathy.

Since that response, another OIA request was submitted to ACC by Mark Honeychurch, co-founder of the Society for Science Based Healthcare, asking for a breakdown of acupuncture spending by condition. To make the request easier, the categories requested were the same as those used by ACC’s Injury Statistics Tool. You can view the results from that request directly as a PDF or as a Google Spreadsheet.

The vast majority of the spending (around 94% for each of the last 2 years) was for soft tissue injuries. However, without a more detailed breakdown we don’t yet know how many of this was supported by ACC’s findings regarding the evidence for acupuncture.

Although the amount of money is piddling in comparison, I have to admit I find it rather concerning to see that, over the past decade, ACC has spend $9,000 on acupuncture for deafness.

EDIT 2014/08/30

In a discussion of this post on Facebook, Jonathan Grady raised a good point about what this might have involved. It’s still just speculation, but I thought it was worth mentioning here:

Well, I know of no evidence that acupuncture is useful in the treatment of hearing impairment per se. There’s no plausible mechanism at work. But it could well, in some instances, be used plausibly to treat tinnitus, as tinnitus has a high component of subjective influences such as mood and levels of physiological and psychological arousal (similar to perceptions of chronic pain and of “phantom limb” syndrome). Placebos have some good efficacy in the treatment of tinnitus and, as we all know, acupuncture is really not much more than a placebo treatment. But it certainly won’t help you hear any better…

Jonathan Grady, Practicing Audiologist and Consulting Expert Member of the Society for Science Based Healthcare


I currently have an outstanding OIA request with ACC, asking for their 2011 review as well as any others they’ve conducted regarding acupuncture, and any guidelines they have for conducting these reviews.

Although my request asked for “copies of or links to all literature reviews regarding the effectiveness of acupuncture for any condition undertaken by ACC”, in their initial response to my request they only identified that I had asked for “A copy of The efficacy of acupuncture in the management of musculoskeletal pain conducted by ACC’s research team”.

I’m still waiting to hear back regarding if this means it’s the only such review, or if there are others as well. However, I have found other reports on the ACC website, like this one on the Effectiveness of acupuncture in selected mental health conditions, so I suspect there should be more reviews than just the 2011 one.

Yesterday, in response to a separate request from Kevin McCready, ACC has released this 2011 review. Although the names and qualifications of the author or authors were also requested, ACC denied this part of McCready’s request and removed that information from the report:

ACC declines to provide the names and qualifications of the staff members who compiled the literature review as there is a need to protect people’s privacy. We have also been unable to identify why the release of this information would be in the public interest.

You can find the full report, minus the name of the reviewer, here: The efficacy of acupuncture in the management of musculoskeletal pain

Here are the “Key findings” of this review:

General

  • There is insufficient evidence to make a recommendation for the use of acupuncture in the management of acute neck, back or shoulder pain
  • There is emerging evidence that acupuncture may enhance/facilitate other conventional therapies (including physiotherapy & exercise-based therapies)
  • There is a paucity of research for the optimal dosage of acupuncture treatment for treating shoulder, knee, neck and lower back pain
  • Studies comparing effective conservative treatments (including simple analgesics, physical therapy, exercise, heat & cold therapy) for (sub) acute and chronic non-specific low back pain (LBP) have been largely inconclusive

Lower back

  • The evidence for the use of acupuncture in (sub)acute LBP is inconclusive
  • There is limited evidence to support the use of acupuncture for pain relief in chronic LBP in the short term (up to 3 months)
  • The evidence is inconclusive for the use of acupuncture for long term (beyond 3 months) pain relief in chronic LBP
  • There is no evidence to recommend the use of acupuncture for lumbar disc herniation related radiculopathy (LDHR)

Neck

  • There is good evidence that acupuncture is effective for short term pain relief in the treatment of chronic neck pain
  • There is moderate evidence that real acupuncture is more effective than sham acupuncture for the treatment of chronic neck pain
  • There is limited evidence that acupuncture has a long term effect on chronic neck pain

Shoulder

  • There is good evidence from one pragmatic trial that acupuncture improves pain and mobility in chronic shoulder pain
  • There is limited evidence for the efficacy of acupuncture for frozen shoulder
  • There is contradictory evidence for the efficacy of acupuncture for subacromial impingement syndrome

Knee

  • There is no evidence to recommend the use of acupuncture for injury-related knee pain

Ankle

  • There is no evidence to recommend the use of acupuncture for ankle pain

Regarding the evidence found for neck and shoulder pain, it seems important to note a couple of caveats:

The “good” evidence for shoulder pain was from a single trial: Molsberger et al. 2010. This trial had a decent sample size of 424, and compared Chinese acupuncture to sham acupuncture.

The abstract doesn’t specify any details of the sham treatment, but the full paper describes it as involving the same number of needles and number and frequency of treatments, but the needles were placed at non-acupuncture points and inserted less than 5 mm (compared with 1-2 cm in the acupuncture group). I was glad to see one of the inclusion criteria was that participants be naïve to acupuncture, so they could be effectively blinded to this.

The full paper makes it clear that “Treatment assignment… was known to the acupuncturist” so this study wasn’t double-blinded.

The report also notes that its conclusions contrast with an earlier Cochrane review on Acupuncture for shoulder pain, which found that:

There is little evidence to support or refute the use of acupuncture for shoulder pain

The Cochrane review was carried out in 2005, though, prior to some of the studies examined in the ACC review such as Molsberger et al 2010.

It's surprisingly hard to find a stock photo of acupuncture involving gloves
It’s surprisingly hard to find a stock photo of acupuncture involving gloves

While the overall evidence for neck pain was described as “good”, when compared with sham acupuncture (which is the most appropriate control if you want to account for the elaborate placebo effect of acupuncture) the evidence was not so strong. It’s not clear what the sham protocol was, or if it allowed for the blinding of both participants and practitioners.

It seems Judith Collins’ description of the review’s conclusions was taken practically verbatim from its “Summary Message”, although she also seems to have opted to remove the sentence regarding the evidence for injuries other than those involving chronic neck and shoulder pain, which I’ve highlighted below:

The evidence for the effectiveness of acupuncture is most convincing for the treatment of chronic neck and shoulder pain. In terms of other injuries, the evidence is either inconclusive or insufficient. The state of the evidence of the effectiveness of acupuncture is not dissimilar to other physical therapies such as physiotherapy, chiropractic and osteopathy.

For a more direct comparison, here’s her wording again:

In 2011, ACC’s research team also conducted a literature review of the efficacy of acupuncture in the management of musculoskeletal pain. It found the most convincing evidence for the effectiveness of acupuncture related to the treatment of chronic neck pain and the improvement of pain and mobility in chronic shoulder pain. Evidence on the effectiveness of acupuncture is not dissimilar to other physical therapies such as physiotherapy, chiropractic and osteopathy.


Despite the findings of this report, ACC still seems to be spending a lot of money on treatments that their own review concludes are not supported by evidence. It’s hard to quantify exactly how much of their expenditure on acupuncture – over $24,000,000 over the 2013/14 financial year – went toward these treatments, but Daniel Ryan (also from the Society for Science Based Healthcare) has submitted another OIA request asking for a further breakdown of spending by body part.

In their response to Kevin McCready’s request, however, they have said that:

ACC funds acupuncture for conditions where prescribed by the client’s medical professional. This includes conditions other than musculoskeletal pain, and for non-chronic pain.

I’m trying to give them the benefit of the doubt, but from this it sounds like their review of the evidence may play no part in the decision of whether or not to fund acupuncture treatment for any particular condition.

Instead, it appears ACC will fund any acupuncture treatment so long as the patient was:

referred for acupuncture by a health professional who is covered by the Health Practitioners Competence Assurance (HPCA) Act.

Perhaps this explains ACC’s worrying funding of such treatments as acupuncture for deafness, but I certainly don’t think it excuses it.


Although the report makes an “Important Note” at the top that “The content [of this document] does not necessarily represent the official view of ACC or represent ACC policy”, I did find one particular section in the background section quite interesting. First off, TCM acupuncture is described as:

Drawing of the human body showing acupuncture meridians

TCM acupuncture involves inserting needles into traditional meridian points with the intention on [sic] influencing energy flow within that meridian

As far as I’m aware, that’s a fairly accurate description of how modern acupuncturists tend to claim their treatments work. For example, the New Zealand Register of Acupuncturists’ explanation of How Acupuncture Works claims that:

Any disruption or blockage of the flow of Qi along the meridians will in time affect the associated organ, resulting, for example, in decreased function or pain. Acupuncture aims to correct this flow of Qi, and thereby restore the balance within the body

This explanation also seems to be reflected in ACC’s Acupuncture Treatment Profiles document. However, the reviewer then went on to say that:

many physicians currently practicing acupuncture reject such pre-scientific notions described above

I think that description of acupuncturists’ claims to be able to manipulate qi flow with their needles as “pre-scientific notions” is absolutely spot on. The existence of qi and meridians is not supported by any evidence, and when this practice was developed it was based more on philosophy than evidence. In that way, it’s similar to extinct medical philosophies such as the “Western” medical philosophy of humorism.

One aspect of acupuncture that I find illustrates quite well the fact that its development was not supported by evidence is that horses are said to have a gall bladder meridian. This gall bladder meridian has even been the subject of published papers in journals such as the “Journal of Acupuncture and Meridian Studies”. What’s so odd about that, though? Horses don’t have a gall bladder.

Statements such as many of those found in ACC’s Acupuncture Treatment Profiles, which was “developed by the New Zealand Register of Acupuncturists Inc. in consultation with the New Zealand Acupuncture Standards Authority in a joint initiative with ACC”, are pretty self-evidently pseudoscientific. For example:

“Qi is the vital force of life which… Is the material substrate of the Universe”

If I want to know about the “vital force of life” or the “material substrate of the Universe”, I’ll ask a biologist and a physicist, respectively. So I asked a biologist and a physicist what they thought of these claims regarding “Qi”. Here’s what they had to say:

From my perspective as a biologist, that quote about Qi bears a lot more relation to the old and discredited concept of ‘vitalism’ than it does to modern biology. There is no evidence Qi exists, let alone that the lines along which Qi is supposed to flow through the body exist (and indeed, different ‘schools’ of acupuncture seem to deal in different ‘meridians’ – they can’t all be right!)

Alison Campbell, Senior biological sciences lecturer at the University of Waikato

I think it is self-evident that modern physics does not hold that qi is the “material substrate of the universe”; I would just say that qi is not something you find in physics textbooks; going beyond that might be beating a dead horse.

Richard Easther, Professor and Head of Department of Physics at the University of Auckland

(Alison and Richard often have interesting things to say. If you’d like to follow them, Alison blogs at Bioblog, and Richard at Excursionset.com. You can also follow him on Twitter @REasther.)


Now you have some insight into what I consider to be the problems with ACC’s funding of acupuncture. I’ve had people suggest to me, though, that some people do seem to benefit from acupuncture, so perhaps I shouldn’t be so interested in removing it from ACC.

Before I respond to that, I want to make one thing clear. I am not and will not argue for preventing people who want acupuncture from having acupuncture. Patient autonomy is an immensely important aspect of healthcare, and people have the right to make their own healthcare choices.

My primary motivation, as with all my involvement in dealing with dodgy healthcare claims, is echoed in the mission statement of the Society for Science Based Healthcare:

We believe that a strong basis in rigorous science is a necessary prerequisite for providing safe and effective healthcare. Decisions regarding public funding of healthcare in New Zealand should therefore be science based. We support public health measures that have a clear basis in science and evidence, and oppose those that do not.

We will work to counter misinformation about health issues propagated by individuals and organisations in New Zealand.

Consumers have the right to make an informed decision about their healthcare, and should not have to worry about being misled by unsubstantiated claims.

People have the right to make their healthcare choices, but they should not be misinformed by claims like “this needle will unblock qi flow through your meridians”.

Also, and this is more relevant to the case at hand, a treatment that is not supported by rigorous evidence is at best an experimental treatment. Generally the only ethically appropriate situation for an experimental treatment to be used is in a well-designed clinical trial, although in certain cases an argument can also be made for compassionate use exceptions.

However, this is not what is being funded. ACC is funding acupuncture for indications for which there is either insufficient evidence, no evidence, or negative evidence, and it is funding it in a clinical setting.

Furthermore, the ACC registered acupuncturists that benefit from this framework are generally proponents of the “pre-scientific notions” of qi flow and meridians. Whether they intend to or not, I think ACC is effectively promoting this via their association with these practitioners, and through official documents such as the Acupuncture Treatment Profiles. I also think this promotion is likely to do harm to the public understanding of science, particularly medical science.

ACC funding implies governmental approval, and many more obviously ineffective services such as “energy therapies” make similar justifications to acupuncturists. If consumers are led to believe that practitioners who base their practice on “pre-scientific notions” like qi flow are reliable and trustworthy, it seems reasonable to me to expect that they will think the same of practitioners who base their practice on other concepts such as aura healing.

So, what do I think should be done? I think ACC needs to review its funding scheme for acupuncture. I think their approach to this should start with reviewing their Acupuncture Treatment Profiles document, ensuring that the only treatments contained within it are those supported by rigorous evidence, and purging pseudoscientific claims from it. If they find they need to undertake further reviews of the evidence for the use of acupuncture for particular indications, then they should do that before approving funding for it.

I think ACC should then only agree to pay for acupuncture treatments that are aligned with their Treatment Profiles document, which they should commit to reviewing at regular intervals to keep it in line with the latest evidence (I’m not sure what time interval would be most appropriate, and I understand that there is a cost involved in that work).

I’m not sure, but it’s possible some changes to legislation may be required before this becomes a reality, but if that’s the case those changes should happen. A government body should not be bound by law to fund healthcare that is not supported by evidence.

There’s one last thing I’d also like to see, although I really feel like this is a long shot. I think ACC should take an active role in discouraging healthcare practice based on the “pre-scientific notions” described in their 2011 review. I think they should do this by distancing themselves from those acupuncturists who promote it and who base their practice on it, by refusing to grant them status as registered ACC practitioners if they are found to rely on it.

But I won’t hold my breath.

ASA Complaint: Osmosis Skincare’s Drinkable Sunscreen

In May this year, One News ran a story on a US skincare company releasing what it was calling “drinkable sunscreen”. Around the world, various sceptical websites also picked up the story, such as Doubtful News and Neurological Blog. The message was roughly that a “drinkable sunscreen” is a cool idea that isn’t entirely implausible, but that this company’s “Harmonized Water” product seemed to be entirely ineffective pseudoscience.

I tweeted about the story from One News, noting that the article seemed like little more than free advertising of what really seemed like quite a dangerous product. Thomas Lumley, a professor of Biostatistics at Auckland University who runs the great blog Stats Chat, picked up this story and wrote about it there: Revolutionary new advertising success

He also pointed out to me on Twitter that this company, Osmosis Skincare, has a New Zealand distributor, and that they have a website. Here’s that website’s Harmonized Water product listing page. If you look at it now, you’ll luckily see that although it does list a large number of “Harmonized Water” products that almost certainly don’t do what is claimed about them, it does not include any products that claim to be able to be used as “drinkable sunscreen”. The reason for this is that the Advertising Standards Authority has upheld a complaint I lodged against their online advertisements for these products.


As part of submitting this complaint, I took screenshots of the advertisements. I’ve embedded these below so you can see the claims as they were originally made:

Osmosis Skincare - UV Neutralizer Tan

Osmosis Skincare - UV Protection No Tan

My full complaint is available for you to read, as well as Osmosis Skincare’s response and the ASA’s decision, on the ASA’s website. I encourage you to read it in full, but I’ve put some of the highlights in this article.


The gist of my complaint was, as usual, that I don’t believe the advertiser has any evidence to support the claims they were making about the product. I also argued that the advertisements “abuse scientific terminology in a way that seems intended to exploit consumers’ lack of knowledge”.

In my complaints I generally also argue that making misleading or unsubstantiated therapeutic claims is socially irresponsible, and when the ASA upholds my complaints they tend to agree. In this case though, I felt the advertiser went a step further:

The New Zealand Cancer Society website writes, on the dangers of unprotected sun exposure:

New Zealand has the highest rate of melanoma in the world, and other skin cancers are also very common. You can help reduce your risk of skin cancer by using sunscreen the right way.

By misleading consumers into believing they are protected when in fact they very likely are not, this advertisement is likely to increase their risk of contacting [sic] melanoma due to unprotected exposure to UV radiation from the Sun. This misrepresentation is highly irresponsible.

Soon after submitting my complaint, I saw that the British Association of Dermatologists had published a response to these products. It’s worth reading in full, but here’s a highlight:

We want to make it immediately clear at this stage, the formulation is 100% water and, as far as our experts are concerned, it is complete nonsense to suggest that drinking water will give you a Sun Protection Factor (SPF) of 30.

They also contacted Osmosis Skincare to ask what the “scientific basis” for their claims was. The full message and its response are available at the link. Osmosis Skincare confirmed that the product is 100% water and didn’t provide them with any evidence to support their claims.


In Osmosis Skincare’s response to my ASA complaint, they said they’d made some changes like calling the products “UV Neutralizer” instead of “UV Protection”. How they thought this made it acceptable is entirely beyond me.

They also said the following:

This is a new type of technology being used in this way and Head office can reference the internal research they did showing the product to be effective, but their independent clinical trial isn’t until the 28th of June, whereby they will put 30 people outside for one hour in San Diego, CA at noon supervised by a plastic surgeon. So perhaps we have some extra time to submit these results? We are told our UK distributor will also be conducting their own study. We have been selling this in New Zealand for the past couple of years without any issue.

I can’t say I was surprised to read that the “independent clinical trial” they were planning on would have a tiny sample size of 30 and be without a control group, let alone adequate randomisation and blinding.

DermNet NZ has a page on sunscreen testing and classification that says sunscreens in New Zealand are now tested in vitro. That makes perfect sense to me, partly because in vivo testing for something like sunscreen seems like it would likely be unethical (which is mentioned on DermNet’s page) and partly because the difference between sunscreen and no sunscreen – blocking UV radiation when placed on the skin surface – would presumably be much easier to test and measure than more complex medical outcomes.

I would also imagine that the placebo effect will not have a strong influence here, but that’s only a guess and I have no evidence to support that. However, a study like this would still need a control group to be able to tell how much of a difference the product made, and in such a design it would still be more rigorous to randomise participants and blind both them and the researchers to eliminate potential sources of bias.

It will be interesting to see if this trial is published, and what its methodology and findings are. Especially since they’ve publicly reported beforehand that an independent trial was due to be done.


The Advertising Standards Complaints Board seemed to agree with my complaint on all its main points. To quote the summary of their decision (which they note is not the decision itself, but the whole decision is available on their website):

The Complaints Board acknowledged the changes made by the Advertiser, however, it said that the amended advertisement was still misleading, abused the trust and exploited the knowledge of the consumer by stating that the product offered sun protection using scientific terminology without adequate substantiation. It said this was exacerbated within New Zealand as sun exposure can have significant negative effects in comparison with other countries.

Accordingly the Complaints Board said the advertisement was in breach of the Therapeutic Products Advertising Code and did not observe a high standard of social responsibility effecting a breach of the Therapeutic Products Advertising Code.

In their full decision, the complaints board noted that although Osmosis Skincare alluded to evidence, they didn’t actually provide any. They also raised the valid point that their US-based clinical trial’s “application in a New Zealand context considering the strength of the sun was questionable”. They also said that:

the advertisement was likely to abuse the trust and exploit the knowledge of the consumer by stating the product offered sun protection “30 x more than normal” and used scientific terminology like “isolates the precise frequencies” without adequate substantiation.

I have to applaud the complaints board here for taking a stand against this sort of language, which abuses scientific jargon in a way that makes the advertisement sound more authoritative than it should. As they would have been able to uphold the complaint on the sole basis that the claims are unsubstantiated, I’m glad they also decided to take on this language as well.


To their credit, Osmosis Skincare has quickly removed the advertisements for these products entirely from their New Zealand website, even before the ASA’s decision was released. However, as you’ll have seen if you clicked on the link to Osmosis Skincare’s “Harmonized Water” product listing page at the top of this article, they still sell a number of these products that also seem to make unsubstantiated therapeutic claims.

All of my complaints about misleading healthcare claims, including this one, are now submitted under the Society for Science Based Healthcare. If you’re interested in these complaints, have a look at their website. You can also keep up to date with their complaints on Twitter @SBHNZ.