Official information is being kept secret for longer than it should be. For the past few months, I have been gathering and analysing data from 12 government agencies, looking at how they handle requests made under the Official Information Act.
My findings reflect what many have observed more anecdotally: responses to requests for information are often sent at the very last minute, and seem to often be delayed unnecessarily. Though it can be difficult to demonstrate that a response was not sent “as soon as reasonably practicable” — a requirement under the law — in any particular case, looking at a larger data set reveals some of the strategies used to delay the release of official information.
Despite a repeated cycle of calls for change, people keep dying in police pursuits in New Zealand. Just today, a teenager and a child died as the result of a police pursuit in Palmerston North.
News stories like these keep appearing. In February this year I asked NZ Police to release a number of statistics regarding police pursuits so I could examine what, if anything, has changed. I began to write about it in March but didn’t end up publishing it, having intended to put it on a new “features” subdomain I’ve been working on where I’ll be able to do some more complicated stuff than WordPress will allow.
One of the statistics I included was the number of people who have died as a result of police pursuits since the most recent review began. I’ve gotten very sick of having to update that number. So out of that frustration, I’ve published my article. You can read it here: NZ Police pursuits keep killing people
What will be done about the Official Information Act’s accessibility problem?
Late last year, I published an open letter on the accessibility of the Official Information Act. I gave examples of inaccessible responses to requests under the OIA, and suggested a solution. I sent this letter to the Ombudsman, and later also forwarded it to Associate Minister of State Services (Open Government) Clare Curran and to Minister for Disability Issues Carmel Sepuloni.
I’ve now heard back from the latter two, and want to share their responses so others interested in this topic can read what’s being done to address it.
I’d also lodged an OIA request with the Ministry of Defence last year, as their Briefing to Incoming Minister document was released as a text-based PDF with watermarks and redaction. I’d asked them what software they used to produce this, and the associated costs, so I’d be able to give a specific recommendation to agencies who send me inaccessible responses in the future.
I’ve received a response to this request as well. As with the responses from Clare Curran and Carmel Sepuloni, you can find the text and PDF link to this response below.
Unfortunately, and somewhat ironically, all three responses were received in the form of image-based PDFs of documents that had been printed, signed, and then scanned. This is the primary accessibility issue I’d raised in my letter regarding OIA responses, as this format cannot be easily converted to text and is inaccessible to people who rely on screen readers. A couple of the responses also contained links that I’ve had to type out by hand instead, due to this inaccessible format. I’ve made the original PDFs available, and also converted them to text (by hand, so please assume any mistakes are mine), which you can find below:
Response from Clare Curran, Associate Minister of State Services (Open Government). Original PDF
Dear Mark Hanna
Thank you for your email on 30 November 2017, regarding the accessibility of information released to requesters under the Official Information Act 1982 (OIA). I note your concerns about the format some agencies choose to provide information in, which can render that information unsearchable by the recipient.
As you are aware, section 16(2) of the Official Information Act provides that information can be made available in a way preferred by the requester, unless there is good reason identified by the agency not to do this. In the first instance it is for the requester to ask the agency to make information, and in particular data sets available in a searchable format. If the agency refuses then this can be the subject of a complaint to the Ombudsman.
The State Services Commissioner has been delegated the functions under section 46 of the OIA by the Ministry of Justice. This function includes providing advice and guidance to agencies to act in accordance with the OIA. SSC has established an official information work programme which includes developing guidance for agencies. I think that the issues you have raised are worthy of further consideration.
I have asked the State Services Commission to consider whether accessibility of information should be part of the official information work programme that the Commission is undertaking.
Hon Clare Curran Associate Minister of State Services (Open Government)
Response from Carmel Sepuloni, Minister for Disability Issues. Original PDF
Dear Mr Hanna
Thank you for your email dated 30 November 2017, regarding accessibility issues with Official Information Act (OIA) practices and across government more generally. The Government is committed to being open and transparent and we expect the same of the public service.
I have been advised that the Official and Parliamentary Information team within the Ministry of Social Development will soon utilise more sophisticated redaction software. This software will enable the Ministry of Social Development to send and publish OIA responses and documents that are text searchable and text recognisable for those requesters who use reading software.
Providing OIA responses in accessible formats should be standard practice as part of an open and transparent government. To this end, the Ministry of Social Development is leading a work programme to increase the accessibility of information and communications across government agencies. This will involve providing clear and practical advice to support government agencies to meet accessibility requirements for all their communications with the public. Participating Chief Executives have signed up to an Accessibility Statement to commit to this. You can find more information about this here: www.odi.govt.nz/nz-disability-strategy/outcome-5-accessibility/
Thank you for writing. I hope this information is helpful.
Hon Carmel Sepuloni Minister for Disability Issues
Thank you for your email of 7 December 2017 regarding the software used to create the redacted and watermarked 2017 Briefing to the Incoming Minister of Defence, published on the Beehive website.
The Ministry of Defence recognises that publishing a document online as a text-searchable PDF file makes is easier to access. We prepare material for release in this form wherever possible, and endeavour to provide alternative formats on request for non-searchable documents, such as older files.
In order to produce searchable, redacted and watermarked files, the source documents are generally prepared using Microsoft Office software and then exported as searchable PDF files. Adobe Acrobat Professional software is then used to make redactions and add watermarks.
The Ministry has an agreement with the New Zealand Defence Force on software licensing. The Defence Force pays for an all-inclusive software bundle and is charged a flat rate. They distribute the licenses for Adobe Acrobat Professional as needed by staff. Please refer to the Adobe Acrobat website for costs: https://acrobat.adobe.com/nz/en/acrobat/pricing.html. The cost to agencies will be affected by volume. Assuming multiple licenses are purchased, the Adobe website indicated pricing of A$22.99 per seat per month (about NZ$25 at present exchange rates) or A$275.86 per year (about NZ$300).
Under section 28(3) of the Official Information Act 1982 you have the right to request the Ombudsman to investigate and review this response.
Helene Quilter Secretary of Defence
It’s not clear yet whether or not accessibility of information will be part of the SSC’s official information work programme that Clare Curran mentioned. Though I’m hopeful that it will be, I’m not sure how likely this is to effect change.
I strongly agree with Carmel Sepuloni’s statement that “Providing OIA responses in accessible format should be standard practice as part of an open and transparent government”, and hope that she will do her best to ensure our government meets that standard. I’m cautiously optimistic, though the fact that this response itself was delivered in an inaccessible format isn’t encouraging.
For the foreseeable future, it sounds like the best course of action will be to ensure that all OIA requests specify that the response should be delivered in an accessible, searchable format. I have included this recommendation in the OIA Guide that I published last year.
If I receive inaccessible PDF responses in the future, I now intend to direct the agency to the Ministry of Defence’s response regarding the software they use.
There is a hashtag on Twitter that’s often used by journalists and activists, myself included, to talk about issues with the Official Information Act: #fixtheOIA
Today Nikki Macdonald, senior feature writer at the Dominion Post, started a discussion about a widespread and rather infuriating practice for responding to requests for official information:
Is there a reason – other than being obstructive – that government departments insist on sending OIA data as scanned pdfs, so you have to transcribe everything to be able to do anything with the numbers? #fixtheOIA
In response to a recent request she made under the OIA, she received a PDF of a scanned document that included a large amount of tabular data.
The issue she mentions is one I have seen many times. It is common practice to take a document that will be released under the OIA, print it, then scan it back into a PDF, then send that PDF instead of the original document.
Printing and scanning documents allows for them to be signed, which is typically done with OIA response letters (though not with documents released alongside them). PDFs cannot be altered by the recipients, so the information is kept intact as it was released.
Also, perhaps most importantly, because these PDFs come from scanned documents they don’t contain any text. Instead, they just contain images of text. This means there is absolutely no way the recipient will be able to view information that has been withheld from released documents.
But releasing documents as images of text has a huge drawback too: accessibility
Most obviously, this will affect requestors who are blind or have impaired vision. If you rely on a screen reader to read these documents, it won’t be able to read a PDF that’s full of images. It’s a pain in the ass for those of us who are fully sighted as well.
There is software that can be used to convert images of text back into text, but it doesn’t always work perfectly, and often these image-based PDFs will include tables of data that are only really useful once converted into a spreadsheet. And, frankly, knowledge of how to use specialised software shouldn’t be a requirement for using the OIA to get usable information.
This isn’t the only accessibility issue with the OIA. Government agencies have also been known to do things like refuse to release documents online for unclear “security reasons“, instead telling the requestor that they must pay to have it posted to them.
Recently, Sam Warburton has been documenting a particularly trying OIA response on Twitter. First, he was sent documents that were locked by a password, which prevented him from copying text out of them:
Spreadsheets were provided on a USB, but password protected. The only thing I could do was view sheets. No copying. Not pasting. No editing.
He then made a second OIA request asking for all correspondence regarding the first request, so he could get to the bottom of how this bizarre twist had come about. The response was, well, not very accessible. He was sent 249 pages of printed documents, with the excuse being that it was “too big” to send digitally:
I think we should be able to expect better. Far better, in fact. And I’ve written the following open letter to the Ombudsman in the hope that this aspect of using the OIA can be improved:
There has been a discussion on the “#fixtheOIA” hashtag on Twitter started by Nikki Macdonald today about the common practice of agencies sending responses to requests for official information in PDF format, where the contents of the PDF must be either transcribed by hand or converted using specialised software before they are useful.
I’ve observed this practice across several agencies, including those I have most commonly made requests to: New Zealand Police, the Department of Corrections, and ACC. It seems as though the response is written up, then it is printed out and the physical copy is signed before being scanned, and finally the scanned copy is sent to the requestor as a PDF. This means that, effectively, the response is delivered in the format of an image of text.
This has obvious accessibility issues for requestors who are blind or have impaired vision, but is also detrimental to sighted requestors.
For example, I recently received such a response from the Department of Corrections which included some long URLs. It is not possible to copy text from these responses without specialised software that many requestors will be unlikely to know much about (OCR – Optical Character Recognition – software can be used convert images of text into actual text). In this case, I had to type the URLs out by hand. Though there were only three of them in this instance, a lengthier response could easily have made this a significant barrier to accessing the released information.
In the case raised by Nikki Macdonald, I understand she received a PDF response that, unexpectedly, also included large amounts of data. However, because the release was effectively an image of this data rather than a more usable format such as a spreadsheet, it will require a conversion process (either by hand or via OCR software) before it can be used.
I received a similar response in 2015, when I requested documents from the Ministry of Health but did not specify a preferred format. One of the documents I had requested was a large spreadsheet, but it was released to me as a PDF. Though this particular PDF was at least text-based, I still had to use specialised software to convert it to a usable format. I have made the PDF I was sent available online: compliance-of-comp-meds-manufacturers-2-oct-2007.pdf
While this issue can be pre-empted by requesting that the information should be released in a specific accessible format, such as an XSLX or CSV spreadsheet for data, this is not always realistic when it’s unclear whether or not the response is likely to include this type of information.
Despite section 16 of the OIA laying out certain responsibilities here, this approach is also not always respected by the agency. I have received multiple datasets as image-based PDFs despite requested explicitly as XSLX or CSV spreadsheets.
Here is an example of one response where I had to transcribe data by hand. The full response contains more data, as it continues onto the next page:
I have also assisted other requestors in transcribing information received this way, during which I was reminded that it can be quite a painful and time-consuming process for a requestor who is neither a fast typist nor able to use specialised OCR software.
In cases where OCR software is used to convert an image-based PDF response into a text-based one, which allows responses to be searched for certain keywords, this is often hampered by “Released under the Official Information Act” watermarks that are commonly overlaid on each page of released documents. These watermarks interfere with the software’s character recognition, so that portions of the text must instead be transcribed by hand. Poor quality scans, such as misaligned pages, can also cause issues with using OCR software to convert released information into an accessible format.
OCR software can also incorrectly transcribe parts of text, so it is always necessary to compare its output with the original PDF. This can be a time-consuming process, and is inaccessible to requestors who are blind or have impaired vision.
My impression, from talking with other users of the Official Information Act, is that this is a common and frustrating experience, widely regarded by requestors as being unnecessary and obstructive.
I accept that there may be cases where it would be difficult for an agency to release a text-based document with appropriate redactions, and where the reason for each piece of information withheld is clear and the withheld information cannot be accessed. However, there are clearly many cases where information could be released in a more accessible format, and where this would happen if the requestor had known to specify such a format in their request. I would hope that it might be expected that, when information can be released in any of several different formats with similar effort, the most accessible format should be used by default.
For example, after having seen me express frustration on social media about having to transcribe length URLs by hand from an OIA response, the Department of Corrections has agreed that they will now include links in the body of the email sent as a response. Although this only affects a small part of the response, this change would make the URLs significantly more accessible by allowing them to be used directly, without having to transcribe them from the PDF attachment.
I have looked through the Official Information Act and the official information legislation guides on your website for guidance on this accessibility issue.
The closest thing I have found is the relatively recent case note regarding a request from a prisoner who was unable to access information publicly available via the internet, after their request was refused under section 18(d).
Unfortunately, I have not found anything specific to this issue of accessibility. I’d greatly appreciate any advice you might be able to give on making requests in a way that should make a more accessible response more likely.
I would also like to suggest that a guideline on this issue might significantly improve both the experience of making requests for official information and the accessibility of released information.
I think it would be reasonable to expect agencies to release information with an accessibility-first mindset. This would be consistent with standards such as the New Zealand Government Web Toolkit, which recommends the following:
“Choose formats that are both easy to use and most likely to be accessible in the future. Choose openly documented formats over closed formats, or ensure that material released in closed formats is also accompanied by an equivalent in open formats.”
Particularly as many requests are made publicly available, either by the agency, through a service such as FYI.org.nz, or by direct sharing from person to person, it is important to consider issues such as accessibility for the blind and visually impaired for every request.
It is my opinion that accessibility should be considered an important part of information being made “available”, as per the Official Information Act’s Principle of Availability.
You have the right to ask the government for information. Because of a law called the Official Information Act (OIA), they’re obliged to give it to you unless there is a good reason not to.
You’ve likely seen the OIA mentioned in the news. Phrases like “Documents released to [news outlet] under the Official Information Act” can often be found in important news stories. It’s in indispensable tool for holding the government to account.
Some of my articles here have also been based on information that I only had access to because of the OIA. For example, my articles about ACC’s funding of acupuncture have all been based heavily on information released under the OIA.
An interactive visualisation I did of police use of force data released under the OIA found a place on the Herald Insights website earlier this year, and the story accompanying it was published on the front page. Organ donation statistics I’d requested also made the front page of the Herald in April of 2016.
It’s such a great and flexible tool, so I want to make sure everyone is able to use it.
It can feel a bit daunting if you think you might need to read the legislation itself to understand what you need to do. So to help make the OIA more accessible, I’ve written a guide to its use based on my experience with it.
Steffan Browning will leave his role as an MP next year, which is a great opportunity for the Green Party to ditch their anti-science baggage.
I have a love-hate relationship with the Green Party. I love their social policies, but as someone who dedicates a lot of my time to fighting pseudoscience I have a hard time justifying support for a political party with anti-science tendencies.
In the lead up to the 2014 general election, when I was considering where I would place my party vote, I emailed the Greens’ then health spokesperson Kevin Hague with some questions about Green Party health policy.
Hague’s response satisfied me that, despite the party’s reputation, references in their health policy to being evidence-based were more than just lip service. I voted for them.
Luckily, the response from Green Party leadership was pretty good. Browning’s “Natural Health” portfolio was taken away from him and folded into Hague’s health portfolio, after which then co-leader Russell Norman was pretty clear:
It’s not something we support and it’s not Green Party Policy.
The Green Party was awarded two awards by the NZ Skeptics at their 2014 conference. One, the Bent Spoon award, goes each year to “the New Zealand organisation which has shown the most egregious gullibility or lack of critical thinking in public coverage of, or commentary on, a science-related issue”. In 2014, it went to Steffan Browning.
But they also chose Russell Norman for a Bravo award
for quickly responding to Steffan Browning’s comments and stating that this was not something the Green Party would support as they take “an evidence based approach”.
After this wobble, it looked like the Greens had recovered and maybe taken another little step away from their anti-science past.
But since then both Russell Norman and Kevin Hague have left the Green Party. Though they are by no means the only great people in the Greens, I feel they had shown themselves to support evidence-based policy. I’ve been worried for some time now that it might signal a return to the Greens’ anti-science past, especially as Steffan Browning still held their GE portfolio despite his anti-science views on that topic.
The Greens’ reputation took another blow in my mind this year, as I discovered when researching DHB candidates for links to quackery that the Greens were backing Sue Kedgley in her stand for the Wellington City Council and the Capital & Coast DHB.
I felt strongly enough about this that I wrote to the Greens to express my disappointment.
When Steffan Browning put his foot in it soon after the election by supporting homeopathy for ebola, I worried I might have made the wrong choice. But the swift reaction from the party’s leadership again convinced me I’d done the right thing.
Now I see that the Greens are supporting Sue Kedgley as one of their candidates for Wellington City Council. I’m really, really disappointed about this. And it makes me worry for the party’s future.
With Kevin Hague now leaving the Greens to his new role at Forest & Bird, seeing this makes me very concerned about the current direction of the Green party. Steffan Browning still holds his GE portfolio despite his unscientific views in that area, and the party is throwing its weight behind a city council candidate like Sue Kedgley. It makes me think perhaps the Greens aren’t the evidence-based party I hoped they could be.
Unfortunately, Kedgley has now been re-elected to both the Greater Wellington Regional Council and Capital & Coast DHB
With all this context, I hope you can all understand why I’m happy to hear the news today that Steffan Browning is not seeking re-election in 2017.
This could be a great opportunity for the Green Party to shed their anti-science baggage and commit themselves to becoming the evidence-based party that many people, including myself, want them to become.
A good start would be re-addressing their stance on GE technology to align it more closely with scientific evidence.
But also, I feel like the time has come for the Greens to cut ties with Sue Kedgley. She hasn’t been on their list since 2011. Although Browning is stepping down as a Green MP voluntarily, this is a chance for the Greens to move past their anti-science past by cutting ties with Sue Kedgley.
Here’s hoping that, in 2017, they will be an evidence-based option.
There are lots of cool science activities you can do at home with light.
Like I’ve done almost every year of my life, I spent my summer break at my family bach at Oakura. Last summer I wrote a post about a trip to the rocks and what could be found living there. This summer, on the relatively few sunny days we had, I had fun playing with light.
Here are three easy, fun, and cheap activities you can try yourself.
The previous year, I made a simple telescope out of a $2 set of two magnifying glasses. Playing with trial and error and a piece of soft wood, I ended up with something that had a zoom of about 2x. However, because it only used two lenses the resulting image was inverted.
This summer, I came prepared with an extra set of magnifying glasses, making four in total. I raided the recycling bin and used some ginger beer bottles to hold them in place, facing an island in the bay. Then I moved them back and forth until the zoom and focus seemed as good as I could get it.
Once I had the placement right, I marked off the distances on a long piece of wood, then taped the magnifying glasses to it. What I ended up with wasn’t the strongest or most portable telescope in the world, but all it took to make was $4 and a fun afternoon.
See Shadows Jump
My brother Jeremy is a concept artist for Weta Workshop, which has left him with a good understanding of light and colour. One evening up at the beach he started talking about some interesting things that shadows do.
Watching shadows of leaves dance on the ground, he wondered if they often form natural pinholes. When we had a partial solar eclipse in Auckland in 2012, my mum (who also has a great artistic understanding of light and colour) mentioned to me how the shadows in her garden looked strange when she went outside during the eclipse. This would have been due to the pinhole effect, and it’s why some of the recommended ways of viewing an eclipse are to make a pinhole in a piece of paper or use a colander.
You’ve probably seen diagrams showing the basics of how a pinhole camera works. Even without a lens, when light passes through a small hole it can project a sharp image on a surface opposite that hole. However, that image will be inverted (like in my first attempt at making a telescope).
I often collect pāua shells from my trips to the rocks when I’m at the beach. A pāua shell has a row of holes along one side. When I held it a certain distance away from a wall, with the Sun low on the horizon, we found it made a row of pinholes. But because a projection of the Sun looks the same inverted as it does normally, in order to tell if the image really was inverted I moved a cardboard roll behind the pāua and watched at the holes “filled up” with shadow backwards – just as we’d expected.
But something else happened which I definitely didn’t expect. Watch this video we took to see the shadow of the pāua shell reach out to touch the cardboard roll’s shadow as they get close together:
I understand why the circles fill up backwards, but why do the shadows jump together like that to start with? pic.twitter.com/G6Zv18LTpq
If instead the pāua shell was held closer to the Sun and the cardboard roll was closer to the wall, then we found it would be the shadow of the cardboard roll that bulged out as they got close.
We immediately took to pen and paper to try to draw out diagrams that would explain how this worked. My initial idea was that we were seeing the area of intersection between the penumbras – the hazy edge of the shadows where the Sun was only partially obscured. But this wouldn’t explain why the bulge would change depending on which object was in front of the other.
Before too long, one of Jeremy’s ray diagrams seemed to explain what was happening. I’ve tried to reproduce them here (I hope you’re all suitably awed by my skills with MS Paint):
This diagram shows a light source on the left casting a shadow from the object in the middle onto the surface on the right. It shows how a non-point light source such as the Sun produces a shadow with an umbra (where none of its light reaches) and a penumbra (where part of its light reaches). The darkest part of the shadow, the umbra, is the middle section between the lines on the right.
Now, what would happen if I insert another object partly between the light source and the first object?
The new object blocks some of the light from reaching the original object. As this ray diagram shows with the red line – where the light is partially blocked – the result of inserting this second object is that the umbra of the first object’s shadow is extended toward the new object. This is the cause of the bulge you can see in the video above.
It turns out this shadow jumping effect is called the shadow blister effect. You can observe it easily for yourself on any sunny day.
Wave at the International Space Station
The sky at Oakura is lovely and dark, with the nearest city being nearly 50 km away. Before the Moon rose one night after Christmas a few of us went up a nearby hill to stare up at the night sky.
With a clear dark sky, you can see the band of the Milky Way galaxy arc across the sky like a pale cloud, as well as the fuzzy blobs that are the Large Magellanic Cloud and Small Magellanic Cloud. These are dwarf galaxies which orbit the Milky Way.
We also saw many meteors, and a surprisingly high number of satellites. From Earth satellites look just like stars, except they move steadily across the sky in a straight line. Usually they appear quite dim, but there is one satellite in particular which can shine brighter than any star in the sky, and even brighter than any of the planets. That is the largest artificial satellite of them all: the International Space Station (ISS).
The ISS orbits the Earth about once every 90 minutes, and although it doesn’t pass over New Zealand each time it does fly over us more often than you might think. But we can’t always see it in the sky; the conditions have to be right first.
Before we can see the ISS the sky needs to be dark enough for it to stand out. Also, it needs to be in the right position for sunlight to reflect down at us off its massive arrays of solar panels. This means that you’ll only be able to see it in the hours after sunset and before sunrise.
It generally takes 1-6 minutes for the ISS to pass visibly overhead. This will usually end with it appearing to fade into darkness as it stops reflecting sunlight back at us – you won’t see it set over the horizon like you would with the Sun or Moon.
NASA has a great online service, which you can subscribe to and get email alerts, that can tell you when and where to look to spot the ISS. It’s called Spot The Station. It lets you enter a city, and will tell you when the next few ISS sightings will be as well as how long they will last, and how it will travel across the sky.
ISS sightings often come in clusters – there will be sightings around a similar time in the morning or evening for several days in a row, followed by a period of no sightings. If you’re extra lucky, you might get to see it twice in one evening as it comes back round an hour and a half later.
I’d be remiss if I didn’t also mention that you can rent our bach if you ever want to see Oakura with your own eyes.
In February this year, an article was published in the Wairarapa Times-Age (both in print and online) headlined Use of natural remedies is on the rise. The article discussed some specific cases of so-called “natural remedies” being used to treat serious diseases, such as intravenous vitamin C for leukaemia and homeopathy to treat various forms of cancer.
Except for a brief mention at the bottom of the article encouraging people with cancer to talk to their doctor before using any “alternative therapies”, there was no mention of the fact that none of the treatments discussed are supported by any reliable evidence. Instead, the article uncritically included various quotes such as this, from homeopath Claire Bleakley:
Featherston-based homeopath Claire Bleakley said she has treated cancer patients using similar natural remedies [to intravenous vitamin C] – significantly extending life expectancy.
She mentioned two of her patients in particular: A man with tumours who lived for seven years after being given two to live, and a woman with ovarian cancer who lived 15 years past her initial prognosis.
“There have been some exceptional results,” Mrs Bleakley said.
“We are indoctrinated to think chemotherapy is the only cure for cancer, but alternative [remedies] have been proven to be more life giving.”
Medical anecdotes such as these unfortunately tend to be very convincing despite the fact that they can also be completely misleading. The reasons why people might get better are varied and complex. Without running a controlled test, there’s no way to know whether or not a particular treatment contributed to an improvement in health. That’s exactly why we need to undertake rigorous clinical trials before we can say with confidence what the effects of any particular treatment are. It’s also why the Medicines Act prohibits the use of health testimonials like this in advertisements, although that restriction of course doesn’t extend to news articles in publications like the Times-Age.
I thought, and still do think, that the lack of balance in this article has the capacity to do serious harm. I wrote to the editor of the Wairarapa Times-Age to make my case, and to give some suggestions for how they might attempt to mitigate the damage this article could do, in a formal complaint:
This article uncritically promotes the use of so-called “natural remedies” such as vitamin C or homeopathy for the treatment of cancer. They are promoted by the inclusion of quotes such as “There have been some exceptional results”, regarding the treatment of cancer with homeopathic products.
None of the relevant controversy regarding these treatments is discussed in the article. Although there is a brief note at the end that “those living with cancer [are encouraged] to consult their doctor or specialist before embarking on any alternative therapies”, this does not sufficiently address the important and relevant fact that these treatments are entirely unsupported by scientific evidence, as well as the utter implausibility of treatments like homeopathy.
The failure to discuss the lack of scientific evidence supporting these treatments, as well as the complete lack of plausibility underlying homeopathy, violates the Press Council’s principle of “Fairness, Accuracy and Balance”. The description of this principle on the Press Council’s website states that:
“Publications should be bound at all times by accuracy, fairness and balance, and should not deliberately mislead or misinform readers by commission or omission. In articles of controversy or disagreement, a fair voice must be given to the opposition view.
Exceptions may apply for long-running issues where every side of an issue or argument cannot reasonably be repeated on every occasion and in reportage of proceedings where balance is to be judged on a number of stories, rather than a single report.”
This is not a long-running issue in which readers can readily be expected to be familiar with the lack of evidence supporting the treatments discussed in the article, so the exception should not apply. There is significant controversy surrounding the issues discussed in this article, but a fair voice has not been given to the opposition view.
Particularly as this article could lead to people living with serious diseases such as cancer to rely on ineffective treatments such as homeopathy, its lack of balance has the potential to cause real and serious harm. Therefore it is important that the Wairarapa Times-Age take appropriate action to prevent this harm by amending the article, publishing a prominent correction, or publishing a followup article linked to from today’s article, that discusses the lack of evidence and plausibility underlying the treatments discussed in today’s article.
Despite sending a follow-up email a few days later, I still hadn’t heard back from the editor over the next 10 working days, which is the deadline set in the Press Council’s complaints process as the time to wait before escalating a complaint to them if you don’t hear back from the editor. After I forwarded my complaint to the Press Council, the editor contacted me to apologise that he’d overlooked my complaint messages, which was apparently due to his having to deal with another complaint about the same article from Peter Griffin, manager of the Science Media Centre (Peter is also the editor and manager of Sciblogs, where my blog is syndicated, and we’d discussed our complaints via email prior to submitting them).
When I forwarded my complaint to the Press Council, I fleshed it out a bit more. I won’t quote the whole thing here as a lot of it would just be repeating myself, although I’d be happy to share my full complaint if anyone would like to see it, but here is one part I added that I think is important and worth sharing:
As far as I’ve been able to tell, the Wairarapa Times-Age has not published a large number of articles regarding this, so it cannot be argued that the counterpoints have already been published in earlier articles.
When it comes to whether or not readers can be expected to be familiar with the important facts not mentioned in this article, I would like to bring the Press Council’s attention to a 2009 study (I am not aware of any more recent data collected on this) published in the New Zealand Medical Journal entitled “Beliefs about homeopathy among patients presenting at GP surgeries”. This study can be accessed for free on Page 94 of this PDF: http://www.nzma.org.nz/__data/assets/pdf_file/0010/17794/Vol-122-No-1295-22-May-2009.pdf
This study found that only 8 out of 124 respondents disagreed to some extent that “There is good scientific evidence that homeopathy works”, and only 24 respondents reported that they believed homeopathic products were either “very dilute” or that there was “nothing there”. In contrast, 82 respondents agreed to some extent that “There is good scientific evidence that homeopathy works”, and 80 believed that homeopathic products are either “Very concentrated”, “Moderately concentrated”, or “Moderately dilute”.
Contrary to these common beliefs, most homeopathic products are diluted to the point that it is astronomically unlikely that there is even a single molecule of the original ingredient present in the product, and there effectiveness is thoroughly unsupported by scientific evidence. For example, a rigorous review undertaken by the Australian Government’s National Health and Medical Research Council (NHMRC) in 2013 investigated the evidence regarding homeopathy for 68 clinical conditions and concluded that “The available evidence is not compelling and fails to demonstrate that homeopathy is an effective treatment for any of the reported clinical conditions in humans” (https://www.nhmrc.gov.au/_files_nhmrc/file/your_health/complementary_medicines/nhmrc_homeopathy_overview_report_october_2013_140407.pdf)
For this reason, and especially because the article discussed the use of ineffective therapies in the treatment of terminal illness, it is very important that stories such as this be balanced. As I stated in my original complaint to the editor, I believe the article in its current form has the capacity to do serious harm and that the Wairarapa Times-Age has a responsibility to mitigate this harm. An appropriate response would be amending the article, publishing a prominent correction, or publishing a followup article linked to from the article from the 10th of February that discusses the lack of evidence and plausibility underlying the treatments discussed in today’s article.
(The NHMRC link I provide there is from their 2013 conclusion. Within 2 weeks of submitting this complaint, however, they released their final statement on homeopathy, which states “there are no health conditions for which there is reliable evidence that homeopathy is effective”. This statement was not reported in the Wairarapa Times-Age)
Once the complaint was escalated to the Press Council, the editor of the Times-Age was given an opportunity to respond, then I had a final opportunity to write a short response to that. His primary argument was that the topic of “alternative medicine” was a long-running issue in a wider context, and that the exemption to the principle of balance should apply because other media have reported on the opposing side of the issue.
I strongly disagree with this argument. Although it’s true that media like the Wairarapa Times-Age do not exist in a vacuum, I don’t think this should mean that they don’t have a responsibility to provide balanced articles for their readers. The way I interpret the Press Council code, the exception can be useful when an article is part of a series of articles on the same issue, and when taken in the context of other articles in the series the overall view still maintains an appropriate balance. In the interest of balanced reporting, I believe exceptions to the principle of balance should be applied very sparingly.
The Press Council agrees with the editor that the debate over alternative remedies is sufficiently well known not to require balancing comment in every story about them. The subject falls within the exception to the principle of balance for issues of enduring public discussion.
The complainant in this case raised the important question of whether the exception can be invoked for an article in a newspaper that may not itself have covered both sides of the debate. The Council considered this point closely and came to the view that the exception has not been applied as narowly as the complainant contends and should not be. A newspaper, even if it is the sole newspaper of its locality, does not exist in a vacuum. Its readers, meeting an uncritical story on the supposed popularity of homeopathy and natural remedies, are likely to be aware the efficacy of these treatments is strongly contested by medical science.
I think this is a very worrying precedent to set. Newspapers such as the Wairarapa Times-Age can now feel justified in publishing unbalanced articles on topics such as homeopathy without feeling bound to uphold the Press Council’s principle of balance. The public have a reasonable expectation, given that the Press Council exists to uphold standards in reporting and its first principle is that articles should be accurate, fair, and balanced. While it’s a good idea to take everything you read with a grain of salt, you should be able to feel justified in expecting media reports on controversial topics to provide a balanced view. I worry that people might read articles such as this with that assumption in mind, and falsely conclude that the views omitted from the article are not merited.
I’m also rather frustrated that the Press Council concluded that anyone reading articles such as this is “likely to be aware the efficacy of these treatments is strongly contested by medical science” even though I provided data from a survey that found only 6% of respondents disagreed that “there is good scientific evidence that homeopathy works”. I understand that the survey I cited was conducted 6 years ago, but as I said in my complaint I’m unaware of anything more recent.
Although I don’t think it is, I really hope that the Press Council’s conclusion that most people are aware that homeopathy is not supported by evidence is correct. Following last year’s story about Green MP Steffan Browning backing homeopathy for ebola and March’s story about the Australian National Health and Medical Research Council concluding that homeopathy does not improve people’s health, I think there is some basis to believe that more people are familiar with the lack of evidence surrounding homeopathy than 6 years ago, but I don’t expect there would be that large a difference.
One positive thing to take away from this, at least, is that the journalist who wrote the article said in a Facebook comment that she understood the article was unbalanced and that she should have done better. I hope she’ll take this as a learning experience and, when she or other Wairarapa Times-Age reporters write on matters of “natural health” in the future, that they get in touch with the Science Media Centre to provide that much-needed balance. If we can’t rely on the Press Council to hold journalists to a high standard of balanced reporting, then we’ll have to rely on journalists’ and editors’ own standards.
EDIT 2015/04/14 10:05 am: Peter Griffin, who also complained to the Press Council about this article, has published his thoughts on the ruling as well: When balance goes out the window
EDIT 2015/04/15 2:51 pm: Andrew Bonallack, the editor of the Wairarapa Times-Age, has published his thoughts on the Press Council decision in an opinion piece for the Times-Age: Your right to choose sacrosanct
This afternoon, the Advertising Standards Authority released their decision to uphold an interesting complaint regarding advertisements for a couple of cleaning products on a website. Here is the ASA’s description of how the products were described on the website:
The Wendyl’s website (http://wendyls.co.nz/) for “100% natural cleaning and beauty products” advertised their products as having “all their ingredients listed and contain no fillers, chemicals or synthetics.”
The webpage for Wendyl’s Oxygen Bleach 1KG (sodium Percarbonate) stated, in part:
This is powdered hydrogen peroxide which is a greener alternative to chlorine bleach because it breaks down to oxygen and water in the environment.
The webpage for Wendyl-San oxygen soaker 1KG stated, in part:
I’ve spent years testing this oxygen soaker and stain remover and I’m so glad to have something which is so free of chemicals and additives. Secret ingredient is sodium percarbonate, a powdered hydrogen peroxide bleach which breaks down in the environment to oxygen and water…
The advertiser uses words such as “100% natural”, and “contains no fillers, chemicals, or synthetics”.
However, the product in question is sodium percarbonate, which is not a naturally occurring product. The main active ingredient, hydrogen peroxide, is also not a naturally occurring product and it is not stable in nature.
Both are synthetic chemicals.
After hearing from the advertiser as well, the Advertising Standards Complaints Board sided with the complainant. Here is a summary from the headnote of their decision:
The Complaints Board said it accepted the Advertiser’s view that “sodium percarbonate is a much safer and more environmnetally friendly alternative to chlorine bleach” but not that it was “chemical free” and “100% natural.” The Complaints Board said the advertisement was likely to mislead consumers into thinking the products were “100% natural” and “chemical free” when they actually contained naturally occurring chemicals, in breach of Principle 2 of the Code for Environmental Claims and had not been prepared with a due sense of social responsibility to consumers in breach of Principle 1 of the Code for Environmental Claims.
Accordingly, the Complaints Board ruled to Uphold the complaint.
The most interesting part of this complaint is, I think, who the advertiser is. As well as selling cleaning and beauty products online, Wendyl Nissen writes a weekly column for the New Zealand Herald called “Wendyl Wants To Know“. The Herald describes the column as:
Each week, Wendyl Nissen takes a packaged food item and decodes what the label tells you about its contents.
Have a look for yourself, but from the columns of hers that I’ve read it seems the main argument is typically along the lines of “natural is good, chemicals are bad”. So I find it very ironic that she’s now had a complaint upheld against her for misleadingly claiming that a product she sells is “100% natural” and “chemical free”.
For a counterexample to the attitude of “natural is good, chemicals are bad”, you need look no further than the “recipes” section of her website. There, she has some pet recipes which she makes available for free including one for De-Flea Powder for Cat Biscuits and another for Doggy De-Flea Treats. In both recipes, she claims the active ingredients are yeast and garlic:
The theory behind this powder is that fleas hate the taste of yeast and garlic so will hop off and look elsewhere.
Recently I’ve run across a couple of New Zealand companies that sell therapeutic products – one a weight loss pill, the other a jet lag drink – that seem to put marketing first and let science take the back seat. This is by no means new behaviour, but I want to use them as examples to illustrate this widespread problem and suggest what can be done to combat it.
Before promoting a therapeutic product, you should first have good reason to believe that it works. This, I hope, is common sense, but it’s also enshrined in the Fair Trading Act and the Therapeutic Products Advertising Code as they prohibit unsubstantiated claims. This means you have to test the product, and do so rigorously. Rigorous clinical trials are expensive to undertake though, so they’re quite a prohibitive first step.
Instead of jumping straight into the deep end, a useful first step can be to undertake a smaller and less rigorous (and therefore less expensive) experiment. In order to answer the question of whether or not a product actually works you need to conduct a more rigorous trial. There’s a great cost involved in doing this, but if the results of a preliminary trial are optimistic then you have reason to expect a more rigorous trial might give similar results, so the expense might be worth it. You may even be able to get some funding to help with a more rigorous trial on the basis that the preliminary results were positive.
As you can see on the trial registration page, this was an uncontrolled trial on overweight adults. The original plan was to recruit 100 volunteers with the hope that at least 60 will complete the trial. I have to say I’m a bit confused about how many people were in the trial, as apparently the recruitment was increased to 200 applicants after applications opened (a change that has not been reflected in the trial’s registration) yet apparently about 400 people applied. One article claims there were 200 participants, a later media release from Tuatara Natural Products seems to imply 100 were recruited, and the analysis of the trial says there were only 81 participants. Either way, 81 participants completed the full 8 weeks, and 52 of them took the recommended dose for the whole duration of the trial.
If there were 19 or 119 participants who didn’t complete the trial, the statistical analysis seems to ignore them with no justification given. This is unusual – a 19% drop out rate is significant and shouldn’t be swept under the rug. A lot of the time Tuatara also seems to ignore the 29 participants who didn’t drop out but also didn’t take the full dose for the whole duration.
The Science Media Centre posted the responses of 2 experts, Associate Professor Andrew Jull and Professor Thomas Lumley, to a press release from Tuatara Natural Products in February. It’s a good analysis of some of the weaknesses with the study, and I recommend you read it: Kiwi diet pill claims – experts respond
This trial was uncontrolled, and therefore also unblinded and unrandomised. As Professor Lumley explains, this is a problem if you want to draw strong conclusions from its results. It is of low methodological quality, but that’s okay. There is no problem with doing less rigorous trials first if they’re done in order to determine if more rigorous trials are necessary. Dr Glenn Vile, Chief Technical Officer of Tuatara Natural Products and the principal investigator for this study, wrote the following in a comment on a post on the “Fat Mates” trial by Dr John Pickering:
The Fat Mates trial was designed by clinical trial specialists to generate information about the Satisfax® capsules that would help Tuatara Natural Products plan a larger and longer double blind, cross over, placebo controlled trial.
We will use this information to proceed with the next clinical trial, but in the meantime we were so excited the weight loss achieved by most of our Fat Mates was much greater than the placebo effect seen in other weight loss clinical trials that we decided to launch the product so that anybody who is overweight can try Satisfax® for themselves.
I think the first part of what I’ve quoted above describes exactly what Tuatara Natural Products should be doing. They’ve conducted their low quality trial, and intend to use its results to proceed with a larger, longer, and more rigorous clinical trial. This is the right way to proceed – they now have an indication that their product might be effective, so they should do the research to find out.
The problem is that that’s not all they’re doing. After performing only a small low-quality trial, they’ve released their product for sale online and have been making a lot of noise about it. In my opinion, they’ve been significantly overstepping the results of their clinical trial. For example, in his comment Dr Vile also said:
our initial trial has shown [Satisfax] to be extremely effective in some overweight people.
Dr Glenn Vile
In their media release on the 20th of February, they reported the average weight lost only by the 52 participants who took the full dose to completion (rounded up from 2.9 kg to “close to 3kg”) but not the average weight lost by all participants. They then reported in bold that the top 26 participants lost more weight, and the top two participants lost even more weight than that!
This cherry picking of the best results appears to have been part of Tuatara Natural Products’ marketing strategy for at least a few months now. In January, Stuff published an article on the trial highlighting the single person who lost the most while participating in it: Blenheim ‘fat mate’ loses 13.5kg in 8 weeks.
That article particularly highlights the person who lost the most weight out of all those in the trial, at 13.5 kg (confusingly, the maximum weight loss reported in the analysis of the trial’s results is 13.3 kg). However, she was one of only 2 participants who lost over 10 kg, and on average the 52 participants who took the recommended dose for the full eight weeks lost 2.9 kg. Losing 13.5 kg is very far from a representative example. I’m not surprised that they didn’t choose instead to focus on the participant who gained 1.2 kg despite taking the recommended dose for the whole duration, but that is actually much closer to the mean change in weight.
The article is, for all intents and purposes, one big testimonial in favour of Satisfax. It was an article, not an advertisement, which is important because in New Zealand it’s illegal to publish any medical advertisement that:
directly or by implication claims, indicates, or suggests that a medicine of the description, or a medical device of the kind, or the method of treatment, advertised… has beneficially affected the health of a particular person or class of persons, whether named or unnamed, and whether real or fictitious, referred to in the advertisement
This effectively bans all health testimonials from advertisements. I think this is a good part of the law, as testimonials can be both very convincing and completely misleading; a quack’s dream. Banning them should force businesses to instead focus on the results of research on their products, but this hasn’t stopped Tuatara Natural Products from getting stories written about the most extreme testimonials they could find from people who have lost weight at the same time as they were taking Satisfax.
More recently, Tuatara Natural Products has put out a press release multiple times (at least on the 20th of February and again on the 4th of March) that I think rather oversteps the results of their small preliminary trial:
A NEW ZEALAND SOLUTION TO A GLOBAL PROBLEM A little pill is providing an exciting answer to one of the worlds greatest and fastest growing problems: Obesity.A NEW ZEALAND SOLUTION TO A GLOBAL PROBLEM
A little pill is providing an exciting answer to one of the world’s greatest and fastest growing problems: Obesity.
I simply don’t think they are at all justified in saying that their new product is “providing an exciting answer to… Obesity”. They are putting marketing ahead of science, and that’s not okay.
Another company that seems to put marketing before research is 1Above. They make a drink which they claim can help you recover faster from jet lag, and have recently been in the news for signing a sponsorship deal with the fantastically successful golfer Lydia Ko.
At the end of that article about their sponsorship deal the reporter, Richard Meadows, made some comments regarding the science behind jet lag relief products and asked some good questions of 1Above’s CEO, Stephen Smith (emphasis mine):
[1Above’s] product contains a mixture of vitamins B and C, electrolytes, and Pycnogenol, a pine bark extract.
The efficacy of flight drinks to combat the effects of jetlag is unproven.
Late last year pharmacists were warned after the Advertising Standards Authority upheld a complaint against an ad saying a homeopathic anti-jet lag pill really worked.
[1Above CEO Stephen] Smith said 1Above would not be doing clinical trials, which were highly expensive and not necessary.
“What we tend to use is testimonials from people who have used the product and swear by it.”
Smith said the key ingredient, Pycnogenol, had itself had been tested in dozens of trials, including its effects on reducing jetlag.
Yes, you read that correctly. The CEO of 1Above literally said that they won’t be doing clinical trials because they are “not necessary” and that they use testimonials instead.
As I said before, using testimonials to promote a therapeutic product, like a drink to help you recover faster from jet lag, can be both very convincing and completely misleading. There’s a reason why testimonials implying health benefits are illegal in New Zealand, and I hope that 1Above’s marketing will not violate this regulation.
Not all testimonials are prohibited, of course. It’s entirely acceptably to provide a testimonial from someone who thinks their drink tastes great, or that they provide great service. Basically anything for which a single person’s experience can provide a useful insight. Therapeutic effects, almost without exception, do not fall into this category, which is a big part of why we need to do clinical trials in the first place. If they quote someone in saying that their product helped them recover faster from jet lag, they may be in danger of breaching the Medicines Act.
For example, I’d expect they probably shouldn’t use a testimonial that says this:
On their website, 1Above currently does refer to research on one of the ingredients in their product, “pycnogenol”. Professor Lumley recently wrote a post about this on his other blog, Biased and Inefficient, regarding these studies and how they are used by 1Above: Clinically Proven Ingredients
I recently contacted 1Above to ask about some discrepancies I found between the abstract of the study they cited for showing pycnogenol reduced the duration of jet lag and their description of it on their website:
I was interested to see the claim your company made that Pycnogenol® has been shown to support circulation and reduce the length and severity of jet lag.
The participants in the study took 50 mg Pycnogenol 3 times per day, but I haven’t been able to find out how much is contained in your products. Is this information available anywhere on your website? I notice the study also says the participants took this regimen for 7 days, starting 2 days prior to departure. Is this comparable with how your product is intended to be used?
I also noticed some differences between the description of the study and its results between the abstract and your website, I would be grateful if you could explain to me the source of these differences.
The abstract states the control group took, on average, 39.3 hours to recover and the experimental group took, on average, 18.2 hours to recover. However your website reports these as 40 and 17 hours respectively.
Also, your website states that the study involved 133 passengers (it’s not clear from the description on your website if they all took Pycnogenol or if some of them were in the control group) who reported the time it took them to recover from jetlag. However, the study’s abstract states that in the first experiment, which is the only one that involved the reporting of the time taken to recover from jetlag, only involved 68 participants – 30 in the control group and 38 in the experimental group.
I would be grateful if you could explain these differences to me, and if you could send me any other relevant scientific information that supports this claim.
To their credit, since receiving my message they did update their website to fix the discrepancies in the reported number of participants and times taken to recover from jet lag, and their CEO replied to thank me for pointing these discrepancies out.
However, they didn’t respond to my other questions about the amount of pycnogenol in their products or the study involving the participants taking pycnogenol for 7 consecutive days, starting 2 days before their flight, which is inconsistent with how 1Above recommends their products be used.
This is just one more company basing their marketing on preliminary trials instead of using them as the basis for research that could actually answer the question of whether or not a product is useful. Worse than Tuatara Natural Products, they even go so far as to consider clinical trials “not necessary” and apparently intend to rely on testimonials instead. It would be much more appropriate for them to spend some of their $2.4 million annualised income on researching their product rather than paying for a sporting celebrity to endorse them.
I try to make my rants constructive, so I want to end this article with the question “What can we do about this?”. If you have any suggestions, I’d love to hear them in the comments section.
I think the most important thing that anyone can do to address this problem is to ask for evidence. If you see a claim made about a product that you think you might buy, then get in touch with the company selling it to let them know you’re considering buying it and to ask for evidence. If they don’t have a good enough answer, then let them know that’s why you won’t be buying their product. If they give you evidence to back up their claim, then great!
Asking for evidence doesn’t have to be a big deal, involving a formal letter or anything like that. When you see a weight loss product advertised on a one day deal site, a copper bracelet that apparently offers pain relief advertised on a store counter, or a jet lag cure promoted on Twitter, make your first response be to politely ask for evidence.
This isn’t a problem that’s going away any time soon. As consumers, we deserve to be able to make informed decisions about the products we buy, and when companies put marketing before research it becomes harder to make these informed choices. But if we work together then we can encourage companies like Tuatara Natural Products and 1Above to improve their behaviour and attitudes toward marketing and research.
Let’s turn “what’s the evidence?” into a frequently asked question for all companies that sell therapeutic products.