#Solo11: Post-publication peer review
- Open science
- Post-publication peer review
- Drowning in data
- Open data -> sooner treatments?
2) Post-publication peer review
You may or may not remember the brouhaha last year over a NASA paper purporting to have found evidence of arsenic-based life (or so the headlines said). The paper was immediately taken to task by scientists on the web, who explained the flaws on their blogs. This led to a big debate about ‘post-publication peer review’, which, essentially, was the topic of the first panel session at #Solo11.
Rosie Redfield, (a microbiologist at the University of British Columbia, who blogs at RRResearch) was one of the most vocal critics of the paper, and the press coverage that followed. She’s also one of the few scientists trying to replicate the paper’s findings — and doing so in an open online labbook to boot. Asked why researchers blog, she gave a multitude of reasons: it allows her to spotlight work less covered, critique relevant papers, and provide context to the public on her patch of science.
Do post-publication blog critiques discourage researchers from building on flawed research? Not really, said Redfield. They don’t read blogs because they’re not peer reviewed.
Ed Gerstner, Senior Editor at Nature Physics, said journals need to think about how they are going to deal with post-publication peer review. At the moment they only know how to say: “No. We will not engage”. He said peer review is not a binary process, even when we have an offiline conversation. But asked whether Nature would open up its (pre-publication) peer review comments to the public he said that would run the risk of losing referees.
Redfield pointed out that the lack of public discussions in journals (particularly those with comments like Nature) could also relate to career concerns: junior scientists are more open to do it but worry about the fallout later in their lives. Jonathan Eiser, an evolutionary biologist and science blogger based at UC Davis, pointed out the flaw in getting people to engage on journals: they live on Twitter and Facebook – they’re not going to go to the journals’ websites to air their gripes.
Ivan Oransky, a Reuters journalist who runs the blog RetractionWatch (does exactly what it says on the tin), spoke from a reporter’s perspective. In an age where many are calling for science journalists to cover stuff other than what comes up in the usual journal publishing cycle, he argued that they can’t do that if scientists won’t cooperate. “We can’t move science journalism upstream if scientists think the policy is ‘no comment if it’s not in peer reviewed literature’ or before publication.”
So what’s the way forward? Gerstner asked if the community gets in the habit of putting comments about papers in public (and became comfortable with the idea that such public comments are ‘official’ and ‘signed off’), would this move pave the way for open science? He pointed out that the British Medical Journal have already made strides towards this. “A culture of open debate will eventually emerge if more post-publication peer review happens. But we have to watch the danger of ‘the loudest voice wins’ factor.”
Oransky also pointed out differences between research communities: clinical science is a bigger community and the thing to remember is not everyone has an understanding of every niche paper (a point echoed by a comment from the audience: in clinical medicine it’s not the published paper that is important, but the timing. You often hear of stuff first at a conference).
In short: it’s complicated.
Want more? Full list of blog coverage of #Solo11 is available on their wiki.