Skip to content

#Solo11: Post-publication peer review

13 Sep, 2011

Last weekend I attended this year’s Science Online London conference (which the Wellcome Trust sponsored). This week I’ll be reporting on a few of the key things I took from the conference:

  1. Open science
  2. Post-publication peer review
  3. ‘Microattribution’
  4. Drowning in data
  5. Open data -> sooner treatments?

2) Post-publication peer review

You may or may not remember the brouhaha last year over a NASA paper purporting to have found evidence of arsenic-based life (or so the headlines said). The paper was immediately taken to task by scientists on the web, who explained the flaws on their blogs. This led to a big debate about ‘post-publication peer review’, which, essentially, was the topic of the first panel session at #Solo11. 

Rosie Redfield, (a microbiologist at the University of British Columbia, who blogs at RRResearch) was one of the most vocal critics of the paper, and the press coverage that followed. She’s also one of the few scientists trying to replicate the paper’s findings — and doing so in an open online labbook to boot. Asked why researchers blog, she gave a multitude of reasons: it allows her to spotlight work less covered, critique relevant papers, and provide context to the public on her patch of science.

Do post-publication blog critiques discourage researchers from building on flawed research? Not really, said Redfield. They don’t read blogs because they’re not peer reviewed.

Ed Gerstner, Senior Editor at Nature Physics, said journals need to think about how they are going to deal with post-publication peer review. At the moment they only know how to say: “No. We will not engage”. He said peer review is not a binary process, even when we have an offiline conversation. But asked whether Nature would open up its (pre-publication) peer review comments to the public he said that would run the risk of losing referees.

Redfield pointed out that the lack of public discussions in journals (particularly those with comments like Nature) could also relate to career concerns: junior scientists are more open to do it but worry about the fallout later in their lives. Jonathan Eiser, an evolutionary biologist and science blogger based at UC Davis, pointed out the flaw in getting people to engage on journals: they live on Twitter and Facebook – they’re not going to go to the journals’ websites to air their gripes.

Ivan Oransky, a Reuters journalist who runs the blog RetractionWatch (does exactly what it says on the tin), spoke from a reporter’s perspective. In an age where many are calling for science journalists to cover stuff other than what comes up in the usual journal publishing cycle, he argued that they can’t do that if scientists won’t cooperate. “We can’t move science journalism upstream if scientists think the policy is ‘no comment if it’s not in peer reviewed literature’ or before publication.”

So what’s the way forward? Gerstner asked if the community gets in the habit of putting comments about papers in public (and became comfortable with the idea that such public comments are ‘official’ and ‘signed off’), would this move pave the way for open science? He pointed out that the British Medical Journal have already made strides towards this. “A culture of open debate will eventually emerge if more post-publication peer review happens. But we have to watch the danger of ‘the loudest voice wins’ factor.”

Oransky also pointed out differences between research communities: clinical science is a bigger community and the thing to remember is not everyone has an understanding of every niche paper (a point echoed by a comment from the audience: in clinical medicine it’s not the published paper that is important, but the timing. You often hear of stuff first at a conference).

In short: it’s complicated.

You can read a full, detailed Storify of the session on Nature Network’s blog. 

Want more? Full list of blog coverage of #Solo11 is available on their wiki.

Researchers: where are you having online discussions about your work? We’d like your input on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: