Skip to content

Opinion: Measure for measuring’s sake?

31 Jan, 2013

Tape measure Can we assess the impact of research by simply counting outputs? We must avoid the temptation to reduce impact to things we can measure, says Liz Allen.

Learning about what works and what doesn’t is an important part of being a science funder. Take the Wellcome Trust, for example: we have multiple grant types spread over many different funding schemes and need to make sure our money is well spent. Evaluating what we fund helps us to learn about successes and failures and to identify the impacts of our funded research.

Progress reporting while a grant is underway is a core component of the evaluation process. Funders are increasingly taking this reporting to online platforms, which come with the ability to easily quantify outputs, compare funded work and identify trends.

As useful as these systems are, they come with an inherent danger: oversimplifying research impact and simply reducing it to things we can count. At the Trust, our attitude recalls a quote attributed to Albert Einstein: “Everything that can be counted does not necessarily count and everything that counts cannot necessarily be counted.” Including qualitative descriptors of progress and impact, alongside quantitative data, is integral to our auditing and reporting.

Some quantifiable indicators, such as bibliometrics, do help to tell us whether research is moving in the right direction; the production of knowledge and its use in future research, policy and practice is important. However, it’s not about how many papers have been published or how many patents have been filed – it’s about what’s been discovered. Without narrative and context alongside ‘lists’ of outputs, how can you know whether research is making a difference?

While different funders place their emphasis on different things when deciding which research to fund, as a sector we need to be responsible and avoid the creation of perverse incentives that distort the system1. If funders send out the message that what’s important is a lot of papers or collaborations, then those seeking our funding will tell us that they’ve produced a lot of papers or collaborations.

The research community needs to be pragmatic in moving the field of impact tracking and evaluation forward. We need to develop better qualitative tools to complement more established indicators of impact – traditional bibliometric indicators, such as citations, can now be complemented with more qualitative tools such as those provided by F1000 Prime. The Trust is also exploring the value that altmetrics can bring. Other channels for the dissemination of research, such as Twitter, are becoming increasingly popular among researchers, and it’s important that we understand their role.

Most of all, we should not forget why we fund research: to make a difference. In gathering reports from those we fund, we should encourage openness and the sharing of successes and failures alongside the products and outputs of research. At the core of evaluation is learning: how and when does research lead to impact, and how we might use our funds more effectively? A research impact agenda that encourages funders to merely count is clearly at odds with that.

Liz Allen

Dr Liz Allen is Head of Evaluation at the Wellcome Trust

1. Allen L. The art of evaluating the impact of medical science. Bulletin of the World Health Organization 2010;88:4-4. doi: 10.2471/BLT.09.074823

Image Credit: GenBug on Flickr
2 Comments leave one →
  1. 4 Feb, 2013 9:26 pm

    In measuring the impact of our knowledge mobilization efforts we use a mix of numbers and narratives. Numbers count things…mostly activities or outputs. Impacts are the stories that illustrate the “so what” of those numbers. I entirely agree that without context and narratives, number tell only part of the story. My stories are great. And so are yours. And that’s where we have a challenge: how do we compare our stories? It’s easy to tell that your numbers are bigger than mine but is your story better than mine? Is mine better than yours? Or maybe its not about compare and contrast but how those stories weave together to tell a larger narrative.

Trackbacks

  1. Measure for measuring’s sake? « AMRC blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,643 other followers

%d bloggers like this: