Skip to content

Alternative impact: Can we track the impact of research outside of academia?

25 Nov, 2014

impact 2

A new type of metric purports to provide a means of measuring the ‘alternative’ impact of research among non-academic audiences such as clinicians, educators, policy makers, and the general public. Adam Dinsmore from the Wellcome Trust’s Evaluation Team explains how improvements to our understanding of the meaning and value of ‘altmetrics’ could expand the tools and techniques available to research evaluators in the research funding sector and beyond…

At any one time, the Wellcome Trust supports around 5000 active grants across biomedical research, developing medical technologies, public engagement with science, and medical humanities. An important part of the Trust’s operations is therefore to track, understand, and learn from the types of impact that this research has.

That job falls to the Trust’s Evaluation Team, who employ a wide range of tools and techniques to gather intelligence on the effects that Trust-funded research is having in academia and beyond. This can include case studies detailing the rich stories of research progression and breakthroughs associated with Wellcome Trust funding, longitudinal tracking surveys such as the Wellcome Trust Career Tracker, and the online system WT e-Val, which allows Trust-funded researchers to provide us with an on-going commentary on the work they do. All of this feeds back into the Trust’s strategic thinking.

C0021303 Tape measureIn addition, the team has long made use of quantitative indicators of scientific productivity. Conventional ‘bibliometric’ analysis can inform us about the use and re-use of the knowledge produced by research. This is typically measured by the frequency that scholarly published outputs are cited by peers in the academic literature.

These are important indicators of the production of new knowledge, but academics are not the only people who use and re-use research outputs such as articles, datasets, and software. Teachers may wish to integrate the latest research into their lesson plans. Policy makers may consult the academic literature to inform their thinking on social issues. Clinicians and patients are interested in reading about the conditions they must either treat or manage.

Metrics that allow us to gauge the ‘alternative’ impact of research away from the academic literature have conventionally been unavailable, but that may be changing.

The vast majority of research outputs are now accessed online, and so it is now possible to track the number of times an output is viewed, downloaded, or tweeted about (etc), and in some cases – by whom. Measures of this kind have come to be known as ‘altmetrics’, and are now provided in some form by most academic publishers and a number of online platforms (e.g., Plum Analytics, ImpactStory).

Some research suggests that the online interactions measured by certain altmetrics may have some value as a predictor of subsequent academic impact, but the new metrics also hold much potential to inform the Trust and other funders about the wider impact of the work they support which can sometimes be invisible to conventional citation metrics. This could be especially helpful for early-career researchers who have not yet had the chance to accrue a large number of traditional citations, researchers who work in fields with little citation culture, or researchers whose specialist roles are rarely recognised in the authorship of academic papers. Altmetrics may also be useful as a discovery tool for researchers who wish to know what is being discussed and read about most in their field.

iamThe Wellcome Trust hosted a conference on altmetrics in late September (1:AM London) and an on-going HEFCE review is exploring whether altmetrics might contribute to the next Research Excellence Framework, likely to take place in 2020.

This wide interest in altmetrics represents a great opportunity to probe questions of the meaning and validity of altmetrics as proxies of research impact, which may provide us with new tools to help make science as efficient as possible.

Read more about the Wellcome Trust’s perspective on altmetrics in an editorial published this week in PLOS Biology. To read more about the work of the Wellcome Trust’s Evaluation Team please visit the WT e-Val webpage.

One Comment leave one →
  1. 30 Nov, 2014 10:58 am

    It is important to note that short term impact is not necessarily what will produce the best science with the longest half life, how ever you measure it. Game changing science usually is the one with a long-term vision and tends to be ignored by the mainstream for a long time, hence will score low in your analytics. The true challenge is to recognise science with true FUTURE potential.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: