Alternative impact: Can we track the impact of research outside of academia?
A new type of metric purports to provide a means of measuring the ‘alternative’ impact of research among non-academic audiences such as clinicians, educators, policy makers, and the general public. Adam Dinsmore from the Wellcome Trust’s Evaluation Team explains how improvements to our understanding of the meaning and value of ‘altmetrics’ could expand the tools and techniques available to research evaluators in the research funding sector and beyond…
At any one time, the Wellcome Trust supports around 5000 active grants across biomedical research, developing medical technologies, public engagement with science, and medical humanities. An important part of the Trust’s operations is therefore to track, understand, and learn from the types of impact that this research has.
That job falls to the Trust’s Evaluation Team, who employ a wide range of tools and techniques to gather intelligence on the effects that Trust-funded research is having in academia and beyond. This can include case studies detailing the rich stories of research progression and breakthroughs associated with Wellcome Trust funding, longitudinal tracking surveys such as the Wellcome Trust Career Tracker, and the online system WT e-Val, which allows Trust-funded researchers to provide us with an on-going commentary on the work they do. All of this feeds back into the Trust’s strategic thinking.
In addition, the team has long made use of quantitative indicators of scientific productivity. Conventional ‘bibliometric’ analysis can inform us about the use and re-use of the knowledge produced by research. This is typically measured by the frequency that scholarly published outputs are cited by peers in the academic literature.
These are important indicators of the production of new knowledge, but academics are not the only people who use and re-use research outputs such as articles, datasets, and software. Teachers may wish to integrate the latest research into their lesson plans. Policy makers may consult the academic literature to inform their thinking on social issues. Clinicians and patients are interested in reading about the conditions they must either treat or manage.
Metrics that allow us to gauge the ‘alternative’ impact of research away from the academic literature have conventionally been unavailable, but that may be changing.
The vast majority of research outputs are now accessed online, and so it is now possible to track the number of times an output is viewed, downloaded, or tweeted about (etc), and in some cases – by whom. Measures of this kind have come to be known as ‘altmetrics’, and are now provided in some form by most academic publishers and a number of online platforms (e.g. Altmetric.com, Plum Analytics, ImpactStory).
Some research suggests that the online interactions measured by certain altmetrics may have some value as a predictor of subsequent academic impact, but the new metrics also hold much potential to inform the Trust and other funders about the wider impact of the work they support which can sometimes be invisible to conventional citation metrics. This could be especially helpful for early-career researchers who have not yet had the chance to accrue a large number of traditional citations, researchers who work in fields with little citation culture, or researchers whose specialist roles are rarely recognised in the authorship of academic papers. Altmetrics may also be useful as a discovery tool for researchers who wish to know what is being discussed and read about most in their field.
The Wellcome Trust hosted a conference on altmetrics in late September (1:AM London) and an on-going HEFCE review is exploring whether altmetrics might contribute to the next Research Excellence Framework, likely to take place in 2020.
This wide interest in altmetrics represents a great opportunity to probe questions of the meaning and validity of altmetrics as proxies of research impact, which may provide us with new tools to help make science as efficient as possible.
Read more about the Wellcome Trust’s perspective on altmetrics in an editorial published this week in PLOS Biology. To read more about the work of the Wellcome Trust’s Evaluation Team please visit the WT e-Val webpage.