A quantitative analysis of 16 million articles related to the Sustainable Development Goals (SDGs) revealed only a modest relationship between conventional indicators of research quality and citation in policy documents as indicators for article influence in formulating policies.
These findings “challenge the assumption that high-quality research, as traditionally measured, directly informs policy”, according to the study published in Sustainable Development on 1 October.
Authored by Basil Mahfouz, Licia Capra and Geoff Mulgan at University College London, UK, the study suggested “a need for both researchers and policy-makers to broaden their strategies for promoting and sourcing research”.
The study said: “Researchers should recognise that academic rigour alone may not ensure policy impact, while policy-makers should improve their methods for identifying and utilising robust research.”
90% of SDG papers are never cited in policy
The study used over 16 million SDG-related research articles in Scopus and over 10 million policy documents in the Overton database to examine the relationship between research quality indicators and policy citations at scale.
After filtering to include only SDG-related articles, the final corpus comprised 16 million scholarly articles, of which 1.5 million were cited in policy documents. The study found that the distribution of the policy citations varied significantly across papers, authors, SDGs and geographical locations, “indicating a highly skewed structure”.
The study also found “a significant skew in the number of policy citations per article” as about 90% of SDG papers are never cited in policy documents, and among those that are cited, nearly half receive only one citation.
A small fraction (7%) of the SDG papers accounts for half of all policy citations, indicating “a concentration of influence within a small subset of research,” the study said.
Variations between SDGs
It also found a large disparity in policy citation ratios, or the proportion of papers cited in policy documents relative to the total number of papers published within each SDG category, suggesting that certain SDGs are better represented in scholarly publications and have a higher influence on policymaking than others.
For example, articles related to SDG 3 (Good health and well-being) made up nearly 40% of all articles and 30% of those cited in policy documents.
While SDG 1 (No poverty) and SDG 10 (Reduced inequalities) had citation ratios exceeding 25%, suggesting that a significant portion of the research in these areas is utilised in policymaking, SDG 7 (Affordable and clean energy) had a citation ratio below 5%.
“The disparity in citation ratios among SDGs could be attributed to the different fields of research they represent,” the authors noted.
Referencing another published study which indicated that policy-makers are more likely to cite research from the social sciences and humanities than from the physical sciences, the authors noted: “Nearly half of the research in SDG 1 (No poverty) is published in sociology and economics journals, while half the research under SDG 7 (Affordable and clean energy) comes from journals in engineering, energy, chemistry and chemical engineering, which are more technical subjects with less direct relevance to policy-makers.”
A modest influence
Mahfouz et al wrote: “While papers with higher scores on established research quality metrics tend to be cited more frequently in policy documents, the extent of their influence remains modest.”
Depending on the SDG, the bibliometric indicators can explain between 3% and 11% of the variation in whether a paper is cited in policy, suggesting that “factors beyond research quality likely play a more significant role in influencing policy citations”.
They wrote: “This indicates that either non-academic factors play a significant role in determining which research is cited in policy documents, or that the traditional bibliometric indicators do not effectively capture research excellence from a policy perspective” and “challenges the prevailing belief that policy-makers utilise high-quality research”.
Although the relationship between bibliometric indicators of research quality and the likelihood of being cited in policy documents was modest for all SDGs, the study did observe some notable differences between SDGs.
“The variation suggests that the relevance of bibliometric indicators of quality varies across SDGs,” it said.
In the case of SDGs 1 (No poverty) and 16 (Peace, justice and strong institutions), the model efficacy, calculated as the proportion of recall to citation ratio, achieved an efficacy rate of 0.77 and 0.68, suggesting a stronger correlation between metrics and policy citations than for SDGs 6 (Clean water and sanitation) and 7 (Affordable and clean energy) with efficacy rates of 0.24 and 0.23, respectively.
By calculating the number of papers published and cited for each country by determining the locations of each co-author based on their institutional affiliation, the study found that a small number of countries lead in terms of both SDG publications and policy citations.
In particular, the United States stands out, contributing to 20% of all publications and 30% of all policy citations – more than the next six countries combined. Also, only 15 countries account for approximately 75% of both the total number of publications and the total number of papers cited in policy.
However, even among these countries, there is a significant disparity, with India, Japan and China collectively accounting for over 21% of the total publications, but only 6.5% of the papers cited in policy.
Unsurprising findings
Professor Ellen Hazelkorn, joint managing partner at BH Associates education consultants, told University World News the article confirmed “what many of us know and experience from working with policy-makers”.
“Academics and policy-makers are looking at issues in different ways,” said Hazelkorn, who is also a professor emerita at Technological University Dublin in Ireland. “Policy-makers are looking for intelligible and translatable papers which can speak to the policy implications of the evidence,” Hazelkorn said.
“But academics are incentivised by a reward system which does not value policy-oriented work.
“Academics often have a poor understanding of how government works and [how to provide] solutions to problems rather than simply being a critic,” she said.
Hazelkorn said governments expect research to provide leaders and policy-makers with the evidence they need to help solve the challenges of the SDGs.
“There has never been a more important time to analyse, design, evaluate, critique, integrate, compare and innovate,” she noted.
Angel Calderon, director of strategic insights at RMIT University in Australia, agreed that the findings were not surprising.
“I am not surprised by the findings of this study. In my research, I have also found a similar pattern and have wondered why this is so,” Calderon told University World News.
“We might have overestimated usage of scholarly outputs in policy documents,” Calderon said, adding it was worth mentioning that the volume of policy documents produced in the world is small compared to the volume of scholarly outputs produced. “So there is little scope for research being cited,” Calderon added.
There are also differences in utilisation between and across countries, world regions and countries as well types of policy think tanks, Calderon noted.
Non-academic factors
“I would also say that among non-academic factors influencing what research is cited is the role of the peer networks of those who write policy papers, and access to published research,” said Calderon.
“In one report which we hope we will get to publish we found that research produced using frontier technologies such as synchrotrons was not aligned to the SDGs.
“In part it’s because it is groundbreaking research but also many researchers are not aligning their research to the SDGs, or simply do not know about SDG terminology,” Calderon said.
“I think more work needs to be done to make good use of SDGs keywords and terminology. It continues to evolve and we need to raise awareness about it among academic and policy experts,” Calderon noted.
Richard Holmes, an independent writer and consultant and the producer of University Ranking Watch blog, told University World News the factors that influence government policy are “so varied” that it is unsurprising that there is not a great correlation with standard measures of research quality.
“It is a plus that there is a modest correlation with the impact of SDGs-related research quality on policy,” Holmes said.
“There should be some discussion about how the research part of the Times Higher Education Impact Rankings, which measure global universities’ success in delivering SDGs, for example, should be assessed.
“Perhaps the quality of research can be assessed by a broader range of criteria, including mentions in the popular media or in social media and by the diversity of responses,” Holmes pointed out.
Curled: University World News
