What difference does it make? The art of quantifying research impact

Anyone who thinks that business school academics are indifferent to sustainability should read some of their recent writings. More open to debate is how to measure the proportion and effect of their work in the field, as well as how to incentivise more such research.

In a paper in the American Economic Review, for example, Derek Lemoine from the University of Arizona and Ivan Rudik from Iowa State University make the case that the best policy for tackling global warming is to introduce carbon taxes — but with prices that increase more slowly than the exponential rises that many advocate.

In an article in the Journal of International Business Studies, Asaf Bernstein and Ryan Lewis at the University of Colorado at Boulder and Matthew Gustafson at Pennsylvania State University explore the price discounts on seafront properties at risk of flooding — with implications for evaluating different climate change policies.

These are just two examples that performed strongly in an analysis by Wilfred Mijnhardt at the Rotterdam School of Management. He uses machine learning to analyse phrases linked to the UN’s Sustainable Development Goals (SDGs) in articles by business school professors in the FT50 — a list of journals that top-ranked schools themselves consider important.

Alternative methodologies throw up different results. Across the full range of the UN’s 17 SDGs, Mijnhardt’s work (see table) assigns high scores to papers on Amazonian deforestation, the influence of corporate climate change targets on companies’ environmental performance, and the role of regulation in cutting pollution by US manufacturers.

Machine learning analysis of SDG-related content in FT50 journals 2018-21

By contrast, Corporate Knights, a Canadian publisher that produces its annual Better World MBA ranking of business schools, uses a different algorithm to identify the societal impact of academic papers, which are then weighted in part based on the number of citations they receive in other articles — a measure of quality.

It ranks top a series of papers on health, including ones on the effect of lockdowns during the pandemic, population risk factors for severe illness and death linked to Covid-19, and the most promising health technologies.

For sustainability, it highlights papers including one by Caroline Flammer at Boston University’s Questrom School of Business which shows that companies issuing green bonds report improved environmental performance and greater investment by both long-term and “green” investors.

Meanwhile, Clarivate, the analytics company that oversees the Web of Science database of academic articles, has developed its own filter for SDG content.

The company does not attribute direct weights to individual papers, but when sorting through FT50 material that links to the 13th SDG related to climate action, it highlights papers on topics including the use of ecolabels to identify a product’s environmental credentials, the importance of consumer trust in the sale of organic food, and reasons for adopting bike-sharing schemes.

Even though these different approaches all highlight a significant amount of business school research that is clearly linked to societal impact, a far greater volume of academic activity is much less directly related to such responsible business objectives.

An analysis last year by academics led by Kathleen Rodenburg at the University of Guelph concluded that only a tenth of the articles published in FT50 journals in 2019 had an explicit link to the SDGs, while 17 per cent had an implicit link.

The academics argued that the list was too reductive, that a wider range of journals and business schools should be assessed, and that academic leaders should encourage greater research on sustainability by crediting authors who have been published in a broader range of outlets.

However, they deployed yet another methodology to evaluate SDG content: enlisting undergraduate and graduate business school students to read and assess papers. The research also did not make public the details of the individual articles considered or seek to weight their relative impact.

As with the other approaches, one challenge is that the 17 SDGs are so broad that almost any academic research could be argued to be indirectly relevant. Another is that it is difficult to quantify the extent to which publications add genuinely new insights, let alone how far they can influence policy or practice to generate a positive social effect.

For those impatient to accelerate research around sustainability, time poses another difficulty in assessing relevant and high-quality academic output. It may take many years for citations by other academics to form a reliable picture of the importance of individual work.

While there is a strong argument for institutions to give credit to staff for articles that feature in a wider range of publications, there is also a need for quality control. This is traditionally provided by journal editors, peer reviewers and “impact factors” based on high average historical citations for past articles.

Analysis of the performance of individual authors or their business schools also remains difficult in the absence of more comprehensive open source data that reliably identifies authors and their institutions — such as the Orcid number, a system of unique identifiers, which launched in 2012.

It is also restrained by a longstanding focus on peer-reviewed journals as the core metric for academic recruitment and career advancement. Arguably a far greater societal impact would come from producing textbooks for students — the future managers and entrepreneurs who represent the largest single “output” from business schools.

A far greater societal impact would come from producing textbooks for students — the future managers and entrepreneurs

Equally, authors could be given credit for writing books grounded in academic research but aimed at a broader external audience of practitioners; and for still more practical activities designed to generate societally beneficial results, including speaking and serving on advisory boards to business and government.

A final concern is that any attempt to focus research on the SDGs upfront could constrain free academic activity and discourage work that may ultimately prove to be beneficial, even if there is no initial sign of its relevance.

But none of these limitations should curtail efforts to encourage and reward research that fosters insights into sustainability. They should rather be the subject of greater academic reflection to produce better outcomes.

Climate Capital

Where climate change meets business, markets and politics. Explore the FT’s coverage here.

Are you curious about the FT’s environmental sustainability commitments? Find out more about our science-based targets here