International funding and evaluation differences impact immunology, allergy research

Citations are often considered the gold standard for research impact, especially when it comes to evaluating funding. The number of times a scientific paper is cited in other publications demonstrates the influence the work has in the field. Coupled with the impact factor of the publishing journal for the original paper and the following citations, the result is a measure of how valuable the research is. But a new report from a research consortium in Japan that collabroated with the country’s ENGAGE Task Force, a panel of experts tasked with implementing a national research strategy for immunological and allergic diseases, has found that such indicators may not reveal the real impact of research.

In a letter to the editor published on Feb. 10 in Allergy, the researchers summarize a comparison of funded research and evaluation measures in Japan, the United Kingdom and America. The comparison was focused on understanding how short-term indicators may inhibit long-term results, how results reach the public, and how to standardize metrics for measuring success within one country and across the globe.  

“In this study, for the first time in the field of immunology and allergy, the results of research grants were compared internationally and multidimensionally, and the results of research in Japan, the United States and the United Kingdom were measured not only in terms of the quantity and quality of papers but also in terms of the ‘substantiality’, and whether they reached a wide range of people, including academia, media, patients and students,” said corresponding author Takeya Adachi, research assistant professor at Kyoto Prefectural University of Medicine and at the Keio Frontier Research & Education Collaborative Square, Keio University. Adachi chairs the ENGAGE Task Force, which stands for Empowering Next-Generation Allergist/immunologist toward Global Excellence. “We have realized that it is difficult to evaluate a research project that uniformly unites multiple types of research, and we further investigated these issues integrating novel scientific methods.”

The researchers analyzed results of allergy and immunology research funded by the Japan Agency for Medical Research and Development (AMED) in accordance with “Strategy 2030,” Japan’s official research strategy to tackle allergic and immunological diseases issued by the country’s Ministry of Health, Labour and Welfare in 2019.

“They inevitably focused on shortsighted indicators because the number of years since the start of the research itself was short,” Adachi said. “We believe it would be essential to track how short-term outcomes will be socially implemented, including further analysis of those future long-term outcomes. It is important to measure inputs/outputs/outcomes from multiple perspectives by combining different kinds of indicators.”

They compared those results to similar analyzes of allergy and immunology-related papers funded by the National Institutes of Health (NIH) in the United States and by the Medical Research Council (MRC) in the United Kingdom. They concluded that while AMED produces a high volume of research publications, the quality of publications and the rate of international collaboration are higher for NIH and MRC. They also found that while research results tend to reach the public through news media in Japan, the rate of open access to the original scientific papers is low. Both the results and rate of open access are higher in the United States and the United Kingdom, the latter of which is likely the result of MRC’s strong open research policy, according to Adachi.

The researchers also analyzed the natural language of the papers’ summaries to better understand the research topics and scope of work tackled. They found that although all funding agencies produced mainly basic allergy or immunology study papers, AMED-funded research had more clinically relevant results. They also found that projects funded in the UK, including by MRC, had more public engagement.

“To understand the performance of research funding agencies, it would be helpful to take similar programs from each and assess them by combining partial indicators from multiple angles,” Adachi said. “It is hoped that further research will add more cases and refine the methodology, leading to the accumulation of knowledge that would improve the management of research funding agencies.” 

Next, the researchers plan to examine whether research projects evaluated with diverse, long-term indicators produce substantial results, as well as study how their research impact analysis method can be used to formulate and revise the research strategies of each country and jurisdiction to produce more impactful research.

“Finally, we would like to contribute to the development of strategies unique to each country, including Japan, to ensure that publicly funded research is promoted with a long-term and international perspective and that its results are well-delivered to patients and citizens,” Adachi said.

Adachi is also affiliated with the Department of Dermatology at Tachikawa Hospital of the Federation of National Public Service Personnel Mutual Aid Associations. Other co-authors who are also members of the ENGAGE Task Force are Katsunori Masaki, Department of Pulmonary Medicine and Department of Medicine, Keio University School of Medicine; Sakura Sato, Department of Allergy, Clinical Research Center for Allergy and Rheumatology, National Hospital Organization Sagamihara National Hospital; Masaki Futamura, Division of Pediatrics, National Hospital Organization Nagoya Medical Center; Keiko Kan-o, Research Institute for Diseases of the Chest, Graduate School of Medical Sciences, Kyushu University; Yosuke Kurashima, Department of Mucosal Immunology, Graduate School of Medicine, Chiba University; Saeko Nakajima, Department of Dermatology, Kyoto University Graduate School of Medicine; Masafumi Sakashita, Department of Otorhinolaryngology Head and Neck Surgery, University of Fukui; and Hideaki Morita, Department of Allergy and Clinical Immunology and Allergy Center, National Research Institute for Child Health and Development.

Other contributors include Yasushi Ogawa, Department of Advanced Medicine and Department of Dermatology, Nagoya University; Tamami Fukushi, Aikichi Iwamoto, Mayumi Tamari and Hajime Iizuka, Japan Agency for Medical Research and Development (AMED); Kei Ito, Accenture Inc.; Amane Koizumi, National Institutes of Natural Science (NINS); Masashi Shirabe, Tokyo Institute of Technology; Masako Toriya, Global Research Institute, Keio University, Global Research Institute; Jun Hirako and Ryohei Sasano, Graduate School of Informatics, Nagoya University; Takenori Inomata, Department of Ophthalmology and Department of Digital Medicine, Juntendo University Graduate School of Medicine; Keigo Kainuma, Institute for Clinical Research, National Hospital Organization Mie National Hospital; and Sankei Nishima, National Hospital Organization Fukuoka National Hospital. Tamari is also affiliated with The Jikei University School of Medicine, Research Center for Medical Science, Division of Molecular Genetics. Hajime Iizuka is also affiliated with the Department of Dermatology, Housui General Medical Clinic.

The Scientific Research Fund of the Ministry of Health, Labour and Welfare, the Japan Society for the Promotion of Science, and the Japan Science and Technology Agency/Research Institute of Science and Technology for Society supported this research.