Development of a novel and more holistic approach for assessing impact in health and medical research: the Research Impact Assessment Framework

Robyn L. Ward; Don Nutbeam; Wilfred Mijnhardt; Philip Nelson; Angela Todd; Mark I. Rees; Janine Richards; Nadia N. Khan; Isaac Ho; Sean Chung

doi:10.1071/AH23152

RESEARCH ARTICLE (Open Access)

Previous Next Contents Vol 47(5)

Development of a novel and more holistic approach for assessing impact in health and medical research: the Research Impact Assessment Framework

Robyn L. Ward

^A ^* , Don Nutbeam ^B , Wilfred Mijnhardt

^C , Philip Nelson ^D , Angela Todd ^B , Mark I. Rees ^A , Janine Richards ^A , Nadia N. Khan ^E , Isaac Ho ^E and Sean Chung ^E

+ Author Affiliations

- Author Affiliations

^A Faculty of Medicine and Health, University of Sydney, NSW 2006, Australia.

^B Sydney Health Partners, Faculty of Medicine and Health, The University of Sydney, NSW 2006, Australia.

^C Rotterdam School of Management, Erasmus University, Rotterdam, Netherlands.

^D Institute of Sound and Vibration Research, University of Southampton, Southampton SO17 1BJ, UK.

^E Deloitte Touche Tohmatsu Limited, Melbourne, Vic. 3000, Australia.

^* Correspondence to: robyn.ward@sydney.edu.au

Australian Health Review 47(5) 589-595 https://doi.org/10.1071/AH23152
Submitted: 9 August 2023 Accepted: 25 August 2023 Published: 11 September 2023

© 2023 The Author(s) (or their employer(s)). Published by CSIRO Publishing on behalf of AHHA. This is an open access article distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND)

Abstract

Considered investment in health and medical research (HMR) is critical for fostering a healthcare system that is sustainable, effective, responsive, and innovative. While several tools exist to measure the impact of research, few assess the research environment that nurtures and supports impactful research and the strategic alignment of research with societal needs. This perspective article discusses the limitations of existing assessment tools and presents a novel Research Impact Assessment Framework designed to enable more strategic and targeted investment towards HMR, having the potential for significant public benefit.

Keywords: governance, health policy, health system, impact, investment decisions, performance and evaluation, research funding, translation, value.

Overview of health and medical research

Australia has an internationally competitive health and medical research (HMR) ecosystem, generating transformative discoveries including the artificial heart valve, in vitro fertilisation, and the human papillomavirus vaccine.¹ This research generates significant socioeconomic benefit, with a recent evaluation highlighting that every A$1 spent on HMR more than triples the return in health benefits.² While research is critical for driving improved health outcomes and economic benefit, not all research generates this impact. Basic research is considered globally as the principal beneficiary of HMR investment,³ however only 2–21% of basic research is associated with clinical advances.⁴ Australia allocates much of its expenditure towards basic research, but scores poorly in international innovation rankings.⁵

Research impact assessment

Research impact assessments are critical in promoting the conduct of impactful and value-based research. Traditional methods focus on productivity metrics, such as bibliometrics and funding track record, with broader health and socioeconomic impacts generally not considered.⁶ These traditional metrics are increasingly recognised by government and policymakers as insufficient in assessing ‘real world’ research value.⁷^–⁹

There has been considerable investment in devising methods for the wider assessment of research impact. Originating in Australia in the early 2000s,¹⁰ these methods were refined in the United Kingdom through the Research Excellence Framework (REF),¹¹ a performance-based research funding system that evaluates the non-academic impacts of research¹²^,¹³ and incentivises universities to promote the impacts. Criticisms of the REF have included its applicability only to universities and its operational cost.¹⁴

Subsequently, several newer frameworks have sought to measure the socioeconomic impacts of research, including notable Australian examples¹⁵^–¹⁷ (Table 1). However these have not typically assessed key attributes such as research environment or strategic alignment to the priorities of funders and beneficiaries. Culture and support for researchers are considered the most influential environmental predictors of research productivity,¹⁸ yet only a few of the identified frameworks assess the research environment.¹²^,¹⁹ Medical research should ultimately contribute towards developing purposeful, real-world solutions aligned to societal needs, yet less than half of the identified frameworks explicitly assess this alignment.¹⁵^,¹⁶^,²⁰^–²² In Australia in the financial year 2022, the Federal, State, and Territory health departments invested billions of dollars into HMR,²³ highlighting the importance of the strategic alignment of research. Additionally, existing frameworks often fail to assess research in non-academic settings,¹²^,¹⁹^,²² discounting the impacts of HMR conducted in the not-for-profit and private sectors.²³

Table 1.Summary of health-related research impact assessment frameworks^A used to guide decision making.

Framework (year published)	Country of origin	Foundational frameworks	Description	Impact assessment categories	Indicators	Target group	Alignment to national, state or community health priorities
Canadian Academy of Health Sciences Impact Framework (2009)²⁰	Canada	Payback, program logic and Canadian Institutes of Health Research	A logic model that measures return on investment in health research. It is not a rating or scoring system and must be adapted based on user’s needs.	Advancing knowledge Building capacity Informing decision making Health impact Broad socioeconomic impacts	Menu of 66 qualitative and quantitative indicators/metrics	Range of funder and research types	Yes (considers health status and function, well-being and economic conditions as a precursor of logic model)
Comprehensive Research Metrics Logic Model for National Institute of Health Environmental Health Group (2008)²⁶	USA	Logic model	A logic model that defines the relationship between environmental health research grant programs and outcomes related to health, society, environment, economics and quality of life. It is not a rating or scoring system.	Resources (inputs) Activities Outputs Outcomes (short, medium and long-term) Contextual factors (environment) Reservoir of knowledge	Various metrics including grants, patents, investigator professional development, community outreach, policy changes, and societal changes	National Institute of Environmental Health Sciences, other government (federal, state, and local) agencies, grantee institutions, business and industry, and community partners	No
CSIRO Impact Evaluation Framework (2020)¹⁶	Australia	Logic model	Aims to inform allocation, analysis, advocacy, and accountability for work undertaken by CSIRO.	Economic impact Environmental impact Social Impact	Cost-benefit/statistical metrics, bibliometrics, qualitative	CSIRO (primarily) and other Australian publicly funded research organisations	Yes (address local, regional, national, or global need)
Excellence in Research for Australia (ERA) (2010)²⁵	Australia	n/a	ERA is the Australian Government’s national research evaluation framework which assesses quality of research, using a 5-point rating system, conducted in Australian universities.	Research quality Research activity Research application	Bibliometrics and case studies	Australian higher education institutions	No
Horizon Europe Key Impact Pathways (2021)²¹	European Union (EU)	n/a	Horizon Europe is the EU’s framework program for research and innovation. The Impact Pathways outline the societal/economic effects and benefits of the program or European science.	Scientific impact Societal impact Towards technological/economic impact	Metrics such as bibliometrics, human capital, collaborations, policy, economic outcomes	Horizon Europe funded research programs	Yes (addressing EU policy priorities and global challenges through research and innovation)
Hunter Medical Research Institute Framework to Assess the Impact from Translational health research (FAIT) (2016)¹⁵	Australia	Modified payback, economic analysis, narratives and program logic model	FAIT encourages and measures research translation and research impact and uses a scorecard approach to reporting outcomes and impact.	Advance knowledge Clinical implementation Community benefit Legislation and policy Economic impact Social return on investment	Benefits (including publications, health outcomes, policy, and societal impact), economic outcomes, case studies	Health-related research programs	Yes (community needs that are being addressed by research is included in case study)
Institute for Translational Health Sciences (ITHS) Kellogg Logic Model – World Health Organization (WHO) Health Services Assessment Model²⁷	USA	Logic model	The ITHS framework is a mix of the Kellog Logic Model and the WHO’s Health Services Assessment Model. It is designed to assess the value added to translational research that is funded by the Clinical and Translational Science Award (CTSA) program.	Relevance Process Impact Equity Sustainability Adequacy Efficiency Effectiveness	Metrics and value-based indicators	CTSA sites, University of Washington	No
The Matrix (2004)²⁸	USA	n/a	The Matrix is a score-based framework which assesses the performance of individual faculty members across the full spectrum of non-clinical engagement.	Research Education Scholarship Administration or services	Metrics such as, grants, patents, teaching hours/student supervision, and bibliometrics	Non-clinical faculties (e.g. teaching, basic science, translation, clinical research) at the Temple University School of Medicine	No
Research Assessment Exercise (RAE) (1993)¹⁹	Hong Kong	n/a	The RAE aims to increase accountability of researchers, guide future research funding decisions, and provide input into the research environment and translation process.	Research outputs Impact Environment	Research outputs, impact overview statements, and impact case studies	Hong Kong University Grants Committee-funded universities	No
Research Excellence Framework (REF) (2014)¹²	UK	n/a	The REF assesses the quality of research in UK higher education institutions. It aims to increase accountability of researchers (and funding allocation) as well as provide input into the research environment and translation process (through benchmarking).	Quality of outputs Impact beyond academia Environment that supports research	Bibliometrics, impact statement, case studies	UK higher education institutions	No
Research Performance Evaluation Framework (RPEF) (2012)²⁹	Australia	n/a	The RPEF is a score-based framework which assesses research performance specifically for a medical research institute and uses outcomes to inform strategic goals and internal funding decisions.	Knowledge creation Inputs to research Commercial, clinical, and public health outcomes	Publications, technical papers, grants, students, outcomes	Murdoch Children’s Research Institute	No
Standard Evaluation Protocol (2015)²²	Netherlands	n/a	A scoring system that assesses the quality and the relevance of research to society and informs opportunities for improvement.	Research quality Relevance to society Viability PhD programs Research integrity Diversity	Self-assessment by research unit (including metrics and narrative around finances, bibliometrics, grants, public engagements, patents, prizes) and site visit	Dutch universities and Dutch Scientific Research Institutes (NWO) and Academy institutes	Yes (relevance to society)

^{CSIRO, Commonwealth Scientific and Industrial Research Organisation; USA, United States of America; UK, United Kingdom.}

^A Note: this is not an exhaustive list of all research impact assessment frameworks but rather a selection of frameworks that are currently widely adopted and operationalised across their respective jurisdictions. Theoretical or conceptual frameworks such as the payback model, the balance score card, and Lean and Six Sigma Techniques upon which more recent operationalised frameworks are based are not listed. Discontinued frameworks such as The Program Assessment Rating Tool are also not listed; adapted from Cruz Rivera et al.,³⁰ Deeming et al.,¹⁷ and Graham et al.³¹

Acknowledging the limitations of existing frameworks, we designed a novel Research Impact Assessment Framework (RIAF) that assesses the health and societal impacts of HMR, the research environment, and the strategic alignment of research with societal needs. The RIAF was developed through an iterative process involving extensive literature review and consultations with global experts, the details of which will be described in a future publication. Ultimately it is intended that the RIAF will enable research funders to support more strategic and targeted investment towards HMR with greater public benefit.

Components of the RIAF

The RIAF introduces a more comprehensive approach to assess the research impact of an organisation, based on two assessment domains; Research Environment (RE) and Alignment and Influence of Research (AIR) (Fig. 1).

Fig. 1.

Overview of the novel RIAF.

Research Environment domain

The RE domain of the RIAF encompasses two sub-domains: Capability and Translation. The Capability sub-domain intends to evaluate an organisation’s ability to cultivate an environment that facilities high-quality and responsible research, and to foster the development of skills and expertise among its researchers to deliver impactful research. The Translation sub-domain assesses the extent to which an organisation’s research environment supports the translation of research findings into tangible impacts, recognising that translation and implementation activity does not come naturally to many researchers, and requires resourcing and partnerships with end users.

By considering the RE as a critical assessment domain, the RIAF emphasises the importance of creating a supportive and responsible ecosystem that fosters high-quality research, innovation, and collaboration. It encourages organisations to invest in the infrastructure, resources, and collaboration mechanisms needed to translate research outputs into impactful outcomes. This goal is consistent with the recent Australian Universities Accord Interim Report highlighting the need for research to deliver widespread impact through translation and use.²⁴

Alignment and Influence of Research domain

The AIR domain goes beyond the traditional focus on academic outputs to encompass the broader societal impacts of research. It is designed to assist health ministries and organisations in aligning their investment in research with health services priorities and community expectations. This domain comprises two sub-domains: Alignment to Priorities and Influence of Research. The Alignment to Priorities sub-domain intends to evaluate the extent to which the research is relevant and congruent with end-user needs. The Influence of Research sub-domain seeks to measure how well organisational research programs impact on policy and practice, health and wellbeing, the economy, sustainability, and the organisation’s reputation and brand.

In the RIAF, assessment processes extend beyond academic audiences, and encompass the health, not-for-profit, and private sectors. This cross-sectoral evaluation of research initiatives provides a holistic view of the HMR ecosystem and creates avenues for non-academic institutions to access funding and resources.

Assessment tools

We are currently pilot-testing two streamlined data collection tools to reduce the administrative burden associated with traditional research impact assessments:¹²^,²⁰ an online survey completed by a sample of research active staff to assess the RE domain, and an impact case study template to assess the AIR domain. Emerging technologies, including intelligent document processing and machine learning, will be trialled to automate data collection.

Unlike traditional ranking-based approaches to funding,¹²^,¹⁹^,²²^,²⁵ the RIAF provides funding bodies with information on capability and/or capacity gaps that could be addressed to promote responsible impact generation, fostering a more sustainable and equitable HMR ecosystem.

Conclusion

The RIAF is undergoing pilot-testing to evaluate its acceptability, feasibility, and effectiveness. The results of the pilot will be used to further refine the RIAF. It is intended that by adopting the RIAF, organisations and funders may comprehensively evaluate their research initiatives, fostering research that has the potential to generate meaningful societal impact. Although the RIAF is currently focused on the assessment of impact in HMR, many of the concepts and methods may be translated to the assessment of other disciplines.

Data availability

The data that support this study will be shared upon reasonable request to the corresponding author.

Conflicts of interest

Nadia N. Khan, Isaac Ho, Joanne Egan, and Sean Chung are employed by Deloitte Touche Tohmatsu Limited and completed this study as part of contractual work funded by NSW Health. Wilfred Mijnhardt and Philip Nelson are contractually affiliated with the University of Sydney as members of the International Advisory Group. NSW Health did not control or influence the decision to submit the final manuscript for publication.

Declaration of funding

This study was supported by funding from NSW Health and in-kind support from Deloitte Touche Tohmatsu Limited.

Acknowledgements

The authors gratefully acknowledge the assistance of Ms Mona Shamshiri for her administrative support and Ms Joanne Egan and Ms Cassie-lee Osborne for contributing to the literature review and data analysis required for the development of the Research Impact Assessment Framework.

References

1 Association of Australian Medical Research Institutes. Health and Medical Research in Australia. 2023. Available at https://aamri.org.au/health-medical-research/fast-facts-on-medical-research/ [cited July 2023].

2 KPMG. Economic Impact of Medical Research in Australia. 2018. Available at https://aamri.org.au/resources/reports/kpmg-medical-research-delivers-roi/ [cited July 2023].

3 Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383(9912): 156-65.
| Crossref | Google Scholar | PubMed |

4 Grant J, Green L, Mason B. Basic research and health: a reassessment of the scientific basis for the support of biomedical science. Res Eval 2003; 12(3): 217-24.
| Crossref | Google Scholar |

5 Department of Education. Australian Universities Accord Panel Discussion Paper. 2023. Available at https://www.education.gov.au/australian-universities-accord/resources/australian-universities-accord-panel-discussion-paper [cited July 2023].

6 Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst 2015; 13: 18.
| Crossref | Google Scholar | PubMed |

7 Bornmann L. Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO Rep 2012; 13(8): 673-6.
| Crossref | Google Scholar | PubMed |

8 Holbrook JB. Assessing the science–society relation: The case of the US National Science Foundation’s second merit review criterion. Technol Soc 2005; 27(4): 437-51.
| Crossref | Google Scholar |

9 Frodeman RJ, Holbrook B. Science’s Social Effects. Issues Sci Technol 2007; 23(3): 28-30.
| Google Scholar |

10 Gunn A, Mintrom M. Measuring Research Impact in Australia. Aust Univ Rev 2018; 60: 9-15.
| Google Scholar |

11 Williams K, Grant J. A comparative review of how the policy and procedures to assess research impact evolved in Australia and the UK. Res Eval 2018; 27(2): 93-105.
| Crossref | Google Scholar |

12 Research Excellence Framework. What is the REF? 2021. Available at https://www.ref.ac.uk/about-the-ref/what-is-the-ref/ [cited June 2023].

13 Sutton E. The increasing significance of impact within the Research Excellence Framework (REF). Radiography 2020; 26: S17-9.
| Crossref | Google Scholar | PubMed |

14 Department for Business, Energy and Industrial Strategy. Building on Success and Learning from Experience An Independent Review of the Research Excellence Framework. 2016. Available at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf [cited June 2023].

15 Searles A, Doran C, Attia J, Knight D, Wiggers J, Deeming S, et al. An approach to measuring and encouraging research translation and research impact. Health Res Policy Syst 2016; 14(1): 60.
| Crossref | Google Scholar | PubMed |

16 CSIRO. CSIRO Impact Evaluation Guide. 2020. Available at https://www.csiro.au/en/about/corporate-governance/ensuring-our-impact/evaluating-our-impact [cited June 2020].

17 Deeming S, Searles A, Reeves P, Nilsson M. Measuring research impact in Australia’s medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks. Health Res Policy Syst 2017; 15(1): 22.
| Crossref | Google Scholar | PubMed |

18 McInnis C, Ramsden P, Maconachie D. A handbook for executive leadership of research development. University of the Sunshine Coast; 2014.

19 Research Assessment Exercise. Framework for Research Assessment Exercise (RAE) 2020. 2020. Available at https://www.ugc.edu.hk/doc/eng/ugc/rae/2020/framework.pdf [cited June 2023].

20 Canadian Academy of Health Sciences. Making an Impact A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. 2009. Available at https://cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf [cited June 2023].

21 European Commission. Horizon Europe programme analysis Impact assessment, evaluation and monitoring of Horizon Europe. 2020. Available at https://research-and-innovation.ec.europa.eu/strategy/support-policy-making/shaping-eu-research-and-innovation-policy/evaluation-impact-assessment-and-monitoring/horizon-europe-programme-analysis_en [cited June 2023].

22 Association of Universities in the Netherlands (VSNU), the Netherlands Organisation for Scientific Research (NWO), and the Royal Netherlands Academy of Arts and Sciences (KNAW). Standard Evaluation Protocol 2015 – 2021: Protocol for Research Assessments in the Netherlands. 2016. Available at https://universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP2015-2021.pdf#:~:text=The%20Standard%20Evaluation%20Protocol%20%28SEP%29%20describes%20the%20methods,as%20well%20as%20the%20aims%20of%20such%20assessments [cited June 2023].

23 Research Australia. Australian H&MR Research Facts. 2016. Available at https://researchaustralia.org/australian-research-facts/ [cited June 2023].

24 Department of Education. Australian Universities Accord Interim Report. 2023. Available at https://www.education.gov.au/australian-universities-accord/resources/accord-interim-report#:~:text=The%20Accord%20Interim%20Report%20outlines%20a%20vision%20for,further%20discussion%20to%20inform%20the%20Review%E2%80%99s%20Final%20Report [cited August 2023].

25 Australian Research Council. Excellence in Research for Australia. 2022. Available at https://www.arc.gov.au/evaluating-research/excellence-research-australia [cited June 2023].

26 Engel-Cox JA, Van Houten B, Phelps J, Rose SW. Conceptual model of comprehensive research metrics for improved human health and environment. Environ Health Perspect 2008; 116(5): 583-92.
| Crossref | Google Scholar | PubMed |

27 Scott CS, Nagasawa PR, Abernethy NF, Ramsey BW, Martin PJ, Hacker BM, et al. Expanding Assessments of Translational Research Programs: Supplementing Metrics With Value Judgments. Eval Health Prof 2014; 37(1): 83-97.
| Crossref | Google Scholar | PubMed |

28 Wiegers SE, Houser SR, Pearson HE, Untalan A, Cheung JY, Fisher SG, et al. A Metric-Based System for Evaluating the Productivity of Preclinical Faculty at an Academic Medical Center in the Era of Clinical and Translational Science. Clin Transl Sci 2015; 8(4): 357-61.
| Crossref | Google Scholar | PubMed |

29 Schapper CC, Dwyer T, Tregear GW, Aitken M, Clay MA. Research performance evaluation: the experience of an independent medical research institute. Aust Health Rev 2012; 36(2): 218-23.
| Crossref | Google Scholar | PubMed |

30 Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ. Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med 2017; 14(8): e1002370.
| Crossref | Google Scholar | PubMed |

31 Graham KER, Langlois-Klassen D, Adam SAM, Chan L, Chorzempa HL. Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada. Front Res Metrics Anal 2018; 3: 25.
| Crossref | Google Scholar |