Register      Login
Australian Health Review Australian Health Review Society
Journal of the Australian Healthcare & Hospitals Association
Shopping Cart: ( empty )
You are here: Home > Journals > AH > AH23152
RESEARCH ARTICLE (Open Access)
Contents Vol 47(5)

Development of a novel and more holistic approach for assessing impact in health and medical research: the Research Impact Assessment Framework

Robyn L. Ward https://orcid.org/0000-0002-6877-8906 A * , Don Nutbeam B , Wilfred Mijnhardt https://orcid.org/0000-0001-9066-0798 C , Philip Nelson D , Angela Todd B , Mark I. Rees A , Janine Richards A , Nadia N. Khan E , Isaac Ho E and Sean Chung E
+ Author Affiliations
- Author Affiliations

A Faculty of Medicine and Health, University of Sydney, NSW 2006, Australia.

B Sydney Health Partners, Faculty of Medicine and Health, The University of Sydney, NSW 2006, Australia.

C Rotterdam School of Management, Erasmus University, Rotterdam, Netherlands.

D Institute of Sound and Vibration Research, University of Southampton, Southampton SO17 1BJ, UK.

E Deloitte Touche Tohmatsu Limited, Melbourne, Vic. 3000, Australia.

* Correspondence to: robyn.ward@sydney.edu.au

Australian Health Review 47(5) 589-595 https://doi.org/10.1071/AH23152
Submitted: 9 August 2023  Accepted: 25 August 2023   Published: 11 September 2023

© 2023 The Author(s) (or their employer(s)). Published by CSIRO Publishing on behalf of AHHA. This is an open access article distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND)

Abstract

Considered investment in health and medical research (HMR) is critical for fostering a healthcare system that is sustainable, effective, responsive, and innovative. While several tools exist to measure the impact of research, few assess the research environment that nurtures and supports impactful research and the strategic alignment of research with societal needs. This perspective article discusses the limitations of existing assessment tools and presents a novel Research Impact Assessment Framework designed to enable more strategic and targeted investment towards HMR, having the potential for significant public benefit.

Keywords: governance, health policy, health system, impact, investment decisions, performance and evaluation, research funding, translation, value.

References

1  Association of Australian Medical Research Institutes. Health and Medical Research in Australia. 2023. Available at https://aamri.org.au/health-medical-research/fast-facts-on-medical-research/ [cited July 2023].

2  KPMG. Economic Impact of Medical Research in Australia. 2018. Available at https://aamri.org.au/resources/reports/kpmg-medical-research-delivers-roi/ [cited July 2023].

3  Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383(9912): 156-65.
| Crossref | Google Scholar | PubMed |

4  Grant J, Green L, Mason B. Basic research and health: a reassessment of the scientific basis for the support of biomedical science. Res Eval 2003; 12(3): 217-24.
| Crossref | Google Scholar |

5  Department of Education. Australian Universities Accord Panel Discussion Paper. 2023. Available at https://www.education.gov.au/australian-universities-accord/resources/australian-universities-accord-panel-discussion-paper [cited July 2023].

6  Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst 2015; 13: 18.
| Crossref | Google Scholar | PubMed |

7  Bornmann L. Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO Rep 2012; 13(8): 673-6.
| Crossref | Google Scholar | PubMed |

8  Holbrook JB. Assessing the science–society relation: The case of the US National Science Foundation’s second merit review criterion. Technol Soc 2005; 27(4): 437-51.
| Crossref | Google Scholar |

9  Frodeman RJ, Holbrook B. Science’s Social Effects. Issues Sci Technol 2007; 23(3): 28-30.
| Google Scholar |

10  Gunn A, Mintrom M. Measuring Research Impact in Australia. Aust Univ Rev 2018; 60: 9-15.
| Google Scholar |

11  Williams K, Grant J. A comparative review of how the policy and procedures to assess research impact evolved in Australia and the UK. Res Eval 2018; 27(2): 93-105.
| Crossref | Google Scholar |

12  Research Excellence Framework. What is the REF? 2021. Available at https://www.ref.ac.uk/about-the-ref/what-is-the-ref/ [cited June 2023].

13  Sutton E. The increasing significance of impact within the Research Excellence Framework (REF). Radiography 2020; 26: S17-9.
| Crossref | Google Scholar | PubMed |

14  Department for Business, Energy and Industrial Strategy. Building on Success and Learning from Experience An Independent Review of the Research Excellence Framework. 2016. Available at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf [cited June 2023].

15  Searles A, Doran C, Attia J, Knight D, Wiggers J, Deeming S, et al. An approach to measuring and encouraging research translation and research impact. Health Res Policy Syst 2016; 14(1): 60.
| Crossref | Google Scholar | PubMed |

16  CSIRO. CSIRO Impact Evaluation Guide. 2020. Available at https://www.csiro.au/en/about/corporate-governance/ensuring-our-impact/evaluating-our-impact [cited June 2020].

17  Deeming S, Searles A, Reeves P, Nilsson M. Measuring research impact in Australia’s medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks. Health Res Policy Syst 2017; 15(1): 22.
| Crossref | Google Scholar | PubMed |

18  McInnis C, Ramsden P, Maconachie D. A handbook for executive leadership of research development. University of the Sunshine Coast; 2014.

19  Research Assessment Exercise. Framework for Research Assessment Exercise (RAE) 2020. 2020. Available at https://www.ugc.edu.hk/doc/eng/ugc/rae/2020/framework.pdf [cited June 2023].

20  Canadian Academy of Health Sciences. Making an Impact A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. 2009. Available at https://cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf [cited June 2023].

21  European Commission. Horizon Europe programme analysis Impact assessment, evaluation and monitoring of Horizon Europe. 2020. Available at https://research-and-innovation.ec.europa.eu/strategy/support-policy-making/shaping-eu-research-and-innovation-policy/evaluation-impact-assessment-and-monitoring/horizon-europe-programme-analysis_en [cited June 2023].

22  Association of Universities in the Netherlands (VSNU), the Netherlands Organisation for Scientific Research (NWO), and the Royal Netherlands Academy of Arts and Sciences (KNAW). Standard Evaluation Protocol 2015 – 2021: Protocol for Research Assessments in the Netherlands. 2016. Available at https://universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP2015-2021.pdf#:~:text=The%20Standard%20Evaluation%20Protocol%20%28SEP%29%20describes%20the%20methods,as%20well%20as%20the%20aims%20of%20such%20assessments [cited June 2023].

23  Research Australia. Australian H&MR Research Facts. 2016. Available at https://researchaustralia.org/australian-research-facts/ [cited June 2023].

24  Department of Education. Australian Universities Accord Interim Report. 2023. Available at https://www.education.gov.au/australian-universities-accord/resources/accord-interim-report#:~:text=The%20Accord%20Interim%20Report%20outlines%20a%20vision%20for,further%20discussion%20to%20inform%20the%20Review%E2%80%99s%20Final%20Report [cited August 2023].

25  Australian Research Council. Excellence in Research for Australia. 2022. Available at https://www.arc.gov.au/evaluating-research/excellence-research-australia [cited June 2023].

26  Engel-Cox JA, Van Houten B, Phelps J, Rose SW. Conceptual model of comprehensive research metrics for improved human health and environment. Environ Health Perspect 2008; 116(5): 583-92.
| Crossref | Google Scholar | PubMed |

27  Scott CS, Nagasawa PR, Abernethy NF, Ramsey BW, Martin PJ, Hacker BM, et al. Expanding Assessments of Translational Research Programs: Supplementing Metrics With Value Judgments. Eval Health Prof 2014; 37(1): 83-97.
| Crossref | Google Scholar | PubMed |

28  Wiegers SE, Houser SR, Pearson HE, Untalan A, Cheung JY, Fisher SG, et al. A Metric-Based System for Evaluating the Productivity of Preclinical Faculty at an Academic Medical Center in the Era of Clinical and Translational Science. Clin Transl Sci 2015; 8(4): 357-61.
| Crossref | Google Scholar | PubMed |

29  Schapper CC, Dwyer T, Tregear GW, Aitken M, Clay MA. Research performance evaluation: the experience of an independent medical research institute. Aust Health Rev 2012; 36(2): 218-23.
| Crossref | Google Scholar | PubMed |

30  Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ. Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med 2017; 14(8): e1002370.
| Crossref | Google Scholar | PubMed |

31  Graham KER, Langlois-Klassen D, Adam SAM, Chan L, Chorzempa HL. Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada. Front Res Metrics Anal 2018; 3: 25.
| Crossref | Google Scholar |