Australian Health Review Australian Health Review Society
Journal of the Australian Healthcare & Hospitals Association
RESEARCH ARTICLE (Open Access)

Health policy analysis: a tool to evaluate in policy documents the alignment between policy statements and intended outcomes

K. Katherine Cheung A , Masoud Mirzaei B D and Stephen Leeder C
+ Author Affiliations
- Author Affiliations

A Faculty of Medicine, The University of Sydney, Sydney, NSW 2006, Australia.

B Yazd Cardiovascular Research Centre, Shahid Sadoughi University, Yazd, 89151-73143 IR, Iran.

C Menzies Centre for Health Policy, The University of Sydney, NSW 2006, Australia.

D Corresponding author. Email: masoud.mirzaei@sydney.edu.au

Australian Health Review 34(4) 405-413 https://doi.org/10.1071/AH09767
Submitted: 31 March 2009  Accepted: 15 February 2010   Published: 25 November 2010

Abstract

Objective. Health policy analysis remains surprisingly undeveloped in Australia given the power that policy exercises over the direction of public health. This paper describes the use of a policy analysis tool to evaluate the alignment between policy statements and intended outcomes of principal chronic illness policy documents in New South Wales (NSW) from 1999 to 2008. In doing so, it demonstrates the utility of a set of predefined criteria for use in retrospective policy analysis and potential for use in reviewing policy proposals and making future health policies.

Methods. We analysed the major health policy for the care of people with chronic disease in NSW, the Chronic Care Program, using a modified set of existing criteria derived from the logic of events theoretical framework, which conceptualises the connection between policy determinants and outcomes. A document map was also developed to identify linkages between the policy documents analysed.

Results. Internal validity, the alignment between policy statements and intended outcomes, was highest for policy background and goal-setting criteria, and lowest for accessibility, resources, public opportunities and monitoring and evaluation criteria. The use of document mapping was vital in determining linkages between the closely related policy documents of this complex initiative.

Conclusions. The use of predefined criteria to identify in policy documents where policy statements are not consistent with intended outcomes, in conjunction with policy mapping, are useful methods of analysing complex policy initiatives. In the Australian context, the use of a validated policy-analysis tool might help achieve greater consistency.

Implications. The use of a tool during policy development to identify in policy documents where statements are not consistent with intended outcomes may increase the likelihood of the successful implementation of future health policy. The tool can also assist those who make and review future policies.

What is known about the topic? Chronic diseases are an increasing burden on the Australian community and effective policy is required for their prevention and management. Evidence-based policy making has much potential in effecting policy impact yet there is very little Australian research into policy making. Health policy analysis has been conducted in the past but there has not been an attempt to evaluate or analyse the documents that communicate policy in NSW.

What does this paper add? This is the first attempt to analyse the policy documentation of a major health initiative for internal validity, that is, the alignment between policy statements and intended outcomes. It offers a framework that may be used to assess policy documents and demonstrates the utility of document maps to identify policy linkages.

What are the implications for practitioners? The use of a predefined set of criteria highlights opportunities where policies such as the NSW Chronic Care Program can potentially be strengthened. The criteria can be used by reviewers of policy proposals to find where policy documents can be improved to better reflect their intention. This may increase the chance of successful implementation. Document maps may clarify the relationships between policy documents in policy-rich programs and improve their accessibility to target audiences.

Additional keywords: chronic disease, criteria, internal validity, policy formulation.


Acknowledgements

The authors acknowledge Professor Daniel M. Fox (President of the Milbank Memorial Fund, USA), Professor Alfred Rütten and Dr Karim Abu-Omar (Friedrich-Alexander-University, Germany), Associate Professor Richard H. Osborne (University of Melbourne), Dr James Gillespie (School of Public Health, University of Sydney), and Ms Laurann Yen (Australian Primary Health Care Research Institute, Australian National University) for their comments on the criteria and help in locating some of the literature and policy documents. The authors also thank Dr Andrew Kemp (Western Clinical School), Professor A. Stewart Truswell (Human Nutrition Unit, University of Sydney) and Dr Angela Beaton (Menzies Centre for Health Policy, University of Sydney) for critical reading of the manuscript. Thanks to the Serious and Continuing Illness Policy and Practice Study (SCIPPS) project team for provision of some of the policy documents. Masoud Mirzaei was supported by a National Health and Medical Research Council Program Grant (No. 402793), on which he was a postdoctoral research fellow.


References


[1] Buse K , Mays N , Walt G . Making Health Policy. London: Open University Press; 2005.

[2] Smith-Merry J,  Gillespie J,  Leeder SR. A pathway to a stronger research culture in health policy. Aust NZ Health Policy 2007; 4 19.
CrossRef |

[3] Redman S,  Jorm L,  Haines M. Increasing the use of research in health policy: the Sax Institute model, The Sax Institute. Australasian Epidemiologist 2008; 15 15–8.


[4] Rütten A,  Luschen G,  von Lengerke T,  Abel T,  Kannas L,  Rodríguez Diaz JA,  Vinck J,  van der Zee J. Determinants of health policy impact: comparative results of a European policymaker study. Soz Praventivmed 2003; 48(6): 379–91.
CrossRef | PubMed |

[5] Mazmanian DA , Sabatier PA . Effective policy implementation. Lexington, MA: Lexington Books; 1981.

[6] Mazmanian DA , Sabatier PA . Implementation and public policy. Glenview, IL: Scott, Foresman; 1983.

[7] Health system expenditure on disease and injury in Australia, 2000–01. Canberra: Australian Institute of Health and Welfare; 2004. Available at http://www.aihw.gov.au/publications/hwe/hsedia00-01/hsedia00-01.pdf [verified 26 June 2008].

[8] Dunstan DW,  Zimmet PZ,  Welborn TA,  de Courten MP,  Cameron AJ,  Sicree RA,  Dwyer T,  Colagiuri S,  Jolley D,  Knuiman M,  Atkins R. The rising prevalence of diabetes and impaired glucose tolerance: the Australian Diabetes, Obesity and Lifestyle Study. Diabetes Care 2002; 25 829–34.
CrossRef | PubMed |

[9] Tobias M,  Taylor R,  Yeh LC,  Huang K,  Mann S,  Sharpe N. Did it fall or was it pushed? The contribution of trends in established risk factors to the decline in premature coronary heart disease mortality in New Zealand. Aust N Z J Public Health 2008; 32 117–25.
CrossRef | PubMed |

[10] Improving health care for people with chronic illness – a blueprint for change 2001–2003. Sydney: New South Wales Health; 2001. Available at http://www.health.nsw.gov.au/pubs/2001/chronic.html [verified 1 December 2008].

[11] Harvey P. Managing health care in Australia: steps on the health care roundabout? Aust J Primary Health 2003; 9 105–8.
CrossRef |

[12] NSW Chronic Care Program. Sydney: New South Wales Health; 2007. Available at http://www.health.nsw.gov.au/health_pr/chronic_care/ [verified 26 April 2007].

[13] Head BW. Three lenses of evidence-based policy. Aust J Publ Admin 2008; 67(1): 1–11.
CrossRef |

[14] von Wright GH . Determinism and the study of man. In: Manninen J, Tuomela R, editors. Essays on explanation and understanding. Dordrecht: Deidel; 1976.

[15] Rütten A,  Luschen G,  von Lengerke T,  Abel T,  Kannas L,  Rodríguez Diaz JA,  Vinck J,  van der Zee J. Determinants of health policy impact: a theoretical framework for policy analysis. Soz Praventivmed 2003; 48(5): 293–300.
CrossRef | PubMed |

[16] Bendor J,  Moe TM,  Shotts KW. Recycling the garbage can: an assessment of the research program. Am Polit Sci Rev 2001; 95(1): 169–90.


[17] Cohen MD,  March JG,  Olsen JP. A garbage can model of organizational choice. Adm Sci Q 1972; 17(1): 1–25.
CrossRef |

[18] Exworthy M. Policy to tackle the social determinants of health: using conceptual models to understand the policy process. Health Policy Plan 2008; 23 318–27.
CrossRef | PubMed |

[19] Althaus C , Bridgman P , Davis G . The Australian policy handbook. 4th edn. Sydney: Allen & Unwin; 2007.

[20] Hodge W , Davies G . Evaluating policy with a modified ‘policy cycle’ – the NSW Healthy School Canteen Strategy. In: Australian Evaluation Society International Conference; 4–7 September 2006; Darwin. 2006. Available at http://www.aes.asn.au/conferences/2006/papers/067%20Wendy%20Hodge.pdf [verified 9 September 2010].

[21] King L,  Turnour C,  Wise M. Analysing NSW state policy for child obesity prevention: strategic policy versus practical action. Aust NZ Health Policy 2007; 4 22.
CrossRef |

[22] Simonsen-Rehn N,  Øvretveit J,  Laamanen R,  Suominen S,  Sundell J,  Brommels M. Determinants of health promotion action: comparative analysis of local voluntary associations in four municipalities in Finland. Health Promot Int 2006; 21(4): 274–83.
CrossRef | PubMed |

[23] Gray JAM . Evidence-based healthcare. New York: Churchill Livingstone; 2001.

[24] WHO. Assistance to national authorities in developing and strengthening national food safety programme. Rome: WHO–European Centre for Environment and Health, Rome Division; 2002. Available at http://www.euro.who.int/document/FOS/gsnfsp6.pdf [verified 12 January 2008].

[25] Nagel SS , Policy Studies Organization. Policy theory and policy evaluation: concepts, knowledge, causes, and norms. New York: Greenwood Press; 1990.

[26] NSW Chronic Care Program Phase Three: 2006–2009. Sydney: NSW Department of Health; 2006. Available at http://www.health.nsw.gov.au/pubs/2006/chroniccare_3.html [verified 12 January 2008].

[27] Raczynski J , Philips M , Bursac Z , Pulley L , West D , Birdsong M , Evans V , Gauss H , Louvring M , Walker J . Establishing a baseline to evaluate Act 1220 of 2003, an act of the Arkansas General Assembly to combat childhood obesity. Little Rock, AR: College of Public Health, University of Arkansas for Medical Sciences; 2005. Available at http://www.uams.edu/coph/reports/ [verified 17 April 2008].

[28] Hammersley M . What’s Wrong With Ethnography? London: Routledge; 1992.

[29] Bennett S . What is Health Policy and Systems Research and why does it matter? 2007. Available at http://www.who.int/alliance-hpsr/resources/AllBriefNote1_5.pdf [verified 22 January 2008].

[30] Rikard-Bell G,  Waters E,  Ward J. Evidence-based clinical policy: case report of a reproducible process to encourage understanding and evaluation of evidence. Intern Med 2006; 36 452–7.
CrossRef | CAS |

[31] Ryan KW,  Card-Higginson P,  McCarthy SG,  Justus MB,  Thompson JW. Arkansas fights fat: translating research into policy to combat childhood and adolescent obesity. Health Aff 2006; 25 992–1004.
CrossRef |

[32] NSW Chronic Care Program: Phase Two 2003–2006. Sydney: NSW Department of Health; 2004. Available at http://www.health.nsw.gov.au/pubs/2005/pdf/chronic_care2.pdf [verified 12 November 2007].

[33] NSW Chronic Care Program 2000–2003: strengthening capacity for chronic care in the NSW health system. Report on Phase one. Sydney: NSW Department of Health; 2004. Available at http://www.health.nsw.gov.au/pubs/2004/pdf/chroniccare2003.pdf [verified 1 December 2007].

[34] NSW Chronic Disease Prevention Strategy 2003–2007. Sydney: New South Wales Department of Health; 2003. Available at http://www.health.nsw.gov.au/pubs/2003/chronic_disease.html [verified 21 January 2009].

[35] Chronic and complex care state-wide performance measures – methodology. Sydney: NSW Department of Health; 2002. Available at http://www.health.nsw.gov.au/pubs/2002/pdf/chronicmeth.pdf [verified 12 January 2008].

[36] Jenkins C . Asthma. Canberra: Commonwealth Department of Health and Ageing; 2003. Available at www.healthinsite.gov.au [verified 31 February 2008].

[37] Lorig K,  Sobel DS,  Stewart AL,  Brown BW,  Bandura A,  Ritter P,  Gonzalez V,  Laurent DD,  Holman H. Evidence suggesting that a chronic disease self-management program can improve health status while reducing utilisation and costs: a randomised trial. Med Care 1999; 37 5–14.
CrossRef | CAS | PubMed |

[38] Nankervis J . Taking carers’ issues into account in discharge planning. Melbourne, Carers Victoria; 2001.

[39] National Chronic Disease Strategy. Canberra: National Health Priority Action Council, Commonwealth Department of Health and Ageing; 2006. Available at http://www.health.gov.au/internet/wcms/publishing.nsf/Content/pq-ncds [verified 17 August 2007].

[40] AcademyHealth. Health Services Research and Health Policy Grey Literature Project: Summary Report. 2006. Available at http://www.nlm.nih.gov/nichsr/greylitreport_06.html [verified 3 January 2008].

[41] McMaster HSC. How to read clinical journals: V. To distinguish useful from useless or even harmful therapy. Can Med Assoc J 1981; 124 1156–62.
PubMed |

[42] Cook DJ,  Sackett DL,  Spitzer W. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis. J Clin Epidemiol 1995; 48 167–71.
CrossRef | CAS | PubMed |


Full Text PDF (403 KB) Export Citation Cited By (5)

View Altmetrics