Accuracy of national key performance indicator reporting from two Aboriginal medical services: potential to underestimate the performance of primary health careIsaac Hill A D , David Johnson A , David Scrimgeour B and Robyn McDermott C
A Aboriginal Health Council of South Australia Inc., 220 Franklin Street, Adelaide, SA 5000, Australia. Email: David.email@example.com
B Spinifex Health Service, PMB 88, Kalgoorlie, WA 6430, Australia. Email: David.firstname.lastname@example.org
C College of Public Health, Medical and Veterinary Sciences, James Cook University, PO Box 6811, Cairns, Qld 4870, Australia. Email: Robyn.Mcdermott@jcu.edu.au
D Corresponding author. Email: Isaac.email@example.com
Australian Health Review - https://doi.org/10.1071/AH16251
Submitted: 7 November 2016 Accepted: 28 March 2017 Published online: 9 May 2017
Objective The aim of the present study was to assess the accuracy of extracting national key performance indicator (nKPI) data for the Online Community Health Reporting Environment for Health Services (OCHREStreams) program using the Pen Computer Systems (Leichhardt, NSW, Australia) Clinical Audit Tool (CAT) from Communicare (Telstra Health Communicare Systems, Perth, WA, Australia), a commonly used patient information management system (PIMS) in Aboriginal primary care.
Methods Two Aboriginal Community-Controlled Health Services (ACCHSs) were recruited to the present study. A sample of regular clients aged ≥55 years from each ACCHS was selected and a subset of 13 nKPIs was examined. A manual case note audit of the nKPI subset within Communicare was undertaken by a clinician at each participating ACCHS and acted as a ‘gold standard’ comparator for three query methods: (1) internal Communicare nKPI reports; (2) PenCS CAT nKPI manual filtering (a third-party data-extraction tool); and (3) nKPI data submitted to the Improvement Foundation qiConnect portal.
Results No errors were found in nKPI data extraction from Communicare using the CAT and subsequent submission to the qiConnect portal. However, the Communicare internal nKPI report included deceased clients and past patients, and we can be very confident that deceased clients and past patients are also included in the qiConnect portal data. This resulted in inflation of client denominators and an underestimation of health service performance, particularly for nKPIs recording activity in the past 6 months. Several minor errors were also detected in Communicare internal nKPI reports.
Conclusions CAT accurately extracts a subset of nKPI data from Communicare. However, given the widespread use of Communicare in ACCHSs, the inclusion of deceased clients and past patients in the OCHREStreams nKPI data program is likely to have resulted in systematic under-reporting of health service performance nationally.
What is known about the topic? There has been limited validation of health data exported via data-extraction tools in Australia. More specifically, there are no current published data describing the accuracy of the CAT in mapping health data extracted from Communicare or the accuracy of internal nKPI reports generated by Communicare. Further, no systematic review has been undertaken to assess the accuracy of the nKPI data submission pathway from PIMSs at the health service level to the OCHREStreams qiConnect portal using the CAT.
What does this paper add? The CAT accurately extracts a subset of nKPI data from Communicare and accurately submits this to the qiConnect portal. Minor errors exist in some Communicare internal nKPI reports. The inclusion of deceased clients and past patients in the nKPI reporting system for ACCHSs is likely to have resulted in systematic under-reporting of health service performance nationally through this program.
What are the implications for practitioners? The inclusion of deceased clients and past patients in the OCHREStreams nKPI program limits the usefulness of these data for local quality improvement activities and national monitoring of health service performance for participating ACCHSs. The use of the CAT by ACCHSs independently from the OCHREStreams program can enable deceased clients and past patients to be excluded from reports that can provide more accurate nKPI data from Communicare for local quality improvement and planning purposes.
References Liaw S, Taggart J, Yu H, de Lusignan S. Data extraction from electronic health records – existing tools may be unreliable and potentially unsafe. Aust Fam Physician 2013; 42 820–3.
 Bailie R, Bailie J, Chakraborty A, Swift K. Consistency of denominator data in electronic health records in Australian primary healthcare services: enhancing data quality. Aust J Primary Health 2015; 21 450–9.
| Consistency of denominator data in electronic health records in Australian primary healthcare services: enhancing data quality.CrossRef |
 HealthConnex. Communicare Client Portal. 2017. Available at: https://portal.healthconnex.com.au/ [verified 5 April 2017].
 Peiris D, Agaloitis M, Patel B, Patel A. Validation of a general practice audit and extraction tool. Aust Fam Physician 2013; 42 816–9.
 Pen CS. Pen CS CAT4. 2017. Available at: http://install.pencs.com.au/ClickOnce/CAT4/publish.htm [verified 7 April 2017].
 Aboriginal Health & Medical Research Council. ACCHS data in ACCHS hands. AH&MRC health data workshop 19 February 2013. Final report. 2013. Available from: http://cqidata.org.au/wp-content/uploads/AHMRC-Data-Workshop-Report-May-2013.pdf [verified 28 August 2015].
 Improvement Foundation. History. 2016. Available at: http://improve.org.au/about-us/history/#2012 [verified 21 June 2016].
 Australian Institute of Health and Welfare. Indigenous primary health care national key performance indicators nKPI. 2016. Available at: http://www.aihw.gov.au/indigenous-primary-health-care-nkpi/ [verified 6 January 2016].
 OCHREStreams. OCHREStreams eBulletin July 2015. 2015. Available at: http://www.ochrestreams.org.au/Pages/eBulletins/July_2015.aspx [verified 18 January 2017].
 Pen Computer Systems. 02 Jun Media release – Pen Computer Systems (PCS). 2015. Available at: http://www.pencs.com.au/media-release/ [verified 4 December 2015].
 Improvement Foundation. A change for good – introducing new data extraction software. 2015. Available at: http://improve.org.au/news-and-media/media-centre/canningtool/#.VqA4pLTi8uM [verified 10 November 2015].
 The Canning Tool. Demonstration Access. 2017. Available at: http://canningtool.com.au/demo.html [verified 5 April 2017].
 OCHREStreams. OCHREStreams eBulletin August 2016. 2016. Available at: http://www.ochrestreams.org.au/Pages/eBulletins/August_2016.aspx [verified 21 December 2016].
 OCHREStreams. OCHREStreams eBulletin December 2016. 2016. Available at: http://www.ochrestreams.org.au/Pages/eBulletins/December_2016.aspx [verified 18 January 2017].
 AusTender. Closed ATM view – HEALTH/016/1617. Australian Government. 2017. Available at: https://www.tenders.gov.au/?event=public.atm.showClosed&ATMUUID=55CA9A31-FB82-3D5F-9811A83E9C029174 [verified 20 January 2017].
 Improvement Foundation. OCHREStreams key results. 2014. Available at: http://improve.org.au/news-and-media/media-centre/key-results/ochrestreams-key-results/#.WOb0QW-GO9I [verified 7 April 2017].
 SMS Technology & Management. Final report. National key performance indicators. Data quality review. 2014. Available at: http://www.health.gov.au/internet/main/publishing.nsf/content/9A3654F963F938B0CA257DDA00771054/$File/nkpquality.pdf [verified 3 March 2017].
 Australian Institute of Health and Welfare. Indigenous primary health care key performance indicators (2015). 2015. Available at: http://meteor.aihw.gov.au/content/index.phtml/itemId/584983 [verified 6 January 2016].
 McDonald K. Communicare rolls out across Indigenous communities in Qld. Pulse+IT 2015. Available at: https://www.pulseitmagazine.com.au/news/australian-ehealth/2577-communicare-rolls-out-across-indigenous-communities-in-qld?highlight=WyJ0ZWxzdHJhIiwidGVsc3RyYSdzIiwidGVsc3RyYXMiXQ== [verified 8 August 2016].
 Stewart J, Sanson-Fisher R, Eades S. Aboriginal and Torres Strait Islander health: accuracy of patient self-report of screening for diabetes, high cholesterol and cervical cancer. Aust N Z J Public Health 2016; 40 S3–6.
| Aboriginal and Torres Strait Islander health: accuracy of patient self-report of screening for diabetes, high cholesterol and cervical cancer.CrossRef |