Going digital: a narrative overview of the clinical and organisational impacts of eHealth technologies in hospital practiceJustin Keasberry A , Ian A. Scott A B D , Clair Sullivan A , Andrew Staib A and Richard Ashby C
B Southern School of Medicine, University of Queensland, Translational Research Institute, 199 Ipswich Road, Brisbane, Qld 4102, Australia.
C Metro South Hospital and Health Service, Garden City Park, 2404 Logan Road, Brisbane, Qld 4113, Australia. Email: email@example.com
D Corresponding author. Email: firstname.lastname@example.org
Australian Health Review 41(6) 646-664 https://doi.org/10.1071/AH16233
Submitted: 8 May 2016 Accepted: 4 November 2016 Published: 9 January 2017
Journal compilation © AHHA 2017 Open Access CC BY-NC-ND
Objective The aim of the present study was to determine the effects of hospital-based eHealth technologies on quality, safety and efficiency of care and clinical outcomes.
Methods Systematic reviews and reviews of systematic reviews of eHealth technologies published in PubMed/Medline/Cochrane Library between January 2010 and October 2015 were evaluated. Reviews of implementation issues, non-hospital settings or remote care or patient-focused technologies were excluded from analysis. Methodological quality was assessed using a validated appraisal tool. Outcome measures were benefits and harms relating to electronic medical records (EMRs), computerised physician order entry (CPOE), electronic prescribing (ePrescribing) and computerised decision support systems (CDSS). Results are presented as a narrative overview given marked study heterogeneity.
Results Nineteen systematic reviews and two reviews of systematic reviews were included from 1197 abstracts, nine rated as high quality. For EMR functions, there was moderate-quality evidence of reduced hospitalisations and length of stay and low-quality evidence of improved organisational efficiency, greater accuracy of information and reduced documentation and process turnaround times. For CPOE functions, there was moderate-quality evidence of reductions in turnaround times and resource utilisation. For ePrescribing, there was moderate-quality evidence of substantially fewer medications errors and adverse drug events, greater guideline adherence, improved disease control and decreased dispensing turnaround times. For CDSS, there was moderate-quality evidence of increased use of preventive care and drug interaction reminders and alerts, increased use of diagnostic aids, more appropriate test ordering with fewer tests per patient, greater guideline adherence, improved processes of care and less disease morbidity. There was conflicting evidence regarding effects on in-patient mortality and overall costs. Reported harms were alert fatigue, increased technology interaction time, creation of disruptive workarounds and new prescribing errors.
Conclusion eHealth technologies in hospital settings appear to improve efficiency and appropriateness of care, prescribing safety and disease control. Effects on mortality, readmissions, total costs and patient and provider experience remain uncertain.
What is known about the topic? Healthcare systems internationally are undertaking large-scale digitisation programs with hospitals being a major focus. Although predictive analyses suggest that eHealth technologies have the potential to markedly transform health care delivery, contemporary peer-reviewed research evidence detailing their benefits and harms is limited.
What does this paper add? This narrative overview of 19 systematic reviews and two reviews of systematic reviews published over the past 5 years provides a summary of cumulative evidence of clinical and organisational effects of contemporary eHealth technologies in hospital practice. EMRs have the potential to increase accuracy and completeness of clinical information, reduce documentation time and enhance information transfer and organisational efficiency. CPOE appears to improve laboratory turnaround times and decrease resource utilisation. ePrescribing significantly reduces medication errors and adverse drug events. CDSS, especially those used at the point of care and integrated into workflows, attract the strongest evidence for substantially increasing clinician adherence to guidelines, appropriateness of disease and treatment monitoring and optimal medication use. Evidence of effects of eHealth technologies on discrete clinical outcomes, such as morbid events, mortality and readmissions, is currently limited and conflicting.
What are the implications for practitioners? eHealth technologies confer benefits in improving quality and safety of care with little evidence of major hazards. Whether EMRs and CPOE can affect clinical outcomes or overall costs in the absence of auxiliary support systems, such as ePrescribing and CDSS, remains unclear. eHealth technologies are evolving rapidly and the evidence base used to inform clinician and managerial decisions to invest in these technologies must be updated continually. More rigorous field research using appropriate evaluation methods is needed to better define real-world benefits and harms. Customisation of eHealth applications to the context of patient-centred care and management of highly complex patients with multimorbidity will be an ongoing challenge.
Considerable financial investment is currently being devoted in many countries to implementing potentially transformative eHealth technologies. For example, England has invested at least £12.8 billion in a National Program for Information Technology (NPfIT) for the National Health Service1 and the Obama administration in the US has similarly committed to a US$34 billion eHealth investment in health care.2,3 The eHealth market worldwide is predicted to reach US$308 billion by 2022.4 Much of this expenditure has been directed towards digitising hospital practice. Queensland Health has embarked on an A$1.3 billion state-wide campaign to digitise all its major public hospitals by 2020.5 Such large-scale expenditure has been justified on the grounds that electronic medical records (EMRs), electronic prescribing (ePrescribing) and associated computerised provider (or physician) order entry systems (CPOE) and computerised decision support systems (CDSS) will help address the problems of variable quality and safety in modern health care, improve efficiency and contain rising healthcare costs.5–9 However, the evidence base underpinning such claims remains uncertain.
The aim of the present study was to generate a narrative overview of systematic reviews and reviews of reviews of contemporary eHealth technologies that have assessed their effects on the quality, safety and resource utilisation of hospital-based health care delivery compared with traditional paper-based systems of care.
Scope of the overview
Given the wide spectrum of eHealth technologies, the foci of this review were those having two functions (with some inevitable overlap): (1) primary technology to enable the storage, retrieval and transmission of clinical data (EMRs); and (2) auxiliary technologies to support clinical decision making (CPOE, ePrescribing and CDSS). Technologies supporting remote care (Telehealth, Telemonitoring and similar) were excluded from the analysis, as were patient-focused interventions (patient-controlled EMRs, electronic messaging or education). Given the rapid evolution of eHealth technologies, we restricted the present review to reports published in the past 5 years to ensure relevance to current deliberations.
Search strategy, data sources and study selection
Established Cochrane-based systematic review principles were used to search for relevant systematic reviews and reviews of such reviews. We used the pre-specified PubMed Clinical Queries search string for ‘electronic health record’ (see Item S1, available as Supplementary Material to this paper) and ‘article type = review’ and ‘publication date = last 5 years’ filters to search PubMed, MEDLINE and Cochrane Library contents for reviews published in English-language journals, either in print or online, from 1 January 2010 to 31 October 2015. Additional searches were performed using PubMed Clinical Queries function (with ‘systematic reviews’ search filter) and relevant search terms for CPOE, ePrescribing and CDSS. The bibliographies of retrieved reviews were scrutinised to find additional reviews. Articles were screened for inclusion on the basis of: (1) reference to the study as being a systematic review or a review of systematic reviews by the authors within the title, abstract or text; and/or (2) evidence from the description of the methods that systematic review principles had been used in searching and appraising the evidence. Reviews that contained randomised trials and/or observational studies were included in the analysis. Studies were excluded if, for any of the four eHealth technologies: (1) they focused only on single disciplines (e.g. oncology, mental health), single class of investigations or medications or single vendor systems; (2) dealt primarily with implementation issues; or (3) were conducted exclusively or predominantly in non-hospital settings or in developing countries. A complete list of inclusion and exclusion criteria is provided in Item S2. All potentially suitable reviews were independently assessed for inclusion by two reviewers (JK and IAS) and agreement reached by consensus. A log of excluded studies and reasons for exclusion is available on request from the authors.
Quality of evidence
The methodological quality and risk of bias of each review was critically appraised using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR; https://amstar.ca/Amstar_Checklist.php, verified 17 September 2015) tool for systematic reviews (Item S3). Reviews were rated as high quality on the basis of AMSTAR scores ≥8.
Data extraction and synthesis
Data from included reviews were abstracted using a standardised format by one reviewer (JK) and cross-checked by a second reviewer (IAS). In synthesising the data, the heterogeneity of reviews was too diverse in terms of eTechnologies and outcome measures to allow meta-analysis of pooled data and therefore a narrative synthesis was undertaken.
The outcome measures, based on a typology of eHealth technology functions and effects developed by Black et al.,6 are listed below.
For EMR alone, the benefits (compared with paper-based systems) were categorised as data security, legibility, accessibility, completeness, comprehensiveness, efficiency and secondary uses. The harms were categorised as paper persistence, patient disengagement, insecure data, increased time for data entry and increased costs.
For CPOE, the benefits were categorised as resource utilisation, indicated care, patient outcomes (all-cause mortality, disease-specific mortality, disease-specific events, symptom measures, quality of life scores (as stated by the authors)), cost savings and time savings. The harms were categorised as increased time to perform task, interruptions, increased costs and workarounds. Where reviews reported findings in the CPOE category that were related to orders for medications, we thought it more appropriate to place these in the ePrescribing category for analysis.
For ePrescribing, the benefits were categorised as surrogate outcomes (as stated by the authors), guideline adherence, safer prescribing, communication, patient outcomes (as listed above), resource and/or cost savings and time savings. The harms were categorised as patient harm (adverse drug-related events (ADEs) or mortality), increased time to perform task and increased costs.
For CDSS, the benefits were categorised as indicated care, guideline adherence, surrogate outcomes (as stated by the authors) and patient outcomes (as stated above). The harms were categorised as practitioner performance and patient outcomes. Although we anticipated many CDSS would be applied to CPOE and ePrescribing functions, we elected to retain them in the CDSS category in keeping with their primary intended function.
Estimates of effect related to outcome measures
Estimates of effect related to outcome measures were defined as: (1) qualitative measures, as stated by the authors (i.e. no, small (modest), moderate or large effect); (2) standardised effect size (ES), where an ES of 0.2 indicates a small effect, an ES of 0.5 indicates a moderate effect and an ES of 0.8 indicates a large effect; (3) relative risk (RR) or odds ratio (OR) for benefit (or harm); and (4) absolute risk reduction (ARR) or increase (ARI) for benefit or harm respectively.
Quality of evidence for each review
Where review authors assessed the quality of the primary studies using numerical scoring methods (expressed as mean quality scores for all studies) or rated the quality of evidence as high, moderate, low, or very low based on the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) system,10 these scores or ratings were reported.
The search strategy retrieved 1197 abstracts for screening, of which 1139 were rejected, leaving 58 articles that were subject to full-text review. Of these, 21 articles were included in the overview (see Fig. 1), comprising 19 systematic reviews7–9,11–26 and two reviews of systematic reviews.6,27 In all, 18 of the 21 (85.7%) articles evaluated CDSS, 6–9,11–19,22–24,26,27 five (23.8%) evaluated data information retrieval and transfer,6,9,20,25,27 six (28.6%) measured effects of CPOE6,9,16,21,22,27 and three (14.3%) assessed ePrescribing.6,16,25 Most articles evaluated more than one eHealth technology and nine of the 21 (42.9%) articles were rated as high quality.6,8,9,15,18,19,20,21,24 A summary of evidence from the studies included and interpretive comments made by their authors are given in Appendix 1 for systematic reviews and in Appendix 2 for reviews of reviews, listed in descending level of quality (and in chronological order within each level); key findings from each type of review are presented below.
Electronic medical records
One low-quality review reported enhanced communication of information between providers and a 22% reduction in documentation time.25 In a high-quality review, there was moderate-quality evidence of a 15% reduction in hospitalisations and small decreases in both the length of hospital stay and the number of patient visits to emergency departments as a result of electronically generated reports of investigations containing care recommendations.20 The same review noted no improvement in timeliness of discharge summaries to primary care providers and no effects on disease-specific processes of care or clinical outcomes.20 Two reviews, one low quality23 and another high quality,9 that examined in-hospital mortality showed no effects from EMR. However, EMRs that included electronic surveillance systems for monitoring patient deterioration were associated with a 15% reduction in in-hospital mortality.9
Reviews of reviews
In one high-quality review, there was limited evidence of improvement in data security, interpretability, accessibility, comprehensiveness and search and retrieval functions for secondary uses such as research.6 One low-quality review noted modest effects of EMRs on information accuracy and completeness, with just over half the included studies reporting positive metrics for data storage, retrieval and transfer.27 There were also trends towards improved organisational efficiency and greater use of health information exchange technology.6,27 Although there were varying levels of sophistication and functionality across home-developed and commercial systems, time efficiency depended on the skills of end-users.6
Computerised provider order entry
One high-quality review reported no effects of CPOE on hospital mortality or length of stay.9
Reviews of reviews
In one low-quality review, positive feedback was reported for CPOE in two-thirds of 287 studies, but there was no effect on clinical performance or overall costs.27 In a high-quality review of 53 reviews, modest effects were seen on turnaround times for processing and delivering laboratory investigation, as well as on practitioner performance.6 The same review noted small time and cost savings overall, but there was also evidence of increased time to interact with CPOE systems in order to address alerts and override or confirm orders, leading to interruptions and workarounds at the point of care.6
One high-quality review quantified the benefits of ePrescribing as a 54% decrease in medication errors and a 53% decrease in ADEs,21 whereas the corresponding reductions seen in a low-quality review were 54% and 34% respectively.25 However, a twofold risk of new prescribing errors (e.g. concurrent submission of duplicate orders) was reported in two of 16 studies, which could potentially be disruptive to work flows but, to date, has not been associated with patient harm.21
Review of reviews
One high-quality review reported moderate evidence of improvement in organisational efficiency, more accurate communication between prescribers and pharmacists, and limited evidence of decreased turnaround times in supplying stock to wards and filling discharge prescriptions.6 The same review reported limited to moderate evidence of reductions in medication errors and ADEs, increases in more appropriate prescribing, greater guideline adherence, improvement in disease outcomes and savings on cost and resource use.6 However, these benefits came at the cost of introducing alert fatigue and inadvertent selection of incorrect type and dosage of medications.6
Computerised decision support systems
Multiple CDSS were described with variable levels of sophistication with regard to inputs, targeted goals and prompts and inference mechanisms. High-quality reviews showed moderate evidence (more than 50% of primary studies were positive) of improvements in processes of care15 and appropriateness of test ordering.24 In one review there was moderate-quality evidence of a 42% increase in preventive care services and a 72% increase in ordering or completion of recommended clinical investigations, and high-quality evidence of a 57% increase in adherence to treatment guidelines.17 However, the same review reported no effects on mortality, ADEs, length of stay or clinician confidence in patient care.17 In contrast, in another high-quality review, morbid events were reduced by 18% across a range of clinical conditions,8 whereas in a third high-quality review, in-hospital mortality was reduced by 17%, which was just below statistical significance (P = 0.05).9 One low-quality review reported a twofold increase in adherence to clinical recommendations, a 42% increase in the use of preventive care services and a 57% increase in the appropriate use of medical treatments, in association with a 12% decrease in the incidence of morbid events.22 However, this same review noted that, in the presence of alert fatigue, adherence to advice decreased by 63%.22 Another low-quality review noted moderate effects in reducing inappropriate diagnostic imaging and modest reductions in overall use, but also slight increase (~7%) in failure to order appropriate tests when indicated.23 Other low-quality reviews noted evidence of less ordering of redundant tests7,11,26 (up to 18% less11), increased use of alerts and reminders12,13 and diagnostic and medication dosing aids,13 improved processes of care,12,14 lower costs,14 a 33% greater adherence to guidelines25 and a 60% decrease in turnaround times.26 With regard to chronic disease management, one low-quality review reported improved management of diabetes only,12 whereas in another review, management of other chronic diseases, such as ischaemic heart disease,18 hypertension and chronic obstructive pulmonary disease, also benefited.13 Various reviews showed more significant optimisation of the use of high-risk drugs in certain patient subgroups, such as those receiving insulin,16 vitamin K antagonists,15 antibiotics and anti-rejection drugs.19
Reviews of reviews
The present narrative overview provides an update on the cumulative evidence of clinical and organisational effects of contemporary eHealth technologies in hospital practice. The current evidence base is limited for EMR simply as a data storage, retrieval and transfer platform, and for CPOE. In contrast, ePrescribing and CDSS have attracted greater research interest, probably because these systems have the greatest potential to directly and significantly affect patient care and outcomes. However, the overall quality of evidence is low, with only nine of 21 reviews being rated as high quality. Moreover, the results of different reviews, even those ostensibly studying the same question, yield somewhat inconsistent results, which may reflect, in part, the heterogeneity of the populations studied, technologies analysed and outcomes measured. Positive effects in many reviews are often small to moderate in magnitude and based on low- to moderate-quality evidence. There is very limited and conflicting evidence of the effects of eHealth technologies on patient-important clinical outcomes, such as morbid events, mortality and unplanned readmissions. Measures of hospital bed use, equity of access, resource utilisation, patient satisfaction or quality of life measures, or provider satisfaction and perceived ease of use, were rarely, if ever, reported. Although some reviews made mention of overall costs, data relating to cost-effectiveness remains sparse and is no doubt explained by the complexity of measuring direct and indirect costs of development and implementation of eHealth technologies over a period of time sufficiently long enough to allow adequate evaluation.
Study strengths and limitations
The present overview provides a current synopsis of the evidence of effect of currently available eHealth technologies that may assist project groups in developing business cases for local design and implementation. Strengths include a comprehensive search for systematic reviews and reviews of reviews from the literature and an assessment of their quality using a validated appraisal instrument. Detailed information on study characteristics, process and outcome measures according to four categories of technology functions, and interpretive comments, were extracted from each review, enabling readers to better assess relevant intervention effects.
Limitations relate to inadequate indexing of eHealth technologies in the literature, although the search method was one developed and endorsed by PubMed and it is unlikely that sentinel articles were missed. The scope of the present review was restricted to articles published in the past 5 years, which may have led to oversight of effective technologies that have withstood the test of time, although many reviews included individual studies dating back more than two decades. We did not search grey literature from websites or other sources given the potential for bias in reports not subject to peer review. We only assessed clinical and organisational effects devoid of any consideration of contextual issues around implementation, which may have affected the results. We concede overviews, by aggregating results from multiple primary studies, can only generate estimates of ‘average’ effects that may totally obscure highly positive results in individual trials. In addition, reviews rarely distinguished between studies that evaluated eHealth technologies as before–after ‘brownfield’ designs (i.e. an existing hospital converting from paper-based to computerised systems) or comparative ‘greenfield’ designs (newly built computerised hospitals or units compared with concurrent or historical controls). However, our aim was to generate effect estimates that were broadly generalisable.
Our categorisation of results of some reviews into CPOE, ePrescribing or CDSS could be regarded as arbitrary, but overlap of these functions was, at times, unavoidable and our choice of category was explained. There is the also the potential for overemphasising the findings of particular reviews (and their included primary studies) by including multiple reviews and reviews of reviews that duplicate the same studies multiple times. However, adopting this approach affords the opportunity for noting consistency in, and hence potential robustness of, the findings of different reviews performed by different authors. Although we were unable to assess the quality of individual trials in every review, we did cite the methods and results used to conduct such assessments whenever reported by review authors, and the AMSTAR quality criteria we used to appraise the reviews take account of whether such assessments had been undertaken. We were unable to pool data across all reviews, but reported quantitative results for those reviews that undertook meta-analyses. Many evaluations were conducted in large academic institutions by potentially conflicted developers of the eHealth technologies, which raises the potential for information and publication bias leading to overestimation of benefits, with most reviews making no mention of potential harms. Where review authors tested for such bias, we have reported their findings.
Implications for clinical practice
The authors have been involved in the recent digitisation of a large tertiary adult hospital in Brisbane that incorporates all four eHealth functions mentioned above. Presenting a synthesis of evidence of effects helped inform clinicians’ and managers’ views of the benefits (and harms) of digitisation, and assisted in gaining hospital-wide acceptance for such a major transformational change.
This overview suggests EMRs have the potential to increase the accuracy and completeness of clinical information and to reduce documentation time. Improvements in information transfer and organisational efficiency may translate into reduced hospitalisations, emergency visits and redundant test requests. Whether EMRs in the absence of auxiliary support systems affect clinical outcomes or overall costs remains unclear. Similarly, although CPOE appears to improve laboratory turnaround times, this is offset by increased interaction times, workflow interruptions and workarounds. Neither EMR nor CPOE has been shown to reduce mortality, length of hospital stay or overall costs.
There is reasonably strong evidence that ePrescribing significantly reduces medication errors and ADEs and may further increase patient safety and organisational efficiency with reminders relating to significant drug interactions that may otherwise be missed. Eliminating illegible handwriting and inappropriate dose prescribing by physicians, and improving communication with pharmacists, are proven benefits. However, there is potential for alert fatigue and duplication or wrong selection of medication type and dosage.
Evidence for improving processes and quality of care that affect clinical outcomes is currently strongest for CDSS, especially those used at the point of care and integrated well into workflows. Reviews have shown that such systems substantially increase clinician adherence to guidelines, appropriateness of disease and treatment monitoring and optimal use of medications. However, customisation of CDSS applications to the context of patient-centred care and management of highly complex patients with multimorbidity remains an ongoing challenge.
This overview of recently published systematic reviews of eHealth technologies may help inform decisions to invest in eHealth technologies in hospitals throughout Australia. Overall, eHealth technologies, especially ePrescribing and CDSS, appear effective in improving health care processes and outcomes across diverse settings using both commercially and locally developed systems.
eHealth technologies are evolving rapidly and much is still to be learned as to how these tools should be designed and used in ways that optimise their effectiveness. Evaluation methodologies most suited to assessing benefits and harms of these technologies, in both quantitative and qualitative terms, need to be deployed and refined over time. More rigorous field research targeting hospitals undergoing digital transformation, and performed by independent, multidisciplinary research groups, is required to narrow the gap between theorised potential benefits of eHealth technologies and empirically demonstrated real-world improvements in patient care and outcomes and efficient use of resources.
References Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 2009; 6 e1000126
| Evaluating eHealth interventions: the need for continuous systemic evaluation.CrossRef |
 Romer C, Berstein J. The job impact of the American Recovery and Reinvestment Plan. 2009. Available at: http://www.ampo.org/assets/library/184_obama.pdf [verified 19 April 2016].
 Joseph M. How President Obama shaped the future of digital health. Available at: https://techcrunch.com/2016/07/27/how-president-obama-shaped-the-future-of-digital-health/ [verified 30 July 2016].
 Grand View Research Inc. eHealth market will reach $308.0 billion by 2022: Grand View Research, Inc. 2015. Available at: https://globenewswire.com/news-release/2015/11/18/788256/0/en/eHealth-Market-Will-Reach-308-0-Billion-By-2022-Grand-View-Research-Inc.html [verified 19 April 2016].
 Queensland Health. 21st century healthcare. eHealth investment strategy. 2015. Available at: https://www.health.qld.gov.au/publications/portal/ehealth-investment-strategy/ehealthinvestmentstrategy.pdf [verified 19 April 2016].
 Black AD, Car J, Pagliari C, et al The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011; 8 e1000387
| The impact of eHealth on the quality and safety of health care: a systematic overview.CrossRef |
 Roshanov P, You JJ, Dhailwal J, et al Can computerized clinical decision support systems improve practitioners’ diagnostic test ordering behaviour. A systematic review. Implement Sci 2011; 6 88–99
| Can computerized clinical decision support systems improve practitioners’ diagnostic test ordering behaviour. A systematic review.CrossRef |
 Moja L, Kwag KH, Lytras T, et al The effects of computerised decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health 2014; 104 e12–22.
| The effects of computerised decision support systems linked to electronic health records: a systematic review and meta-analysis.CrossRef |
 Thompson G, O’Horo JC, Pickering BW, et al Impact of the electronic medical record on mortality, length of stay and cost in the hospital and ICU: a systematic review and meta-analysis. Crit Care Med 2015; 43 1276–82.
| Impact of the electronic medical record on mortality, length of stay and cost in the hospital and ICU: a systematic review and meta-analysis.CrossRef |
 Balshem H, Helfand M, Schünemann HJ, et al GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011; 64 401–6.
 Main C, Moxham T, Wyatt JC, et al Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic review of the effects and cost-effectiveness of systems. Health Technol Assess 2010; 14 48
| Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic review of the effects and cost-effectiveness of systems.CrossRef |
 Sahota N, Lloyd R, Ramakrishna A, et al Computerised decision support systems for acute care management: a decision maker researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci 2011; 6 91–104.
| Computerised decision support systems for acute care management: a decision maker researcher partnership systematic review of effects on process of care and patient outcomes.CrossRef |
 Roshanov P, Misra S, Gerstein HC, et al Computerised decision support systems for chronic disease management: systematic review. Implement Sci 2011; 6 92–107.
| Computerised decision support systems for chronic disease management: systematic review.CrossRef |
 Hemens B, Holbrook A, Tonkin M, et al Computerised decision support systems for drug prescribing and management: systematic review. Implement Sci 2011; 6 89–105.
| Computerised decision support systems for drug prescribing and management: systematic review.CrossRef |
 Nieuwlaat R, Connolly SJ, Mackay JA, et al Computerised decision support systems for therapeutic drug monitoring and dosing: systematic review. Implement Sci 2011; 6 90–103.
| Computerised decision support systems for therapeutic drug monitoring and dosing: systematic review.CrossRef |
 Nirantharakumar R, Chen YF, Marshall T, et al Computerised decision support systems in the care of in patients with diabetes in non-critical care setting: systematic review. Diabet Med 2012; 29 698–708.
| Computerised decision support systems in the care of in patients with diabetes in non-critical care setting: systematic review.CrossRef | 1:CAS:528:DC%2BC38XhtFSru7nL&md5=8f3626a296fbdc70a20f6f9f98d6109bCAS |
 Bright T, Wong A, Dhurjati R, et al Effects of computerised decision support systems. Ann Intern Med 2012; 157 29–43.
| Effects of computerised decision support systems.CrossRef |
 Anchala R, Pinto MP, Shroufi A, et al The role of decision support system (DSS) in prevention of cardiovascular disease: a systematic review and meta-analysis. PLoS One 2012; 7 e47064
| The role of decision support system (DSS) in prevention of cardiovascular disease: a systematic review and meta-analysis.CrossRef | 1:CAS:528:DC%2BC38XhsFOkurnP&md5=84b8f03e102ae74553379468f9b01debCAS |
 Gillaizeau F, Chan E, Trinquart L, et al Computerized advice on drug dosage to improve prescribing practice. Cochrane Database Syst Rev 2013; 11 CD002894
| Computerized advice on drug dosage to improve prescribing practice.CrossRef |
 Ontario Health Technology Assessment Electronic tools for health information exchange: an evidence based analysis Ont Health Technol Assess Ser 2013; 13 1–76.
 Nuckols T, Smith-Spangler C, Morton SC, et al The effectiveness of computerised physician order entry at reducing preventable adverse drug events and medication errors in hospital settings. Syst Rev 2014; 3 56–67.
| The effectiveness of computerised physician order entry at reducing preventable adverse drug events and medication errors in hospital settings.CrossRef |
 Murphy CDS. Effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J Biol Med 2014; 87 187–97.
 Goldzweig C, Orshansky G, Paige NM, et al Electronic health record based interventions for improving appropriate diagnostic imaging. Ann Intern Med 2015; 162 557–65.
| Electronic health record based interventions for improving appropriate diagnostic imaging.CrossRef |
 Goldzweig CL, Orshansky G, Paige NM, Electronic health record based interventions for reducing inappropriate imaging in the clinical setting: a systematic review of the evidence. Veteran Affairs ESP Project #05-226; 2014.
 Campanella P, Lovato E, Marone C, et al The impact of electronic health records on healthcare quality: systematic review. Eur J Public Health 2016; 26 60–4.
| The impact of electronic health records on healthcare quality: systematic review.CrossRef |
 Fraccaro P, Casteleiro MA, Ainsworth J, et al Adoption of computerised decision support in multimorbidity: a systematic review. JMIR Med Inform 2015; 3 e4–17.
| Adoption of computerised decision support in multimorbidity: a systematic review.CrossRef |
 Lau F, Kuziemsky C, Price M, Gardner J. A review on systematic reviews of health information system studies. J Am Med Inform Assoc 2010; 17 637–45.
| A review on systematic reviews of health information system studies.CrossRef |