Health Promotion Journal of Australia Health Promotion Journal of Australia Society
Journal of the Australian Health Promotion Association
RESEARCH ARTICLE (Open Access)

Designing evaluation plans for health promotion mHealth interventions: a case study of the Milk Man mobile app

Becky K. White A D , Sharyn K. Burns A B , Roslyn C. Giglia A C and Jane A. Scott A B
+ Author Affiliations
- Author Affiliations

A School of Public Health, Curtin University, GPO Box U1987, Bentley, Perth, WA 6845, Australia.

B Collaboration for Evidence, Research and Impact in Public Health (CERIPH), Faculty of Health Sciences, Curtin University, PO Box U1987, Bentley, WA 6845, Australia.

C Telethon Kids Institute, University of Western Australia, 100 Roberts Road, Subiaco, WA 6008, Australia.

D Corresponding author. Email: becky.white@curtin.edu.au

Health Promotion Journal of Australia 27(3) 198-203 https://doi.org/10.1071/HE16041
Submitted: 5 May 2016  Accepted: 7 September 2016   Published: 26 October 2016

Abstract

Evaluating complex health promotion interventions that use mobile apps requires comprehensive and adaptive evaluation plans. As mobile usage becomes increasingly sophisticated and personalised, broad evaluation plans are important in determining the impact and efficacy of a mobile health (mHealth) app. Evaluation should consider user feedback and outcome measures, as well as examine elements such as the robustness of the technology, the intervention principles and engagement strategies, and the interaction of the user with the technology. This paper introduces four mHealth evaluation models and tools and describes the evaluation plan that has been developed for Milk Man, a breastfeeding app targeting new and expectant fathers. Milk Man is a socially connected, gamified app that is being tested in a large Randomised Control Trial (RCT). While there is a need for mobile apps to be evaluated in adequately powered RCTs, trialling mobile apps over a long period of time presents challenges. Incorporating robust evaluation design will help ensure that technological performance, app intervention principles, as well as health and behavioural outcomes are measured. The detail and scope of the Milk Man app evaluation plan will ensure the findings add to the evidence base and have broad relevance to health promotion practitioners.

So what?Evidence about the efficacy of mHealth interventions is an emerging area and appropriate evaluation skills are needed. This paper illustrates an evaluation planning approach for mHealth interventions that could be adapted for use by health promotion practitioners and researchers.

Key words: breastfeeding, evaluation methods, evidence-based practice, information and communication technology.


References

[1]  Research2guidance. mHealth app developer economics 2014: the state of the art of mHealth app publishing. 2014. Available from: http://www.research2guidance.com/r2g/research2guidance-mHealth-App-Developer-Economics-2014.pdf [Verified 10 April 2015].

[2]  Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al (2013) The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 10, e1001362
The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review.CrossRef | 23349621PubMed |

[3]  Kratzke C, Cox C (2012) Smartphone technology and apps: rapidly changing health promotion. Int Electron J Health Educ 15, 72–82.

[4]  Becker S, Miron-Shatz T, Schumacher N, Krocza J, Diamantidis C, Albrecht U (2014) mHealth 2.0: experiences, possibilities, and perspectives. JMIR Mhealth Uhealth 2, e24
mHealth 2.0: experiences, possibilities, and perspectives.CrossRef | 25099752PubMed |

[5]  Fanning J, Mullen SP, McAuley E (2012) Increasing physical activity with mobile devices: a meta-analysis. J Med Internet Res 14, e161
Increasing physical activity with mobile devices: a meta-analysis.CrossRef | 23171838PubMed |

[6]  Boushey CJ, Harray AJ, Kerr DA, Schap TE, Paterson S, Aflague T, et al (2015) How willing are adolescents to record their dietary intake? The mobile food record. JMIR Mhealth Uhealth 3, e47
How willing are adolescents to record their dietary intake? The mobile food record.CrossRef | 26024996PubMed |

[7]  White BK, Martin A, White JA, Burns SK, Maycock BR, Giglia RC, et al (2016) Theory-based design and development of a socially connected, gamified mobile app for men about breastfeeding (Milk Man). JMIR Mhealth Uhealth 4, e81
Theory-based design and development of a socially connected, gamified mobile app for men about breastfeeding (Milk Man).CrossRef | 27349756PubMed |

[8]  Heron KE, Smyth JM (2010) Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. Br J Health Psychol 15, 1–39.
Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments.CrossRef | 19646331PubMed |

[9]  Deloitte. Mobile consumer survey 2014. The Australian cut: revolution and evolution. 2014. Available from: http://landing.deloitte.com.au/rs/deloitteaus/images/Deloitte_Mobile_Consumer_Survey_2014.pdf [Verified 22 April 2016].

[10]  Tecmark. Tecmark survey finds average user picks up their smartphone 221 times a day. 2014. Available from: http://www.tecmark.co.uk/smartphone-usage-data-uk-2014/ [Verified 9 April 2016].

[11]  White B, White J, Giglia R, Tawia S (2016) Feed safe: a multidisciplinary partnership approach results in a successful mobile application for breastfeeding mothers. Health Promot J Austr 27, 111–7.
Feed safe: a multidisciplinary partnership approach results in a successful mobile application for breastfeeding mothers.CrossRef |

[12]  Kenny R, Dooley B, Fitzgerald A (2015) Feasibility of “CopeSmart”: a telemental health app for adolescents. JMIR mental health 2, e22
Feasibility of “CopeSmart”: a telemental health app for adolescents.CrossRef | 26552425PubMed |

[13]  Gilliland J, Sadler R, Clark A, O’Connor C, Milczarek M, Doherty S (2015) Using a smartphone application to promote healthy dietary behaviours and local food consumption. Biomed Res Int 2015, 841368
Using a smartphone application to promote healthy dietary behaviours and local food consumption.CrossRef | 26380298PubMed |

[14]  Agarwal S, Perry HB, Long LA, Labrique AB (2015) Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Trop Med Int Health 20, 1003–14.
Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review.CrossRef | 25881735PubMed |

[15]  Agarwal S, Labrique A (2014) Newborn health on the line: the potential mHealth applications. JAMA 312, 229–30.
Newborn health on the line: the potential mHealth applications.CrossRef | 1:CAS:528:DC%2BC2cXhs1egtbbI&md5=d2b852296a8a3ef8455ae4ad3f61fb62CAS | 24953141PubMed |

[16]  Qiang CZYM, Hausman V, Altman DG. Mobile applications for the health sector. 2011. Available from: http://siteresources.worldbank.org/INFORMATIONANDCOMMUNICATIONANDTECHNOLOGIES/Resources/mHealth_report.pdf [Verified 22 April 2016].

[17]  Baskerville NB, Struik LL, Hammond D, Guindon GE, Norman CD, Whittaker R, et al (2015) Effect of a mobile phone intervention on quitting smoking in a young adult population of smokers: randomized controlled trial study protocol. JMIR Res Protoc 4, e10
Effect of a mobile phone intervention on quitting smoking in a young adult population of smokers: randomized controlled trial study protocol.CrossRef | 25599695PubMed |

[18]  Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, et al (2015) Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res 17, e166
Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies.CrossRef | 26155878PubMed |

[19]  O’Connor-Fleming ML, Parker E, Higgins H, Gould T (2006) A framework for evaluating health promotion programs. Health Promot J Austr 17, 61–6.
A framework for evaluating health promotion programs.CrossRef | 16619938PubMed |

[20]  Lobo R, Burns SK, Petrich M (2014) Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health 14, 1315
Supporting health promotion practitioners to undertake evaluation for program development.CrossRef | 25532523PubMed |

[21]  O’Grady L, Witteman H, Bender J, Urowitz S, Wiljer D, Jadad A (2009) Measuring the impact of a moving target: towards a dynamic framework for evaluating collaborative adaptive interactive technologies. J Med Internet Res 11, e20
Measuring the impact of a moving target: towards a dynamic framework for evaluating collaborative adaptive interactive technologies.CrossRef | 19632973PubMed |

[22]  Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M (2015) Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 3, e27
Mobile app rating scale: a new tool for assessing the quality of health mobile apps.CrossRef | 25760773PubMed |

[23]  Hides L, Kavanagh D, Stoyanov S, Zelenko O, Tjondroegoro D, Mani M. Mobile Application Rating Scale (MARS): a new tool for assessing the quality of health mobile applications. Melbourne: Young and Well Cooperative Research Centre; 2014.

[24]  Agarwal S, Lefevre AE, Lee J, L’Engle K, Mehl G, Sinha C, et al (2016) Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 352, i1174
Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist.CrossRef | 26988021PubMed |

[25]  Maycock BR, Scott JA, Hauck YL, Burns SK, Robinson S, Giglia R, et al (2015) A study to prolong breastfeeding duration: design and rationale of the Parent Infant Feeding Initiative (PIFI) randomised controlled trial. BMC Pregnancy Childbirth 15, 159
A study to prolong breastfeeding duration: design and rationale of the Parent Infant Feeding Initiative (PIFI) randomised controlled trial.CrossRef | 26231519PubMed |

[26]  Howat P, Brown G, Burns S, McManus A. Project Planning, using the Precede Proceed Model. In Jirowong S, LIamputtong P, editors. Population health, communities and health promotion. Sydney: Oxford University Press; 2009.

[27]  Jaspers MWM (2009) A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform 78, 340–53.
A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence.CrossRef |

[28]  Dennis C-L., Ross L (2006) Women’s perceptions of partner support and conflict in the development of postpartum depressive symptoms. J Adv Nurs 56, 588–99.
Women’s perceptions of partner support and conflict in the development of postpartum depressive symptoms.CrossRef | 17118038PubMed |

[29]  Dennis CL (2003) The breastfeeding self-efficacy scale: psychometric assessment of the short form. J Obstet Gynecol Neonatal Nurs 32, 734–44.
The breastfeeding self-efficacy scale: psychometric assessment of the short form.CrossRef | 14649593PubMed |


Full Text PDF (98 KB) Export Citation Cited By (1)

View Altmetrics