Digital health interventions have enormous potential as scalable tools to improve
health and healthcare delivery by improving effectiveness, efficiency, accessibility,
safety, and personalization. Achieving these improvements requires a cumulative knowledge
base to inform development and deployment of digital health interventions. However,
evaluations of digital health interventions present special challenges. This paper
aims to examine these challenges and outline an evaluation strategy in terms of the
research questions needed to appraise such interventions. As they are at the intersection
of biomedical, behavioral, computing, and engineering research, methods drawn from
all of these disciplines are required. Relevant research questions include defining
the problem and the likely benefit of the digital health intervention, which in turn
requires establishing the likely reach and uptake of the intervention, the causal
model describing how the intervention will achieve its intended benefit, key components,
and how they interact with one another, and estimating overall benefit in terms of
effectiveness, cost effectiveness, and harms. Although RCTs are important for evaluation
of effectiveness and cost effectiveness, they are best undertaken only when: (1) the
intervention and its delivery package are stable; (2) these can be implemented with
high fidelity; and (3) there is a reasonable likelihood that the overall benefits
will be clinically meaningful (improved outcomes or equivalent outcomes at lower cost).
Broadening the portfolio of research questions and evaluation methods will help with
developing the necessary knowledge base to inform decisions on policy, practice, and
research.
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to American Journal of Preventive MedicineAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Smoking cessation support delivered via mobile phone text messaging (txt2stop): a single-blind, randomised trial.Lancet. 2011; 378: 49-55https://doi.org/10.1016/S0140-6736(11)60701-0
- Adaptive e-learning to improve dietary behaviour: a systematic review and cost-effectiveness analysis.Health Technol Assess. 2011; 15: 1-160https://doi.org/10.3310/hta15370
- Website-delivered physical activity interventions a review of the literature.Am J Prev Med. 2007; 33: 54-64https://doi.org/10.1016/j.amepre.2007.02.041
- Interactive computer-based interventions for sexual health promotion.Cochrane Database Syst Rev. 2010; 9: CD006483https://doi.org/10.1002/14651858.cd006483.pub2
- Can stand-alone computer-based interventions reduce alcohol consumption? A systematic review.Addiction. 2011; 106: 267-282https://doi.org/10.1111/j.1360-0443.2010.03214.x
- Interactive Health Communication Applications for people with chronic disease.Cochrane Database Syst Rev. 2005; 4: CD004274https://doi.org/10.1002/14651858.cd004274.pub4
- Digital interventions to promote self-management in adults with hypertension systematic review and meta-analysis.J Hypertens. 2016; 34: 600-612https://doi.org/10.1097/HJH.0000000000000859
- Computer-based diabetes self-management interventions for adults with type 2 diabetes mellitus.Cochrane Database Syst Rev. 2013; 3: CD008776https://doi.org/10.1111/jebm.12042
- Technological innovations in mental healthcare: harnessing the digital revolution.Br J Psychiatry. 2015; 206: 263-265https://doi.org/10.1192/bjp.bp.113.142612
- Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness.PLoS One. 2014; 9: e98118https://doi.org/10.1371/journal.pone.0098118
- Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults.Cochrane Database Syst Rev. 2015; 3: Cd011565https://doi.org/10.1002/14651858.cd011565
- Computerised cognitive behavioural therapy for psychological distress in patients with physical illnesses: a systematic review.J Clin Psychol Med Settings. 2015; 22: 20-44https://doi.org/10.1007/s10880-015-9420-0
- Internet-based cognitive behavioral therapy for patients with chronic somatic conditions: a meta-analytic review.J Med Internet Res. 2014; 16: e88https://doi.org/10.2196/jmir.2777
- Rapid rate of technological development and its implications for research on digital behavior interventions.Am J Prev Med. 2016; (In press)
- Methodological challenges in online trials.J Med Internet Res. 2009; 11: e9https://doi.org/10.2196/jmir.1052
Michie S, West R. Guide to Development and Evaluation of Digital Behaviour Change Interventions in Healthcare. London: UCL Centre for Behaviour Change; In press
- MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression.Psychiatr Serv. 2015; 66: 1043-1050https://doi.org/10.1176/appi.ps.201400477
- Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data.BMJ. 2013; 346: f540https://doi.org/10.1136/bmj.f540
- DIgital Alcohol Management ON Demand (DIAMOND) feasibility randomised controlled trial of a web-based intervention to reduce alcohol consumption in people with hazardous and harmful use versus a face-to-face intervention: protocol.Pilot Feasibility Stud. 2015; 1: 1-8https://doi.org/10.1186/s40814-015-0023-1
- Does brief telephone support improve engagement with a web-based weight management intervention? Randomized controlled trial.J Med Internet Res. 2014; 16: e95https://doi.org/10.2196/jmir.3199
- Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance.PLoS ONE. 2010; 5: e10939https://doi.org/10.1371/journal.pone.0010939
- Developing and evaluating complex interventions: the new Medical Research Council guidance.BMJ. 2008; 337: a1655https://doi.org/10.1136/bmj.a1655
- Designing and evaluating complex interventions to improve health care.BMJ. 2007; 334: 455-459https://doi.org/10.1136/bmj.39108.379965.BE
- Normalisation process theory: a framework for developing, evaluating and implementing complex interventions.BMC Med. 2010; 8: 63https://doi.org/10.1186/1741-7015-8-63
- Designing and undertaking a health economics study of digital health interventions.Am J Prev Med. 2016; (In press)
- Understanding Development Methods From Other Industries to Improve the Design of Consumer Health IT: Background Report. (Prepared by Westat, under Contract No. HHSA290200900023I.) AHRQ Publication No. 11-0065-EF.Agency for Healthcare Research and Quality, Rockville, MD2011
- Methods to support human-centred design.Int J Hum Comput Stud. 2001; 55: 587-634https://doi.org/10.1006/ijhc.2001.0503
- Interaction Design-Beyond Human-Computer Interaction.John Wiley & Sons, New York2015
- Sketching User Experiences: Getting the Design Right and the Right Design.Morgan Kaufmann, Burlington, MA2010
- A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm.J Am Med Inform Assoc. 1995; 2: 46-57https://doi.org/10.1136/jamia.1995.95202548
- Design challenges and principles for Wizard of Oz testing of location-enhanced applications.IEEE Pervasive Comput. 2007; 6: 70-75https://doi.org/10.1109/MPRV.2007.28
- How we design feasibility studies.Am J Prev Med. 2009; 36: 452-457https://doi.org/10.1016/j.amepre.2009.02.002
Hekler EB, Michie S, Rivera DE, et al. Developing and refining models and theories suitable for digital health interventions. Am J Prev Med. In press.
Yardley L, Spring B, Riper H, et al. Understanding and promoting engagement with digital behavior change interventions. Am J Prev Med. In press.
- Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART).Clin Trials. 2014; 11: 426-434https://doi.org/10.1177/1740774514536795
- Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of HIV/AIDS.AIDS Behav. 2016; 20: S197-S214https://doi.org/10.1007/s10461-015-1145-4
- Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST).Transl Behav Med. 2014; 4: 252-259https://doi.org/10.1007/s13142-013-0247-7
- Translational research in South Africa: evaluating implementation quality using a factorial design.Child Youth Care Forum. 2012; 41: 119-136https://doi.org/10.1007/s10566-011-9164-4
- Web-based smoking-cessation programs: results of a randomized trial.Am J Prev Med. 2008; 34: 373-381https://doi.org/10.1016/j.amepre.2007.12.024
- Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs.Psychol Methods. 2009; 14: 202-224https://doi.org/10.1037/a0015826
- Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations.Psychol Methods. 2012; 17: 153-175https://doi.org/10.1037/a0026972
- Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research.Transl Behav Med. 2014; 4: 260-274https://doi.org/10.1007/s13142-014-0265-0
- A control systems engineering approach for adaptive behavioral interventions: illustration with a fibromyalgia intervention.Transl Behav Med. 2014; 4: 275-289https://doi.org/10.1007/s13142-014-0282-z
- Using engineering control principles to inform the design of adaptive interventions: a conceptual introduction.Drug Alcohol Depend. 2007; 88: S31-S40https://doi.org/10.1016/j.drugalcdep.2006.10.020
- Factorial experiments: efficient tools for evaluation of intervention components.Am J Prev Med. 2014; 47: 498-504https://doi.org/10.1016/j.amepre.2014.06.021
- “SMART” design for building individualized treatment sequences.Annu Rev Clin Psychol. 2012; 8: 21-48https://doi.org/10.1146/annurev-clinpsy-032511-143152
- Unaddressed privacy risks in accredited health and wellness apps: a cross-sectional systematic assessment.BMC Med. 2015; 13: 214https://doi.org/10.1186/s12916-015-0444-y
- System Identification—Theory for the User.2nd ed. PTR Prentice Hall, Upper Saddle River, NJ1999
- A dynamical systems approach to understanding self-regulation in smoking cessation behavior change.Nicotine Tob Res. 2014; 16: S159-S168https://doi.org/10.1093/ntr/ntt149
Martin CA, Deshpande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on Social Cognitive Theory. Paper presented at: American Control Conference (ACC), July 1-3, 2015; Chicago, IL.
- Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies.J Med Internet Res. 2015; 17: e166https://doi.org/10.2196/jmir.4391
- A trial of calcium and vitamin D for the prevention of colorectal adenomas.N Engl J Med. 2015; 373: 1519-1530https://doi.org/10.1056/NEJMoa1500409
- Hormone therapy and cardiovascular disease—are we back to the beginning?.Climacteric. 2015; 18: 437-438https://doi.org/10.3109/13697137.2015.1057958
- A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results.Trials. 2015; 16: 495https://doi.org/10.1186/s13063-015-1023-4
- External validity of randomised controlled trials: “to whom do the results of this trial apply?”.Lancet. 2005; 365: 82-93https://doi.org/10.1016/S0140-6736(04)17670-8
- Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies.Health Technol Assess. 2012; 16: 1-82https://doi.org/10.3310/hta16350
- Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide.BMJ. 2014; 348: g1687https://doi.org/10.1136/bmj.g1687
- Benefit-Risk Factors to Consider When Determining Substantial Equivalence in Premarket Notifications [510(k)] with Different Technological Characteristics Draft Guidance for Industry and Food and Drug Administration Staff.U.S. DHHS, Rockville, MD2014
- Negotiating the ‘grey area between normal social drinking and being a smelly tramp’: a qualitative study of people searching for help online to reduce their drinking.Health Expect. 2015; 18: 2011-2020https://doi.org/10.1111/hex.12351
- “Let’s get wasted!” and other apps: characteristics, acceptability, and use of alcohol-related smartphone applications.JMIR Mhealth Uhealth. 2013; 1: e9https://doi.org/10.2196/mhealth.2709
- CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions.J Med Internet Res. 2011; 13: e126https://doi.org/10.2196/jmir.1923
Article Info
Footnotes
This article is part of a theme section titled Digital Health: Leveraging New Technologies to Develop, Deploy, and Evaluate Behavior Change Interventions.
Identification
Copyright
© 2016 American Journal of Preventive Medicine. Published by Elsevier Inc. All rights reserved.
