Evaluating Digital Health Interventions

Key Questions and Approaches
      Digital health interventions have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety, and personalization. Achieving these improvements requires a cumulative knowledge base to inform development and deployment of digital health interventions. However, evaluations of digital health interventions present special challenges. This paper aims to examine these challenges and outline an evaluation strategy in terms of the research questions needed to appraise such interventions. As they are at the intersection of biomedical, behavioral, computing, and engineering research, methods drawn from all of these disciplines are required. Relevant research questions include defining the problem and the likely benefit of the digital health intervention, which in turn requires establishing the likely reach and uptake of the intervention, the causal model describing how the intervention will achieve its intended benefit, key components, and how they interact with one another, and estimating overall benefit in terms of effectiveness, cost effectiveness, and harms. Although RCTs are important for evaluation of effectiveness and cost effectiveness, they are best undertaken only when: (1) the intervention and its delivery package are stable; (2) these can be implemented with high fidelity; and (3) there is a reasonable likelihood that the overall benefits will be clinically meaningful (improved outcomes or equivalent outcomes at lower cost). Broadening the portfolio of research questions and evaluation methods will help with developing the necessary knowledge base to inform decisions on policy, practice, and research.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to American Journal of Preventive Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Free C.
        • Knight R.
        • Robertson S.
        • et al.
        Smoking cessation support delivered via mobile phone text messaging (txt2stop): a single-blind, randomised trial.
        Lancet. 2011; 378: 49-55
        • Harris J.
        • Felix L.
        • Miners A.
        • et al.
        Adaptive e-learning to improve dietary behaviour: a systematic review and cost-effectiveness analysis.
        Health Technol Assess. 2011; 15: 1-160
        • Vandelanotte C.
        • Spathonis K.M.
        • Eakin E.G.
        • Owen N.
        Website-delivered physical activity interventions a review of the literature.
        Am J Prev Med. 2007; 33: 54-64
        • Bailey J.V.
        • Murray E.
        • Rait G.
        • et al.
        Interactive computer-based interventions for sexual health promotion.
        Cochrane Database Syst Rev. 2010; 9: CD006483
        • Khadjesari Z.
        • Murray E.
        • Hewitt C.
        • Hartley S.
        • Godfrey C.
        Can stand-alone computer-based interventions reduce alcohol consumption? A systematic review.
        Addiction. 2011; 106: 267-282
        • Murray E.
        • Burns J.
        • See Tai S.
        • Lai R.
        • Nazareth I.
        Interactive Health Communication Applications for people with chronic disease.
        Cochrane Database Syst Rev. 2005; 4: CD004274
        • McLean G.
        • Band R.
        • Saunderson K.
        • et al.
        Digital interventions to promote self-management in adults with hypertension systematic review and meta-analysis.
        J Hypertens. 2016; 34: 600-612
        • Pal K.
        • Eastwood S.V.
        • Michie S.
        • et al.
        Computer-based diabetes self-management interventions for adults with type 2 diabetes mellitus.
        Cochrane Database Syst Rev. 2013; 3: CD008776
        • Hollis C.
        • Morriss R.
        • Martin J.
        • et al.
        Technological innovations in mental healthcare: harnessing the digital revolution.
        Br J Psychiatry. 2015; 206: 263-265
        • Arnberg F.K.
        • Linton S.J.
        • Hultcrantz M.
        • Heintz E.
        • Jonsson U.
        Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness.
        PLoS One. 2014; 9: e98118
        • Olthuis J.V.
        • Watt M.C.
        • Bailey K.
        • Hayden J.A.
        • Stewart S.H.
        Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults.
        Cochrane Database Syst Rev. 2015; 3: Cd011565
        • McCombie A.
        • Gearry R.
        • Andrews J.
        • Mikocka-Walus A.
        • Mulder R.
        Computerised cognitive behavioural therapy for psychological distress in patients with physical illnesses: a systematic review.
        J Clin Psychol Med Settings. 2015; 22: 20-44
        • van Beugen S.
        • Ferwerda M.
        • Hoeve D.
        • et al.
        Internet-based cognitive behavioral therapy for patients with chronic somatic conditions: a meta-analytic review.
        J Med Internet Res. 2014; 16: e88
        • Patrick K.
        • Hekler E.B.
        • Estrin D.
        • et al.
        Rapid rate of technological development and its implications for research on digital behavior interventions.
        Am J Prev Med. 2016; (In press)
        • Murray E.
        • Khadjesari Z.
        • White I.R.
        • et al.
        Methodological challenges in online trials.
        J Med Internet Res. 2009; 11: e9
      1. Michie S, West R. Guide to Development and Evaluation of Digital Behaviour Change Interventions in Healthcare. London: UCL Centre for Behaviour Change; In press

        • Titov N.
        • Dear B.F.
        • Staples L.G.
        • et al.
        MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression.
        Psychiatr Serv. 2015; 66: 1043-1050
        • Bower P.
        • Kontopantelis E.
        • Sutton A.
        • et al.
        Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data.
        BMJ. 2013; 346: f540
        • Hamilton F.L.
        • Hornby J.
        • Sheringham J.
        • et al.
        DIgital Alcohol Management ON Demand (DIAMOND) feasibility randomised controlled trial of a web-based intervention to reduce alcohol consumption in people with hazardous and harmful use versus a face-to-face intervention: protocol.
        Pilot Feasibility Stud. 2015; 1: 1-8
        • Dennison L.
        • Morrison L.
        • Lloyd S.
        • et al.
        Does brief telephone support improve engagement with a web-based weight management intervention? Randomized controlled trial.
        J Med Internet Res. 2014; 16: e95
        • Titov N.
        • Andrews G.
        • Davies M.
        • McIntyre K.
        • Robinson E.
        • Solley K.
        Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance.
        PLoS ONE. 2010; 5: e10939
        • Craig P.
        • Dieppe P.
        • Macintyre S.
        • Michie S.
        • Nazareth I.
        • Petticrew M.
        Developing and evaluating complex interventions: the new Medical Research Council guidance.
        BMJ. 2008; 337: a1655
        • Campbell N.C.
        • Murray E.
        • Darbyshire J.
        • et al.
        Designing and evaluating complex interventions to improve health care.
        BMJ. 2007; 334: 455-459
        • Murray E.
        • Treweek S.
        • Pope C.
        • et al.
        Normalisation process theory: a framework for developing, evaluating and implementing complex interventions.
        BMC Med. 2010; 8: 63
        • McNamee P.
        • Murray E.
        • Kelly M.
        • et al.
        Designing and undertaking a health economics study of digital health interventions.
        Am J Prev Med. 2016; (In press)
        • Agarwal R.
        • Anderson C.
        • Crowley K.
        • Kannan P.K.
        Understanding Development Methods From Other Industries to Improve the Design of Consumer Health IT: Background Report. (Prepared by Westat, under Contract No. HHSA290200900023I.) AHRQ Publication No. 11-0065-EF.
        Agency for Healthcare Research and Quality, Rockville, MD2011
        • Maguire M.
        Methods to support human-centred design.
        Int J Hum Comput Stud. 2001; 55: 587-634
        • Preece J.
        • Sharp H.
        • Rogers Y.
        Interaction Design-Beyond Human-Computer Interaction.
        John Wiley & Sons, New York2015
        • Buxton B.
        Sketching User Experiences: Getting the Design Right and the Right Design.
        Morgan Kaufmann, Burlington, MA2010
        • Detmer W.M.
        • Shiffman S.
        • Wyatt J.C.
        • Friedman C.P.
        • Lane C.D.
        • Fagan L.M.
        A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm.
        J Am Med Inform Assoc. 1995; 2: 46-57
        • Li Y.
        • Hong J.
        • Landay J.
        Design challenges and principles for Wizard of Oz testing of location-enhanced applications.
        IEEE Pervasive Comput. 2007; 6: 70-75
        • Bowen D.J.
        • Kreuter M.
        • Spring B.
        • et al.
        How we design feasibility studies.
        Am J Prev Med. 2009; 36: 452-457
      2. Hekler EB, Michie S, Rivera DE, et al. Developing and refining models and theories suitable for digital health interventions. Am J Prev Med. In press.

      3. Yardley L, Spring B, Riper H, et al. Understanding and promoting engagement with digital behavior change interventions. Am J Prev Med. In press.

        • Collins L.M.
        • Nahum-Shani I.
        • Almirall D.
        Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART).
        Clin Trials. 2014; 11: 426-434
        • Collins L.M.
        • Kugler K.C.
        • Gwadz M.V.
        Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of HIV/AIDS.
        AIDS Behav. 2016; 20: S197-S214
        • Wyrick D.L.
        • Rulison K.L.
        • Fearnow-Kenney M.
        • Milroy J.J.
        • Collins L.M.
        Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST).
        Transl Behav Med. 2014; 4: 252-259
        • Caldwell L.L.
        • Smith E.A.
        • Collins L.M.
        • et al.
        Translational research in South Africa: evaluating implementation quality using a factorial design.
        Child Youth Care Forum. 2012; 41: 119-136
        • Strecher V.J.
        • McClure J.B.
        • Alexander G.L.
        • et al.
        Web-based smoking-cessation programs: results of a randomized trial.
        Am J Prev Med. 2008; 34: 373-381
        • Collins L.M.
        • Dziak J.J.
        • Li R.
        Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs.
        Psychol Methods. 2009; 14: 202-224
        • Dziak J.J.
        • Nahum-Shani I.
        • Collins L.M.
        Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations.
        Psychol Methods. 2012; 17: 153-175
        • Almirall D.
        • Nahum-Shani I.
        • Sherwood N.E.
        • Murphy S.A.
        Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research.
        Transl Behav Med. 2014; 4: 260-274
        • Deshpande S.
        • Rivera D.E.
        • Younger J.W.
        • Nandola N.N.
        A control systems engineering approach for adaptive behavioral interventions: illustration with a fibromyalgia intervention.
        Transl Behav Med. 2014; 4: 275-289
        • Rivera D.E.
        • Pew M.D.
        • Collins L.M.
        Using engineering control principles to inform the design of adaptive interventions: a conceptual introduction.
        Drug Alcohol Depend. 2007; 88: S31-S40
        • Collins L.M.
        • Dziak J.J.
        • Kugler K.C.
        • Trail J.B.
        Factorial experiments: efficient tools for evaluation of intervention components.
        Am J Prev Med. 2014; 47: 498-504
        • Lei H.
        • Nahum-Shani I.
        • Lynch K.
        • Oslin D.
        • Murphy S.A.A.
        “SMART” design for building individualized treatment sequences.
        Annu Rev Clin Psychol. 2012; 8: 21-48
        • Huckvale K.
        • Prieto J.T.
        • Tilney M.
        • Benghozi P.J.
        • Car J.
        Unaddressed privacy risks in accredited health and wellness apps: a cross-sectional systematic assessment.
        BMC Med. 2015; 13: 214
        • Ljung L.
        System Identification—Theory for the User.
        2nd ed. PTR Prentice Hall, Upper Saddle River, NJ1999
        • Timms K.P.
        • Rivera D.E.
        • Collins L.M.
        • Piper M.E.
        A dynamical systems approach to understanding self-regulation in smoking cessation behavior change.
        Nicotine Tob Res. 2014; 16: S159-S168
      4. Martin CA, Deshpande S, Hekler EB, Rivera DE. A system identification approach for improving behavioral interventions based on Social Cognitive Theory. Paper presented at: American Control Conference (ACC), July 1-3, 2015; Chicago, IL.

        • Mohr D.C.
        • Schueller S.M.
        • Riley W.T.
        • et al.
        Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies.
        J Med Internet Res. 2015; 17: e166
        • Baron J.A.
        • Barry E.L.
        • Mott L.A.
        • et al.
        A trial of calcium and vitamin D for the prevention of colorectal adenomas.
        N Engl J Med. 2015; 373: 1519-1530
        • Fenton A.
        • Panay N.
        Hormone therapy and cardiovascular disease—are we back to the beginning?.
        Climacteric. 2015; 18: 437-438
        • Kennedy-Martin T.
        • Curtis S.
        • Faries D.
        • Robinson S.
        • Johnston J.
        A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results.
        Trials. 2015; 16: 495
        • Rothwell P.M.
        External validity of randomised controlled trials: “to whom do the results of this trial apply?”.
        Lancet. 2005; 365: 82-93
        • Savovic J.
        • Jones H.
        • Altman D.
        • et al.
        Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies.
        Health Technol Assess. 2012; 16: 1-82
        • Hoffmann T.C.
        • Glasziou P.P.
        • Boutron I.
        • et al.
        Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide.
        BMJ. 2014; 348: g1687
      5. Benefit-Risk Factors to Consider When Determining Substantial Equivalence in Premarket Notifications [510(k)] with Different Technological Characteristics Draft Guidance for Industry and Food and Drug Administration Staff.
        U.S. DHHS, Rockville, MD2014
        • Khadjesari Z.
        • Stevenson F.
        • Godfrey C.
        • Murray E.
        Negotiating the ‘grey area between normal social drinking and being a smelly tramp’: a qualitative study of people searching for help online to reduce their drinking.
        Health Expect. 2015; 18: 2011-2020
        • Weaver E.R.
        • Horyniak D.R.
        • Jenkinson R.
        • Dietze P.
        • Lim M.S.
        “Let’s get wasted!” and other apps: characteristics, acceptability, and use of alcohol-related smartphone applications.
        JMIR Mhealth Uhealth. 2013; 1: e9
        • Eysenbach G.
        • CONSORT−EHEALTH Group
        CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions.
        J Med Internet Res. 2011; 13: e126