Mobile Health Technology Evaluation

The mHealth Evidence Workshop


      Creative use of new mobile and wearable health information and sensing technologies (mHealth) has the potential to reduce the cost of health care and improve well-being in numerous ways. These applications are being developed in a variety of domains, but rigorous research is needed to examine the potential, as well as the challenges, of utilizing mobile technologies to improve health outcomes. Currently, evidence is sparse for the efficacy of mHealth. Although these technologies may be appealing and seemingly innocuous, research is needed to assess when, where, and for whom mHealth devices, apps, and systems are efficacious.
      In order to outline an approach to evidence generation in the field of mHealth that would ensure research is conducted on a rigorous empirical and theoretic foundation, on August 16, 2011, researchers gathered for the mHealth Evidence Workshop at NIH. The current paper presents the results of the workshop. Although the discussions at the meeting were cross-cutting, the areas covered can be categorized broadly into three areas: (1) evaluating assessments; (2) evaluating interventions; and (3) reshaping evidence generation using mHealth. This paper brings these concepts together to describe current evaluation standards, discuss future possibilities, and set a grand goal for the emerging field of mHealth research.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to American Journal of Preventive Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Krishna S.
        • Boren S.A.
        • Bales E.A.
        Healthcare via cell phones: a systematic review.
        Telemed J E Health. 2009; 15: 231-240
        • Patrick K.
        • Griswold W.
        • Raab F.
        • Intille S.S.
        Health and the mobile phone.
        Am J Prev Med. 2008; 35: 177-181
        • Riley W.T.
        • Rivera D.
        • Atienza A.
        • Nilsen W.
        • Allison S.M.
        • Mermelstein R.
        Health behavior models in the age of mobile interventions: are our theories up to the task?.
        Transl Behav Med. 2010; 1: 53-71
        • Quinn C.C.
        • Shardell M.
        • Terrin M.
        • Barr E.
        • Ballew S.
        • Gruber-Baldini A.L.
        • Cluster A.
        Randomized trial of a mobile phone personalized behavioral intervention for blood glucose control.
        Diabetes Care. 2011; 34: 1934-1942
      1. Gupta S, Chang P, Anyigbo N, Sabharwal A. mobileSpiro: accurate mobile spirometry for self-management of asthma. In Proceedings of ACM mHealthSys 2011; Article 1.

        • Bexelius C.
        • Lof M.
        • Sandin S.
        • Lagerros Y.T.
        • Forsum E.
        • Litton J.E.
        Measures of physical activity using cell phones: validation using criterion methods.
        J Med Internet Res. 2010; 12: e2
        • Patrick K.
        • Raab F.
        • Adams M.
        • et al.
        A text message-based intervention for weight loss: randomized controlled trial.
        J Med Internet Res. 2009; 11: 1-9
      2. Ali A, Hossain SM, Hovsepian K, Rahman M, Kumar S. SmokeTrack: automated detection of cigarette smoking in the mobile environment from respiration measurements. In Proceedings of ACM/IEEE Conference on Information Processing in Sensor Networks 2012;269–80.

      3. Plarre K, Raij AB, Hossain M, et al. Continuous inference of psychological stress from sensory measurements collected in the natural environment. In Proceedings of ACM/IEEE Conference on Information Processing in Sensor Networks 2011;97–108.

        • Burns M.N.
        • Begale M.
        • Duffecy J.
        • et al.
        Harnessing context sensing to develop a mobile intervention for depression.
        J Med Internet Res. 2011; 13: e55
        • D'eglise C.
        • Suggs L.S.
        • Odermatt P.
        Short message service (SMS) applications for disease prevention in developing countries.
        J Med Internet Res. 2012; 14: e3
        • Abroms L.C.
        • Padmanabhan N.
        • Thaweethai L.
        • Phillips T.
        iPhone apps for smoking cessation: a content analysis.
        Am J Prev Med. 2011; 40: 279-285
        • Barnhart H.X.
        • Haber M.J.
        • Lin L.I.
        An overview on assessing agreement with continuous measurements.
        J Biopharm Stat. 2007; 17: 529-569
        • Piantadosi S.
        Clinical trials: a methodologic perspective.
        Wiley, New York NY2005
        • Ioannidis J.P.A.
        Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials.
        JAMA. 1998; 279: 281-286
        • West S.G.
        Alternatives to randomized experiments.
        Curr Direct Psych Sci. 2009; 18: 299-304
        • Shadish W.R.
        • Cook T.D.
        • Campbell D.T.
        Experimental and quasi-experimental designs for generalized causal inference.
        Houghton Mifflin, Boston2002
        • Kravitz R.L.
        • Duan N.
        • Niedzinski E.I.
        • Hay M.C.
        • Subramanian S.K.
        • Weisner T.S.
        Whatever happened to n-of-1 trials? Insiders’ perspectives and a look to the future.
        Milbank Q. 2008; 86: 533-555
        • Brown C.
        • Lilford R.
        The stepped wedge trial design: a systematic review.
        BMC Med Res Methodol. 2006; 6: 54
        • Hussey M.A.
        • Hughes J.P.
        Design and analysis of stepped wedge cluster randomized trials.
        Contemp Clin Trials. 2007; 28: 182-191
        • Ratanawongsa N.
        • Handley M.
        • Quan J.
        • et al.
        Quasi-experimental trial of diabetes self-management automated and real-time telephonic support (SMARTSteps) in a Medicaid managed care plan: study protocol.
        BMC Health Services Res. 2012; 12: 22
      4. Mohr DC, Duan N. Continuous evaluation of evolving interventions. mHealth Evidence Workshop, August 16, 2011, Washington DC.

        • Collins L.M.
        • Murphy S.A.
        • Strecher V.
        The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART): new methods for more potent e-health interventions.
        Am J Prev Med. 2007; 32: S112-S118
      5. Lizotte DJ, Bowling M, Murphy SA. Efficient reinforcement learning with multiple reward functions for randomized controlled trial analysis. In Proceedings of Inter Conf Machine Learning (ICML) 2010;695–702.

        • Shortreed S.M.
        • Laber E.
        • Lizotte D.J.
        • Stroup T.S.
        • Pineau J.
        • Murphy S.A.
        Informing sequential clinical decision-making through reinforcement learning: an empirical study.
        Machine Learning. 2010; 84: 109-136
        • Mitka M.
        Strategies sought for reducing cost, improving efficiency of clinical research.
        JAMA. 2011; 306: 364-365
        • Shusterman V.
        • Goldberg A.
        • London B.
        Upsurge in T-wave alternans and nonalternating repolarization instability precedes spontaneous initiation of ventricular tachyarrhythmias in humans.
        Circulation. 2006; 113: 2880-2887
        • Hedeker D.
        • Mermelstein R.J.
        • Demirtas H.
        An application of a mixed-effects location scale model for analysis of ecological momentary assessment (EMA) data.
        Biometrics. 2008; 64: 627-634
        • Steyvers M.
        • Griffiths T.
        Probabilistic topic models.
        in: McNamara D. Dennis S. Kintsch W. Handbook of latent semantic analysis. Psychology Press, Oxford, UK2007
      6. Lafferty J, McCallum A, Pereira F. Conditional random fields: probabilistic models for segmenting and labeling sequence data. Proceedings of the 18th International Conference on Machine Learning 2001;282–89.

      7. Rabiner LR. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 1989;77(2):257–86.

        • Preece S.J.
        • Goulermas J.Y.
        • Kenney L.P.J.
        • et al.
        Activity identification using body-mounted sensors—a review of classification techniques.
        Physiol Meas. 2009; 30: R1-R33
      8. Rahman M, Ali AA, Plarre K, al'Absi M, Ertin E, Kumar S. mConverse: inferring conversation episodes from respiratory measurements collected in the field. In Proceedings of ACM Wireless Health 2011.

        • Mitchell H.B.
        Multi-sensor data fusion– an introduction. Springer-Verlag, Berlin2007
      9. Mustang M, Raij AB, Ganesan D, Kumar S, Shiffman S. Exploring micro-incentive strategies for participant compensation in high burden studies. In Proceedings of ACM UbiComp 2011;435–44.

      10. Estrin D, Sim I. Open mHealth., 2010.

        • Estrin D.
        • Sim I.
        Open mHealth architecture: an engine for health care innovation.
        Science. 2010; 330: 759-760
      11. Raij AB, Ghosh A, Kumar S, Srivastava MB. Privacy risks emerging from the adoption of innocuous wearable sensors in the mobile environment. In Proceedings of ACM CHI 2011;11–20.

        • Mare S.
        • Sorber J.
        • Shin M.
        • Cornelius C.
        • Kotz D.
        Adaptive security and privacy for mHealth sensing.
        In USENIX Workshop on Health Security (HealthSec). August, 2011;