Challenges in Assessing Mobile Health App Quality

A Systematic Review of Prevalent and Innovative Methods
Published:September 19, 2016DOI:


      Mobile apps are poised to become a major source of health guidance; however, concerns about their quality and safety remain. Governments, clinicians, and health researchers are struggling to determine how to adequately evaluate the content and function of mobile health apps to guide consumers toward apps that will effectively and safely support their health. Researchers are both adapting existing research methods, such as those for systematic reviews, and exploring novel methods toward this aim. The purpose of this systematic review was to describe prevalent and emerging methods for searching, data extraction, and analysis in the context of mobile health apps and commercial app stores.

      Evidence acquisition

      A systematic search was conducted on April 6, 2016, according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, for published studies that evaluated health-related apps targeted at consumers in a commercial app store, and a descriptive content analysis of their methods was performed.

      Evidence synthesis

      Although the majority of studies self-identified as “reviews” or “systematic reviews,” their claims to have performed an exhaustive, replicable, and systematic search and data extraction are unsupported, given the nature of commercial app stores’ search engines and personalized app content. Currently, studies of health-related app content are largely descriptive, and most use surrogate and one-dimensional outcome measures for app content quality.


      This field is rapidly developing, and innovative methods for systematic searches, multidimensional quality assessment, and contextualizing content assessment are emerging. Areas for future methodologic development are highlighted, including expanding definitions of app “content” and assessing risk of bias.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to American Journal of Preventive Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Huckvale K.
        • Prieto J.
        • Tilney M.
        • Benghozi P.-J.
        • Car J.
        Unaddressed privacy risks in accredited health and wellness apps: a cross-sectional systematic assessment.
        BMC Med. 2015; 13: 214
        • research2guidance
        mHealth App Developer Economics 2015.
        research2guidance, Berlin2015
        • U.S. Food and Drug Administration (FDA)
        Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff.
        U.S. FDA, Silver Spring, MD2015
      1. Therapeutic Goods Administration. Regulation of medical software and mobile medical “apps.” Published 2013. Accessed October 22, 2015.

        • European Commission
        Manual on Borderline and Classification in the Community Regulatory Framework for Medical Devices.
        European Commission, Brussels2015
        • de la Vega R.
        • Miro J.
        mHealth: a strategic field without a solid scientific soul. A systematic review of pain-related apps.
        PLoS One. 2014; 9: e101312
        • Buijink A.W.
        • Visser B.J.
        • Marshall L.
        Medical apps for smartphones: lack of evidence undermines quality and safety.
        Evid Based Med. 2013; 18: 90-92
        • Neubeck L.
        • Lowres N.
        • Benjamin E.J.
        • Freedman S.B.
        • Coorey G.
        • Redfern J.
        The mobile revolution—using smartphone apps to prevent cardiovascular disease.
        Nat Rev Cardiol. 2015; 12: 350-360
        • BinDhim N.F.
        • Hawkey A.
        • Trevena L.
        A systematic review of quality assessment methods for smartphone health apps.
        Telemed J E Health. 2015; 21: 97-104
      2. IDC Research Inc. Smartphone OS market share, 2015 Q2. Published 2015. Accessed October 16, 2015.

      3. Tan A. How to Choose the Best Platform for Your Mobile App. Published February 9, 2015. Accessed October 16, 2015.

      4. Apple Inc. iTunes Connect developer guide. Published 2015. Accessed December 14, 2015.

      5. Google Play. Launch checklist. Published 2015. Accessed December 14, 2015.

        • Huckvale K.
        • Adomaviciute S.
        • Prieto J.T.
        • Leow M.K.-S.
        • Car J.
        Smartphone apps for calculating insulin dose: a systematic assessment.
        BMC Med. 2015; 13: 1-10
      6. Google Play. Core app quality. Published 2015. Accessed December 14, 2015.

      7. Apple Inc. App review. Published 2015. Accessed December 14, 2015.

        • Stoyanov S.R.
        • Hides L.
        • Kavanagh D.J.
        • Zelenko O.
        • Tjondronegoro D.
        • Mani M.
        Mobile app rating scale: a new tool for assessing the quality of health mobile apps.
        JMIR mHealth uHealth. 2015; 3: e27
        • Direito A.
        • Dale L.P.
        • Shields E.
        • Dobson R.
        • Whittaker R.
        • Maddison R.
        Do physical activity and dietary smartphone applications incorporate evidence-based behaviour change techniques?.
        BMC Public Health. 2014; 14: 646
      8. Perez S. Apple’s app store gets a smarter search engine. TechCrunch. Published November 13, 2015. Accessed December 14, 2015.

      9. Rampton J. The beginner’s guide to app store optimization. Forbes Tech. Published October 16, 2015. Accessed December 14, 2015.

        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • Group T.P.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 62009: e1000097
      10. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration; 2011: Accessed April 3.

        • Wurzer P.
        • Parvizi D.
        • Lumenta D.B.
        • et al.
        Smartphone applications in burns.
        Burns. 2015; 41: 977-989
        • van Mechelen D.M.
        • van Mechelen W.
        • Verhagen E.A.
        Sports injury prevention in your pocket?! Prevention apps assessed against the available scientific evidence: a review.
        Br J Sports Med. 2014; 48: 878-882
        • Lister C.
        • West J.H.
        • Cannon B.
        • Sax T.
        • Brodegard D.
        Just a fad? Gamification in health and fitness apps.
        JMIR Serious Games. 2014; 2: e9
        • Mobasheri M.H.
        • Johnston M.
        • King D.
        • Leff D.
        • Thiruchelvam P.
        • Darzi A.
        Smartphone breast applications—what’s the evidence?.
        Breast. 2014; 23: 683-689
        • Knight E.
        • Stuckey M.I.
        • Prapavessis H.
        • Petrella R.J.
        Public health guidelines for physical activity: is there an app for that? A review of Android and Apple app stores.
        JMIR mHealth and uHealth. 2015; 3 (e43–e43)
        • Lalloo C.
        • Jibb L.A.
        • Rivera J.
        • Agarwal A.
        • Stinson J.N.
        “There’s a pain app for that”: Review of patient-targeted smartphone applications for pain management.
        Clin J Pain. 2015; 31: 557-563
        • Cowan L.T.
        • Van Wagenen S.A.
        • Brown B.A.
        • et al.
        Apps of steel: are exercise apps providing consumers with realistic expectations?: A content analysis of exercise apps for presence of behavior change theory.
        Health Educ Behav. 2013; 40: 133-139
        • Reynoldson C.
        • Stones C.
        • Allsop M.
        • et al.
        Assessing the quality and usability of smartphone apps for pain self-management.
        Pain Med. 2014; 15: 898-909
        • Williams J.P.
        • Schroeder D.
        Popular glucose tracking apps and use of mHealth by Latinos with diabetes: review.
        JMIR mHealth uHealth. 2015; 3
        • Basilico A.
        • Marceglia S.
        • Bonacina S.
        • Pinciroli F.
        Advising patients on selecting trustful apps for diabetes self-care.
        Comp Biol Med. 2016; 71: 86-96
        • Xu W.
        • Liu Y.
        mHealthApps: a repository and database of mobile health apps.
        JMIR mHealth uHealth. 2015; 3: e28
        • Bardus M.
        • van Beurden S.B.
        • Smith J.R.
        • Abraham C.
        A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management.
        Int J Behav Nutr Phys Act. 2016; 13: 35
        • Lee H.
        • Sullivan S.J.
        • Schneiders A.G.
        • et al.
        Smartphone and tablet apps for concussion road warriors (team clinicians): a systematic review for practical users.
        Br J Sports Med. 2015; 49: 499-505
        • Plaza I.
        • Demarzo M.M.P.
        • Herrera-Mercadal P.
        • García-Campayo J.
        Mindfulness-based mobile applications: literature review and analysis of current features.
        JMIR mHealth uHealth. 2013; 15
        • Huckvale K.
        • Morrison C.
        • Ouyang J.
        • Ghaghda A.
        • Car J.
        The evolution of mobile apps for asthma: an updated systematic assessment of content and tools.
        BMC Med. 2015; 13: 58
        • Ubhi H.K.
        • Kotz D.
        • Michie S.
        • et al.
        Comparative analysis of smoking cessation smartphone applications available in 2012 versus 2014.
        Addict Behav. 2016; 58: 175-181
        • Jutel A.
        • Lupton D.
        Digitizing diagnosis: a review of mobile applications in the diagnostic process.
        Diagnosis (Berl). 2015; 2: 89-96
        • Lupton D.
        • Jutel A.
        “It’s like having a physician in your pocket!” A critical analysis of self-diagnosis smartphone apps.
        Soc Sci Med. 2015; 133: 128-135
        • Zahry N.R.
        • Cheng Y.
        • Peng W.
        Content analysis of diet-related mobile apps: a self-regulation perspective.
        Health Comm. In press. Online. March 3, 2016;
        • Azar K.M.
        • Lesser L.I.
        • Laing B.Y.
        • et al.
        Mobile applications for weight management: theory-based content analysis.
        Am J Prev Med. 2013; 45: 583-589
        • Davis S.F.
        • Ellsworth M.A.
        • Payne H.E.
        • Hall S.M.
        • West J.H.
        • Nordhagen A.L.
        Health behavior theory in popular calorie counting apps: a content analysis.
        JMIR mHealth uHealth. 2016; 4: e19
        • Hoeppner B.B.
        • Hoeppner S.S.
        • Seaboyer L.
        • et al.
        How smart are smartphone apps for smoking cessation? A content analysis.
        Nicotine Tob Res. 2016; 18: 1025-1031
        • Hundert A.S.
        • Huguet A.
        • McGrath P.J.
        • Stinson J.N.
        • Wheaton M.
        Commercially available mobile phone headache diary apps: a systematic review.
        JMIR mHealth uHealth. 2014; 2: e36
        • Gan K.O.
        • Allman-Farinelli M.
        A scientific audit of smartphone applications for the management of obesity.
        Aust N Z J Public Health. 2011; 35: 293-294
        • Nicholas J.
        • Larsen M.E.
        • Proudfoot J.
        • Christensen H.
        Mobile apps for bipolar disorder: a systematic review of features and content quality.
        JMIR mHealth uHealth. 2015; 17: e198
        • Lupton D.
        Apps as artefacts: towards a critical perspective on mobile health and medical apps.
        Societies. 2014; 4: 606
        • Ouhbi S.
        • Luis Fernandez-Aleman J.
        • Rivera Pozo J.
        • El Bajta M.
        • Toval A.
        • Idri A.
        Compliance of blood donation apps with mobile OS usability guidelines.
        J Med Syst. 2015; 39: 63
        • Scott K.
        • Gome G.
        • Richards D.
        • Caldwell P.Y.
        How trustworthy are apps for maternal and child health?.
        Health Technol (Berl). 2015; 4: 329-336
        • App Annie
        App Annie Index Market Q1 2015.
        App Annie, San Francisco, CA2015
      11. Orland K. Pay to rank: gaming the app store in the age of Flappy Bird. Ars Technica. Published February 11, 2014. Accessed December 24, 2015.

        • Krieger W.H.
        Medical apps: public and academic perspectives.
        Perspect Biol Med. 2013; 56: 259-273