Continuous Evaluation of Evolving Behavioral Intervention Technologies


      Behavioral intervention technologies (BITs) are web-based and mobile interventions intended to support patients and consumers in changing behaviors related to health, mental health, and well-being. BITs are provided to patients and consumers in clinical care settings and commercial marketplaces, frequently with little or no evaluation. Current evaluation methods, including RCTs and implementation studies, can require years to validate an intervention. This timeline is fundamentally incompatible with the BIT environment, where technology advancement and changes in consumer expectations occur quickly, necessitating rapidly evolving interventions. However, BITs can routinely and iteratively collect data in a planned and strategic manner and generate evidence through systematic prospective analyses, thereby creating a system that can “learn.”
      A methodologic framework, Continuous Evaluation of Evolving Behavioral Intervention Technologies (CEEBIT), is proposed that can support the evaluation of multiple BITs or evolving versions, eliminating those that demonstrate poorer outcomes, while allowing new BITs to be entered at any time. CEEBIT could be used to ensure the effectiveness of BITs provided through deployment platforms in clinical care organizations or BIT marketplaces. The features of CEEBIT are described, including criteria for the determination of inferiority, determination of BIT inclusion, methods of assigning consumers to BITs, definition of outcomes, and evaluation of the usefulness of the system. CEEBIT offers the potential to collapse initial evaluation and postmarketing surveillance, providing ongoing assurance of safety and efficacy to patients and consumers, payers, and policymakers.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to American Journal of Preventive Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Mohr D.C.
        • Burns M.N.
        • Schueller S.M.
        • Clarke G.
        • Klinkman M.
        Behavioral intervention technologies: evidence review and recommendations for future research in mental health.
        Gen Hosp Psychiatry. 2013; 35: 332-338
        • Oh H.
        • Rizo C.
        • Enkin M.
        • Jadad A.
        What is eHealth (3): a systematic review of published definitions.
        J Med Internet Res. 2005; 7: e1
        • Fogg B.J.
        • Hreha J.
        Behavior wizard: a method for matching target behaviors with solutions.
        Persuasive Technol Proc. 2010; 6137: 117-131
        • Tomlinson M.
        • Rotheram-Borus M.J.
        • Swartz L.
        • Tsai A.C.
        Scaling up mHealth: where is the evidence?.
        PLoS Med. 2013; 10: e1001382
      1. Houston TK, Frisbee K, Moody R, et al. Implementation of Veterans Affairs (VA) Mobile Health (Panel). Medicine 2.0. Boston, MA, 2012.

      2. research2guidance. Mobile health market report 2013-2017: the commercialization of mHealth applications (Vol. 3). Berlin: research2guidance, 2013.

        • Abroms L.C.
        • Padmanabhan N.
        • Thaweethai L.
        • Phillips T.
        iPhone apps for smoking cessation: a content analysis.
        Am J Prev Med. 2011; 40: 279-285
      3. U.S. Food and Drug Administration. Draft guidance for industry and Food and Drug Administration staff—mobile medical applications.

        • Chambless D.L.
        • Hollon S.D.
        Defining empirically supported therapies.
        J Consult Clin Psychol. 1998; 66: 7-18
        • Rounsaville B.J.
        • Carroll K.M.
        • Onken L.S.
        A stage model of behavioral therapies research: getting started and moving on from stage I.
        Clin Psychol Science Pract. 2001; 8: 133-142
        • Balas E.A.
        • Boren S.A.
        Managing clinical knowledge for healthcare improvements. In:.
        in: van Bemmel J.H. McCray A.T. Yearbook of medical informatics 2000. Schattauer, Stuttgart2000: 65-70
        • Glasgow R.E.
        • Lichtenstein E.
        • Marcus A.C.
        Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition.
        Am J Public Health. 2003; 93: 1261-1267
        • Brown C.H.
        • Kellam S.G.
        • Kaupert S.
        • et al.
        Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: experiences of the prevention science and methodology group.
        Adm Policy Ment Health. 2012; 39: 301-316
        • Christensen C.M.
        • Johnson C.W.
        • Horn M.B.
        Disrupting class: how disruptive innovation will change the way the world learns.
        McGraw Hill, New York2008
      4. Kumar S, Nilsen W, Abernethy A, et al. Mobile health technology evaluation: the Mhealth evidence workshop. Am J Prev Med 45(2):228–36.

      5. Duan N, Cheung YK. Design of implementation studies for quality improvement programs: an effectiveness/cost-effectiveness framework. Am J Pub Health 2013 (In Press).

        • Levin B.
        • Robbins H.
        Selecting the highest probability in binomial or multinomial trials.
        Proc Natl Acad Sci U S A. 1981; 78: 4663-4666
        • Cheung Y.K.
        • Gordon P.H.
        • Levin B.
        Selecting promising ALS therapies in clinical trials.
        Neurology. 2006; 67: 1748-1751
        • Christensen H.
        • Griffiths K.M.
        • Farrer L.
        Adherence in internet interventions for anxiety and depression.
        J Med Internet Res. 2009; 11: e13
        • Donkin L.
        • Christensen H.
        • Naismith S.L.
        • Neal B.
        • Hickie I.B.
        • Glozier N.
        A systematic review of the impact of adherence on the effectiveness of e-therapies.
        J Med Internet Res. 2011; 13: e52
        • Guadagnoli E.
        • Ward P.
        Patient participation in decision-making.
        Soc Sci Med. 1998; 47: 329-339
        • Brewin C.R.
        • Bradley C.
        Patient preferences and randomised clinical trials.
        BMJ. 1989; 299: 313-315
        • Lavori P.W.
        • Rush A.J.
        • Wisniewski S.R.
        • et al.
        Strengthening clinical effectiveness trials: equipoise-stratified randomization.
        Biol Psychiatry. 2001; 50: 792-801
        • Stroup T.S.
        • McEvoy J.P.
        • Swartz M.S.
        • et al.
        The National Institute of Mental Health Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) project: schizophrenia trial design and protocol development.
        Schizophr Bull. 2003; 29: 15-31
        • Thompson W.R.
        On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.
        Biometrika. 1933; 25: 285-294
        • Wei L.J.
        • Durham S.
        Randomized play—winner rule in medical trials.
        J Am Stat Assoc. 1978; 73: 840-843
        • Cheung Y.K.
        Simple sequential boundaries for treatment selection in multi-armed randomized clinical trials with a control.
        Biometrics. 2008; 64: 940-949
        • Cheung Y.K.
        • Inoue L.Y.
        • Wathen J.K.
        • Thall P.F.
        Continuous Bayesian adaptive randomization based on event times with covariates.
        Stat Med. 2006; 25: 55-70
        • Rosenbaum P.R.
        Observational studies.
        Springer, New York NY2002
        • Rosenbaum P.R.
        • Rubin D.B.
        The central role of the propensity score in observational studies for causal effects.
        Biometrika. 1983; 70: 41-55
        • Rosenbaum P.R.
        • Rubin D.B.
        Reducing bias in observational studies using subclassification on the propensity score.
        J Am Stat Assoc. 1984; 79: 516-524
        • Finkelstein M.O.
        • Levin B.
        • Robbins H.
        Clinical and prophylactic trials with assured new treatment for those at greater risk: II. Examples.
        Am J. Public Health. 1996; 86: 696-705
        • Finkelstein M.O.
        • Levin B.
        • Robbins H.
        Clinical and prophylactic trials with assured new treatment for those at greater risk: I. A design proposal.
        Am J Public Health. 1996; 86: 691-695
        • West S.G.
        • Duan N.
        • Pequegnat W.
        • et al.
        Alternatives to the randomized controlled trial.
        Am J Public Health. 2008; 98: 1359-1366
      6. Smith A. Mobile access 2010: Pew Internet and the American Life Project; 2010.