Measuring Preventive Care Delivery

Comparing Rates Across Three Data Sources


      Preventive care delivery is an important quality outcome, and electronic data reports are being used increasingly to track these services. It is highly informative when electronic data sources are compared to information manually extracted from medical charts to assess validity and completeness.


      This cross-sectional study used a random sample of Medicaid-insured patients seen at 43 community health centers in 2011 to calculate standard measures of correspondence between manual chart review and two automated sources (electronic health records [EHRs] and Medicaid claims), comparing documentation of orders for and receipt of ten preventive services (n=150 patients/service). Data were analyzed in 2015.


      Using manual chart review as the gold standard, automated EHR extraction showed near-perfect to perfect agreement (κ=0.96–1.0) for services received within the primary care setting (e.g., BMI, blood pressure). Receipt of breast and colorectal cancer screenings, services commonly referred out, showed moderate (κ=0.42) to substantial (κ=0.62) agreement, respectively. Automated EHR extraction showed near-perfect agreement (κ=0.83–0.97) for documentation of ordered services. Medicaid claims showed near-perfect agreement (κ=0.87) for hyperlipidemia and diabetes screening, and substantial agreement (κ=0.67–0.80) for receipt of breast, cervical, and colorectal cancer screenings, and influenza vaccination. Claims showed moderate agreement (κ=0.59) for chlamydia screening receipt. Medicaid claims did not capture ordered or unbilled services.


      Findings suggest that automated EHR and claims data provide valid sources for measuring receipt of most preventive services; however, ordered and unbilled services were primarily captured via EHR data and completed referrals were more often documented in claims data.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to American Journal of Preventive Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • DHHS, Centers for Medicare and Medicaid Services
        Medicare and Medicaid programs; electronic health record incentive program. Final rule.
        Fed Regist. 2010; 75: 44313-44588
      1. Center for Medicaid and Medicare Services. 2011–2012 Eligible Professional Clinical Quality Measures (CQMs). Published 2012. Accessed January 26, 2016.

        • National Committee for Quality Assurance (NCQA)
        HEDIS 2013: Technical Specifications for Physician Measurement.
        NCQA, Washington, DC2013
        • Weiner J.P.
        • Fowles J.B.
        • Chan K.S.
        New paradigms for measuring clinical performance using electronic health records.
        Int J Qual Health Care. 2012; 24: 200-205
        • Kern L.M.
        • Barron Y.
        • Dhopeshwarkar R.V.
        • Edwards A.
        • Kaushal R.
        Electronic health records and ambulatory quality of care.
        J Gen Intern Med. 2013; 28: 496-503
        • Lobach D.F.
        • Detmer D.E.
        Research challenges for electronic health records.
        Am J Prev Med. 2007; 32: S104-S111
        • Baer H.J.
        • Cho I.
        • Walmer R.A.
        • Bain P.A.
        • Bates D.W.
        Using electronic health records to address overweight and obesity: a systematic review.
        Am J Prev Med. 2013; 45: 494-500
        • Tinoco A.
        • Evans R.S.
        • Staes C.J.
        • Lloyd J.F.
        • Rothschild J.M.
        • Haug P.J.
        Comparison of computerized surveillance and manual chart review for adverse events.
        J Am Med Inform Assoc. 2011; 18: 491-497
        • Vinson D.R.
        • Morley J.E.
        • Huang J.
        • et al.
        The accuracy of an electronic pulmonary embolism severity index auto-populated from the electronic health record: setting the stage for computerized clinical decision support.
        Appl Clin Inform. 2015; 6: 318-333
        • Kern L.M.
        • Kaushal R.
        Accuracy of electronically reported “meaningful use” clinical quality measures.
        Ann Intern Med. 2013; 159: 73
        • Chan K.S.
        • Fowles J.B.
        • Weiner J.P.
        Review: electronic health records and the reliability and validity of quality measures: a review of the literature.
        Med Care Res Rev. 2010; 67: 503-527
        • Tang P.C.
        • Ralston M.
        • Arrigotti M.F.
        • Qureshi L.
        • Graham J.
        Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures.
        J Am Med Inform Assoc. 2007; 14: 10-15
        • Kerr E.A.
        • Smith D.M.
        • Hogan M.M.
        • et al.
        Comparing clinical automated, medical record, and hybrid data sources for diabetes quality measures.
        Jt Comm J Qual Improv. 2002; 28: 555-565
        • Baker D.W.
        • Persell S.D.
        • Thompson J.A.
        • et al.
        Automated review of electronic health records to assess quality of care for outpatients with heart failure.
        Ann Intern Med. 2007; 146: 270-277
        • Kmetik K.S.
        • O’Toole M.F.
        • Bossley H.
        • et al.
        Exceptions to outpatient quality measures for coronary artery disease in electronic health records.
        Ann Intern Med. 2011; 154: 227-234
        • Persell S.D.
        • Wright J.M.
        • Thompson J.A.
        • Kmetik K.S.
        • Baker D.W.
        Assessing the validity of national quality measures for coronary artery disease using an electronic health record.
        Arch Intern Med. 2006; 166: 2272-2277
        • Parsons A.
        • McCullough C.
        • Wang J.
        • Shih S.
        Validity of electronic health record-derived quality measurement for performance monitoring.
        J Am Med Inform Assoc. 2012; 19: 604-609
        • Agnew-Blais J.C.
        • Coblyn J.S.
        • Katz J.N.
        • Anderson R.J.
        • Mehta J.
        • Solomon D.H.
        Measuring quality of care for rheumatic diseases using an electronic medical record.
        Ann Rheum Dis. 2009; 68: 680-684
        • Benin A.L.
        • Vitkauskas G.
        • Thornquist E.
        • et al.
        Validity of using an electronic medical record for assessing quality of care in an outpatient setting.
        Med Care. 2005; 43: 691-698
        • Devoe J.E.
        • Gold R.
        • McIntire P.
        • Puro J.
        • Chauvie S.
        • Gallia C.A.
        Electronic health records vs Medicaid claims: completeness of diabetes preventive care data in community health centers.
        Ann Fam Med. 2011; 9: 351-358
        • MacLean C.H.
        • Louie R.
        • Shekelle P.G.
        • et al.
        Comparison of administrative data and medical records to measure the quality of medical care provided to vulnerable older patients.
        Med Care. 2006; 44: 141-148
        • Linder J.A.
        • Kaleba E.O.
        • Kmetik K.S.
        Using electronic health records to measure physician performance for acute conditions in primary care: empirical evaluation of the community-acquired pneumonia clinical quality measure set.
        Med Care. 2009; 47: 208-216
        • Rosenman M.B.
        • Tao G.
        • Szucs K.A.
        • Wang J.
        • Ambuehl R.
        • Mahon B.E.
        Prenatal syphilis screening rates measured using medicaid claims and electronic medical records.
        Sex Transm Dis. 2008; 35: 387-392
        • Heintzman J.
        • Bailey S.R.
        • Hoopes M.J.
        • et al.
        Agreement of Medicaid claims and electronic health records for assessing preventive care quality among adults.
        J Am Med Inform Assoc. 2014; 21: 720-724
        • Kottke T.E.
        • Baechler C.J.
        • Parker E.D.
        Accuracy of heart disease prevalence estimated from claims data compared with an electronic health record.
        Prev Chronic Dis. 2012; 9: E141
        • Solberg L.I.
        • Engebretson K.I.
        • Sperl-Hillen J.M.
        • Hroscikoski M.C.
        • O’Connor P.J.
        Are claims data accurate enough to identify patients for performance measures or quality improvement? The case of diabetes, heart disease, and depression.
        Am J Med Qual. 2006; 21: 238-245
        • Angier H.
        • Gold R.
        • Gallia C.
        • et al.
        Variation in outcomes of quality measurement by data source.
        Pediatrics. 2014; 133: e1676-e1682
        • Gold R.
        • Angier H.
        • Mangione-Smith R.
        • et al.
        Feasibility of evaluating the CHIPRA care quality measures in electronic health record data.
        Pediatrics. 2012; 130: 139-149
      2. Centers for Disease Control and Prevention. Summary Recommendations: Prevention and Control of Influenza with Vaccines: Recommendations of the Advisory Committee on Immunization Practices—(ACIP)—United States, 2013–14. Published 2013. Accessed January 26, 2016.

      3. U.S. Preventive Services Task Force. USPSTF A and B recommendations. Published 2014. Accessed January 26, 2016.

        • Cantor A.
        Sample-size calculations for Cohen’s kappa.
        Psychol Methods. 1996; 1: 150-153
      4. OCHIN. OCHIN 2012 Annual report. Published 2013. Accessed January 26, 2016.

        • Devoe J.E.
        • Sears A.
        The OCHIN community information network: bringing together community health centers, information technology, and data to support a patient-centered medical village.
        J Am Board Fam Med. 2013; 26: 271-278
        • Devoe J.E.
        • Gold R.
        • Spofford M.
        • et al.
        Developing a network of community health centers with a common electronic health record: description of the Safety Net West Practice-based Research Network (SNW-PBRN).
        J Am Board Fam Med. 2011; 24: 597-604
        • Harris P.A.
        • Taylor R.
        • Thielke R.
        • Payne J.
        • Gonzalez N.
        • Conde J.G.
        Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.
        J Biomed Inform. 2009; 42: 377-381
        • Morissette J.T.
        • Khorram S.
        Exact binomial confidence interval for proportions.
        Photogramm Eng Remote Sens. 1998; 64: 281-282
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Gao H.
        • Aiello Bowles E.J.
        • Carrell D.
        • Buist D.S.
        Using natural language processing to extract mammographic findings.
        J Biomed Inform. 2015; 54: 77-84
        • Denny J.C.
        • Choma N.N.
        • Peterson J.F.
        • et al.
        Natural language processing improves identification of colorectal cancer testing in the electronic medical record.
        Med Decis Making. 2012; 32: 188-197