Author + information
- Robert J. Mentz, MD∗ (, )
- L. Kristin Newby, MD, MHS,
- Ben Neely, MS,
- Joseph E. Lucas, PhD,
- Sean D. Pokorney, MD, MBA,
- Meena P. Rao, MD, MPH,
- Larry R. Jackson II, MD,
- Maria V. Grau-Sepulveda, MD, MPH,
- Michelle M. Smerek, BS,
- Pamela Barth, MRP,
- Charlotte L. Nelson, MS,
- Michael J. Pencina, PhD and
- Bimal R. Shah, MD, MBA
- ↵∗Duke Clinical Research Institute, 2301 Erwin Road, P.O. Box 17969, Durham, North Carolina 27715
To improve the efficiency of clinical research, there is interest in using electronic health record (EHR) data for outcome ascertainment (1). However, EHR data are designed for clinical care and billing and their utility for event capture is largely untested (2,3). We investigated 4 published computable phenotype definitions on the basis of administrative data for acute myocardial infarction (AMI) compared with a gold standard AMI classification created through chart review. The 4 algorithms were published by: 1) the Agency for Healthcare Research and Quality; 2) Centers for Medicare & Medicaid Services (CMS) Chronic Conditions Warehouse; 3) CMS Quality Net; and 4) Mini-Sentinel (Table 1).
For the present analysis, 150 encounters at Duke between January 2010 and July 2011 were selected for review via stratified random sampling on the basis of all 4 definitions. The population included Durham County residents age ≥18 years. After implementation of the algorithms, the population was stratified according to 1 of 3 criteria: 1) those encounters for which all 4 phenotypes indicated that an AMI occurred; 2) 1 to 3 phenotypes indicated that an AMI event occurred; and 3) all 4 phenotypes indicated no AMI occurred. The sample size was calculated on the basis of assumptions about positive predictive value and 1 − negative predictive value within each strata and was chosen to minimize the variance estimate of sensitivity (4). Cardiologist reviewer pairs independently assessed AMI events on the basis of the Third Universal Definition of Myocardial Infarction (UDMI) (5) through chart review. Reviewers were blinded to algorithm output. Disagreements were resolved by a third adjudicator. Following adjudication, the true AMI status was established for each encounter against which the administrative algorithms were evaluated. Inter-rater agreement was assessed using Fleiss’s kappa statistics. The sensitivity/specificity and 95% confidence intervals of each definition compared with chart review were calculated (4). Sensitivity/specificity estimates were computed on the basis of proportions in all records during the sampling period.
Of the 150 encounters, 70 (47%) were an AMI by the UDMI and 79 (53%) did not meet the UDMI definition; the specific encounter could not be located for 1 chart. Overall, the physician chart reviewers agreed on the assessment of AMI 100% of the time (kappa = 1) in the no positive phenotype stratum. There was 72% agreement in the strata in which 1 to 3 phenotypes were positive (kappa = 0.51), and 78% agreement in the strata in which all phenotypes were positive (kappa = 0.42). When assessing which components of the UDMI were most strongly associated with disagreements, we found that cardiac biomarker changes and symptoms of ischemia were most strongly associated with disagreements compared with electrocardiogram changes, imaging evidence, or cardiac death/autopsy findings of AMI.
Of the 70 AMIs by chart review, all algorithms were positive for 19 (27%) encounters, and 1 to 3 algorithms were positive for 51 (73%). Table 1 presents the sensitivity/specificity of each algorithm for AMI identification compared with the UDMI.
In summary, the CMS Chronic Conditions Warehouse was found to have the highest sensitivity and a high specificity via assessment of inpatient claims with the International Classification of Diseases, Ninth Revision codes of 410.X1 in the first or second position for discharge or final diagnosis. The high sensitivity supports its use to rule out an AMI in those with negative administrative data, and the high specificity rules in an AMI event in those with positive administrative data. Importantly, these findings were observed in a single-center analysis of Durham County residents. We did not specifically document details related to completeness of electrocardiography data or subsequent revascularization.
Importantly, prior analyses demonstrated that classification of nonfatal events in routinely-recorded EHR-type data is imperfect (2). EHRs are heterogeneous, and data elements are inconsistently defined (3). The complexity of the healthcare system can affect the utility of the EHR for outcome ascertainment, as patients receive care from multiple providers and institutions. An improved understanding of how to make administrative algorithms more accurate would facilitate outcomes research. Future research is needed to validate these observations and investigate other endpoints including revascularization, stroke, and heart failure events.
Please note: The projects and the work described in this research letter are supported in part by grant number 1C1CMS331018-01-00 from the Department of Health and Human Services, Centers for Medicare & Medicaid Services, and in part by the Bristol-Myers Squibb Foundation Together on Diabetes program, respectively. Dr. Mentz receives research support from the National Institutes of Health (U10HL110312). Dr. Newby has received research funding from PCORI and the National Institutes of Health; has received honoraria from JACC: Basic to Translational Science and the Journal of the American Heart Association; has served as a consultant to Roche Diagnostics, Philips Healthcare, Metanomics, Merck, Inc., and BioKier; has served on the advisory board of MedScape/theHeart.org; has performed research for Metanomics, Verily (formerly Google Life Sciences), GlaxoSmithKline, Amylin/Bristol-Myers Squibb, and Sanofi; and has served on the data and safety monitoring board of DemeRx.
Dr. Pokorney has received modest research or education grants from Gilead, Boston Scientific, and AstraZeneca; and has received modest consulting support from Boston Scientific and Medtronic. Dr. Rao has received a research grant from Medtronic. Dr. Shah is employed by Premier Inc. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- American College of Cardiology Foundation