Author + information
- Received October 19, 2009
- Revision received February 8, 2010
- Accepted February 9, 2010
- Published online May 4, 2010.
- Eric D. Peterson, MD, MPH⁎,⁎ (, )
- David Dai, PhD⁎,
- Elizabeth R. DeLong, PhD⁎,
- J. Matthew Brennan, MD⁎,
- Mandeep Singh, MD†,
- Sunil V. Rao, MD⁎,
- Richard E. Shaw, PhD‡,
- Matthew T. Roe, MD, MHS⁎,
- Kalon K.L. Ho, MD∥,
- Lloyd W. Klein, MD¶,
- Ronald J. Krone, MD#,
- William S. Weintraub, MD††,
- Ralph G. Brindis, MD, MPH§,
- John S. Rumsfeld, MD, PhD‡‡,
- John A. Spertus, MD, MPH⁎⁎,
- NCDR Registry Participants
- ↵⁎Reprint requests and correspondence:
Dr. Eric D. Peterson, Duke Clinical Research Institute, 2400 Pratt Street, Room 7009, North Pavilion DUMC, Durham, North Carolina 27715
Objectives We sought to create contemporary models for predicting mortality risk following percutaneous coronary intervention (PCI).
Background There is a need to identify PCI risk factors and accurately quantify procedural risks to facilitate comparative effectiveness research, provider comparisons, and informed patient decision making.
Methods Data from 181,775 procedures performed from January 2004 to March 2006 were used to develop risk models based on pre-procedural and/or angiographic factors using logistic regression. These models were independently evaluated in 2 validation cohorts: contemporary (n = 121,183, January 2004 to March 2006) and prospective (n = 285,440, March 2006 to March 2007).
Results Overall, PCI in-hospital mortality was 1.27%, ranging from 0.65% in elective PCI to 4.81% in ST-segment elevation myocardial infarction patients. Multiple pre-procedural clinical factors were significantly associated with in-hospital mortality. Angiographic variables provided only modest incremental information to pre-procedural risk assessments. The overall National Cardiovascular Data Registry (NCDR) model, as well as a simplified NCDR risk score (based on 8 key pre-procedure factors), had excellent discrimination (c-index: 0.93 and 0.91, respectively). Discrimination and calibration of both risk tools were retained among specific patient subgroups, in the validation samples, and when used to estimate 30-day mortality rates among Medicare patients.
Conclusions Risks for early mortality following PCI can be accurately predicted in contemporary practice. Incorporation of such risk tools should facilitate research, clinical decisions, and policy applications.
Percutaneous coronary intervention (PCI) has become one of the most widely applied treatments in current-day cardiology, facilitating the relief of angina and (in the setting of acute ST-segment elevation myocardial infarction [STEMI]), saving lives (1). Although the periprocedural complications of PCI have declined over time, tangible risks remain. Estimating patients' PCI mortality risk is important for several reasons. At the individual-patient level, knowing one's procedural risk can help physicians and patients make informed clinical decisions (2). Identification and quantification of clinical factors associated with procedural risk can also facilitate observational comparative effectiveness research (3). Finally, at a policy level, predicted risk estimates can help “level the playing field” of provider outcome metrics, helping to adjust for potential differences in cases treated (4).
To date, several PCI mortality risk models have been published. Yet many have become outdated and do not reflect contemporary care or outcomes (5–13). Other risk models were developed on select populations and may not be generalizable (7–9,11,14–19). Additionally, many models failed to consider angiographic features that are associated with procedural risk (9,20,21). The National Cardiovascular Data Registry (NCDR) for catheterization percutaneous coronary intervention (CathPCI) provides the ideal infrastructure to derive procedure risk models in a national representative contemporary U.S. sample. This database has a very large patient population, contains rich and complete clinical information, and is reflective of contemporary practice.
Using the NCDR CathPCI database, our goals were to: 1) develop PCI risk tools for estimating mortality risks for both elective and primary PCI; 2) determine the incremental prognostic value of angiographic details beyond pre-procedural risk factors; 3) develop a simplified, user-friendly, PCI risk score; 4) internally validate the PCI risk model and risk score in important subpopulations; and 5) assess the models' ability to estimate 30-day PCI mortality risk among Medicare patients whose status is defined via claims data.
The NCDR CathPCI Registry database
The NCDR CathPCI Registry is cosponsored by the American College of Cardiology and the Society for Cardiovascular Angiography and Interventions (22,23). The registry catalogs data on patient characteristics, clinical features, angiographic and procedural details, and in-hospital outcomes. Participating centers agree to submit complete information and outcomes from consecutive interventional cases performed at their institutions. The NCDR also has a comprehensive data quality program, including data abstraction training, data quality thresholds for inclusion, site data quality feedback reports, independent auditing, and data validation (22). Data elements and definitions are available at: http://www.ncdr.com/WebNCDR/ELEMENTS.ASPX#1. The Duke Clinical Research Institute (DCRI) serves as the primary analytic center for the CathPCI Registry, and performed the analyses for this report.
The NCDR established a risk adjustment model committee of American College of Cardiology volunteers to provide oversight for model development, including input on candidate variable selection and review of the model results. This group strictly adhered to current standards of model creation (24). The outcome of interest for these models was all-cause in-hospital mortality. Candidate variables were selected based on their relevance, as identified in prior research, or as identified in the committee's clinical experience.
The rates of overall missing data in the NCDR CathPCI database are very low. Of the final model variables, only ejection fraction (EF) percentage had more than a 5% rate of missing data. For those few cases that contained missing information, the following imputation rules were used: 1) for elements dealing with a patient's past medical history, use of a pre-procedural intra-aortic balloon pump, presence of subacute thrombosis, and coronary lesion with highest risk lesion, missing data were imputed to “no”; 2) for body mass index (BMI), missing values were imputed to the gender-specific median; 3) for glomerular filtration rate (GFR), missing values were imputed to the gender-, prior renal failure-, and STEMI-specific median; and 4) for EF, missing data were imputed by stratifying the population based on a history of congestive heart failure (CHF), prior myocardial infarction, pre-procedural cardiogenic shock, and the presence of STEMI. Neither age nor the Society for Cardiovascular and Angiography and Interventions Lesion Class were imputed. We also performed a sensitivity analysis using multiple imputation methods. However, these results were nearly identical to the overall findings and are, therefore, not presented.
Two separate patient populations were identified: one for model development and one for prospective validation. For the model development phase, patients were included if they received their first PCI procedure at any of the 470 hospitals submitting PCI records between January 1, 2004, and March 30, 2006. Patients were excluded if they transferred out or were missing more than 2 candidate variables (Fig. 1).The model development population was further randomly allocated to an initial model development dataset (60% of total), and a second group (40% of total) was used for an initial validation sample. A second prospective validation sample was identified from cases performed at the 608 NCDR hospitals submitting PCI cases between March 31, 2006, and March 30, 2007, using the same inclusion and exclusion criteria as noted in the previous text (Fig. 1).
Additionally, we examined the robustness of our models to predict 30-day mortality, as opposed to in-hospital mortality, in a Medicare-eligible population (25). Since outcomes beyond the initial hospital stay are not routinely collected in the NCDR, we linked NCDR records for those age 65 years or older to the national Centers for Medicare and Medicaid Services (CMS) inpatient claims data. The process used to do this has been previously described (26). For this specific linkage to occur, we began with Medicare-eligible NCDR CathPCI patients undergoing a PCI procedure between January 2005 and December 2006 (the last available data from CMS). Of the possible 348,370 records, we linked 253,081 records (72.7%), representing 204,111 unique patients. Baseline characteristics of the linked population and unlinked records were similar.
An initial candidate variable list was generated using clinical judgment and prior known PCI risk factors. Univariate analysis was then used to identify which of the potential candidate variables had a statistical association with in-hospital mortality (e.g., p < 0.05). Based on this univariate analysis, the risk adjustment model committee selected the most clinically meaningful variables as potential candidates for inclusion in the multivariable model. Multivariate logistic regression with a backward selection method (p < 0.05 to remain in the model) was then performed to identify independent predictors of outcomes.
Three separate models were developed. First, a “full” model was created, which included all candidate variables (e.g., demographic, pre-catheterization clinical variables, and angiographic variables). Second, we contrasted this full model with a second “pre-cath” model, excluding detailed NCDR angiographic data. This second model assessed the incremental value of angiographic information for mortality prediction. Finally, we developed a “limited” pre-cath risk prediction model, which included only those variables with the strongest explanatory power based on their Wald chi-square value. The regression coefficients from the simplified pre-cath model were then converted into whole integers to create an NCDR CathPCI Risk Prediction score (27).
Model performance characteristics
After development, we applied these 3 models to the prospective validation sample sets. Model discrimination was assessed using the c-index. A model c-index can range from 0.50 (no predictive value) to 1.0 (perfect prediction). To assess model calibration, patients were rank-ordered from lowest- to highest-predicted risk. Comparison was then made of predicted versus observed event rates within risk strata. Model discrimination and calibration were assessed in the overall population, within the 2 validation samples, and among select subpopulations of both of these groups. Finally, we assessed the models' discrimination among patients age 65+ years who had been linked to CMS data to assess both in-hospital and 30-day mortality.
Between January 2004 and March 2007, 600,533 consecutive PCI admissions were recorded in the NCDR CathPCI Registry. Following exclusions, 588,398 total patients were included in our overall model development and validation cohort. From this population, a model development sample (n = 181,775) was created from a random sample comprised of two-thirds the cases performed between January 2004 and March 2006. The final one-third of these cases was used to create a contemporary model validation sample (n = 121,183). Cases performed between March 2006 and March 2007 were used as a prospective validation sample (n = 285,440) (Fig. 1).
Table 1provides demographic, clinical, and angiographic features of those patients in the development set, as well as in the 2 validation sets. The mean patient age was 64 years, 33% were female, 32% had diabetes mellitus, and 10% had a prior history of CHF. Overall, 51% of the patients underwent nonelective procedures, and 14% underwent multivessel PCI. The results were similar across the 3 samples, except that in-hospital mortality was slightly lower in the second prospective validation sample (1.17%), relative to the other 2 samples (1.24% and 1.27%).
Risk factors for in-hospital mortality
Table 2provides observed in-hospital mortality rates for various patient subgroups. These mortality rates ranged from 0.65% in the non-primary PCI population to 4.81% in the STEMI population (Table 2). Older patients, women, and diabetic patients experienced higher unadjusted in-hospital mortality rates than younger patients, men, and non-diabetic patients (2.25% vs. 0.76%, 1.63% vs. 1.04%, and 1.44% vs. 1.15%, respectively).
Table 3provides the final full model, which includes 21 separate clinical variables, as well as interaction terms for STEMI/shock, BMI, GFR, dialysis, New York Heart Association (NYHA) functional class, highest-risk lesion segment category, and PCI status. When model chi-square value was used as the metric, cardiogenic shock was the most predictive of in-hospital mortality, followed by renal function (estimated glomerular filtration rate [eGFR]) and age. In contrast, angiographic predictors were generally less prognostic. The angiographic feature most highly associated with in-hospital mortality was lesion location (e.g., left main lesions and proximal left anterior descending lesions).
NCDR PCI bedside risk prediction score
Predictors containing the strongest association with mortality are described in Table 3. These risk factors were then converted to an integer score (based on their relative magnitude of association with mortality), to create the NCDR CathPCI Risk Prediction Score (Table 4).Using this scoring system, the risk of in-hospital mortality can be estimated by summating point scores between 0 and 100.
The full NCDR CathPCI Mortality Risk Prediction model in the contemporary and prospective validation cohorts performed exceptionally well, with a c-index of 0.925 and 0.924, respectively. Additionally, the full model performed well in each of the 8 predefined patient subgroups, with c-indices ranging from 0.892 to 0.930 (Table 5).Of note, the exclusion of angiographic details and EF from the full model resulted in only a slight decrement in the overall model accuracy. Similarly, there was limited loss in model discrimination when the model was transformed into the final, simplified NCDR CathPCI Risk Score, with c-indices of 0.901 and 0.905, respectively, in the validation samples. This simplified score also had good operating characteristics in all predefined patient subgroups.
Model calibration plots are shown in Figures 2 and 3.⇓⇓Notably, the majority of patients had a relatively low mortality risk (92.6% of patients had a predicted mortality risk between 0% and 2.5%). However, there was high concordance between model predicted risk and that which was actually observed. The simplified NCDR CathPCI Risk Score was also well calibrated in both low- and moderate-risk populations, with only a slight underestimation of predicted risk in high-risk patients (Fig. 3).
Finally, we examined the full and simplified models' ability to estimate 30-day mortality among patients age 65 years or older who had been linked to CMS data. Among 204,111 Medicare patients, 4,068 (1.99%) died in-hospital and 6,011 (2.94%) died within 30 days of the procedure. C-indices for the full model in this population were: c= 0.90 for in-hospital and c= 0.86 for 30-day mortality, respectively. C-indices for the Simplified Risk Score in this population were: c= 0.89 for in-hospital and c= 0.83 for 30-day mortality, respectively.
Despite tremendous advances in PCI over the past decade, early periprocedural mortality remains a concern. Using data from the NDCR, we identified demographics, clinical factors, and angiographic features associated with PCI in-hospital mortality. These were summarized into a full risk model (with both pre-procedure and angiographic features) and a simplified 8-item NCDR CathPCI Risk Score, to support both robust hospital outcome comparisons and patient-level pre-procedural risk estimation, respectively. Both the full and simplified models retain their predictive accuracy in important patient subsets, in separate internal validation samples, and when estimating 30-day mortality in Medicare patients.
Several risk-adjustment models have been previously developed for the prediction of mortality following PCI. However, many of these were developed using data that predates the generalized use of stents and/or contemporary adjuvant antithrombotic therapy (5–13). Other models have been developed from select referral centers or regional populations and may not be as generalizable across the nation (7–9,11,14–19). Still, other models were developed using databases that included only elderly patients, or used administrative data which lacked the clinical details necessary to capture the important clinical and angiographic risks factors associated with periprocedural mortality (9,20,21).
The models derived in this study expand on these prior models. First, the comprehensive and complete nature of the NCDR's clinical data allows for a more complete assessment of multiple risk predictors. For example, female sex has long been a feature predictive in many prior studies, yet this feature is no longer significantly associated with mortality after adjusting for multiple potential confounders (e.g., BMI, eGFR, and so on) and in the contemporary populations (28,29). Additionally, we have demonstrated that the inclusion of angiographic details (as they are defined in the NCDR CathPCI Registry) to a pre-cath risk prediction model, add marginal overall improvements in our ability to predict in-hospital mortality. Rather, in-hospital mortality was driven primarily by pre-existing patient comorbidities and markers of clinical instability. This finding is consistent with the work of others (16) and has important clinical implications in that it allows patients and physicians to obtain a reasonable estimate of procedural risk, prior to angiography.
In the aggregate population, angiographic risk factors added modest value, whereas in individual cases, their impact was more substantial. For example, the mean predicted PCI risk for patients with left main stenosis was 4.5% versus 2.4%, depending on whether or not the prediction included the angiographic left main risk feature. Other risk scores (such as the SYNTAX score), which arguably focus more heavily on collecting exhaustive angiographic data, have found some additional benefit from these angiographic variables (30).
We also found that patients presenting for PCI in the setting of STEMI, faced substantially higher procedural risk. However, the scope and relative impact of risk factors needed to predict risk in acute versus elective cases, were quite similar. Based on this observation, we were able to develop a unified model of risk estimation for all PCI cases, as opposed to separate STEMI and elective models. This unified model (e.g., the simplified NCDR PCI Mortality Risk Score) accurately predicts mortality in both acute and elective cases.
Utility of risk models
The NCDR CathPCI risk prediction tools developed and validated in this analysis cover the broad spectrum of anticipated model uses and address the needs of researchers, administrators, physicians, and patients. The full NCDR model provides a comprehensive tool to: 1) permit the most accurate adjustment of both pre-procedural and angiographic features for research projects; and 2) “level the playing field” for provider-level mortality results comparisons. Yet the full model is complex, inclusive of multiple data elements, spline-transformed continuous variables, and interaction terms—thus, the model is not practical to estimate patients' individual risk without computer assistance. Therefore, we also created the NCDR CathPCI Risk Score, whose simplified 8-item additive risk score can be used for bedside risk estimation.
Participation in the NCDR CathPCI centers is voluntarily and slightly under-represents smaller clinical practices. That said, the NCDR CathPCI Registry remains the largest, most generalizable U.S. data source. In-patient mortality, rather than 30-day mortality, has limitations as an end point (31). However, at the provider level, in-hospital and 30-day mortality results are highly correlated. Additionally, the only source of complete 30-day outcomes is Medicare data, which do not capture outcomes in those <65 years of age. When our models were applied to predict 30-day mortality in the Medicare population, they retained good discrimination (c= 0.86).
As the practice of medicine continues to evolve, so will the use of risk prediction models. Clinically, computer-generated risk scores are being used to aid in the personalization of the procedural consent process (2). Although mortality is clearly an important outcome, modeling other modifiable outcomes, such as myocardial infarction, renal failure, bleeding complications, restenosis, stent thrombosis, and angina relief, could further advance the Institute of Medicine's goals for evidence-based, patient-centered, medical care (2,32). As advanced procedural support devices (e.g., hemodynamic support devices) continue to develop, risk prediction tools can be utilized to more clearly define the patient populations in which they will be maximally effective. From an administrative standpoint, the importance of these tools for provider-based risk-adjusted outcomes comparisons will continue to increase, as public reporting and pay-for-performance initiatives continue to grow in popularity. Finally, from a research perspective, these risk tools will be used to mitigate treatment selection bias when conducting comparative effectiveness analyses in observational data.
Using data from the NCDR CathPCI Registry, we have developed and validated contemporary models for assessing periprocedural PCI mortality risk. Each of these has excellent predictive accuracy throughout the full spectrum of patient risk, and important patient subgroups. We anticipate that these models will have multiple applications (including bedside risk estimation using the simplified risk score, comparison of hospital performance, and risk adjustment).
Dr. Peterson has received research grants from Bristol-Myers Squibb, Merck Group, Sanofi-Aventis, St. Jude Medical, Inc., American Heart Association, American College of Cardiology, and Society of Thoracic Surgeons; and is a consultant for Kansas City Cardiomyopathy Questionnaire, Peripheral Artery Questionnaire, PRISM, and Seattle Angina Questionnaire. Bristol-Myers Squibb, Centocor, Eli Lilly & Company, and Pfizer. Dr. Rao has received research funding from Cordis Corporation(modest). Dr. Roe has received research grants from Bristol-Myers Squibb, Eli Lilly & Company, Portola Pharmaceutical, Sanofi-Aventis, and Schering-Plough Corporation; and is a consultant for Adolor Corporation, AstraZeneca, Bristol-Myers Squibb, Daiichi Sankyo, Eli Lilly & Company, Merck Group, Novartis Pharmaceutical Company, Sanofi-Aventis, and Schering-Plough Corporation. Dr. Ho serves on an independent clinical events committee for Harvard Clinical Research Institute, providing event adjudication services for Boston Scientific Corporation; and is a clinical advisor for Massachusetts Data Analysis Center under contract to the Department of Public Health for the Commonwealth of Massachusetts (an unpaid volunteer position). Dr. Rumsfeld is Chief Science Officer for the National Cardiovascular Data Registry. Dr. Spertus has a research contract from ACCF; grants from AHA, Amgen, Johnson & Johnson, and NIH; grant support from Atherotech and Roche Diagnostics; is a consultant for Novartis, United Healthcare, and St. Jude Medical; and holds copyrights/patents for Kansas City Cardiomyopathy Questionnaire, Peripheral Artery Questionnaire, PRISM, and Seattle Angina Questionnaire.
The authors would like to acknowledge Erin LoFrese for her editorial contributions to this manuscript.
This project was supported by grant number U18HS016964from the Agency for Healthcare Research and Quality (AHRQ). The content is solely the responsibility of the authors and does not necessarily represent the official views of the AHRQ. The funding source had no role in the design or implementation of the study, or in the decision to seek publication. For full author disclosures, please see the end of this article.
- Abbreviations and Acronyms
- body mass index
- catheterization percutaneous coronary intervention
- congestive heart failure
- Centers for Medicare and Medicaid Services
- ejection fraction
- glomerular filtration rate
- National Cardiovascular Data Registry
- New York Heart Association
- percutaneous coronary intervention
- ST-segment elevation myocardial infarction
- Received October 19, 2009.
- Revision received February 8, 2010.
- Accepted February 9, 2010.
- American College of Cardiology Foundation
- Smith S.C. Jr..,
- Feldman T.E.,
- Hirshfeld J.W. Jr..,
- et al.
- Arnold S.V.,
- Decker C.,
- Ahmad H.,
- et al.
- Curry L.A.,
- Nembhard I.M.,
- Bradley E.H.
- Califf R.M.,
- Peterson E.D.,
- Gibbons R.J.,
- et al.
- Kimmel S.E.,
- Berlin J.A.,
- Strom B.L.,
- Laskey W.K.
- Moscucci M.,
- Kline-Rogers E.,
- Share D.,
- et al.
- Negassa A.,
- Monrad E.S.,
- Bang J.Y.,
- Srinivas V.S.
- O'Connor G.T.,
- Malenka D.J.,
- Quinton H.,
- et al.
- Wu C.,
- Hannan E.L.,
- Walford G.,
- et al.
- Singh M.,
- Rihal C.S.,
- Roger V.L.,
- et al.
- Brindis R.G.,
- Fitzgerald S.,
- Anderson H.V.,
- Shaw R.E.,
- Weintraub W.S.,
- Williams J.F.
- Krumholz H.M.,
- Brindis R.G.,
- Brush J.E.,
- et al.
- Curtis J.P.,
- Schreiner G.,
- Wang Y.,
- et al.
- Singh M.,
- Rihal C.S.,
- Gersh B.J.,
- et al.
- Lichtman J.H.,
- Spertus J.A.,
- Reid K.J.,
- et al.
- Institute of Medicine