Author + information
- Received August 13, 2001
- Revision received November 29, 2001
- Accepted January 9, 2002
- Published online April 3, 2002.
- Richard E Shaw, PhD, FACC*,* (, )
- H.Vernon Anderson, MD, FACC*,
- Ralph G Brindis, MD, FACC*,
- Ronald J Krone, MD, FACC*,
- Lloyd W Klein, MD, FACC*,
- Charles R McKay, MD, FACC*,
- Peter C Block, MD, FACC*,
- Leslee J Shaw, PhD*,
- Kathleen Hewitt, MS, RN*,
- William S Weintraub, MD, FACC*,
- ↵*Reprint requests and correspondence:
Richard E. Shaw, PhD, FACC, San Francisco Heart Institute at Seton Medical Center, 1900 Sullivan Avenue, Daly City, California 94015, USA.
Objectives We sought to develop and evaluate a risk adjustment model for in-hospital mortality following percutaneous coronary intervention (PCI) procedures using data from a large, multi-center registry.
Background The 1998–2000 American College of Cardiology–National Cardiovascular Data Registry (ACC–NCDR) dataset was used to overcome limitations of prior risk-adjustment analyses.
Methods Data on 100,253 PCI procedures collected at the ACC–NCDR between January 1, 1998, and September 30, 2000, were analyzed. A training set/test set approach was used. Separate models were developed for presentation with and without acute myocardial infarction (MI) within 24 h.
Results Factors associated with increased risk of PCI mortality (with odds ratios in parentheses) included cardiogenic shock (8.49), increasing age (2.61 to 11.25), salvage (13.38) urgent (1.78) or emergent PCI (5.75), pre-procedure intra-aortic balloon pump insertion (1.68), decreasing left ventricular ejection fraction (0.87 to 3.93), presentation with acute MI (1.31), diabetes (1.41), renal failure (3.04), chronic lung disease (1.33); treatment approaches including thrombolytic therapy (1.39) and non-stent devices (1.64); and lesion characteristics including left main (2.04), proximal left anterior descending disease (1.97) and Society for Cardiac Angiography and Interventions lesion classification (1.64 to 2.11). Overall, excellent discrimination was achieved (C-index = 0.89) and application of the model to high-risk patient groups demonstrated C-indexes exceeding 0.80. Patient factors were more predictive in the MI model, while lesion and procedural factors were more predictive in the analysis of non-MI patients.
Conclusions A risk adjustment model for in-hospital mortality after PCI was successfully developed using a contemporary multi-center registry. This model is an important tool for valid comparison of in-hospital mortality after PCI.
The establishment of quality standards based on patient outcome data is a rational means for differentiating the quality of health care in the marketplace. Institutional variation in the baseline clinical risks of patients precludes the direct comparison of outcomes across institutions. The application of risk adjustment methodology to account for patient differences in these treatment outcomes is imperative for legitimate comparison of institutional results in the modern era of cardiovascular intervention. Mortality is the indicator that has been most widely used to evaluate the quality of cardiac treatment procedures. Cardiac surgeons addressed the problems associated with the publication in the late 1980s of unadjusted surgical mortality results in New York and other states by forming the Society of Thoracic Surgeons National Database and developing risk adjustment models for coronary artery bypass graft (CABG) surgical mortality (1–4). Studies conducted in the Veterans Administration Health Care System have also demonstrated that risk adjustment approaches provide a fair comparison of cardiac surgical outcomes across a broad spectrum of institutions (5).
There have been numerous efforts in recent years to incorporate risk adjustment methodology to evaluate differences in mortality rates for interventional procedures (6–14). These efforts have been limited by inconsistent definitions of the factors used in the models, small sample sizes, limited geographic representation, inclusion of programs that do not necessarily represent the standard of practice across the country, and patient samples that do not reflect contemporary percutaneous coronary intervention (PCI) practice. The goal of the current study was to analyze the initial experience of the American College of Cardiology–National Cardiovascular Data Registry (ACC–NCDR) to develop a risk-adjusted model for mortality associated with PCI. The experience of this registry is described in a companion publication (15). The strengths of the ACC–NCDR registry experience include the use of standardized data definitions, data completeness procedures, geographic and institutional diversity, a large sample size, and analysis of contemporary PCI practice. These features offer a significant advantage over previous efforts to develop a risk-adjustment model for in-hospital mortality after PCI.
The data collection process has been described (15). For institutions with submissions passing the inclusion/exclusion criteria for data completeness, the first PCI procedure performed during a qualifying hospitalization was chosen for analysis. In all, 100,292 procedures passed the initial screening for inclusion into the risk model analysis. Because initial screening required that 99% completeness be achieved on all outcome variables, 39 hospitalizations were missing data on mortality. These 39 PCI procedures were excluded from the risk model development, leaving a total of 100,253 procedures.
Data elements entered into the mortality risk model included patient demographic data, cardiac risk factors, coronary revascularization status, anginal status, non-coronary disease processes, angiographic findings and procedural variables (Tables 1 through 5). ⇓⇓⇓⇓⇓A variable was constructed combining the lesion codes for the ACC/AHA type A-B-C lesion class along with the presence or absence of an occlusion, based on the work of Krone et al. (16). This classification scheme, referred to as the Society for Cardiac Angiography and Interventions Lesion Class (SCAI LC), produces four categories (I, II, III, IV): I—non–type C/patent; II—type C/patent; III—non–type C/occlusion; and IV—type C/occlusion. In a preliminary analysis, this classification system was highly correlated with PCI outcomes. Congestive heart failure was not included in the model, because there was a problem with one of the software vendor packages that allowed out-of-range values to be included in the database that could not be interpreted.
Standard univariate methods were used to generate odds ratios (ORs) with 95% confidence intervals (CIs) in evaluating the relationship between individual factors and mortality. These were tested using Pearson’s chi-square test. Stepwise logistic regression analysis was performed using the SPSS 10.1 statistical software (SPSS Inc; Chicago, Illinois) to assess the independent relationship of all significant univariate factors with in-hospital mortality. Variables were entered and removed by a stepwise selection process, using residual Wald chi-square p values for entry of 0.5 and 0.1 for removal. Improvements in chi-square and maximum log likelihood stepwise methods, including Hosmer-Lemeshow goodness of fit chi-square estimates, were used to evaluate the regression model (17). Model discrimination was assessed using the area under the receiver operating characteristic (ROC) or C-index (18). The level of missing data varied from a low of 0.1% for gender to a high of 26.1% for left ventricular ejection fraction (LVEF) (Tables 1 through 5). Missing values were assigned values using multiple imputation methods (19).
Risk model development
Univariate analyses were used to identify patient demographic and risk factor (Table 1), cardiac history and anginal status (Table 2), non-coronary disease processes (Table 3), angiographic (Table 4) and procedural (Table 5) factors significantly associated with mortality that were included in the regression model. Of the 32 variables evaluated, hypertension, previous myocardial infarction (MI), previous CABG and lesion in a graft did not achieve a significance level of <0.01 and were not included in the regression model. Hypercholesterolemia was also omitted from the model because of its counterintuitive relationship to the mortality outcome, perhaps related to the advanced sickness of many of the patients treated. Others (8,9)also have noted this peculiarity.
A standard training set/test set approach was used. A randomly generated training set was used to develop the regression model, and the test set consisting of the remaining patients was used to assess the performance of the model against observed mortality results. After the risk factors were determined and their regression weights calculated from the training set, the standard probability formula was applied to the test set to determine the risk of mortality for each patient. The ROC curves were generated for the training set and the test set.
The model was validated in two ways. First, the dataset was ordered using the values for each patient’s probability of mortality generated from the regression model. The dataset was then divided into deciles of risk, and the observed mortality rate was calculated for each decile. The observed versus the expected mortality was plotted and evaluated using the R-square statistic. The model was also validated by identifying patient subgroups that were known to have high mortality rates. In addition, separate logistic regression models were generated for patients presenting with acute MI within 24 h of PCI and those presenting without acute MI within 24 h.
In 100,253 PCI procedures, in-hospital mortality occurred in 1,422 (1.4%). There was wide variation in mortality among subgroups of patients, with the highest mortality rates observed in patients presenting for emergent salvage procedures (30.9%), PCI indication for shock (28.0%), insertion of an intra-aortic balloon pump (IABP) pre-PCI (19.1%) and LVEF <10% (10.1%) (Tables 1–5).
The training set consisted of 50,123 PCI procedures randomly selected from the overall patient population. In this group, 707 deaths occurred (1.4%). Multivariate logistic regression analysis identified PCI indication for shock, increasing age, the need for urgent or emergent PCI, pre-procedure placement of an IABP, decreasing LVEF, acute MI within 24 h of hospital admission, diabetes, renal failure, chronic lung disease, treatment approaches including use of thrombolytics and non-stent devices, and lesion characteristics including presence of left main or proximal left anterior descending (LAD) disease and SCAI lesion class as factors independently associated with in-hospital mortality (Table 6) with a C-index of 0.89, demonstrating excellent model discrimination. The Hosmer-Lemeshow statistic was not significant, indicating little departure from a perfect fit.
The model was then applied to the test set that consisted of 50,130 PCI procedures in which 715 deaths occurred (1.4%). This generated a C-index of 0.89, demonstrating equally good model discrimination for the test set. The predicted risks of patients were ranked and divided into deciles. The number of predicted deaths was plotted against the number of actual deaths in each decile (Fig. 1). The computed R-square was 0.96, demonstrating excellent correlation between the expected and observed mortality. Table 7shows the observed and predicted mortalities, area under the ROC curve and 95% CIs for the application of the model to the high-risk groups of patients in the test set. The predictive model appears to be relatively stable across these high-risk patients, with areas under the ROC curve exceeding 0.80 in almost every category. The lowest values occurred in the patients with shock and those undergoing emergent salvage procedures. A similar finding was reported by O’Connor et al. (8).
Separate models were developed for patients presenting with acute MI within 24 h of their PCIs and those presenting without acute MI. These results are reported in Tables 8 and 9. ⇓⇓The area under the ROC curve was less in these models than in the overall model. The C-index for the analysis of acute MI patients was 0.87 (0.85 to 0.89) and for non-MI patients was 0.86 (0.83 to 0.88). It is interesting to note the pattern of factors in each of these models compared with the overall model. In the model for acute MI patients, the strongest factors were age, diabetes, left main disease, a proximal LAD lesion, shock and use of devices other than stents. Procedures classified as urgent, placement of an IABP and chronic lung disease were not significant in this model, and the effects of factors such as LVEF, SCAI lesion class and classification as an emergent procedure were not as strong as in the overall model, in which acute MI and non-MI patients were mixed (Table 8). For patients presenting without MI, presentation in shock, acuteness of the procedure, pre-placement of an IABP, SCAI lesion class, renal failure and chronic lung disease emerged as strongly predictive factors (Table 9). The effects of age, diabetes, left main disease and the use of devices other than stents were lesser, while lesions in the proximal LAD and use of thrombolytic therapy dropped out of the model entirely. The pattern observed in the acute MI model seems to be consistent with the highest-risk patients presenting as older, with diabetes, in shock and having involvement of the left coronary system. The pattern observed in the non-MI model seems to be consistent with lesion morphology and classification of the acuteness of the procedure as being the factors most related to patient outcome.
Over the last decade, advancements in technology and new pharmaceutical agents have helped to reduce the morbidity and mortality associated with PCI (20). However, PCI still carries significant risk, especially in subgroups in which a more complex clinical condition may lead to higher adverse event rates. Databases have emerged as tools crucial for evaluating the quality of cardiac care (21). The ACC–NCDR provides a way for institutions to collect high-quality local data and to merge those data with data from other centers across the country through a national data registry. The current study demonstrates the potential value of using this national database for developing a robust risk-adjustment mortality model for PCI.
Prior risk modeling has been limited by inconsistent definitions, small sample sizes, lack of institutional diversity, restricted geographic representation and patient samples that do not reflect contemporary PCI practice. Hannan et al. (6)published results from the New York State mandated registry for PCI. This represented one of the first efforts to develop a mortality risk model using data from several centers. The patient sample that was included, however, reflected an early era in the development of interventional technology and pharmaceutical treatment. More recently, Kimmel et al. (7)analyzed the experience in 10,622 first-time PCI procedures from the Society for Cardiac Angiography and Interventions Registry. O’Connor et al. (8)reported on the development of a risk-adjusted mortality model using 15,331 patients who underwent PCI over a three-year period at six regional centers in northern New England. Although the data definitions and collection protocols were well-established and produced high-quality data in both of these studies, the sample sizes were relatively small for model development. In the Kimmel et al. (7)study, the geographic distribution was broad, but the number of centers was small, while the northern New England experience represented a very narrow geographic region. Neither of these studies included patient samples comparable to contemporary PCI cohorts, wherein the proportion of patients receiving stents and glycoprotein IIb/IIIa inhibitors exceeds 75%. Two more recent analyses (13,14)have included patient samples that reflect the widespread use of stents and glycoprotein IIb/IIIa inhibitors, although both were analyses of a single center’s experience.
Block et al. (9)combined the experience of eight registries in a pooled meta-analysis of 158,273 cases to identify factors associated with risk of Q-wave MI, emergent cardiac surgery and death. This effort was helpful in providing a broad view of data elements across a large sample of patients. It was limited for the purpose of model development, however, in that source data were not available to apply standard statistical analyses. Other efforts, such as the multi-center study of Ellis et al. (10), the National Heart, Lung, and Blood Percutaneous Transluminal Coronary Angioplasty registries (11)and the New Approaches in Coronary Intervention Registry (12), have utilized risk-adjustment techniques. However, the centers involved in these studies were highly selective and not necessarily representative of a broad assessment of PCI experience.
Comparison with models from other studies
The variables that were generated from the ACC–NCDR mortality model are consistent with a number of other studies published from local databases and registries. O’Connor et al. (8)identified increasing age, cardiogenic shock, urgent and emergent procedures, LVEF, pre-procedure IABP placement and attempt of type C lesions as significant predictors in their model. They also noted that heart failure and creatinine levels ≥2.0 mg/dl were predictors in their model, but these two variables were not included in the analysis of the ACC–NCDR experience, although the definition of renal failure in the current analysis was based on similar creatinine levels. The ACC–NCDR data analysis included usage of devices, which was not addressed in the O’Connor et al. (8)model. The only factor common to both studies that was significant in the O’Connor et al. (8)model and that fell out of the ACC–NCDR model was the presence of peripheral vascular disease. In an analysis of patients undergoing more contemporary PCI, Resnic et al. (13)identified similar factors, including cardiogenic shock, class 3 or 4 heart failure, left main intervention, tachycardia, chronic renal insufficiency, age ≥75 years, type B2 or C lesions, acute MI, unstable angina and stent use (a negative relationship to mortality).
Block et al. (9)pooled data from eight different sources and identified a number of factors associated with in-hospital death. Variables they identified that overlap with the current analysis include age, LVEF, acute MI, procedure acuteness, cardiogenic shock, use of IABP, diabetes, renal failure and lesion type. There is remarkable consistency for many of the factors across all databases.
Other studies have focused on the unique relationship of lesion factors to adverse outcomes. Ellis et al. (14)found 10 lesion factors that were related to ischemic complications after PCI. The most significant factors were a non-chronic total occlusion and degenerated saphenous vein graft. In the current analysis, the SCAI LC lesion classification combined the effect of type C and occlusion. However, treatment of a lesion in a vein graft did not have a significant relationship to in-hospital mortality. The assessment of vein graft irregularities in the Ellis et al. (14)study probably represents a more comprehensive analysis of the condition of the saphenous vein graft, which may have enhanced the predictive utility of this variable. Zaacks et al. (22)found that focused characteristics of lesions (presence of thrombus, inability to protect a side branch and degenerated vein graft lesions) were more likely to be related to complications, while the more general classification of lesions using the ACC/AHA A-B-C system were more predictive of procedural success. Several lesion factors that are consistent with other studies emerged in the current analysis as predictors of mortality. One of the most important aspects of the current analysis, however, is the development of separate models for acute MI and non-MI patients. This analysis showed that lesion factors were more highly predictive of in-hospital mortality for non-MI patients than for acute-MI patients.
The utility of risk models
It is important to emphasize that the development of predictive models is as much an art as it is a science. Models are dependent on the quality and accuracy of the data and the relative rate of the outcome event being studied. If the quality of the data is suspect, the modeling process is unpredictable. Likewise, when the outcomes assessed occur infrequently, as in PCI mortality, the modeling process is even more challenging. These limitations of modeling must be kept in mind when evaluating and applying the results of the models presented herein as well as other models developed for PCI outcomes.
There were variables in the ACC–NCDR that could not be reliably used for the current analysis. The variable for assessing the status of congestive heart failure had uninterpretable data resulting from a software vendor problem. This problem was corrected, and the heart failure variable will be available for future analyses. The logistic regression approach has an upper limit of predictive capability, with a C-statistic of around 0.87 (23). Techniques such as neural networks are capable of achieving indexes to 0.93 and may play a role in future modeling efforts. It is also not clear to what extent these models built on national datasets can be generalized to local datasets. Comparisons of several models used in cardiac surgery have demonstrated similar predictive capabilities (24,25), but application from one setting to another has limitations (26). These same issues are present for risk-adjusted models developed for PCI mortality.
Perhaps the most significant limitation of the current study is the lack of a systematic approach to auditing the data. Although many consistency checks were instituted in the data collection process, the extent to which these data reflect clinical reality at each institution is not known. It is imperative that future databases used for institutional evaluation be subjected to valid and objective audit processes.
This study represents the development of a risk-adjustment model for in-hospital mortality after PCI in a large, contemporary, multi-institutional national database. This analysis has generated a powerful tool for evaluating and comparing mortality outcomes across institutions, which is crucial in the current era of cardiovascular intervention.
The authors and the ACC–NCDR Oversight and Planning Task Force thank the members of the Publications and Development Subcommittee: Ben D. McCallister, MD; Charles R. McKay, MD; David O. Williams, MD; H. Vernon Anderson, MD; John F. Williams, Jr., MD; Leslee J. Shaw, PhD; Lloyd W. Klein, MD; Martha J. Radford, MD; Michael A. Kutcher, MD; Michael J. Wolk, MD; Peter C. Block, MD; Ralph G. Brindis, MD, MPH; Raymond J. Gibbons, MD; Richard E. Shaw, PhD; Ronald J. Krone, MD; Ronald N. Riner, MD; Ross A. Davies, MD; William S. Weintraub, MD. The authors acknowledge the technical assistance provided by Jim Emlet, MS, in the preparation of the data analyzed and Michael B. Pliam, MD, PhD, for his work on the risk adjustment methodology.
- American College of Cardiology National Cardiovascular Data Registry
- coronary artery bypass graft surgery
- confidence interval
- intra-aortic balloon pump
- left anterior descending
- left ventricular ejection fraction
- myocardial infarction
- odds ratio
- percutaneous coronary intervention
- receiver operating characteristic
- SCAI LC
- Society for Cardiac Angiography and Interventions Lesion Classification
- Received August 13, 2001.
- Revision received November 29, 2001.
- Accepted January 9, 2002.
- American College of Cardiology Foundation
- for the Registry Committee of the Society for Cardiac Angiography and Interventions,
- Kimmel S.E,
- Berlin J.B,
- Strom B.L,
- Laskey W.K
- for the Northern New England Cardiovascular Disease Study Group,
- O’Connor G.T,
- Malenka D.J,
- Quinton H,
- et al.
- Block P.C,
- Peterson E.C,
- Krone R,
- et al.
- Ellis S.G,
- Omoigui N,
- Bittl J.A,
- et al.
- Williams D.O,
- Holubkov R,
- Yeh W,
- et al.
- Ellis S.G,
- Guetta V,
- Miller D,
- Whitlow P.L,
- Topol E.J
- Anderson H.V,
- Shaw R.E,
- Brindis R.G,
- et al.
- Hosmer D.W,
- Lemeshow S
- Hanley J.A,
- McNeil B.J
- for the Northern New England Cardiovascular Diseases Study Group,
- McGrath P,
- Malenka D.J,
- Wennberg D.E,
- et al.
- Peterson E.D,
- DeLong E.R,
- Muhlbaier L.H,
- et al.
- Ivanov J,
- Tu J.V,
- Naylor D