Author + information
- Received April 11, 2003
- Revision received May 16, 2003
- Accepted May 26, 2003
- Published online November 19, 2003.
- Mandeep Singh, MD*,* (, )
- Charanjit S Rihal, MD*,
- Faith Selzer, PhD†,
- Kevin E Kip, PhD†,
- Katherine Detre, MD, DrPH† and
- David R Holmes Jr, MD*
- ↵*Reprint requests and correspondence:
Dr. Mandeep Singh, Mayo Clinic, 200 First Street SW, Rochester, Minnesota 55905, USA.
Objectives We sought to validate the recently proposed Mayo Clinic risk score model for complications after percutaneous coronary interventions (PCI), using an independent data set.
Background The Mayo Clinic risk score has eight simple clinical and angiographic variables for the prediction of complications defined as either death, Q-wave myocardial infarction, emergent or urgent coronary artery bypass graft surgery, or cerebrovascular accident after PCI. External validation using an independent data set is lacking.
Methods A total of 3,264 patients undergoing PCI at each of the 17 sites in the National Heart, Lung, and Blood Institute's Dynamic Registry during two enrollment periods (July 1997 to February 1998 and February to June 1999) were studied. Logistic regression was used to model the calculated risk score and major procedural complications. The expected number of complications, with 95% confidence bounds (CBs), was also calculated.
Results There were 96 (2.94%) observed procedural complications, and the Mayo Clinic risk score predicted 93.5 events (2.86%; 95% CB 2.32% to 3.41%; p = NS). The Hosmer-Lemeshow goodness-of-fit p value was 0.28, and the area under the receiver operating curve was 0.76, indicating excellent overall discrimination. There were no statistical differences between observed and predicted procedural complications using the Mayo Clinic risk score among the most selected high- and low-risk subgroups.
Conclusions Eight variables were combined into a convenient risk scoring system that accurately predicts cardiovascular complications after PCI. The Mayo clinic predictive model for procedural complications yielded excellent results when applied to a multi-center external data set.
With increasing operator experience, refinement in technology, and the availability of improved stent designs, percutaneous coronary intervention (PCI) is now considered the treatment of choice for many high-risk subgroups in which PCI was previously contraindicated (1–3). Current risk adjustment models for PCI have been restricted to procedural mortality, thereby ignoring other important complications that can significantly increase morbidity and the length of hospital stay (4–8). Using Mayo Clinic data on patients undergoing PCI, we published a procedural complications risk score model that included not only death but also Q-wave myocardial infarction (MI), emergent or urgent coronary artery bypass graft surgery (CABG), and cerebrovascular accident (9). This internally validated model, comprised of eight clinical and angiographic variables, however, lacked external validation. The multi-center National Heart, Lung, and Blood Institute's (NHLBI) Dynamic Registry was designed to obtain periodic updates of the practice of interventional cardiology, with emphasis on patient and lesion selection criteria, procedural performance, and early and intermediate-term outcomes. Given the similarities in data collection and definitions between the Mayo Clinic and Dynamic Registry, we sought to validate the Mayo Clinic risk score for procedural complications after PCI, using the NHLBI Dynamic Registry data set.
Mayo Clinic risk score
Briefly, the Mayo Clinic procedural complications risk score was comprised of the following eight clinical and angiographic variables: age, cardiogenic shock, serum creatinine >264 μmol/l, urgent or emergent procedure, New York Heart Association (NYHA) functional class ≥III heart failure, thrombus, and left main and multi-vessel disease (Table 1). The risk score was based on 5,463 patients' data collected between January 1, 1996, and December 31, 1999 (9).
The NHLBI Dynamic Registry is a prospective cohort study of consecutive patients treated with PCI at participating clinical sites across the U.S. and Canada. The registry enrolls and follows patients in “waves.” The first wave of the registry enrolled 2,524 patients from 15 clinical sites between July 1997 and February 1998, and the second wave enrolled 2,105 patients from 16 clinical sites between February and June 1999. This report combines data from waves 1 and 2 because of the absence of differences in the clinical characteristics of patients in both waves (10). The present analysis was confined to 3,264 of these patients. Patients excluded from this analysis were those whose age at baseline was unknown (n = 9), those who reported a previous PCI (n = 1,343), and those who needed elective CABG during the index hospitalization for severe residual disease (n = 14). The latter two exclusions were stipulations made by the Mayo Clinic risk score analysis. Each clinical center received approval from its Institutional Review Board.
The outcome for this analysis was the incidence of major procedural complications, defined as one or more of the following: 1) in-hospital death; 2) Q-wave MI; 3) urgent or emergent CABG; and 4) cerebrovascular accident during the index hospital admission.
The definitions used by the Dynamic Registry are similar to those used in the original Mayo Clinic risk score for in-hospital death, Q-wave MI, CABG, and cerebrovascular accidents. Current physician-diagnosed congestive heart failure (CHF) during index hospitalization was used in place of the NYHA classification for congestive heart failure that was used by the Mayo Clinic risk score. In addition, a history of chronic kidney disease or end-stage renal disease was used in place of high serum creatinine because creatinine values were not routinely collected. Lastly, there are differences in the definition of “vessel disease,” with the Dynamic Registry using ≥50% diameter stenosis and the Mayo Clinic using ≥70%. Detailed baseline patient demographic, angiographic, and procedural variables have been described in the original Mayo Clinic risk score report (9)and in previous Dynamic Registry publications (10,11).
Crude incidence rates of major in-hospital complications were calculated for baseline demographic, procedural, and angiographic characteristics and compared using the Pearson chi-square test. Similarly, odds ratios and standard 95% confidence intervals were calculated using the Mantel-Haenszel method. Risk scores using the Mayo Clinic risk score model were calculated from coefficients derived from the multivariable risk factor equation for major procedural complications associated with coronary intervention procedures (9). The probabilities of procedural complications were then summed to determine an expected number of in-hospital complications for the overall Dynamic Registry sample and for patient subgroups. The 95% confidence bounds (CBs) around the expected procedural complication rate were calculated with the normal approximation to a binomial distribution. Logistic regression was used to model the calculated risk score and major procedural complications for the overall sample and specific subgroups. Each model's goodness-of-fit was assessed using the Hosmer-Lemeshow method (12). Model discrimination was assessed using the area under the receiver operating curve (ROC) or the cstatistic. Observed versus expected major procedural complications by rank-ordered deciles of risk were plotted. Logistic regression was also used to model predictors of major procedural complications specific to the Dynamic Registry. Baseline demographic data, patient characteristics, and angiographic and procedural variables were screened univariately, using a level of significance of 0.15. Standard stepwise procedures were then used to select variables to include in the final multivariable model. A two-tailed p value ≤0.05 was considered significant.
In the Dynamic Registry, there were 96 procedural complications (2.94%) among the 3,264 patients undergoing PCI. The majority of patients experienced a single procedural event (53 deaths, 7 Q-wave MIs, 6 strokes, 13 urgent CABG operations, and 11 emergent CABG operations), although six patients experienced combined outcomes (1 urgent CABG/death, 1 emergent CABG/death, 3 Q-wave MIs/death, and 1 stroke/death).
Demographic, clinical, and procedural characteristics
The mean age of the patients was 62.5 ± 11.9 years (data not shown); 36.5% were female; 42.1% presented with unstable angina (data not shown); and 27.1% presented with acute MI (Table 2). At the time of the procedure, 2.3% of patients were in cardiogenic shock. The prevalence of hypertension was 59.2% (data not shown); 26.4% presented with diabetes; 6.1% had peripheral vascular disease; and 3.9% reported a history of chronic kidney disease or end-stage renal disease.
Univariate associations between baseline demographic characteristics and major adverse complications are also shown in Table 2. The factors significantly associated with procedural complications included older age, lower body mass index, acute MI, cardiogenic shock, urgent or emergent procedures, diabetes, CHF on presentation, low left ventricular ejection fraction (<40%), and prevalent concomitant pulmonary, peripheral vascular, or renal disease.
High-risk angiographic characteristics were frequently present among the study patients, including thrombus, multi-vessel coronary artery disease, and American College of Cardiology/American Heart Association (ACC/AHA) type C lesions (Table 3). Major procedural complications after PCI were significantly associated with the presence of two- or three-vessel coronary disease, total occlusion(s), lesion(s) containing intracoronary thrombus or calcification, ACC/AHA type C lesion, and four or more significant lesions (vs. one significant lesion).
Validation of Mayo Clinic risk score using the NHLBI Dynamic Registry
Using the Mayo Clinic risk score, the observed and predicted in-hospital complications rates after PCI in the Dynamic Registry are shown in Table 4. Overall, there were 96 complications (2.94%) among the 3,264 patients studied. Using the coefficients from the Mayo Clinic risk score's logistic regression model (Table 1), 93.5 procedural complications (2.86%) were predicted (95% CB 2.32% to 3.41%; p = NS). The area under the ROC was 0.76, which indicates a good ability to discriminate between patients who had complications during the index hospitalization and those who did not. The data did not deviate significantly from the logistic model, as indicated by the non-significant Hosmer-Lemeshow goodness-of-fit test (p = 0.28).
The discriminatory ability of the prediction equation, as measured by the area under the ROC (cstatistic), was reasonably consistent across most subgroups (0.61 to 0.81) (Table 4). The model performed slightly worse for patients who had elective procedures (c= 0.61). However, the model performed especially well for older patients (c= 0.81) and for patients with diabetes (c= 0.81). Furthermore, the observed incident rates of procedural complications fell within the 95% CB (null hypothesis) for all subgroups, with the exception of patients who presented with a previous CABG. In this instance, the model significantly overpredicted the number of procedural complications. With the exception of patients undergoing urgent or emergent PCI, the non-significant Hosmer-Lemeshow goodness-of-fit p values indicated little departure in model fit.
Integer risk score
Based of the integer portion of the risk score, each patient was categorized into one of five post-PCI procedural complication risk groups. Among the Dynamic Registry patients, 1,669 (51.1%) were considered very low risk, 1,089 (33.4%) as low risk, 351 (10.8%) as moderate risk, 102 (3.1%) as high risk, and 53 (1.6%) as very high risk. The observed rates of procedural complications (and the expected range of events based on the Mayo Clinic data) in these strata were as follows: 1.26% (≤2%) for very low-risk procedures, 1.93% (>2% to 5%) for low-risk procedures, 6.84% (>5% to 10%) for moderate-risk procedures, 8.82% (>10% to 25%) for high-risk procedures, and 39.6% (>25%) for very high-risk procedures (Fig. 1). These data demonstrate an overall linear relationship between risk strata and the incidence of procedural complications, with exceptionally high relative risk among patients categorized as very high risk of PCI complications.
Predictors of major procedural complications in the Dynamic Registry
A multivariate model of predictors of major in-hospital outcomes specific to the Dynamic Registry included the following variables: older age, CHF during index hospitalization, cardiogenic shock, total occlusion attempted, the number of attempted lesions, urgent or emergent procedure, and the presence of severe concomitant renal disease (Table 5). Although there are some differences, mainly involving angiographic variables, many of the same variables are included in both the Dynamic Registry and Mayo Clinic models (Table 1) for procedural complications. The non-significant Hosmer-Lemeshow goodness-of-fit p value indicated that the model was adequate.
In this report, we assessed the validity of the Mayo Clinic risk score for the prediction of procedural complications after PCI by using an external data set. Using the Dynamic Registry, we found essentially no difference between the observed procedural complication rate and predicted complications based on the Mayo Clinic risk model. There was no departure in model fit, and model discrimination was high. Furthermore, observed and predicted complication rates among most low- and high-risk subgroups were generally similar, suggesting that the Mayo Clinic risk score applies well across most patient subgroups. Thus, our results indicate that readily available clinical and angiographic variables—those used in the Mayo Clinic risk score, in particular—can be used for patient risk stratification at the time of initial presentation.
Previous studies with validation of predictors of complications
Previously published studies on predictors (e.g., risk function) of complications after PCI have had methodologic limitations. Many included in-hospital mortality only and were based on internally validated samples (8,9,13,14), using the same data set in a different year or derived from the same consortium. Moreover, in some studies, important procedural and angiographic variables that improve the discriminatory accuracy of the risk score have been lacking (7,8,13). The recent score from the Michigan Cardiovascular Consortium identified several clinical and angiographic variables that overlap with the Mayo Clinic score; however, an independent validation data set was obtained from the same consortium from which the initial model was developed, limiting its broader applicability (8). Kimmel et al. (13)identified major predictors of complications from prospectively collected data for the Society for Cardiac Angiography and Interventions for the year 1992, which were validated in patients undergoing PCI in 1993. Importantly, since the development of this model, interventional cardiology has changed, with widespread use of stents and glycoprotein IIb/IIIa inhibitors. The New York State percutaneous transluminal coronary angioplasty mortality model was applied to patients undergoing stent implantation at the Mayo Clinic, with excellent prediction of in-hospital and late mortality (7). However, this model did not include any specific lesion characteristics that might influence the prediction of in-hospital mortality.
The present study
There are several strengths of the present study that address the limitations noted in the previous paragraph. First, our risk score has been validated and predicts not only mortality but also important adverse cardiovascular events, including stroke, MI, and the need for urgent/emergent CABG. Including these events is equally important in decision-making, as they increase a patient's morbidity and length of stay in the hospital. Second, our model was tested and validated in an independent data set. Third, patients in both the data sets were treated with a contemporary interventional stent-based approach relevant to current practice. Fourth, the Mayo Clinic risk score demonstrated consistency in predicting patient risk across many subgroups and levels of overall risk. Finally, the variables that were identified from the NHLBI Dynamic Registry significantly overlapped with the Mayo Clinic risk score, with the exception of some angiographic features, adding credence to this score.
The issue of the applicability of the Mayo Clinic model to non-referral, low-volume centers and to low-volume interventionists cannot be evaluated from the present study. In the Dynamic Registry, the overall event rates were low, and there were few high-risk patients. There was also an indication that the Mayo Clinic risk model did not perform optimally at the extremes (high- and low-risk patients), with significant overprediction of procedural complications in patients with a previous CABG. The suboptimal fit among low-risk subgroups most likely stems from the inclusion of high-risk variables such as cardiogenic shock, left main coronary disease, and CHF. Operator volume, a variable not addressed in the current study, has also been found to be significantly associated with adverse events after PCI (15–19). This variable was not addressed in the present study. Due to differences in definitions across studies, it is possible that the calculated probabilities of procedural complication may be inflated in the Dynamic Registry. Misclassification may have occurred with respect to renal, multi-vessel, and left main coronary disease variables. Finally, though the Mayo Clinic risk score was accurate in predicting procedural complications, no predictive model can ameliorate the effect of chance and unanticipated circumstances and complications inherently encountered in invasive treatments.
External validation of the Mayo Clinic risk score using the multi-center NHLBI Dynamic Registry study confirms the broader applicability of this score in predicting in-hospital complications, defined as either death, Q-wave MI, emergent or urgent CABG, or cerebrovascular accidents, using eight simple clinical and angiographic variables. Most of the variables can be obtained at the time of first contact with the patient. Risk stratification may help the operator to individualize the risk of procedural complications from PCI and to counsel patients at the time of PCI.
☆ The Dynamic Registry is funded by the National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland, and supported by grant HL33292-14.
- coronary artery bypass graft surgery
- confidence bound
- congestive heart failure
- myocardial infarction
- National Heart, Lung, and Blood Institute
- New York Heart Association
- percutaneous coronary intervention
- receiver operating curve
- Received April 11, 2003.
- Revision received May 16, 2003.
- Accepted May 26, 2003.
- American College of Cardiology Foundation
- Ellis S.G.,
- Tamai H.,
- Nobuyoshi M.,
- et al.
- Ellis S.G.,
- Guetta V.,
- Miller D.,
- Whitlow P.L.,
- Topol E.J.
- Northern New England Cardiovascular Disease Study Group,
- O'Connor G.T.,
- Malenka D.J.,
- Quinton H.,
- et al.
- Holmes D.R. Jr..,
- Berger P.B.,
- Garratt K.N.,
- et al.
- Moscucci M.,
- Kline-Rogers E.,
- Share E.,
- et al.
- Singh M.,
- Lennon R.J.,
- Holmes D.R. Jr..,
- et al.
- Williams D.O.,
- Holubkov R.,
- Yeh W.,
- et al.
- Hosmer D.W. Jr..,
- Lemeshow S.
- Kimmel S.E.,
- Berlin J.A.,
- Strom B.L.,
- Laskey W.K.
- Mick M.J.,
- Piedmonte M.R.,
- Arnold A.M.,
- Simpfendorfer C.
- Ellis S.G.,
- Omoigui N.,
- Bittl J.A.,
- et al.
- Ellis S.G.,
- Weintraub W.,
- Holmes D.,
- Shaw R.,
- Block P.C.,
- King S.B. III.
- Northern New England Cardiovascular Disease Study Group,
- McGrath P.D.,
- Wennberg D.E.,
- Malenka D.J.,
- et al.