Author + information
- Received August 28, 1998
- Revision received April 7, 1999
- Accepted May 16, 1999
- Published online September 1, 1999.
- Mauro Moscucci, MD∗,†,*,
- Gerald T O’Connor, PhD, DSc†,‡,
- Stephen G Ellis, MD, FACC†,†,
- David J Malenka, MD, FACC†,‡,
- Jennifer Sievers, MSc∗,†,
- Eric R Bates, MD, FACC∗,†,
- David W.M Muller, MBBS, MD, FACC†,†,
- Steven W Werns, MD, FACC∗,†,
- Eva Kline Rogers, RN, MSc∗,†,
- Dean Karavite∗,† and
- Kim A Eagle, MD, FACC∗,†
- ↵*Reprint requests and correspondence: Dr. Mauro Moscucci, Heart Care Program, University of Michigan Medical Center, B1, Room F245, 1500 East Medical Center Drive, Ann Arbor, Michigan 48109-0022
We sought to validate recently proposed risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty (PTCA) mortality on an independent data set of high risk patients undergoing PTCA.
Risk adjustment models for PTCA mortality have recently been reported, but external validation on independent data sets and on high risk patient groups is lacking.
Between July 1, 1994 and June 1, 1996, 1,476 consecutive procedures were performed on a high risk patient group characterized by a high incidence of cardiogenic shock (3.3%) and acute myocardial infarction (14.3%). Predictors of in-hospital mortality were identified using multivariate logistic regression analysis. Two external models of in-hospital mortality, one developed by the Northern New England Cardiovascular Disease Study Group (model NNE) and the other by the Cleveland Clinic (model CC), were compared using receiver operating characteristic (ROC) curve analysis.
In this patient group, an overall in-hospital mortality rate of 3.4% was observed. Multivariate regression analysis identified risk factors for death in the hospital that were similar to the risk factors identified by the two external models. When fitted to the data set, both external models had an area under the ROC curve >0.85, indicating overall excellent model discrimination, and both models were accurate in predicting mortality in different patient subgroups. There was a trend toward a greater ability to predict mortality for model NNE as compared with model CC, but the difference was not significant.
Predictive models for PTCA mortality yield comparable results when applied to patient groups other than the one on which the original model was developed. The accuracy of the two models tested in adjusting for the relatively high mortality rate observed in this patient group supports their application in quality assessment or quality improvement efforts.
The development of report cards for cardiac surgery (1–4)has been associated with a major effort toward the development and validation of risk adjustment models for coronary artery bypass graft surgery (CABG) mortality (5–9). More recently, risk adjustment models for percutaneous transluminal coronary angioplasty (PTCA) mortality have been proposed and used for the comparison of the results of different operators (10–14), but external validations on high risk patient groups and independent data sets are lacking. The objectives of our study were to identify risk factors for in-hospital PTCA mortality in a tertiary care high risk patient group, to compare and validate recently proposed risk adjustment models for PTCA mortality in this patient group and to assess the effect of case-mix on model accuracy.
The study sample consisted of 1,476 consecutive percutaneous revascularization procedures performed between July 1, 1994 and June 1, 1996 at the University of Michigan Medical Center. Conventional PTCA was used in 1,070 procedures and new device angioplasty (directional coronary atherectomy, coronary stenting, rotational atherectomy and transdevice synergy) was used in 406 procedures. Clinical and laboratory data were obtained through a computerized data base and chart review. Angiographic data were obtained through cine film review. Coronary artery stenoses were classified according to the modified American College of Cardiology/American Heart Association classification (15).
Data are expressed as the mean value ± SD or as a percentage. Statistical analysis was performed using Stata (Stata Corporation, College Station, Texas). To assess the consistency of variable selection between different models and different patient groups, univariate predictors of in-hospital death were identified using logistic regression analysis. Independent predictors of death in the hospital were identified using stepwise multivariate logistic regression analysis of a model containing univariate factors (p < 0.05). The internal model was tested for goodness-of-fit using the Hosmer-Lemeshow statistic (16).
Validation of external models
Predicted probabilities of in-hospital death for individual patients were calculated using 1) a risk adjustment logistic regression model developed at the Cleveland Clinic (17)from a multi-institutional data base including 12,985 procedures (model CC) and 2) a risk adjustment logistic regression model developed by the Northern New England (NNE) Cardiovascular Disease Study Group (18)from a multi-institutional data base including 15,331 procedures (model NNE) (Table 1). The predicted probabilities for the whole population and for various subgroups were then calculated.
The NNE risk adjustment model includes the ejection fraction (EF) as a risk factor for in-hospital death. The EF was measured using the method of Sandler and Dodge (19). To avoid exclusion of a small portion of the data set from the analysis, when missing, the EF was estimated using a multiple imputation method with inclusion of left ventricular end-diastolic pressure, history of acute myocardial infarction (AMI), history of heart failure, history of CABG, age, gender and procedure priority (20–22). Receiver operating characteristic (ROC) curves (23)were developed using the method of maximal likelihood estimation through the program CLABROC developed by Metz (24). Separate ROC curves were constructed using models CC and NNE. Model accuracy was assessed by comparing the errors and the areas under the ROC curves for the two external models. The areas were compared using a univariate Z score test statistic through the CLABROC program. Model accuracy was further assessed by constructing calibration curves, and goodness-of-fit was tested using the Hosmer-Lemeshow statistic (16).
Baseline demographic and clinical characteristics are listed in Table 2. There was a high percentage of patients presenting within <24 h of a myocardial infarction (14.5%) and with a myocardial infarction complicated by cardiogenic shock (3.3%). Twenty-one percent of patients had a history of CABG, 41% had a history of PTCA and 61% had a history of myocardial infarction.
In this high risk patient group, an overall in-hospital mortality rate of 3.4% was observed. The highest mortality rates were observed in patients presenting with AMI (12.7%) and with cardiogenic shock (48%). The high mortality rate observed in patients with AMI was a reflection of the high percentage of patients referred for cardiogenic shock in this group (17%). Further analysis revealed a mortality rate of 4.7% in patients presenting with AMI not complicated by cardiogenic shock. Multivariate analysis identified emergency procedure, age, female gender, cardiogenic shock, number of diseased vessels, congestive heart failure and creatinine level >2 mg/dl as independent predictors of in-hospital mortality. The area under the ROC curve was 0.88, indicating excellent model discrimination, and the Hosmer-Lemeshow statistic was not significant, indicating little departure from a perfect fit (chi-square = 4.3; p = 0.8).
Validation of external models
Predicted probabilities of in-hospital death estimated using the two external models are shown in Table 3. There was a trend for model CC to underestimate the risk of death for each patient subgroup. Receiver operating characteristic curve analysis revealed an area under the ROC curve of 0.88 ± 0.03 for model NNE and an area of 0.85 ± 0.03 for model CC (p = 0.09) (Fig. 1). The overall calibration was excellent for both models, and the Hosmer-Lemeshow statistic was not significant, indicating little departure from a perfect fit (Fig. 2).
Predictors of death in the hospital
In this analysis, we assessed the accuracy and potential differences of risk adjustment models for PTCA mortality on a high risk tertiary-care patient population. The mortality rate observed in this patient group was relatively high when compared with mortality rates previously reported in angioplasty registries (25–27), reflecting the high percentage of patients presenting with cardiogenic shock and AMI.
In agreement with what has been shown by other investigators, higher mortality rates were observed in elderly patients, in patients presenting with AMI, in women, in patients with multivessel disease and in patients undergoing emergency revascularization procedures (10–14). In particular, the variables identified in the internal logistic regression model as independent predictors of in-hospital death were similar to the variables identified by the two external models, supporting overall consistency in variable selection across different patient groups. In addition, these variables were identical to variables included in a list that was recently developed from combining eight large multicenter data bases (28), and which has been proposed as a tool for efficient data collection to risk adjust outcomes of coronary interventions.
Validation of the two external models
Both external models had a high area under the ROC curve (e.g., >0.8), indicating that they had excellent discrimination in this high risk patient group. The area under the ROC curve of model NNE of 0.88 observed in this study was in fact similar to the area under the ROC curve observed in an internal validation set of 6,037 patients during the development of this model (area 0.88). The area under the ROC curve for model CC was also similar to the area under the ROC curve reported during its development (17)(area 0.84). The comparison of the two models on different subsets of patients and on the overall patient population suggested that the NNE model performed slightly better than model CC in predicting mortality, although the difference of the area under the ROC curve was not statistically significant.
Differences in model performance
The slight difference in model performance can be explained on the basis of differences in variables entered in each model and their coefficients. Age, AMI, cardiogenic shock and lesion severity were variables common to both models, whereas EF, emergent or urgent priority, renal insufficiency, history of congestive heart failure, history of peripheral vascular disease and intraaortic balloon pump requirement were variables present in the NNE model only. The slight difference in model performance may also reflect baseline differences between the cohorts from which each model was derived and the present data set. For example, our patient group was characterized by a high incidence of cardiogenic shock and AMI (3% and 14%, respectively). In the data base used for the development of model CC, the overall incidence of AMI was 6.9% and the overall mortality rate was 1.3%. In the data base used for the development of the NNE model, the overall incidence of cardiogenic shock was only 0.6% and the observed mortality rate was 1.08%. Although the observed mortality in the group of patients with cardiogenic shock in this study is similar to mortality rates reported by other registry studies and in the recently completed SHOCK (SHould we emergently revascularize Occluded Coronaries for cardiogenic shocK?) trial (29,30), both models had a tendency to underestimate the mortality rate in this group of patients.
Preprocedural morbidity and PTCA mortality
It is important to underscore the fact that the high mortality rate observed in the group of patients presenting with cardiogenic shock is more likely a reflection of the natural history of the disease rather than a consequence of the procedure itself. In support of this statement, the mortality rate observed in our group of patients presenting with cardiogenic shock was similar to the mortality rate observed in the PTCA group of the SHOCK trial, and it was lower than the mortality rate observed in the medical treatment arm of this study (30).
The issue of preprocedural morbidity becomes rather important when trying to differentiate between “procedure-related deaths” and “disease-related deaths.” As of today, death after PTCA is a relatively rare event, and most of the deaths are related to preprocedural morbidity rather than to the procedure itself. Few deaths will be related to an interaction between preprocedural morbidity and the procedure itself, and very few of them will be secondary to the procedure alone. The value of risk adjustment resides in its ability to identify and to “correct” for those comorbidities that affect the relation between observed outcomes and the procedure itself.
Implications of this analysis
The important finding of this study is that despite some differences in variable selection, the two external models performed well in predicting mortality in this patient group. In particular, both models were rather accurate in adjusting for comorbidities and accounting for the relatively high mortality rate observed, thus adding supportive mathematic evidence to the initial impression of a patient group at higher risk of death in the hospital. Thus, our study shows that models that have been developed from large multi-institutional data bases can perform well also when applied to smaller, high risk, single-center patient populations. This finding is particularly relevant because models that are most effective in mathematically accounting for variation in outcomes allow physicians and institutions greater confidence that the relation between observed outcomes and procedures is influenced by explanatory variables related to case-mix rather than by other unrelated variables. In addition, the same models ensure that sheer chance is being minimized as the explanation for changing rates of outcomes during quality improvement efforts. Thus, for internal quality assessment and quality improvement purposes that typically utilize such formulas to generate expected versus observed outcomes over a given time frame (31), institutions should identify the model that best fits their patient group and their data collection process.
A perceived potential limitation of this analysis is that we did not validate the internal model on an independent data set. However, the internal model was generated only for comparison purposes and to determine consistency of variable selection across different patient groups, rather than to develop a new local model. Second, both the CC model and the NNE model were validated on a relatively small number of patients from a single center. Therefore, we should be cautious in generalizing these findings. Finally, although elective or bailout stenting was available at our institution during the study period, we cannot exclude that a more widespread use of elective stenting could result in a further decrease in the observed complications rates. Thus, risk adjustment models should be viewed as a moving target, and they should be refined over time according to further advancements in medical technology.
Despite differences in variable selection, predictive models for PTCA mortality yield comparable results when applied to patient groups other than the one on which the original model was developed. The accuracy of the two models tested in adjusting for the relatively high mortality rate observed in this patient group supports their application in quality assessment or quality improvement efforts.
- acute myocardial infarction
- coronary artery bypass graft surgery
- Cleveland Clinic
- ejection fraction
- Northern New England
- percutaneous transluminal coronary angioplasty
- receiver operating characteristic
- Received August 28, 1998.
- Revision received April 7, 1999.
- Accepted May 16, 1999.
- American College of Cardiology
- Zimman D.
- A Consumer Guide to Coronary Artery Bypass Graft Surgery. Vol. I. 1990 Data. Harrisburg (PA): Pennsylvania Health Care Cost Containment Council, 1994.
- “Surgical Scorecards.” American Broadcasting Company, June 4, 1992, Prime Time Live Transcript #248, with Diane Sawyer and Sam Donaldson.
- Northern New England Cardiovascular Disease Study Group,
- O’Connor G.T.,
- Plume S.K.,
- Olmstead E.M.,
- et al.
- Williams S.V.,
- Nash D.B.,
- Goldfarb N.
- Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired heart disease. Circulation 1989;79 Suppl I:I-3–12. (Erratum, Circulation 1990;89:1078.)
- Hannan El,
- Arani D.T.,
- Johnson L.W.,
- et al.
- Kimmel S.E.,
- Berlin J.A.,
- Strom B.J.,
- Laskey W.K.
- Ellis S.G.,
- Omoigui N.,
- Bittl J.A.,
- et al.
- Ellis S.G.,
- Vandormael M.G.,
- Cowley M.J.,
- et al.
- Hosmer D.W.,
- Lemeshow S.
- Ellis S.G.,
- Weintraub W.,
- Holmes D.,
- et al.
- ↵O’Connor GT, Malenka DJ, Quinton H, et al., Northern New England Cardiovascular Disease Study Group. Multivariate prediction of in-hospital death following percutaneous coronary interventions in 1994–1996. J Am Coll Cardiol 1999;34.
- Little R.J.A.,
- Rubin D.B.
- Greenland S.,
- Finkle W.D.
- Egan J.P.
- Metz C.E.
- Holmes D.R. Jr.,
- Holubkov R.,
- Vliestra R.E.,
- et al.
- King S.B. III.,
- Yeh W.,
- Holubkov R.,
- et al.
- Block P.C.,
- Peterson E.C.,
- Krone R.,
- et al.
- ↵Hochman JS. SHould we emergently revascularize Occluded Coronaries for cardiogenic shocK (SHOCK)?—an international randomized trial of emergency PTCA/CABG: results of the SHOCK trial. 48th Annual Scientific Session of the American College of Cardiology, New Orleans, Louisiana, March 1999.