Author + information
- Received August 25, 2005
- Revision received September 16, 2005
- Accepted September 20, 2005
- Published online February 7, 2006.
- Chuntao Wu, MD, PhD⁎,
- Edward L. Hannan, PhD, FACC⁎,
- Gary Walford, MD, FACC†,
- John A. Ambrose, MD, FACC‡,
- David R. Holmes Jr, MD, FACC§,
- Spencer B. King III, MD, FACC∥,
- Luther T. Clark, MD, FACC¶,
- Stanley Katz, MD, FACC#,
- Samin Sharma, MD, FACC⁎⁎ and
- Robert H. Jones, MD, FACC††,⁎ ()
- ↵⁎Reprint requests and correspondence:
Dr. Robert H. Jones, Duke Clinical Research Institute, P.O. Box 17969, Durham, North Carolina 27715.
Objectives Our purpose was to develop a risk score to predict in-hospital mortality for percutaneous coronary intervention (PCI) using a statewide population-based PCI registry.
Background Risk scores predicting adverse outcomes after PCI have been developed from a single or a small group of hospitals, and their abilities to be generalized to other patient populations might be affected.
Methods A logistic regression model was developed to predict in-hospital mortality for PCI using data from 46,090 procedures performed in 41 hospitals in the New York State Percutaneous Coronary Intervention Reporting System in 2002. A risk score was derived from this model and was validated using 2003 data from New York.
Results The risk score included nine significant risk factors (age, gender, hemodynamic state, ejection fraction, pre-procedural myocardial infarction, peripheral arterial disease, congestive heart disease, renal failure, and left main disease) that were consistent with other reports. The point values for risk factors range from 1 to 9, and the total risk score ranges from 0 to 40. The observed and recalibrated predicted risks in 2003 were highly correlated for all PCI patients as well as for those in the higher-risk subgroup who suffered myocardial infarctions within 24 h before the procedure. The total risk score for mortality is strongly associated with complication rates and length of stay in the 2003 PCI data.
Conclusions The risk score accurately predicted in-hospital death for PCI procedures using future New York data. Its performance in other patient populations needs to be further studied.
Statistical models have been developed to predict adverse outcomes such as mortality and complications after percutaneous coronary intervention (PCI) (1–11). Using these models, a patient’s risk of adverse outcomes can be predicted based on the presence of pre-procedural risk factors. However, calculating the predicted risk by applying logistic regression models is not simple and requires the use of a calculator or a computer. To simplify the process of risk prediction, risk scores have been proposed (5–8). In these score systems, each risk factor is assigned a score; risk factors scores can be easily summed to calculate a patient’s total risk score. The predicted risk of adverse outcomes for each total risk score is then available to clinicians.
Risk scores provide useful statistical information in a more clinically useful form than corresponding logistic regression models. Most risk scores have been developed from data from a single or a small group of hospitals, and their abilities to be generalized to other patient populations is suspect (5–8). In theory, the generalizability of risk scores could be improved when they are derived from a larger patient population. The objective of this study is to develop a risk score system for the prediction of in-hospital mortality for PCI using the data from the New York State Percutaneous Coronary Intervention Reporting System, which contains all PCI procedures performed in the state.
Database and study population
The New York State Percutaneous Coronary Intervention Reporting System is a population-based registry that collects detailed information on each patient’s demographic characteristics, pre-procedural risk factors, complications, and discharge status. The risk score system was derived from all 46,090 patients who underwent PCI procedures in 41 hospitals in New York State and were discharged in 2002. It was validated using data from all 50,046 PCI patients who were discharged from New York State hospitals in 2003.
A logistic regression model was developed to predict in-hospital mortality using a cross-validation strategy. The first step of the model development was to examine the bivariate relationship between mortality and each pre-procedural risk factor using 2002 PCI data. The risk factors examined included age, gender, body surface area, hemodynamic state, ventricular function, vessels diseased, a variety of comorbidities, and previous interventions. Risk factors that were significantly (p < 0.05) related to mortality were chosen as candidate variables that were used to develop the logistic regression model. Next, the 2002 PCI data were split into two groups of the same population size, with the groups having identical in-hospital mortality rates and similar prevalences of all candidate variables. A stepwise logistic regression model was developed using one group, and significant risk factors (p < 0.1) were used as candidate variables to develop another stepwise logistic regression model in the other group. The variables that remained significant (p < 0.1) were used as candidate variables to fit a logistic regression model using the whole 2002 PCI database. All variables with p values <0.05 were included in the final model. To evaluate the fit of the final logistic regression model, the C-statistic was used to measure discrimination (12) and the Hosmer-Lemeshow test was used to measure calibration (13).
The risk score was developed based on the final logistic regression model using the method described by Sullivan et al. (14). The first step was to break the only continuous variable, age, into four groups: 55 years or younger, 56 to 64 years, 65 to 74 years, and 75 years and older. This was done because the spline functions used for age in the previously mentioned logistic regression model established that the risk of mortality was essentially flat for patients who were under 55 years old, and that risk increased linearly for patients older than 55 years. The youngest age group was used as the base category. Reference values for the other three groups were defined as the differences between their mid-point values and 55. For the oldest age group, the mid-point was between 75 and the 99% percentile for age (88 years), which was used to minimize the influence of extreme values. Then the distance of each age group from the base age category in regression coefficient units was computed by multiplying its reference value by the regression coefficient of age (0.0635). For instance, the distance for the age group of 56 to 64 years was 0.0635 × 5 = 0.3175.
All other risk factors in the model were categorical variables, and the distance between a variable and its base (reference) category in regression coefficient units was equal to the size of the coefficient. Spline functions confirmed that the risk of mortality was essentially constant for patients with ejection fractions above 30% and linear for patients with ejection fractions below 30%, and this led to the choice of three categories (one the base category with a zero score) for ejection fraction in the risk index. The definitions of all risk factors are included in the Appendix.
The constant of the scoring system was defined as the increase of risk in regression units associated with an increase of five years in age (i.e., 0.3175). This constant corresponded to one point in the risk score system. For each risk factor, its distance from the base category in regression coefficient units was divided by this constant and rounded to the nearest integer to get its point value. For example, the point value for the risk factor “left main coronary artery disease” was calculated as 0.8456/0.3175 and rounded to 3.
A patient’s total risk score was calculated by adding up the points for all existing risk factors. For each risk score, the predicted risk of in-hospital mortality (p) was calculated using the method described by Sullivan et al. (14),where −7.6597 is the intercept of the logistic regression model (Table 1)and 0.3175 is the constant used in the scoring system.
The comparison of observed and predicted risk of in-hospital mortality for each risk score unit was used to evaluate the accuracy of the risk score system. The accuracy was first evaluated using 2002 PCI data that were used to develop the risk score. Then the risk score based on 2002 data was applied to 2003 New York data. When the 2003 PCI data were used, the predicted risk of in-hospital death was recalibrated to reflect the differences in patient mix and observed mortality rate between the data used for the development of the risk score and the database to which it was applied. The predicted risk associated with each risk score was recalibrated by multiplying it by the ratio of the observed mortality rate (0.58%) in 2003 and the mortality rate predicted (0.66%) by applying the 2002 logistic regression model to the 2003 patient population.
The correspondence between the risk score and the probability of complications as well as length of stay among patients undergoing PCI in 2003 was also examined. Complications collected in the data system include stroke, myocardial infarction (MI), acute occlusion in the targeted lesion or in a significant side branch, vessel injury at the catheter entry site requiring intervention, renal failure, emergency cardiac surgery, stent thrombosis, and emergency PCI.
All statistical analyses except the hierarchical logistic regression analyses were conducted in SAS version 9.1 (SAS Institute, Cary, North Carolina).
A total of 46,090 PCI procedures were performed in 41 hospitals in New York State in 2002; a total of 321 (0.70%) patients died during their hospital stay. Table 1presents the logistic regression model that was developed to predict in-hospital death using 2002 PCI data. There were nine significant risk factors in the model. Age was represented as a continuous variable, number of years >55; its odds ratio (OR) of 1.07 means that a patient who was over 55 years was 1.07 times likely to die in the hospital compared to another patient who was one year younger. The other eight risk factors were categorical. Two risk factors that were associated with a very high risk of death were shock (OR = 19.92, 95% confidence interval [CI] 11.92 to 33.30) and acute MI (within 24 h before procedure) with stent thrombosis (OR = 18.75, 95% CI 7.27 to 48.37). The model fit the data well in terms of discrimination (C-statistic = 0.886) and calibration (Hosmer-Lemeshow goodness-of-fit test, p = 0.12).
Table 2presents the point scores for all risk factors in the logistic regression model. Point scores ranged from 1 for the age group 56 to 64 years and for female gender to 9 for shock and for acute MI with stent thrombosis. The minimum total risk score was 0 for a patient without any risk factors listed in Table 2, and the maximum possible score is 40. The predicted probabilities of in-hospital death for each risk score ranged from 0.05% for a patient with a score of 0 to 99.36% for a patient with the highest possible score of 40 (Table 3).
To illustrate the application of this risk score, suppose that we have a prospective PCI patient who is a 67-year-old woman, presents with an ejection fraction of 25%, suffered an MI five days ago, and has peripheral arterial disease but none of the other risk factors in Table 2. Then, according to Table 2, the patient has a total risk score of 3 + 1 + 2 + 4 + 2 = 12. In Table 3, the predicted risk of in-hospital mortality for this patient is 2.08%.
Figure 1presents the distribution of patients and the correspondence between observed and predicted risks by risk score group using 2002 PCI data. About half of the patients had total scores of 4 or less with a predicted risk of death <0.2%. Only 0.5% of the patients had total scores of at least 19, and the highest observed score was 31. In general, the observed and predicted risks were very close to each other even for total scores of at least 13 as evidenced by the fact that the predicted risks were within the 95% CIs of the observed risks. Figure 2shows that the correspondence between observed and predicted risks was also good for the 4,974 patients who suffered acute MI, a higher-risk subgroup that had an observed in-hospital mortality rate of 2.87%.
There were 50,046 PCI procedures performed in 45 hospitals in New York in 2003, and 292 (0.58%) patients died in-hospital during or after these procedures. The predicted in-hospital mortality rate was 0.66% when applying the 2002 logistic regression model to the 2003 PCI data. The discrimination of the model was very high (C = 0.905) when it was applied to the 2003 data. The ratio of observed and predicted mortality rates was used to recalibrate the predicted risk for each individual risk score. Figure 3examines the correspondence between observed and recalibrated predicted risks by risk score group using the 2003 PCI data. The predicted risks were relatively close to the observed risks for each risk score group and were always within the 95% CIs of the latter. A good correspondence was also illustrated in Figure 4for the 5,401 patients who suffered an acute MI before PCI and had an observed in-hospital mortality rate of 2.48%.
Figure 5shows that higher risk scores corresponded to higher complication rates among the patients discharged in 2003, who experienced an average complication rate of 2.39%. The complication rate increased from 1.6% for the scores from 0 to 2 to 15.4% for scores of at least 19.
Figure 6demonstrates that the length of stay also continuously increased with the total risk score. The mean length of stay was 2.1 days in 2003. The length of stay was 1.3 days for scores from 0 to 2, and it was 11.7 days for scores of at least 19.
In this study, a risk score system for predicting the risk of in-hospital mortality for PCI was developed based on the data of 46,090 procedures performed New York in 2002. The system was then validated using data collected from 50,046 procedures in 2003. This study had the advantage of using the data collected by a well-established population-based registry, the New York State Percutaneous Coronary Intervention Reporting System, whose accuracy of data is maintained by continuous auditing of medical records (1,9). Unlike other published risk score systems that were developed from data derived from a single or a small group of hospitals (5–8), this new risk score was developed from data of all 41 hospitals who were approved to perform PCI procedures in 2002, and its ability to be generalized to other populations is expected to be high.
The risk score developed in this study was designed to be a handy tool to predict the risk of in-hospital mortality for PCI based on a patient’s pre-procedural risk factors. The point score assigned to each risk factor was derived from a well-fit logistic regression model, which included risk factors consistent with other published models (1–8). For example, all risk factors used in this study to compute the risk score were also included in at least one of four other recent risk score systems. Renal failure was included in all four other systems; age, shock, and history of MI were included in three other systems (5–8). Also, shock and acute MI had the highest scores in the other systems that included them (5–8).
It should be noted that the risk factors in the New York PCI risk score are very similar to the ones used in a coronary artery bypass graft surgery risk score based on New York data (15). Seven of the nine variables (age, female gender, hemodynamic state, ejection fraction, pre-procedural MI, peripheral arterial disease, and renal failure) in the PCI score are the same. The PCI risk score also includes congestive heart failure and left main disease, and the coronary artery bypass graft surgery risk score also includes chronic obstructive pulmonary disease, extensively calcified ascending aorta, and previous open heart operations.
When this risk score (or any risk score) is used in other patient populations for purposes of predicting mortality for individual patients, we recommend, if possible, recalibrating the predicted risk to achieve optimal accuracy. This is because underlying outcomes may be quite different between the development patient population and other populations. Recalibration requires multiplying the predicted risk for each risk score presented in Table 3by the ratio of the observed mortality rate in a new population and its expected mortality rate predicted by the 2002 New York logistic regression model. The expected mortality rate could also be approximated by using the predicted risks in Table 3if computing the predicted probability using a logistic regression model is not feasible. It was demonstrated in this study that the observed and recalibrated predicted risks were close to each other for each risk score in the 2003 PCI data for all PCI patients as well as for those who suffered acute MIs before the procedure.
It was also observed that the risk score proposed in this study was positively correlated with the complication rate and the length of stay in the 2003 PCI data. This finding indicates it can be used as an indicator of the chance of experiencing complications and longer lengths of stay even though it was derived from a logistic regression predicting in-hospital death for PCI patients.
It is worth emphasizing that, although a good risk score can predict adverse outcomes after PCI procedures well on average, it is not designed to precisely predict a single patient’s risk. In addition, an individual patient may have other medical problems not included in the model that are rare but associated with significant risk (e.g., hypoxic coma from a cardiac arrest or newly diagnosed leukemia). Therefore, a risk score should be used as a tool to help physicians and patients to make informed decisions, but not to foresee a patient’s specific outcome.
We look forward to attempts to test the generalizability of the risk score developed in this study by testing it on other patient populations, and to attempt to compare its accuracy with various other risk scores (5–8).
The authors would like to thank Kenneth Shine, MD, the Chair of New York State’s Cardiac Advisory Committee (CAC), and the remainder of the CAC for their encouragement and support of this study; and Paula Waselauskas, Donna Doran, Kimberly Cozzens, and the 45 participating hospitals for their tireless efforts to ensure the timeliness, completeness, and accuracy of the registry data.
For the definition of risk factors in the logistic regression model for PCI in-hospital deaths in New York State in 2002, please see the online version of this article.
- Abbreviations and Acronyms
- confidence interval
- myocardial infarction
- odds ratio
- percutaneous coronary artery intervention
- Received August 25, 2005.
- Revision received September 16, 2005.
- Accepted September 20, 2005.
- American College of Cardiology Foundation
- Ellis S.G.,
- Weintraub W.,
- Holmes D.,
- Shaw R.,
- Block P.C.,
- King S.B.
- O’Connor G.T.,
- Malenka D.J.,
- Quinton H.,
- et al.
- Shaw R.E.,
- Anderson H.V.,
- Brindis R.G.,
- et al.
- Moscucci M.,
- Kline-Rogers E.,
- Share D.,
- et al.
- Singh M.,
- Lennon R.J.,
- Holmes D.R. Jr.,
- Bell M.R.,
- Rihal C.S.
- Qureshi M.A.,
- Safian R.D.,
- Grines C.L.,
- et al.
- New York State Department of Health
- Holmes D.R.,
- Selzer F.,
- Johnston J.M.,
- et al.
- Hosmer D.W.,
- Lemeshow S.
- Hannan E.L.,
- Wu C.,
- Bennett E.V.,
- et al.