Author + information
- Received July 14, 2010
- Accepted September 2, 2010
- Published online February 22, 2011.
- Frederic S. Resnic, MD, MSc⁎,⁎ (, )
- Sharon-Lise T. Normand, PhD†,‡,
- Thomas C. Piemonte, MD§,
- Samuel J. Shubrooks, MD∥,
- Katya Zelevinsky, BA†,
- Ann Lovett, RN, MA† and
- Kalon K.L. Ho, MD, MSc∥
- ↵⁎Reprint requests and correspondence:
Dr. Frederic S. Resnic, Brigham and Women's Hospital, 75 Francis Street, Boston, Massachusetts 02115
Objectives This study investigated the impact of adding novel elements to models predicting in-hospital mortality after percutaneous coronary interventions (PCIs).
Background Massachusetts mandated public reporting of hospital-specific PCI mortality in 2003. In 2006, a physician advisory group recommended adding to the prediction models 3 attributes not collected by the National Cardiovascular Data Registry instrument. These “compassionate use” (CU) features included coma on presentation, active hemodynamic support during PCI, and cardiopulmonary resuscitation at PCI initiation.
Methods From October 2005 through September 2007, PCI was performed during 29,784 admissions in Massachusetts nonfederal hospitals. Of these, 5,588 involved patients with ST-segment elevation myocardial infarction or cardiogenic shock. Cases with CU criteria identified were adjudicated by trained physician reviewers. Regression models with and without the CU composite variable (presence of any of the 3 features) were compared using areas under the receiver-operator characteristic curves.
Results Unadjusted mortality in this high-risk subset was 5.7%. Among these admissions, 96 (1.7%) had at least 1 CU feature, with 69.8% mortality. The adjusted odds ratio for in-hospital death for CU PCIs (vs. no CU criteria) was 27.3 (95% confidence interval: 14.5 to 47.6). Discrimination of the model improved after including CU, with areas under the receiver-operating characteristic curves increasing from 0.87 to 0.90 (p < 0.01), while goodness of fit was preserved.
Conclusions A small proportion of patients at extreme risk of post-PCI mortality can be identified using pre-procedural factors not routinely collected, but that heighten predictive accuracy. Such improvements in model performance may result in greater confidence in reporting of risk-adjusted PCI outcomes.
- American College of Cardiology National Cardiovascular Data Registry CathPCI
- hierarchical risk prediction models
- percutaneous coronary intervention
- predictive models
Beginning in 2003, Massachusetts followed New York, becoming the second state in the nation to require public release of risk-adjusted mortality statistics after percutaneous coronary interventions (PCIs). The program was established to monitor the quality of invasive cardiac procedures performed within the state and to provide rigorous statistical assessments of any differences in the risk-adjusted mortality among hospitals and individual PCI operators. The Massachusetts program uses Bayesian hierarchical logistic regression models to evaluate the quality of cardiac care as assessed by the in-hospital, risk-adjusted, all-cause mortality after PCI (1). The hierarchical model methodology has been shown to be more specific in the detection of performance outliers than nonhierarchical methods (2,3).
The public release of risk-adjusted mortality reports is controversial (4–8) and can have a significant impact on the hospitals and operators through adverse publicity associated with being identified as a negative outlier in quality performance (8). From the outset of the PCI reporting effort, the Massachusetts program actively engaged interventional cardiologists in the case review and adjudication process for all patients with high-risk clinical features. In addition, clinicians were directly involved in the process of developing the risk-prediction models, based on the covariates available in the National Cardiovascular Data Registry (NCDR) CathPCI Registry (version 3.04) in an effort to establish the maximum confidence by the clinical community in the risk-adjustment methodology. Based on physician input, Massachusetts divided the population of patients undergoing PCI into 2 groups: those presenting with ST-segment elevation myocardial infarction (STEMI) or cardiogenic shock (the shock or STEMI or SOS group), and all other, typically less acute PCI admissions (non-SOS group).
Although the NCDR CathPCI Registry dataset is broadly used as a quality monitoring tool throughout the U.S., it had not been originally intended for nor designed to be applied to mandatory public reporting as used in Massachusetts. Recognizing limitations in the breadth of the variables collected in the existing CathPCI Registry dataset, interventional cardiologists in Massachusetts became concerned that the risk-adjustment models were limited in their ability to account fully for the clinical acuity of the most critically ill patients. There was a specific concern that clinicians could become increasingly risk averse due to fears that the risk-model methodology would not adequately adjust for the highest risk patient populations being treated. The concerns centered on the possibility that uncaptured (and thus unmodeled) covariates, such as impaired neurologic status on presentation, might be independently predictive of in-hospital mortality. Exclusion of such potentially strong, independent predictors of mortality that could vary in prevalence across hospitals would reduce the clinical credibility of the models and might lead to inaccurate prediction of post-PCI mortality.
The concerns raised by Massachusetts clinicians were based, in part, on the observation that there had been a gradual decrease in the proportion of patients treated with PCI for cardiogenic shock in New York in the years since that state began publishing reports of clinician-specific risk-adjusted outcomes (4). Supporting this view, several surveys of New York State PCI operators and cardiac surgeons indicated an increasing unwillingness to treat the most severely ill patients after the introduction of public reporting of mortality outcomes, despite validated risk-adjustment models (7). Moreover, it had been observed that New York interventionalists and cardiac surgeons were unique among geographic locales in providing lower rates of revascularization for patients included in a national cardiogenic shock registry (9), despite mounting evidence of the clinical benefits of early revascularization of most patients presenting in cardiogenic shock (10). In 2008, New York State began excluding patients presenting in cardiogenic shock from the publicly released risk-adjusted mortality reports. In light of these findings, clinical advisors to the Massachusetts public reporting program recommended inclusion of additional high-risk clinical factors not routinely collected in the CathPCI Registry dataset in an effort to improve the performance of the risk-adjustment models as well as to bolster clinician acceptance of the risk-adjustment process.
We sought to examine the feasibility of collecting a set of novel factors likely predictive of post-PCI demise and their impact in the Massachusetts risk-adjusted models of in-hospital mortality.
Patient and hospital populations
We studied all patients age 18 years or older presenting with cardiogenic shock or STEMI and who underwent PCI at any of the Massachusetts, nonfederal, acute care hospitals between October 1, 2005, and September 30, 2007. Because it was possible for patients to undergo >1 PCI during a single admission, we selected the first PCI as the unit of observation in accordance with NCDR risk adjustment methods and to reflect the clinical status of the patient on presentation. Patients were assigned to the hospital where the initial PCI was performed, even if transferred elsewhere for subsequent procedures.
We used clinical and billing data submitted to the Massachusetts Data Analysis Center (Mass-DAC), the data coordinating center located in Harvard Medical School under contract with the Department of Public Health of the Commonwealth of Massachusetts to collect, clean, and analyze the PCI data. Three distinct data sources were used to assemble information. Detailed clinical information obtained during the hospital admission was collected prospectively by trained data managers using variables defined in the NCDR CathPCI Registry dataset (American College of Cardiology, 2010) and submitted electronically every 3 months. Massachusetts hospitals were also mandated by regulation to supplement the NCDR data with additional elements incorporating patient identification information and details regarding the identity of hospitals from which and to which patients may have been transferred. To ensure completeness of the admissions submitted to Mass-DAC, inpatient discharge billing data available from the Division of Health Care Finance and Policy in the Massachusetts Department of Public Health were linked to the Mass-DAC data using patient and hospital identifiers. The billing data included patient demographic information, as well as diagnoses and procedures performed during the hospitalization. Finally, while data managers report patient vital status at discharge, Mass-DAC confirmed this information by linking to the Massachusetts Registry of Vital Records and Statistics.
Beginning October 1, 2005, Mass-DAC added 3 unique covariates to those available in the NCDR CathPCI dataset and collected this information electronically at regular intervals. The new covariates were termed “compassionate use” (CU) criteria and were intended to identify extremely high risk clinical scenarios not currently characterized by the existing NCDR data instrument. The CU criteria included coma on presentation for emergent PCI, use of a percutaneous ventricular assist device or extracorporeal bypass, and cardiopulmonary resuscitation at the initiation of the procedure. Of note, both coma on presentation and cardiopulmonary resuscitation at initiation of the procedure required that the case also be coded as an emergent or salvage procedure, whereas a planned or emergent ventricular assist device–supported procedure would also qualify as CU. Table 1 summarizes the definitions of the additional CU covariates.
All CU cases were independently reviewed, adjudicated, and confirmed by physician reviewers trained in the definitions of items in the NCDR data instrument (e.g., cardiogenic shock, PCI status as emergent or salvage) as well as the CU criteria. All reviewers underwent training in research on human subjects and were approved by the Harvard Medical School's Institutional Review Board. The CU case adjudication process scrutinized every case coded as CU using supplemental clinical documentation forwarded to Mass-DAC. This documentation included emergency department and emergency medical services records (when appropriate), cardiac catheterization laboratory reports, and clinical documentation for the inpatient hospital course. If a physician reviewer was uncertain about a particular adjudication, the case was discussed with the other members of the review committee, and a decision was reached by consensus. An appeal process was established so that medical centers could challenge the adjudication of any particular case that was originally ruled to not meet the CU criteria. All appeals required additional clinical documentation explicitly justifying the use of CU designation. Appeal adjudications were made by board-certified interventional cardiologists with extensive experience in the adjudication process.
The primary patient outcome was mortality, regardless of cause, during the hospitalization (or outpatient procedure, if applicable) in which the PCI was performed. Secondary outcomes included survival at 30 days and 12 months, as assessed by linked data analysis with Massachusetts death index, and assessment of impact of inclusion of CU adjustment on hospital quality classification.
Univariate analyses were performed using the chi-square test, or the Fisher exact test was implemented using SAS version 9 (SAS Institute, Cary, North Carolina) (11). Hierarchical logistic-normal regression models that included random hospital-specific intercepts were used to account for within-hospital clustering and estimated using the WinBUGS application (12). The CU covariate was operationalized as a binary composite variable that assumed a value of 1 if any of the specified conditions was observed to be true; otherwise, it assumed a value of 0. In-hospital mortality prediction models were developed using hierarchical regression models, including and excluding the compassionate use covariate.
The discriminating ability of the model was assessed by the area under the receiver-operating characteristic curve. We calculated the difference in the area under the receiver-operating characteristic curve between a model with and without the CU variables using a paired test (13). Because we were also interested in how the addition of the variables would affect risk prediction, we undertook a reclassification analysis to assess the proportion of patients whose predicted mortality changed by at least 1 quintile of risk (14). This was accomplished by calculating the risk quintile for each patient using both the predictive model excluding and including the compassionate risk covariate. The Net Reclassification Improvement measure that reflects the overall gain in prediction due to the inclusion of the compassionate use covariate was also calculated (15).
Between October 1, 2005, and September 30, 2007, at least 1 PCI occurred in a total of 29,784 admissions in Massachusetts licensed (nonfederal) hospitals. Among these admissions, 5,588 patients (18.8%) underwent a PCI after presenting with shock or STEMI (SOS cases). Within this SOS cohort of patients, a total of 96 patients (1.7% of SOS cases) were adjudicated as having qualified for CU.
CU admissions were significantly more likely to have presented in cardiogenic shock (Table 2) (65.6% vs. 8.2%, p < 0.001), more likely to have received an intra-aortic balloon pump (49.0% vs. 12.5%, p < 0.001), and more likely to have pre-existing renal insufficiency on presentation (15.6% vs. 4.1%, p < 0.001) compared with SOS patients without CU clinical features.
In-hospital outcomes for CU patients were dramatically worse than for the high-risk SOS cohort of patients without CU clinical features as a whole (Table 3). The likelihood of procedural success was significantly lower for CU patients compared with non-CU SOS patients (79.2% vs. 94.2%, p < 0.001). The CU patients were significantly more likely to experience new post-procedure cardiogenic shock and more likely to experience bleeding and renal complications after the index procedure than non-CU patients. The unadjusted in-hospital mortality rate was 15.6 times higher for CU patients than for non-CU SOS patients (69.8% vs. 4.5%, p < 0.001). Although CU patients represented only 1.7% of overall SOS patients, these patients accounted for more than 21% of the overall mortality after PCI in the SOS cohort. Post-procedure death from neurological causes was significantly more frequent in the CU population compared with the standard-risk SOS patients, likely driven by the concentration of patients presenting with coma in the CU cohort. The observed mortality for CU patients with use of percutaneous ventricular support, resuscitation at the start of the procedure, and coma on presentation were 50%, 84%, and 70% respectively. Due to the relatively small individual sample sizes, there was no significant difference in the mortality rates between these 3 categories.
After adjusting for all other known predictors of in-hospital mortality, CU designation was associated with an odds ratio for in-hospital death of 27.3 (95% confidence interval: 14.5 to 47.6) relative to the non-CU SOS patients. The final multiple logistic regression model included patient age, pre-procedure renal insufficiency, documented pre-procedure left ventricular ejection fraction <30%, emergent or salvage procedure status, presence of left main coronary artery disease (of severity >50%), presentation with cardiogenic shock, and the CU indicator (Table 4).
Among those CU patients who survived to hospital discharge, 83% (24 of 29) were alive at 30 days, whereas 76% (22 of 29) were alive at 1 year.
Inclusion of the CU covariates significantly improved the in-hospital mortality risk prediction model performance. The discrimination of the hierarchical mortality prediction model significantly improved from an area under the receiver-operating characteristic of 0.87 to 0.90 (p < 0.001) with preserved goodness of fit. Individual cases were assigned to 5 risk strata according to predicted risk before and after incorporation of CU criteria. Incorporation of the CU covariate led to the reclassification of the risk by at least 1 risk stratum for 347 SOS patients (6.2%). The net reclassification improvement was 8.7% (p = 0.43), indicating that 8.7% more patients who died appropriately moved up a category of risk than down compared with survivors. Most of the impact of inclusion of the new covariate was observed for those who died during the hospitalization (Fig. 1).
In this analysis, overall classification of hospital performance did not change significantly with the inclusion of the CU indicator variable (Fig. 2). However, there were measurable changes in the width and range of the estimated posterior intervals, which made certain centers “closer” to a change in classification as an outlier institution. For example, the posterior interval for hospital 1 was shifted to the left after including the CU indicator, indicating a reclassification of cases, which results in improved estimated quality relative to the overall state performance, as compared with the method that excluded the CU indicator. In contrast, the posterior interval estimate for hospital 12 was shifted to the right after inclusion of the CU indicator, indicating that reclassification of cases resulted in a less favorable estimate of risk-adjusted mortality performance. Therefore, hospital 12 was closer to being identified as an outlier for poorer performance after incorporation of CU indicators than for the models that did not include CU. However, there were no changes in the posterior interval estimates that led to a change in overall classification by institution as either above, below, or within expectations for risk-adjusted mortality, although the observed changes in posterior intervals confirms the potential impact of the inclusion of CU indicators on assessment of hospital quality.
Although the impact on physician perception of the risk-adjustment methods was not directly measured in this study, a temporal association was apparent after the introduction of the CU indicator in October 2005, with a substantial increase in the prevalence of cardiogenic shock among PCI admissions (Fig. 3). The proportion of overall PCI patients presenting in cardiogenic shock monotonically decreased from 2.3% in 2003 to 1.3% in 2005. Beginning with the inclusion of CU criteria in late 2005, this trend reversed, with an increasing proportion of patients treated with PCI for cardiogenic shock for both 2006 and 2007 to a level of 1.7%. Of note, there was no concomitant change to the risk-adjustment methodologies used for predicting mortality after coronary artery bypass graft (CABG) surgery, and the proportion of CABG cases with shock has declined steadily since the inception of public reporting of post-revascularization outcomes (Fig. 3).
The Massachusetts experience demonstrates that a small proportion of patients at extremely high risk of in-hospital mortality can be identified using objective, pre-procedure clinical factors that had not been previously collected as part of traditional quality monitoring efforts. Incorporation of these CU covariates in risk-adjustment models led to significant improvements in model performance as well as reclassification of predicted risk in a substantial proportion of cases. The NCDR dataset was supplemented with CU covariates without significant changes in data collection methods, and these additional variables predictive of mortality were conducive to comprehensive case-level review and adjudication.
In Massachusetts between 2005 and 2007, CU patients represented 1.72% of all high-risk cardiogenic shock or STEMI patients, with an in-hospital mortality rate of 69.8% (vs. 4.45% for other shock or STEMI patients). The adjusted odds ratio for CU was 27.3 (95% confidence interval: 14.5 to 47.6) compared with the non-CU shock or STEMI patients. Nearly two thirds of CU patients were already coded as presenting in cardiogenic shock, indicating the extreme instability of these CU patients on arrival to the cardiac catheterization laboratory. Importantly, the presence of the CU covariate was highly, and independently, predictive of in-hospital mortality, conferring substantial additional risk of in-hospital death beyond that of cardiogenic shock alone. Despite a prevalence of CU criteria in only 1.7% of the analyzed cohort, the inclusion of the covariate significantly improved model discrimination and risk prediction.
Beyond improving the performance of the risk-prediction models, the incorporation of CU covariates was intended to increase the confidence of participating physicians in the risk-adjustment methodology used in Massachusetts. Although the potential association between inclusion of the CU covariate and changes in physician willingness to treat high-risk patients with PCI cannot be directly inferred from the available data, the observed change in proportion of patients treated for cardiogenic shock (Fig. 3) may indicate a positive influence on physician acceptance of the risk-adjustment methodology.
There was debate within the medical and regulatory communities in Massachusetts as to the best approach to handling cases identified as CU. Although many physicians advocated excluding such cases altogether from the risk-prediction models, the regulator community held that CU was an appropriately “behaved” predictor variable for in-hospital mortality, which added to the discrimination of the prediction models and should therefore be included in the final model. The current study confirms the utility of including CU as an independent predictor of in-hospital mortality, and the temporal association of inclusion of CU in the model with the increasing proportion of the sickest patients in cardiogenic shock being treated with PCI may demonstrate a physician acceptance of the improvement of modeling performance and, therefore, patient selection behavior.
The generalizability of incorporation of CU variables into voluntary PCI registries such as the NCDR CathPCI registry hinges on the accuracy of institutionally reported CU status compared with the central adjudication process used in Massachusetts. In the first year of use of the CU criteria, 69 cases were coded by the participating hospitals as CU, of which 49 (71%) were adjudicated as meeting criteria for CU. This proportion of adjudication rejection for a high-risk clinical factor is similar to the Massachusetts experience for the institutional coding of cardiogenic shock and may therefore be representative of the accuracy of institutionally reported high-risk features. Therefore, it appears that incorporation of CU in a voluntary registry may be reasonable, given the current reliance on noncentrally adjudicated high-risk factors with similar adjudication rejection rates.
This analysis was based on clinical data from a mandatory statewide observational registry with a well-publicized and rigorous audit program, and its generalizability is therefore limited. Specifically, each fatal, high-risk, and CU case was individually reviewed and key variables adjudicated by volunteer practicing interventional cardiologists trained in use of definitions from both the NCDR data instrument and the CU variables, resulting in high credibility of the final dataset. However, not all PCI cases and not all variables in the risk prediction models could be uniformly reviewed. Post-PCI mortality occurring after transfer to another facility was also not evaluated.
The incorporation of CU covariates in a statewide clinical outcomes registry is feasible, significantly improves mortality risk prediction model performance, and was temporally associated with an increase in the prevalence of cardiogenic shock among patients undergoing PCI in Massachusetts. Improvements to risk model performance with regard to extremely high-risk patients may provide clinicians, regulators, and health care consumers with increased confidence regarding the results of mandatory public reporting of risk-adjusted PCI outcomes.
The authors thank the members of the Mass-DAC PCI Data Adjudication Committee who participated in numerous data audits, Matthew Cioffi (Mass-DAC) for data management, Treacy Silbaugh (Mass-DAC) for programming expertise, and Hillary Greene (Mass-DAC) for technical assistance.
Dr. Resnic was supported, in part, by the National Library of Medicine (grant NIH R01-LM008142); has received research funding support from The Medicines Company; and has received modest consulting support from St. Jude Medical, Medtronic, and Agfa Medical Imaging. The work of Dr. Normand, Ann Lovett, and Katya Zelevinsky was supported through a contract with the Department of Public Health of the Commonwealth of Massachusetts (620022A624PRE). Dr. Normand is the principle investigator of Mass-DAC; receives modest consulting income from Kaiser Permamente, the Institute of Clinical Evaluative Sciences, American Heart Association, St. Luke's Hospital System, The Medicines Company, Best Practices Project Management, Ingenix Inc.; and receives significant support from Yale Medical School. Ms. Zelevinsky reports being employed (50%) by Mass-DAC, the sponsoring research organization for this study. Dr. Ho receives moderate consulting support as a member of an independent clinical events committee administered by the Harvard Clinical Research Institute that provides adjudication services for patients enrolled in clinical trials sponsored by Boston Scientific. All other authors have reported that they have no relationships to disclose.
- Abbreviations and Acronyms
- coronary artery bypass graft
- compassionate use
- Massachusetts Data Analysis Center
- National Cardiovascular Data Registry
- percutaneous coronary intervention
- ST-segment elevation myocardial infarction
- Received July 14, 2010.
- Accepted September 2, 2010.
- American College of Cardiology Foundation
- Krumholz H.M.,
- Brindis R.G.,
- Brush J.E.,
- et al.
- Resnic F.S.,
- Welt F.G.
- Cutlip D.E.,
- Ho K.K.,
- Kuntz R.E.,
- Baim D.S.
- Apolito R.A.,
- Greenberg M.A.,
- Menegus M.A.,
- et al.
- Cook N.R.