Author + information
- Received February 3, 1997
- Revision received July 1, 1997
- Accepted July 10, 1997
- Published online November 1, 1997.
- ↵*Dr. Jack V. Tu, Institute for Clinical Evaluative Sciences, G-106, 2075 Bayview Avenue, North York, Ontario, Canada, M4N 3M5.
- for the Steering Committee of the Cardiac Care Network of Ontario1
Objectives. We sought to determine whether more comprehensive risk-adjustment models have a significant impact on hospital risk-adjusted mortality rates after coronary artery bypass graft surgery (CABG) in Ontario, Canada.
Background. The Working Group Panel on the Collaborative CABG Database Project has categorized 44 clinical variables into 7 core, 13 level 1 and 24 level 2 variables, to reflect their relative importance in determining short-term mortality after CABG.
Methods. Using clinical data for all 5,517 patients undergoing isolated CABG in Ontario in 1993, we developed 12 increasingly comprehensive risk-adjustment models using logistic regression analysis of 6 of the Panel’s core variables and 6 of the Panel’s level 1 variables. We studied how the risk-adjusted mortality rates of the nine cardiac surgery hospitals in Ontario changed as more variables were included in these models.
Results. Incorporating six of the core variables in a risk-adjustment model led to a model with an area under the receiver operating characteristic (ROC) curve of 0.77. The ROC curve area slightly improved to 0.79 with the inclusion of six additional level 1 variables (p = 0.063). Hospital risk-adjusted mortality rates and relative rankings stabilized after adjusting for six core variables. Adding an additional six level 1 variables to a risk-adjustment model had minimal impact on overall results.
Conclusions. A small number of core variables appear to be sufficient for fairly comparing risk-adjusted mortality rates after CABG across hospitals in Ontario. For efficient interprovider comparisons, risk-adjustment models for CABG could be simplified so that only essential variables are included in these models.
Assessing the quality of cardiac surgical care through interhospital and intersurgeon comparisons of mortality rates after coronary artery bypass graft surgery (CABG) is an increasingly prevalent phenomenon in the United States as the managed care revolution leads to increasing demands for information on quality of care [1, 2]. For these comparisons to be undertaken fairly, differences in patient case-mix between different providers must be taken into consideration in the relevant statistical analyses. The publication of hospital- and surgeon-specific CABG outcomes data in New York State and Pennsylvania has led to many complaints on the part of the clinicians being assessed that the risk-adjustment methods utilized do not adequately adjust for differences in patient case-mix [3–5]. Many clinicians believe that it is not possible to compare providers fairly until an exhaustive array of surgical risk factors are accounted for in a statistical model. This belief has led to the development of increasingly comprehensive cardiac surgery risk models for assessing the results of CABG [6–9].
In contrast, a recent consensus report from the Working Group Panel on the Collaborative CABG Database Project , hereafter referred to as the Panel, suggests that a large amount of the prognostic information in patients undergoing CABG is contained in relatively few clinical variables. Members of the Panel include representatives from several of the largest multi-institutional cardiac surgery registries in the United States. The Panel has identified and proposed uniform definitions for a list of 7 core variables (i.e., age, gender, acuity of operation, left ventricular function [LVF], previous operation, left main coronary artery disease and number of diseased coronary arteries) that they consider must be present in any database of patients undergoing CABG because they are unequivocally related to operative mortality. The Panel has also identified an additional 13 level 1 variables that they suggest should also be included in the database of every patient undergoing CABG, given that these variables are most likely related to short-term CABG mortality, and 24 optional level 2 variables that have not clearly been shown to predict short-term CABG mortality but have potential research or administrative interest . Whether adjusting for level 1 variables after adjusting for core variables has a significant impact on risk-adjusted mortality rates after CABG has not been empirically addressed.
In Ontario, Canada’s largest province, we have been using data from the Cardiac Care Network (CCN) of Ontario surgical registry to provide feedback to all hospitals providing cardiac surgery on their risk-adjusted mortality rates since 1993 [11, 12]. Surgical “report cards” are released to each surgical center in Ontario on an annual basis as part of a quality improvement program. Case-mix adjustment has been performed using two methods: 1) a simple six-variable cardiac surgical risk index , and 2) slightly more comprehensive logistic regression models . Clinicians in Ontario have responded positively to our outcomes feedback program, although concerns have occasionally been raised about whether the current risk-adjustment methods fairly adjust for differences in patient case-mix because they do not include every surgical risk factor shown to be important in other studies [6–10]. More complex risk-adjustment models may be preferable to some clinicians, but they also require more extensive and costly data collection efforts that may include some unimportant variables . At least in theory, it may be better to collect essential data assiduously rather than to overwhelm data collectors with large numbers of variables and adversely affect data quality. To compare the impact of more comprehensive models versus those limited to crucial risk factors, we conducted a study evaluating the incremental impact of increasingly more comprehensive risk-adjustment models on hospital risk-adjusted mortality rates in Ontario, using the Panel’s classification of surgical risk factors into core and level 1 variables.
1.1 Data Sources
The data for this study were taken from the CCN of Ontario surgical registry, a population-based clinical registry containing prospectively collected data for all patients undergoing cardiac surgery in the province since April 1, 1991. The clinical information in the CCN database has been linked to outcomes (e.g., in-hospital mortality status) and comorbidity information in the Canadian Institute for Health Information (CIHI) administrative database, as described elsewhere [11, 12]. The combined CCN/CIHI database contains information on 12 surgical risk factors, 6 of which would be considered core variables and 6 of which would be considered level 1 variables by the Panel’s classification, as shown in Table 1. Data on the number of diseased vessels (i.e., one-, two- or three-vessel disease), the seventh core variable, were not included in the study because they were not defined in the same manner as that recommended by the Panel. For the present study, we used data from this registry for all 5,517 patients undergoing isolated CABG (without concomitant valve surgery) in Ontario in fiscal year 1993 (April 1, 1993 to March 31, 1994). We chose to conduct our study using the 1993 data because the fiscal 1991 and 1992 data were used in the development of our original risk index . The methods used in the current study were also applied to the 1991 and 1992 data, but the results are not presented here because the findings are similar to those obtained with the 1993 data.
1.2 Risk Factor Definitions
The definitions of the risk factors in the current study are identical to those used by us in previous studies [11, 12]. Emergency surgeryis defined as operation required within 24 h; grade 3and grade 4LVF correspond to a left ventricular ejection fraction of 20% to 34% and <20%, respectively; and recent myocardial infarction(MI) indicates an infarction within 1 week of operation. Left main diseasewas defined as a ≥50% stenosis of the left main coronary artery.
Comorbidities(e.g., diabetes, peripheral vascular disease [PVD], cerebral vascular disease [CVD], chronic obstructive pulmonary disease [COPD]) were determined using the International Classification of Diseases, 9th Revision, Clinical Modification (ICM-9-CM) codes found in the Deyo adaptation of the Charlson comorbidity index [13, 14]. The data in the CCN registry are periodically audited, with the most recent audit indicating a 97.5% agreement rate between the data in the registry and that recorded in patient charts .
1.3 Risk-Adjustment Models
All statistical analyses were conducted using the statistical program STATA 5.0 . Initially, univariate analyses of the 12 surgical risk factors in the CCN/CIHI database were conducted. The prevalence and in-hospital mortality rates of each risk factor were determined. Next, a series of 12 increasingly complex risk-adjustment models for predicting in-hospital mortality after CABG were constructed using logistic regression modeling. Variables were added to these models in a forward stepwise manner on the basis of a prespecified order, as shown in Table 1. This order was chosen for several reasons: 1) Age and female gender were included in the models because these variables represent the minimal level of risk-adjustment that is possible using administrative data sources; 2) emergency surgery, previous CABG, LVF and left main disease were added to the models because they are the other core variables in the Panel’s classification; 3) recent MI, Canadian Cardiovascular Society (CCS) class 4 angina , PVD, CVD, diabetes and COPD were added to the models, in that order. These variables are designated as level 1 variablesin the Panel’s classification. In constructing these models, patient age was modeled using two indicator variables—age 65 to 74 years and age ≥75 years—and LVF was modeled using two indicator variables—grade 3 and grade 4 LVF—although they are counted as one variable in this report.
We evaluated these 12 increasingly complex risk-adjustment models in several ways: 1) We determined the odds ratios and p values associated with the regression coefficient of each variable after each new variable was added to the model. The odds ratiosrepresent the odds of someone with a risk factor dying relative to that of someone in the reference category. 2) We calculated the area under the receiver operating characteristic (ROC) curve of each increasingly complex model in the 1993 data set . The area under the ROCcurve is a measure of the discriminating ability of a model, with higher areas indicating better predictive performance. An area of 1.00 indicates a model that predicts mortality perfectly, whereas an area of 0.50 indicates a model that predicts no better than chance alone. The area under the ROC curve of different models was compared using the methods described by Hanley and McNeil .
1.4 Risk-Adjusted Mortality Rates
A total of nine hospitals provided CABG in Ontario in 1993. We used each of the 12 increasingly comprehensive risk-adjustment models to calculate each hospital’s risk-adjusted mortality rate for CABG in the following manner: 1) The regression coefficients from each risk-adjustment model were used to calculate an expected mortality rate on the basis of the prevalence of patient characteristics at that hospital. 2) The actual mortality rate for that hospital was divided by the expected mortality rate, and the result was multiplied by the overall mortality rate in the province (3.14%) to determine the risk-adjusted mortality rate based on that particular model. The risk-adjusted mortality ratecan be interpreted as the mortality rate a hospital would have if the case-mix at that hospital was similar to the average case-mix in the province. This process was repeated for each of the 12 risk-adjustment models. 3) The change in the different hospital risk-adjusted mortality rates and relative rankings was determined after different levels of adjustment; unadjusted, age and gender adjusted, core variables adjusted and completely adjusted (core and level 1 variables). Both the Pearson correlation coefficient of risk-adjusted mortality rates and the Spearman rank correlation coefficient of relative hospital rankings were determined .
2.1 Mortality Rates
Table 2shows the prevalence and in-hospital mortality rates for 12 surgical risk factors used in the present study compared with the average mortality rate (3.14%) in the province. Emergency surgery, previous CABG, grade 4 LVF and recent MI were associated with the highest in-hospital mortality rates in this analysis.
2.2 Risk-Adjustment Models
The odds ratios and p values associated with each variable’s regression coefficient in the 12 increasingly complex risk-adjustment models are shown in Table 3. Table 3shows how the odds ratios of individual variables changed as the risk-adjustment models were made increasingly complex. The results show that increasing the complexity of the model did not have a major effect on the magnitude of the odds ratio of most variables, with a few exceptions. One of these was emergency surgery, which had an odds ratio of 3.72 when only 3 variables were in the model but an odds ratio of 2.17 after all 12 variables were entered into the model. COPD, CVD and diabetes were not significant predictors (p > 0.05) of mortality when the other factors were adjusted for.
2.3 Area Under the ROC Curve
The area under the ROC curve of each increasingly complex model is shown in Fig. 1. After adjusting for the six core variables, the area under the ROC curve in the 1993 data set was 0.77. Adjusting for an additional six level 1 variables only slightly improved the ROC curve area of the model to 0.79 (p = 0.063). Thus, the marginal gains with level 1 variables beyond the core variables were relatively minor in terms of the model’s discriminating ability.
2.4 Risk-Adjusted Mortality Rates
The effect of the increasingly complex risk-adjustment models on hospital risk-adjusted mortality rates is shown in Fig. 2for the nine cardiac surgery hospitals in Ontario and demonstrates that adjusting for age and female gender alone did not change the adjusted mortality rates significantly from the unadjusted mortality rates. The Pearson correlation coefficient between the unadjusted and the age- and gender-adjusted mortality rates was extremely high (r = 0.997), probably because the distribution of these variables was similar at the different institutions (data not shown). However, adjusting for the four other core variables (i.e., emergency surgery, previous CABG, LVF and left main disease) had a significant impact, with most hospital risk-adjusted mortality rates and relative rankings changing. Once the six core variables were adjusted for, adjusting for the six level 1 variables available in the database had a minimal impact on the risk-adjusted mortality rates and their relative rankings. The correlation between hospital risk-adjusted mortality rates after adjustment for core variables and complete adjustment (core and level 1 variables) was 0.99, whereas the correlation of their relative rankings was 0.98. There was a lower correlation (r = 0.80) between the unadjusted hospital rankings versus the completely adjusted rankings, showing that risk-adjustment did have an impact on overall hospital rankings, although the same hospitals were in the highest and lowest groupings before and after adjustment.
In the present study, we evaluated the effect of increasingly complex risk-adjustment models on hospital risk-adjusted mortality rates after CABG in Ontario, Canada. Our study showed that adjusting for several important clinical predictors of mortality after CABG had a moderate impact on hospital mortality rates and their relative rankings with each other. Adjusting for the six core variables suggested by the Working Group Panel on the Cooperative CABG Database Project accounted for most of the interhospital variation in mortality rates that was attributable to case-mix differences at the different hospitals providing cardiac surgery in Ontario. Incorporating six additional level 1 variables into a model that already had six core variables had minimal impact on the discriminating ability of the model or hospital risk-adjusted mortality rates or their relative ranking. The results of our study suggest that simpler models may be just as effective as more complex models for interprovider comparisons of the short-term mortality risks of CABG. They also support the recommendation that data collection efforts should be directed toward accurately collecting a small number of the most important prognostic variables in patients undergoing CABG .
3.1 Related Studies
To the best of our knowledge, a study similar to the current study has not previously been conducted. Hannan et al. conducted a related study comparing the hospital rankings achieved by risk-adjusting using clinical data from New York State’s Cardiac Surgery Reporting System (CSRS) and administrative data from New York State’s Statewide Planning and Research Cooperative System (SPARCS). They concluded that a risk model based on the clinical database was a better predictor of mortality after CABG and that there was only a moderately high correlation (r = 0.75 to 0.80) between hospital rankings using adjustment from the clinical and administrative databases. They attributed the discrepancy to three factors unique to the clinical database: 1) left ventricular ejection fraction, 2) reoperation, and 3) left main disease. Our study also suggests that these three factors have an important impact on hospital risk-adjusted mortality rates and that these factors should be part of any risk-adjustment model for assessing the short-term results of CABG.
3.2 Are More Risk Factors Better?
Despite our findings and the conclusions of the Panel, the trend in the area of cardiac surgery risk assessment appears to be toward increasingly more complex models. For example, the Society of Thoracic Surgeons database , the largest surgical registry in the United States, with >706 participating hospitals, uses a Bayesian risk-adjustment model with 23 variables. A recent report from Magovern et al. from Allegheny General Hospital also suggested using a risk model with 24 variables. Whether there are any significant marginal predictive benefits with these more complex models remains to be determined.
The risk-adjustment models presented in the current study are similar to those previously published by us using data from 1991 and 1992 [11, 12]. Our previous work showed that mortality, intensive care unit stay and overall postoperative length of stay after cardiac surgery can be predicted by using a simple, additive six-variable risk index containing: 1) age, 2) gender, 3) urgency of operation, 4) type of operation, 5) LVF, and 6) reoperation. The results of the current study also support the conclusion that most prognostic information in patients undergoing cardiac surgery can be captured by a few key variables.
Why do many clinicians believe that more complex models must be better (yet the results of our current study suggest that simple models may be just as effective)? Several explanations of a statistical nature may be given. For any variable to have a significant impact on a hospitals’ risk-adjusted mortality rate, it must have at least three characteristics: 1) It must be a statistically significant predictor of the outcome; 2) the distribution of the variable must be significantly different among the hospitals being compared; 3) it must be present in the database in a relatively high frequency and not be highly correlated with other factors in the database that are already included in a risk-adjustment model. Our study demonstrates some of these principles. For example, age and female gender were both highly statistically significant predictors of in-hospital mortality, yet adjusting for these variables minimally changed hospital risk-adjusted mortality rates because the distribution of these variables is similar at the nine cardiac surgery hospitals in Ontario. Another risk factor, recent MI was also highly significant, but its prevalence was only 2.2% in the database, and thus, adjusting for this variable did not have a major impact. Including too many variables in a model can lead to the statistical phenomenon of “overfitting,” with a more complex model paradoxically performing worse than a simpler model in an independent test data set . The risks of “upcoding” of risk factors and the expense of data collection are also directly proportional to the number of variables in a risk-adjustment model.
3.3 Study Limitations
Our study has certain limitations: 1) We did not have data for all the level 1 variables listed in the report by the Panel (e.g., creatinine levels, mitral regurgitation), and it is possible that some of these other level 1 variables may have had a greater impact on hospital risk-adjusted mortality rates than those that were available for use in our study. 2) Although involving complete data from nine centers, our study was conducted in one jurisdiction only, and the results need to be verified in other jurisdictions. The impact of risk-adjustment may be less in a regionalized system such as Ontario, where all centers have a high surgical volume and serve a relatively well defined catchment area. 3) We did not address the issue of the effect of increasingly complex models on risk assessment at the individual patient level. More complex models could potentially provide a more accurate assessment of mortality risk in high risk patients, even if they do not affect overall results at the hospital level. 4) We analyzed only the outcome of in-hospital mortality in this study, and the ability of more comprehensive models to predict other outcomes (e.g., stroke, post-CABG MI) was not assessed.
Our study suggests that a small number of core variables (i.e., age, gender, emergency operation, previous CABG, LVF and left main disease) appear to be sufficient for fairly comparing hospital risk-adjusted mortality rates after CABG in Ontario. Including additional variables beyond a core set of variables in a risk-adjustment model did not have a substantial impact on the overall results. Residual differences in hospital mortality rates may be a function of random variation, unmeasured case-mix differences or differences in quality of care. Our findings should be of value to clinicians interested in using simple models for monitoring and improving the results of CABG. Further studies will be required to assess the generalizability of our findings and to determine an optimal set of risk factors for accurately assessing the short-term mortality risks of CABG.
We thank Francine Duquette for assistance in preparing the figures. We also thank all the cardiovascular medical and surgical practitioners, nurses and registry personnel who comprise the Cardiac Care Network of Ontario.
A.1 Steering Committee of the Cardiac Care Network of Ontario
Sudbury: S. Sewa Aul, MB, Sudbury Memorial Hospital. Ottawa: Donald S. Beanlands, MD, Lorna Bickerton, BScN, University of Ottawa Heart Institute. Toronto: Robert Chisholm, MD, Jeffrey Lozon, MHA, St. Michael’s Hospital; June Hylands, BA RN, Hospital for Sick Children; Chris Lai, MD, Thunder Bay Regional Hospital; Barry J. Monaghan, BComm DHA, West Park Hospital; Christopher D. Morgan, MD, Sunnybrook Health Science Centre; George H. Pink, PhD, University of Toronto; Hugh Scully, MD, Toronto Hospital. Kingston: W. John S. Marshall, MB ChB, Kingston General Hospital. London: Martin Goldbach, MD, Victoria Hospital; Neil McKenzie, MB ChB, University Hospital. Hamilton: B. William Shragge, MD, Hamilton General Hospital. Willowdale: Mark A. Vimr, MScN, Cardiac Care Network of Ontario. Brampton: Bob Youtz, Peel District Health Council.
↵fn1 Dr. Naylor is supported by a Career Scientist Award from the Ontario Ministry of Health, Toronto, Ontario, Canada. This work was supported by an operating grant from the Sunnybrook Trust for Medical Research, North York, Ontario, Canada.
- coronary artery bypass graft surgery
- Cardiac Care Network of Ontario
- Canadian Cardiovascular Society
- Canadian Institute for Health Information
- chronic obstructive pulmonary disease
- cerebral vascular disease
- left ventricular function
- myocardial infarction
- peripheral vascular disease
- receiver operating characteristic
- Received February 3, 1997.
- Revision received July 1, 1997.
- Accepted July 10, 1997.
- The American College of Cardiology
- (1995) A Consumer Guide to Coronary Artery Bypass Graft Surgery: Hospital and Physician Comments, Vol. IV, 1993 (Pennsylvania Health Care Cost Containment Council, Harrisburg (PA)).
- Parsonnet V,
- Dean D,
- Bernstein AD
- Magovern JA,
- Sakert T,
- Magovern GJ,
- et al.
- Jones RH,
- Hannan EL,
- Hammermeister KE,
- et al.
- ↵Tu JV, Jaglal SB, Naylor CD, for the Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Circulation 1995;91:677–84.
- ↵Tu JV, Naylor CD, for the Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Coronary artery bypass mortality rates in Ontario: a Canadian approach to quality assurance in cardiac surgery. Circulation 1996;94:2429–33.
- ↵(1992) The International Classification of Diseases, 9th Revision, Clinical Modification: ICD-9-CM (Commission on Professional and Hospital Activities, Ann Arbor (MI)).
- ↵(1997) Stata Statistical Software: Release 5.0 (Stata Corporation, College Station (TX)).
- Campeau L
- Rosner B