Author + information
- Received July 28, 2000
- Revision received October 16, 2000
- Accepted November 29, 2000
- Published online March 1, 2001.
- Fred H Edwards, MD, FACC∗,* (, )
- Eric D Peterson, MD, MPH, FACC†,
- Laura P Coombs, PhD†,
- Elizabeth R DeLong, PhD†,
- W.R.Eric Jamieson, MD, FACC‡,
- A.Laurie W Shroyer, PhD§ and
- Frederick L Grover, MD, FACC§
- ↵*Reprint requests and correspondence: Dr. Fred H. Edwards, Division of Cardiothoracic Surgery, University of Florida Health Science Center, 653-2 West Eighth Street, Jacksonville, Florida 32209-6511
We sought to develop national benchmarks for valve replacement surgery by developing statistical risk models of operative mortality.
National risk models for coronary artery bypass graft surgery (CABG) have gained widespread acceptance, but there are no similar models for valve replacement surgery.
The Society of Thoracic Surgeons National Cardiac Surgery Database was used to identify risk factors associated with valve surgery from 1994 through 1997. The population was drawn from 49,073 patients undergoing isolated aortic valve replacement (AVR) or mitral valve replacement (MVR) and from 43,463 patients undergoing CABG combined with AVR or MVR. Two multivariable risk models were developed: one for isolated AVR or MVR and one for CABG plus AVR or CABG plus MVR.
Operative mortality rates for AVR, MVR, combined CABG/AVR and combined CABG/MVR were 4.00%, 6.04%, 6.80% and 13.29%, respectively. The strongest independent risk factors were emergency/salvage procedures, recent infarction, reoperations and renal failure. The c-indexes were 0.77 and 0.74 for the isolated valve replacement and combined CABG/valve replacement models, respectively. These models retained their predictive accuracy when applied to a prospective patient population undergoing operation from 1998 to 1999. The Hosmer-Lemeshow goodness-of-fit statistic was 10.6 (p = 0.225) for the isolated valve replacement model and 12.2 (p = 0.141) for the CABG/valve replacement model.
Statistical models have been developed to accurately predict operative mortality after valve replacement surgery. These models can be used to enhance quality by providing a national benchmark for valve replacement surgery.
Risk stratification has become an essential element in the practice of cardiac surgery (1). In the field of coronary artery bypass graft surgery (CABG), numerous studies have identified the most important patient risk factors and have confirmed the advantage of risk-adjusted results compared with raw mortality statistics (2–6). Unfortunately, outcomes research in valve replacement surgery has been relatively limited.
Using data from the Society of Thoracic Surgeons National Cardiac Surgery Database (STS Database), we characterized operative outcomes after isolated valve replacement and CABG plus valve surgery. We identified major preoperative predictors of mortality and developed risk models that would accurately estimate operative mortality (OM). We then assessed the performance of these models in a prospective patient population.
The STS Database
The STS Database was initiated in 1986 and now contains detailed clinical information on more than one million registered patients undergoing cardiac surgery. In total, over 487 academic, private, military and Veterans Affairs hospitals from 47 U.S. states and five Canadian provinces participate.
Participants in the STS Database receive individualized, semi-annual, risk-adjusted surgery outcomes, along with direct comparisons with the aggregate database population. This feedback is used for internal quality improvement efforts and is not publicly released. The mortality risk models are an integral part of the participants’ software, which allows the users to generate risk-adjusted outcomes data locally, compared with the national experience.
The models were developed using data on all patients undergoing aortic valve replacement (AVR) or mitral valve replacement (MVR) either alone or in combination with CABG from January 1, 1994 through December 31, 1997. This included those patients receiving either mechanical or bioprosthetic valve replacement, but excluded those receiving valve repair surgery. In addition, patient records were excluded from the study if they were missing age or gender (<0.1% of records), if they were receiving tricuspid or pulmonary valve surgery or if they were having multivalve replacement surgery. In addition, all patients having a diagnosis of congenital heart disease were excluded. Finally, those undergoing cardiac surgery combined with other cardiac procedures were excluded. The resultant database contains clinical information on 92,536 patients. A total of 49,073 patients received isolated AVR or MVR, and 43,463 patients received combined CABG/valve surgery.
The external validation sample consisted of 51,492 patients in the STS Database who received valve replacement from January 1, 1998 to December 31, 1999. In this group, 25,640 patients received isolated valve replacement, whereas the remaining 25,852 patients underwent combined CABG/valve surgery.
Operative mortality is defined as 1) all deaths occurring during the hospital period in which the operation was performed; and 2) those deaths occurring after hospital discharge, but within 30 days of the procedure, unless the cause of death is clearly unrelated to the operation. Definitions of risk factors used in this analysis are provided in the STS web site.
Each of the STS “core data set” risk factors were considered for inclusion into the model. Patient demographic data were compared (Table 1), and univariate OM rates for each risk factor were examined. This information, along with clinical evaluation of the risk factors, was used to determine the model variables. We constructed two models—one that included isolated valve procedures (AVR and MVR) and another that included combined (CABG plus valve replacement) procedures.
For modeling purposes, the median value was used to replace missing values for continuous variables, whereas missing categorical values were replaced with the most prevalent value. With these standard assumptions, all patients in the population were included in the modeling process. For each model, stepwise logistic regression was performed to determine those factors that were most significantly associated with OM. In the stepwise process, variables were entered and removed according to whether the Wald p values were <0.10 for entry and <0.01 for retention. Logistic regression models were then developed in this form: where P is the probability of postoperative death; and X = B0+ B1X1+ B2X2+ … + BkXk, where kis the number of risk factors in the model; Biis the constant associated with the ithrisk factor; and Xiindicates the status of the ithrisk factor for an individual patient.
Assessment of model performance
Model performance was assessed both internally and externally. The internal approach involved direct comparison of predicted versus observed results within the data on which the model was derived. The c-index, which reflects the ability of a model to discriminate patients who died during the operation from survivors, was calculated to evaluate model discrimination. For model calibration, patients were ordered by their predicted OM and then classified into 10 groups of equal size. Average predicted versus observed OM rates were then compared for each of the groups across the risk spectrum.
We also tested the model in a population separate from the derived population. For this independent data set, we selected registered patients undergoing operation in 1998 or 1999. The screening criteria described earlier were applied to all patients in this database, yielding 25,640 patients in the “isolated valve test set” and 25,852 patients in the “CABG plus valve test set.” Average predicted and observed OM rates were then compared for each group. The c-index for the model applied to this population was also calculated.
Patient demographic data
Of the 92,536 patients, 32,968 had AVR; 16,105 had MVR; 32,538 had CABG/AVR; and 10,925 had CABG/MVR. The clinical characteristics of these four patient groups are shown in Table 1. Compared with those receiving combined CABG/valve surgery, those receiving isolated valve procedures (AVR or MVR) were younger and tended to have fewer cardiac risk factors. Patients undergoing MVR were younger, had a smaller body surface area, were more likely to have had previous cardiac surgery and had a higher incidence of congestive heart failure and cardiogenic shock, compared with those receiving AVR.
Outcomes for the four procedural groups are presented in Table 2. Operative mortality was higher in those receiving combined procedures than in those receiving isolated valve surgery. Likewise, OM was higher in those receiving MVR than in those receiving AVR. Higher complication rates and longer postoperative hospital stays were also noted in those receiving combined CABG/valve surgery (vs. isolated valve surgery) and in those receiving mitral (vs. aortic) valve procedures.
Operative mortality risk factors
Certain preoperative clinical characteristics markedly increased the risk of mortality after valve surgery. Table 3provides OM rates in specific subsets of patients receiving valve surgery. As indicated previously, our strategy was to build two models—one for isolated valve procedures and one for combined (CABG plus valve replacement) procedures. Tables 4 and 5⇓show the independent risk factors that emerged for these two models, excluding those factors demonstrating significant interactions. In patients having isolated valve replacement, the factors most strongly associated with an adverse outcome were the need for reoperation, emergency operation and the presence of renal failure (Table 4). These same factors were also the most prominent in the CABG/valve replacement group (Table 5). Although the two models contain similar risk predictors in general, the combined CABG/valve replacement procedure has a number of risk factors traditionally associated with coronary artery disease. In addition, most clinical factors had a similar impact on risk in both AVR and MVR. For the isolated valve model, age was the only risk factor that had a significant interaction with the type of valve replacement, indicating that MVR, but not AVR, is associated with an adverse odds ratio that increases with age. The isolated valve model also included an interaction between emergency/salvage status and age. Emergency/salvage status had less influence on older patients. The CABG/valve model included four risk factor interactions: the risk associated with previous cardiac operation(s), pulmonary hypertension, salvage status and peripheral vascular disease was increased among the MVR group compared with the AVR group. The CABG/valve model also contains an interaction between age and previous cardiac operation(s), indicating that older patients who have had at least one previous cardiac operation have a greater risk of dying compared with younger patients.
Model performance assessment
The c-index was 0.766 for the isolated valve model and 0.739 for the CABG/valve replacement model. Figures 1 and 2⇓⇓compare predicted versus observed results for subgroups across the risk spectrum. Both models demonstrate very good agreement between observed and predicted values.
The “test set” population consisted of 25,640 patients undergoing isolated AVR or MVR procedures and 25,852 undergoing combined CABG/valve surgery between January 1998 and December 1999. Each patient record in the test set was entered into the appropriate model. The c-index for the isolated valve model was 0.773 and that for the CABG/valve model was 0.730. A measure of calibration was determined by using the Hosmer-Lemeshow goodness-of-fit statistic, which was 10.6 (p = 0.225) for the isolated valve models and 12.2 (p = 0.141) for CABG/valve models.
The overall predicted mortality rates for the 1998 to 1999 test set were very similar to the observed mortality rates (Fig. 3 and 4). ⇓⇓For the isolated valve model, the overall predicted mortality rate was 4.8%, compared with an observed rate of 4.7%. For the CABG/valve model, the predicted mortality rate was 8.6%, compared with an 8.2% observed mortality rate.
The quality of care provided by cardiac surgery programs has traditionally been measured by the OM associated with CABG. The use of statistical models of CABG mortality has become commonplace, allowing sophisticated risk stratification of patients undergoing CABG (2–6).
The results of valve replacement surgery should also be considered an important element in the evaluation of cardiac surgery results. Unfortunately, there have been few national or large regional studies of valve surgery comparable to those of coronary surgery. Accordingly, the risk factors associated with valve surgery are more obscure, and acceptable outcomes are more difficult to define. Risk models have only recently been used to evaluate patients undergoing valve replacement surgery (7,8).
The use of a national database has obvious value in this context. The STS Database has detailed preoperative, intraoperative and postoperative information on patients undergoing valve replacement surgery in a wide variety of centers throughout the country. The centers include academic, private, military and Veterans Affairs practices, which, in aggregate, should represent a reasonable estimate of the “average” national experience.
The use of this population to develop statistical risk models should serve as a unique benchmark for quality assurance. With the risk models presented in this report, it becomes possible for centers to obtain an objective and reproducible estimate of OM based on national results. It is instructive to illustrate this concept with an example: a given institution seeking to assess its valve surgery OM may enter its patient population into the STS models to generate a predicted OM for each patient. The average predicted mortality (P%) for the population is then calculated. The following statement is warranted: “Based on the accumulated experience of the STS Database, of every 100 patients with comparable risk factors, P would be expected to die after valve replacement surgery.” The actual OM for the population should be compared with the predicted OM. If there is no significant difference, it is reasonable to conclude that the results of this institution conform with an accepted national average. If there is a significant difference, it should prompt the institution to investigate the issue more thoroughly. The STS holds the position that differences between predicted and actual OM rates do not invariably signal substandard care, but such differences should be a cause for more detailed examination (9).
Comparison with previous studies
The initial attempt to model valve OM using the STS Database was begun five years ago (7). This effort focused on the population from 1986 to 1995, which, in retrospect, constitutes a more heterogeneous group of patients undergoing a broader array of valve procedures. Furthermore, data gathered in the 1980s were not subject to the more rigorous screening criteria imposed in the 1990s, so that a significant portion of the 1986 to 1995 population may contain information that does meet current quality standards. In addition, the STS “core data set” variables had not been developed, so the model risk factors include several variables that are not now considered necessary for collection. These early models served a useful purpose and provided a necessary step in the evolution of our outcomes analyses; however, the present models represent a clear improvement by obviating most of the aforementioned shortcomings.
The pioneering work of the Veterans Affairs Cardiac Surgery Consultants Committee has been particularly important in this field (5,8). The scientific development and the practical application of risk models by this organization have been exemplary, but their population, by design, is restricted to Veterans Affairs centers. The select population of Veterans Affairs hospitals may well preclude application of their models on a broader scale.
Statistical risk models are valuable as quality assurance tools and as aids to medical decision making; however, one should not place an undue premium on their value. Models should not dictate therapy, nor should they arbitrarily brand a practice as “acceptable” or “substandard.” They are designed to serve as aids, not to serve as the answer. Results of risk models should be generally considered as one would regard the results of a laboratory test—one piece of the puzzle that should be evaluated in concert with clinical judgment and other more traditional forms of assessment (9–11).
The broad scope of the STS Database suggests that this aggregate experience defines an acceptable standard of care. Although this is a logical conclusion, it may not be true. It is possible that regional geographic differences and select referral patterns are so disparate that it is unreasonable to hold all practices to a single standard. This issue is presently under investigation by the STS Database Committee. For now, it may be most appropriate to consider these data as a “national average” rather than a “national standard” (9).
Operative mortality is certainly the most important and the most easily measured of surgical outcomes, but other outcomes also have a major impact on patient care. We are presently developing models to predict various forms of operative morbidity, with special emphasis on postoperative neurologic dysfunction and the need for prolonged ventilatory support. Models to predict length of hospital stay and cost are also being investigated.
The STS has assisted with several regional and state initiatives to gather and analyze data on a regional level. These studies will be compared with national data to determine whether significant differences exist. At this point, one state has developed a risk model of CABG mortality based solely on its state population. After extensive testing of this “state model,” it was found to be essentially identical to the national model. Future efforts along these lines should determine whether the national experience with cardiac surgery should be considered as a standard of care or whether regionalization is necessary to establish fair standards.
These STS risk models fill a void in our ability to assess surgical outcomes associated with valve replacement surgery. They allow one to determine the net impact of all important preoperative risk factors to predict the probability of postoperative mortality in patients undergoing valve surgery. This information, in turn, can be used as a benchmark to compare risk-stratified patient groups with a large multi-institutional experience that approximates a national standard of care.
- aortic valve replacement
- body surface area
- coronary artery bypass grafting
- myocardial infarction
- mitral valve replacement
- operative mortality
- Society of Thoracic Surgeons
- STS Database
- Society of Thoracic Surgeons National Cardiac Surgery Database
- United States
- Received July 28, 2000.
- Revision received October 16, 2000.
- Accepted November 29, 2000.
- American College of Cardiology