Author + information
- Received March 31, 2000
- Revision received August 2, 2000
- Accepted December 14, 2000
- Published online March 15, 2001.
- Jack V Tu, MD, PhD, FRCPC∗,e,* (, )
- Peter C Austin, PhD∗,e,
- Randy Walld, BSc†,
- Leslie Roos, PhD†,
- Jean Agras, PhD‡ and
- Kathryn M McDonald, MM§
- ↵*Reprint requests and correspondence: Dr. Jack V. Tu, Institute for Clinical Evaluative Sciences, G-106, 2075 Bayview Avenue, Toronto, Ontario, M4N 3M5, Canada
To develop and validate simple statistical models that can be used with hospital discharge administrative databases to predict 30-day and one-year mortality after an acute myocardial infarction (AMI).
There is increasing interest in developing AMI “report cards” using population-based hospital discharge databases. However, there is a lack of simple statistical models that can be used to adjust for regional and interinstitutional differences in patient case-mix.
We used linked administrative databases on 52,616 patients having an AMI in Ontario, Canada, between 1994 and 1997 to develop logistic regression statistical models to predict 30-day and one-year mortality after an AMI. These models were subsequently validated in two external cohorts of AMI patients derived from administrative datasets from Manitoba, Canada, and California, U.S.
The 11-variable Ontario AMI mortality prediction rules accurately predicted mortality with an area under the receiver operating characteristic (ROC) curve of 0.78 for 30-day mortality and 0.79 for one-year mortality in the Ontario dataset from which they were derived. In an independent validation dataset of 4,836 AMI patients from Manitoba, the ROC areas were 0.77 and 0.78, respectively. In a second validation dataset of 112,234 AMI patients from California, the ROC areas were 0.77 and 0.78 respectively.
The Ontario AMI mortality prediction rules predict quite accurately 30-day and one-year mortality after an AMI in linked hospital discharge databases of AMI patients from Ontario, Manitoba and California. These models may also be useful to outcomes and quality measurement researchers in other jurisdictions.
Because management of coronary disease affects millions of patients worldwide, the assessment of the outcomes of acute myocardial infarction (AMI) using population-based hospital discharge data bases is an important activity. AMI “report cards” listing hospital-specific rates have been publicly released in several U.S. states and some countries in Europe (1–4). However, to properly conduct these studies, an appropriate statistical model must be developed to adjust
for patient case-mix differences between institutions. A few statistical models that have already been implemented using administrative databases to predict AMI mortality are not widely used because they require many variables and complex models and often have not been validated in other jurisdictions (3,5,6).
In Ontario, we published the first hospital-specific AMI “report card” in Canada in 1999 (7). This report contained information on the 30-day and one-year risk-adjusted mortality rates for 52,616 AMI patients at 167 hospitals in Ontario between April 1, 1994, and March 31, 1997. As part of the development of the Ontario AMI report, we created a simple 11-variable prediction rule using the secondary diagnosis fields in the Ontario hospital discharge data bases to adjust for regional and interinstitutional differences in AMI case mix. To evaluate the potential usefulness of this model to clinicians and researchers in other jurisdictions, we tested the model in two completely independent datasets of AMI patients in another Canadian province, Manitoba, and a large area of the U.S., the state of California. This study describes the derivation and validation of the “Ontario AMI mortality prediction rules.”
The Ontario data for the study were taken from the Ontario Myocardial Infarction Database (OMID), which links together all of Ontario’s major health care administrative databases to create a large database for monitoring the quality of AMI care in Ontario. For the present study, we linked data on all patients discharged with a most responsible diagnosis of an AMI (International Classification of Diseases [ICD]-9 code 410) in Ontario between fiscal year 1994 and 1996 (April 1, 1994, to March 31, 1997). The index hospitalization data were obtained from the Canadian Institute for Health Information hospital discharge database while long-term follow-up data were obtained through the Ontario Registered Persons Database, which records the vital status of all Ontario residents. The Ontario discharge data contain 15 secondary diagnosis fields coded using the ICD ninth revision codes and 15 corresponding diagnosis type indicators that indicate whether a diagnosis is a preexisting comorbidity or a complication (Type II diagnosis) occurring after hospital admission. Only diagnoses coded as comorbidities were used in the present study.
Two independent cohorts of AMI patients were created using similar methods from administrative databases in Manitoba at the Manitoba Centre for Health Care Policy and Evaluation, and in California from the California Office of Statewide Health Planning and Development hospital discharge database.
In the California dataset, the “most responsible diagnosis” is specified as the “principal” diagnosis, and secondary diagnosis codes do not distinguish between complications and comorbid conditions.
Similar inclusion/exclusion criteria were used to create the linked AMI datasets in the three jurisdictions. Patients were included in the AMI cohorts if they were admitted with a “most responsible” diagnosis (ICD-9 code 410) of AMI in Ontario or Manitoba, or a “principal” diagnosis of AMI in California. Previous studies have shown the similarities of these two types of diagnoses (8). The exclusion criteria included patients admitted to a noncardiac surgical service, those admitted as transfers from another acute care facility, those admitted with an AMI in the previous year, those discharged alive with a total length of stay of less than four days and those whose AMI was coded as an in-hospital complication. Transferred patients were only counted once based on their first admission, with subsequent admissions linked to the first one. The rationale for these criteria are described elsewhere (9). After these inclusion/exclusion criteria were applied, 52,616 patients were left in the Ontario AMI cohort. A comparable cohort was created using the Manitoba hospital discharge data over the same time frame and yielded a total of 4,386 AMI patients. The California cohort for the same time period consisted of 112,234 AMI patients.
Potential risk factors for prediction model development
Forty-three potential candidate variables in addition to age and gender were considered for inclusion in the AMI mortality prediction rules (Table 1). These candidate variables were taken from a list of risk factors used to develop previous report cards in the California Hospital Outcomes Project and Pennsylvania Health Care Cost Containment Council AMI “report card” projects (3,5). Each of these comorbidities was created using appropriate ICD-9 codes from the 15 secondary diagnosis fields in OMID. The Ontario discharge data are based on ICD-9 codes rather than ICD-9-CM codes used in the U.S., so the U.S. codes were truncated. Some risk factors used in these two projects do not have an ICD-9 coding analog (e.g., infarct subtype, race) and therefore were not included in our analysis. The frequency of each of these 43 comorbidities was calculated, and any comorbidity with a prevalence of <1% was excluded from further analysis. Comorbidities that the authors felt were not clinically plausible predictors of AMI mortality were also excluded. The remaining variables were then entered into a multivariate logistic regression model and backward stepwise regression was used to eliminate variables until only variables significant at the p < 0.05 level were left in the final model. The discrimination of the resulting models was calculated by measuring the area under the receiver operating characteristic (ROC) curve (10).
Prediction model validation
The logistic regression models were developed to predict 30-day mortality and one-year mortality in the Ontario dataset. Separate regression coefficients were fit for 30-day and one-year mortality. We also determined the prevalence of each of the comorbidities using the Deyo adaption of the Charlson comorbidity index score (11)and calculated disease-specific regression coefficients for the Charlson comorbidities so that we could compare the predictive performance of the Charlson model with those in the Ontario AMI mortality prediction rules. The resulting coefficients from the Ontario models were then applied in the independent Manitoba and California AMI datasets to evaluate the generalizability of the models. The areas under the ROC curves were compared in both the derivation and validation datasets. The calibration of the model was assessed by comparing the mean observed and predicted 30-day AMI mortality rates among patients sorted into deciles of ascending risk. All statistical analyses were conducted using SAS version 6.12 (SAS Institute Inc., Cary, North Carolina).
Prevalence of risk factors
A list of the final 11 variables in the Ontario AMI mortality prediction rules is shown in Table 2along with the ICD-9 codes used to generate them. The corresponding 30-day and one-year mortality rates for each of these risk factors is also presented. These results show that congestive heart failure was the most common comorbidity, followed by cardiac dysrhythmias. The presence of shock was associated with the highest one-year mortality rate (82.7%) followed by acute renal failure (70.4%). All of these variables were significant in the univariate analyses for both 30-day and one-year mortality at the p < 0.001 level.
The overall 30-day mortality rate in the Ontario AMI patients was 14.8% and the one-year mortality rate was 23.2%.
Table 3shows the logistic regression coefficients and associated odds ratio (OR) with 95% confidence intervals for both the 30-day and one-year mortality prediction rules. The presence of shock at hospital admission was the strongest predictor of mortality at 30 days (OR = 22.31, 95% CI, 19.30 to 25.79) followed by age >75 years (OR = 12.24, 95% CI, 10.18 to 14.71).
In the Ontario derivation set, both the 30-day and one-year mortality prediction rules performed well with areas under the ROC curve of 0.78 and 0.79, respectively (Table 4). In comparison, a disease-specific Charlson comorbidity index score yielded an ROC area of 0.74 and 0.77 respectively.
Figure 1shows an assessment of the calibration characteristics of the model. There is a high correlation (R2= 0.985) between the predicted and observed 30-day AMI mortality rates within each decile of patient risk.
The performance of the models in the independent California and Manitoba datasets is shown in Table 4. The models predicted well in these datasets, with areas under the ROC curve of 0.77 for 30-day mortality and 0.78 for one-year mortality in both regions. In contrast, the Charlson comorbidity index validated less well in the two external cohorts of AMI patients. We also recalibrated each of the models to develop Manitoba- and California-specific regression coefficients and found that the ORs were similar for each of the 11 variables in the Ontario mortality prediction rules (data not shown).
Hospital report cards have been published in several jurisdictions in North America and are likely to become increasingly prevalent in the 21st century as the public demand for information on health care increases. Acute myocardial infarction is likely to be a focus of many of these reports because it is a very common and highly lethal condition for which there are many effective therapies that lower short-term mortality rates. A simple risk-adjustment model that could be easily generated using administrative databases would be very useful, as it would allow researchers to conduct risk-adjusted outcome analyses across and within jurisdictions. In this study, we developed and validated a simple 11-variable model to predict short- and long-term mortality after an AMI. The model performed well not only in the Ontario AMI patient population from which it was derived, but also in two completely independent cohorts of AMI patients from California and Manitoba.
Other AMI prediction rules
Several prediction rules have been developed by other investigators to predict AMI mortality using administrative databases. However, each of these models has a number of limitations. Normand et al. (6)developed a 40-variable prediction rule using the U.S. Medicare claims database. However, the performance of this model was only 0.72 in terms of its ROC curve area for two-year mortality in the dataset from which it was derived. The developers of the Pennsylvania and California report cards have also developed prediction rules using their administrative databases. However, in both jurisdictions two separate models are required. The Pennsylvania group requires one model for direct admissions and a separate model for transferred-in patients (3). For the California report card, separate models were developed for patients with no prior hospital admissions versus those with prior hospital admissions (5). In contrast, the Ontario AMI mortality prediction rules can be applied to all AMI patients and predict both 30-day and one-year mortality after an AMI using the same set of predictor variables.
Clinical prediction rules
Although AMI outcome prediction rules developed using clinical data remain the ultimate “gold standard” for risk adjustment, it is very expensive and time-consuming to collect these data on a population basis. For these reasons, it is likely that most report cards on AMI care will continue to be developed using routinely collected hospital discharge databases. A recent study using data abstracted from the charts of elderly AMI patients in the U.S. Cardiovascular Cooperative Project yielded a prediction rule with an ROC curve area of 0.79 for 30-day mortality (12), which is only slightly superior to that which we were able to achieve using administrative data. Furthermore, we were also able to demonstrate that this disease-specific prediction rule predicts 30-day AMI mortality with a higher ROC curve area than the Charlson comorbidity index, which is the most commonly used method of adjusting for comorbid conditions using administrative databases (13).
Strengths and limitations of the Ontario rules
The Ontario AMI prediction rules have several strengths. First, they use a relatively small number of variables that can be easily generated using the appropriate ICD-9 codes from hospital discharge databases. The variables in the model are clinically sensible and are similar to those found in other studies. Second, although factors, such as blood pressure at presentation and type of infarct, are not included in the model, other variables in the current model (i.e., shock, congestive heart failure) may be correlated with these factors and contribute to the models’ overall predictive performance. Third, the model has been externally validated in two completely different jurisdictions from which it was derived, which represents a rigorous test of its potential generalizability. Fourth, the rules predict both 30-day and one-year mortality, whereas most other models were designed only to predict short-term mortality.
The Ontario AMI mortality prediction rules also have their limitations. First, we were not able to directly compare their predictions against those that would occur with a prediction rule derived from clinical data. Second, it remains to be established whether risk-adjusted mortality rates calculated using this rule are a marker of better quality in-hospital care (e.g., higher rates of use of aspirin, beta-blockers, thrombolytics). These types of studies are planned in the future.
In summary, we have developed and validated the Ontario AMI mortality prediction rules: simple logistic regression models that predict 30-day and one-year mortality after an AMI using variables that can be easily generated from hospital discharge administrative databases. The models are easy to use, have clinical sensibility and have been externally validated. These models were recently used to generate the first hospital-specific AMI report card in Ontario, Canada’s largest province. They have also been recently adopted for use in the Technological Change in Health Care (TECH) project, which involves the comparison of AMI care using administrative databases from 16 countries around the world (14). We believe the models will also likely prove useful to clinicians and outcomes researchers in other jurisdictions around the world.
The authors thank Pam Slaughter and Mark Cheung for comments on earlier versions of this manuscript and the TECH research network for coordinating the research teams. They also acknowledge the St. Boniface Hospital Research Centre and are indebted to Health Information Services, Manitoba Health, for providing the data from Manitoba.
☆ Supported in part by an operating grant from the Canadian Institutes of Health Research, the National Institute on Aging, TECH grant AG17154, and the Heart and Stroke Foundation of Manitoba. Dr. Tu is supported by a Canada Research Chair in Health Services Research.
- acute myocardial infarction
- International Classification of Diseases
- Ontario Myocardial Infarction Database
- odds ratio
- receiver operating characteristic
- Technological Change in Health Care
- Received March 31, 2000.
- Revision received August 2, 2000.
- Accepted December 14, 2000.
- American College of Cardiology
- Pennsylvania Health Care Cost Containment Council
- Romano P.S.,
- Luft H.S.,
- Rainwater J.,
- Remy L.L.
- Tu J.V.,
- Austin P.,
- Naylor C.D.,
- Iron K.,
- Zhang H.
- Tu J.V.,
- Naylor C.D.,
- Austin P.
- Krumholz H.M.,
- Chen J.,
- Wang Y.,
- Radford M.J.,
- Chen Y.T.,
- Marciniak T.A.