Author + information
- Received December 12, 2006
- Revision received March 9, 2007
- Accepted March 13, 2007
- Published online July 17, 2007.
- Daniel E. Weiner, MD, MS⁎,⁎ (, )
- Hocine Tighiouart, MS†,
- Essam F. Elsayed, MD⁎,
- John L. Griffith, PhD†,
- Deeb N. Salem, MD, FACC‡,
- Andrew S. Levey, MD⁎ and
- Mark J. Sarnak, MD, MS⁎
- ↵⁎Reprint requests and correspondence:
Dr. Daniel E. Weiner, Division of Nephrology, Box #391, Tufts-New England Medical Center, Boston, Massachusetts 02111.
Objectives We sought to determine the utility of the Framingham equations in individuals with chronic kidney disease (CKD).
Background The Framingham equations predict incident coronary disease. The utility of these equations is unknown in CKD.
Methods We pooled individuals without pre-existing coronary disease age 45 to 74 years from the ARIC (Atherosclerosis Risk In Communities) and CHS (Cardiovascular Health Study) trials with CKD, defined by an estimated glomerular filtration rate of 15 to 60 ml/min/1.73 m2. Using gender-specific models, we determined 5- and 10-year risk of incident myocardial infarction and fatal coronary disease, and evaluated discriminative and calibration ability of the Framingham equations for predicting coronary events.
Results There were 577 women and 357 men with CKD. Thirty-five men (9.8%) and 30 women (5.2%) and 74 men (20.7%) and 56 women (9.7%) had cardiac events within 5 and 10 years, respectively; 5-year events were predicted in 6.0% and 1.9% and 10-year events in 13.9% and 4.8% of men and women, respectively. For 5-year events, C-statistics assessing discrimination were 0.62 and 0.77, while 10-year C-statistics were 0.60 and 0.73 for men and women, respectively. Calibration was also poor, with Framingham scores generally underpredicting events in individuals with CKD at 5 and 10 years. Discrimination was significantly improved by refitting models with population-specific coefficients, while recalibration improved prediction in women.
Conclusions The Framingham instrument demonstrates poor overall accuracy in predicting cardiac events in individuals with CKD, although refit models can substantially improve discrimination. Calibration in women can be moderately improved with adjustment for higher event rates. Development of CKD-specific equations is needed.
The Framingham predictive instrument allows clinicians to estimate individual patient risk of incident coronary heart disease by accounting for traditional cardiac risk factors, including gender, age, blood pressure, cholesterol, diabetes, and smoking (1,2). While the Framingham equation has been validated in racially diverse populations including the ARIC (Atherosclerosis Risk In Communities) trial and CHS (Cardiovascular Health Study), its applicability in individuals with chronic kidney disease (CKD) is unknown (3).
Stage 3 to 4 CKD, defined by a glomerular filtration rate (GFR) between 15 and 60 ml/min/1.73 m2, is extremely common in the U.S., with estimated prevalence of 8 million adults (4). Chronic kidney disease is an independent risk factor for cardiovascular disease (CVD), and individuals with CKD have a high burden of CVD risk factors and cardiac events (5,6). The mechanism of increased cardiovascular risk in CKD is uncertain, but is likely secondary to increased severity of traditional CVD risk factors, most notably hypertension and diabetes (7). Nontraditional risk factors, including inflammation, oxidative stress, and anemia, may also contribute (8).
Given the importance of identifying individuals with CKD at highest risk for cardiac events, we evaluated the utility of the Framingham predictive instrument in a predominantly stage 3 CKD population.
We utilized 2 limited-access databases that evaluated CVD in community-based populations: ARIC and CHS. Pooling these population-based studies allowed evaluation of individuals in the Framingham study age range while increasing statistical power and generalizability. From 1987 to 1989, ARIC enrolled 15,792 participants age 45 to 64 years from 4 communities (9). From 1989 to 1990, CHS enrolled 5,201 subjects age 65 years and older in 4 communities; an additional 687 African Americans were recruited from 1992 to 1993 (10). To match the Framingham studies (11,12) we excluded individuals over 74 years old. Data from the limited-access database of the 11th examination of Framingham Heart study cohort and the baseline examination of the Framingham Offspring study (1971 to 1974) were used to reproduce beta-coefficients and survival functions of the Framingham predictive instrument (11,12).
We quantified kidney function as GFR estimated with the 4-variable Modification of Diet in Renal Disease study equation (13–15). We calibrated the ARIC and CHS laboratories indirectly using NHANES (National Health and Nutrition Examination Survey)-III data (16–18). We defined kidney disease as estimated GFR below 60 ml/min/1.73 m2(4). Subjects with GFR below 15 ml/min/1.73 m2were excluded to avoid individuals likely to require dialysis in the immediate future.
Baseline characteristics included demographics (age, gender, race); medical history (coronary heart disease, diabetes, smoking); systolic and diastolic blood pressure; and laboratory variables (total cholesterol, high-density lipoprotein cholesterol, creatinine). Race was defined as white or African American. Cigarette smoking was dichotomized by current use. Diabetes was defined by use of insulin, oral hypoglycemic medications, or fasting glucose level ≥140 mg/dl (7.8 mmol/l) in order to match the original Framingham definition (12). Baseline coronary disease included a history of coronary angioplasty, coronary bypass surgery, and both recognized and silent myocardial infarction as defined by consensus committees for the respective studies (9,10).
After exclusions for missing age, race, gender, or creatinine data or of nonwhite/non-African American race (n = 402), age over 74 years (n = 1,915), and missing baseline coronary heart disease status (n = 317), there were 19,046 subjects. Of these, 1,106 individuals had estimated GFR between 15 and 60 ml/min/1.73 m2. A further 168 with coronary heart disease, 2 with missing follow-up data, and 2 with missing blood pressure or laboratory data were excluded, yielding a final population of 934 individuals.
The primary outcome included myocardial infarction and fatal coronary heart disease. Myocardial infarction was defined by both clinically recognized and silent infarctions (noted on screening electrocardiograms).
Baseline characteristics for the CKD cohort were compared with the Framingham derivation cohort using chi-square and ttests. All p values are 2-sided, and, as in Framingham, all analyses are gender-specific.
Discrimination is the ability of a prediction model to separate those who had events from those who did not have events and was quantified by the C-statistic, analogous to the area under a receiver operating characteristic curve. We obtained the 10-year Framingham survival function from previously published data and reproduced the 5-year survival function for myocardial infarction and fatal coronary heart disease at the mean values of the Framingham risk factors by using individual patient data from the Framingham (11th visit) and Framingham Offspring (baseline visit) datasets, replicating Framingham techniques (2,12,19). Using these survival functions for predicting myocardial infarction and fatal coronary heart disease (defined as “hard” outcomes by Framingham investigators as the more subjective outcome of angina is excluded) and the values for traditional coronary risk factors (blood pressure and lipid categories, age, diabetes, and smoking) from our study population, we utilized coefficients developed by the Framingham investigators to calculate the Framingham risk score for each individual and further derive the 5- and 10-year Framingham probability of a coronary event (2,3,20).
We then created gender-specific “best Cox” models. These utilize Cox proportional hazards regression with covariates identical to those in the Framingham risk score. Coefficients for these covariates are generated based on the results of the predictive model in the CKD population and yield different coefficients than the original Framingham models. For each risk factor, the regression coefficients for the CKD cohort from “best Cox” models and the original Framingham cohort were compared using a 2-tailed z statistic, where z = (b[F]− b[C])/SE. The beta coefficients for individual Framingham and CKD covariates are represented by b[F]and b[C], respectively, while the standard error (SE) is defined as (SE[F]2+ SE[C]2)½. Lastly, we computed 2 gender-specific C-statistics: the first applied the Framingham function to the CKD cohort and the second utilized the “best Cox” model in the CKD cohort. For comparison purposes, we duplicated this technique in individuals (n = 16,689) from the pooled cohort with GFR ≥60 ml/min/1.73 m2and no history of coronary artery disease. C-statistics are compared using a nonparametric approach (21).
Calibration assesses whether predicted outcomes and actual outcomes agree. Individuals with CKD were divided into quintiles of predicted risk based on their Framingham probabilities, and plots of 5- and 10-year predicted and actual events adjusted for informative censoring using Kaplan-Meier estimates were created. Differences between predicted and actual rates were compared using a modified Hosmer-Lemeshow chi-square statistic (3). High chi-square values indicate poor calibration.
Calibration may be poor in cases where the event rate for the population being studied is markedly different than in the Framingham population. In this situation, recalibration is performed, whereby the event rate of the population being studied replaces the event rate of the Framingham population, accounting for a systematic difference in event rates between the 2 populations (3,20). Importantly, recalibration does not affect discrimination. To further account for possible differences, we also calculated calibration using the gender-specific “best Cox” models.
Because of the high risk of mortality as a competing outcome in CKD, such that individuals with a preponderance of cardiac risk factors may die before having a cardiac event, we examined a composite outcome of cardiac events and all-cause mortality (22). As the Framingham equations were designed to predict cardiac events and not all-cause mortality, these analyses were performed using “best Cox” models.
All data were analyzed using SAS Version 9.1 (SAS Institute, Cary, North Carolina). The Institutional Review Board at Tufts-New England Medical Center approved this project.
Among 934 individuals with CKD and no history of coronary heart disease, 577 (61.7%) were women. Mean serum creatinine was 1.5 ± 0.3 mg/dl (133 ± 27 μmol/l) for men and 1.2 ± 0.3 mg/dl (106 ± 27 μmol/l) for women, yielding mean estimated GFR of 52.9 ± 7.5 ml/min/1.73 m2. There were 131 (14.0%) individuals with diabetes and 617 (66.1%) with hypertension (Table 1).Among men with CKD, there were 35 (9.8%) and 74 (20.7%) cardiac events within 5 and 10 years, respectively. Among women, there were 30 (5.2%) and 56 (9.7%) cardiac events, respectively. Additionally, 53 (14.8%) and 126 (35.3%) men, and 54 (9.4%) and 120 (20.8%) women died within 5 and 10 years, respectively (Table 2).
Beta-coefficients and hazard ratios both for the original Framingham cohort and for CKD patients based on “best Cox” models are presented in Table 3.Among men, beta-coefficients significantly differed between Framingham and CKD for both hyperlipidemia and the highest blood pressure group (p < 0.05), with the markedly increased risk associated with these characteristics in the Framingham population not appreciated in CKD patients. Among women with CKD, being in the highest blood pressure groups was associated with significantly increased risk when compared with the Framingham population. Diabetes was associated with a trend toward increased risk in both men and women with CKD when compared with the Framingham population (p < 0.10).
In models using the CKD cohort, discrimination was low in men for both 5- and 10-year probabilities. Models correctly identified individuals who would develop an event 62% and 60% of the time, respectively, in comparison with discrimination of 72% and 69% in the non-CKD cohort pooled from ARIC and CHS and 79% and 73% in the original Framingham cohort. In men with CKD, “best Cox” models significantly improved discrimination.
In models using the CKD cohort, discrimination in women ranged from 77% for 5-year events to 73% for 10-year events; this was similar to that seen in individuals without CKD. In individuals with CKD, discrimination improved with use of “best Cox” models, approaching that seen in the original Framingham cohort (Table 4).
Among men with CKD, the Framingham equation consistently underpredicted cardiac events in quintiles 1 to 4. However, in quintile 5, the Framingham equation overpredicted cardiac events; notably there was 55% mortality in this quintile. Overall 5- and 10-year calibration for men was poor, with chi-square of 33.4 and 71.3, respectively (p < 0.001 for both) (Fig. 1).Among women, the Framingham equations consistently underpredicted events resulting in poor 5- and 10-year calibration, with chi-square of 61.2 and 75.1, respectively (p < 0.001 for both) (Fig. 2).
Recalibrated models performed somewhat better. Although prediction remained poor in men (5- and 10-year chi-square of 13.7 (p = 0.01) and 32.3 (p < 0.001), respectively (Fig. 1), there was no longer a significant difference in predicted and observed events in 5- and 10-year probability models for women (Fig. 2). “Best Cox” models performed well for 5- and 10-year probabilities in both men and women, with chi-square values of 4.2 and 4.0 for men and 0.8 and 2.5 for women, respectively (p > 0.20 for all).
Because individuals with CKD are at increased risk for noncardiovascular mortality and this competing risk may affect cardiac outcomes, we examined calibration of the Framingham equations for a composite outcome of myocardial infarction and all-cause mortality. In gender-specific 5- and 10-year models, the composite event rate increased as Framingham risk rose (Fig. 3).“Best Cox” models performed well for 5- and 10-year probabilities in men, with chi-square values of 2.1 and 2.3 (p > 0.20 for both). In women, models also performed relatively well, with chi-square values of 2.6 and 7.1, respectively (p > 0.10 for both).
In the current study, we used a bi-racial, community-based cohort to assess the utility of the Framingham predictive equation for cardiac events in individuals with CKD. Our findings were as follows: 1) discrimination of the baseline Framingham equation in individuals with CKD, particularly men, is poor; 2) discrimination can be significantly improved with use of “best Cox” models, which utilize the same traditional risk factors as the Framingham equation but assign different weight to each factor, indicating that traditional Framingham risk factors are important in CKD but that these risk factors carry different influence on cardiac events; 3) calibration of the Framingham equation in men and women with CKD is poor; and 4) recalibration, which accounts for a systematic underestimation of cardiac events by the Framingham equation in individuals with CKD, improves accuracy in women but not in men, and this failure in men likely is due to the competing outcome of death.
Discrimination is the ability of a prediction model to separate those who experience a cardiac event from those who do not. The C-statistic expresses the probability of correctly identifying from a random pair (containing an individual who will develop and an individual who will not develop coronary disease) the individual who will develop coronary disease (23). In models using the original Framingham equation, discrimination in men with CKD was only 60% to 62%, whereas in men without CKD it was 69% to 72%, and in the Framingham derivation cohort it was 73% to 79%. Discrimination was better in women but still appreciably worse than in the Framingham population. This is particularly remarkable as individuals comprising the CKD cohort have relatively high GFR (mean 53 ml/min/1.73 m2) and fairly “normal” appearing serum creatinine levels (mean 1.3 mg/dl). Importantly, this cohort is representative of the large majority of the 8 million individuals in the U.S. with CKD. When we assigned different weights to each Framingham covariate with “best Cox” models, discrimination improved significantly, approaching that seen in the non-CKD population and demonstrating that traditional cardiovascular risk factors are important in individuals with CKD but that they differ in magnitude of importance.
Calibration is the measure of how closely predicted outcomes agree with actual outcomes. In models using the original Framingham equation, calibration was poor in both men and women. While the Framingham predictive equations are an integral component of CVD prevention guidelines in the U.S., they tend to overestimate cardiac risk in non-U.S. and minority populations (3,20,24,25). This is often correctable with recalibration for the event rate within the population, reflecting a systematic error of the Framingham equation such that it is either overestimating or underestimating events by a constant factor (20,23,26–29). Although the predictive utility of the Framingham equations in CKD improve after recalibration for higher event rates, with improvement particularly notable in women, recalibrated Framingham equations remained inadequate in men with stage 3 to 4 CKD.
This manuscript dramatically expands on recent findings by our group where we showed that adding a CKD term to the Framingham equations identified additional at-risk individuals but did not improve discrimination in a population that included individuals both with and without CKD (30). Focusing only on individuals with CKD, we explored the relationship between specific traditional risk factors and outcomes in individuals with CKD and found that, although many traditional risk factors remained important in CKD, they carried different weight. While it is not entirely apparent why the Framingham instrument is inaccurate in CKD, the regression coefficients presented in Table 3suggest several hypotheses. In men with CKD, elevated blood pressure and elevated total cholesterol do not carry the same risk that is seen in the Framingham cohort; further, there appears to be no relationship between high-density lipoprotein cholesterol and outcomes, and the import of diabetes appears greater. In women, cholesterol again has less of an effect while diabetes assumes more import and blood pressure has a marked “J”-shaped relationship. The Framingham equations, as currently designed, anticipate that the relationship between risk factors like elevated cholesterol and elevated blood pressure with outcomes will be linear when examined on a population scale. However, in individuals with a chronic condition like CKD, lower blood pressure and lower cholesterol may be identifying individuals with greater infirmity; this is well described in dialysis patients (31). Additionally, the mortality rate in men with CKD is very high. These factors make development of prediction models challenging as: 1) many risk factors for mortality are also risk factors for cardiac events; and 2) in the most infirm individuals, risk factors and outcomes may have an altered relationship. In women, where the mortality rate is lower, this is less important. These altered, nonlinear relationships may reflect subclinical CVD; for example, the coexistence of diabetes and kidney disease identifies individuals with severe enough diabetes to develop end-organ damage. Accordingly, similar findings for the impact of diabetes on cardiac outcomes were appreciated in an assessment of the Framingham risk score in kidney transplant recipients (32). Therefore, these findings may suggest that the dual presence of CKD and traditional risk factors like diabetes and hypertension indicates a greater duration or severity of these risk factors—this would not be accounted for in the Framingham risk score.
Similarly, recalibration failure in CKD may be explained by the competing high mortality rate seen in this population, as individuals who die are also those at highest risk for coronary disease. In subjects with CKD, 10-year mortality exceeded 35% in men and 20% in women, rates nearly 4 times higher than those appreciated in the Framingham derivation cohort. In the lowest 4 quintiles of Framingham risk in men with CKD, observed events reliably exceed predicted events; however, in the highest risk quintile, predicted events far exceeded observed events. This paradox is explained by the effects of mortality and is presented in Figure 3, where, after evaluating a composite outcome of cardiac events and all-cause mortality, 5- and 10-year event rates in men reliably increase as quintile of Framingham risk rises. In women, where for any given age the mortality rate is lower than seen in men, the competing risk presented by mortality is less important and recalibration is more successful.
Our study has several limitations. Because most participants with kidney disease had estimated GFR above 40 ml/min/1.73 m2, we cannot comment on the utility of Framingham equations in more advanced disease. However, it is notable that the vast majority of individuals in the U.S. with CKD and impaired kidney function, nearly 8 million people, fall into this GFR range. We do not have data on microalbuminuria, a component of kidney disease that independently predicts CVD (4,33,34). Additionally, because of the limited size of the CKD population, we lack the statistical power to perform race-specific analyses.
Our study has several strengths. The pooled cohort comprises a large population-based cohort with decreased kidney function and represents a diverse population with a wide age range. Both ARIC and CHS, reflecting their design as community-based cardiovascular studies, have thorough cardiac and mortality event ascertainment. Additionally, we have detailed medical histories and data that allow creation of a cohort of individuals with CKD without known prior coronary disease.
Although traditional risk factors are important predictors of cardiac events in individuals with CKD, the Framingham equations do not accurately weight these risk factors in predicting coronary heart disease events in CKD. Much of this failure is driven by competing events with death as well as the significantly higher cardiac event rate in CKD patients. While it is possible to somewhat improve coronary heart disease prediction in individuals with CKD, particularly in women, both by recalibrating the equations and by assigning different importance to traditional Framingham risk factors, it is likely that future predictive equations with adequate sample size for development and validation of a predictive model in CKD will need to examine both cardiac events and mortality in this high risk population with the goal of identifying shared modifiable risk factors for adverse outcomes.
The ARIC study, CHS, and the Framingham Heart and Framingham Offspring studies are conducted and supported by the National Heart, Lung, and Blood Institute in collaboration with the individual study investigators. Grant support was received by Dr. Sarnak (R21 DK068310), Dr. Elsayed (T32 DK007777), and Dr. Weiner (K23 DK071636). Amgen Inc., Thousand Oaks, California, provided partial support for the creation of the pooled database. The substance of this article was presented at the 2006 Annual Meeting of the American Society of Nephrology in San Diego, California, November 16, 2006.
- Abbreviations and Acronyms
- chronic kidney disease
- cardiovascular disease
- glomerular filtration rate
- Received December 12, 2006.
- Revision received March 9, 2007.
- Accepted March 13, 2007.
- American College of Cardiology Foundation
- Wilson P.W.,
- D’Agostino R.B.,
- Levy D.,
- Belanger A.M.,
- Silbershatz H.,
- Kannel W.B.
- Weiner D.E.,
- Tighiouart H.,
- Amin M.G.,
- et al.
- Sarnak M.J.,
- Levey A.S.,
- Schoolwerth A.C.,
- et al.
- The ARIC Investigators
- Dawber T.R.,
- Kannel W.B.
- Kannel W.B.,
- Feinleib M.,
- McNamara P.M.,
- Garrison R.J.,
- Castelli W.P.
- Levey A.S.,
- Greene T.,
- Kusek J.W.,
- Beck G.J.
- Fried L.F.,
- Katz R.,
- Sarnak M.J.,
- et al.
- D’Agostino R.B.,
- Griffith J.L.,
- Schmid C.H.,
- Terrin N.
- Grundy S.M.,
- Balady G.J.,
- Criqui M.H.,
- et al.,
- American Heart Association
- (2002) Third report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation 106:3143–3421.
- Empana J.P.,
- Ducimetiere P.,
- Arveiler D.,
- et al.
- Thomsen T.F.,
- McGee D.,
- Davidsen M.,
- Jorgensen T.
- Marrugat J.,
- D’Agostino R.,
- Sullivan L.,
- et al.
- Hense H.W.,
- Schulte H.,
- Lowel H.,
- Assmann G.,
- Keil U.
- Kasiske B.L.,
- Chakkera H.A.,
- Roel J.
- Hillege H.L.,
- Fidler V.,
- Diercks G.F.,
- et al.