Author + information
- Ronald S. Freudenberger, MD, MBA∗ ()
- ↵∗Address for correspondence:
Dr. Ronald S. Freudenberger, Lehigh Valley Hospital and Health Network, Walter and Hazel May Chair for Excellence in Cardiology, Physician in Chief-Heart Institute, 707 Hamilton Street, 7th Floor, 7B, Allentown, Pennsylvania 18101.
“Derivatives are like sex. It's not who we're sleeping with, it's who they're sleeping with that's the problem.”
—Warren Buffett (1)
Financial derivatives are contracts that derive their value from the use of underlying assets as foundation and building blocks to outcomes. Similarly, medical risk scores derive their values from the underlying derivation cohorts. Use of this approach in both cases confers aggregated risk; therefore, the further one travels from the original cohort or underlying asset, the greater likelihood of inaccuracy and risk. Using risk scores to predict outcomes in heart failure (HF) has become the holy grail of the HF investigator. Between 1994 and 2012, there were >117 different models described in 55 papers aiming to predict mortality or HF hospitalizations (2). Most risk scores and predictive models are derived from cohorts taken from clinical trials, which represent a very select population that is difficult to generalize to the individual patient. Often, the trials that make up the cohort are from a different population and era for which the model is intended to predict outcomes. For example, the commonly used Seattle Heart Failure Model was developed by analyzing 1 cohort of 1,125 subjects in the Prospective Randomized Amlodipine Survival Evaluation study, published in 1996 (3); however, because of its period of study, it had limited use of contemporary therapies (i.e., beta-blockers, implantable cardioverter-defibrillators, and aldosterone antagonists). To accommodate for these deficiencies, the Seattle Heart Failure Model authors decided that “for these medications and devices, benefits were estimated from large, published randomized trials or meta-analyses to determine the β-coefficients (natural log of the hazard ratio) for adding the medication/device to a patient’s regimen.” In spite of these multiple derivations from different cohorts in different eras and estimates of benefits of game-changing therapies, the model performed fairly well in 5 validation cohorts. This model was subsequently used in further derivations of this derivative (4–7), including 1 to predict benefits from implantable cardioverter-defibrillators (8), 1 of the β-coefficients added to the original model. Hence, one can certainly see the danger of generalizing derivatives of derivatives to an individual patient.
Therefore, to ensure the validity of a model, 4 aspects must maintain integrity: 1) the derivation of the original cohort; 2) its generalizability; 3) its ability to discriminate; and 4) accurate calibration. Discrimination relates to the ability of the model to differentiate those patients who had events from those who did not. This is commonly assessed using the C-statistic, which is equivalent to the area under the receiver-operating characteristic curve (9). The calibration and “goodness-of-fit” of a model involve investigating how close the values predicted by the model are to the actual observed values (i.e., does the model work?).
In this issue of the Journal, the paper by Ferriera et al. (10) seems to reflect that, having exhausted the avenue of mortality and hospitalization models, we are at the start of a new approach to mining the data, this time, in search of answers relating to stroke (editorialist included).
The authors of this paper report of a pooled analysis of several large clinical trials involving acute myocardial infarction, patients with HF or moderate left ventricular dysfunction, and systolic dysfunction. The purpose of this analysis is to identify risk factors for stroke and to develop a risk score to ascertain “risk enhancement strategies” for future trials of stroke prevention in this patient population. The dataset is derived from a pooled analysis of 3 studies and an additional dataset used for external validation. The validation cohort includes patients with acute myocardial infarction with signs and symptoms of congestive HF or diabetes mellitus without signs or symptoms of congestive HF (11). The discriminatory ability of this model to predict death or stroke (c-index) is 0.67. Most authors deem model discrimination as poor if the C-statistic is between 0.50 and 0.70, modest if between 0.70 and 0.80, and acceptable if >0.80 (12). Calibration as tested and reported in this model is good with the observed 3-year stroke event rate increasing steeply for each category of the risk score (1.8%, 2.9%, 4.1%, 5.6%, 8.3%, and 10.9%, respectively). This well-constructed risk model is the only such model developed to guide clinicians conducting clinical trial on strategies to enhance the design of future studies in this population.
In this paper, the primary endpoint is stroke (death was a competing risk). The term “stroke” is not consistently defined in clinical practice or assessments of public health (13). As with acute myocardial infarction, there was no “universal” definition of stroke until recently. The term is neither defined nor adjudicated in many HF clinical trials, particularly when stroke is not a primary endpoint. Rather, it is often defined by the patient, family member, study nurse, or cardiologist, so this endpoint is derived from case report forms, which is yet another derivation. Fortunately, in this report, the authors indicate that all stroke endpoints were defined and adjudicated, representing a significant strength of this analysis.
The authors correctly note that in this analysis, patients who received anticoagulants in the judgment of their treating physicians after their myocardial infarction were excluded from the cohort. These patients presumably were at the highest risk of stroke and therefore deemed to need anticoagulation, which may have affected the validity of the stroke risk model.
The utility of CHA2S2DS2-VASc (congestive heart failure, hypertension, age ≥75 [doubled], diabetes, stroke [doubled], vascular disease, age 65–74, and sex category [female]) predicting stroke even in the absence of atrial fibrillation was recently reported (14) by retrospectively evaluating patients who presented with stroke and applying the pre-stroke CHA2S2DS2-VASc score. The current study is the only 1 that identifies post MI, HF patients without atrial fibrillation and provides a score with good calibration.
Given the inherent difficulty in using clinical trials as the derivation cohort in risk models that may be clinically useful, Califf has pointed out that the information contained in the electronic medical record may provide a basis for investigating a more heterogeneous population that can be used to develop better tools with greater generalizability (15). Greater generalizability may result in our ability to apply these tools for population health management to identify those that may require more intensive observation and subsequent clinical decision making, rather than its applicability to individual patients. These risk tools may be helpful but certainly must be used with caution and proper training as with any other tool. As tools, they should be used to help inform clinical decision-making, but not become the absolute arbiter.
↵∗ Editorials published in the Journal of the American College of Cardiology reflect the views of the authors and do not necessarily represent the views of JACC or the American College of Cardiology.
Dr. Freudenberger has reported that he has no relationships relevant to the contents of this paper to disclose.
- 2018 American College of Cardiology Foundation
- ↵Goodreads. Warren Buffet quotes. Available at: https://www.goodreads.com/quotes/871616-derivatives-are-like-sex-it-s-not-who-we-re-sleeping-with. Accessed December 19, 2017.
- Ouwerkerk W.,
- Voors A.A.,
- Zwinderman A.H.
- Adlbrecht C.,
- Hulsmann M.,
- Neuhold S.,
- Strunk G.,
- Pacher R.
- Lanfear D.E.,
- Levy W.C.,
- Stehlik J.,
- et al.
- Bilchick K.C.,
- Wang Y.,
- Cheng A.,
- et al.
- Ferreira J.P.,
- Girerd N.,
- Gregson J.,
- et al.
- Balakrishnan N RCAR,
- Byung-Ho N.
- Sacco R.L.,
- Kasner S.E.,
- Broderick J.P.,
- et al.
- Califf R.M.,
- Pencina M.J.