Author + information
- ↵⁎Reprint requests and correspondence:
Dr. David M. Shahian, Center for Quality and Safety, Bulfinch 270, Massachusetts General Hospital, 55 Fruit Street, Boston, Massachusetts 02114
In this issue of the Journal, Brown and colleagues (1) propose an approach to cardiac surgery patient referral which is based on their understanding of ethical obligation:
Risk-adjusted cardiac surgery mortality rate data for individual cardiac surgeons are currently available for 25% of the U.S. population … surveys of cardiologists and analysis of market share data indicate this information is not being used to refer to cardiac surgeons with the lowest mortality rates. … Because the welfare of a patient will be maximally promoted by referring that patient to the best available surgeon, cardiologists have a responsibility to refer their patients to the surgeon with the best risk-adjusted outcomes. … The only way for cardiologists to fulfill this ethical obligation when referring patients to cardiac surgeons is to refer patients to those available surgeons who have the best outcomes …
Brown et al. (1) assert an ethical responsibility for cardiologists to refer patients to the surgeon(s) with the lowest risk-adjusted mortality rate. With an ethicist and philosopher lending gravitas, this essay has far-reaching implications. Practical considerations notwithstanding, their thesis will have visceral appeal to many stakeholders. How could one possibly argue with the notion that cardiologists should refer their patients to the “best” available surgeon with the “lowest mortality rates”?
We are strong supporters of healthcare transparency and accountability, and we have implemented several public reporting initiatives. We also believe in the value of more objective, evidence-based approaches to shared decision making and informed consent (2,3). However, we have both ethical and methodological concerns with the current proposal of Brown et al. (1). It presupposes an extremely restrictive view of what it means to be the “best,” it makes paternalistic judgments about what should be most important to patients, and its expectations regarding the capabilities of provider profiling are unrealistic. We examine these concerns in reverse order, starting with what one can reasonably expect from statistical profiling.
Provider Profiling: Real or Random Differences
Brown et al. (1) define “best” surgeon solely on the basis of having the lowest risk-adjusted mortality rate. The appealing simplicity of this algorithm quickly evaporates the more closely that we examine it. For example, what exactly do the authors mean when they say the “lowest risk-adjusted mortality rate”?
Medical research is conducted largely on the basis of the principles of inferential statistics. Whether analyzing the results of 2 treatment strategies in a clinical trial or comparing performance among providers, the available data are typically limited to a sample from a larger potential universe of observations. The statistician's objective is to detect the signal within the noise. Frequentist tests of statistical significance (p values and confidence intervals), Bayesian probabilities (with credible intervals), and graphical approaches such as funnel plots have all been used to address this fundamental question. So we return to our original question: What do Brown et al. (1) mean by “lowest risk-adjusted mortality rate” or “best risk-adjusted outcomes”? We assume that they are not suggesting a simple ranking of surgeons by point estimates of mortality, because absolute differences in mortality are often not statistically significant. If they are making their recommendation on the basis of current report cards, then they are most likely referring to low-mortality outliers. These providers have results that are statistically significantly different from the average of other providers, and they are very unlikely to have occurred by chance.
Classification of providers as high- or low-mortality outliers is inherently complex, and numerous methodological limitations must be considered when using such results for profiling, public reporting, or informed consent (4). For the purposes of this commentary, we accept that risk adjustment using audited clinical data is reasonably effective in accounting for the increased risk faced by providers who care for the most severely ill patients. However, there are other equally challenging problems common to all coronary artery bypass graft (CABG) profiling initiatives. First, the event rate (risk-adjusted mortality rate) is low, typically 1% to 3%, yielding relatively few events on which to base statistical inferences. Second, overall sample sizes for an individual surgeon are small if a homogeneous target population (isolated CABG) is chosen, appropriately excluding combined procedures (e.g., CABG plus valve replacement) that are associated with a higher risk of mortality. To mitigate small sample size, many states aggregate multiple years of data, although the number of resulting cases may still be <100 for many surgeons. Other states use statistical estimators that “shrink” the results of low-volume surgeons toward the overall mean, thereby obtaining more precise estimates of their true underlying rates (5–7). The Society of Thoracic Surgeons prefers to profile at the program rather than the individual surgeon level. This largely solves sample size issues, brings substantial peer pressure to bear on underperforming members of a group, and may reduce the potential for risk aversion. It also is more consistent with the team approach that currently characterizes most complex medical care.
Given these practical and statistical constraints, it is not surprising that cardiac surgeon report cards often show wide and overlapping 95% confidence intervals, most of which intersect with unity and are therefore not classified as outlying. Using hierarchical statistical models to account for small sample sizes and clustered observations, there have been no statistical outliers among approximately 60 Massachusetts cardiac surgeons in recent years (5). With a completely different statistical approach, the 2008 to 2009 results for Pennsylvania CABG procedures identified only 3 of approximately 180 cardiac surgeons (1.7%) whose risk-adjusted mortality rates were statistically better than expected (8).
To further complicate matters, even confirmed status as a low-mortality outlier for 1 reporting period does not provide assurance that a surgeon is the “best.” If a particular surgeon were truly the best, one would expect consistency in their superior performance rating over time. In reality, examination of sequential state report cards often reveals that the lowest mortality rate, outlying surgeons change over successive reporting periods. Consequently, if using lowest mortality rate as the sole criterion for cardiac surgeon referral, selection determined on the basis of the most recently published results (which may be 6 months to several years old) does not guarantee that current performance is also superior.
The ultimate lowest mortality rate is 0%, but even this confers no certainty of future performance. Brown et al. (1) assert that it is “extremely unlikely” that referral to a surgeon with the lowest historical mortality rates could result in a worse clinical outcome. In fact, dozens of surgeons in the 2006 to 2008 New York cardiac surgery report cards are reported as having a 0% risk-adjusted mortality rate at a particular hospital (9), but many are determined on the basis of very low volumes and have wide 95% confidence intervals. Aside from the futility of comparing results on the basis of such a small number of cases, regression to the mean suggests that surgeons with a 0% mortality rate on the basis of historical small samples could just as likely have average or worse than average results a year later, as illustrated by the study of Dimick and Welch (10).
It is important to recognize that even statistically significant differences among provider results may not have practical significance. Although not usually the case with physician profiling, it is theoretically possible for almost any small absolute difference to be statistically significant if the sample size is sufficiently large, but such differences may be irrelevant compared with other considerations.
Finally, virtually all report cards use indirect standardization to compute expected mortality rates, and this approach imposes constraints on their interpretation. The resulting risk-standardized mortality rates assess the performance of surgeons for the cases they actually treated. Some surgeons may perform very well given their particular patient cohort, but this does not guarantee a similar level of performance if the same surgeons were confronted with a different (e.g., more complex) mix of patients (11).
The preceding discussion outlines the statistical problems of classifying a provider as having the “lowest mortality rate.” For the sake of argument, assume for the moment that this determination can be made with acceptable accuracy. Given that assumption, would it then be reasonable to equate “lowest mortality rate” with “best”?
Beneficence is one of the most important obligations of physicians to their patients, a foundational principle of medical ethics since the time of Hippocrates (12). Physicians have a responsibility to promote the welfare of their patients, by both doing good and actively avoiding potential harm. On the basis of this principle, it would seem that referral only to the “best surgeon” would be perfectly consistent with this ethical goal. But is beneficence the only ethical principle to be considered?
The goal of beneficence often runs afoul of another equally important ethical principle—patient autonomy—and balancing the two is among the most contentious issues in medical ethics (12). Who should ultimately define what is in the best interest of patients? Before the current era of patient centeredness, promoted by the Institute of Medicine as 1 of 6 key aims to improve health care, the physician was typically assumed to have the answer to this question. Exhibiting various degrees of paternalism, physicians either presumed that their actions were promoting the perceived goals of their patients or, in extreme cases, simply substituted their own values and judgments, believing them to better accomplish what the patient should want.
So how do the potentially conflicting ethical principles of beneficence and autonomy play out in the referral scheme of Brown et al. (1)? On the one hand, it seems completely reasonable to presume that short-term (in-hospital or 30 days) survival is the most important criterion for patients in selecting a provider. But is that necessarily the case? In selecting caregivers for their own families, most physicians we know would not feel compelled to only use a specialist with the lowest reported mortality rate, because they recognize there are many facets to being an outstanding physician. If published mortality rates are not the only criterion that physicians would use to select a provider for their own care, perhaps it is not so unreasonable to ask why this should be any different for their patients.
Consider some real-life scenarios. For example, what if the surgeon with the lowest risk-adjusted mortality rate used operative techniques that resulted in substantially increased incidence of nonfatal but debilitating complications such as sternal infection or stroke? What if the surgeon with the “best” 30-day CABG mortality rates had inferior long-term outcomes because they infrequently used internal mammary artery grafts or inconsistently prescribed important medications for secondary prevention (e.g., aspirin, statin drugs)? What if this “best” surgeon was arrogant, aloof, and never available to answer questions? What if the “best” surgeon did not follow-up with patients and their referring doctors after discharge to coordinate long-term care? What if the “best” surgeon were located hundreds of miles across the state? Any of these considerations might lead even a sophisticated patient to rationally choose a physician other than the one with the “lowest mortality rate.” Patients do, in fact, often choose on the basis of reasoning that physicians may find illogical and frustrating, but which is still their right. For example, Finlayson and colleagues (13) found that patients would often risk 3 to 4 times greater mortality for pancreatic surgery rather than have to travel to a distant regional referral center.
A patient's choice of a physician may involve many considerations, not just short-term survival but also complication rates, patient-reported outcomes, convenience, familiarity, and interpersonal skills. As noted by Professor Judith Hibbard (14), “consumers' conceptualization of quality of care differs from the way in which it is measured and reported publicly.” Interestingly, the current proposal by Brown et al. (1) differs dramatically from the more inclusive, patient-centered approach taken by the lead author in another recent publication (15), which we find much more persuasive:
Historically, end points studied in retrospective analyses of CABG outcomes are those that can be relatively easily measured such as in-hospital or 30-day mortality. However, in my experience, patients (and their families) are equally concerned about different outcomes that are not so easily captured and analyzed but are certainly related to quality—at least from the patient's perspective. That is, they want to know the chances that they are going to survive the operation without major complications such as stroke, cognitive dysfunction, kidney failure, major myocardial infarction, wound infection, ventilator dependence, or the need for a prolonged recovery in a nursing home.
Although it is the only outcome that is publicly reported, mortality is just one dimension of many relevant outcomes. In addition, out-of-hospital outcomes beyond 30 days, which are rarely studied, are also important from the patient perspective. These include survival, return to normal or improved function, relief of symptoms, and avoidance of hospital readmission and additional procedures … . it is possible that patients fear a stroke more than they fear death. There are no data regarding the relative value placed on different adverse outcomes following CABG.
Ultimately, the tension between beneficence and autonomy is best resolved through the process of shared decision making (2). This optimally includes all relevant information that would assist patients in making their decision, presented clearly and impartially by the physician in such a way that their own biases and preferences do not dominate the process.
Justice and Risk Aversion
In addition to beneficence and patient autonomy, the proposal of Brown et al. (1) also affects a third ethical obligation of physicians—justice. If mortality becomes the sole criterion on which to base cardiac surgery referrals, then surgeons may become even more risk averse than has already been observed in some public reporting states (16). This unintended negative consequence may limit access to surgery for 2 classes of patients: those with a high calculated risk of postoperative mortality (e.g., using The Society of Thoracic Surgeons risk model), and those patients belonging to racial or ethnic minorities who may be subjectively regarded by surgeons as being high risk (4,17).
We agree with Brown et al. (1) that there is an ethical obligation to assess and publish provider performance results, although there remains legitimate controversy about whether this should be at the hospital or individual physician level (or both). Such reports must be made more understandable and accessible to consumers (18). Patient-specific risk estimates and provider performance results should be included in the portfolio of relevant information conveyed to patients during the process of shared decision making. Although not emphasized by Brown and colleagues (1), we also believe that ethical conduct by cardiologists always requires full and unbiased discussion of alternative treatment strategies (e.g., percutaneous coronary intervention, CABG), not just the choice of an interventionalist or surgeon once that decision is made.
We do not accept that it is scientifically reasonable, ethically justified, or realistic to require that cardiologists must refer only to the “best” cardiac surgeon as defined by having the lowest risk-adjusted mortality rate. From a statistical perspective, it is difficult to identify significantly high-performing outliers on the basis of relatively small sample sizes and low event rates, and year-to-year fluctuations in results suggest that random variation continues to play a major role even with sophisticated profiling techniques. Consistent low-mortality outlier status over successive reporting periods suggests that the high performance is systematic rather than random, but this is uncommonly observed. Conversely, patients should be wary of surgeons or hospitals with consistently low performance (e.g., high mortality).
Methodological issues notwithstanding, it is unreasonable to define “best” surgeon on the basis of only one outcome, risk-adjusted mortality rate, as important as that may be. Patient autonomy must always take precedence, and patients appropriately consider many different priorities—operative survival, avoidance of serious complications, a compassionate and caring provider, the likelihood of early and sustained return of function, long-term survival, freedom from recurrent symptoms, and convenience. Composite measures of performance that encompass multiple domains of quality must be developed. Not only do they promote the Institute of Medicine goal of multidimensional quality measurement (19), but they also provide additional endpoints and thus facilitate statistical discrimination among providers.
Ideally, any decision regarding referrals must thoughtfully attempt to match the history, clinical presentation, intended procedure, goals, and personality of a specific patient with the experience, capabilities, performance, and demeanor of available surgeons. This approach is most likely to yield the “best” outcome from all relevant perspectives.
Dr. Normand directs the Massachusetts Data Analysis Center (MassDAC), which produces public report cards for CABG and PCI for the Commonwealth of Massachusetts. Dr. Shahian is Senior Surgical Advisor for MassDAC. Both authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- American College of Cardiology Foundation
- Brown D.L.,
- Clarke S.,
- Oakley J.
- ↵MA cardiac surgery report card. http://massdac.org/. Accessed November 29, 2011.
- ↵PA cardiac surgery, 2008-9. http://www.phc4.org/reports/cabg/09/docs/cabg2009report.pdf. Accessed November 29, 2011.
- ↵NY cardiac surgery report, 2006-8. http://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2006-2008_adult_cardiac_surgery.pdf. Accessed November 29, 2011.
- Shahian D.M.,
- Normand S.L.
- Beauchamp T.L.,
- Childress J.F.
- Werner R.M.,
- Asch D.A.,
- Polsky D.
- Committee on Redesigning Health Insurance Performance Measures, Payment, and Performance Improvement Programs