Author + information
- Received August 31, 2011
- Revision received November 10, 2011
- Accepted December 17, 2011
- Published online June 19, 2012.
- Edward L. Hannan, PhD⁎,⁎ (, )
- Kimberly Cozzens, MA⁎,
- Spencer B. King III, MD†,
- Gary Walford, MD‡ and
- Nirav R. Shah, MD§
- ↵⁎Reprint requests and correspondence:
Dr. Edward L. Hannan, School of Public Health, State University of New York, University at Albany, One University Place, Rensselaer, New York 12144-3456
In 1988, the New York State Health Commissioner was confronted with hospital-level data demonstrating very large, multiple-year, interhospital variations in short-term mortality and complications for cardiac surgery. The concern with the extent to which these differences were due to variations in patients' pre-surgical severity of illness versus hospitals' quality of care led to the development of clinical registries for cardiac surgery in 1989 and for percutaneous coronary interventions in 1992 in New York. In 1990, the Department of Health released hospitals' risk-adjusted cardiac surgery mortality rates for the first time, and shortly thereafter, similar data were released for hospitals and physicians for percutaneous coronary interventions, cardiac valve surgery, and pediatric cardiac surgery (only hospital data). This practice is still ongoing. The purpose of this communication is to relate the history of this initiative, including changes or purported changes that have occurred since the public release of cardiac data. These changes include decreases in risk-adjusted mortality, cessation of cardiac surgery in New York by low-volume and high-mortality surgeons, out-of-state referral or avoidance of cardiac surgery/angioplasty for high-risk patients, alteration of contracting choices by insurance companies, and modifications in market share of cardiac hospitals. Evidence related to these impacts is reviewed and critiqued. This communication also includes a summary of numerous studies that used New York's cardiac registries to examine a variety of policy issues regarding the choice and use of cardiac procedures, the comparative effectiveness of competing treatment options, and the examination of the relationship among processes, structures, and outcomes of cardiac care.
By 1988, Dr. David Axelrod, the innovative New York State Commissioner of Health, had become concerned about the frequently 5-fold variation in hospital mortality rates for coronary artery bypass graft (CABG) surgery in the state. Unfortunately, the hospital-level information available at that time was woefully inadequate in terms of explaining this variation. The dilemma facing the Commissioner and the State's Cardiac Advisory Committee (CAC), a group that was charged by the New York State Department of Health (DOH) to help oversee the quality and provision of cardiac care, was to assess relative quality of care taking into account that some hospitals may have been treating much sicker patients than other hospitals.
A precedent for solving this problem was the Health Care Financing Administration (HCFA), the predecessor of the Center for Medicare and Medicaid Services (CMS), mortality studies, which were released annually between 1986 and 1992 (1). These studies used Medicare administrative data to assign medical and surgical patients into diagnostic groups. Each group was analyzed separately, and risk-adjusted mortality rates were developed and released to the public for each hospital/group.
The HCFA mortality studies were discontinued in 1992 in the face of considerable criticism regarding the use of administrative data, the grouping of patients, and several other criticisms/concerns. Because of this reaction to the use of administrative data for evaluating hospital performance, the DOH decided to create a patient-level clinical database that could be used to assess institutional outcomes for CABG surgery while taking into account interhospital differences in patient acuity. This decision appears to have been wise because subsequent studies using the registry data combined with administrative data demonstrated that the registry data are more predictive of mortality than administrative data, and that the 2 types of databases arrive at somewhat different conclusions about relative hospital quality (2,3).
The CAC perused the current literature to identify patient risk factors that were related to short-term adverse outcomes for CABG surgery, and these risk factors were included in the data system along with demographics, complications of care, admission and discharge dates, procedures performed, and patient disposition at discharge. The new registry was used for the first time to assess hospital performance in a 1990 paper published in the Journal of the American Medical Association (JAMA) (4). This study identified independent significant risk factors for CABG/valve surgery in-hospital mortality, as well as the observed, expected, and risk-adjusted mortality rates and volumes for all of the 28 hospitals (numbered from 1 to 28 but not named) in the state that were approved through Certificate of Need to perform these procedures (4). On the same day that the JAMA paper was published, Dr. Axelrod released the names of the hospitals along with their risk-adjusted mortality rates to the New York Times (5). Although this was the first public release of the data, hospitals had received their own data for the first half of 1989, and the 1990 release noted that the mortality rates for the first half of 1990 had declined by 14% from the first half of 1989.
Shortly after the publication of hospital mortality rates, the Long Island newspaper Newsday pressed the DOH to release surgeon-specific mortality rates to hospitals and the public. The DOH was reluctant to do this, in part because surgeon volumes are lower than hospital volumes, and there is a danger of outcomes being misleading as a result of lower statistical power. Newsday sued the DOH for surgeon-specific information, and prevailed in the law suit. The CAC then recommended that hospitals submit the data in a manner that would make it impossible to identify individual surgeons. The final resolution was to release surgeon-specific outcomes on a rolling 3-year basis for surgeons with >200 cases in that time frame (to accumulate enough cases so there would be adequate statistical power in assessing performance; now, all surgeons with at least 1 procedure in each of the 3 years are also included) (6). The first surgeon level data were released in December 1992.
In 1992, a coronary angioplasty reporting system, now called the Percutaneous Coronary Interventions Reporting System (PCIRS) was created, and annual PCIRS reports have been released since 1997. The Pediatric Cardiac Surgery Reporting System was developed in 1991, and reports are released periodically.
Methodology for Collecting and Reporting Data/Format of Reports/Information Reported
Hospitals with Certificate of Need approval to perform the invasive cardiac procedures in New York submit data for CSRS and PCIRS to the DOH Cardiac Services Program/University at Albany School of Public Health for cleaning, oversight of auditing, and analysis. The data elements and their definitions are determined by CAC subcommittees of cardiac surgeons and cardiologists, revised periodically, and sent to the hospitals. All hospitals have clinical data coordinators who interact with the DOH staff through biannual training and information sessions and through open access by telephone or e-mail on specific questions.
Assuring data accuracy
Cleaning consists of minimizing the amount of missing data through interactions with hospitals, and ensuring that all patients undergoing the procedures are in the registry and that their discharge disposition is accurate. This is done by matching the data with the DOH's administrative hospital database, the Statewide Planning and Research Cooperative System. Auditing consists of inspection of risk factor coding in hospital medical records by the DOH's utilization review agent. Hospitals are chosen for auditing on the basis of time since last audit, problems identified in the last audit, and high reported prevalence of important risk factors in relation to the statewide average reporting rate for those risk factors. Additional auditing consists of medical record review by Cardiac Services Program staff to verify accuracy of selected risk factors and procedures reported. When minor data accuracy problems are detected, hospitals are asked to re-abstract the data and provide accompanying documentation; major problems have sometimes led to the DOH requiring the hospital to pay for the cost of an independent abstractor.
Although this auditing process delays the production of public reports, hospitals are aware of their own results as they are submitted. First, because the risk model changes minimally from year to year, hospitals have immediate feedback on their approximate risk-adjusted mortality rates for internal hospital quality improvement purposes. Second, the DOH sends alert letters to hospitals whose mortality is high during the course of the year in between reports. These letters have frequently been the impetus for change within hospitals.
The annual reports contain each hospital's volume, unadjusted in-hospital/30-day mortality rate, expected mortality rate, risk-adjusted mortality rate, and outlier status (significantly higher, significantly lower, not different from the statewide mortality rate) for surgeons/interventional cardiologists every year on a rolling 3-year basis to accumulate more data. For cardiac surgery, CABG and valve with or without CABG are reported, with the valve data reported on a rolling 3-year basis (7,8).
Risk-adjusted data are obtained by calculating an expected probability of mortality for each patient using a logistic regression model. Backward stepwise elimination methods are used along with training and validation samples to cross-validate the models. The final model is estimated on the full dataset, and the coefficients of the models are used to obtain the probability of mortality for a given patient as a function of his or her own risk factors. The probabilities of mortality are averaged across all patients for each provider to obtain that provider's expected mortality rate, and the ratio of the provider's expected and crude mortality rates is multiplied by the statewide rate to obtain a risk-adjusted rate for the provider (7,8). The 95% confidence intervals for risk-adjusted rates are constructed to identify high and low outliers as hospitals with respective confidence intervals entirely above and below the statewide mortality rate.
In more recent years (since 2004), short-term mortality is defined as death occurring in the index admission or death within 30 days of the index procedure, either in or out of the hospital. In the earlier years, in-hospital mortality was used as the outcome measure because of the inability to obtain data on out-of-hospital deaths.
Hospital quality improvement activities related to registry information
Although many hospitals have not been responsive to information supplied in the release of risk-adjusted outcomes information from the cardiac registries, there have been several reported hospital-specific quality improvement initiatives related to the release. Three of these initiatives are described in the following text.
St. Peter's Hospital in Albany, New York, was identified as having significantly higher than expected mortality in the early years of the program (1991 and 1992). The excess mortality was discovered to be a result of emergency cases, for which St. Peter's experienced a 26% mortality rate, compared with a 7% rate for the state. Elective and urgent cases at St. Peter's had roughly the same mortality as the statewide rate. After a multidisciplinary review of its management of emergency cases, St. Peter's concluded that the patients were not being sufficiently stabilized before surgery. That led to major changes in the management of these patients, and in 1993, there were no deaths among the 54 emergency patients who underwent CABG surgery (9).
Winthrop Hospital, Mineola, New York, had one of the highest risk-adjusted mortality rates in the state in the first public report, and an outside review commissioned by the DOH led to probation for its cardiac surgery program. The hospital hired a new chief of cardiac surgery, who concentrated the service on a single floor, hired clinical nurse specialists and physician assistants who were dedicated to cardiac surgery, reviewed each case pre-operatively, and installed a dedicated cardiac anesthesia service (10). Winthrop's risk-adjusted mortality rate dropped from 9.2% to 4.6% to 2.3% between 1989 and 1991 (Edward Hannan, personal communication, April 2012).
Erie County Medical Center, Buffalo, New York, was another hospital that experienced high-risk adjusted mortality in the early years of the program, with the highest risk-adjusted mortality in the state for the first 6 months of 1989. A site visit by the CAC led to a series of recommended changes, and the hospital voluntarily suspended operations in 1990 to implement these recommendations. Changes included the establishment of a cardiac surgery quality assurance program, credentialing and ongoing evaluation of surgeon performance, dedicated cardiac anesthesiologists and cardiac intensive care beds, and the recruitment of a permanent, full-time service chief (10). The new chief of cardiac surgery hired new operating room nurses, cardiopulmonary bypass technicians, and intensive care staff who were all dedicated to cardiothoracic surgery. The hospital's risk-adjusted mortality fell from 7.31% in 1989 to 1991 to 2.51% in 1993 to 1995, just below the statewide average of 2.57%. The very low annual volume of just over 100 cases rose to 219 cases per year in the 1996 to 1998 time period, during which the mortality was 1.77% (10,11).
Changes in Cardiac Outcomes and in Practice in New York
There have been numerous changes in cardiac outcomes, as well as in factors such as market share, the practice of cardiac surgery/cardiology, contracting with managed care plans, and choice of patients that have either occurred or alleged to have occurred since the inauguration of New York's registries and public release of outcomes. The following is a short synopsis of studies related to these topics. We acknowledge that our interpretation of these studies is unavoidably influenced by our active involvement as developers and defenders of the systems.
Changes in cardiac outcomes
There have been a few studies that have examined the change in short-term CABG surgery mortality after the release of the New York reports. For example, Hannan et al. (12) found that the in-hospital mortality for CABG patients decreased from 3.52% in 1989 to 2.78% in 1992. After risk-adjustment to reflect differences across the years in patient severity of illness, the decrease in risk-adjusted mortality was 41% in this time frame, from 4.17% in 1989 to 2.45% in 1992 (12).
Since CABG surgery outcomes have been improving across the country and world as a result of new techniques and processes of care, the most definitive manner of assessing the outcome changes in New York is to compare them to other regions over the same period. The first such attempt to do this was a study by Peterson et al. (13) that used Medicare data between 1987 and 1992 (before and after initiation of the program) to examine the CABG mortality rate and changes in the rate. The researchers found that New York had the lowest risk-adjusted mortality rate of any state in 1992, and that the decrease in mortality between 1987 and 1992 was higher than for any other state with below-average mortality (10,13). A more recent study by Hannan et al. (14) compared CABG surgery in-hospital/30-day risk-adjusted mortality between 1994 and 1999 in states/regions of the country with public reporting and/or formal quality improvement programs with the mortality in the remainder of the United States. Results showed that the risk-adjusted odds for mortality in New York for the 1994 to 1999 period was only 0.66 (95% confidence interval: 0.57 to 0.77) times the odds in the remainder of the country. In other words, after adjusting for patients' pre-procedural severity of illness, patients in the remainder of the country were 52% more likely to experience short-term mortality (14).
Avoidance of high-risk patients by not providing procedures or by out-of-state referrals
Several studies have contended that an adverse reaction to the public dissemination of cardiac data in New York has been to avoid high-risk patients by either refusing to recommend cardiac surgery or percutaneous coronary intervention (PCI), or by referring the patient out of state. One of the earliest studies of this nature was by Omoigui et al. (15), who compared the number and acuity level of New York patients undergoing CABG surgery from 1980 to 1988 in the Cleveland Clinic before the inauguration of the New York CABG registry with similar patients in the 1989 to 1993 time frame. The investigators found that there was an average of 61.4 patients per year in the first time period, and 96.2 per year after the founding of the New York registry. Also, New York patients treated at the Cleveland Clinic were sicker than New York patients treated in New York (15).
It should be noted that the first public release of New York CABG surgery data occurred in 1990 and it was not expected, so the earliest that hospitals would have been tempted to refer out of state for fear of adverse publicity would have been in 1991, 2 years after the beginning of the “after” time period in the study. The expected mortality of the patients referred from New York, although higher than that of other Cleveland Clinic patients changed little between the 1989 to 1990 period and the 1991 to 1993 period (10,15).
It should also be noted that in the study by Hannan et al. (14) that compared Medicare patients undergoing CABG surgery in regions with public dissemination of outcomes and formal quality improvement programs with the remainder of the United States, it was reported that out-of-state referrals in New York were lower in 1994 than the remainder of the country (9.9% vs. 10.4%) and also lower in 1999 (10.4% vs. 10.5%), so there does not appear to be any widespread outmigration in New York in comparison with states without public data releases.
A widely quoted study by Moscucci et al. (16) compared the pre-procedural severity of illness of New York PCI patients with PCI patients in 8 Michigan hospitals in 1998 to 1999. The researchers found that New York patients had significantly lower prevalence of acute myocardial infarction (AMI), cardiogenic shock, and congestive heart failure than the Michigan patients. Although the unadjusted in-hospital mortality was significantly lower in New York than in Michigan, the risk-adjusted mortality rates in the 2 regions were not significantly different. Moscucci et al. (16) concluded that New York cardiologists may not be intervening as much on high-risk patients because of their fear of public reporting of mortality rates.
There are some reasons why this conclusion may not be correct. First, although Moscucci et al. (16) demonstrate that some risk factor definitions in the 2 systems are nearly identical, they do not mention definitions of other risk factors. In particular, shock is defined in the New York registry as systolic blood pressure <80 mm Hg or cardiac index <2.0 at the time of the procedure despite pharmacologic or mechanical support. Most definitions of shock do not require low blood pressure or low output even after treatment. Second, the prevalence of high-risk conditions is measured as the number of patients with that condition undergoing PCI divided by the total number of patients undergoing PCI. The number could be lower in New York because there are more low-risk patients per capita undergoing PCI rather than that there are fewer high-risk patients per capita undergoing PCI.
Another study, by Dranove et al. (17), used Medicare data to conclude that the severity of illness (measured by costs in the year preceding hospitalization) of CABG patients versus AMI patients in New York and Pennsylvania hospitals declined between 1987 and 1994 relative to hospitals in states without public reporting, as evidenced by lower hospital utilization in the year before admission for surgery. Presumably the reason for using this unconventional measure of patient severity is that it is not subject to surgeon manipulation, is not controlled for in the risk-adjustment methodology, and could be used by surgeons to identify lower-risk patients. This presumes that surgeons would take the time to identify patients with lower hospital utilization who appeared not to have a commensurately low severity of illness on the basis of measures that are used in the risk-adjustment methodology (17,18).
In a more direct and traditional study of Medicare patients undergoing CABG surgery between 1994 and 1999 in New York, other states/regions with public reporting/quality improvement efforts, and the remainder of the United States by Hannan et al. (14), New York CABG patients had significantly higher prevalence of AMI, age ≥80 years, emergency admissions, and females than the remainder of the country. Also, Peterson et al. (13) found that New York Medicare CABG patients had comparable prevalence of AMI, congestive heart failure, diabetes mellitus, and peripheral vascular disease as other U.S. Medicare CABG patients between 1987 and 1992, and New York patients had higher rates of AMI, age >80 years, and females in 1992 than in 1989. It is important to note that the prevalence of risk factors in these studies was coded by hospital personnel who do not work in the cardiac surgery departments, and are therefore not as susceptible to manipulation or subtle over-coding of risks that may bias risk assessment.
Some surveys of surgeons and cardiologists in the state indicate that they believe that avoidance of high-risk patients is a problem. Burack et al. (19) reported that 67% of New York surgeons claimed they had refused to treat at least 1 patient in the previous year, and 18% refused to treat ≥5 patients. Narins et al. (20) found that 83% of interventional cardiologists in New York “agreed” or “strongly agreed” that the publication of statewide PCI report cards decreased the chance that patients needing PCI actually received it.
Although there appeared to be no evidence of widespread avoidance of high-risk cases, cardiologists and surgeons in the state and on the CAC continued to express concern about including patients in cardiogenic shock in the computation of risk-adjusted mortality rates. After a recommendation by the CAC, the DOH decided to no longer include shock patients in the risk-adjusted rates starting with the 2006 report because of the concern that these patients were not undergoing revascularization as often as it was needed. However, it was decided to continue to collect data on shock patients undergoing revascularization to determine the impact of the new policy. In 2005, when shock patients were in the reports, 83 shock patients underwent PCI, with a in-hospital/30-day mortality rate of 34%. During the next 3 years, 133, 146, and 138 shock patients underwent PCI with a combined mortality rate of 45%. Thus, the policy of omitting shock patients from public reporting resulted in an average increase of 67% per year in the next 3 years, and the shock patients undergoing PCI were more severely compromised on average. The trend for CABG surgery was weaker, with 32 shock patients in 2005, followed by 46, 41, and 43 in the next 3 years. However, the average mortality rate in the subsequent 3 years was 38%, compared with a 22% mortality rate in 2005.
More recently, a decision was made to exclude from analysis and public reporting PCI patients who experience anoxic brain injury and die of withdrawal of support subsequent to a documented pre-intervention AMI and cardiac arrest. Since this policy took effect with 2010 procedures, the data are still being reviewed, and the impact of this policy change is unknown at this time.
Impact of public reporting on market share
There have been a few studies that have examined the impact of public reporting of CABG surgery mortality on hospitals' future market shares, and the evidence is mixed. An earlier study (Hannan et al. ) found no substantial changes in hospital volumes during the course of 1989 to 1992. Mukamel and Mushlin (22) found that in the 1990 to 1993 period, CSRS reports had a small effect on hospital volume, but a larger effect on surgeon volume. Romano and Zhou (23) found that New York CABG hospitals noted as having significantly fewer CABG deaths than expected experienced significantly increased CABG volume in the first month after publication (61 patients predicted and 75 admitted, a 22% increase). These increases were not seen for non–CABG-related admissions (23). The latest and most comprehensive study, by Jha and Epstein (24), found no evidence that performance in the reports was associated with a significance increase (for hospitals in the best quartile or best decile) or decrease (for hospitals in the worst quartile or worst decile) in market share between 1989 and 2002. In general, each hospital's market share remained very similar over time (24). The differences in conclusions of these studies may be due in part to differences in time frames that were used, but are probably more likely due to differences in the methods used.
Whether there are changes in market share or not, they do not seem to have been strongly impacted by changes in referrals, at least in the early years of the system when such information was obtained. In a 1996 to 1997 survey, Hannan and Stone (25) found that only 22% of New York cardiologists routinely discussed the report with their patients, and only 38% of cardiologists used the report card information for referrals. These findings may no longer be true, given how dated they are.
Another way that change in market share can occur is by changes in contracting by managed care organizations for tertiary services. Romano et al. (26) found in 1999 that about half of their surveyed hospital administrators used the report cards in health-plan negotiations. Mukamel et al. (27) found that 60% of managed care organizations responding to a 1998 survey ranked surgeon quality as the most important factor in contracting, but only 20% indicated that the report cards were a major factor in their contracting decision. Conversely, a later study by Mukamel et al. (28) found that the probability of managed care organizations contracting with a surgeon in the Downstate region of the state was significantly higher if surgeon was a high-quality outlier or had a lower risk-adjusted mortality rate.
Impact of public reporting on surgeons
Another hospital-level activity that was prompted by the release of the reports was the monitoring and policing of low-volume surgeons. Earlier publications highlighted the fact that lower volume surgeons were associated with worse outcomes, and that became clear with the release of surgeon-level data (29,30). In reaction to these studies and reports, some hospitals restricted the privileges of low-volume surgeons, and between 1989 and 1992, 27 low-volume surgeons ceased practicing cardiac surgery in the state. Some left the state, some retired, and others restricted their practice to noncardiac surgery (30). This group's risk-adjusted CABG surgery mortality in the last year in which they practiced in the state was 11.9%, compared with the statewide rate of 3.1% in that time interval (10).
A more recent study by Jha and Epstein (24) used 3-year reports (the periods used to report surgeon data) from 1989 to 1991 through 1994 to 1996 to examine the surgeons who discontinued performing cardiac surgery in the 2-year period after the publication of the reports. They found that >20% of surgeons with patient risk-adjusted mortality rates in the highest (worst) quartile stopped practicing CABG surgery within 2 years after publication of the reports, in comparison to roughly 5% of surgeons in the top 3 quartiles. This was a statistically significant difference when all 5 3-year reports were combined. It is notable that the group of surgeons with risk-adjusted mortality rates in the highest quartile all performed >150 CABG procedures over the 3-year periods in the report (24).
Ability to predict performance over time
The ability of New York's public CABG reports to predict performance over time is another attribute of the system that has been studied. This is important because it relates to the use of the reports to choose hospitals and surgeons. Jha and Epstein (24) found that if patients undergoing CABG surgery in 1996 selected a hospital in the top decile using the latest available (1993) data, the mean 1996 risk-adjusted mortality rate of those hospitals was 1.82%, compared to a rate of 2.89% in 1996 for hospitals that were in the bottom decile in the 1993 report. Summing data of this nature across 6 years between 1996 and 2002, the mean risk-adjusted mortality of hospitals in the top decile 3 years earlier was 1.59%, compared to 2.78% for hospitals in the bottom decile in the index year (24).
Glance et al. (31) used a different method to compare the ability of the New York CABG report card data to predict mortality 2 and 3 years later. The researchers compared quality ratings in 2 different time periods (first 2 years apart, then 3 years apart) on the basis of ratios of observed to expected mortality. They concluded that hospital assessments made on the basis of 2-year-old data is a strong predictor of future performance, but that 3-year-old data may not be useful for identifying low-performance hospitals (31).
Use of Registry Data for Numerous Purposes Other Than Producing Report Cards
The registry data have been used for a variety of purposes other than report cards, including examination of the volume-mortality relationship for cardiac procedures; short-term readmissions; comparison of outcomes for different types of stents; the impact of incomplete revascularization for PCI; comparison of CABG surgery and PCI; comparison of off-pump and on-pump CABG surgery; development of risk indices to use for informed consent and pre-procedural risk; access to cardiac procedures by race, ethnicity, sex, and payer; evaluation of the impact of various processes of care on outcomes; evaluation of the impact of structures of care on outcomes; geographical variations in use of cardiac procedures; identification and exploration of significant risk factors for cardiac procedures; evaluation of risk-adjustment methods and of clinical versus administrative databases; appropriateness of cardiac procedures; and improving quality of pediatric cardiac surgery through the development of statistical models for assessing outcomes, and the examination of the relationship between processes, structures and outcomes of care (see the Online Appendix for a list of references to these topics).
Many of these studies required matching New York's CABG surgery and/or PCI registry data with other databases. The registries were matched and merged with themselves and each other to create longitudinal databases to be used to identify repeat revascularization and to compare CABG surgery and PCI outcomes. They were matched to Statewide Planning and Research Cooperative System data to obtain information on readmissions for AMI that did not result in revascularization, and they were matched to New York vital statistics data and to National Death Index data to identify short-term and longer-term deaths after discharge to measure comparative effectiveness of various treatment options (e.g., CABG surgery vs. PCI, drug-eluting stents vs. bare-metal stents).
The formation of the cardiac registries in New York has coincided with and likely spurred decreases in mortality since their inception. There has also been a lively debate as to whether these decreases have been in part due to the avoidance of or out-migration of high-risk patients. Our view is that although there have undoubtedly been some patients who have been refused procedures, this number is quite small as a percentage of all procedures performed, and some of those patients were not viable candidates for the procedures. In addition, if the assessment of a given hospital is that a patient may be too high of a risk, then referral to a hospital with more experience or more resources may be in that patient's best interest.
Several other impacts of the New York cardiac registries (mostly CSRS) have been studied, including the impact on hospitals' and surgeons' market share, the impact on low-rated surgeons continuing to perform cardiac surgery in the state, and the ability to use reports to predict future performance. The evidence on market share is dated and ambivalent. There does seem to be evidence (also dated) that surgeons with poor ratings were more likely to discontinue cardiac surgery in the state in relation to other surgeons, and there is evidence that the reports are reasonably good predictors of future performance, although it is important that they be as timely as possible if used for this purpose. The registries have also been used to examine a variety of research topics pertinent to policy issues regarding the choice and use of cardiac procedures, and the comparative effectiveness of competing treatment options, and have made many contributions to the establishment of policies related to cardiac procedures.
Regardless of opinions about the pros and cons of public reporting, it is clear that it is a phenomenon that is here to stay for the foreseeable future. Consequently, there is a need to focus on lessons to be learned from existing public report cards. In our view, the most valuable lessons we have learned are as follows: 1) it is critically important to assure the completeness and accuracy of the data being used because the reports can impact quality of patient care as well as the success and profitability of healthcare providers; 2) the acceptance and use of the reports is dependent on the manner in which they are presented to providers and the public, and the degree to which these constituencies are part of the process of developing the reports; and 3) being an outlier or fear of being a high outlier/desire to be a low outlier are powerful motivators.
It can be expected that reporting will be either accidentally or deliberately inaccurate in an era of public reporting, and concerted measures must be taken to assure accuracy and completeness. During the course of releasing cardiac data for the past 20 years, we have encountered many instances of inaccurate data that were threats to the validity of our reports. In the early years, a hospital submitted >300 isolated CABG cases without a reported death. When these data were matched to New York's administrative data, it was discovered that about 50 more patients had undergone surgery and 18 of them had died in the index admission. Other hospitals have reported extremely high prevalence for some risk factors, but auditing revealed that their prevalence was very similar to the statewide average. In another hospital, numerous patients were reported as discharged alive (when in-hospital mortality was used as the outcome measure), but it was discovered that they were “discharged” to an in-hospital hospice that was not certified and died of complications of CABG surgery.
When mortality after discharge but within 30 days of the index procedure was combined with in-hospital mortality as the new outcome measure, hospitals were asked to report this measure to the registry. Matching these reported deaths against the National Death Index revealed that the vast majority of these out-of-hospital deaths were not identified by the hospitals. Consequently, the National Death Index or New York vital statistics data in conjunction with the Social Security Death Master File have subsequently been used to capture out-of-hospital deaths. The former is preferable as it includes more deaths for patients under age 65 years, but funding issues have sometimes required the use of the latter.
With regard to completeness, the best safeguard is to match the data against another database if possible. In New York, we have matched the registries to Statewide Planning and Research Cooperative System data. It should be noted that assuring completeness is probably not a problem when using CMS data because MedPAR should be complete (for Medicare patients) given that it is related to reimbursement.
With respect to accuracy, different methods are probably necessary to assure the accuracy of risk factors and outcomes as noted in the preceding text, our experience is that out-of-hospital deaths should be captured by matching to death indices. Risk factor accuracy can probably only be assured by auditing medical records. This is an expensive proposition, and considerable thought needs to be given to choosing cases/hospitals to be audited. As noted earlier, New York has chosen hospitals on the basis of past problems, time since last audit, and presence of risk factors in the statistical models with unusually high prevalence. Because of resource constraints, under-reporting problems are left to hospitals to monitor (32).
If administrative data are used in the risk-adjustment process, as is the case with the CMS report cards for AMI and congestive heart failure, the data accuracy problem is exacerbated because the International Classification of Diseases codes used in MedPAR are not very detailed, and a specified level of severity cannot be assured. For example, in the New York registry, creatinine levels are used as a measure of renal failure, and chronic obstructive pulmonary disease is defined in terms of expiratory volume, PO2, and PCO2 levels, whereas the International Classification of Diseases codes have no clinically objective definition. In general, this means that more subjectivity is involved in assigning risk factors to patients, and this can lead to increases over time in reporting prevalence in an era of pay for performance.
In terms of improving acceptance of reports as well as improving their quality, it is important to seek the advice of multiple constituencies (hospital administrators, clinicians from inside and outside the healthcare system being studied, health policy experts, ethicists, researchers, and so forth) and to keep hospitals and physicians apprised of decisions that are being made, as well as to provide them with a forum for making recommendations.
The New York CAC includes several of the most prominent clinicians and researchers in the world, and Chairs of the CAC have included John Kirklin, Kenneth Shine, and current Chair Spencer King. Also, town hall meetings have been convened across the state on several occasions to present the methods used and findings as well as to entertain suggestions and questions. Nevertheless, although the in-state and out-of-state members of the CAC are highly supportive of the reporting systems, there are undoubtedly numerous detractors of the systems. Surveys, although now quite old, have demonstrated many complaints and lukewarm enthusiasm (25). It is possible that even more concerted efforts to educate the clinicians throughout the state and to accommodate more of the recommendations may improve satisfaction with the reports. Also, more efforts/resources probably need to be expended to make the reports more understandable to both clinicians and prospective patients. Furthermore, future reports need to include additional outcome measures, including appropriateness, be tailored to the disease rather than the procedure used to treat it, and possibly include process measures.
Regarding the impetus created by outlier status, the earlier description of specific initiatives taken by hospitals in the state to improve their risk-adjusted mortality rates and outlier status attests to the motivation inspired by outlier status in conjunction with public reporting. There are numerous other off-the-record reports of hospitals who concede that they would not have looked more carefully at their processes of care without having been identified as an outlier or been on the cusp of outlier status. Although we have not formally studied the difference in quality initiatives of hospitals in the middle of the pack compared to outliers or near outliers, anecdotal information points to the benefits of potential outlier status in conjunction with data available for hospitals to investigate the impact of various processes and structures of care. Thus, reports that contain very few outliers may not have the ability to effect change as much as reports that tend to distinguish hospitals from each another. Also, reports that do not distinguish quality among providers offer little guidance to patients and referring physicians (18).
The New York “experiment,” on the heels of a maligned effort by HCFA to release provider data to the public, has proven to be robust over the course of 20 years, has helped spawn several similar statewide efforts (33–36), and has served as a model for current federal and professional association initiatives. Much has been learned from this experience, but many more challenges exist to make public reporting of healthcare outcomes as accurate, informative, and beneficial as possible to patients, providers, regulators and policy makers.
For supplemental references, please see the online version of this article.
The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- acute myocardial infarction
- coronary artery bypass graft
- Center for Medicare and Medicaid Services
- Cardiac Surgery Reporting System
- Department of Health
- Health Care Financing Administration
- percutaneous coronary intervention
- Percutaneous Coronary Interventions Reporting System
- Received August 31, 2011.
- Revision received November 10, 2011.
- Accepted December 17, 2011.
- American College of Cardiology Foundation
- Hannan E.L.,
- Racz M.J.,
- Jollis J.G.,
- Peterson E.D.
- Altman L.K.
- ↵Coronary artery bypass surgery in New York State: 2006–2008 (New York State Department of Health), http://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2006-2008_adult_cardiac_surgery.pdf. Accessed April 23, 2012. December 2010.
- Percutaneous coronary angioplasty in New York State: 2006–2008 (New York State Department of Health), http://www.health.ny.gov/statistics/diseases/cardiovascular/docs/pci_2006-2008.pdf. Accessed April 23, 2012. December 2010.
- Chassin M.R.
- Coronary artery bypass surgery in New York State: 1996–1998 (New York State Department of Health), http://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/1996-1998_adult_cardiac_surgery.pdf. Accessed April 23, 2012. January 2001.
- Peterson E.D.,
- DeLong E.R.,
- Jollis J.G.,
- Muhlbaier L.H.,
- Mark D.B.
- Omoigui N.A.,
- Miller D.P.,
- Brown K.J.,
- et al.
- Moscucci M.,
- Eagle K.A.,
- Share D.,
- et al.
- Dranove D.,
- Kessler D.,
- McClellan M.,
- Satterthwaite M.
- Epstein A.J.
- Hannan E.L.,
- Kumar D.,
- Racz M.,
- Siu A.L.,
- Chassin M.R.
- ↵Romano PS, Zhou H. Do well-publicized risk-adjusted outcomes reports affect hospital volume? Med Care 42:367–77.
- Jha A.,
- Epstein A.M.
- Romano P.S.,
- Rainwater J.A.,
- Antonius D.
- Hannan E.L.,
- Siu A.L.,
- Kumar D.,
- Kilburn H. Jr..,
- Chassin M.R.
- ↵Cardiac surgery in Pennsylvania 2007–2008 (Pennsylvania Health Care Cost Containment Council), http://www.phc4.org/reports/cabg/09/default.htm. Accessed April 23, 2012. September 2010.
- The California report on coronary artery bypass graft surgery: 2007 hospital data: California CABG Outcomes Reporting Program (California Office of Statewide Health Planning and Development), http://www.oshpd.ca.gov/HID/Products/Clinical_Data/CABG/07-08Breakdown.html. Accessed April 23, 2012. June 2011.
- Adult coronary artery bypass graft surgery in the Commonwealth of Massachusetts: fiscal year 2009 report (Harvard Medical School), http://www.massdac.org/sites/default/files/reports/CABG%20FY2009.pdf. Accessed April 23, 2012. January 2011.
- Cardiac surgery in New Jersey: 2008 (Health Care Quality Assessment, Office of Policy and Strategic Planning, New Jersey Department of Health and Senior Services), http://www.state.nj.us/health/healthcarequality/cardiacsurgery.shtml#CSR. Accessed April 23, 2012. June 2011.