Author + information
- Received August 14, 2010
- Revision received October 25, 2010
- Accepted October 28, 2010
- Published online April 5, 2011.
- Paul S. Chan, MD, MSc⁎,†,⁎ (, )
- Ralph G. Brindis, MD, MPH‡,§,
- David J. Cohen, MD, MSc⁎,†,
- Philip G. Jones, MSc⁎,
- Elizabeth Gialde, RN⁎,
- Richard G. Bach, MD§∥,
- Jeptha Curtis, MD¶,
- Charles F. Bethea, MD#,
- Marc E. Shelton, MD⁎⁎ and
- John A. Spertus, MD, MPH⁎,†
- ↵⁎Reprint requests and correspondence:
Dr. Paul S. Chan, Saint Luke's Mid America Heart Institute, 5th Floor, 4401 Wornall Road, Kansas City, Missouri 64111
Objectives The objective of this study was to compare the consistency in appropriate use criteria (AUC) ratings among a broad range of practicing cardiologists and the AUC Technical Panel.
Background AUC for coronary revascularization have been developed by selected experts.
Methods Before AUC publication, 85 cardiologists from 10 U.S. institutions assessed the appropriateness of coronary revascularization for 68 indications that had been evaluated by the AUC Technical Panel. Each indication was classified as appropriate, uncertain, or inappropriate, based on the physician group's median rating. Rates of concordance between the physician group and the AUC Technical Panel (i.e., same appropriateness category assignment) and rates of nonagreement within the physician group (≥25% of panelists' ratings outside the group's appropriateness category assessment) were determined.
Results Overall concordance between the 2 groups was 84%. Among indications classified as appropriate by the AUC Technical Panel, concordance between the 2 groups was excellent (94% [34 of 36]); however, nonagreement within the physician group was 44% (16 of 36). Among indications classified as uncertain, there was 73% (16 of 22) concordance between the 2 groups. Among inappropriate indications, concordance was moderate (70% [7 of 10]), but nonagreement occurred frequently (70% [7 of 10]). Moreover, there was substantial variation in appropriateness ratings between individual physicians and the AUC Technical Panel (weighted kappa range: 0.05 to 0.76).
Conclusions Although there was good concordance in assessments of appropriateness for coronary revascularization between physicians and the AUC Technical Panel, nonagreement within the physician group was common and there was marked variation in ratings between individual physicians and the AUC Technical Panel.
Although few debate the potential of coronary artery bypass grafting or percutaneous coronary intervention to confer substantial mortality and quality of life benefits in selected patients, concerns about potential overuse and underuse of these procedures have emerged. As a result, national societies have developed the appropriate use criteria (AUC) for coronary revascularization to support the efficient and rational use of these procedures.
Although the technical panel that created the AUC included a diverse range of perspectives and based their recommendations upon clinical guidelines, it is unknown whether appropriateness assessments might differ among a broader range of cardiologists, including those in community-based settings. Understanding the extent of concordance in appropriateness ratings between the expert panel and a more diverse group of practitioners can validate scenarios for which there is broad consensus or highlight the need for either more research or better translation of existing knowledge to clinical practice. It can also indicate the likelihood of widespread acceptance and adoption of the AUC. Accordingly, we surveyed cardiologists with a representative sample of the same clinical indications as those rated by the AUC Technical Panel and examined both their concordance and the variation in their distribution of appropriateness ratings with the AUC Technical Panel.
Development of the AUC
The methodology to develop the AUC for coronary revascularization has been previously described (1). Briefly, the AUC Technical Panel included 17 members (4 interventional cardiologists, 8 noninterventional cardiologists, 4 cardiac surgeons, and 1 payer) who rated the appropriateness of coronary revascularization for 174 distinct clinical indications based upon different combinations of: 1) clinical presentation (acute vs. nonacute); 2) angina severity (asymptomatic or Canadian Cardiovascular Society [CCS] class); 3) extent of ischemia on noninvasive functional testing (low, intermediate, or high risk); 4) high-risk clinical features (e.g., left ventricular dysfunction); 5) extent of anti-ischemic medical therapy; and 6) extent of anatomic disease based on angiography (number of diseased coronary arteries with or without proximal left anterior descending [LAD], left main, or bypass graft disease). Moreover, the AUC Technical Panel evaluated the relative appropriateness of percutaneous coronary intervention and coronary artery bypass grafting for 24 additional scenarios.
Based upon the median score of the AUC Technical Panel for each clinical indication (range 1 to 9), clinical indications were categorized as “appropriate” (median 7 to 9), “uncertain” (4 to 6), or “inappropriate” (1 to 3). In addition, rates of nonagreement were determined to assess variations in panelists' ratings of appropriateness for a given clinical scenario. Nonagreement occurred when ≥25% of the individual panelists' ratings fell outside the 3-point region for that indication (2). To accomplish this, the American College of Cardiology Foundation (ACCF) made available to us the final ratings of each AUC Technical Panel member.
Description of study survey
In 2008, before the release of the AUC, permission was obtained from the ACCF to replicate the appropriateness ratings process in a cohort of clinicians. Two of the investigators (P.S.C. and J.A.S.) reviewed and selected 68 representative indications evaluated by the AUC Technical Panel for this study's survey (Online Appendix). For the survey, one-third of the AUC's indications were chosen to minimize respondent burden, but the selected indications reflected a diverse range of clinical scenarios.
Select cardiologists from 10 institutions (Online Appendix) were electronically mailed the study survey. Because the AUC were not yet published, none of the survey participants had access to the AUC Technical Panel's ratings. Study participants were provided the same explanatory documents, tables and figures, references, and appendices that had been given to the AUC Technical Panel.
For each of the 68 clinical indications, the median and distribution of appropriateness ratings from study participants were determined. From the median value, the physician group's assignment of appropriateness category was determined for each indication using the method described earlier.
Concordance of appropriateness category assignments (e.g., appropriate, uncertain, inappropriate) for each clinical indication was compared between the physician group and the AUC Technical Panel with descriptive plots. Summaries of concordance results were tabulated overall and stratified by whether the AUC Technical Panel had categorized an indication as appropriate, uncertain, or inappropriate. Analyses were then repeated after dividing the physician group by interventional status, years in practice (<15 years vs. ≥15 years), and percent of time dedicated to research (<10% vs. ≥10%).
For examination of variation in appropriateness assignments, the presence of nonagreement for each indication within the physician group was assessed. In addition, to examine the extent of variation between individual physicians and the AUC Technical Panel, we computed weighted kappa statistics between each individual physician and the Technical Panel for all 68 clinical indications and examined the distribution of weighted kappas within the physician group.
Finally, we examined which clinical factors (coronary anatomy, extent of ischemia on noninvasive functional testing, severity of symptoms, and intensity of anti-ischemic therapy) predicted nonagreement within the physician group using multivariable hierarchic regression, with indications clustered by physician and physicians clustered by institution. All analyses were conducted with SAS version 9.2 (SAS Institute, Inc., Cary, North Carolina) and R version 2.6.2.
A total of 85 physicians from 10 (2 community, 7 university-affiliated, and 1 university-owned) institutions completed the study survey. Among the physician respondents, 44 (51.8%) were interventional and 41 (48.2%) were noninterventional cardiologists. All but 2 (97.6%) were board certified in cardiology, the median number of years in practice was 14.5 (interquartile range: 7.5 to 20.0 years; range: 1.0 to 43.0 years), and the median proportion of time spent in research was 5% (interquartile range: 0% to 15%; range 0% to 75%). Overall, there was good concordance (84%) in appropriateness category assignments between the physician group and the AUC Technical Panel for the 68 indications. However, rates of nonagreement were 66% in the physician group.
Clinically appropriate indications
For the 36 clinical indications identified as appropriate by the AUC Technical Panel, there was excellent concordance, with the physician group rating the indications as appropriate 94% (34 of 36) of the time (Fig. 1, Table 1). Both interventional and noninterventional cardiologists rated the vast majority of these indications as appropriate (Online Appendix), and physician ratings did not differ by number of years in practice or percent of time dedicated to research (Table 1). There was, however, greater variation (i.e., wider distribution) in the ratings among physicians in this study than in the AUC Technical Panel. Whereas the Technical Panel had nonagreement in 31% (11 of 36) of the clinical indications categorized as appropriate, the physician group had nonagreement in 44% (16 of 36) of the indications (Table 2).
Clinically uncertain indications
For the 22 clinical indications rated as uncertain by the AUC Technical Panel, both groups rated the indications as uncertain 73% (16 of 22) of the time (Table 1). Of the 6 discordant indications, the physician group rated 3 scenarios (12c, 14c, and 57b) as appropriate and 3 (25a, 28a, and 50a) as inappropriate (Fig. 2). Although there was similar concordance between interventionalists and noninterventionalists with the AUC Technical Panel, physicians with ≥10% time dedicated to research had higher concordance with the AUC Technical Panel than physicians with <10% research time (Table 1). Finally, rates of nonagreement for scenarios categorized as uncertain were 100% in the physician group (22 of 22) and 82% (18 of 22) in the AUC Technical Panel.
Clinically inappropriate indications
For the 10 clinical indications identified as inappropriate by the AUC Technical Panel, the physician group assigned an inappropriate classification for only 70% (7 of 10) of the indications categorized as inappropriate by the AUC Technical Panel (Fig. 3, Table 1), with 100% concordance among noninterventional cardiologists and 70% among interventional cardiologists (Online Appendix). Importantly, there was significant variation among physician ratings for these clinical scenarios, with nonagreement rates of 70% in the physician group compared with 20% in the AUC Technical Panel (Table 2). Interventionalists and physicians with ≥10% research time had lower rates of nonagreement than their counterparts.
Predictors of nonagreement among physicians
In a multivariable model, the presence of a proximal LAD stenosis was associated with higher rates of physician nonagreement (adjusted rate ratio [RR]: 1.29 [95% confidence interval (CI): 1.11 to 1.51]; p = 0.001), suggesting greater variability in ratings of appropriateness within the physician group when the proximal LAD was involved (Table 3). In contrast, a high-risk noninvasive study for ischemia (adjusted RR: 0.51 [95% CI: 0.40 to 0.65]; p < 0.0.001), maximal intensity (i.e., 2 or more agents) of anti-ischemic therapy (adjusted RR: 0.75 [95% CI: 0.69 to 0.82]; p < 0.001), and substantial (CCS class III to IV) symptoms (adjusted RR: 0.46 [95% CI: 0.38 to 0.57]; p < 0.001) were each associated with significantly lower rates of physician nonagreement.
Variation between individual physicians and AUC Technical Panel
The distribution of weighted kappa statistics between individual physicians and the AUC Technical Panel for all 68 indications was wide, ranging from 0.05 to 0.76, with the average weighted kappa for all physicians of 0.52 (Fig. 4). This suggests marked variation in appropriateness assignments between individual physicians and the AUC Technical Panel, with certain physicians almost never agreeing with the AUC Technical Panel for any of the surveyed indications and no physician achieving uniform concordance with the AUC Technical Panel.
In this study, we compared ratings of clinical appropriateness for coronary revascularization between practicing clinicians and members of the AUC Technical Panel. We found there was excellent concordance (94%) between the 2 groups for clinical indications categorized as appropriate but only modest concordance (70%) for clinical indications categorized as inappropriate. However, there was wide variation (i.e., nonagreement) in ratings of appropriateness among physicians, with more than 25% of physicians assigning an appropriateness category different than the group as a whole in 2 of every 3 scenarios. Moreover, there was substantial variation in appropriateness category assignments between individual physicians and the AUC Technical Panel, with some physicians almost never agreeing with the AUC Technical Panel and no physician achieving more than 80% agreement. Collectively, our findings suggest that although there is general concordance in ratings of clinical appropriateness between practicing cardiologists and the AUC Technical Panel, there is not uniform agreement between the 2 groups, with markedly different opinions among individual physicians, even after reviewing existing evidence.
An important strength of this study is that the physician ratings were obtained before the publication of the ACC's AUC for coronary revascularization in early 2009. This avoided potential contamination of survey results by respondents from the views of the AUC Technical Panel. Importantly, the physician group received similar instructions and access to clinical guidelines for coronary revascularization as the AUC Technical Panel to derive their appropriateness ratings.
Our findings suggest that there was substantial within-group heterogeneity in ratings of appropriateness among the physicians surveyed. More than 1 in 4 physicians rated an indication for coronary revascularization as uncertain or inappropriate for 43% of indications categorized as appropriate by the AUC and rated an indication as uncertain or appropriate for 70% of the inappropriate indications. Notably, physicians' rates of nonagreement were lower for those indications involving severe CCS class III to IV angina symptoms and high-risk noninvasive studies for ischemia, when the benefits of coronary revascularization are likely to be greater. Conversely, there was more divergence of opinion in the setting of significant proximal LAD obstruction, suggesting that some clinicians presumably viewed its treatment as life prolonging, whereas others did not.
The broad range of kappa statistics between individual physicians and the AUC Technical Panel, none of which exceeded 0.76, highlights the variability in current opinions about the roles of coronary revascularization in the care of patients with coronary artery disease. More research and better translation of existing knowledge to clinical practice to clarify the appropriateness of certain clinical indications, especially for those indications for which appropriateness was uncertain or when nonagreement existed, are needed. Given this variability in the appropriateness ratings, it is expected that clinicians will vary greatly in their AUC ratings for coronary revascularization. We believe that it will be important to both measure and provide feedback to clinicians about the appropriateness of patients that they treat, while concurrently educating them about the AUC, if more uniform practice is to be achieved.
Our study should be interpreted in the context of the following limitations. Our survey methodology differed somewhat from that of the AUC Technical Panel because there were no face-to-face meetings to discuss cases for which divergent assessments were made. Second, our survey was conducted among clinical cardiologists from only 10 U.S. institutions, and we did not assess ratings of appropriateness among cardiac surgeons. Third, although our panel was larger and more geographically representative of clinical practices than the AUC Technical Panel, it did not capture all geographic regions of the U.S. Fourth, we did not have sufficient sample size to examine other subgroups of interest or obtain more detailed data on characteristics of survey respondents.
We found good concordance for ratings of appropriateness for coronary revascularization between a diverse group of cardiologists and the Technical Panel for the AUC for coronary revascularization. However, there was substantial variation in ratings of appropriateness between individual physicians and the AUC Technical Panel, as well as nonagreement in appropriateness category assignments among cardiologists for a number of indications. These findings suggest that more research to understand these variations, along with additional education about procedural appropriateness, may help achieve greater uniformity in the appropriate use of coronary revascularization.
The authors wish to thank the ACCF for making available to us the deidentified ratings of the Technical Panel for the AUC for coronary revascularization, without which comparisons of rates of agreement between the AUC Technical Panel and the physician group would not have been possible.
For a description of the survey instrument used for physicians in this study, the list of institutions recruited for this study, and comparisons of appropriateness ratings, please see the online version of this article.
Dr. Chan is supported by a Career Development Grant Award (K23HL102224) from the National Heart, Lung, and Blood Institute. Dr. Spertus has a contract with the American College of Cardiology Foundation to analyze the National Cardiovascular Data Registry and to develop appropriate use criteria ratings in the catheterization/percutaneous coronary intervention registry. All other authors have reported that they have no relationships to disclose.
- Abbreviations and Acronyms
- American College of Cardiology Foundation
- appropriate use criteria
- Canadian Cardiovascular Society
- confidence interval
- left anterior descending
- rate ratio
- Received August 14, 2010.
- Revision received October 25, 2010.
- Accepted October 28, 2010.
- American College of Cardiology Foundation
- Patel M.R.,
- Dehmer G.J.,
- Hirshfeld J.W.,
- Smith P.K.,
- Spertus J.A.
- Fitch K.,
- Bernstein S.,
- Aguilar M.,
- et al.