Author + information
- Received July 16, 2007
- Revision received September 27, 2007
- Accepted October 23, 2007
- Published online April 1, 2008.
- Raymond J. Gibbons, MD, FACC⁎,⁎ (, )
- Todd D. Miller, MD, FACC⁎,
- David Hodge, MSc†,
- Lynn Urban†,
- Philip A. Araoz, MD‡,
- Patricia Pellikka, MD, FACC⁎ and
- Robert B. McCully, MD, FACC⁎
- ↵⁎Reprint requests and correspondence:
Dr. Raymond J. Gibbons, Mayo Clinic, Gonda 5, 200 First Street SW, Rochester, Minnesota 55905.
Objectives The purpose of this study was to apply published appropriateness criteria for single-photon emission computed tomography (SPECT) myocardial perfusion imaging (MPI) in a single academic medical center.
Background The American College of Cardiology Foundation (ACCF) and the American Society of Nuclear Cardiology (ASNC) have developed appropriateness criteria for stress SPECT MPI to address concern about the growth in cardiac imaging studies.
Methods We retrospectively examined 284 patients who underwent stress SPECT MPI and 298 patients who underwent stress echocardiography before publication of these criteria.
Results The overall level of agreement in characterizing appropriateness between 2 experienced cardiovascular nurse abstractors was modest (kappa = 0.56), but noticeably poorer (kappa = 0.27) for patients with previous SPECT or echo studies. Similar percentages of each imaging modality were assigned to the 3 appropriateness categories: 64% of stress SPECT and 64% of stress echo studies were classified appropriate; 11% of stress SPECT and 9% of stress echo were of uncertain appropriateness; and 14% of stress SPECT and 18% of stress echo were inappropriate. Of the inappropriate studies, 88% were performed for 1 of 4 indications. Approximately 10% of the patients were unclassifiable.
Conclusions Application of existing SPECT MPI appropriateness criteria is demanding and requires an established database or detailed data collection, as well as a number of assumptions. Fourteen percent of stress SPECT studies and 18% of stress echo studies were performed for inappropriate reasons. Quality improvement efforts directed at reducing the number of these inappropriate studies may improve efficiency in the health care system.
There is a growing recognition of the impending healthcare crisis in the U.S. (1). Many have argued that much of the care currently delivered is not “efficient,” as advocated by the Institute of Medicine (2).
One aspect of cardiovascular care that has drawn increasing scrutiny is cardiovascular imaging. Both the MedPAC commission and other third-party payers have expressed concern about rapid rates of growth in these services (3). A cross-sectional, population-based study of Medicare patients from 1993 to 2001 (4) demonstrated a 6.1% average annual increase in cardiovascular imaging stress tests, compared with 2.0% for cardiac catheterization, 0.8% for percutaneous coronary intervention (PCI), and 0.1% for acute myocardial infarction.
The American College of Cardiology Foundation (ACCF) initiated an effort to develop appropriateness criteria for imaging studies in early 2005. Using a modification of the RAND/UCLA method (5), the ACCF and American Society of Nuclear Cardiology (ASNC) developed appropriateness criteria for single-photon emission computed tomography (SPECT) myocardial perfusion imaging (MPI) (6). The American Heart Association has officially supported the development of appropriateness criteria and endorsed the ACCF/ASNC criteria for SPECT MPI (7).
Unlike the original RAND/UCLA approach (8), which covered thousands of individual patient situations, the published ACCF/ASNC criteria for SPECT MPI only describe 52 specific patient indications, divided into 9 tables (6). Five of the indications address rest SPECT MPI, viability, or ventricular function. The present study was focused on the 47 indications in 7 tables related to stress SPECT MPI.
The specific aims of this study were to examine the existing stress imaging studies (echocardiography and SPECT) performed in clinical practice in 1 academic medical center to determine the following:
1. How many of the studies performed at Mayo Clinic Rochester (before publication of the ACCF/ASNC criteria) were appropriate, uncertain, or inappropriate according to the ACCF/ASNC criteria?
2. What percentage of patients were not included in the existing ACCF/ASNC appropriateness criteria?
Although the ACCF/ASNC appropriateness criteria were developed for SPECT MPI, we decided to apply them more broadly to both stress SPECT MPI and stress echocardiography in our practice. Although the ACCF has announced plans to develop criteria for stress echocardiography, these had not yet been published at the time of this study. Since the existing American College of Cardiology/American Heart Association (ACC/AHA) guidelines have similar recommendations for stress SPECT MPI and stress echocardiography, we anticipated that the appropriateness criteria for stress echocardiography would be very similar to those for stress SPECT MPI. We felt that simultaneous collection of appropriateness data from both the Mayo nuclear cardiology and echocardiography laboratories was necessary to obtain a broader cross section of the use of stress imaging procedures in our institution for comparison to the published Medicare data (4), which does not distinguish between stress SPECT and stress echocardiography studies. The application of the criteria to both laboratories would also avoid preferential “shunting” of patients eligible for stress imaging studies between nuclear cardiology and echocardiography as ordering physicians in our institution became aware of this study. Such shunting would potentially confound any meaningful follow-up studies to assess a change in ordering patterns after internal dissemination of the results of this study within the Mayo Clinic.
Both the Mayo Nuclear Cardiology and Echocardiography Laboratories maintain prospective electronic databases on all patients undergoing these stress imaging procedures, which capture similar but not identical elements. Symptoms recorded at the time of testing include chest pain and dyspnea. Chest pain is categorized as typical angina, atypical angina, or noncardiac chest pain according to the criteria of Diamond (9). Both databases have been used in previously published studies (10,11).
Because the application of these criteria to our databases had not been tested, we elected to do a pilot project consisting of a single day of tests from both laboratories from the month of January 2005. All 5 physicians participating in the project, as well as 2 experienced cardiovascular nurse abstracters (neither of whom worked in either laboratory), independently reviewed these 50 test patients and attempted to apply the appropriateness criteria. A meeting of all 7 participants was then held to review this experience, assess discrepancies, and identify potential obstacles to consistent application of the criteria. We sought to use the existing nuclear cardiology and echocardiography databases wherever possible to limit the resources needed for completion of the project. We consulted each patient’s Mayo electronic medical record only when information necessary to apply a specific appropriateness indication was not recorded in the existing database. Elements that required consultation of the Mayo record included the specific type of surgery planned and the clinical assessment of the patient’s exercise capacity for patients undergoing pre-operative assessment for noncardiac surgery, and recent cholesterol values to calculate the patient’s Framingham risk score.
We also found that a number of assumptions were required to subsequently apply the appropriateness criteria in a standardized, accurate manner.
1. Patients who were symptomatic with dyspnea rather than chest pain were regarded as symptomatic and having “atypical angina” for determining pre-test probability of coronary artery disease. The appendix to the ACCF/ASNC appropriateness criteria document specifically included dyspnea in the description of a chest pain syndrome.
2. We calculated the Framingham score that is appropriate to determine the future risk of hard cardiac events (12), and not the score that incorporates subsequent angina as an event (13). Any patient already on a statin for hyperlipidemia was imputed to have a value of +2 for the low-density cholesterol component of the score.
3. The patient’s exercise tolerance was classified on the basis of a “best estimate” from the clinical notes, as the patient’s exercise tolerance in estimated metabolic equivalents (14) was not commonly specified.
4. Surgical procedures that were not listed among the examples in the ACC/AHA pre-operative testing guidelines (14) (e.g., endoscopic inguinal hernia repair) were generally regarded as “low risk.”
5. Some indications within the same table of the ACCF/ASNC appropriateness criteria were not mutually exclusive. For example, an intermediate-risk patient with normal exercise tolerance undergoing intermediate-risk surgery could be assigned to either indication 32 (inappropriate) or 33 (appropriate) in the ACCF/ASNC appropriateness document. These patients were regarded as unclassified.
6. Only the most recent revascularization procedure was considered if the patient had previously undergone more than 1 procedure. For example, for a patient who had bypass surgery 6 years ago and PCI 2 years ago, the PCI was used to establish the indication for stress imaging.
7. For multiple tables of the appropriateness document, the pilot demonstrated that a single patient could potentially be considered under multiple different indications. For example, a patient with a high-risk Framingham score who was asymptomatic 6 months after PCI for treatment of symptoms could be considered under 2 different categories—asymptomatic (and then classified as appropriate) or previous revascularization (and then classified as inappropriate). We felt that the appropriateness criteria tables in the ACCF/ASNC document should be considered in the order outlined in Table 1 to best reflect clinical decision making and appropriate test utilization. The final patient classification would be the first applicable category in this sequence. Using this approach, the patient cited in the example was, therefore, labeled inappropriate.
Final study group
The final study group consisted of all patients who underwent stress SPECT sestamibi studies and stress echocardiograms at Mayo Clinic Rochester from May 1, 2005 to May 15, 2005 (before the publication of the ACCF/ASNC criteria). Exclusions were as follows:
1. Patients who did not grant research authorization in accordance with Minnesota state law. There were 6 SPECT patients and 11 echocardiogram patients excluded for this reason.
2. Patients who underwent testing at off-site locations as part of Mayo outreach programs.
3. Patients who underwent stress echo hemodynamic studies for the assessment of valvular heart disease.
Each patient was classified independently by 2 experienced cardiovascular nurse abstracters into 1 of the following possibilities:
1. The patient did not qualify under any of the existing 47 indications. Such patients were considered unclassified.
2. The patient qualified under 1 of the existing indications, and the appropriateness of his/her study could be established. The patient was then classified as appropriate, uncertain, or inappropriate.
If the 2 nurse assessments agreed, the classification was final. If they did not, discrepancies were settled by the consensus of 2 staff physicians, 1 from the nuclear cardiology laboratory (R.J.G.) and 1 from the echocardiography laboratory (R.B.M.). The results that follow represent the final classification. There was little difficulty in defining the “broad indication” for the test, such as previous coronary revascularization, which assigned the patient to a specific tablet. Linking the patient information with a specific indication within each table was the source of disagreement.
Agreement in the classification of the indications were compared between the 2 nurses using kappa statistics. Comparisons of categorical factors were completed using the chi-square test for independence.
The study was approved by the Mayo Institutional Review Board.
There were 284 eligible stress SPECT MPI patients and 298 eligible stress echocardiography patients studied between May 1, 2005 and May 15, 2005. Their demographics are shown in Table 2. The patients undergoing stress SPECT MPI were more likely to be men, and more likely to have hypertension, hyperlipidemia, prior myocardial infarction, prior revascularization, and obesity. Stress echocardiography patients were more likely to have dyspnea as their presenting symptom and to have a normal resting electrocardiogram (ECG).
Indication for testing
The indications for testing, categorized for the stress SPECT MPI and stress echo patients using the sequential approach described in the Methods section, are shown in Figure 1. There were significant differences in the indications for testing between the stress SPECT and stress echo patients. Stress SPECT patients were more likely to be referred for follow-up testing post-revascularization (Table 7 of the appropriateness criteria) and for follow-up testing after prior SPECT testing (Table 4 of the appropriateness criteria). In contrast, stress echo patients were more likely to be referred for pre-operative assessment before noncardiac surgery (Table 5 of the appropriateness criteria) and for the assessment of symptoms (Table 1 of the appropriateness criteria). Very few patients were referred for risk assessment after acute coronary syndromes (Table 6 of the appropriateness criteria). A similar percentage (14% to 15%) of both groups were asymptomatic patients without other indications, who were being screened for suspected coronary artery disease.
The agreement between the 2 nurses is shown in Table 3. The overall level of agreement was modest with a kappa of 0.56. There was little difference between the stress SPECT patients (kappa = 0.58) and the stress echo patients (kappa = 0.54). However, the level of agreement was much poorer for patients who were referred for follow-up testing (Table 4) (kappa = 0.27) compared with all the other tables in the ACCF/ASNC document combined (kappa = 0.60). The lower rate of agreement for these patients reflected the difficulty of applying the listed indications, and required subjective judgments, which we did not appreciate during our pilot study. For example, should a SPECT MPI study performed 17 months after a normal SPECT MPI study in a stable patient be considered “annual?” This group also accounted for most (72%) of the unclassified patients, because of the obvious “gaps” between the indications. For example, indication #27 applies to patients with a previous coronary calcium score of <100 and indication #28 applies to patients with a previous coronary calcium score of >400, but there is no indication for patients with a calcium score between 100 and 400.
The overall classification of the stress SPECT studies and stress echo studies is shown in Figure 2. Eleven percent (95% confidence interval [CI] 8% to 15%) of the stress SPECT patients and 9% (95% CI 6% to 13%) of the stress echo patients were unclassifiable. The same percentage of the 2 groups was appropriate 64% (95% CI 58% to 70%) for stress SPECT and 64% (95% CI 58% to 70%) for stress echo. A similar percentage of both patient groups were of uncertain appropriateness 11% (95% CI 7% to 15%) for stress SPECT, 9% (95% CI 6% to 13%) for stress echo. Fourteen percent (95% CI 11% to 19%) of stress SPECT studies and 18% (95% CI 14% to 23%) of stress echo studies were inappropriate.
Appropriateness by indication for testing
The classification of both stress SPECT patients and stress echo patients by the indication for testing, as defined by the tables of the appropriateness criteria, is shown in Table 4. The classification of patients falling under all but one of the tables of indications was similar for both stress SPECT and stress echo patients. There was a significant difference between the stress SPECT patients and the stress echo patients for previous revascularization (Table 7 of the indications), where more echo patients were unclassified (6 vs. 1 for SPECT) and more SPECT patients were inappropriate (5 vs. 0 for echo), but these differences involved only small numbers of patients.
The inappropriate studies for either stress SPECT or stress echo are tabulated in Table 5 according to the indications for testing defined in the appropriateness criteria. Almost one-half of the inappropriate tests (48%) were asymptomatic patients with a low-risk Framingham score who were referred for testing to screen for coronary artery disease. The second largest group of inappropriate tests (17%) was performed in patients who were under consideration for intermediate-risk surgery who had good exercise capacity and no or minor risk predictors. The third sizable group of inappropriate patients (13%) was symptomatic patients with chest pain who had a low pre-test probability, an interpretable ECG, and were able to exercise. The final sizable group (10%) was patients under consideration for low-risk noncardiac surgery.
This study represents one of the first attempts to apply the recently published ACCF/ASNC appropriateness criteria for SPECT perfusion imaging to current clinical practice in an academic medical center. Our experience suggests that the application of these criteria will require an established database or detailed data collection, as well as a number of assumptions, which were described in the preceding text. Broad application of the criteria to general clinical practice is certainly desirable, but likely will be very difficult. The ACCF/ASNC appropriateness criteria were not intended to cover all possible clinical indications, and thereby differ noticeably from the previous RAND/UCLA criteria. Our results suggest that the ACCF/ASNC appropriateness criteria apply to approximately 90% of current stress imaging patients in our center. Omissions from the criteria, as well as ambiguities, leave 9% to 11% of patients unclassified. Further refinements in the criteria, particularly to the indications for follow-up testing, can be expected to reduce this number. Potential suggestions would include elimination of the overlap between indication 32 and 33, addition of an indication for prior calcium scores between 100 and 400, and addition of indications for repeat testing in patients who are not high risk by Framingham score and have normal initial studies. In May 2005, before publication of these criteria, 14% of stress SPECT studies and 18% of stress echo studies in our institution were for inappropriate indications, and another 11% of stress SPECT studies and 9% of stress echo studies were for indications of uncertain appropriateness. These results suggest room for improvement in selection of patients for stress cardiac imaging to improve efficiency in our medical system.
Despite different baseline clinical characteristics, and a different distribution of indications, the overall results with respect to appropriateness were quite similar for stress SPECT studies and stress echo studies in our institution. In almost all cases, these studies were ordered by members of the Mayo Clinic Rochester staff. Although individual staff physicians may preferentially refer their patients to stress SPECT studies or stress echo studies, our results suggest that their overall clinical judgment in ordering stress imaging studies is similar. The similar results between the 2 laboratories may also reflect the fact that the overall patient population on which the studies are ordered comes from the same institution. All of the major findings reported in this study require confirmation in other academic medical centers, as well as nonacademic settings. The unexplained practice variation in the utilization of medical procedures, including SPECT imaging, has been well documented by the Dartmouth Healthcare Atlas for many years (15). We would, therefore, expect that the findings of similar studies performed in different locations may be very different.
The inappropriate studies in our stress imaging practice were restricted to only a few patient indications. Of the 13 inappropriate indications listed in the original ACCF/ASNC criteria, only 7 occurred in our study, and 4 of the 7 accounted for 88% of all the inappropriate studies. The 4 inappropriate indications are well established in previous guidelines and scientific statements. Previous ACC/AHA guidelines for SPECT and echo do not recommend stress imaging in low-risk asymptomatic individuals. The ACC/AHA guideline for pre-operative cardiovascular evaluation recommends against stress testing before low-risk surgery, or before intermediate-risk surgery in patients with good exercise tolerance and no or minor risk predictors. The ACC/AHA guideline for stable angina recommends against stress imaging in patients with a normal ECG who are able to exercise. This suggests great potential for improvement in our center, as an educational effort directed toward ordering physicians need focus only on a few specific situations with strong support in existing guidelines. It is our intent to perform follow-up studies in our institution after such an effort.
The difficulty that we encountered in applying the appropriateness criteria was considerably greater than we first anticipated. The results of the pilot study were surprising to the 5 participating physicians and 2 participating nurses. Using our existing databases wherever possible, the multiple assumptions outlined in the preceding text, and 2 experienced cardiovascular nurse abstracters, our results were modestly consistent, with a kappa value of 0.56 for agreement between the 2 nurses. (The level of agreement was noticeably poorer for patients undergoing follow-up testing.) Efforts to apply these criteria without all of these key components may lead to less consistent and reproducible results. Additional training of our experienced nurses may have improved the level of agreement between them, but such extensive training is unlikely in the “real-world” application of these criteria. Our existing laboratory databases require extensive data collection at “point of service.” Our electronic medical record provided the necessary supplemental data from “point of ordering.” Further real-world application will be necessary to determine whether a “point of service” or “point of ordering” approach is best. If, however, our results can be confirmed in other studies, and the primary goal of applying appropriateness criteria is quality improvement through the elimination of inappropriate studies, our results suggest that this could potentially be achieved at the “point of ordering” with collection of a limited number of data elements (Fig. 3).
The major limitation of this study is its performance at a single academic medical center. All of the ordering physicians, and the stress laboratory physicians, were salaried Mayo staff physicians who, therefore, had no direct financial incentive for the performance of additional tests and no financial interest in the imaging equipment. The assumptions that we made in applying these criteria may not be accepted by others. In particular, third-party payers, who have been reluctant to reimburse stress imaging studies performed for an indication of dyspnea, may question the assignment of such patients to “atypical angina.” However, there are now published studies for both stress SPECT (16) and stress echo (17) showing their utility in the assessment of dyspnea. Our application of stress SPECT criteria to stress echo studies is also a potential limitation. We await the ACCF appropriateness criteria for stress echocardiography with interest. However, we felt that this approach would give us a broader assessment of stress imaging studies in our institution to relate to the published Medicare data (4) and avoid preferential shunting of patients to a different laboratory, which would confound our future attempts to demonstrate improvement. We only studied 2 weeks of patients owing to the extensive effort required, but the 95% CIs for our estimates are acceptable. Finally, as in many evaluations of quality of care, incomplete documentation may explain some of the inappropriate studies.
This study demonstrates the application of appropriateness criteria to attempt quality improvement in the clinical use of stress cardiac imaging. We encourage others to perform similar studies in their own institutions, hospitals, and practices. Such efforts are only a first step toward the ultimate goal of quality improvement in this area (i.e., to improve the appropriateness of stress imaging studies) and thereby contribute to increased efficiency in our healthcare system.
The authors acknowledge the invaluable assistance of Beth Kaping, RN, and Jo Carryer, RN, who performed the final patient classifications.
Supported by a grant from the Mayo Clinic Departments of Medicine and Radiology. Dr. Gibbons has received a research grant from King Pharmaceuticals, but it is not for support of this study. Dr. Miller has received a research grant from Bristol-Myers Squibb, but it is not for support of this study.
- Abbreviations and Acronyms
- American College of Cardiology/American Heart Association
- American College of Cardiology Foundation
- American Society of Nuclear Cardiology
- confidence interval
- myocardial perfusion imaging
- percutaneous coronary intervention
- single-photon emission computed tomography
- Received July 16, 2007.
- Revision received September 27, 2007.
- Accepted October 23, 2007.
- American College of Cardiology Foundation
- ↵(2001) Crossing the Quality Chasm (Institute of Medicine, Washington, DC), pp 1–22.
- Lucas F.L.,
- DeLorenzo M.A.,
- Siewers A.E.,
- Wennberg D.E.
- Patel M.,
- Spertus J.,
- Brindis R.G.,
- et al.
- Brindis R.G.,
- Douglas P.S.,
- Hendel R.C.,
- et al.
- Gibbons R.,
- Eckel R.,
- Jacobs A.K.
- Diamond G.A.
- Poornima I.G.,
- Miller T.D.,
- Christian T.F.,
- Hodge D.O.,
- Bailey K.R.,
- Gibbons R.J.
- McCully R.B.,
- Roger V.L.,
- Mahoney D.W.,
- et al.
- ↵Risk Assessment Tool for Estimating 10-Year Risk of Developing Hard CHD (Myocardial Infarction and Coronary Death). http://hp2010.nhlbihin.net/atpiii/calculator.asp?usertype=prof. Accessed January 29, 2007.
- ↵Estimating Coronary Heart Disease (CHD) Risk Using Framingham Heart Study Prediction Score Sheets. http://www.nhlbi.nih.gov/about/framingham/riskabs.htm. Accessed January 29, 2007.
- Eagle K.A.,
- Berger P.B.,
- Calkins H.,
- et al.
- Wennberg J.E.,
- Birkmeyer J.D.,
- Birkmeyer N.J.O.,
- et al.
- Bergeron S.,
- Ommen S.R.,
- Bailey K.R.,
- Oh J.K.,
- McCully R.B.,
- Pellikka P.A.