Pediatric Appropriate Use Criteria Implementation ProjectA Multicenter Outpatient Echocardiography Quality Initiative
Author + information
- Ritu Sachdeva, MBBS∗∗ (sachdevar{at}kidsheart.com),
- Joseph Allen, MA†,
- Oscar J. Benavidez, MD‡,
- Robert M. Campbell, MD∗,
- Pamela S. Douglas, MD§,
- Benjamin W. Eidem, MD‖,
- Lara Gold, MA†,
- Michael S. Kelleman, MS, MSPH∗,
- Leo Lopez, MD¶,
- Courtney E. McCracken, PhD∗,
- Kenan W.D. Stern, MD#,
- Rory B. Weiner, MD‡,
- Elizabeth Welch, MD¶ and
- Wyman W. Lai, MD, MPH∗∗
- ∗Emory University School of Medicine and Children's Healthcare of Atlanta Sibley Heart Center Cardiology, Atlanta, Georgia
- †American College of Cardiology, Washington, DC
- ‡Massachusetts General Hospital, Boston, Massachusetts
- §Duke University, Durham, North Carolina
- ‖Mayo Clinic, Rochester, Minnesota
- ¶Miami Children’s Hospital, Miami, Florida
- #Children’s Hospital at Montefiore, New York, New York
- ∗∗Morgan Stanley Children’s Hospital of NewYork-Presbyterian, New York, New York
- ↵∗Reprint requests and correspondence:
Dr. Ritu Sachdeva, Emory University School of Medicine, Children’s Healthcare of Atlanta, 1405 Clifton Road, Atlanta, Georgia 30322.
Abstract
Background Recently published appropriate use criteria (AUC) for initial pediatric outpatient transthoracic echocardiography (TTE) have not yet been evaluated for clinical applicability.
Objectives This study sought to determine the appropriateness of TTE as currently performed in pediatric cardiology clinics, diagnostic yield of TTE for various AUC indications, and any gaps in the AUC document.
Methods Data were prospectively collected from patients undergoing initial outpatient TTE in 6 centers. TTE indications (appropriate [A], may be appropriate [M], or rarely appropriate [R]) and findings (normal, incidental, or abnormal) were recorded.
Results Of the 2,655 studies ordered by 102 physicians, indications rated A, M, and R were found in 1,876 (71%), 316 (12%), and 319 studies (12%), respectively, and 144 studies (5%) were unclassifiable. Twenty-four of 113 indications (21%) were not used. Innocent murmur and syncope or palpitations with no other indications of cardiovascular disease, a benign family history, and normal electrocardiogram accounted for 75% of indications rated R. Pathologic murmur had the highest yield of abnormal findings (40%). Odds of an abnormal finding in an A or M TTE were 6 times that of R (95% confidence interval [CI]: [2.8 to 12.8]). Abnormal findings were more common in patients <1 year of age than in those >10 years of age (odds ratio: 6.4; 95% CI: 4.7 to 8.7). Age was a significant predictor of an abnormal finding after adjusting for indication and site (p < 0.001).
Conclusions Most TTEs ordered in pediatric cardiology clinics were for indications rated A. AUC ratings successfully stratified indications based on the yield of abnormal findings. This study identified differences in the yield of TTE based on patient age and most common indications rated R. These findings should inform quality improvement efforts and future revisions of the AUC document.
Transthoracic echocardiography (TTE) has emerged as a powerful imaging tool in the field of cardiology, as it is relatively widespread, easy to use, and poses negligible risk. In 2005, the American College of Cardiology established the Appropriate Use Criteria (AUC) Task Force, which has developed documents providing appropriateness of indications for cardiac imaging and other procedures (1,2). AUC are designed to guide providers with individual patient care decisions, evaluate patterns of care by physicians over time, and to assist in improving quality of care. After years of experience in adult cardiology, this process was extended to pediatrics, and the first pediatric AUC in any area was recently published for TTE (3). This document focuses on the initial outpatient evaluation of pediatric patients, using TTE, which is an area of high use. The document was developed using a well-defined methodology that has been refined over the years (4).
Implementation studies evaluating the use of AUC in adult cardiology have been instrumental in recognizing the appropriateness of the imaging studies and identifying any deficiencies in the document (5–7). Furthermore, these studies have been helpful in engaging physicians in quality improvement projects related to test-ordering behavior and in lowering the number of tests for indications rated as rarely appropriate (8,9). The purpose of this study was to determine the clinical applicability of the AUC document in pediatric cardiology clinics in terms of care provided prior to the release of this document, to evaluate the diagnostic yield of the various AUC indications, and to identify any missing and unused indications in this document.
Methods
Study design
Six centers participated in this prospective cross-sectional quality improvement project, including Emory University School of Medicine and Children’s Healthcare of Atlanta, Children’s Hospital at Montefiore, Mayo Clinic Rochester, Massachusetts General Hospital, Miami Children’s Hospital, and Morgan Stanley Children’s Hospital of NewYork-Presbyterian. The proposal for this project was presented to the respective Institutional Review Boards of the participating sites and did not qualify as a research project for all centers except for Morgan Stanley Children’s Hospital of NewYork Presbyterian, where the study was reviewed and approved by their Institutional Review Board. Data were collected from April to September 2014, before the online release of the AUC document. All clinicians participating in the study were made aware of the data collection but were unaware of the contents of the AUC document. All patients ≤18 years of age undergoing initial TTE evaluation in a pediatric cardiology clinic were included. Patients previously evaluated by TTE during hospitalization or as outpatients, and those who were referred to a cardiology clinic only for a TTE without any clinical evaluation by the pediatric cardiologist, and those seen by investigators of this study were excluded.
Data collection and rating of indications
Patient and visit data, including patient demographics, date of study, name of the ordering physician, and reason for TTE, were entered on a data collection sheet by the clinician prior to ordering the TTE. The reasons for TTE were divided into 10 broad categories: murmur, chest pain, syncope, palpitations, abnormal electrocardiogram (ECG) results, abnormal test results other than ECG, systemic disorder, family history of cardiovascular disease, conditions that increase cardiovascular risk, and other reasons that did not fit into any of the above-listed categories. For patients with more than 1 indication for TTE, 1 was designated as the primary indication by the clinician, and the others were considered secondary indications. Site investigators reviewed the data collection sheet and medical records from the clinic visit to assign the AUC indication for the TTE by using the tables and flow diagrams in the AUC document (3). Indications were then rated as appropriate (A), may be appropriate (M), or rarely appropriate (R) based on the AUC document (4). If the clinical scenario did not fit into any of the 113 indications listed in the document, it was considered unclassifiable. If there was a question about assignment of indication to a case scenario, it was referred for adjudication (R.S. and W.W.L.).
All de-identified data were entered into the Research Electronic Data Capture (REDCap) system, which was maintained by the core site at Emory University School of Medicine and Children’s Healthcare of Atlanta. Individual sites maintained their own enrollment logs with their corresponding REDCap-assigned identification numbers.
Classification of TTE findings
Site investigators reviewed TTE findings and graded the severity of abnormal findings. TTE findings were classified as normal, incidental, or abnormal. Incidental findings included patent foramen ovale, small patent ductus arteriosus (PDA) in a neonate, peripheral pulmonary artery stenosis, left superior vena cava, tiny coronary artery fistula, retroaortic innominate vein, common origin of innominate artery and left common carotid artery, and left aortic arch with an aberrant right subclavian artery. Abnormal findings were divided into those that were related to the indication and those that were not. Each of these findings were further classified based on their clinical relevance as minor, moderate, or severe. Minor findings were other than those listed under incidental findings that might require follow-up but no intervention would be anticipated. Minor findings included small atrial septal defect (ASD), small ventricular septal defect (VSD), small PDA beyond the neonatal period, mild pulmonary stenosis, mitral valve prolapse, or bicuspid aortic valve without stenosis or insufficiency. Moderate findings were those that altered patient management but did not require urgent intervention (e.g., moderate-to-large secundum ASD, ASD other than secundum type, moderate-to-large VSD, moderate-to-large PDA, bicuspid aortic valve with stenosis or insufficiency, and anomalous aortic origin of a coronary artery). Severe findings were those that required urgent hospitalization or intervention (e.g., critical valvular stenosis, moderate or severe coarctation of aorta, severely reduced cardiac function, large pericardial effusion, or anomalous origin of coronary artery from pulmonary artery).
Study outcomes
The primary outcome measure was the number of studies ordered for indications rated R. Secondary outcome measures were the yield of abnormal findings for various AUC indications, influence of age on the diagnostic yield of TTE, indications for studies that were unclassifiable, and unused AUC indications.
Statistical analysis
Statistical analyses were performed using SAS version 9.4 software (Cary, North Carolina). Statistical significance was assessed at the 0.05 level. Descriptive statistics were calculated for all variables of interest and included medians and 25th to 75th interquartile (IQR) ranges and counts and percentages, when appropriate. Normality of continuous variables was assessed using histograms, normal probability plots, and the Anderson-Darling test for normality. Odds ratios (ORs) and 95% confidence intervals (CIs) were used to compare risks of abnormal findings in different patient subgroups by using logistic regression models. When we examined the odds of an abnormal finding among patients with both primary and secondary TTE indications versus those with only a primary indication, patients with only a primary indication were treated as the reference group. In addition, when we assessed the impact of TTE rating on odds of an abnormal finding, indications rated R were treated as the reference group and were compared to those with indications rated A or M. In a multivariable logistic regression model, we examined the impact of primary indication and age on the outcome abnormal finding, while controlling for clustering of patients within sites.
Results
During the study period, there were 2,655 initial outpatient TTE evaluations ordered by 102 physicians. The median age of the patients was 10 years (IQR: 3 to 15 years of age). The median number of TTE studies ordered per physician was 16 (IQR: 5 to 37 studies).
Indications and ratings of TTEs
Of 2,655 studies, the indications for ordering TTE were rated A in 1,876 studies (71%), M in 316 studies (12%), and R in 319 studies (12%). One hundred forty-four studies (5%) had a reason for TTE that was not listed in the AUC document and were therefore unclassifiable (Central Illustration). The 3 most common indications for TTEs rated A were pathologic murmur (n = 446), exertional chest pain (n = 355), and abnormal ECG without symptoms (n = 279). For those rated M, the 3 most common indications were positive family history of congenital left-sided heart lesion (n = 71); chest pain with other symptoms or signs of cardiovascular disease, a benign family history and a normal ECG (n = 41); and unexplained pre-syncope (n = 38). For those rated R, the 3 most common indications were presumptively innocent murmur (n = 172), syncope with no other symptoms or signs of cardiovascular disease, a benign family history and a normal ECG (n = 26); and palpitations with no other symptoms of signs of cardiovascular disease, a benign family history, and a normal ECG (n = 23).
The 5 most commonly used AUC indications, in order of frequency, were pathologic murmur (n = 446), exertional chest pain (n = 355), abnormal ECG without symptoms (n = 279), presumptively innocent murmur (n = 172), and presumptively innocent murmur with signs, symptoms, or findings of cardiovascular disease (n = 160). These 5 indications accounted for 53% of studies. One or more secondary indications were present in 533 patients (20%).
Diagnostic yield of TTE
Overall, 2,251 TTEs results (85%) were normal, 130 (5%) had only an incidental finding other than an isolated patent foramen ovale, and 274 (10%) had an abnormal finding with or without an incidental finding. The percentages of abnormal findings in each appropriateness category and unclassifiable indications are shown in the Central Illustration. The OR for an abnormal finding on TTE rated as A or M versus R was 6.0 (95% CI: 2.8 to 12.8). Table 1 shows the diagnostic yield of the 3 most frequently used AUC indications for each of the 9 tables provided in the AUC document (3). The frequency of abnormal findings varied significantly depending on the indication. Among the top 3 TTE indications rated A, pathologic murmur (AUC indication 41) had the highest yield of abnormal findings (178 of 446 [40%]), whereas exertional chest pain (AUC indication 30) had the lowest yield (2 of 355 [0.6%]). Abnormal findings were noted in 10.7% (227 of 2123) of studies with only a primary indication and 9% (47 of 532) of studies with a primary and a secondary indication (OR: 0.8; 95% CI: 0.6 to 1.2).
Diagnostic Yield of the 3 Most Frequently Used AUC Indications for Each Table Provided in the AUC Document
The 3 most common abnormal findings for each appropriateness category, severity of abnormal findings and the proportion of those related to the indication are shown in Table 2. The majority of abnormal findings (226 of 245 [92%]) for TTEs done for indications rated A were related to the indication, whereas for indications rated R, only 1 of 7 (14%) was related. That patient had a TTE performed for obesity, without having other cardiovascular risk factors (AUC indication 76) and was noted to have mild left ventricular hypertrophy on TTE. All 5 TTEs with severe abnormalities were done for indications rated A. The findings on these TTEs included double-outlet right ventricle with coarctation of aorta, dilated cardiomyopathy, severe pulmonary valve stenosis, moderate pulmonary hypertension with pericardial effusion, and coarctation of aorta. The 3 TTEs with moderate abnormalities for indications rated R had incidental discovery of a coronary anomaly that was unrelated to the indication for the study. All of the abnormal findings in TTEs done for unclassifiable reasons were noted to be minor, and 3 of the 8 were related to the reason for the study (finding of a bicuspid aortic valve in 2 patients and mitral valve prolapse in 1 patient, when the reason for the study was a click). Of the 130 studies that had only incidental findings, peripheral pulmonary artery stenosis (n = 87) and patent foramen ovale (n = 60) were the most common findings on TTE.
Most Common Abnormal Findings for Each Appropriateness Category, Severity of Abnormal Findings, and Percentage of Abnormal Findings Related to the Indication
Influence of age on diagnostic yield of TTEs
The age distribution for TTEs based on their primary indication is shown in Figure 1. The most common indication for TTE in children <10 years of age was a murmur (innocent or pathologic), whereas chest pain was the most common indication for those ≥10 years of age. Children with abnormal findings were much younger than those with normal findings (median age: 1 vs. 10 years, respectively; p < 0.001). The median age of patients who had TTEs performed for indications rated R was 6 years compared to 10 years for those in whom it was done for indications rated A or M (p = 0.003). The likelihood of detecting an abnormal finding was highest when TTE was performed in those <1 year of age versus in those >10 years of age (OR: 6.4; 95% CI: 4.7 to 8.7). After we adjusted for indication and site, younger age remained a significant predictor of an abnormal finding (OR [per 1-year decrease]: 1.04; 95% CI: 1.01 to 1.07; p = 0.0004).
Age Distribution of Patients at the Time of TTE Based on Primary Indication
The most common indication for transthoracic echocardiography (TTE) in children <10 years of age was a murmur (innocent or pathologic murmur), whereas in those ≥10 years of age, the most common indication was chest pain. CV = cardiovascular; ECG = electrocardiogram; FH = family history.
Gaps in the current AUC document
Of 2,655 TTEs, 144 (5%) were unclassifiable. Table 3 shows a summary of the unclassifiable scenarios. Family history in a member other than a first-degree relative and click on auscultation were the 2 most common unclassifiable reasons for echo. Twenty-four of the 113 indications (21%) were not used (Table 4) for 1 of the following reasons: overlap with other indications (7); rare condition in outpatient setting (7); overall rare condition (6); and indication rated R for which TTE was not ordered on any study patient (4).
Common Unclassifiable Reasons for Ordering Transthoracic Echocardiograms
Indications in the AUC Document That Were Not Used in the Current Study
Discussion
With the release of the first pediatric AUC, the field of pediatrics has entered territory that has been familiar to the world of adult cardiology for the past decade. This is the first pediatric AUC implementation study that has evaluated the appropriateness of TTEs performed in a large number of patients seen in pediatric cardiology clinics in multiple centers. We found that most TTEs ordered in pediatric cardiology clinics were for indications rated A (Central Illustration). AUC ratings successfully stratified the indications based on the yield of abnormal findings and identified the common indications rated R. The likelihood of detecting an abnormal TTE finding was the greatest when TTE was done in infants and the indication was rated A or M. This study also identified a small number of unclassifiable indications. One of the unique features of this study was that it was done prior to the release of the AUC document; therefore the physicians ordering the TTEs were unaware of the AUC ratings. This approach provided us with baseline information regarding the appropriateness of TTEs in pediatric cardiology practice. Furthermore, it identified the gaps in the current AUC document that will be important during future revisions to this document.
Pediatric Echocardiography AUC Implementation Study: TTE Appropriateness Rating and TTE Findings for Each Appropriateness Rating and Unclassifiable Studies
(Left) Appropriateness rating of 2,655 transthoracic echocardiogram (TTE) studies ordered by pediatric cardiologists. (Right) Percentages of normal, incidental, and abnormal findings by TTE for indications rated appropriate (n = 1,876), may be appropriate (n = 316), or rarely appropriate (n = 319) and unclassifiable indications (n = 144). AUC = appropriate use criteria.
Rating and findings of TTEs
Most TTE's performed for a wide array of clinical indications were rated A, and the rate of studies with indications rated R was only 12%. Although 85% of the findings on TTE were normal, most of the abnormal findings were seen in indications rated A, and the findings noted in R were either minor or not related to the indication itself. This suggests that the current AUC document can reasonably stratify clinical indications based on their yield of abnormal findings. This study identified some commonly used indications rated R, such as presumptively innocent murmurs, information that can be used to lay the framework for educational interventions to reduce the rate of TTEs ordered for these indications. In addition, the presence of 1 or more secondary findings did not affect the yield of abnormal findings.
In contrast, implementation studies using the initial adult AUC document for echocardiography revealed significant variations in appropriateness, ranging from 56% to 95% for the studies rated A (10–14). A recent meta-analysis reporting temporal changes in appropriateness of cardiac imaging showed that there was a temporal improvement in appropriateness of TTE (15), suggesting that as AUC are more widely accepted in clinical practice providers may be more conscious of their own ordering patterns.
Influence of age on diagnostic yield of TTE
Our study showed that there was a significant variation in the indications for a TTE in different age groups. Also, there was a much higher yield of abnormal findings in studies done in infants compared to older children. These findings are not surprising because a murmur was the most common indication in younger patients, and the highest yield of abnormal findings was in those with murmurs. Similarly, chest pain was the most common indication in older children and had a very low yield of abnormal findings. Although these findings may not be new to experienced physicians, their validation through the AUC document is of interest.
Gaps in the current AUC document
The current AUC document captured 95% of the clinical scenarios for TTEs ordered in pediatric cardiology clinics. The yield of abnormal findings in the unclassifiable studies was quite low in our study, and all findings were minor. Two noteworthy scenarios that formed a large proportion of unclassifiable studies were a family history in a member other than a first-degree relative, such as in cousins or grandparents, and the auscultatory finding of a click. It is important to note that the term “family history” in the current AUC document is applicable only to first-degree relatives. Finding of an isolated click or other well-known auscultatory abnormalities such as a gallop or rub are clinically important indications that should be included in future revisions. Furthermore, the indications for syncope could probably be refined by including children younger than 5 years of age, apnea, or apparent life-threatening events in infants, postural orthostatic tachycardia syndrome, dizziness, and unexplained syncope. Some additional indications that warrant attention are sinus tachycardia and hemangiomas prior to initiation of propranolol therapy. We realize that, despite our large sample size, there are some other situations in which TTE is used in clinical practice that may have escaped capture during our 6-month study period. One example involves TTE referral for sports clearance, either by a primary physician or upon parental request. It is also important to recognize that at times there are situations in the pediatric cardiology clinic when patients, parents, or referring physicians have an expectation that TTE will be performed, despite the cardiologist’s preference to not do it. These issues are not addressed by the current AUC document.
It is important to note that our study identified a relatively small gap compared to some of the studies using the initial adult AUC document, which reported as much as one-third of their studies as unclassifiable (11). It is understandable that an AUC document can by no means include every single possible case scenario one could encounter in clinical practice.
Our study also identified 24 unused indications and the possible reasons for lack of their use. During future revisions of the document due consideration could be given to avoid the overlap and consolidate some of the rare conditions in outpatient settings.
Future directions
This study has laid the foundation for future efforts designed to improve the appropriateness of TTEs ordered in outpatient pediatric setting. Based on the information provided by this study, educational interventions could be targeted to reduce the use of TTE for common indications rated R. Such interventions for adult providers have included webinars, lectures, pocket cards, and personal feedback via e-mails. Studies have shown variable results in terms of achieving this goal with some showing a high rate of success and others showing none (8,9,16). Although our study extensively evaluated the diagnostic yield of various indications, the next step could be evaluation of the clinical impact of the AUC in terms of any change in patient care and its outcomes. A study in adults showed that despite the fact that 90% of their TTEs were rated A, fewer than 1 in 3 resulted in an active change in care (17). Despite the results of this study, it is important to emphasize that when dealing with the pediatric population, there may be significant value in having negative study results (i.e., normal examination findings) in several of the indications rated A. For example, the yield of TTE for chest pain with exertion, an indication rated A, was quite low for any abnormal findings, and results for most studies were normal. However, the value of such negative results lies in reassurance and lifting of any restrictions on sports activities that may have been placed on these children prior to their visit to the cardiologist.
Study limitations
Determination of indication for TTE may not have been uniform among centers. In addition, there may be an element of bias, because site investigators were not blinded to the results of TTE, even though the results of TTE were reviewed after assigning the AUC indication. Also, the clinicians were made aware of data collection for the study, and this could have influenced their decision to order TTEs. Although the AUC document applies to a TTE ordered by any provider, our study included only patients that were evaluated by a pediatric cardiologist because the orders received for TTEs by other providers lacked details that were needed to assign an AUC indication. Even though the bulk of outpatient TTE's in the real world are ordered by pediatric cardiologists, the findings of this study may not be generalizable as the distribution of appropriateness rating may vary when the TTE is ordered by primary care providers versus a specialist. We would also like to acknowledge that this investigation was not performed by an independent team as some of the authors on this study were involved with the development of the AUC document. In addition, this study was supported by the American College of Cardiology Appropriate Use Criteria Task Force.
Conclusions
The clinical application of the first pediatric AUC was feasible in this large multicenter study. Most TTEs ordered by pediatric cardiologists for initial outpatient evaluation were appropriate, and the majority of the abnormal findings noted in this study were for indications rated A or M. The pediatric AUC document was able to successfully classify most studies. Future refinement of the AUC document should consider the relatively small but important gaps identified through this effort.
COMPETENCY IN SYSTEM-BASED PRACTICE: Baseline data for the use of transthoracic echocardiography in pediatric patients before the release of evidence-based AUC could be used by individual providers and hospitals as benchmarks against which to compare uses of this resource as ambulatory practice evolves.
TRANSLATIONAL OUTLOOK: Educational and quality improvement programs should address the impact of AUC not only on use of the targeted resource but on unintended use of alternative diagnostic technologies, cost, and ultimately on clinical outcomes.
Acknowledgments
The authors acknowledge American College of Cardiology Appropriate Use Criteria Task Force for support throughout this project; Destinee Adams and Carol Dickens for building and managing Children's Healthcare of Atlanta REDCap data collection project; Tuan Nguyen, MD, for data entry at Miami Children's Hospital; Heather Friedman, MPH, Cardiovascular Imaging Research Core, Children's Healthcare of Atlanta, for help with data entry into REDCap; and Emory University Children’s Pediatric Biostatistics Core for help with data analysis.
Footnotes
The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
Listen to this manuscript's audio summary by JACC Editor-in-Chief Dr. Valentin Fuster.
- Abbreviations and Acronyms
- A
- appropriate
- ASD
- atrial septal defect
- AUC
- appropriate use criteria
- ECG
- electrocardiogram
- M
- may be appropriate
- OR
- odds ratio
- PDA
- patent ductus arteriosus
- R
- rarely appropriate
- TTE
- transthoracic echocardiography
- VSD
- ventricular septal defect
- Received April 29, 2015.
- Revision received June 16, 2015.
- Accepted June 22, 2015.
- American College of Cardiology Foundation
Podcast