Author + information
- Received March 27, 2017
- Accepted April 9, 2017
- Published online June 5, 2017.
- Julia H. Indik, MDa,∗ (, )
- Lauren M. Duhigg, MPHb,
- Furman S. McDonald, MD, MPHb,
- Rebecca S. Lipner, PhDb,
- Jonathan D. Rubright, PhDc,
- Steven A. Haist, MD, MSc,
- Naomi F. Botkin, MDd and
- Jeffrey T. Kuvin, MDe
- aUniversity of Arizona College of Medicine, Tucson, Arizona
- bAmerican Board of Internal Medicine, Philadelphia, Pennsylvania
- cNational Board of Medical Examiners, Philadelphia, Pennsylvania
- dUMass Memorial Medical Center, Worcester, Massachusetts
- eDartmouth-Hitchcock Medical Center, Lebanon, New Hampshire
- ↵∗Address for correspondence:
Dr. Julia H. Indik, Sarver Heart Center, University of Arizona College of Medicine, 1501 North Campbell Avenue, Tucson, Arizona 85724.
The American College of Cardiology In-Training Exam (ACC-ITE) is incorporated into most U.S. training programs, but its relationship to performance on the American Board of Internal Medicine Cardiovascular Disease (ABIM CVD) Certification Examination is unknown. ACC-ITE scores from third-year fellows from 2011 to 2014 (n = 1,918) were examined. Covariates for regression analyses included sex, age, medical school country, U.S. Medical Licensing Examination Step, and ABIM Internal Medicine Certification Examination scores. A secondary analysis examined fellows (n = 511) who took the ACC-ITE in the first and third years. ACC-ITE scores were the strongest predictor of ABIM CVD scores (p < 0.0001), and the most significant predictor of passing (p < 0.0001). The change in ACC-ITE scores from first to third year was a strong predictor of the ABIM CVD score (p < 0.001). The ACC-ITE is strongly associated with performance on the ABIM CVD Certification Examination.
- American Board of Internal Medicine
- cardiovascular disease fellowship program
- cardiovascular education
- medical knowledge assessment
Competency assessment during cardiovascular training is complex, and incorporates a variety of formative and summative evaluations (1). Assessment of medical knowledge is critical during fellowship and is required by the Accreditation Council for Graduate Medical Education. A national standardized in-training examination (ITE) is an ideal way to provide meaningful feedback that can be used across training programs, and they have also been developed in other specialties (2–5). We have previously shown that the American College of Cardiology (ACC) ITE is an effective method of testing fellow-in-training (FIT) knowledge of cardiovascular medicine in a model that mirrors the American Board of Internal Medicine (ABIM) Cardiovascular Disease (CVD) Certification Examination (6).
The ACC-ITE is an online, secure, multiple choice examination that includes clinical questions often requiring data and image analysis. In addition to feedback provided by scores for each section within the ABIM blueprint (7), fellows and program directors receive the competency statement (8) and diagnosis for each question answered incorrectly. Thus, the ITE quantifies FIT knowledge gaps and informs training programs about areas in need of focused education for the individual fellow, as well as the fellowship training program. The ITE is able to discern beginner medical knowledge from that of an advanced fellow (6), and the test has been incorporated into most U.S. training programs, with 195 cardiology fellowship programs participating in 2014, in comparison with a total of 210 Accreditation Council for Graduate Medical Education–accredited cardiology fellowship programs in the United States as of the 2016 to 2017 academic year (9). Whether or not FIT performance on the ITE correlates with future scores on the ABIM CVD Certification Examination is not known. This analysis was performed to determine the association between FIT performance on the ACC-ITE to subsequent ABIM CVD Certification Examination scores.
The study sample consisted of CVD FITs who took the ACC-ITE between 2011 and 2014, and subsequently attempted the ABIM CVD Certification Examination. CVD FITs were matched with ABIM data by first name, last name, and date of birth. FITs that lacked program director ratings were excluded. The primary analyses used the ACC-ITE score from the third year of fellowship training, with a secondary analysis that incorporated performance from the first year of fellowship training. Performance-related covariates included first-attempt scores on the ABIM CVD Certification Examination, the ABIM Internal Medicine (IM) Certification Examination, and the U.S. Medical Licensing Examination (USMLE) Steps 1, 2 (clinical knowledge), and 3. Demographic data included sex, age at the ACC-ITE, and medical school country.
The ACC-ITE has been previously described (6). In brief, this examination was developed by a committee (Online Appendix), the majority of whom are cardiology fellowship program directors, and all of whom were trained in question writing by the National Board of Medical Examiners. The ITE uses the blueprint from the ABIM CVD Certification Examination. All ITE questions are written in a single best answer, multiple choice format, and still images and videos are often incorporated. The examination comprises a total of 150 questions. The National Board of Medical Examiners oversees the performance of questions, including psychometric analyses. The examination is given on specific dates in the fall via a secure website. Scores are equated using item response theory (10), allowing for comparisons between different FIT years and across years, and are scaled to a standardized score scale ranging from 1 to 999, with a mean of 500 and SD of 100 points.
A unique aspect of the ACC-ITE is the feedback provided to each trainee and program director. A report is generated that includes the diagnosis tested and the ACC curricular competency (8) for each question answered incorrectly. Each examination is graded with an overall score, as well as a score within each major content area. In addition, FIT performance is compared with that of other trainees at the same year of training within programs and across the United States. Percentile ranks are also provided. Program directors receive a histogram to describe the distribution of mean percent correct scores of all programs, in addition to the performance report of each of their FITs.
The USMLE Steps are intended to assess knowledge and patient care principles, and are taken at 3 different time points in medical school and training. Satisfactory completion of these examinations is required for medical licensure of allopathic physicians. The USMLE Steps included here are computer-based examinations, which include both multiple choice questions and case simulations. Each Step examination is scored with item response theory on a range from 1 to 300, with a mean set to 200 and SD of 20. The reliability coefficient (Cronbach’s alpha) (11) for the USMLE Steps has been reported as at least 0.87 (4,5).
ABIM IM Certification Examination
The ABIM IM Certification Examination is generally taken within the first year after completion of IM residency training. The ABIM IM Certification Examination is a computer-based examination of 240 multiple choice questions, of which 205 compose the overall score. The score is computed using item response theory, and scores are equated and standardized on a scale of 200 to 800, mean of 500 and SD of 100 points. The reliability coefficient (Cronbach’s alpha) (11) of the ABIM IM Certification Examination has ranged from 0.89 to 0.91 across administrations.
ABIM CVD Certification Examination
The ABIM CVD Certification Examination is generally taken in the first year after completion of general CVD fellowship training. Similar to the ABIM IM Certification Examination, the ABIM CVD Certification Examination consists of 240 multiple choice questions, of which 200 compose the overall multiple choice questions score. The scaled score is from 200 to 800, with a mean of 500 and SD of 100 points. The examination’s reliability coefficient (Cronbach’s alpha) (11) was 0.85 to 0.87 across administrations in our study period.
The ABIM CVD Certification Examination also has separate portions of the examination for electrocardiography and cardiovascular imaging interpretation. A passing score on these parts is required to pass the ABIM CVD Certification Examination. In this investigation, we only analyzed the multiple choice question section of the ABIM CVD Certification Examination.
Descriptive data are presented as mean ± SD, and statistical analyses were on the basis of scaled scores. A Pearson correlation coefficient was used to assess the correlation between the ACC-ITE and ABIM CVD Certification Examination scores. Multiple linear regression was used to examine the relationship between scores on the ACC-ITE and the ABIM CVD Certification Examination. Multiple logistic regression was used to assess the association between ACC-ITE score and passing the ABIM CVD Certification Examination. Covariates in both models included scores on the ACC-ITE taken in the third year of fellowship training, USMLE Steps 1, 2 (clinical knowledge), and 3 (ABIM IM Certification Examination), as well as categorical demographic variables, including sex (women as reference group) and medical school country (United States/Canadian or international), with international being the reference group. A receiver-operating characteristic analysis was performed to assess the overall accuracy of the model’s ability to predict passing the ABIM CVD Certification Examination. The ACC-ITE score from the third year of fellowship training was used for these primary analyses.
In a secondary analysis, the ACC-ITE score from the first year of fellowship training was used for a subset of individuals who had taken the examination in both the first and third years of fellowship training to assess the relative predictability of examination attempts in the first compared with the third year, and for the change in examination score from the first to the third year. Regression (β) coefficients (unstandardized and standardized) are reported for linear regression, and odds ratios for logistic regression. Adjusted R2 and maximum rescaled R2 values are reported to describe the strength of the models. Significance was set at p < 0.05. Statistical tests were performed using SAS 9.3 software (SAS Institute Inc., Cary, North Carolina). Data were deidentified, and Institutional Review Board approval with exempt status was granted by the American Institutes for Research, Washington, DC.
From 2011 to 2014, there were 2,015 third-year CVD FITs who took the ACC-ITE, of whom 2,011 matched with ABIM examination data. Of these 2,011 FITs, 77 had no ABIM CVD Certification Examination score. An additional 16 FITS were excluded for missing program director ratings (although this parameter was not used in analyses), yielding a study population of 1,918 individuals. Men were 82% of the cohort, and 59% of individuals attended medical school in the United States or Canada (Table 1).
ACC-ITE scores (from the third year of fellowship training) were significantly correlated with scores on the first attempt at the ABIM CVD Certification Examination, with a correlation coefficient of 0.68 (p < 0.0001) (Figure 1). In a multiple linear regression analysis, ACC-ITE score was the strongest predictor of the ABIM CVD Certification Examination score, (β = 0.38; t = 20.31; p < 0.0001), followed by ABIM IM Certification Examination score (β = 0.25; t =12.32; p < 0.0001). In other words, higher scores on the ACC-ITE and the ABIM IM Certification Examination predicted higher scores on the ABIM CVD Certification Examination. Other significant predictors included USMLE Steps 1, 2, and 3 scores, age, sex, and medical school country (Table 2). Simply stated, better performance on the Step exams, being younger when taking the ITE examination, being male, and having trained in the United States or Canada predicted higher scores on the ABIM CVD Certification Examination. The adjusted R2 for the full model was 0.61.
In the multiple logistic regression model, the strongest predictors of passing the ABIM CVD Certification Examination on the first attempt were the ACC-ITE and ABIM IM Certification Examination scores (Table 3). The maximum rescaled R2 for the full logistic model incorporating all covariates was 0.38. The area under the curve was 0.9 in a receiver-operating characteristic analysis for the model (Figure 2). The probability of passing the ABIM CVD Certification Examination exceeded 90% for an ACC-ITE score of approximately 500 (Central Illustration).
A secondary analysis was performed to assess the predictive strength of the change in ACC-ITE score from the first to the third year of fellowship training. The model included the ACC-ITE score in the first year, the change in ITE score from the first to third year, ABIM IM Certification Examination score, USMLE Step examination scores, age, sex, and medical school country. There were 511 individuals who had taken the ACC-ITE in both the first and third fellowship years. The multiple linear regression model (Table 4) showed the ACC-ITE score in the first year to be the strongest predictor (β = 0.46; t = 11.42; p < 0.001), followed by the change in ITE score (β = 0.29; t = 9.45; p < 0.001), and the ABIM IM Certification Examination score (β = 0.24; t = 5.97; p < 0.001).
This analysis showed a significant correlation between scores on the ACC-ITE and the ABIM CVD Certification Examination. In fact, the strongest predictor of the ABIM CVD Certification Examination score was the ITE score, followed by scores on prior standardized testing, such as the ABIM IM Certification Examination and USMLE Step examinations. This indicates that the ACC-ITE is a valuable predictive tool, and the results should be taken seriously by both the trainee and the training program. In addition, the secondary analysis demonstrated that the improvement in scores from the first to the third year was independently predictive of passing the ABIM CVD Certification Examination. This supports a strategy to use the ITE examination in all years of training to identify knowledge gaps and provide program directors the opportunity to address these gaps early on so as to improve the likelihood of passing the ABIM CVD Certification Examination.
Since 2011, the ACC-ITE has made a substantial impact on fellows and training programs by assessing medical knowledge and providing direct feedback to further direct learning. Nearly all ACMGE-accredited cardiology fellowship programs (210 programs as of the academic year 2016 to 2017) (9) participate in the ACC-ITE examination, with 195 cardiology fellowship programs having participated in 2014, the last year of this analysis. Thus, the ACC-ITE has become the primary testing tool during training in U.S. CVD fellowship programs. ACC-ITE questions are developed by cardiologists, and focus on key learning objectives and competencies. It was previously reported that ACC-ITE scores increase with each fellowship year as a cohort progresses through training (6). The report given to the trainee and program director provides the competency statement and diagnosis of every question answered wrong, which gives program directors and trainees the level of detail they need to formulate educational interventions. The ACC-ITE has become an important aid for curriculum design and tailored learning activities, by providing an improved understanding of the educational needs and gaps among cardiology FITs throughout the 3 years of fellowship training. Until now, it has been unclear as to whether or not scores on the ITE correlate with outcomes on the ABIM CVD Certification Examination. This investigation now establishes the association between performance on the ACC-ITE examination taken in the third year of training with performance on the ABIM CVD Certification Examination, and provides evidence that closing knowledge gaps reflected in an increasing ACC-ITE score from the first to third years also is independently associated with better performance on the ABIM CVD Certification Examination.
Shortly after the completion of general cardiovascular fellowship training, cardiologists take the ABIM CVD Certification Examination. The ABIM CVD Certification Examination is designed to test the Accreditation Council for Graduate Medical Education core competencies of medical knowledge and patient care in the major cardiovascular areas, including arrhythmias, coronary artery disease, acute coronary syndromes, valvular disorders, congenital disorders, pericardial disease, aortic and peripheral vascular disease, hypertension and pulmonary hypertension, pharmacology, congestive heart failure, physiology, and biochemistry (7). Candidates are assessed with multiple choice questions, and a separate section of image and electrocardiogram analysis. Certification is not government mandated for physicians to practice medicine; rather, it is used as an indicator of competency, and is often mandated by employers, hospitals, and payers (12). With regard to IM, certification by the ABIM has been shown to be associated with providing better care to patients (13), and ABIM certification, as well as a high performance on the ABIM-IM examination, is associated with a lower risk of disciplinary actions from state medical boards (14,15). A majority of first-time takers (>90%) pass the ABIM CVD Certification Examination (16). Before the development of the ACC-ITE, cardiology FITs did not have a standardized tool to measure medical knowledge and assess the likelihood of passing the ABIM CVD Certification Examination.
Numerous graduate medical training programs, including specialty programs, use ITEs to help trainees identify knowledge gaps and prepare for certification. Previous studies have shown a positive correlation between ITEs and outcomes of certifying examinations in both specialty and subspecialty programs (2–5). The USMLE Steps have also been correlated with perform ance on the IM ITE (17). This study adds to the growing body of published reports demonstrating the association of an ITE to board certification examination performance. ITEs serve as an objective guide for personal and curricular development. The ITE can also inform trainees and program directors about potential risks of failing an upcoming certification examination and likely scores, which can have future implications for the early career professional.
There may be other variables or interactions with unknown factors not included in this analysis that may be associated with performance on the ABIM CVD Certification Examination. Although the entire cohort was large, the secondary analysis to assess the predictive strength of the change in ACC-ITE score from the first to the third fellowship year was limited to only approximately one-fourth of the group. Second, the data only reflect training programs in the United States. Certainly, because the ITE is administered across the world, further global and regional analyses are warranted. Finally, although passing the ABIM CVD Certification Examination is an important milestone, assessment of physician competency does not rest on 1 examination.
Given the acceptance of the ACC-ITE in general cardiology training programs across the United States, along with its prognostic importance, there is growing interest in developing subspecialty ITEs in electrophysiology, heart failure, interventional cardiology, and adult congenital heart disease, to name a few. Further expansion of the ITE outside the United States could provide internationally trained fellows valuable feedback in their knowledge of cardiology throughout their training.
The ACC-ITE, developed in conjunction with the National Board of Medical Examiners, is a robust tool that assesses medical knowledge, identifies gaps, and importantly, is a predictor of passing the ABIM CVD Certification Examination. The ITE should help cardiology FITs and program directors to focus on learning and curricular development during the training years to improve medical knowledge and address knowledge deficits. In addition, the ITE is a good indicator of how a trainee will perform on the ABIM CVD Certification Examination.
The authors thank the American College of Cardiology for developing tools to help educate and train cardiology fellows. The authors also thank the National Board of Medical Examiners and the American Board of Internal Medicine for collaborating on this project.
For a list of the In-Training Examination Writing Group (2011 to 2016), please see the online version of this article.
The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- American Board of Internal Medicine
- American College of Cardiology
- cardiovascular disease
- internal medicine
- in-training examination
- National Board of Medical Examiner
- U.S. Medical Licensing Examination
- Received March 27, 2017.
- Accepted April 9, 2017.
- 2017 American College of Cardiology Foundation
- Kuvin J.T.,
- Williams E.S.
- Collichio F.A.,
- Hess B.J.,
- Muchmore E.A.,
- et al.
- Grabovsky I.,
- Hess B.J.,
- Haist S.A.,
- et al.
- Lohr K.M.,
- Clauser A.,
- Hess B.J.,
- et al.,
- American College of Rheumatology Committee on Rheumatology Training and Workforce Issues
- Kuvin J.T.,
- Soto A.,
- Foster L.,
- et al.
- ↵American Board of Internal Medicine Cardiovascular Disease Certification Examination Blueprints. American Board of Internal Medicine. 2017. Available at: http://www.abim.org/∼/media/ABIM%20Public/Files/pdf/exam-blueprints/certification/cardiovascular-disease.pdf. Accessed April 13, 2017.
- Williams E.S.,
- Halperin J.L.,
- Arrighi J.A.,
- et al.
- ↵Number of Accredited Programs Academic Year 2016-2017 United States. ACGME. 2017. Available at: https://apps.acgme.org/ads/Public/Reports/Report/3. Accessed April 13, 2017.
- Lord F.M.
- Lipner R.S.,
- Young A.,
- Chaudhry H.J.,
- Duhigg L.M.,
- Papadakis M.A.
- ↵American Board of Internal Medicine. First-time taker pass rates – initial certification. Available at: http://www.abim.org/∼/media/ABIM%20Public/Files/pdf/statistics-data/certification-pass-rates.pdf. Accessed April 13, 2017.