Author + information
- Received October 13, 2014
- Revision received January 6, 2015
- Accepted January 14, 2015
- Published online March 31, 2015.
- Jeffrey T. Kuvin, MD∗∗ (, )
- Amanda Soto, MSEd, EdD†,
- Lauren Foster, MS†,
- John Dent, MD, MS, MHCM‡,
- Andrew M. Kates, MD§,
- Donna M. Polk, MD‖,
- Barry Rosenzweig, MD¶ and
- Julia Indik, MD, PhD#
- ∗Cardiovascular Center, Tufts Medical Center, Boston, Massachusetts
- †National Board of Medical Examiners, Philadelphia, Pennsylvania
- ‡Cardiovascular Division, University of Virginia, Charlottesville, Virginia
- §Cardiovascular Division, Washington University School of Medicine, St. Louis, Missouri
- ‖Cardiovascular Medicine Division, Brigham and Women’s Hospital, Boston, Massachusetts
- ¶Leon H. Charney Division of Cardiology, New York University School of Medicine, New York, New York
- #Division of Cardiology, University of Arizona, Tucson, Arizona
- ↵∗Reprint requests and correspondence:
Dr. Jeffrey T. Kuvin, Cardiovascular Center, Tufts Medical Center, 800 Washington Street, Box 315, Boston, Massachusetts 02111.
Background The American College of Cardiology (ACC), in collaboration with the National Board of Medical Examiners (NBME), developed the first standardized in-training examination (ITE) for cardiovascular disease fellows-in-training (FITs). In addition to testing knowledge, this examination uses the newly developed ACC Curricular Milestones to provide specific, competency-based feedback to program directors and FITs. The ACC ITE has been administered more than 5,000 times since 2011.
Objectives This analysis sought to report the initial experience with the ITE, including feasibility and reliability of test development and implementation, as well as the ability of this process to provide useful feedback in key content areas.
Methods The annual ACC ITE has been available to cardiovascular disease fellowship programs in the United States since 2011. Questions for this Web-based, secure, multiple-choice examination were developed by a group of cardiovascular disease specialists and each question was analyzed by the NBME to ensure quality. Scores were equated and standardized to allow for comparability. Trainees and program directors were provided detailed feedback, including a list of the curricular competencies tested by those questions answered incorrectly.
Results The ITE was administered 5,118 times. In 2013, the examination was taken by 1,969 fellows, representing 194 training programs. Among the 3 training years, there was consistency in the examination scores. Total test scores and scores within each of the content areas increased with each FIT year (there was a statistically significant difference in each cohort’s average scale score across administration years). There was also significant improvement in examination scores across the fellowship years.
Conclusions The ACC ITE is a powerful tool available to all training programs to assess medical knowledge. This examination also delivers robust and timely feedback addressing individual knowledge gaps, and thus, may serve as a basis for improving training curricula.
- cardiovascular education
- clinical competence
Assessment of medical knowledge is an essential part of graduate medical education. An in-training examination (ITE) is a formal method for the evaluation of medical knowledge, and many specialty and subspecialty post-graduate training programs incorporate ITEs into their curricula. In fact, formal validated assessment of medical knowledge, such as an ITE, is a key component of the Accreditation Council for Graduate Medical Education (ACGME) program requirements (1,2). Internal Medicine training programs have incorporated a national ITE since 1988 (3). In recent years, medicine subspecialties, such as nephrology (4) and oncology (5), have developed computer-based ITEs in association with the National Board of Medical Examiners (NBME).
In 2011, the American College of Cardiology (ACC) developed and implemented the first standardized ITE for cardiology fellows-in-training (FITs). This secure examination is executed by the NBME and serves as a benchmark for trainees as they progress through training and prepare for the American Board of Internal Medicine (ABIM) certification examination. The goal of the cardiology ITE, like most ITEs, is to assess trainee medical knowledge through multiple-choice questions in the content areas specified by the ABIM. Moreover, the ACC ITE incorporates a unique feedback process whereby each examination question is tagged to the recently developed, competency-based ACC Curricular Milestones. Thus, after examination completion, program directors and FITs receive information that identifies knowledge gaps and areas in need of further curricular emphasis. During the past 3 years, the ITE has become the standard in-training examination in adult cardiology fellowship programs across the United States. This paper offers an analysis of the data accrued thus far, and outlines future directions for the ITE examination.
The cardiology ITE was developed by a test-writing committee consisting of a chair and 11 members (Online Appendix). Eight members of the test writing committee were fellowship training program directors and all committee members underwent instructional training for test question writing provided by the NBME. Questions were developed specifically for this examination and were reviewed in face-to-face committee meetings. Each question was designed to begin with a clinical stem and incorporate a single-best-answer, multiple-choice format. Many questions were accompanied by high-resolution still or video images. The NBME editors and psychometricians analyzed all questions for structure, content, and statistical performance.
The ITE was designed to assess the knowledge of general cardiovascular disease FITs. Each year’s examination consisted of 150 questions, separated into 5 equal sections of 30 questions apiece. FITs were allotted 1 h per session, with breaks after each session. The examination blueprint mirrors the ABIM certifying cardiovascular disease examination in terms of the percent of questions dedicated to each content area (Table 1).
The ACC provides information to cardiology fellowship training program directors and coordinators regarding the ITE via an internet listserv and FITs are registered for the examination online. The examinations are scheduled on 2 consecutive days in October, with a makeup date 1 week later. FITs take the examinations online via a secure website provided by the NBME. Neither test takers nor program directors are allowed access to the questions after the examination.
The goal of the ITE is to test medical knowledge across the six core ACGME competencies (1,2). Each examination question is linked to an ACC Curricular Milestone (6), which allows for direct feedback to fellows and program directors. Thus, after the examination responses are scored, fellows receive a detailed assessment of scores along with a list of curricular milestones associated with core clinical diagnoses (Figure 1) for each question answered incorrectly, with the objective of providing insight into knowledge gaps. In addition, each FIT receives an analysis of his or her own performance on the total test, represented by a score for the proportion of the 150 test questions answered correctly. FITs additionally receive scores for the proportion of items answered correctly in each of the major content areas, as well as the mean proportion answered correctly in each area by fellows in each year of training.
FIT scores are equated using Item Response Theory (7) and are placed on a standardized score scale set to a mean of 500 points and a standard deviation of 100 points. Equating allows for comparability in scale scores across each fellow year (first, second, and third) within a testing year, and across testing years. Thus, these processes allow for comparison across fellowship years and provide a measure of improvement during training. FITs are able to compare their individual performance with peers nationally by using the provided percentile ranks (for comparison within fellow year) and the equated scale scores (for comparisons across administrations of the test).
Program directors are provided with summary reports outlining overall FIT performance within an individual training program and across the country in each content area (with the FITs divided by fellowship year). Each program is provided with a histogram representing the distribution of mean percent correct scores across all programs. Program directors are provided with a breakdown of the individual performances of each of their individual FITs, the same performance data provided to the fellows themselves, including scale score, an overall percent correct score, and scores within each of the major content areas.
The ITE was administered to 3,388 FITs over a 3-year period with a total of 5,118 examination administrations (Table 2). The ITE was given to 1,319 FITs in 127 cardiology fellowship training programs in the first examination year (2011), to 1,732 FITs in 165 programs in the second year (2012), and to 1,969 FITs in 194 programs in the third year (2013). All training programs that participated in the initial 2 examinations participated in the third examination.
The ITE maintained comparable reliability during the 3 administration years, and had a reliability coefficient (alpha) of between 0.73 and 0.81 each year (Table 3) (8). The standard error of measurement, indicating the degree of precision in scores, fluctuated between 51 and 53 points across administrations, thereby indicating that if an individual were to retest on the same content, his or her expected score would be within approximately 53 scale score points of their original score.
A comparison of the means and standard deviations in scale scores across administrations and fellow years is provided (Figure 2). There was consistency in the examination scores between the 3 training years, with first-year fellows scoring similarly on each examination administration, with incremental improvement in average scores with additional years of fellowship training. Summary plots of training program and individual examinee performance are shown (Figures 3 and 4⇓⇓).
FIT scores tended to increase on the basis of total test performance and performance within the major content areas as trainees moved through the training years. This is reflected in a statistically significant difference in each cohort’s average scale score across administration years (Table 4). For example, second-year fellows in 2012 scored higher, on average, than first-year fellows did the year prior. Although there is not complete fellow overlap across years (that is, not all second-year FITs took the examination during their first year), the trend among those who took the ITE during consecutive training years is toward a higher score with more advanced training. There was an incremental and statistically significant improvement in examination scores across the fellowship years, indicating that within each administration, fellows with more advanced training scored higher. Second-year and third-year FIT performance increased significantly across administrations on the basis of 1-way analysis of variance (F[2,1316 = 114.56], F[2,1729 = 218.78], F[2,1966 = 221.37], p < 0.05).
The ACC’s cardiology ITE has become an important part of fellowship training (Central Illustration). The ITE assists in assessing fellow knowledge and allows training programs to adjust curriculum to address specific content areas. As shown in this analysis, the examination is robust in its ability to discern beginner knowledge compared with that of the advanced trainee. This type of knowledge assessment is key to allowing the trainee to focus on specific topics that are underdeveloped, and perhaps, under-recognized. Providing FITs with timely feedback on knowledge deficits allows the development of remediation plans during training, instead of discovering these knowledge deficits years later, and should help in preparation for the ABIM certifying examination. An ACC survey of cardiology training programs indicates that training directors are now using the results of the ITE to provide individualized counseling to fellows and to further develop training curricula (personal communication from Julia Bainbridge, American College of Cardiology, October 1, 2014). Future data will help determine whether scores on the ITE correlate with ABIM certifying examination scores. In years past, some training programs developed unique in-training modules to assess trainees. Now, there is a national, secure examination that allows trainees and programs to assess an individual fellow’s knowledge and provide comparison with trainees from around the country. The high number of repeating examinees and programs indicate that the ITE is a useful tool and is relevant to today’s training needs.
The ACC ITE provides a unique assessment tool for cardiology trainees and programs. The examination is linked to the newly developed ACC Curricular Milestones and provides valuable information about specific areas in need of further education and training. Each question is tagged to a specific competency statement within the curricular milestones and is designed to direct the examinee to content areas related to questions answered incorrectly. In addition, the examination was built on the ABIM cardiovascular medicine examination blueprint, with special emphasis on the 6 core competency areas. Thus, in addition to assisting in preparation for the ABIM certifying examination, the ITE provides fellows with the opportunity to identify areas of strength and weakness and to compare themselves to their peers across the country. On a national level, data derived from the ITE will also engender competency-based training by providing a broad understanding of the educational gaps of cardiology fellows.
The 3-year data presented indicate that the test is reliable and robust, as demonstrated by the statistically significant improvement in scores as FITs progress through training. This pattern is to be expected on several fronts. First, feedback received by first-year fellows allows them to better prepare for the second and third years of cardiovascular training. Second, training experiences and curricular knowledge increase commensurate with fellowship level.
There are limitations noted in the ITE experience thus far. Approximately 20% of questions are repeated across examination forms to allow for equating scores across administration years, and as such, it is possible that fellows may improve their score due to familiarity with previously viewed questions. Although the number of questions available for reuse across years is limited due to the ITE having only been in existence for 3 years, we expect this to change in ensuing years with the development of additional questions. Another limitation of the examination is the lack of a defined section dedicated to electrocardiograms and images. The test-writing committee, along with the NBME, is now working to develop new modules to better mirror the present-day ABIM certifying examination. Furthermore, there is potential for additional ITEs in the subspecialty areas of cardiovascular medicine, including electrophysiology, heart failure, interventional cardiology, and adult congenital heart disease. The ITE has been used in a few countries outside the United States. In future years, further international penetration of the ITE appears warranted. Finally, after a few more years of data are accrued, ACC ITE data will be compared to ABIM certification examination data. This will help identify trainees at risk for failing the ABIM examination and provide further guidance to program directors and fellows.
The ACC ITE, developed in conjunction with the NBME, tests medical knowledge of cardiology fellows in content areas that mirror the blueprint of the ABIM Cardiovascular Medicine certification examination. The ITE is a robust method of testing medical knowledge of cardiology FITs across years of training and an invaluable tool for program directors.
COMPETENCIES IN MEDICAL KNOWLEDGE, PRACTICE-BASED LEARNING, AND IMPROVEMENT: The ACC ITE tool assesses the medical knowledge of cardiovascular fellows-in-training, identifies knowledge gaps and provides competency-based feedback addressing each of the 6 core competencies defined by the ACGME.
TRANSLATIONAL OUTLOOK: Additional work is needed to identify test-related, trainee-related, and program-related factors that correlate with subsequent passing of the ABIM cardiovascular disease certifying examination or with clinical competence, but further experience and analysis of data from this examination will shed light on these relationships.
The authors thank Kristin Doermann and Julie Bainbridge, along with the leadership of the American College of Cardiology, for helping to develop this important teaching and learning tool for training programs. In addition, the authors thank the entire staff of the National Board of Medical Examiners, in particular, Molly Meehan, Drew Dillon, and Tricia Manning.
Ms. Foster and Ms. Soto are employees of the National Board of Medical Examiners. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- American Board of Internal Medicine
- American College of Cardiology
- Accreditation Council for Graduate Medical Education
- in-training examination
- National Board of Medical Examiners
- Received October 13, 2014.
- Revision received January 6, 2015.
- Accepted January 14, 2015.
- 2015 American College of Cardiology Foundation
- ↵ACGME Common Program Requirements, focused revision. American Council for Graduate Medical Education. 2013. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. Accessed January 21, 2015.
- ↵ACGME Program Requirement for Graduate Medical Education in Cardiovascular Disease (Internal Medicine). American Council for Graduate Medical Education. 2013. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/2013-PR-FAQ-PIF/141_cardiovascular_disease_int_med_07132013.pdf. Accessed January 21, 2015.
- Rosner M.H.,
- Berns J.S.,
- Parker M.,
- et al.,
- for the ASN In-Examination Training Committee
- Collichio F.A.,
- Kayoumi K.M.,
- Hande K.R.,
- et al.
- ↵American College of Cardiology competency statements. Available at: http://www.Cardiosource.Org/corecompetencies. Accessed January 19, 2015.
- Lord F.M.