Author + information
- Published online January 15, 2013.
- Alice K. Jacobs, MD, FACC, FAHA, Chair, Workgroup Authors⁎,
- Frederick G. Kushner, MD, FACC, FAHA, Workgroup 1 Chair, Workgroup Authors,
- Steven M. Ettinger, MD, FACC, Workgroup 2 Chair, Workgroup Authors,
- Robert A. Guyton, MD, FACC, Workgroup 3 Chair, Workgroup Authors,
- Jeffrey L. Anderson, MD, FACC, FAHA, Workgroup 4 Chair, Workgroup Authors,
- E. Magnus Ohman, MD, FACC, Workgroup 5 Chair, Workgroup Authors,
- Nancy M. Albert, PhD, CCNS, CCRN, FAHA, Workgroup Author,
- Elliott M. Antman, MD, FACC, FAHA, Workgroup Author,
- Donna K. Arnett, PhD, MD, FAHA, Workgroup Author,
- Marnie Bertolet, PhD, Workgroup Author,
- Deepak L. Bhatt, MD, MPH, FACC, FAHA, Workgroup Author,
- Ralph G. Brindis, MD, MPH, MACC, Workgroup Author,
- Mark A. Creager, MD, FACC, FAHA, Workgroup Author,
- David L. DeMets, PhD, Workgroup Author,
- Kay Dickersin, PhD, Workgroup Author,
- Gregg C. Fonarow, MD, FACC, FAHA, Workgroup Author,
- Raymond J. Gibbons, MD, FACC, FAHA, Workgroup Author,
- Jonathan L. Halperin, MD, FACC, FAHA, Workgroup Author,
- Judith S. Hochman, MD, FACC, FAHA, Workgroup Author,
- Marguerite A. Koster, MA, MFT, Workgroup Author,
- Sharon-Lise T. Normand, PhD, Workgroup Author,
- Eduardo Ortiz, MD, MPH, Workgroup Author,
- Eric D. Peterson, MD, MPH, FACC, FAHA, Workgroup Author,
- William H. Roach Jr, Esq, Workgroup Author,
- Ralph L. Sacco, MD, MS, FAHA, Workgroup Author,
- Sidney C. Smith Jr, MD, FACC, FAHA, Workgroup Author,
- William G. Stevenson, MD, FACC, FAHA, Workgroup Author,
- Gordon F. Tomaselli, MD, FAHA, Workgroup Author,
- Clyde W. Yancy, MD, FACC, FAHA, Workgroup Author,
- William A. Zoghbi, MD, FACC, FAHA, Workgroup Author,
- John G. Harold, MD, MACC, FAHA, Guest⁎,
- Yulei He, MS, PhD, Guest,
- Pamela B. Mangu, MA, Guest,
- Amir Qaseem, MD, PhD, MHA, FACP, Guest,
- Michael R. Sayre, MD, FAHA, Guest and
- Mark R. Somerfield, PhD, Guest
ACCF/AHA Task Force Members
Alice K. Jacobs, MD, FACC, FAHA, Immediate Past Chair
Jeffrey L. Anderson, MD, FACC, FAHA, Chair
Jonathan L. Halperin, MD, FACC, FAHA, Chair-Elect
Nancy M. Albert, PhD, CCNS, CCRN, FAHA
Mark A. Creager, MD, FACC, FAHA
David L. DeMets, PhD
Steven M. Ettinger, MD, FACC
Robert A. Guyton, MD, FACC
Judith S. Hochman, MD, FACC, FAHA
Frederick G. Kushner, MD, FACC, FAHA
E. Magnus Ohman, MD, FACC
William G. Stevenson, MD, FACC, FAHA
Clyde W. Yancy, MD, FACC, FAHA
Table of Contents
1. Workgroup 1: Clinical Practice Guidelines We Can Trust......219
1.1 Workgroup 1 Questions for Consideration......220
1.2 Workgroup 1 Recommendations......223
2. Workgroup 2: Standards for Initiating a Systematic Review......223
2.1 Workgroup 2 Questions for Consideration......224
2.2 Workgroup 2 Recommendations......226
3. Workgroup 3: Standards for Finding and Assessing Individual Studies......226
3.1 Workgroup 3 Recommendations......227
4. Workgroup 4: Standards for Synthesizing the Body of Evidence......229
4.1 Workgroup 4 Questions for Consideration......229
4.2 Workgroup 4 Recommendations for Assessing and Addressing the Quality of a Body of Evidence......231
4.3 Workgroup 4 Recommendations for Standards for Quantitative Analysis Across a Body of Evidence......232
4.4 Workgroup 4 Recommendations Regarding ACCF/AHA Grading Methodology and Nomenclature......236
4.5 Workgroup 4 Additional Recommendations......238
5. Workgroup 5: Standards for Reporting Systematic Reviews......238
5.1 Workgroup 5 Recommendations......241
Appendix 1. Author Listing of Comprehensive Relationships With Industry and Others......243
Appendix 2. Workgroup Comparison Tables......250
Workgroup 1 Comparison Table......250
Workgroup 2 Comparison Table......253
Workgroup 3 Comparison Table......256
Workgroup 4 Comparison Table......261
Workgroup 5 Comparison Table......263
The American College of Cardiology Foundation (ACCF) and the American Heart Association (AHA) have jointly engaged in the development of clinical practice guidelines (CPGs) for nearly 3 decades, based on the shared vision of their responsibility to provide guidance to cardiovascular healthcare professionals and the patients they serve by translating the best available evidence into clinical practice.
The ACCF/AHA Task Force on Practice Guidelines (Task Force) oversees and directs the CPG development process and methodology that have been the foundation of our documents and responsible for their widespread recognition. In brief, once a topic is selected for a new, revised, or updated CPG, selected organizations and professional societies with similar interests and expertise are invited to participate as partners or collaborators, with the overall ACCF/AHA policy of being inclusive and collaborative. Next, a guideline writing committee (GWC) chair is selected by the Task Force, and together with the chair, potential GWC members are identified, based on a detailed and specific relationship with industry and other entities (RWI) policy stating that the chair and the majority of GWC members must have no relevant RWI. Once formed, the GWC outlines the content of the document, performs a detailed and specific evidence acquisition and review, drafts recommendations with limited preliminary text, and attends a consensus conference with GWC members of related CPGs where recommendations are vetted and reconciled across existing guidelines. After the conference, the document is finalized. Every recommendation is voted on by each GWC member with appropriate recusal based on relevant RWI. The finished document then undergoes extensive peer review and response, and any recommendation that is changed is voted on again by each GWC member. The final document is reviewed, potentially revised and approved by the Task Force, and sent to the ACCF Board of Trustees, the AHA Science Advisory and Coordinating Committee, and partnering or collaborating organizations for final approval. After this very thorough and robust process, the CPG has essentially been vetted throughout the academic and clinical community and is jointly published in the Journal of the American College of Cardiology and Circulation. Once published, it serves as official policy of both organizations, informing strategic initiatives, advocacy, programs, products, and services.
Of critical importance is the continual evolution of the development process and methodology that has characterized the ACCF/AHA CPGs and the ongoing challenge to timely respond to and integrate the continuous stream of new knowledge.
Commissioned by the U.S. Congress as part of the Medicare Improvement for Patients and Providers Act of 2008 to set standards for CPGs, the Institute of Medicine (IOM) issued 2 reports: “Clinical Practice Guidelines We Can Trust” (1) and “Finding What Works in Health Care: Standards for Systematic Reviews” (2) in March 2011. The IOM redefined CPGs as follows:
“Clinical practice guidelines are statements that include recommendations intended to optimize patient care that are informed by a systematic review of the evidence and an assessment of the benefits and harms of alternative care options.” (1, p. 29)
The underpinnings of this definition are that a CPG should be based on a systematic review (SR) of existing evidence; be developed by a knowledgeable, multidisciplinary panel of experts and key stakeholders; consider important patient subgroups and preferences as appropriate; be based on a transparent process that minimizes conflicts of interest and biases; provide a clear explanation of the logical relationships between alternative care options and health outcomes; provide the ratings of both the strength of the recommendation and the quality of evidence; and be revised as appropriate based on new evidence (1,2).
In view of IOM reports and the ACCF/AHA ongoing improvement processes, the Task Force commissioned 5 Workgroups to address the IOM recommendations and present their findings at a Methodology Summit.
Each Workgroup was charged with reviewing each IOM standard and supporting elements with respect to the Workgroup topic; reviewing the current ACCF/AHA CPG methodology that addresses the standard; understanding the gaps and potential barriers between the proposed standard and the current method; and proposing recommendations for the Task Force to incorporate and/or to maintain or change existing methodology.
The 5 Workgroups were commissioned in August 2011 and worked by teleconference to draft the preliminary report. Each Workgroup focused on a series of questions to address the specific standards proposed by the IOM. The IOM reports were vetted by using a table (see Appendix 2) that presented each of the proposed 21 standards and 82 supporting elements, the existing applicable ACCF/AHA CPG methodology, and the Workgroup's proposal related to the standard. Each Workgroup report consisted of a brief introduction with an overview of the section content and questions, the complete table, a discussion of the salient issues and challenges that the Workgroup addressed, and preliminary recommendations.
The Workgroups' reports were distributed before the Summit, which was held over 2 days in mid-December 2011. Each report was discussed by the conference attendees and revised by the Workgroups with revisions and final recommendations presented again to the full group. The near-final draft report of each Workgroup was completed by the close of the Summit.
After the Summit, the reports were finalized and then edited for consistency. The completed “ACCF/AHA Clinical Practice Guideline Methodology Summit Report” was reviewed by the Task Force at its semiannual meeting in June 2012. Priorities, resources, implementation, and operational issues were discussed. After this meeting, additional minor edits were made, and the Report was reviewed and accepted by the leadership of both the ACCF and the AHA in September 2012.
Featured Additions to Existing Methodology
Inclusion of Patient Representatives
Fortified by the AHA's focus on the patient and the public it serves and its lay membership, and by the ACCF's current initiative on patient-centered care, the Task Force will invite patient representatives (defined as patients and former patients, patient advocates, or patient/consumer organization representatives) to participate as a member of the Task Force and a GWC. Methods to identify, select, train, and manage RWI and intellectual perspectives of patient representatives are proposed. Patient representatives will participate in topic selection, patient choices, values, preferences, and shared decision making.
Evidence Review Committee
The current CPG creation process will expand to include a separate evidence review committee (ERC), tasked with the SR process, in addition to a GWC, tasked with creating the scientific CPG. When appropriate, other stakeholders (e.g., policy makers and payers) will be invited to participate as members of the ERC. The ERC will be responsible for all phases of the SR process, including the identification, abstraction, and quality assessment of the evidence base.
Systematic Review Using Standardized Protocols
The ACCF/AHA methodology will incorporate a formal SR of the evidence, initially, with a focused approach to a confined topic. Standardized protocols may serve to streamline and enhance the process by developing topic-specific questions that guide the direction of the SR (e.g., search for studies, data extraction, synthesis, and presentation of findings). The use of a PICO(TS) format (mnemonic: population, intervention, comparator, outcomes, timing, and setting) will be used to develop evidence question(s) for the CPG SR.
Intellectual and Practice Perspectives
The current ACCF/AHA CPG process includes a detailed and specific RWI policy and overall mandate to ensure balance (race, ethnicity, sex, and intellectual expertise and experience) among the GWC. In addition, the concept of intellectual and practice perspective (the latter term is operative when an individual's income is enhanced by performing a specific test or procedure relevant to the guideline topic) will be defined, recognized, and managed. Similar to the ACCF/AHA choice of the term RWI rather than COI (conflict of interest) related to industry, the ACCF and AHA have chosen the terms intellectual perspective and clinical practice perspective (rather than bias) to denote intellectual and practice-related opinions and expertise based on evidence and/or experience.
Expanded Review Process
An expanded group of external reviewers will be added to the extensive peer review process for the completed CPG before publication. The current review process includes scientific and clinical content experts as well as partnering and collaborating organizations and other related professional societies. External reviewers will comprise a full spectrum of relevant stakeholders, including public representatives and constituencies such as governmental agencies (e.g., the Agency for Healthcare Research and Quality [AHRQ] and the U.S. Food and Drug Administration [FDA]). Moreover, the SR protocol and completed evidence review will be opened to public comment. However, because of the ability to introduce potential interference and bias that cannot be adequately controlled or managed, the completed CPG will not be subject to public comment before publication.
Workgroup Recommendations (See Full Report for Complete List)
Workgroup 1: Clinical Practice Guidelines We Can Trust
1. At least 1 patient representative should be a full voting member of the Task Force and of a GWC. Patient representatives could include patients, former patients, members of patients' families, caregivers, and laypeople with “health literacy,” including scientists, statisticians, engineers, and science writers.
2. All nonprofessional members of a GWC should avail themselves of the U.S. Cochrane Center and Consumers United for Evidence-Based Medicine's “Understanding Evidence-Based Healthcare: A Foundation for Action” or an equivalent online learning module before accepting their position.
3. Patient representatives' responsibilities should include the formulation of key clinical questions; topic selection; patient choices, values, and preferences; and issues surrounding quality of life. Patient representatives should be encouraged to provide input on the selection of diagnostic tests and treatment modalities.
4. CPGs should be provided to an expanded group of external reviewers before publication. External reviewers would comprise a full spectrum of relevant stakeholders (in addition to the current group of scientific and clinical content experts, as well as to partnering, collaborating, and other relevant professional societies), such as healthcare specialty societies, agencies (e.g., federal government), and representatives of the public.
5. The ACCF/AHA CPG development policy should be expanded to incorporate intellectual and clinical practice perspectives. This information is discussed by the Task Force during GWC formulation, and the expertise of the members is noted in the guideline section on writing committee composition. This review and selection process should be made transparent to the GWC during their orientation meeting/teleconference, and GWC members should be asked to verbally update the group if any changes occur. For specific CPGs, consideration should be given to capturing clinical practice perspectives on the author RWI disclosure table if the GWC member's income is enhanced by performing a specific test or procedure that is relevant to the CPG topic.
Workgroup 2: Standards for Initiating a Systematic Review
1. An ERC composed of content experts, methodologists, statisticians, and other identified stakeholders (policy makers and payers) should participate in the creation of ACCF/AHA CPGs (initially, with a focused approach to a confined topic). The ERC will interact with members of the GWC; however, the responsibilities of the ERC will be separate, distinct, independent, and clearly delineated.
2. In an effort to standardize the process by which ACCF/AHA CPGs are created, the PICO(TS) format should be applied to develop the SR questions. It will be the responsibility of the ERC to create the format of the SR.
3. The current peer review process of ACCF/AHA CPGs should be expanded to include a formal peer assessment of the SR protocol. External stakeholders and patients' representatives should be included in this expanded process, which will serve to enhance the applicability of the document to real-world decision-making policies and clinical scenarios.
Workgroup 3: Standards for Finding and Assessing Individual Studies
1. The system and tool used for critical appraisal of each study, currently under development by the Task Force (ACCF/AHA Evidence Grading Tool), will be used in the pilot SR conducted in conjunction with CPGs. In this case, it is anticipated that the assessment of the quality of the study will be focused within the context of a PICO(TS) question.
2. The pilot system for critical appraisal of each study will be composed of a quantitative assessment, a yes/no assessment of key features of the studies, and a scoring of each of the 3 domains identified by the IOM: risk of bias, relevance, and fidelity of implementation.
3. The quantitative assessment will need some revision for application to observational studies and for focused assessment of the 3 domains identified by the IOM.
4. A scoring of the quality of the study in each of the 3 domains will be qualitatively accomplished by a judgmental scoring by the members of the ERC. The domain, risk of bias, should be renamed “freedom from bias” (or “internal validity”) so that the judgment system is directionally similar for all 3 domains. For each of the 3 domains, which are 1) freedom from bias, 2) relevance, and 3) fidelity of implementation, one of following judgments will be assigned: fatally flawed, low (low freedom from bias, low relevance, or low fidelity), intermediate, or high. In addition, there will be a summary assessment of each study: fatally flawed study, low-quality study, intermediate-quality study, and high-quality study.
5. For other studies being considered in other parts of the CPG, a similar evaluation of the quality of the individual studies may be done. It will be necessary, however, for each proposed recommendation, to clearly identify the population under consideration, intervention, comparator, outcome, and setting, at least in a broad and general way.
Workgroup 4: Standards for Synthesizing the Body of Evidence
Workgroup Recommendations for Assessing and Addressing the Quality of a Body of Evidence
1. Standardize the method of assessment and description of the quality of the body of evidence across studies and perform a qualitative assessment of individual studies and aggregated studies.
2. Depict qualitative assessment across studies addressing the key elements for each PICO(TS) question in a summary table.
3. Where appropriate, provide a risk-of-bias assessment table across studies. This table would be similar to the Cochrane format, which includes items such as random sequence generator, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selection report, or other bias.
4. Generate standardized summary and evidence table templates specific to ACCF/AHA requirements.
5. When available, high-quality SRs from reputable organizations (e.g., the Cochrane Collaboration and AHRQ evidence-based practice centers [EPCs]) should be used. If a new SR is needed, resources will need to be secured to pursue de novo analyses.
6. Notwithstanding these efforts to standardize and improve the qualitative analysis of bodies of evidence, every effort should be made to minimize delays in time for guideline development/revision.
Workgroup Recommendations for Standards for Quantitative Analysis (i.e., Meta-Analysis) Across a Body of Evidence
1. The GWC, in conjunction with the Task Force, should determine when a specific meta-analysis is needed.
2. With statistical consultation, preferred, acceptable methods for meta-analysis (including a Bayesian analysis when appropriate) should be defined in each instance.
3. When available, high-quality meta-analyses from reputable organizations (e.g., the Cochrane Collaboration and AHRQ EPCs) should be used. If a new meta-analysis is needed, resources should be secured to pursue de novo analyses.
4. Notwithstanding efforts to standardize and improve the approach to quantitative analyses across studies, every effort should be made to minimize delays in time for guideline development/revision.
Workgroup Recommendations for ACCF/AHA Grading Methodology and Nomenclature
1. Retain the current basic ACCF/AHA Class of Recommendation (COR) and Level of Evidence (LOE) structure/nomenclature.
2. Standardize how LOE: A, B, C are determined by using a validated ACCF/AHA Evidence Grading Tool (under development) that will incorporate features of existing tools.
3. Change the wording of LOE to Quality of Evidence (QOE) (once the ACCF/AHA Evidence Grading Tool is operational).
4. Add a separate category for QOE: E (expert opinion) and generate specific definitions and examples for QOE: E.
5. Change “Treatment Effect” on the COR/LOE table to “Intervention Effect” and indicate that “Intervention” includes medications, devices, therapeutic strategies, procedures, diagnostic tests, and other.
6. Add adjectives that “map” COR/LOE to the National Heart, Lung, and Blood Institute (NHLBI) (and Grading of Recommendations Assessment, Development and Evaluation [GRADE]) (i.e., I [strong]; IIa [moderate]; IIb [weak]; III [against]).
Workgroup 5: Standards for Reporting Systematic Review
1. An SR included in CPGs should be published as a separate peer-reviewed manuscript(s) when feasible.
2. Recommendations supported by an SR should be identified in CPGs in addition to the appropriate COR and LOE; for example: (Class I; LOE: A)SR.
3. Within the CPG, salient tables and figures from the SR should be included to support recommendations. The remaining pieces of methodology will be hyperlinked to the original SR publication. By doing so, key elements of the SR will be available for public access after publication of the CPGs.
4. CPG review tables should be incorporated into CPGs for non–SR-based recommendations and be available as online supplemental tables.
5. CPG review tables should incorporate most study components that are provided in SR tables.
It is critically important that we continue to monitor the impact of the proposed IOM standards and ACCF/AHA Summit Workgroup recommendations on the overall timeliness and usefulness of our CPGs, in addition to the inherent resource consumption required. Most important will be the ongoing assessment of the changes in methodology in relation to improved patient care and outcomes. Yet to be determined is whether the inclusion of patients' representatives in CPG development and performance of a formal SR of the evidence will enhance the translation of recommendations as anticipated. What is clear, however, is that these changes have the potential to move us into the future as we prepare to incorporate cost and value into our CPG recommendations and to incorporate these recommendations into electronic clinical support systems.
The overarching goal in the evolution of the process and methods used to develop ACCF/AHA CPGs is to combine an ever-expanding evidence base with multidisciplinary expertise and experience while being mindful of patient values so as to improve clinical decision making and thereby improve the quality of care and outcomes for patients with cardiovascular disease. The ACCF and AHA remain steadfast in their commitment to this endeavor.
Alice K. Jacobs, MD, FACC, FAHA
Chair, ACCF/AHA Clinical Practice Guideline Methodology Summit
COR = Class of Recommendation
CPG = clinical practice guideline
EPC = evidence-based practice center
ERC = evidence review committee
GRADE = Grading of Recommendations Assessment, Development and Evaluation
GWC = guideline writing committee
IOM = Institute of Medicine
LOE = Level of Evidence
PICO(TS) format = (mnemonic: population, intervention, comparator, outcomes, timing, and setting)
QOE = Quality of Evidence
RCT = randomized controlled trial
RWI = relationship with industry and other entities
SR = systematic review
The American College of Cardiology Foundation (ACCF) and American Heart Association (AHA) have jointly developed and published clinical practice guidelines (CPGs) since the early 1980s, when, in response to a governmental concern over the potential overutilization of pacemakers, the ACCF and AHA were asked to evaluate the evidence and provide recommendations for practice. In 1984, the “ACCF/AHA Guidelines for Permanent Cardiac Pacemaker Implantation” was published (3). Since then, the guideline effort has continued to expand, and currently, the ACCF/AHA Task Force on Practice Guidelines (Task Force) oversees and directs 17 CPGs, the majority of which are broadly disease based. In 2011, 2 revised, 2 new, and 3 focused updates to the CPGs were published.
The CPG development process and methodology continue to evolve. Over the past few years, several process improvement initiatives resulted in CPGs that have limited text and include evidence and summary tables. The writing process includes a consensus conference where members of multiple guideline writing committees (GWCs) meet to reach consensus and concordance on overlapping recommendations. Methodological enhancements include the development and testing of a scoring tool to consistently assess the quality of randomized controlled trials (RCTs) that inform recommendations and a thorough analysis of Bayesian approaches to evidence synthesis. This continued evolution is in response to the primary goal of providing evidence-based guidelines for healthcare professionals practicing cardiovascular medicine while maintaining relevancy and ease of use at the point of care. The recommendations are articulated in the time-honored and widely recognized ACCF/AHA Class of Recommendation (COR) and Level of Evidence (LOE) scheme. Currently, we are working to incorporate patient preference and shared decision making into the development and translation of our CPG recommendations.
In March 2011, the Institute of Medicine (IOM) published 2 reports, “Clinical Practice Guidelines We Can Trust” (1) and “Finding What Works in Health Care: Standards for Systematic Reviews” (2). There are 8 standards for developing trustworthy CPGs that include establishing transparency, managing conflict of interest, creating multidisciplinary development groups, basing the CPG recommendations on systematic reviews (SRs), establishing evidence foundations for rating the strength of recommendations, articulating recommendations, establishing methods for external review, and updating. It is noteworthy that the ACCF/AHA CPGs were recognized as being compliant with the majority of these standards (Workgroup 1 Comparison Table). However, in view of these detailed reports and our ongoing challenge to respond to the continuous stream of new evidence in a timely manner while maintaining our robust processes for CPG generation and approval, the Task Force decided to hold a Methodology Summit that would focus on the standards for and performance of systematic evidence reviews and on the inclusion of patients and consumers in the CPG development process.
The purpose of this Methodology Summit was to compare and contrast the current ACCF/AHA CPG methodology with the standards proposed by the IOM and consider what, if any, changes or improvements should be implemented to enhance our development process and evidence review and evaluation. In August 2011, the Task Force commissioned 5 Workgroups:
1. Clinical Practice Guidelines We Can Trust
2. Standards for Initiating a Systematic Review
3. Standards for Finding and Assessing Individual Studies
4. Standards for Synthesizing the Body of Evidence
5. Standards for Reporting Systematic Reviews
The invited members of each Workgroup, in addition to guests at the Summit (and including all Task Force members), brought a diversity of experience and expertise to this initiative. Specifically, the membership included those with special interest and experience in CPG development independent of the ACCF/AHA in addition to those with extensive experience in development of ACCF/AHA CPGs. Participants also included methodologists, biostatisticians, clinical and research cardiologists, epidemiologists, and nurses.
Each Workgroup was charged with 1) reviewing the IOM recommendations and sections of the current tools available for developing/conducting SR relevant to its topic; 2) comparing and contrasting the recommendations with current ACCF/AHA methodology, including an analysis/discussion of the gaps and barriers; and 3) drafting recommendations/considerations for changes and improvements to the evidence review process and the COR/LOE, including a discussion as to why changes may or may not be implemented. The report of the Workgroups follows.
Alice K. Jacobs, MD, FACC, FAHA
Immediate Past Chair, ACCF/AHA Task Force on Practice Guidelines
1 Workgroup 1: Clinical Practice Guidelines We Can Trust
(Authors: Dr. Frederick G. Kushner, Chair; Drs. Ralph G. Brindis, Mark A. Creager, Ralph L. Sacco, William A. Zoghbi, and Mr. William H. Roach, Jr.)
The Task Force has recommendations for patient-centered care within its methodology manual (4). GWC are encouraged to consider the role of patient preferences in decisions with substantial personal choice or values and to consider patient-specific modifiers, comorbidities, and issues of patient preference. The IOM report on “Clinical Practice Guidelines We Can Trust” (Section 3.2 and Section 3.3) (1) recommends the involvement of patients (or former patients) and the public (patient advocates, patient/consumer organizations) in the CPG process to formulate clinical questions and review draft CPGs. The IOM report also recommends that strategies for training participants should be adopted by guideline developers. Consumer involvement may provide transparency and a means of establishing relevancy and credibility for the application of evidence-based medicine to patients and other stakeholders and thereby dispel myths and fears through the mutual understanding of issues and values for both patients and providers. Consumers can be powerful allies as supporters of quality initiatives. They can also provide an important perspective from the patient's point of view in areas of uncertainty, where alternative options exist, or where there are substantial gaps in evidence. Finally, consumer involvement can provide valuable perspectives for enhancing shared decision making. Currently, there is a paucity of evidence surrounding the process and impact of integrating consumers into the CPG process. For the purposes of this document, “consumers” will be referred to as patient representatives.
1.1 Workgroup 1 Questions for Consideration
1. How can the Task Force and GWC identify the appropriate “patient representative”? What are the criteria to be met for selection of “patient representatives” who serve? Will they be volunteers or contracted for their work?
As defined by the Cochrane Consumer Network, consumers (“patient representatives” herein) include all users or receivers of health care, including patients, members of the public, caregivers, family members, and members of consumer advocacy groups. Several organizations have identified or are in the process of identifying individuals to serve on GWC. The 4 largest of these are the Cochrane Collaboration through Cochrane Consumer Network and a U.S. subsidiary, Consumers United for Evidence Based Medicine, Agency for Healthcare Research and Quality (AHRQ), Consumers Union, and Guidelines International Network. Consumers Union is currently developing a list of “qualified” patient representatives to serve on evidence-based medical guideline committees under the auspices of a grant from the American Institutes for Research/AHRQ. They solicited interested parties from their subscriber base of 300,000 and received 2,000 expressions of interest. The AHA has a long history of inviting lay volunteers to serve on its boards and committees, in mission and fundraising activities, and in advocacy, and regularly recommends consumer representatives to private and governmental panels and committees. The American Society of Clinical Oncology has included consumers on their GWCs. Patient representatives with defined constituencies can be particularly valuable because of their institutional involvement. On the basis of the comments from Consumers Union and the American Society of Clinical Oncology and discussions among its members, Workgroup 1 recommends the following principles concerning the identification of appropriate patient representative candidates:
A. A job description, including desired and necessary attributes and expectations, should be available to the Task Force and GWC chairs and to organizations that may recommend candidates.
B. The Task Force and GWC should seek nominations for patient representatives from trusted organizations with knowledge of the individual, such as the AHA, ACCF, Consumers Union, Cochrane Collaboration, and AHRQ.
C. Patient representative candidates and/or their sponsoring organizations should submit their resumes, curriculum vitae, or personal statements.
D. Patient representatives should be advised of the time commitment and compensated for travel expenses identical to physician and other Task Force and GWC members.
E. Patient representative candidates, similar to all members, should complete the relationships with industry and other entities (RWI) disclosure.
F. The Task Force and/or GWC chair or designee should interview potential consumer candidates and identify other conflicts where they exist.
G. The development of a questionnaire similar to those used by major charitable organizations to elicit possible bias should be considered.
H. Prospective members should be required to confirm that they will give precedence to ultimate patient health, well-being, quality care, and value in their input, deliberation, and voting.
I. Patient representatives, as is true for other members of the GWC, must be sensitive to their role as impartial members and not permit financial and nonfinancial conflicts, including personal, intellectual, or organizational relationships to influence their judgment.
J. Patient representatives on GWCs should be engaged in the formulation of key clinical questions, topic selection, patient choices, values, preferences, and issues surrounding quality of life. They should be encouraged to provide input on the selection of diagnostic tests and treatment modalities.
K. Patient representatives should be acknowledged as members of the GWC at publication, and a complete summary of their RWI should be provided. They should be subject to the same RWI and recusal rules as other members.
2. Specifically, how is the patient representative to be integrated into the CPG process and/or peer review and/or external stakeholder review? What expectations are there for patient representative involvement?
As a member of the Task Force, patient representatives would assist in directing and overseeing CPG development and establishing policy with a focus on issues such as shared decision making, patient preference, value, translation, and implementation. Patient representatives may be best suited to participate in the initial work of the GWC, during the formulation of key clinical questions, topic selection, and outline development, and, particularly, to comment on the translation and communication of CPGs to the general public. They may not be as helpful or comfortable with SR or assignment of COR and LOE to specific recommendations. Patient representatives with certain technical skills such as science journalists and writers may aid in a patient-provider communication and summary section. Patient representatives should be expected to contribute key questions to be answered by the GWC. They should provide input about patient choices, values, preferences, and issues surrounding quality of life in selecting diagnostic modalities, therapies, medications, and follow-up.
3. What form of training will be required for patient representative participants?
To integrate patient representatives effectively into the CPG creation process, substantial preparatory training is needed before they begin their service on the Task Force or GWC. One study observed that without preparatory training, patient representatives feel that they are “participating observers of technical language to which they could hardly offer input.” Patient representatives contributed infrequently to the discussions, had difficulty with the technical language, only contributed during discussions of patient education, and in general felt that their contributions were not subsequently acted on (5). Involvement of patient representatives requires their understanding of the evidence. Difficulty with medical terminology or other jargon is an important barrier to active or meaningful involvement. Well-informed and experienced patient representatives are more likely to have meaningful exchanges with the GWC than those less informed or less familiar with medical terminology (6). The capacity for active participation in GWCs presupposes foundations of access, knowledge, information, understanding, confidence, agency, engagement, and advocacy (7).
Although resource intensive, it is feasible to train patient representatives to understand the technical elements of CPG development. It is not expected that these members of GWCs understand most of the science related to specific medical issues. They may also have limitations in their ability to understand the details of SR or health economics. Nonetheless, focused instruction in the CPG development process will allow them to fulfill their role. Training is required for understanding elements of recommendation classifications and LOE, including treatment risks and benefits, comparative efficacy, biostatistics, and clinical trial design (i.e., the value of a single center case report or retrospective observational data versus prospective blinded RCT along with meta-analysis).
The Guideline International Network Patient and Public Involvement Working Group has been created to support the development, implementation, and evaluation of guideline-oriented patient and public involvement programs. They have found that patient representative training should cover the fundamentals of CPG development and approaches for reporting back to patient constituencies. Their participants concluded that training and support may facilitate understanding of the technical aspects of CPG development, address financial and organizational barriers to participation, and enhance mutual understanding. Guideline International Network Patient and Public Involvement Working Group collaboration priorities include the development of recruitment methods, training and support strategies, information material and tools, and glossaries of technical terms used in CPGs (8).
The U.S. Cochrane Center and Consumers United for Evidence-Based Medicine have created a web-based course, “Understanding Evidence-based Healthcare: A Foundation for Action,” through a grant from AHRQ (http://us.cochrane.org/understanding-evidence-based-healthcare-foundation-action) (9). This course is divided into 6 modules: INTRO (what is evidence-based health care and why is it important), ASK (importance of research questions in evidence-based health care), ALIGN (research design, bias, and LOE), ACQUIRE (assessing harms and benefits), APPRAISE (understanding healthcare statistics), and APPLY (critical appraisal and making better decisions for evidence-based care; determining causality) (9). The Cochrane Collaboration also has created the Cochrane Consumer Network to engage patient representatives in the development of SR, raise awareness among patient representatives, that is, serve as a clearinghouse of patient representatives for advisory groups, commission plain language summaries, and recruit coauthors for reviews. New ideas to promote patient representative involvement, such as videos, workshops, learning materials, evaluations, and use of social networks have been recently implemented.
4. Should the Task Force support the standard to have CPGs publicly reviewed? How will that affect the final product? How much of a burden will that put on volunteers?
The IOM has recommended that public agencies, patients, and representatives of the public should be external reviewers. The IOM has also recommended that before publication, a draft of the CPG should be made available to the general public for comment and reasonable notice of impending publication should be provided to interested public stakeholders. The IOM believes that for transparency, fairness, completeness, and credibility, these recommendations are reasonable.
Currently, while the ACCF/AHA CPGs undergo an extensive peer review process that includes scientific and clinical content experts in addition to partnering, collaborating, and other relevant professional societies, they are not open to public review and comment. Fundamental to the CPG development process is the ability to develop the CPG without bias from commercial interests. The Workgroup carefully considered the value provided by opening the draft CPG to public opinion. It was recognized that a period of open public comment would introduce a window for potential interference in the process by industry and other external stakeholders that cannot be adequately controlled or managed. However, review of the CPG by public representatives and stakeholders such as governmental agencies, for example, the AHRQ and the U.S. Food and Drug Administration (FDA), as well as scientific and professional experts, is reasonable and recommended.
5. How will conflicts of interest (organizational, intellectual, and practice based) be adjudicated by the GWC and Task Force? What is an “intellectual conflict” or “intellectual bias”?
The ACCF/AHA CPGs are in substantial compliance with 7 of the 8 standards for developing trustworthy CPGs proposed by the IOM, (Workgroup 1 Comparison Table); the CPG-SR intersection is the subject of the remainder of this report. However, although the ACCF and AHA have a rigorous policy for defining, disclosing, and managing RWI (4), the policy is currently undergoing evaluation and updating for potential conflicts related to intellectual bias and practice (or employment) bias.
Intellectual or clinical practice bias (the latter term is operative when an individual‘s income is enhanced by performing a certain test or procedure relevant to the guideline topic) may result in a conflict of interest. An intellectual or clinical practice bias is a predisposition of an individual based on a reaction to a past or current event, treatment, relationship, or other circumstance, or an opinion, belief, or position so strongly held that it might prevent the individual from exercising objective judgment about a matter relevant to the work of the GWC. In the infrequent scenario where, in the opinion of the Task Force or GWC chair, a true bias (with its negative connotation) exists, that individual will not be invited to participate as a member of the GWC. In contrast, and similar to the ACCF/AHA choice of the term RWI rather than COI related to industry, the ACCF and AHA have chosen the terms intellectual perspective and clinical practice perspective to denote intellectual and practice-related opinions and expertise based on evidence and/or experience. Both science and patient representative GWC members with intellectual perspective and clinical practice perspective must be identified and included and their perspective disclosed and managed. It is understood that they have an open-minded approach to evidence and opinion that distinguishes them from individuals with actual intellectual or practice bias.
Identification of Potential Intellectual or Clinical Practice Perspectives of Candidates for Appointment to a GWC
Unlike potential conflicts arising from RWI, determination of the existence of an intellectual or clinical practice perspective may be subjective and less readily apparent to the Task Force. The key to mitigate effectively any conflict is early identification and management of the conflict by the Task Force initially and then by the GWC chair once the work begins. The Task Force chair should be responsible for implementing applicable intellectual and clinical practice perspective procedures. Following are ways to identify and adjudicate the perspectives:
1. Nominations to GWCs should be sought from trusted organizations with a deep knowledge of the individuals nominated. For example, both the ACCF and the AHA, as well as many of their collaborating organizations, have a thorough understanding of CPG development, and each has a long relationship with numerous consumer and scientific volunteers who would have the credentials to serve on a GWC. From that relationship, each organization has knowledge of individual volunteers' life experiences, professional work, and personal and professional views. It has been the practice of the Task Force to maintain a balance of members with and without RWI so that a minimum of 50% of the members have no relevant RWI. In addition, the expertise of the GWC members is reviewed carefully to ensure that there is a balance of perspectives. The ACCF and AHA could request that other organizations meet these same requirements. In addition, it is the practice of the ACCF and AHA to provide additional scrutiny of nominations by officers and senior staff before the Task Force review. Seeking nominations from similar organizations would reduce the likelihood of nominees with unknown perspectives.
2. Any candidate for GWC membership may be requested to submit a detailed curriculum vitae, which the Task Force and the GWC chair and staff can review for possible intellectual or clinical practice perspectives (or bias).
3. The GWC chair, at his or her discretion, should be empowered to interview proposed GWC members who are not well known or who have possible or unknown intellectual or clinical practice perspectives.
4. Candidates for GWC membership may be asked to complete a questionnaire through which they disclose circumstances that might give rise to a potential or actual perspective or bias related to the GWC assignment. The questionnaire could include inquiries found in questionnaires typically used by charitable health organizations with respect to director and officer perspectives or bias. As some individuals may have difficulty recognizing or disclosing their own intellectual or practice perspectives that may lead to a conflict, questionnaire inquiries should be broadly worded to obtain information from which the GWC chair, Task Force, or staff could identify actual or potential intellectual or practice perspectives.
5. At each meeting of the GWC, remind the members of the policy whereby the group was formulated to incorporate a balance of RWI and varying intellectual and clinical practice perspectives that we continue to manage throughout the guideline writing effort.
6. Prospective GWC members should be educated about being transparent with respect to their individual position and be required to confirm that they will give priority to ultimate patient health and well-being, quality care, and value in their input and deliberation and when voting on issues before the GWC.
Inclusion of Essential GWC Members With Known Intellectual and Clinical Practice Perspectives
To develop the most effective CPGs, a GWC may require the participation of an individual who is outspoken in support of or against a particular procedure, medication, or other matter relevant to the GWC's work in the absence of supportive data and therefore has a known intellectual or clinical practice perspective. In such cases, the Task Force should make certain that the GWC chair is informed of the individual's perspective and that the GWC membership is balanced in viewpoint. All GWCs should be composed of members determined to be able to apply fair judgment with respect to all issues and particularly those in contention, so that they will be able to make objective assessments of all the information relevant to the GWC task. In addition, the GWC chair should make certain that the individual follows the policy and procedures noted in the ACCF/AHA Methodology Manual and Policies (4) as it applies to recusal from discussion at the request of the chair.
Importance of Intellectual, in Comparison to Financial Conflict and RWI, to External Organizations and Government Regulators
Although conflicts arising from RWI have received most of the attention by external groups, the Task Force should effectively address all potential conflicts, whether arising from RWI or intellectual or clinical practice perspective. The Task Force should be prepared to demonstrate to external groups that its policies for managing any conflict are practical and effective.
Maintaining a Majority of GWC Members Without RWI and a Balance of GWC Members With Intellectual and Clinical Practice Perspectives Throughout the Duration of CPG Development
All GWC members should be assessed by the Task Force and GWC chair for potential relevant RWI at the outset of the process. A majority of members who have no relevant RWI must be assigned. They will be asked to commit to developing no new RWI or other relationships that may represent potential intellectual perspectives (e.g., serve as principal investigator of a new study relevant to the CPG topic) during the course of their service on the GWC. As RWI or intellectual perspectives may arise in connection with any one of the many recommendations a GWC may make in developing a CPG, the GWC chair and staff must be vigilant in identifying any relationships as they arise and adjudicating them in accordance with established policies.
1.2 Workgroup 1 Recommendations
1. At least 1 patient representative should be a full voting member of the Task Force and each GWC. Patient representatives could include patients, former patients, members of patients' families, caregivers, and laypeople with “health literacy,” including scientists, statisticians, engineers, and science writers.
2. All nonprofessional members of GWCs should avail themselves of the U.S. Cochrane Center and Consumers United for Evidence-Based Medicine's “Understanding Evidence-Based Healthcare: A Foundation for Action” (9) or an equivalent online learning module before accepting their position.
3. Patient representatives' responsibilities should include the formulation of key clinical questions, topic selection, patient choices, values, and preferences, and issues surrounding quality of life. They should be encouraged to provide input about the selection of diagnostic tests and treatment modalities.
4. CPGs should be provided to an expanded group of external reviewers before publication. External reviewers would comprise a full spectrum of relevant stakeholders (in addition to the current group of scientific and clinical content experts, as well as to partnering, collaborating, and other relevant professional societies), such as healthcare specialty societies, agencies (e.g., federal government), and representatives of the public.
5. The ACCF/AHA CPG development policy should be expanded to incorporate intellectual and clinical practice perspectives. This information is discussed by the Task Force during GWC formulation and the expertise of the members is noted in the guideline section on writing committee composition. This review and selection process will be made transparent to the GWC during their orientation meeting/teleconference, and GWC members will be asked to verbally update the group if any changes occur. For specific CPGs, consideration will be given to capturing clinical practice perspectives on the author RWI disclosure table if the GWC member's income is enhanced by performing a specific test or procedure that is relevant to the guideline topic.
2 Workgroup 2: Standards for Initiating a Systematic Review
(Authors: Dr. Steven M. Ettinger, Chair; Drs. Donna K. Arnett, Gregg C. Fonarow, Judith S. Hochman, Sharon-Lise T. Normand, and Gordon F. Tomaselli)
SRs and the resulting evidence-based CPGs serve as resources for healthcare decision-making policies. The ideal guideline provides comprehensive protocols and plans that are based on a thorough and extensive understanding of the medical literature, are scientifically valid, and are void of clinical bias. The IOM defines 8 standards (Workgroup 2 Comparison Table), which guide the creation of a focused SR policy, that are essential for the creation of a scientifically valid CPG.
Workgroup 2 was charged with the task of reviewing current ACCF/AHA standards for an SR and identifying those elements that diverged from the recommendations outlined by the IOM. Where applicable, the Workgroup proposed alternative strategies relating to SR in an effort to enhance current ACCF/AHA policies. The Workgroup acknowledged that one of the challenges posed by modifications of existing ACCF/AHA evidence review policies is related to the rapid expansion of medical literature seen with publication of late-breaking clinical trials and RCTs. The Workgroup expressed concern that any modification in current ACCF/AHA evidence review policy would result in significant delays in the creation of evidence-based CPGs and the loss of clinical relevance of the document. Recognizing that constraints relating to staffing and financial issues would also affect the SR process, the Workgroup proposed 3 modifications of the current ACCF/AHA evidence review policy. These modifications were developed based on a review of the following key questions:
2.1 Workgroup 2 Questions for Consideration
1. Should other stakeholders/patient representatives be involved in the development of the SR protocol? If yes, to what level and what types of stakeholders/consumers should be included (e.g., payers, policy makers, patients, caregivers, family, and advocacy groups)? What type of training would this group need to be able to provide value to the current process? Would the ACCF/AHA provide the training or outsource it?
In an effort to expand the current ACCF/AHA process related to the creation of CPGs (initially, with a focused approach to a confined topic), 2 separate teams would be created: an evidence review committee (ERC), tasked with the SR process, and a GWC, tasked with creating the scientific CPG. When appropriate, other stakeholders (e.g., policy makers and payers) would be invited to participate as members of the ERC. The ERC will be responsible for all phases of the SR process. By expanding the professional and clinical expertise of the ERC, the “trustworthiness” of the final document as well as its applicability to real-world decision making may be enhanced (2). The Workgroup acknowledges the added benefit of reviewing and incorporating unique data sets provided by the “new” stakeholders that are potentially outside the scope of current published scientific literature. Although “unpublished data may not be used to support a recommendation” (4), information collected by policy makers and payers (e.g., economic and health plan data) may provide additional insight into the effectiveness of healthcare recommendations as it relates to various subpopulations (e.g., racial and ethnic minorities, women, and the aged). Stakeholders invited to participate in the ERC would undergo formal training (by ACCF/AHA staff) related to understanding current CPG writing policies of the ACCF/AHA. In addition, SRs require a rigorous scientific analysis of case reports, case series, cohort studies, clinical trials, RCTs, and blinded RCTs. An understanding of various point statistics (e.g., absolute risk difference, number needed to treat, number needed to harm) and relative treatment effects (e.g., odds ratio, relative risk, hazard ratio, and incidence rate ratio) are essential components of the process. Individuals invited to serve on the ERC may require additional training related to methodology and statistical analysis (or possess this expertise) with the expectation that this would be made available by the ACCF/AHA.
2. Should we redefine how RWI and intellectual and clinical practice perspectives are presently managed for those involved in the development of the SR as it relates to intellectual, professional, and personal association? Would this make the process better?
The ACCF/AHA has an official RWI policy (4). The definition of relevant is published in the first Appendix of each CPG along with the authors' relevant RWI. The ACCF/AHA does not yet have an official policy regarding intellectual and clinical practice perspectives (see Workgroup 1), although a balance of perspectives and opinions is considered by the Task Force when selecting a GWC, each author's institution and title are provided in the document, and a general statement about the expertise of the members are noted under writing committee composition.
The following assumes that GWC structure will not change, but when appropriate, a separate ERC will be impaneled to draft and formalize a SR and include methodologists and clinical content experts. Although the GWC and ERC are separate committees, some members may be selected to serve as liaison to both. As such, the ERC will have the responsibility of developing the SR for use by the GWC in generating the scientific clinical document. The ERC will then review the developed CPG to confirm that the recommendations are consistent with the findings of the clinical trials and studies.
In general, we agree and are compliant with the IOM recommendations on many aspects of RWI and intellectual balance when constituting an SR team. The principle that disclosure is necessary but may not be sufficient is important. We also agree that individuals should be excluded if their participation would diminish the public perception of the independence and integrity of the newly created CPG.
The discussion around disclosure of conflict of interest/RWI in the IOM report (Section 2.2.1) (2) included the question of the structure of the ERC. We recommend that the ERC include members completely free of RWI and balanced with respect to intellectual and clinical practice perspectives (as defined by Workgroup 1) but that the GWCs adhere to the current ACCF/AHA policy of the chairs and >50% of the committee being free of relevant RWI. It is the consensus of the Workgroup that absence of experts (perhaps with relevant RWI) on the GWC would undermine the credibility of the review and CPG as much or more than the presence of committee members with RWI.
As stated by Workgroup 1, individuals thought to have true bias would not be eligible to serve on a GWC. The major issue pertains to the definition of intellectual and clinical practice perspective and how this standard would be adjudicated. It is unclear whether any method or protocol for assessing intellectual and professional perspective can be standardized and codified to allow for use across all document-generating initiatives. We attempted to address the various metrics that could be used to define intellectual or professional perspective considering a number of possibilities, including the percentage of one's income or whether income is enhanced if derived from a procedure, test, or topic relevant to the CPG topic (this could also be considered under financial perspective); the publication and speaking record of the candidate committee member on the topic; and whether or not the home institution or practice would benefit in a way that might promote a particular perspective, among others (see Workgroup 1). However, regardless of the metric(s) used, we recommend consideration of expanding disclosure to include relevant professional/intellectual or clinical practice perspectives in the published document for selected guidelines.
We recognize that some of the changes outlined would require an increase in the size and broadening of the expertise of the groups constituted to develop CPGs. Perhaps more importantly, the timeline for the entire process most certainly will be lengthened. In this context, the IOM report commented that an SR may take 1 year or more to produce, so RWI and intellectual and practice perspectives should be updated at regular intervals, but there is no guidance as to what those intervals should be. Currently, the ACCF/AHA process includes the provision that GWC members avoid accepting new relevant RWI throughout the duration of the development process.
3. Should the SR topics be formulated using the PICO(TS) questions, AHRQ, or an alternative source? Should we consider using a governmental agency, and how would this affect the process (cost, time, and other considerations)? How would we train physicians and/or stakeholders/patient representatives?
In an effort to confirm the need for a focused update (or new or revised CPG), summaries of late-breaking clinical trials presented at major scientific conferences are compiled. The goal is to identify evidence that is important to patient-care issues (including but not limited to benefits relating to quality of life, morbidity and mortality, and economic outcomes). Members of the GWC and Task Force are balloted as to whether the findings of the new study may change a current recommendation. The Task Force reviews the results and a decision is made as to whether a focused update (or new or revised guideline) GWC should be convened.
Once the decision is reached to create a new, revised, or updated CPG, every effort is made to ensure the development of an objective, transparent, and scientifically valid document. The ACCF and AHA currently use a detailed standard format to generate an evidence review for the creation of CPGs (4). The ACCF, AHA, and IOM acknowledge that creation of a CPG and formulation of specific SR questions relating to the topic are challenging processes. Standardized protocols may serve to streamline and enhance the process by developing topic-specific questions that guide the direction of the SR (e.g., search for studies, data extraction, synthesis, and presentation of findings ). The IOM supports an SR that focuses on the development of questions that deal with the “uncertainties that underlie disagreement in practice and the outcomes and interventions that are of interest to patients and clinicians” (2). A CPG with newly developed recommendations created in such a manner may find improved real-world application and expand its value to healthcare decision makers. The use of a PICO(TS) format (2) is recommended by the Workgroup to develop evidence question(s) for the SR of a CPG (initially, with a focused approach to a confined topic). This effort will require additional resources for the education and training of committee members. Collaboration with other guideline developers and government organizations should be explored.
4. Should we alter the current process of peer review as it relates to the assessment of the SR protocol—electronic registration, ACCF/AHA Web sites—or continue the current process of an internal review by the GWC?
As stated previously, SR and resulting evidence-based CPGs serve as a resource for healthcare decision-making policies. The IOM suggests that a formal review process be created to publicly vet the SR as early as possible in the process. The benefits of this recommendation include minimizing bias, providing an opportunity to identify ongoing SR by other groups, thereby avoiding unnecessary (duplicate) efforts, and encouraging collaboration among various organizations (2). Given these potential benefits, the Workgroup acknowledges that a formal peer review process of the SR may enhance the clinical value of the final document. It is essential that these SRs be rigorous, credible, and, to the extent possible, free of commercial, professional, and intellectual bias. Unlike the ACCF/AHA, the IOM does not distinguish between intellectual bias and perspective. Rather, the IOM defines certain standards, which are designed to minimize bias, that guide the creation of a focused SR policy. These include a defined protocol that ensures balanced and unbiased user and stakeholder input to the SR before the initiation of the process and a well-defined policy to manage bias and RWI for all individuals involved in the SR.
Having a mechanism for various stakeholders, including the public, payers, and governmental agencies, to be able to provide input on which topics would benefit from an SR, how the review is designed and conducted, and potential data sources is important. The Workgroup supports the recommendation that the SR questions be vetted by agencies in the public realm (e.g., FDA, Centers for Medicare and Medicaid Services, health plans such as Kaiser Permanente, and United Healthcare). ERC and GWC members would be responsible for determining the scientific validity of the comments, if there would be a need to modify the PICO(TS) question(s), and whether additional data/studies should be included in the SR. Concerns relating to the number of comments and potential bias of the public reviewers may be problematic, and therefore no direct response would be anticipated or required by the ERC and GWC. With the creation of an ERC and the inclusion of new stakeholders (e.g., policy makers and payers), the formal peer review and SR process would be enhanced. It is recommended that members of the ERC be responsible and make final decisions about the design, analysis, and reporting of the identified and selected data.
2.2 Workgroup 2 Recommendations
1. An ERC composed of content experts, methodologists, statisticians, and other identified stakeholders (policy makers and payers) should participate in the creation of ACCF/AHA CPGs (initially, with a focused approach to a confined topic). The ERC should interact with the members of the GWC; however, the ERC's responsibilities will be separate, distinct, independent, and clearly delineated.
2. In an effort to standardize the process by which ACCF/AHA CPGs are created, the PICO(TS) format should be applied to develop the SR questions. It will be the responsibility of the ERC to create the format of the SR.
3. The current peer review process of ACCF/AHA CPGs should be expanded to include a formal peer assessment of the SR protocol. External stakeholders and patient representatives should be included in this expanded process that will serve to enhance the applicability of the document to real-world decision-making policies and clinical scenarios.
3 Workgroup 3: Standards for Finding and Assessing Individual Studies
(Authors: Dr. Robert A. Guyton, Chair; Drs. Jonathan L. Halperin, Sidney C. Smith, Jr., and Marnie Bertolet, and Ms. Marguerite A. Koster)
The 2 publications from the IOM released in 2011, “Clinical Practice Guidelines We Can Trust” (1) and “Standards for Systematic Reviews” (2), present a challenge to developers of CPGs. A compelling argument is made that CPGs are currently not at the level that we seek as we allocate scarce societal resources to fund an expanding repertoire of sophisticated preventive measures and therapies for human illness. In particular, the authors of “Clinical Practice Guidelines We Can Trust” stated that quality healthcare practice is guided by the combination of critically appraised and synthesized scientific evidence, clinical experiential knowledge and skill, and patient values and preferences (1). In this context, a new definition of CPGs has been created: CPGs are statements that include recommendations intended to optimize patient care that are informed by an SR of evidence and an assessment of the benefits and harms of alternative care options (1).
Relevant to the issue of finding and assessing individual studies, Standard 4.1 states that developers of CPGs should use SRs that meet the standards set by the IOM Committee on Standards for Systematic Reviews of Comparative Effectiveness Research (2). In addition, Standard 5.1 states explicitly that a summary of available evidence (and evidentiary gaps) and a description of the quality of evidence (including applicability), quantity (including completeness), and consistency of the aggregate available evidence should be a part of the CPG process, including a rating of the level of confidence in the evidence supporting the recommendation (1).
The charge to this Workgroup is also highlighted in the criteria for inclusion of CPGs in the National Guideline Clearinghouse. These criteria state that corroborating documentation must be available to prove that a systematic literature search of existing scientific evidence from peer-reviewed journals was performed during the CPG development process. It is notable that CPGs are not excluded from the National Guideline Clearinghouse if gaps in scientific evidence for some recommendations are present and there is documentation that a search has been made for this evidence and the gaps are specified in detail. The IOM report also states that (guidelines) that have not included a thorough SR of the relevant scientific evidence should be excluded from the National Guideline Clearinghouse (1).
Challenges in Finding and Assessing Individual Studies
The area of evidence appraisal is described in the IOM report as “besieged with problems.” However, there is a consensus that a standardized rating of evidence quality facilitates the application of individual studies to the risk-benefit analysis that is the foundation of a clinical recommendation.
There are major concerns about bias and relevance of published studies. A 2005 study of trials published in the New England Journal of Medicine, the Journal of the American Medical Association, and The Lancet found that approximately three fourths of the trials were funded by industry (1). Another study revealed that high-quality commercial trials in this study were found to be 5.3 times more likely to endorse the product of the funding industry than noncommercially funded studies of the same product (1). This is certainly understandable, as many of these trials constitute preapproval research conducted for approval or labeling of either devices or medications. The trials have been designed to prove that the drug or device is safe and effective. These trials often tend to utilize young healthy patients who differ from the actual target population of the drug or device. The trials may intentionally use as a comparator a placebo rather than a competitor drug, and often surrogate endpoints are used to allow a shorter period of time for follow-up.
Even when studies have minimal bias (i.e., they have high internal validity), they may not be relevant for the patient population being considered by the GWC. This is referred to as relevance, generalizability, or external validity. RCTs often have an underrepresentation of older patients, of patients with comorbidities, of ethnic minorities, and of low-income patients. The IOM report quoted a 2007 study of evidence quality in cardiovascular risk management and found that only 28% of 369 recommendations were based on high-quality evidence. The most common reason for downgrading the quality of evidence in RCTs was concern about generalizability, that is, extrapolating from the carefully selected trial enrollees to the general population in question (1).
The IOM report “Standards for Systematic Reviews” offered a set of standards for conducting the SR and elements of performance within these standards (2). In the area of finding and assessing individual studies, 6 standards were offered, composed of 30 elements that are addressed specifically in the Workgroup 3 Comparison Table. The Workgroup quickly recognized a number of challenges. First, ACCF/AHA CPGs have been focused on clinical topics rather than specific clinical questions. The standards for SR in general expect the SR to be focused around a limited number of clinical questions that may be optimally organized around the PICO(TS) format.
The second challenge the Workgroup faced was the labor-intensive and resource-intensive nature of SR. In the past, ACCF/AHA CPGs have used literature databases with search engines and relevant topics, key words, or medical subject headings. This has been augmented by the extensive knowledge of the literature derived from the experience of the GWC. The primary potential problem with the current system is the possibility of bias, in particular, the perpetuation of bias from year to year and from guideline to guideline.
The third challenge discussed by the Workgroup was the difficulty in transitioning from the current CPG process to a process focused around SR of relevant clinical questions. An effort was made to find an iterative transition from the current process to a process more fully and carefully informed by SR with implementation of multiple elements of performance from the IOM report.
3.1 Workgroup 3 Recommendations
IOM Standard 3.1: Conduct a comprehensive, systematic search for evidence
A formal SR of the evidence for the entire scope of topics in an ACCF/AHA CPG may not be feasible given existing resources. The Workgroup believes that a more formal review of the literature for ACCF/AHA CPGs, incorporating many of the performance elements proposed by the IOM committee, is an appropriate and necessary step for our guidelines to remain a trustworthy source for recommendations in cardiovascular clinical practice. If a comprehensive SR is not feasible, the processes and procedures used to review the evidence need to be predetermined and documented by an ERC that works with the GWC but is functionally separate.
The Workgroup believes that all evidence reviews should be conducted by staff research specialists who have had training in developing comprehensive search strategies and conducting searches.
The Workgroup believes that the ACCF/AHA CPG process needs to move in iterative manner toward a more robust SR of the literature. Acknowledging that a complete formal SR for all CPG topics is not possible at this time, given existing resources, the Workgroup recommends that 2 or 3 key clinical questions within selected upcoming CPGs be used to pilot an SR process. These key clinical questions could be part of a focused update or a focused approach to a confined topic. It should be noted, however, that if a focused update is to be the pilot of an SR, then the appropriate PICO(TS) question would need to be formulated for the focused update, and all relevant evidence, not just new evidence, would need to be reviewed. A “focused” update does not imply a limited SR. It does imply that the update question might be adequately framed in a single PICO(TS) question for efficient SR.
IOM Standard 3.2: Take action to address potentially biased reporting of research results
Gray literature databases should be searched and relevant findings included as appropriate as evidence in the SR. Gray literature sources would include clinical trial registries (e.g., http://www.clinicaltrials.gov), clinical databases such as the National Cardiovascular Data Registry or the Society of Thoracic Surgery database, industry reports to the FDA, and unpublished abstracts <2 years old. Lack of complete information and lack of peer review are issues that need to be considered with the gray literature. The gray literature should be searched especially for signals of unreported safety issues.
Communication with researchers about information to assess the risk of bias in a study is appropriate, as is communication about unpublished data that may supplement the published data of a study. Again, the absence of peer review is a particularly prominent issue, as is the potential perpetuation and exaggeration of the bias of the sponsor and/or researcher.
IOM Standard 3.3: Screen and select studies
For each pilot SR, criteria should be prespecified for inclusion or exclusion of studies. The use of clinical questions in a PICO(TS) format would greatly facilitate this process. Observational studies in addition to RCTs should be included. A separate ERC should conduct an SR of the literature. Members of the ERC should be without RWI relevant to the topic of the SR. The Workgroup recommends training a cadre of volunteer members to conduct the SR, including screening and selecting studies, abstracting data from the studies, and assessing the quality of the studies. ACCF/AHA members early in their academic careers may find benefit in becoming involved in this process with no financial remuneration. Academic credit might include letters to department chairs, listing as part of a group that conducts SR, and listing of ERC members as coauthors of the CPG along with the GWC. Individuals who screen and select studies would need to work from written procedures and forms. They would need training and verification of accuracy and consistency. The ACCF/AHA research staff should perform the literature searches. The titles and abstracts of all articles should be reviewed with dual screening by 2 ERC members, with a third member adjudicating conflicts. In addition to RCTs, observational studies should be included to address gaps in evidence from RCTs. Staff and screeners should also look for observational studies and unpublished studies seeking benefits and potential harm that would otherwise be missed if only RCTs were reviewed.
IOM Standard 3.4: Document research
The current ACCF/AHA CPG methodology is compliant with this IOM standard with the exception of documentation of the reason for exclusion of peer-reviewed RCTs and large observational trials published in major journals. This exclusion judgment should be documented and justified for each such study.
IOM Standard 3.5: Manage data collection
For each pilot SR, a standardized data extraction form should be developed that includes some standard elements and additional elements unique to the PICO(TS) question under consideration. The data extraction form should be piloted in a few studies and then revised. For the broader clinical questions in the more expansive CPGs (the portions of the guideline not subject to a formal SR), a similar data extraction form should be developed.
IOM Standard 3.6: Critically appraise each study
Documentation of the assessment of 1) the risk of bias, 2) the relevance of the study's populations, interventions, and outcomes measures, and 3) the fidelity of implementation of interventions is essential. The system used to assess the studies in these 3 domains is controversial. Multiple scoring tools exist, including the one under evaluation by the ACCF/AHA.
The Cochrane Risk of Bias tool uses a qualitative, domain-based evaluation and judges each study across 6 domains of bias: selection, performance, detection, attrition, reporting, and “other.” In each domain, bias is judged as low risk of bias, high risk of bias, or unclear risk of bias, with brief supporting comments for each judgment. There also is a summary judgment for each study summarizing the overall risk of bias incorporating information from all 6 domains. It is recommended that the summary judgment be outcome specific (10). The Cochrane tool is a nonquantitative tool of the type thought by many (including members of the Workgroup) to be preferable to quantitative tools or checklists.
For each study, the same 3 members of the ERC who performed data extraction could simultaneously perform a quality assessment (2 members perform a quality assessment with the third member to adjudicate differences). Whether a scoring tool is used or a nonquantitative tool such as the Cochrane tool is used, a separate tool is needed for observational studies and registries that is distinct from the tool used for RCTs.
Proposal for the Critical Assessment of Each Study
The critical appraisal of each study has 3 required elements of performance:
1. Systematic assessment of the risk of bias, using predefined criteria
2. Assessment of the relevance of the study's populations, interventions, and outcomes measures
3. Assessment of the fidelity of the implementation of the interventions
The Workgroup proposes the following system for critical appraisal of each study:
1. This system will be used in the pilot SR, conducted in conjunction with CPGs. In this case, it is anticipated that the assessment of the quality of the study will be focused within the context of a PICO(TS) question.
2. It is expected that a 3-member team from the ERC associated with the GWC will conduct the critical assessment of each study. For efficiency, it is suggested that these 3 people be the same 3 people who conduct the data extraction from each selected study.
3. The pilot system for critical appraisal of each study will be composed of a quantitative assessment, a yes/no assessment of key features of the studies, and a scoring of each of the 3 domains identified by the IOM: risk of bias, relevance, and fidelity of implementation.
4. A quantitative assessment is currently undergoing evaluation and validation by the Task Force. This quantitative assessment will need some revision for application to observational studies and for focused assessment of the 3 domains identified by the IOM.
5. The qualitative assessment of the key features of each study may be the yes/no answers on the quantitative assessment.
6. A scoring of the quality of the study in each of the 3 domains will be qualitatively accomplished judgmental scoring by the 3 members of the ERC. The domain “risk of bias” should be renamed “freedom from bias” (or “internal validity”) so that the judgment system is directionally similar for all 3 domains. Otherwise, a high-quality assessment of the “risk of bias” domain would be “low,” while such an assessment of relevance or fidelity would be “high.” By renaming “risk of bias” as “freedom from bias,” all 3 domains can have similar scoring of high to low, indicating high to low quality in that domain. For each of the 3 domains—1) freedom from bias, 2) relevance, and 3) fidelity of implementation—one of the following judgments will be assigned: fatally flawed, low (low freedom from bias, low relevance, or low fidelity), intermediate, or high. In addition, there will be a summary assessment of each study: fatally flawed study, low-quality study, intermediate-quality study, and high-quality study.
7. Although the freedom from bias of the study is somewhat independent of the PICO(TS) question being addressed, the other 2 domains, relevance and fidelity of implementation, are necessarily dependent on the population, intervention, comparator, outcome, and setting under consideration. Therefore, the assessment of the quality of the study must be made in the context of the PICO(TS) question under consideration.
8. For other studies being considered in other parts of the CPG, a similar evaluation of the quality of the individual studies may be carried out. It will be necessary, however, to clearly identify, for each proposed recommendation, the population under consideration, intervention, comparator, outcome and setting, at least in a broad and general way.
9. With this system, each study, when considered as part of the body of evidence for assessment of the LOE for a specific recommendation, will have an overall quality assessment. Each study will also have a qualitative assessment of quality in each of the 3 domains: freedom from bias, relevance, and fidelity of implementation. The scoring tool, if a quantitative tool is used, and the yes/no answers on key elements of the study will be used to inform the judgment on the 3 domains and the summary judgment of the quality of the study.
10. It is suggested that the proposed system might combine the advantages of the quantitative tool, the qualitative yes/no assessment of key points in the study, and the need for judgment, which is study specific and context specific in the assessment of quality in each of the 3 domains and the summary study quality assessment.
4 Workgroup 4: Standards for Synthesizing the Body of Evidence
(Authors: Dr. Jeffrey L. Anderson, Chair; Drs. Nancy M. Albert, Elliott M. Antman, David L. DeMets, Eduardo Ortiz, Eric D. Peterson, and Clyde W. Yancy, and Ms. Marguerite A. Koster)
Workgroup 4 was assigned 2 overarching tasks: 1) to address setting standards for synthesizing a body of evidence, and 2) to review and make recommendations on grading of guidelines, that is, on the ACCF/AHA system of establishing COR/LOE.
The first task was subdivided into 4 sections, following the IOM's report on “Standards for Systematic Reviews.” These 4 sections comprised the following standards: 1) assessing the quality of a body of evidence, 2) describing the quality of a body of evidence, 3) qualitative synthesis of a body of evidence, and 4) quantitative assessment of a body of evidence (e.g., meta-analysis of the universe of studies to address a specific research question) and how to determine the need for qualitative and/or quantitative evaluations. It was suggested that the formulation of specific research questions be guided by the mnemonic PICO(TS).
Workgroup 4's scope of work and charge included the important task of comparing and contrasting the IOM report with ACCF/AHA CPG methodology for synthesizing the body of evidence, which is summarized in the Workgroup 4 Comparison Table. Where differences or deficiencies in current ACCF/AHA CPG processes were identified, the Workgroup was to propose recommendations for bringing these into compliance with the IOM recommendations or to defend the current processes where the Workgroup thought a change was not warranted.
Three key areas of discussion emerged with respect to the differences between the IOM recommendations and the ACCF/AHA processes: 1) what was the strength of evidence backing the advantage to patient care (superiority) of the IOM proposal? 2) what resources/cost implications would be associated with the proposed change (an overlapping concern)? and 3) how disruptive would a major change be to current ACCF/AHA programs (caused by a domino effect throughout the 2 organizations and the cardiovascular universe) from the current CPG COR/LOE grading system?
Overall, the Workgroup thought that the current ACCF/AHA CPG methodology has served the cardiovascular community well, as evidenced by improved outcomes associated with adherence to CPG recommendations (11,12). Thus, the Workgroup endorsed a plan of incremental change occurring over time, that is, “evolution rather than revolution,” assessing the impact and value of change at each step along the way.
4.1 Workgroup 4 Questions for Consideration
1. Qualitative synthesis of a body of evidence: assessing and describing the quality of a body of evidence. Qualitative synthesis deals with analyzing the essential characteristics of a study, that is, risk of bias, consistency, precision, and directness; and, for observational studies, additionally dose-response association, confounding factors that would change the observed effect, and strength of association. Questions posed to the Workgroup included the following:
• How does the current ACCF/AHA methodology look at the qualitative characteristics of a study and then look at the main variables across a body of evidence?
• How do other evidence grading methodologies (i.e., Grading of Recommendations Assessment, Development and Evaluation [GRADE], NHLBI) address the synthesis and analysis of characteristics across a body of evidence? What is required for such an analysis, for example, data summary tables?
• How does the qualitative synthesis/analysis change when some of the studies are observational?
• What are some key qualitative characteristics that are important for RCTs as well as observational studies?
• What is the optimal and most practical/efficient way to synthesize evidence looking across all studies and to focus on trends, themes, relevance, and observations?
Because Workgroup 3 addressed assessment of quality at the individual study level, Workgroup 4's recommendations apply to assessment of evidence quality across the body of evidence.
The term qualitative synthesis refers to evaluating and summarizing the findings of a body of evidence for each prespecified outcome of interest in a specific clinical question. The IOM further elaborates on the definition of qualitative synthesis as
… an assessment of the body of evidence that goes beyond factual descriptions or tables that, for example, simply detail how many studies were assessed, the reasons for excluding other studies, the range of study sizes and treatments compared, or quality scores of each study as measured by a risk of bias tool. While an accurate description of the body of evidence is essential, it is not sufficient (2, p. 196).
The IOM report recommends 4 elements of performance for conducting a qualitative synthesis of a body of evidence for an outcome of interest:
1. Describe the clinical and methodological characteristics of the included studies, such as their size, inclusion or exclusion of important subgroups, timeliness, and other relevant factors.
2. Describe the strengths and limitations of individual studies and patterns across studies in plain terms, how flaws in the design or execution of the study (or groups of studies) could bias the results, explaining the reasoning behind these judgments.
3. Describe the relationships between the characteristics of the individual studies and their reported findings and patterns across studies.
4. Discuss the relevance of individual studies to the populations, comparisons, cointerventions, settings, and outcomes or measures of interest.
The IOM report described 8 “basic characteristics of quality” that are essential for assessing the quality of a body of evidence for a particular outcome of interest, and a discussion of these characteristics is an important component of the qualitative synthesis. Characteristics are risk of bias (limitations in study design and conduct); consistency of results across studies; precision of the estimates of effect; directness of the evidence in terms of study design, populations, interventions, comparisons, or outcomes; and reporting bias. For observational studies, additional characteristics include dose-response association, plausible confounding factors that would change an observed effect, and strength of association (i.e., large magnitude of effect). Initially developed by the GRADE Working Group (13), these characteristics for assessment of a body of evidence have been endorsed by the Cochrane Collaboration, the AHRQ Effective Health Care Program, and many medical professional societies and healthcare organizations involved in evidence review efforts, especially for development of CPGs.
The GRADE system (13) designates 4 levels for assessing the overall quality of a body of evidence: “high,” “moderate,” “low,” and “very low.” These levels refer to the degree of confidence in the estimate of effect. A body of evidence consisting of RCTs begins at the highest level and can be downgraded on a point system based on methodological limitations of the studies (risk of bias), indirectness of evidence, inconsistency of reported results, imprecision of the effect estimates, and/or publication bias. A body of evidence consisting of observational studies begins at a “low” level and can be upgraded based on a large magnitude of effect, a dose-response association, or few confounding factors that would change the reported effect. The GRADE Working Group has developed a software program, GRADEprofiler (“GRADEpro”), for conducting the quality assessment of a body of evidence (14). The software generates 2 types of tables that are useful for summarizing the quality assessment: “evidence profile” tables and a “summary of findings” table.
The AHRQ Effective Health Care program (15) has essentially adopted the GRADE framework; however, it uses the term “strength” to assess what GRADE calls the “quality” of a body of evidence. Rather than use the term “very low” to categorize the lowest quality of evidence, the Effective Health Care program uses the term “insufficient.” Furthermore, the Effective Health Care program has an additional domain to assess “applicability” of the body of evidence to account for the degree to which the evidence is relevant to patient populations, settings, diseases or conditions, interventions, comparators, and outcomes of interest. At this time the Effective Health Care program does not mandate use of GRADEpro or other software and allows for use of other weighting systems or a qualitative approach.
Many medical professional societies (e.g., American College of Physicians, American College of Chest Physicians, and the American Academy of Neurology), government organizations (e.g., the NHLBI), and healthcare organizations (e.g., Kaiser and the Institute for Clinical Systems Improvement) have revised their grading systems and qualitative syntheses of a body of evidence to incorporate all or some of the characteristics or domains used by GRADE, AHRQ, the Cochrane Collaboration, and other leaders in SR methodology. Some organizations use a combination of the GRADEpro software evidence profiles and summary of findings tables and additional narrative description to characterize the body of evidence, whereas others rely on an entirely descriptive approach.
Most ACCF/AHA CPGs currently include narrative descriptions and evidence tables detailing the clinical characteristics of individual studies (e.g., patient populations, inclusion/exclusion criteria, endpoints, point estimates, and confidence intervals). Current ACCF/AHA CPGs do not, however, explicitly evaluate a body of evidence by outcome, nor do they explicitly take into consideration an evaluation of the basic characteristics of quality using a specific checklist or quality scale tool. Rather, the assessment of quality (and strengths and weaknesses of each study) is inherent to the overall evidence review and is based on the recognized expertise of the GWC.
4.2 Workgroup 4 Recommendations for Assessing and Addressing the Quality of a Body of Evidence
1. Accept the IOM-recommended basic characteristics of quality and elements for assessing quality across studies.
2. Standardize the method of assessment and description of the quality of the body of evidence across studies and carry out a qualitative assessment of individual studies and aggregated studies. (This will require allocation of resources for additional methodological expertise, because this exercise is judged to be beyond the expertise and time commitment expected of GWC members alone; an ERC would likely assume this role.)
3. Depict qualitative assessment across studies addressing the key elements for each PICO(TS) question in a summary table (e.g., similar to GRADE or other formats [Table 4.1]).
4. Where appropriate, provide a risk-of-bias assessment table across studies (e.g., similar to the Cochrane format (10) that includes items such as random sequence generator, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selection report, or other bias.
5. Generate standardized summary and evidence table templates specific to ACCF/AHA requirements.
6. When available, high-quality SRs from reputable organizations (e.g., the Cochrane Collaboration and AHRQ evidence-based practice centers [EPC]) should be used. If a new SR is needed, resources will need to be secured to pursue de novo analyses.
7. Notwithstanding these efforts to standardize and improve the qualitative analysis of bodies of evidence, every effort should be made to minimize delays in time for CPG development/revision.
2. Standards for quantitative analysis (i.e., meta-analysis): Quantitative analysis refers to conducting a (multiple) meta-analysis and deals with the statistical manipulation of data along with the aggregation of data presented in individual studies. This analysis should address heterogeneity among study effects, determine the statistical uncertainty for all estimates, and assess the sensitivity of analysis based on the changes/differences in protocol, assumptions, and study selection.
Quantitative Analysis of Evidence
The standard evidence review to date has focused on qualitative attributes of relevant data to determine the LOE that might inform a CPG. This approach, although very pragmatic and reasonably successful in the past, involves subjective assessments and may be less appropriate for increasingly complex data in contemporary clinical trials and/or observational data.
Addressing Conventional Quantitative Analyses
Broadly speaking, a quantitative analysis would consider a more detailed assessment of the strength of evidence beyond the ordinary “positive” RCT. The notion of a positive trial as the ultimate arbitrator of an evidence base does not speak to the magnitude of the evidence or the potential impact of the evidence if applied to de novo patient cohorts. Specifically, a threshold p value of 0.05 is known to be an arbitrary threshold to determine significance and practically can be defined as the likelihood that the given finding is a random occurrence in 5%, or 1 chance in 20. This may or may not be compelling for certain interventions, and a stronger threshold, p<0.01, or the randomness is <1 chance in 100, may be more appropriate. This may be especially the case for more invasive or expensive interventions. An even more stringent threshold might be appropriate for large observational datasets that have a large number of observations, and a p value of 0.05 might be easily reached for nonsensical (nonclinically significant) observations. In these cases, a threshold of p<0.001, or <1 chance in 1,000 that the results are due to random accident, might be the preferred strategy. Thus, a more appropriate fit of statistical threshold with a data source and within the context of the given data might be a more reasonable approach and would add a helpful quantitative analysis to the current ACCF/AHA evidence review. It is likely that current GWCs exercise this kind of thoughtful review empirically, but as a policy statement, direction for these types of quantitative assessment have not previously existed.
Considering Newer Quantitative Analyses
Beyond the determination of p values, additional quantitative insight is determined from a review of confidence levels. Understanding the “splay” of data may put the top line findings of a given study in a different context and either strengthen the LOE despite a modest level of statistical significance or temper the LOE for a reasonable level of significance but with broad confidence levels. Similarly, evaluating relative risk reduction versus absolute risk reduction can be quite informative. A large multithousand-patient study with a relative risk reduction of 0.15 might reflect an actual risk reduction of only 1% or 2%. This is important information that generates a new perspective/interpretation of clinical trial results. The number needed to treat versus the number needed to harm represents a very informative quantitative representation of available data and further allows discrimination of the veracity of the findings. Importantly, the comparison of number needed to treat and number needed to harm captures the burden of risk associated with emerging new therapies for cardiovascular disease.
Incorporating More Rigorous Quantitative Methodology
Meta-analyses and Bayesian analyses are powerful data analytical tools that may further inform the significance of candidate data to support the generation of CPGs. However, care must be exercised in the use of both of these rigorous methodologies. Meta-analyses are subject to the quality of the studies that are aggregated to either identify a stronger signal of benefit or harm. Publication bias is a consistent concern, and a funnel plot analysis is necessary to mitigate this known source of bias. Moreover, many trials are sufficiently dissimilar so that the construct of a meta-analysis may be fundamentally flawed at the outset. However, a well-constituted meta-analysis may confirm findings from isolated trials and can be of sufficient quality to inform CPGs at the highest tier of evidence.
Bayesian analyses are similarly not perfect tools. The prior probability of an event determined from clinical judgment sets the threshold from which the posterior probability of a Bayesian analysis might be constructed. Bias may influence even a Bayesian analysis. However, a Bayesian approach is the closest statistical analytic model that approximates clinical judgment and decision making; in clinical practice, all practitioners begin with a “prior probability,” that is, an informed perspective of the expected clinical response to or outcomes of a test or intervention. Thus, the results of a well-performed Bayesian analysis may be very helpful in the construct of a CPG.
A more contemporary evidence review is needed for novel datasets that include not only traditional RCTs but also large observational datasets, adaptive clinical trial designs, genomic datasets, and nonrandomized data. As more quantitative data standards are adopted, it will be necessary to have statistical expertise embedded in the infrastructure of all CPG generation paradigms.
4.3 Workgroup 4 Recommendations for Standards for Quantitative Analysis (i.e., Meta-Analysis) Across a Body of Evidence
1. GWCs, in conjunction with the Task Force, should determine when a specific meta-analysis is needed.
2. With statistical consultation, preferred acceptable methods for meta-analysis (including a Bayesian analysis when appropriate) should be defined in each instance.
3. When available, high-quality meta-analyses from reputable organizations (e.g., the Cochrane Collaboration and AHRQ EPC) should be used. If a new meta-analysis is needed, resources will need to be secured to pursue de novo analyses.
4. Notwithstanding efforts to standardize and improve the approach to quantitative analyses across studies, every effort should be made to minimize delays in time for CPG development/revision.
3. Review of ACCF/AHA COR/LOE versus other grading methodologies such as GRADE and NHLBI methodology for grading evidence and recommendations. Questions posed to the Workgroup on grading recommendations included the following:
• What features of the COR/LOE allow for the synthesis and grading of evidence across multiple studies?
• What features of GRADE, AHRQ EPC, and NHLBI methodology allow for the synthesis and grading of evidence across multiple studies?
• What are some changes that may help the COR/LOE incorporate/improve on its current methods of synthesizing and grading evidence across multiple studies?
The objective of this question was to take a “10,000-foot” view of the ACCF/AHA COR/LOE approach and how this current methodology addresses and/or allows for the overall synthesis of evidence across multiple studies. Then, the objective was to compare the ACCF/AHA methodology with some other prominent methodologies in current use, such as GRADE, NHLBI, and AHRQ EPC (Tables 4.2 to 4.7).⇓⇓⇓⇓⇓⇓⇓ The final goal then was to recommend whether and what practical changes/additions might be made to current ACCF/AHA COR/LOE methodology to improve its clarity and accuracy.
Review of COR/LOE Versus Other Methodologies
Professional societies that write CPGs typically provide the end user with a recommendation that has 2 dimensions. The first dimension is a statement of the strength of the recommendation, signifying the GWC's synthesized interpretation of the evidence with respect to how strongly the committee believes that the action/treatment may contribute to providing optimum care of a patient with the target condition discussed in the CPGs. The second dimension is a statement about the strength of the evidence on which the recommendation is based. The purpose of the strength of evidence component is to offer the end user insight into the range and type of evidence reviewed by the GWC when formulating the recommendation. The strength of recommendation and strength of evidence notations are provided in a hierarchical scheme that is ordinal and does not imply any mathematical relationship along the continuum of recommendations. The scheme by which the ordinal bins are depicted to end users varies by organization and may contain combinations of numbers, letters, or phrases that have face validity for conveying the rank order intended by the GWC.
Although the universally agreed-on purpose of CPGs is to offer an instruction set to clinicians caring for patients, it is also understood that individual clinical circumstances may necessitate deviation from the recommended path. Clinicians are advised to document their reasons for deviation from the recommendations. Beyond these straightforward statements, there is variation in how organizations view the intended use of their CPGs. Some organizations write their CPGs with anticipation that they will be intended for end users beyond clinicians, including policy makers responsible for reimbursement for medical care and even patients themselves. When such extended targets of the recommendations are contemplated by GWCs, the instructions include consideration of resources and perceived patient preferences.
Perspective on ACCF/AHA CPG Grading System Relative to Other Systems
With the background noted previously, it is useful to reflect on the ACCF/AHA CPG grading system. The intended audience is clinicians in North America caring for patients with cardiovascular disease. Although it is anticipated that the recommendations may be read with interest by policy makers and even patients, the instructions to GWCs for ACCF/AHA CPG development do not explicitly ask that the wording of recommendations take into account cost or patient preferences (although the GWC addresses the latter in each preamble, noting that patients should be informed of the risks, benefits, and alternatives to a particular treatment and should be involved in shared decision making whenever feasible, particularly for COR IIa and IIb, where the benefit-to-risk ratio may be lower). The GWC may comment in the accompanying text regarding resource implications, but such considerations are not intended to drive the wording of a recommendation. Underlying the ACCF/AHA methodology for CPG development is a set of considerations for focusing the ACCF/AHA recommendations specifically on the science of the data and are: 1) to reserve a place in the set of instructions for clinicians where science trumps other considerations because cost is dynamic and variable from system to system, 2) to allow those bodies in our society that determine how resources are to be allocated to have a reference point for the science review, and 3) in the absence of formal patient data using time tradeoffs and utility function testing, it is difficult to describe with certainty the distribution of patient preferences.
As with the other grading schemes, recommendations with the largest estimated size of intervention effect in clinical practice are those that are the strongest for (green rows in accompanying Table 4.3) or against (red rows in accompanying Table 4.3) providing an intervention because of lack of benefit or even some evidence of harm. All other recommendations are discretionary, and clinical responses to them are heavily influenced by the characteristics of the patient and the circumstances surrounding that patient's care.
There are no rigorously performed studies comparing the consequences of clinical actions that adhere to recommendations provided in one scheme versus another. Data are available, largely from registry reports, that adherence to the current ACCF/AHA system (Table 4.3), which is familiar to cardiovascular clinicians worldwide and drives ACCF/AHA performance measures, is associated with improved patient outcomes (11,12). In the absence of evidence of superiority of one grading system over another, a decision was made to retain the current ACCF/AHA COR/LOE structure and nomenclature, with minor modifications to “map” to terminology in other well-known grading systems. Data from individual studies and multiple studies (pooled by either the random effects or fixed effects methods) can be accommodated by the current ACCF/AHA system.
4.4 Workgroup 4 Recommendations Regarding ACCF/AHA Grading Methodology and Nomenclature
1. Retain the current basic ACCF/AHA COR and LOE structure/nomenclature.
2. Standardize how LOE: A, B, C are determined using a validated ACCF/AHA Evidence Grading Tool (under development) that will incorporate features of existing tools.
3. Change the wording of LOE to Quality of Evidence (QOE) (once the ACCF/AHA Evidence Grading Tool is operational).
4. Add a separate category for QOE: E (expert opinion), and generate specific definitions and examples for QOE: E.
5. Change “Treatment Effect” on the COR/LOE table to “Intervention Effect” and indicate that “Intervention” includes medications, devices, therapeutic strategies, procedures, diagnostic tests, and other.
6. Add adjectives that “map” COR/LOE to the NHLBI (and GRADE) (i.e., I [strong]; IIa [moderate]; IIb [weak]; III [against]).
7. Take every opportunity to align COR as closely as possible with LOE/QOE.
8. Provide a chart to map ACCF/AHA COR/LOE to other major grading systems (e.g., Table 4.2).
1. Plan for regular review (e.g., annually) of CPG recommendations and modify as appropriate.
2. Develop a list of metrics that should be measured and assessed.
4.5 Workgroup 4 Additional Recommendations
Workgroup 4 recommends that additional assessment of the methodological quality (risk of bias) in individual studies, as well as an evaluation of essential characteristics of quality across the body of evidence, by outcome, be incorporated into a qualitative synthesis of evidence in ACCF/AHA CPGs. To accomplish this, Workgroup 4 recommends the following:
1. Consider use of a standardized tool, such as the Cochrane Risk of Bias Tool, for assessing risk of bias within and across studies (10). For observational studies, use a tool that specifically evaluates the methodological limitations of such studies, such as risk of selection bias, potential confounding factors that may affect the estimate of effect, and biases in the design and execution of nonrandomized studies. The Cochrane Collaboration is currently modifying its Risk of Bias tool for use in nonrandomized studies.
2. For assessment of consistency, precision, directness, reporting bias, and, with observational studies, large magnitude of effect, a dose-response association, or confounding factors that would change the reported effect, follow the GRADE or AHRQ definitions for assessment of each characteristic across the body of evidence for each outcome.
3. Consider depicting the summary evaluation of the quality characteristics for a body of evidence in a table format. Use the GRADE evidence profiles and summary of findings tables as models of table formats.
4. In addition to summary quality evaluation tables, provide a high-level narrative synthesis of the quality of a body of evidence that includes the IOM's 4 elements of performance for conducting a qualitative synthesis, as described above.
5 Workgroup 5: Standards for Reporting Systematic Reviews
(Authors: Dr. E. Magnus Ohman, Chair; Drs. Deepak L. Bhatt, Kay Dickersin, Raymond J. Gibbons, and William G. Stevenson)
CPGs have become an important part of the practice of cardiology and other aspects of healthcare delivery. They represent some of the most downloaded and quoted papers published in the literature and therefore are of major importance to the practicing clinician. Furthermore, several outcomes studies have suggested that adherence to ACCF/AHA CPGs is associated with both better in-hospital as well as long-term outcomes (11,12,21). There are 2 crucial factors for dissemination and acceptance of CPGs. They need to be readable and not merely tables of synthesized information. They should also allow the reader to understand the logic and process of evidence synthesis to form coherent recommendations that inform the practice of medicine (22). Therefore, in producing CPGs, GWCs need to recognize that although the body of data for a recommendation ranges from minimal to extensive; the presentation must be comprehensive, logical, and readable to be a relevant resource for clinical practice.
As recommended throughout this report, our goal is for SR to underpin our CPG recommendations. Recognizing that it is not possible to complete SR for all CPGs over a short period of time, we recommend that our CPGs be transformed sequentially, with SR being sought or performed first for those topics deemed to be of greatest importance. When high-quality SRs are available and up to date, they should be summarized and used as a basis for our CPGs. In the event that “conflicting” meta-analyses have been published, thoughtful consideration should be given to developing a formal SR to provide clarity and therefore enable better recommendations for the CPGs. When high-quality SRs do not exist or an update is needed, the ACCF/AHA could commission a new review. In cases where an SR has been determined to be of lower importance to other topics at that time, an informal assessment of evidence combined with expert opinion is conducted.
On occasion, SR may be performed on topics for which there are few or no RCTs. With only 1 trial or a few small trials, one is not sure whether reporting biases may threaten the validity of the summary findings. Where no RCTs exist and the question of intervention effectiveness is important or RCTs are difficult to perform (e.g., during cardiac arrest), investigators may decide to seek data from observational studies such as analysis of administrative or registry data. SR of observational studies are potentially more complex than SR of RCTs, as searching methods for observational studies are not well established and important observational data could be missed. In addition, observational studies are subject to selection and other forms of bias that can make their findings difficult to interpret. Thus, SR of observational data should be used sparingly and will require careful review (see Workgroups 2, 3, and 4).
Incorporating Systematic Reviews as Part of the CPGs
CPGs are, by design, summaries of SRs that can be quite lengthy and detailed. In areas such as hypertension or prevention, for example, literally hundreds of studies are potentially eligible for SR of intervention effectiveness. With the use of methods for summarizing all relevant data, an SR fully encapsulates information in a condensed format that can be translated into CPGs.
In the early 1990s, the AHRQ supported the University of California San Francisco−Stanford EPC to partner with the ACCF/AHA team producing the “ACC/AHA/American College of Physicians−American Society of Internal Medicine Guidelines for the Management of Patients with Chronic Stable Angina,” published in 1999 (23,24). An EPC representative of that group attended the meetings of the GWC, and the EPC made several contributions to the CPG. First, the EPC performed a number of small literature searches that led to studies being incorporated into the CPG, for example, a summary of RCTs and meta-analyses of garlic therapy for treatment of risk factors appeared in the tables and figures of the final CPG. The largest portion of the work done by the EPC was a careful SR of the evidence for medical therapy for chronic stable angina and, in particular, comparisons of the commonly used drugs (beta blockers and calcium antagonists) that were used in the treatment of chronic stable angina. This was published separately (24). An example of incorporating a salient summary of the EPC finding into the text of the CPG is shown in Tables 5.1a and 5.1b, and Figures 5.1a and 5.1b. This approach allowed the natural flow of the CPG to be maintained and allows the complete SR, representing original research on its own, to be published independently.
Independent review of SRs in addition to that of CPGs is an important check on scientific rigor and validity. We believe that the independent review performed in the course of consideration for publication achieves this aim. Previously completed SRs that have been published in a peer-reviewed journal already meet this standard. We recommend that a newly commissioned SR also be published separately and independently from the CPGs and subject to the journal's peer review process, which will ensure that they are of the highest standard. Independent publication would also allow the academic group that is charged with performing the SR to achieve academic recognition for its work. However, independent publication does create challenges. Ideally, the SR should be accepted and at least in press before it is incorporated into the CPGs so that any changes in response to peer review have been included. The publication of SR findings separately from evidence that is based on the SR and incorporated into the CPGs should not be considered duplicate publication. The ACCF/AHA CPGs focus on the summary information and any salient figures or tables that are included are modified and cited appropriately. Perhaps, a more difficult issue relates to the timing of publication and embargo. Although publication of the SR potentially provides a preview of information from the CPGs to interested parties, we do not feel that this would diminish the CPGs, as the SR is only a portion of the information that will be provided in the CPG and does not include recommendations and additional clinical synthesis. SRs do not make practice recommendations; they are simply a summary of relevant data addressing a clinical question. The GWC will provide COR/LOE on review of the SR and in conjunction with other recommendations applicable to a similar clinical setting. Thus, the SR and CPG can be developed simultaneously (with potential edits/revisions made in response to peer review) and the SR published before, without embargo, or nearly simultaneously with the CPG.
Discordance between the ERC and the GWC will be resolved through liaison committee members (see Workgroup 2). A key feature of the review will be transparency. The GWC should specifically cite the SR and indicate how it was incorporated into the CPG.
Guideline Review Tables in the Absence of a Systematic Review
Because SR will not initially be possible for all recommendations, it will be important to distinguish recommendations based on SR from those based on narrative reviews and/or expert opinion. For transparency, the evidence base for recommendations based on non-systematically collected information should be provided in summary tables called “CPG review tables” as well. Items presented would be similar to those presented for trials included in the SR evidence tables, for example, characteristics of the individual studies and factors that allow an assessment of potential bias (i.e., study “quality”), measures of effect size for each study, and list of primary and secondary outcomes reported. Particular attention will be paid to ensuring that all relevant studies are included and that the components noted above are included. CPG review tables from nonsystematic reviews should not be given more prominence than evidence tables from SRs and so can be referred to in the text but are better placed in online data supplements. It is important, though, that the printed or online text, tables, and figures summarize the information to enhance readability of the document and that the recommendations flow naturally from the text. The components of such tables and examples are listed in Table 5.2.
Like the evidence tables from SRs, the CPG review tables will undergo peer review as part of the standard review process.
5.1 Workgroup 5 Recommendations
CPGs are strengthened when they are based on SR. Because it is not possible to complete SR immediately for all questions covered by ACCF/AHA CPGs, SR will be completed in a sequential fashion in order of importance of the question. Non-SRs and accompanying summary tables will be used for areas where SRs are not yet completed; however, SRs may be performed over time. It is believed that this combination will bring the current approach of the ACCF/AHA CPGs in compliance with the recommendations of the IOM, yet allow the CPGs to be readable, timely, and relevant.
1. SRs included in CPGs should be published as separate peer-reviewed manuscript(s) when feasible. By doing so, all of the recommendations regarding methodology, peer review, and access will be compliant with the IOM recommendations (2).
2. Recommendations supported by SR should be identified in CPGs in addition to the appropriate class and LOE. Example: (Class I; LOE: A)SR.
3. Within the CPG, salient tables and figures from SRs should be included to support recommendations. The remaining pieces of methodology will be hyperlinked to the original SR publication. By doing so, key elements of the SR will be available for public access once the CPG are published.
4. CPG review tables should be incorporated into CPGs for non−SR-based recommendations and be available as online supplement tables.
5. CPG review tables should incorporate the majority of study components that are provided in SR tables.
Presidents and Staff
American College of Cardiology Foundation
William A. Zoghbi, MD, FACC, President
Thomas E. Arend, Jr., Esq., CAE, Interim Chief Staff Officer
William J. Oetgen, MD, MBA, FACC, Senior Vice President, Science and Quality
Charlene L. May, Senior Director, Science and Clinical Policy
Erin A. Barrett, MPS, Senior Specialist, Science and Clinical Policy
American College of Cardiology Foundation/American Heart Association
Lisa Bradfield, CAE, Director, Science and Clinical Policy
Debjani Mukherjee, MPH, Associate Director, Evidence-Based Medicine
Ezaldeen Ramadhan III, Specialist, Science and Clinical Policy
American Heart Association
Donna K. Arnett, PhD, MSPH, BSN, FAHA, President
Nancy Brown, Chief Executive Officer
Rose Marie Robertson, MD, FAHA, Chief Science Officer
Gayle R. Whitman, PhD, RN, FAHA, FAAN, Senior Vice President, Office of Science Operations
Judy Bezanson, DSN, RN, CNS-MS, FAHA, Science and Medicine Advisor, Office of Science Operations
Appendix 1 ACCF/AHA Clinical Practice Guideline Methodology Summit Report—Author Listing of Comprehensive Relationships With Industry and Others (December 2011)
|Committee Member||Employment||Consultant||Speaker's Bureau||Ownership/Partnership/Principal||Personal Research||Institutional, Organizational, or Other Financial Benefit||Expert Witness|
|Alice K. Jacobs||Boston University School of Medicine and Boston Medical Center, Department of Medicine, Section of Cardiovascular Medicine—Professor of Medicine; Cardiac Catheterization Laboratories and Interventional Cardiology—Director||None||None||None||None|
|Nancy M. Albert||Cleveland Clinic, Kaufman Center for Heart Failure, Nursing Research, Innovation and CNS—Senior Director||None||None||None||None|
|Jeffrey L. Anderson||University of Utah School of Medicine—Professor of Medicine; Intermountain Medical Center, Intermountain Healthcare—Associate Chief of Cardiology; Cardiovascular Department—Director of Research; Internal Medicine—Vice-Chair of Research||None||None||None|
|Elliott M. Antman||Harvard Medical School—Associate Dean for Clinical/Translational Research||None||None||None||None||None|
|Donna K. Arnett||University of Alabama at Birmingham, EpidemiologySchool of Public Health—Chair and Professor||None||None||None||None||None||None|
|Marnie Bertolet||University of Pittsburgh, Department of Epidemiology, Graduate School of Public Health—Assistant Professor||None||None||None||None||None||None|
|Deepak L. Bhatt||VA Boston Healthcare System—Chief of Cardiology; Brigham and Women's Hospital—Senior Physician; TIMI Study Group—Senior Investigator; Harvard Medical School—Professor of Medicine||None||None||None||None|
|Ralph G. Brindis||Northern California Kaiser Permanente—Senior Advisor for Cardiovascular Disease; University of California, San Francisco—Clinical Professor of Medicine||None||None||None||None||None|
|Mark A. Creager||Brigham and Women's Hospital, Vascular Center—Director; Harvard Medical School—Professor of Medicine||None||None||None||None|
|David L. DeMets||University of Wisconsin—Max Halperin Professor of Biostatistics and Medical Informatics||None||None||None|
|Kay Dickersin||Johns Hopkins Bloomberg School of Public Health—Professor; Center for Clinical Trials— Director; U.S. Cochrane Center, Department of Epidemiology—Director||None||None||None||None||None|
|Steven M. Ettinger||Penn State University, Penn State Heart & Vascular Institute—Professor of Medicine and Radiology; Interventional Program Director||None||None||None||None||None|
|Gregg C. Fonarow||UCLA, Division of Cardiology—Professor of Medicine; Ahmanson-UCLA Cardiomyopathy Center—Director||None||None||None|
|Raymond J. Gibbons||Mayo Clinic—Professor of Medicine||None||None||None||None|
|Robert A. Guyton||Emory University School of Medicine—Distinguished Charles Ross Hatcher, Jr., Professor of Surgery; Emory University Hospital—Chief of Cardiothoracic Surgery||None||None||None||None|
|Jonathan L. Halperin||Mount Sinai School of Medicine—Robert and Harriet Heilbrunn Professor of Medicine (Cardiology)||None||None||None||None|
|John G. Harold||Cedars-Sinai Heart Institute and David Geffen School of Medicine at UCLA—Clinical Professor of Medicine||None||None||None||None||None|
|Yulei He||Harvard Medical School, Department of Health Care Policy—Assistant Professor of Health Care Policy||None||None||None||None||None||None|
|Judith S. Hochman||New York University School of Medicine—Harold Snyder Family Professor of Cardiology; NYU-HHC Clinical and Translational Science Institute—Co-Director; Leon Charney Division of Cardiology—Clinical Chief; Cardiovascular Clinical Research Center—Director||None||None||None||None|
|Marguerite A. Koster||Kaiser Permanente Southern California, Permanente Medical Group, Technology Assessment & Guidelines Unit—Practice Leader||None||None||None||None||None||None|
|Frederick G. Kushner||Tulane University Medical Center—Clinical Professor; Heart Clinic of Louisiana—Medical Director||None||None||None||None||None|
|Pamela B. Mangu||American Society of Clinical Oncology—Senior Practice Guideline Specialist||None||None||None||None||None||None|
|Sharon-Lise T. Normand||Harvard Medical School, Department of Health Care Policy—Professor of Health Care Policy- Biostatistics||None||None||None||None|
|E. Magnus Ohman||None||None||None|
|Eduardo Ortiz||NIH/NHLBI, Division for the Application of Research Discoveries—Senior Medical Officer||None||None||None||None||None||None|
|Eric D. Peterson||Duke University Medical Center—Professor of Medicine & Director of Clinical Research Institute||None||None||None||None||None|
|Amir Qaseem||American College of Physicians—Director, Clinical Policy Medical Education Division||None||None||None||None||None||None|
|William H. Roach, Jr.||AHA—Chairman of the Board||None||None||None||None|
|Ralph L. Sacco||University of Miami, Miller School of Medicine, Department of Neurology—Chairman||None||None||None||None||None|
|Michael R. Sayre||Ohio State University—Associate Professor of Emergency Medicine||None||None||None||None||None|
|Sidney C. Smith, Jr.||University of North Carolina at Chapel Hill—Professor of Medicine||None||None||None||None||None||None|
|Mark R. Somerfield||American Society of Clinical Oncology—Director, Clinical Affairs||None||None||None||None||None||None|
|William G. Stevenson||Brigham and Women's Hospital, Cardiac Arrhythmia Program—Director; Harvard Medical School—Professor of Medicine||None||None||None|
|Gordon F. Tomaselli||None||None||None||None||None|
|Clyde W. Yancy||Northwestern University, Feinberg School of Medicine—Magerstadt Professor of Medicine; Division of Cardiology—Chief||None||None||None||None||None|
|William A. Zoghbi||Methodist DeBakey Heart and Vascular Center—Cardiovascular Imaging Director; William L. Winters Chair in Cardiovascular Imaging||None||None||None||None||None|
This table represents all healthcare relationships of authors with industry and other entities that were reported by authors, including those not deemed to be relevant to this document, at the time this document was under development. The table does not necessarily reflect relationships with industry at the time of publication. A person is deemed to have a significant interest in a business if the interest represents ownership of ≥5% of the voting stock or share of the business entity, or ownership of ≥$10,000 of the fair market value of the business entity; or if funds received by the person from the business entity exceed 5% of the person's gross income for the previous year. Relationships that exist with no financial benefit are also included for the purpose of transparency. Relationships in this table are modest unless otherwise noted. Please refer to http://www.cardiosource.org/Science-And-Quality/Practice-Guidelines-and-Quality-Standards/Relationships-With-Industry-Policy.aspx for definitions of disclosure categories or additional information about the ACCF/AHA Disclosure Policy for Writing Committees.
ACCF indicates American College of Cardiology Foundation; AHA, American Heart Association; C-PORT, The Atlantic Cardiovascular Patient Outcomes Research Team; CANVAS, Canagliflozin Cardiovascular Assessment; CNS, Clinical Nurse Specialists; COAG, Clarification of Optimal Anticoagulation Through Genetics; CORAL, Cardiovascular Outcomes in Renal Atherosclerotic Lesions; CoumaGenII, Applying Pharmacogenetic Algorithms to Individualize Dosing of Warfarin; DAPT, dual antiplatelet therapy; DSMB, data safety monitoring board; EP, electrophysiology; FDA, U.S. Food and Drug Administration; GIFT, Genetics Informatics Trial of Warfarin to Prevent DVT; GWTG, Get With The Guidelines; IMPROVE HF, Registry to Improve the Use of Evidence-Based Heart Failure Therapies in the Outpatient Setting; NHLBI, National Heart, Lung, and Blood Institute; NIAID, National Institute of Allergy and Infectious Diseases; NIH, National Institutes of Health; NYU-HHC; New York University–New York City Health and Hospitals Corporation; OSHPD, Office of Statewide Health Planning and Development; PCI, percutaneous coronary intervention; PI, primary investigator; RCT, randomized controlled trial; STEMI-PCI, ST-elevation myocardial infarction–percutaneous coronary intervention; TIMI, Thrombolysis In Myocardial Infarction; UCLA, University of California, Los Angeles; and VA, Veterans Affairs.
↵⁎ Significant relationship.
↵† No financial benefit.
Appendix 2 Workgroup Comparison Tables
The following tables present each element proposed by the IOM and the Workgroups' comments and recommendations for each element.
Workgroup 1 Comparison Table
|1. Clinical Practice Guidelines We Can Trust|
|IOM Standards and Elements||ACCF/AHA Current Methodology||Workgroup 1 Proposal|
|1. Establishing transparency|
|1.1 The processes by which a CPG is developed and funded should be detailed explicitly and publicly accessible.||The ACCF/AHA Methodology Manual (4) and processes are available to the public on the ACCF and AHA websites. Selected processes are also detailed in the preamble published with each CPG.||The ACCF/AHA CPG methodology is currently compliant.|
|2. Management of COI|
|2.1 Prior to selection of the Guideline Development Group (GDG), individuals being considered for membership should declare all interests and activities potentially resulting in COI with development group activity, by written disclosure to those convening the GDG.||The ACCF/AHA has a very detailed RWI policy that includes this standard.||Patient representatives and all other GWC members should explicitly declare any intellectual bias or perspective for any topic covered by the guideline. The GWC chair is responsible for balancing competing intellectual perspectives during deliberations and in the final document.|
|• Disclosure should affect all current and planned commercial (including services from which a clinician derives a substantial proportion of income), noncommercial, intellectual, institutional, and patient/public activities pertinent to the potential scope of the CPG.||The ACCF/AHA has a very detailed RWI policy that includes this standard.||The ACCF/AHA CPG methodology is currently compliant.|
|2.2 Disclosure of COIs within GDG|
|• All COI of each GDG member should be reported and discussed by the prospective development group prior to the onset of their work.||The ACCF/AHA has a very detailed RWI policy that includes this standard.||The ACCF/AHA CPG methodology is currently compliant.|
|• Each panel member should explain how their COI could influence the CPG development process or specific recommendations.||The ACCF/AHA has a very detailed RWI policy that includes this standard.||The ACCF/AHA CPG methodology is currently compliant.|
|• Members of the GDG should divest themselves of financial investments they or their family members have in, and not participate in marketing activities or advisory boards of, entities whose interests could be affected by CPG recommendations.||The ACCF/AHA RWI policy does not require divestment but requires that RWI be balanced and managed appropriately.||Divestment is not required by ACCF/AHA policy.|
|2.4 Exclusions||See Section 2.3||See Section 2.3|
|• Whenever possible GDG members should not have COI.||The ACCF/AHA policy requires that a majority of the GWC (including the chair) have no relevant RWI.||The ACCF/AHA policy requires that a majority of the GWC (including the chair) have no relevant RWI.|
|• In some circumstances, a GDG may not be able to perform its work without members who have COIs, such as relevant clinical specialists who receive a substantial portion of their incomes from services pertinent to the CPG.||The ACCF/AHA has a very detailed RWI policy that includes this standard.||The ACCF/AHA CPG methodology is currently compliant.|
|• Members with COIs should represent not more than a minority of the GDG.||The ACCF/AHA policy requires that a majority of the GWC (including the chair) have no relevant RWI.||The ACCF/AHA CPG methodology is currently compliant.|
|• The chair or co-chairs should not be a person(s) with COI.||The chair/co-chairs may not have relevant RWI, but a vice chair is permitted to do so.||The ACCF/AHA CPG methodology is currently compliant.|
|• Funders should have no role in CPG development.||The ACCF/AHA does not allow any outside funding for CPG development.||The ACCF/AHA CPG methodology is currently compliant.|
|3. GDG composition|
|3.1 The GDG should be multidisciplinary and balanced, comprising a variety of methodological experts and clinicians, and populations expected to be affected by the CPG.||The ACCF/AHA GWC is a multidisciplinary group that may include (in addition to clinicians and clinical scientists) a methodologist, pharmacologist, nurse, internist, and other subspecialty physicians.||The ACCF/AHA CPG methodology is currently compliant.|
|3.2 Patient and public involvement should be facilitated by including (at least at the time of clinical question formulation and draft CPG review) a current or former patient and a patient advocate or patient/consumer organization representative in the GDG.||The ACCF/AHA GWC does not currently include patients and public representatives.||Inclusion of a patient representative in the GWC (and on the Task Force) is recommended. Patient representatives with IOM-defined “health literacy” skills are preferred. Relevant RWI and intellectual perspectives should be declared and managed by the GWC chair. Patient representatives will be expected to participate in the development of key clinical questions, choices, and opportunities for shared decision making by incorporating patient-centered values and preferences, issues concerning quality of life, and patient-oriented summaries of recommendations.|
|3.3 Strategies to increase effective participation of patient and consumer representatives, including training in appraisal of evidence, should be adopted by GDGs.||The ACCF/AHA GWC does not currently include patients and public representatives||All nonprofessional members of the GWC should avail themselves of the U.S. Cochrane Center and CUE's “Understanding Evidence-Based Healthcare: A Foundation for Action” (9) or an equivalent online learning module before accepting their position.|
|4. CPG-SR intersection|
|4.1 CPG developers should use SRs that meet standards set by the IOM's Committee on Standards for Systematic Reviews of Comparative Effectiveness Research.||Current ACCF/AHA CPG evidence-review methodology does not incorporate a formal SR, and this is under review by Workgroups 2–5.||Workgroups 3–5 will provide recommendations.|
|4.2 When SRs are conducted specifically to inform particular guidelines, the GDG and SR team should interact regarding the scope, approach, and output of both processes.||Current ACCF/AHA CPG evidence-review methodology does not incorporate a separate ERC, and this is under review by Workgroup 2.||Workgroup 2 will provide recommendations.|
|5. Establishing evidence foundations for and rating strength of recommendations|
|Workgroup 4 will provide recommendations.|
|6. Articulation of recommendations|
|6.1 Recommendations should be articulated in a standardized form detailing precisely what the recommended action is and under what circumstances it should be performed.||The ACCF/AHA COR/LOE methodology includes this information.||The ACCF/AHA CPG methodology is currently compliant.|
|6.2 Strong recommendations should be worded so that compliance with the recommendation(s) can be evaluated.||The ACCF/AHA CPG methodology is currently compliant.|
|7. External review|
|7.1 External reviewers should comprise a full spectrum of relevant stakeholders, including scientific and clinical experts, organizations (e.g., health care, specialty societies), agencies (e.g., federal government), patients, and representatives of the public.||CPG should be provided to an expanded group of external reviewers before publication. External reviewers would comprise a full spectrum of relevant stakeholders (in addition to the current group of scientific and clinical content experts and partnering, collaborating, and other relevant professional societies), such as healthcare specialty societies, agencies (e.g., federal government), and representatives of the public.|
|7.2 The authorship of external reviews submitted by individuals and/or organizations should be kept confidential unless that protection has been waived by the reviewer(s).||The reviewer names (and RWI) are known to the GWC and are published as an appendix to the guideline.||See 7.1 above.|
|7.3 The GDG should consider all external reviewer comments and keep a written record of the rationale for modifying or not modifying a CPG in response to reviewers' comments.||The ACCF/AHA CPG peer review process is managed with spreadsheets of all comments and responses. Official reviewers receive responses to their comments during the adjudication process.||The ACCF/AHA CPG methodology is currently compliant.|
|7.4 A draft of the CPG at the external review stage or immediately following it (i.e., prior to the final draft) should be made available to the general public for comment. Reasonable notice of impending publication should be provided to interested public stakeholders.||The ACCF/AHA CPG development process does not include external access to and review by the general public.||The public posting for comment was considered by the Workgroup. The CPG process very carefully avoids undue influence and pressure from external stakeholders during CPG development. We value the confidentiality and embargo policies of the organizations and are concerned about the potential influence of industry and others on the process that cannot be adequately managed during public review.|
|8.1 The CPG publication date, date of pertinent systematic evidence review, and proposed date for future CPG review should be documented in the CPG.||The ACCF/AHA CPG process currently includes the dates and evidence review period in the introduction section and notes that new evidence will be reviewed twice yearly.||The ACCF/AHA CPG methodology is currently compliant.|
|8.2 Literature should be monitored regularly following CPG publication to identify the emergence of new, potentially relevant evidence and to evaluate the continued validity of the CPG.||The ACCF/AHA CPG process includes review of all LBCTs presented at the major medical meetings twice yearly and an environmental scan of other literature is requested from the experts on the current GWC.||The ACCF/AHA CPG methodology is currently compliant.|
|8.3 CPG should be updated when new evidence suggests the need for modification of clinically important recommendation. For example, a CPG should be updated if new evidence shows that a recommended intervention causes previously unknown substantial harm; that a new intervention is significantly superior to a previously recommended intervention from an efficacy or harms perspective; or that a recommendation can be applied to new populations.||All LBCTs are reviewed from the major medical meetings twice yearly and an environmental scan of other literature is requested from the experts on the current GWC. The decision to update an existing CPG is made according to specific criteria that include (among others) large RCT or nonrandomized data deemed important on the basis of results having an impact on current safety and efficacy assumptions.||The ACCF/AHA CPG methodology is currently compliant.|
ACCF indicates American College of Cardiology Foundation; AHA, American Heart Association; COI, conflict of interest; COR, class of recommendation; CPG, clinical practice guideline; CUE, Consumers United for Evidence Based Medicine; GDG, guideline development group; GWC, guideline writing committee; IOM, Institute of Medicine; LBCT, late-breaking clinical trial; LOE, level of evidence; RCT, randomized controlled trial; RWI, relationships with industry and other entities; and SR, systematic review.
Workgroup 2 Comparison Table
|2. Standards for Initiating a Systematic Review|
|IOM Standards and Elements||ACCF/AHA Current Methodology||Workgroup 2 Proposal|
|2.1 Establish a team with appropriate expertise and experience to conduct the SR|
|2.1.1 Include expertise in the pertinent clinical content areas||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. However, GWC have representation across all content areas, including pertinent clinical areas.||A committee should be formed of individuals with appropriate expertise in SR and the clinical content being reviewed. The ERC and GWC should remain separate, but individuals should not be excluded from serving on both in a liaison role.|
|2.1.2 Include expertise in SR methods||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. However, depending on the CPG topic and availability of a methodologist with content expertise, some GWC do include a methodologist with expertise in SR.||The ERC should potentially include a methodologist and/or statistician with experience in scientific SR. If a formal SR is necessary, an ERC to perform the review should be organized, but if an SR is not necessary, the ACCF/AHA will build on the current process by adding more experts to the GWC. The GWC should be expanded with methodologists and statisticians and include stakeholders and policy makers.|
|2.1.3 Include expertise in searching for relevant evidence||The ERC will be responsible for searching and identifying evidence relevant to the CPG. In addition, a request should be made to payers and policy makers (and other identified stakeholders) to provide evidence relevant to the SR. The GWC should be expanded; see 2.1.2. Consumers should help with providing advice on specific topics that will affect an SR.|
|2.1.4 Include expertise in quantitative methods||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. However, depending on the CPG topic and availability of a methodologist with content expertise, some GWC do include a methodologist with expertise in quantitative methods.||As stated in 2.1.2, the ERC should potentially include a methodologist and statistician who have experience in scientific SR.|
|2.1.5 Include other expertise as appropriate||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. However, depending on the CPG topic and the need for unique expertise, other experts are included on GWC as necessary, that is, nurses and pharmacists.||The members of the ERC may include experts from the health field, that is, nurses, pharmacists, physician assistants, and nurse practitioners in addition to essential stakeholders. The ERC should include a librarian and/or an in-house information specialist(s) who will work with the ERC and GWC to plan the search strategy (see 3.1.1 in Workgroup 3 Comparison Table).|
|2.2. Manage bias and COI of the team conducting the SR|
|2.2.1 Require each team member to disclose potential COI and professional or intellectual bias|
|2.2.2 Exclude individuals with a clear financial conflict||The ACCF/AHA does not exclude all individuals with a relevant RWI. Rather, a minimum of 50% of the GWC in addition to the GWC chair may have no relevant RWI.||Individuals with a clear financial conflict will be excluded from membership in the ERC.|
|2.2.3 Exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users||The ACCF/AHA does not have an official policy regarding intellectual bias, but this is discussed during the initiation of each GWC to ensure that a wide range of experts and opinions are brought to the process.||Members of the ERC and GWC will disclose all potential relationships as required by the ACCF/AHA policy (4). The disclosure policy should be expanded to address relevant professional/intellectual or clinical practice perspectives as well so that the information is identified, included, disclosed, and managed.|
|2.3. Ensure user and stakeholder input as the review is designed and conducted|
|2.3.1 Protect the independence of the review team to make the final decisions about the design, analysis, and reporting of the review||The ACCF/AHA does not have a separate SR team.||The SR protocol should be open to stakeholder and public review and comment. It is recommended that the members of the ERC be responsible and make final decisions about the design, analysis, and reporting of the identified and selected data.|
|2.4. Manage bias and COI for individuals providing input into the SR|
|2.4.1 Require individuals to disclose potential COI and professional or intellectual bias||Individuals providing input into the SR will disclose all potential relationships as required by the ACCF/AHA official RWI policy (4). These individuals should be free of RWI and balanced with respect to intellectual perspective. The disclosure policy should be expanded to address relevant professional/intellectual or clinical practice perspectives as well so that the information is identified, included, disclosed, and managed.|
|2.4.2 Exclude input from individuals whose COI or bias would diminish the credibility of the review in the eyes of the intended users||The ACCF/AHA does not have an official policy regarding intellectual bias, but this is discussed during the initiation of each GWC to ensure that a wide range of experts and opinions are brought to the process.||Individuals providing input into the SR will disclose all potential relationships as required by the ACCF/AHA policy (4). Input from individuals with professional or intellectual bias would be excluded.|
|2.5 Formulate the topic for the SR|
|2.5.1 Confirm the need for a new review||The ACCF/AHA guideline process includes review of all LBCTs presented at the major medical meetings twice yearly and an environmental scan of other literature is requested from the experts on the current GWC. The decision to update an existing CPG (or develop a new or revised CPG) is made by the Task Force and GWC according to specific criteria that include (among others) large RCT or nonrandomized data deemed important on the basis of results impacting current safety and efficacy assumptions. Recommendations that impact multiple CPGs are updated in all relevant CPGs simultaneously to maintain concordance.|
|2.5.2 Develop an analytic framework that clearly lays out the chain of logic that links the health intervention to the outcomes of interest and defines the key clinical questions to be addressed by the SR||The GWC does not focus on PICO(TS) questions because ACCF/AHA CPGs cover broad topic areas. Instead, the GWC focuses on reaching consensus on the scope and clinical objective of the CPG.||PICO(TS) questions should be developed when new, revised, or focused updates are being considered.|
|2.5.3 Use a standard format to articulate each clinical question of interest||Checklist 1 in the Methodology Manual (4) details the standards used to formulate CPG objectives.||The ACCF/AHA CPG methodology is currently compliant.|
|2.5.4 State the rationale for each clinical question||Checklist 1 in the Methodology Manual (4) details the standards used to formulate CPG objectives.||The ACCF/AHA CPG methodology is currently compliant.|
|2.5.5 Refine each question based on user and stakeholder input||Checklist 1 in the Methodology Manual (4) details the standards used to formulate CPG objectives.||The ACCF/AHA CPG methodology is currently compliant.|
|2.6 Develop a systematic review protocol|
|2.6.1 Describe the context and rationale for the review from both a decision-making and research perspective||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. No rationale is developed from either a decision-making or research perspective.||The ERC, with input from the GWC, should describe the context and rationale for the review from both a decision-making and research perspective and post this information online.|
|2.6.2 Describe the study screening and selection criteria (inclusion/exclusion criteria)||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. Literature searches include inclusion and exclusion criteria based on defined search limits for MeSH searches.||The ACCF/AHA CPG methodology is currently compliant.|
|2.6.3 Describe precisely which outcome measures, time points, interventions, and comparison groups will be addressed||The ACCF/AHA does not perform a formal SR according to IOM standards, but rather the literature is reviewed and incorporated into a CPG. Because CPGs are broad-topic based, the goal of the GWC is to capture all outcome measures, time points, interventions, and comparison groups of interest.||The ACCF/AHA CPG methodology is currently compliant.|
|2.6.4 Describe the search strategy for identifying relevant evidence||The guidelines search strategy is described in the Methodology Manual (Literature Search Methodology) (4).||The ACCF/AHA CPG methodology is currently compliant.|
|2.6.5 Describe the procedures for study selection||Study selection is individualized for each topic based on availability and quality of evidence.||The IOM standard should be followed.|
|2.6.6 Describe the data extraction strategy||No explicit data extraction is completed. However, pivotal recommendation supporting evidence is included in evidence tables created by either staff or GWC members.||The IOM standard should be followed.|
|2.6.7 Describe the process for identifying and resolving disagreement between researchers in study selection and data extraction decisions||This process is not currently part of the ACCF/AHA methodology.||The IOM standard should be followed.|
|2.6.8 Describe the approach to critically appraising individual studies||An ACCF/AHA Evidence Grading Tool is being developed to systematize the analysis of individual studies.||The ACCF/AHA Evidence Grading Tool should be used once validity and reliability testing is completed.|
|2.6.9 Describe the method for evaluating the body of evidence, including the quantitative and qualitative synthesis strategies||This is done implicitly by GWC members.||Workgroup 4 will provide recommendations.|
|2.6.10 Describe and justify any planned analyses of differential treatment effects according to patient subgroups, how an intervention is delivered, or how an outcome is measured||This is done implicitly by GWC members.||The current practice of rarely using differential treatment effects according to patient subgroups to inform a recommendation and only when prespecified, adequately powered, and supported by heterogeneity testing should remain.|
|2.6.11 Describe the proposed timetable for conducting the review||The proposed time within which the SR will be conducted in addition to the anticipated time for completion of the SR protocol development and review should be described.|
|2.7 Submit the protocol for peer review|
|2.7.1 Provide a public comment period for the protocol and publicly report on disposition of comments||The ACCF/AHA CPGs do not undergo a public comment period; however, a thorough peer review is conducted.||The current peer review process of ACCF/AHA CPGs should be expanded to include a formal peer assessment of the SR protocol. External stakeholders and patient representatives should be included in this expanded process. Response to selected comments, with changes made in protocol or whether no changes were made, may be publicly reported if not precluded by time constraints.|
|2.8 Make the final protocol publicly available, and add any amendments to the protocol in a timely manner||The ACCF/AHA CPGs are simultaneously published in the Journal of the American College of Cardiology and in Circulation.||The final SR protocol should be made publicly available and amendments to the protocol added in a timely manner.|
ACCF indicates American College of Cardiology Foundation; AHA, American Heart Association; COI, conflict of interest; CPG, clinical practice guideline; ERC, evidence review committee; GWC, guideline writing committees; IOM, Institute of Medicine; LBCT, late-breaking clinical trial; MeSH, medical subject heading; PICO(TS), mnemonic: population, intervention, comparator, outcomes, timing and setting; RCTs, randomized controlled trials; RWI, relationships with industry and other entities; SR, systematic review; and Task Force, ACCF/AHA Task Force on Practice Guidelines.
Workgroup 3 Comparison Table
|3. Standards for Finding and Assessing Individual Studies|
|IOM Standards and Elements||ACCF/AHA Current Methodology||Workgroup 3 Proposal|
|3.1 Conduct a comprehensive, systematic search for evidence||Two researchers are proficient in the SR process and performing literature searches. Three project managers are proficient in using PubMed. Researchers work closely with the GWC to perform literature searches with key search terms and obtain full-text copies of articles. However, some GWC members conduct their own literature searches. Data supplement tables are created for key studies.||The following are general recommendations for conducting SR:|
|3.1.1 Work with a librarian or other information specialist trained in performing SR to plan the search strategy||No librarian is on staff. Key questions in PICO(TS) format are not used; however, key search terms are used based on the CPG table of contents. GWC members often ask research staff to perform literature searches using a literature search form; however, some GWC members conduct their own literature searches.|
|3.1.2 Design the search strategy to address each key research question||GWC members determine the search strategy using the CPG table of contents, a spreadsheet of LBCTs, and literature used in the previous CPG. New CPGs or full revisions have no time limit on searches, whereas searches for focused updates are limited based on the time following the last full revision of the CPG.|
|3.1.3 Use an independent librarian or other information specialist to peer review the search strategy||GWC members may provide peer examination, input into search strategies, and key words.||Contracting with a librarian or information specialist on an as-needed or project basis to review search strategies, especially in the pilot phase, should be considered.|
|3.1.4 Search bibliographic databases||Databases used include MEDLINE, PubMed, clinicaltrials.gov, and the list of trials on the ACCF Cardiosource Web site.||The ACCF/AHA CPG methodology is currently compliant.|
|3.1.5 Search citation indexes||PubMed's “search for additional articles” and Google Scholar are both used.||Citation indexes (e.g., Scopus and Web of Science) should be searched whenever possible.|
|3.1.6 Search literature cited by eligible studies||See 3.1.5. Unpublished data are only considered if the data were 1) presented at a major scientific meeting, 2) not older than 2 years, 3) not used to support a recommendation, and 4) not used in CPG figures and tables (unless the data have important public health implications; this is reviewed by the Task Force on an-ad hoc basis). Additional references may be submitted to the ERC by the GWC.||Reference lists of eligible studies should be reviewed to identify additional relevant studies. Every effort should be made to review all clinically relevant literature. Derivative studies, secondary analyses, substudies, and confirmatory/refuting research reports are by definition relevant but provide a different LOE than the primary results of a properly conducted clinical trial. Additional references may be submitted to the ERC by the GWC.|
|3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question being addressed||The time period for searches is stated in the CPG introduction. Articles in press are followed to determine if they will be published within the CPG development period. Cardiovascular scientific meetings that use the English language are routinely monitored. ACCF/AHA staff research specialists currently conduct biannual reviews of emerging literature, abstracts, and presentations. Task Force and GWC members also identify important studies and emerging literature.|
|3.1.8 Search subject-specific databases if other databases are unlikely to provide all relevant evidence||See 3.1.4. for search subject-specific databases used.||Subject-specific databases should be reviewed as suggested by the ERC and the GWC.|
|3.1.9 Search regional bibliographic databases if other databases are unlikely to provide all relevant evidence||EMBASE is currently not used, but the department is obtaining authorization to use EMBASE, and staff will be trained in its use.|
|3.2 Take action to address potentially biased reporting of research results|
|3.2.1 Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies||Grey-literature databases searched include monitoring registries such as the NCDR, FDA Websites, LBCTs presented at scientific sessions, and, when appropriate, ongoing trials covered within guidelines. Unpublished abstracts <2 years old may be addressed/included. Online databases such as MEDLINE/PubMed, Cardiosource Clinical Trials Database, and the Cochrane library are also searched.|
|3.2.2 Invite researchers to clarify information related to study eligibility, study characteristics, and risk of bias||When necessary, GWC members work jointly with CPG staff to clarify study eligibility, study design, and/or study characteristics in order to refine literature searches. Primary study authors may be contacted about questions pertaining to the study.||If a large or important study is potentially downgraded by the absence of information about study eligibility, study characteristics, or control of bias, that study should be flagged and an inquiry sent to authors. The corresponding author would be the primary point person in this regard. Unfortunately, this has the potential for continuation of bias or introduction of additional bias, because the response of the corresponding author might neither be audited nor peer reviewed.|
|3.2.3 Invite all study sponsors and researchers to submit unpublished data, including unreported outcomes, for possible inclusion in the SR||If important unpublished data are identified, the ERC should request the data, document the request for additional information, and report on the disposition of the request in a publicly accessible appendix to the CPG.|
|3.2.4 Hand search selected journals and conference abstracts||Tables of contents of major medical journals (e.g., Journal of the American Medical Association, New England Journal of Medicine, The Lancet, Journal of the American College of Cardiology, and Circulation) are received weekly by guideline staff, and full-text articles pertinent to GWC members are sent electronically. Guideline staff maintains electronic copies of articles relevant to published CPG or CPG in development.||The ACCF/AHA CPG methodology is currently compliant.|
|3.2.5 Conduct a Web search||Web searches are conducted by GWC and CPG staff. Sites searched include PubMed, Cardiosource clinical trials, clinicaltrials.gov, , Google, and Google Scholar.||The ACCF/AHA CPG methodology is currently compliant.|
|3.2.6 Search for studies reported in languages other than English if appropriate||PubMed search features allow the inclusion of English-only journals. Each CPG introduction clarifies if the searches were limited to the English language. Articles published in a language other than English but translated into English may be used in the CPG reference list (e.g., articles in the Canadian Medical Association Journal often report trials in French and English). Non-English articles are identified by the primary language reported in parentheses as cited in the reference list.||The ACCF/AHA CPG methodology recommends inclusion of studies published simultaneously in English and other languages. However, studies published in a language other than English and that do not include an English version are neither translated nor included in the evidence reviews. It is recommended that the ERC or GWC be permitted to consider studies reported in languages other than English for consideration.|
|3.3 Screen and select studies|
|3.3.1 Include or exclude studies based on the protocol's prespecified criteria||Studies are included or excluded on the basis of GWC-determined prespecified criteria. The scope of the CPGs and review of LBCTs also influence which studies may be included or excluded, as well as studies pertinent to the CPGs.||The ACCF/AHA CPG methodology should be more explicit in identifying the prespecified criteria for inclusion or exclusion of studies. The use of clinical questions in a PICO(TS) format would facilitate this process.|
|3.3.2 Use observational studies in addition to RCT to evaluate harms of interventions||Literature searches focus on published articles only (i.e., RCT followed by observational studies, meta-analyses, and SR). The majority of literature searches focus on RCTs but also include nonrandomized studies (i.e., observational studies, case studies, and opinion documents).|
|3.3.3 Use two or more members of the review team, working independently, to screen and select studies||CPG sections may be drafted by GWC members who have expertise and in-depth knowledge of the subject or by a group of GWC members. As a result, relevant studies for each section may be screened independently by multiple GWC members. However, the screening and selection of studies is not explicitly done by multiple members.|
|3.3.4 Train screeners using written documentation; test and retest screeners to improve accuracy and consistency||GWC committee members are given an in-depth orientation to the ACCF/AHA Methodology Manual (4), as well as in-depth orientation about how to populate data summary tables.|
|3.3.5 Use one of two strategies to select studies: 1) read all full-text articles identified in the search, or 2) screen titles and abstracts of all articles and then read the full text of articles identified in initial screening||Research staff perform literature searches, and a document containing the study reference and abstract is emailed to the appropriate GWC member(s). Full-text copies of relevant articles are also identified by the research analyst and sent to GWC members. The final selection of relevant studies is the purview of the GWC member, so the actual strategy used is not explicit or mandated.|
|3.3.6 Taking account of the risk of bias, consider including observational studies to address gaps in the evidence from RCT on the benefits of interventions||All trials (e.g., RCT and observational trials) are examined by GWC members for possible inclusion in the evidence review.|
|3.4 Document the search|
|3.4.1 Provide a line-by-line description of the search strategy, including the date of every search for each database, web browser, etc.||Research staff maintains an electronic file of searches conducted by either staff or GWC members. A hard copy of the standard literature search form is also kept on file. Text describing literature search criteria and key search terms is included in the CPG introduction, thereby allowing CPG users access to the comprehensive literature search conducted.||The ACCF/AHA CPG methodology is currently compliant.|
|3.4.2 Document the disposition of each report identified including reasons for their exclusion if appropriate||Excluded studies or reports are not called out specifically within the CPG.|
|3.5 Manage data collection|
|3.5.1 At a minimum, use two or more researchers, working independently, to extract quantitative or other critical data from each study. For other types of data, one individual could extract the data while the second individual checks for accuracy and completeness. Establish a fair procedure for resolving discrepancies—do not simply give final decision-making power to the senior reviewer||Two members of the ERC should extract quantitative or other critical data from each study. A third individual should be involved in resolution of discrepancies. These 3 individuals should be members of the ERC, not the GWC. Their work should lead to coauthorship of the finished guideline. In some CPG, there may be more than one 3-person ERC team for review of evidence in different parts of the CPG, depending on the scope.|
|3.5.2 Link publications from the same study to avoid including data from the same study more than once||Multiple publications of the same study are identified by GWC members, thereby decreasing the possibility of repeating information or data.||The ACCF/AHA CPG methodology is currently compliant.|
|3.5.3 Use standard data extraction forms developed for the specific SR||Evidence tables with a uniform set of key characteristics identified by GWC members for each guideline are used to standardize data collection.|
|3.5.4 Pilot-test the data extraction forms and process||The format of data supplement tables undergoes multiple revisions before their adoption and use, but no formal pilot-testing is done.||The data extraction form should be piloted in a few studies and then revised.|
|3.6 Critically appraise each study|
|3.6.1 Systematically assess the risk of bias, using predefined criteria||GWC expertise is used to assess the risk of bias. The ACCF/AHA system for assessing bias is currently subjective and qualitative.|
|3.6.2 Assess relevance of the study's populations, interventions, and outcome measures||The ACCF/AHA system for assessing bias is currently subjective and qualitative. GWC members are relied on to implicitly assess the relevance of the study population, intervention, and outcome measures.||The relevance of each study's populations, interventions, comparators, outcomes, timing, and setting to the key clinical question/topic of interest should be assessed.|
|3.6.3 Assess the fidelity of the implementation of interventions||The GWC reviews the study interventions implicitly to ensure/assess the fidelity of the implementation of interventions.||For each study, an explicit assessment of the extent to which the intervention was delivered as planned and the quality of delivery of the intervention should be included.|
ACCF indicates American College of Cardiology Foundation; AHA, American Heart Association; AHRQ, Agency for Healthcare Research and Quality; CMS, Centers for Medicare and Medicaid Services; CPG, clinical practice guideline; EPC, evidence-based practice center; ERC, evidence review committee; ESC, European Society of Cardiology; FDA, U.S. Food and Drug Administration; GWC, guideline writing committee; HRS, Heart Rhythm Society; IOM, Institute of Medicine; LBCT, late-breaking clinical trial; LOE, level of evidence; NCBI, National Center for Biotechnology Information; NCDR, National Cardiovascular Data Registry; PICO(TS), mnemonic: population, intervention, comparator, outcomes, timing, and setting; RCT, randomized controlled trial; SR, systematic review; and Task Force, ACCF/AHA Task Force on Practice Guidelines.
Workgroup 4 Comparison Table
|4. Standards for Synthesizing the Body of Evidence|
|IOM Standards and Elements||ACCF/AHA Current Methodology||Workgroup 4 Proposal|
|4.1 Use a prespecified method to evaluate the body of evidence|
|Currently, the ACCF/AHA does not systematically look at these characteristics. However, the ACCF/AHA does implicitly judge consistency, precision, and directness when using the COR/LOE to write recommendations.||The Workgroup proposes that these IOM-recommended basic characteristics of quality and elements be accepted for assessing and describing quality across studies.|
|A systematic assessment of these characteristics is performed on a case basis, depending on the study analyzed.|
|4.1.3 For each outcome specified in the protocol, use consistent language to characterize the level of confidence in the estimates of the effect of an intervention||By using the COR/LOE, the ACCF/AHA uses consistent language to characterize the level of confidence in the estimates of the effect of an intervention.||See above. The IOM standard should be followed.|
|4.2 Conduct a qualitative synthesis|
|4.2.1 Describe the clinical and methodological characteristics of the included studies, including their size, inclusion or exclusion of important subgroups, timeliness, and other relevant factors||This is currently presented in the form of evidence tables but may be presented in the text as well.|
|4.2.2 Describe the strengths and limitations of individual studies and patterns across studies||This is currently presented in the form of evidence tables but may be presented in the text as well.||The IOM standard should be considered.|
|4.2.3 Describe, in plain terms, how flaws in the design or execution of the study (or groups of studies) could bias the results, explaining the reasoning behind these judgments||This is currently presented in the form of evidence tables but may be presented in the text as well.||The IOM standard should be considered.|
|4.2.4 Describe the relationships between the characteristics of the individual studies and their reported findings and patterns across studies||This is currently presented in the form of evidence tables but may be presented in the text as well.||The IOM standard should be considered.|
|4.2.5 Discuss the relevance of individual studies to the populations, comparisons, cointerventions, settings, and outcomes or measures of interest||This is currently presented in the form of evidence tables but may be presented in the text as well.||The IOM standard should be considered.|
|4.3 Decide if, in addition to a qualitative analysis, the SR will include a quantitative analysis (meta-analysis)|
|4.3.1 Explain why a pooled estimate might be useful to decision makers||Reporting meta-analyses for the subject is encouraged. Limitations regarding meta-analyses are explained in the text.||This may be described in text. See above.|
|4.4 If conducting a meta-analysis, then do the following:|
|4.4.1 Use expert methodologists to develop, execute, and peer review the meta-analyses||Meta-analyses are not performed by GWC members, but published meta-analyses are reported. Meta-analyses used must be published in peer-reviewed journals, and any limitations of the meta-analysis are explained.|
|4.4.2 Address the heterogeneity among study effects||The ACCF/AHA does not explicitly do this but implicitly judges heterogeneity among study effects, which determines the need to downgrade recommendations.||The IOM standard should be considered.|
|4.4.3 Accompany all estimates with measures of statistical uncertainty||This is not currently performed.||The IOM standard should be considered.|
|4.4.4 Assess the sensitivity of conclusions to changes in the protocol, assumptions, and study selection (sensitivity analysis)||This is not currently performed.||The IOM standard should be considered.|
|The ACCF/AHA currently uses COR (I, IIa, IIb, III)/LOE (A, B, C), whereas other popular systems differ.|
ACCF indicates American College of Cardiology Foundation; AHA, American Heart Association; COR, class of recommendation; CPG, clinical practice guideline; EPC, evidence-based practice center; GRADE, Grading of Recommendations Assessment, Development and Evaluation; GWC, guideline writing committee; IOM, Institute of Medicine; LOE, level of evidence; NHLBI, National Heart, Lung, and Blood Institute; PICO(TS), mnemonic: population, intervention, comparator, outcomes, timing and setting; QOE, quality of evidence; SR, systematic review; and Task Force, ACCF/AHA Task Force on Practice Guidelines.
Workgroup 5 Comparison Table
|5. Standards for Reporting Systematic Reviews|
|IOM Standards and Elements||ACCF/AHA Current Methodology||Workgroup 5 Proposal|
|5.1 Prepare final report using a structured format|
|5.1.1 Include a report title||Each CPG is given a title; if the CPG is revised or updated, the title usually follows the title of the previous version. However, ACCF/AHA CPGs are not generally based on formal SR.||Each published SR should have a title that reflects the CPG for which the evidence is provided. If the ACCF/AHA conducts an SR that is not published independently, nevertheless, it will have a carefully constructed research question that can be noted on the tables summarizing the findings of the SR.|
|5.1.2 Include an abstract||All CPGs have an executive summary that is published with the CPG and available online. No abstract is created by GWC members.||The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA publishes an SR as a stand-alone document, relevant IOM Standards and Elements should be used.|
|5.1.3 Include an executive summary||All CPGs have an executive summary that is published with the CPG and available online.||The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA publishes an SR as a stand-alone document, relevant IOM Standards and Elements should be used.|
|5.1.4 Include a summary when written for the lay public||All ACCF/AHA CPGs and derivative products are produced mainly for cardiologists as well as other medical practitioners. As a result, no summary is created for the lay public. CPGs are intended for a professional audience and for that reason are very technical. Creation of a summary for the lay public could be challenging.||The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA publishes an SR as a stand-alone document, relevant IOM Standards and Elements should be used.|
|5.1.5 Include an introduction (rationale and objectives)||SRs are not generally conducted for ACCF/AHA CPGs. Therefore, 5.1.5 is not applicable with respect to the current ACCF/AHA development methodology. All ACCF/AHA CPGs have an introduction that includes details of the evidence review.||The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA publishes the SR as a stand-alone document, relevant IOM Standards and Elements should be used.|
|SRs are generally not conducted for ACCF/AHA CPGs. Therefore, 5.1.6 is not applicable for the current ACCF/AHA development methodology.||The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA publishes an SR as a stand-alone document, relevant IOM Standards and Elements should be used. See Workgroups 3 and 4.|
|SRs are generally not conducted for ACCF/AHA CPGs. Therefore, 5.1.7 is not applicable for the current ACCF/AHA development methodology.|
|SRs are generally not conducted for ACCF/AHA CPGs. Therefore, 5.1.8 is not applicable for the current ACCF/AHA development methodology.|
|5.1.9 Include a section describing funding sources and COI||SRs are generally not conducted for ACCF/AHA CPGs. Therefore, 5.1.9 is not applicable for the current ACCF/AHA development methodology. However, the ACCF/AHA does not accept outside funding for CPG and has a very robust RWI policy for GWC that is available on the Web.|
|5.2 Peer review the draft report|
|5.2.1 Use a third party to manage the peer review process||ACCF/AHA CPG peer review is managed by ACCF/AHA document staff.||The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA conducts its own SR and publishes it, it will be independently reviewed through the journal's review process.|
|5.2.2 Provide a public comment period for the report and publicly report on disposition of comments||Currently, while the ACCF/AHA CPGs undergo an extensive peer review process that includes scientific and clinical content experts in addition to partnering, collaborating, and other relevant professional societies. However, CPGs are not opened to public review and comment. See Workgroups 1 and 2.||The public is able to comment on CPGs through letters to the editor (postpublication). The ACCF/AHA has no control over independently conducted SR, but when the ACCF/AHA conducts its own SR, external stakeholders and patient representatives should be included in this expanded process, and the SR protocol should be open to public comment.|
|5.3 Publish the final report in a manner that ensures free public access||SRs commissioned by the ACCF and AHA will be hyperlinked from the CPG and made available through free public access. SRs performed by other groups and used as part of the CPG process may or may not be available for free public access.|
ACCF indicates American College of Cardiology Foundation; AHA, American Heart Association; COI, conflict of interest; CPG, clinical practice guideline; GWC, guideline writing committee; IOM, Institute of Medicine; RWI, relationships with industry and other entities; and SR, systematic review.
The findings and conclusions in this report are those of the Summit Participants and do not necessarily reflect the official position of the American College of Cardiology Foundation and American Heart Association.
The American College of Cardiology Foundation requests that this document be cited as follows: Jacobs AK, Kushner FG, Ettinger SM, Guyton RA, Anderson JL, Ohman EM, Albert NM, Antman EM, Arnett DK, Bertolet M, Bhatt DL, Brindis RG, Creager MA, DeMets DL, Dickersin K, Fonarow GC, Gibbons RJ, Halperin JL, Hochman JS, Koster MA, Normand S-LT, Ortiz E, Peterson ED, Roach WH Jr., Sacco RL, Smith SC Jr., Stevenson WG, Tomaselli GF, Yancy CW, Zoghbi WA. ACCF/AHA clinical practice guideline methodology summit report: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2013;61:213–65. doi:10.1016/j.jacc.2012.09.025
This article is copublished in Circulation.
Copies: This document is available on the World Wide Web sites of the American College of Cardiology (http://www.cardiosource.org) and the American Heart Association (my.americanheart.org). For copies of this document, please contact the Elsevier Inc. Reprint Department, fax (212) 633-3820, e-mail .
Permissions: Multiple copies, modification, alteration, enhancement, and/or distribution of this document are not permitted without the express permission of the American College of Cardiology Foundation. Please contact.
- American College of Cardiology Foundation and the American Heart Association, Inc.
- Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Institute of Medicine
- Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, Institute of Medicine
- Frye R.L.,
- Collins J.J.,
- DeSanctis R.W.,
- et al.
- ACCF/AHA Task Force on Practice Guidelines
- van Wersch A.,
- Eccles M.
- Bathgate T.,
- Romios P.
- Boivin A.,
- Currie K.,
- Fervers B.,
- et al.
- U.S. Cochrane Center and Consumers United for Evidence-Based Medicine (CUE)
- Higgins J.P.,
- Altman D.G.,
- Gotzsche P.C.,
- et al.
- Brozek J.,
- Oxman A.,
- Schunemann H.
- U.S. Preventive Services Task Force Grade Definitions. 2008 http://www.uspreventiveservicestaskforce.org/uspstf/grades.htm. Accessed May 2, 2012.
- Smith S.C.
- Gibbons R.J.,
- Chatterjee K.,
- Daley J.,
- et al.
- ACCF/AHA Task Force Members
- Table of Contents
- Executive Summary
- Abbreviation List
- Full Report
- 1 Workgroup 1: Clinical Practice Guidelines We Can Trust
- 2 Workgroup 2: Standards for Initiating a Systematic Review
- 3 Workgroup 3: Standards for Finding and Assessing Individual Studies
- 4 Workgroup 4: Standards for Synthesizing the Body of Evidence
- 5 Workgroup 5: Standards for Reporting Systematic Reviews
- Presidents and Staff
- Appendix 1 ACCF/AHA Clinical Practice Guideline Methodology Summit Report—Author Listing of Comprehensive Relationships With Industry and Others (December 2011)
- Appendix 2 Workgroup Comparison Tables