Author + information
- Received July 20, 2011
- Revision received December 14, 2011
- Accepted December 15, 2011
- Published online May 22, 2012.
- Sumeet Subherwal, MD, MBA⁎,⁎ (, )
- Eric D. Peterson, MD, MPH⁎,
- David Dai, PhD, MS⁎,
- Laine Thomas, PhD⁎,
- John C. Messenger, MD†,
- Ying Xian, MD, PhD⁎,
- Ralph G. Brindis, MD‡,
- Dmitriy N. Feldman, MD§,
- Shaun Senter, MD∥,
- Lloyd W. Klein, MD∥,
- Steven P. Marso, MD¶,
- Matthew T. Roe, MD, MHS⁎ and
- Sunil V. Rao, MD⁎
- ↵⁎Reprint requests and correspondence:
Dr. Sumeet Subherwal, Duke University Medical Center, PO Box 17969, 2400 Pratt Street, Room 0311 Terrace Level, Durham, North Carolina 27705
Objectives The purpose of this study was to examine temporal trends in post-percutaneous coronary intervention (PCI) bleeding among patients with elective PCI, unstable angina (UA)/non–ST-segment elevation myocardial infarction (NSTEMI), and ST-segment elevation myocardial infarction (STEMI).
Background The impact of bleeding avoidance strategies on post-PCI bleeding rates over time is unknown.
Methods Using the CathPCI Registry, we examined temporal trends in post-PCI bleeding from 2005 to 2009 among patients with elective PCI (n = 599,524), UA/NSTEMI (n = 836,103), and STEMI (n = 267,632). We quantified the linear time trend in bleeding using 3 sequential logistic regression models: 1) clinical factors; 2) clinical + vascular access strategies (femoral vs. radial, use of closure devices); and 3) clinical, vascular strategies + antithrombotic treatments (anticoagulant ± glycoprotein IIb/IIIa inhibitor [GPI]). Changes in the odds ratio for time trend in bleeding were compared using bootstrapping and converted to risk ratio.
Results An approximate 20% reduction in post-PCI bleeding was seen (elective PCI: 1.4% to 1.1%; UA/NSTEMI: 2.3% to 1.8; STEMI: 4.9% to 4.5%). Radial approach remained low (<3%), and closure device use increased marginally from 44% to 49%. Bivalirudin use increased (17% to 30%), whereas any heparin + GPI decreased (41% to 28%). There was a significant 6% to 8% per year reduction in annual bleeding risk in UA/NSTEMI and elective PCI, but not in STEMI. Antithrombotic strategies were associated with roughly half of the reduction in annual bleeding risk: change in risk ratio from 7.5% to 4% for elective PCI, and 5.7% to 2.8% for UA/NSTEMI (both p <0.001).
Conclusions The nearly 20% reduction in post-PCI bleeding over time was largely due to temporal changes in antithrombotic strategies. Further reductions in bleeding complications may be possible as bleeding avoidance strategies evolve, especially in STEMI.
Bleeding complications after percutaneous coronary intervention (PCI) are associated with increased morbidity and mortality (1–5). The use of bleeding avoidance strategies, including safer approaches for vascular access and hemostasis, and selection of antithrombotic strategies with improved safety profiles, may reduce bleeding risk. Studies have demonstrated that radial artery access for catheterization is associated with fewer bleeding complications (6,7). Vascular closure devices have the potential to decrease bleeding complications in a select population, although further trials are required to definitively test their benefit (8–11). Additionally, randomized trials have demonstrated that an anticoagulation strategy, such as the use of bivalirudin, in elective PCI, invasive treatment of unstable angina (UA)/non–ST-segment elevation myocardial infarction (NSTEMI), and primary PCI for ST-segment elevation myocardial infarction (STEMI), has the potential to reduce bleeding complications with similar efficacy (12–14).
Despite the availability of this evidence, it remains unclear to which extent these bleeding avoidance strategies have been implemented in clinical practice, and subsequently led to measureable reductions in post-PCI bleeding. Although a pooled analysis of 4 different Canadian registries of NSTEMI patients demonstrated that bleeding rates did not change from 1999 to 2008 (15), data from the overall NSTEMI and STEMI population in the National Cardiovascular Data Registry CathPCI Registry suggested a modest 19% temporal reduction in post-PCI bleeding from 2.49% in 2005 to 2.01% in 2009 (16). To date, no study has investigated whether temporal changes in bleeding avoidance strategies would help explain changes in bleeding rates.
Therefore, using the CathPCI Registry, we examined changes in post-PCI bleeding from 2005 to 2009 among 3 groups: 1) STEMI; 2) UA/NSTEMI; and 3) elective PCI, to determine whether or not the temporal reduction in post-PCI bleeding was significant. More importantly, we sought to examine the degree to which any temporal reduction in post-PCI bleeding was associated with temporal changes in bleeding avoidance strategies (vascular access strategies [radial artery catheterization and/or use of vascular closure devices] and/or choice of antithrombotic strategy).
The CathPCI Registry, a partnership between the American College of Cardiology and the Society for Cardiovascular Angiography and Interventions, is a national quality improvement program that includes in-hospital data on patients undergoing cardiac catheterization and PCI procedures, and has been previously described (17,18).
The analysis utilized version 3 of the CathPCI Registry, which contained data from 1,708,449 patient admissions for PCI from January 1, 2005, to June 30, 2009 at 1,031 sites. Patients who died on the day of their PCI (n = 5,125) or those with missing data on bleeding (n = 65) were excluded. Only index PCI procedures performed during the study were included in this analysis.
Full description of the data elements in version 3 of the CathPCI Registry (19). Major bleeding was defined as: 1) bleeding requiring a blood transfusion or prolonged hospitalization; 2) a decrease in hemoglobin >3.0 g/dl from any location, including percutaneous entry site, retroperitoneal, gastrointestinal, genitourinary, and other/unknown location; or 3) percutaneous entry site hematoma >10 cm for femoral access, >2 cm for radial access, or >5 cm for brachial access.
Estimated glomerular filtration rate was calculated using the abbreviated modification of diet in renal disease formula (20). The CathPCI Bleeding Model (2) includes the following variables: acute coronary syndrome type (STEMI, UA/NSTEMI), cardiogenic shock, female gender, previous congestive heart failure (CHF), no previous PCI, New York Heart Association functional class IV CHF, peripheral vascular disease, age, and glomerular filtration rate. Access site bleeding was defined as any bleeding recorded as bleeding at percutaneous entry site or retroperitoneal bleeding if the patient had femoral artery catheterization or use of an intra-aortic balloon pump. Nonaccess site bleeding was defined as gastrointestinal bleeding, genital–urinary bleeding, bleeding for any other cause, or spontaneous retroperitoneal bleeding (not related to catheterization or balloon pump).
Abbreviated statistical analysis
Given that patients' baseline risk and procedurally associated risk of bleeding differed according to indication for PCI, patients were initially stratified into 3 groups by indication for PCI: 1) elective PCI; 2) UA/NSTEMI; and 3) STEMI. Changes in patient and procedural characteristics, and bleeding outcomes for each subgroup were evaluated annually (from 2005 until the second quarter of 2009). Continuous variables were described as medians with 25th and 75th percentiles, whereas categorical variables were expressed as frequencies with percentages. The Kruskal-Wallis test was used to determine whether there was any difference in continuous or ordinal variables, whereas the Pearson chi-square tests were used to determine any difference in categorical variables.
The primary endpoint of interest was major bleeding. To determine whether the decreasing trend in bleeding over time was significant, we added a linear time trend covariate to the previously published CathPCI Bleeding Model (2) for each of the 3 subgroups: Model 1, time (per unit year) + covariates in the CathPCI Bleeding Model.
We then added additional covariates to this model to address the hypothesis that specific changes in practice contributed to the temporal change in bleeding rates. The p values for the comparison of adjusted odds ratios (ORs) for time between models were obtained by bootstrapping. Specifically, Model 2 is Model 1 + vascular access strategies. With respect to vascular access strategies, the data collection form captured whether catheterization was performed by radial artery access, femoral artery access, or if closure devices were used. Sheath size, timing of sheath removal, or duration of manual compression (which could potentially influence bleeding rates) were not collected. To the extent that vascular access strategies explain the temporal reduction in bleeding, we expected the adjusted OR for time to attenuate toward 1 in Model 2 compared with Model 1.
Similarly, Model 3 is Model 2 + anticoagulation strategy. Anticoagulation strategies were limited to 4 different strategies: 1) heparin only; 2) bivalirudin only; 3) glycoprotein IIb/IIIa inhibitor (GPI) plus heparin; and 4) bivalirudin plus GPI. For the STEMI population, lytics were included in the covariates of Model 3, because lytics are known to influence the risk of bleeding (21–23). Bivalirudin was the only direct thrombin inhibitor studied because it was the predominant direct thrombin inhibitor used in >99% of the cases. Fondaparinux was not included as an anticoagulation strategy, because it was used in <1% of the cases. Similar to previously described techniques, controlling for anticoagulation strategies would increase the OR for time toward 1 if anticoagulation strategies meaningfully explained the trend in bleeding rates.
We illustrated the differences between the subsequent models in terms of changes in annual bleeding events. Using the 2005 data as a reference population (observed bleeds: n = 1,375 bleeds in elective PCI; n = 3,327 in UA/NSTEMI; n = 1,978 in STEMI), we calculated the difference in predicted (expected) bleeding events obtained by adding 1 year (time) to the 2005 patient cohort. For each model, this provided the expected annual change, for a fixed set of covariates. Given the complexity of the analysis, the Online Appendix discusses a more elaborated statistical plan.
For each of the 3 groups, the patient demographics and procedural characteristics during the first year (2005) and the most recent year of data collection (first 2 quarters of 2009) are shown in Table 1 (Note: The table in the Online Appendix demonstrates annual data during the study). Patients in the STEMI group were more likely to be male and have lower rates of previous myocardial infarction (MI), CHF, diabetes, previous PCI or coronary artery bypass grafting, and had higher use of an intra-aortic balloon pump compared with the other 2 groups. Thrombolytics were administered to 11.8% of the overall STEMI population, although its use decreased annually from 17.4% in 2005 to 8.1% by 2009 (Online Appendix).
Major bleeding over time
There was a modest ∼20% temporal reduction in post-PCI bleeding overall (from 2.4% in 2005 to 2.0% in 2009) and for each group (1.4% to 1.1% in the elective PCI group, 2.3% to 1.8% in the UA/NSTEMI group, and 4.9% to 4.5% in the STEMI group) (Fig. 1). In comparison of access site versus nonaccess site bleeding, there was a significant temporal reduction in access site bleeding among all 3 groups (p <0.001) (Fig. 1). Meanwhile, the annual rate of nonaccess site bleeding remained relatively constant for elective PCI (p = 0.104) and UA/NSTEMI (p = 0.827), and increased for the STEMI group (from 2.6% to 3.1%) during the study (p < 0.001).
As demonstrated by Model 1 in Table 2, there was a significant 6% to 8% per year reduction in risk of annual bleeding among the elective PCI and UA/NSTEMI groups after adjusting for clinical characteristics associated with bleeding (i.e., covariates in the CathPCI Bleeding Model). The largest relative annual reduction in risk was observed for the elective PCI group (OR: 0.919; 95% confidence interval [CI]: 0.895 to 0.945). Although there was a trend, there was not a significant temporal reduction in annual bleeding risk among the STEMI group (p = 0.088).
Temporal changes in vascular access strategies and anticoagulation strategies
As seen in Figure 2, radial approach to catheterization was infrequent (<1.5% overall). Vascular closure device use was more common and increased slightly during the study by a relative 10% to 13% among all 3 groups (Fig. 3).
The changes in antithrombotic strategies over time were much greater than that of vascular strategies (Fig. 4); overall, bivalirudin use increased from 17% to 30%, whereas heparin + GPI decreased from 41% to 28% during the study. This was largely driven by the adoption of bivalirudin only antithrombotic strategy in the elective PCI and UA/NSTEMI groups, given that its use remained relatively low (<10%) in the STEMI group.
Association of changes in bleeding avoidance strategies and outcomes: changes in vascular access strategies
Model 2 was developed to assess whether the temporal changes in vascular strategies could partly explain the temporal reduction in annual bleeding events during the study. As noted in Figure 5, after accounting for the vascular strategies (Model 2), there was a small, but significant, increase in the OR for time trend of annual bleeding toward the null (i.e., 1) when comparing the adjusted OR of annual bleeding estimated by Model 1 and Model 2 for the elective PCI subgroup (OR: 0.920 to 0.924; p < 0.001) and for the UA/NSTEMI subgroup (OR: 0.939 to 0.942; p = 0.015) after accounting for vascular strategies (STEMI was not reported because there was no temporal reduction in bleeding for STEMI). This <1% change in relative risk of annual bleeding suggests that the temporal increase in vascular access strategies (either radial artery access or use of vascular closure devices) was associated minimally with the temporal reduction in annual major bleeding rates.
Association of changes in bleeding avoidance strategies and outcomes: changes in antithrombotic strategies
Model 3 (Fig. 5) investigated the impact of the 4 different anticoagulation strategies on the adjusted odds of in-hospital major bleeding each year by comparing the adjusted OR between Model 2 and Model 3. The odds of in-hospital bleeding each year increased from 0.924 to 0.959 in the elective PCI group (p < 0.001) after adjusting for antithrombotic use, translating to a reduction from 7.5% to 4% annual relative risk of bleeding. For UA/NSTEMI, the OR changed from 0.942 to 0.971 (p < 0.001), translating to a reduction from 5.7% to 2.8% annual relative risk of bleeding. This suggests that either the increase in bivalirudin use, the decrease in the use of heparin + GPI, or the combination of the 2 are associated with a larger portion of the temporal reduction in post-PCI bleeding events. Consequently, anticoagulation strategies (Model 3) are associated with roughly 51% of the annual reduction in risk of annual bleeding in UA/NSTEMI and roughly 47% reduction in elective PCI.
The present analysis of the CathPCI Registry found a modest, but significant, decrease in the annual rate and risk of in-hospital bleeding from 2005 to 2009 among elective PCI and UA/NSTEMI patients who underwent PCI; there was a temporal reduction in access site bleeding, suggesting that this might have driven the reduction in overall bleeding. We found that vascular access strategies did not change much over this period, and thus, were minimally associated with these observed reductions in bleeding rates over time. In contrast, there was a marked temporal change in concomitant antithrombotic strategy use during the study, which appeared to be associated with half of the reduction in risk of annual bleeding among elective PCI and UA/NSTEMI.
Temporal trend in bleeding rates
In contrast to the study by Elbarouni et al. (15), we found a significant temporal reduction in bleeding rates. The contrasting conclusions might be attributable to differences in treatment strategies, different populations, study period, bleeding definition, and patient sample size. Approximately half of the patients in the Elbarouni et al. (15) analysis underwent cardiac catheterization, and roughly half of these underwent subsequent PCI. Meanwhile, our analysis was limited to PCI patients only. Additionally, one-third of their data was collected before 2003, during a time when GPI use was high (23). Because the use of GPI is associated with higher bleeding than heparin or bivalirudin alone, it is possible that high rates of GPI use could help explain why these investigators might not have noticed a reduction in bleeding over time. Additionally, because our analysis included a later time period, we were able to capture greater use of direct thrombin inhibitors; as shown in the present analysis, greater use of bivalirudin might explain some of the reduction in annual bleeding events. Other potential explanations of the reduction in annual bleeding events include improvements in reducing excess dosing during the study (24) and greater attention to femoral access (25).
In our analysis, the largest relative reduction in annual bleeding risks was noted to be in the elective PCI subgroup. We did not find a significant temporal reduction in the STEMI population, which might be because patients with STEMI received more aggressive anticoagulation with thrombolytics (11.8% of STEMI population) and had greater use of intra-aortic balloon pumps (10% of STEMI population), both of which are associated with increased bleeding risk (26,27). The rate of bleeding in the STEMI subgroup was more than twice that of the UA/NSTEMI group and nearly 4 times that of the elective PCI group. Potentially, increased use of bivalirudin might have reduced bleeding rates in the STEMI group because previous data showed improved bleeding outcomes in STEMI with use of bivalirudin (14). Similarly, patients with STEMI had lower rates of radial catheterization and closure device use, which were associated (though minimally) with the temporal reduction in bleeding in the elective PCI and UA/NSTEMI groups. The increase in nonaccess site bleeding seen with the STEMI group might have offset the benefit in access site bleeding reduction over time. Further analysis to determine why nonaccess site bleeding in the STEMI group trended up during the study is required, and a concerted effort to develop and implement novel bleeding avoidance strategies in the STEMI population is necessary. Greater adoption of the radial approach to catheterization in STEMI has the potential to reduce vascular complications and potentially ischemic vascular events as seen in the recent RIVAL (Radial Versus Femoral Access for Coronary Intervention) trial (28).
Vascular access strategies
The present study also found a small association between vascular access strategies to the temporal reduction in bleeding. We may not have seen a greater influence of the arterial access site given the infrequent use of radial catheterization; in the CathPCI database, <1.5% of the overall population had their PCI via radial access. It remains to be seen if greater adoption of transradial PCI in the United States alongside use of antithrombotics with improved safety profiles further affects the temporal trends in bleeding rates, especially in the STEMI population.
Given that the use of radial artery access was infrequent and did not change much over time, it is possible that the relative 10% to 13% increase in closure device utilization during the study accounted for the small, but significant, influence of vascular access strategies on reduction in annual bleeding risk among the elective PCI and UA/NSTEMI groups. The use of closure devices was recently shown to be associated with lower bleeding in the CathPCI Registry population (10) and the UA/NSTEMI population in the ACUITY (Acute Catheterization and Urgent Intervention Triage Strategy) trial (9). Of note, a meta-analysis suggested that there might be no difference in bleeding comparing closure devices with manual compression (29). Additionally, greater attention by operators to minimize femoral access complications (e.g., fluoroscopic guided puncture) might have attenuated the impact of vascular access strategies. Consequently, further trials looking at the use of closure devices and other femoral access avoidance strategies to reduce periprocedural bleeding are needed.
As seen in Figure 4, the rates of bivalirudin use increased linearly alongside a linear decrease in the heparin + GPI strategy among the elective PCI and UA/NSTEMI populations; however, the use of bivalirudin remained infrequent among the STEMI population by the end of the study (<10% use of bivalirudin alone). The 40% to 50% reduction in the risk of bleeding each year after controlling for these antithrombotic strategies suggests that either a temporal decrease in heparin + GPI use and/or increase in bivalirudin use was associated with the temporal reduction in bleeding rates. It remains to be seen if a greater adoption of a bivalirudin strategy will result in further improvements in bleeding outcomes, especially among the STEMI population, in which a randomized trial demonstrated similar efficacy with reduction in bleeding using a bivalirudin strategy (14).
Moreover, it must be emphasized that changes in anticoagulation strategy only partly explain the decreasing trend in bleeding. Unmeasured variables such as clinician bias in selecting patients for intervention with lower complication risk, bias in choice of vascular access strategy, bias in selection of antithrombotic choice, improvements in antithrombotic dosing, and greater physician awareness of bleeding risk may also contribute to this decreasing temporal trend of bleeding. Further analysis of other populations is necessary to duplicate these findings and investigate whether changes in antithrombotic strategy contribute to reduction in access site bleeding. Further investigation is needed on minimizing nonaccess site bleeding.
First, the definition of bleeding used was designed for a quality improvement registry. As a result, bleeding was site-reported and not adjudicated, likely leading to an underestimate of actual bleeding rates. A more comprehensive bleeding definition may have demonstrated a greater annual reduction in bleeding. Second, it is possible that patients with elective PCI may have been discharged early, subsequently, underestimating in-hospital bleeding rates. Similarly, length of stay may have influenced an underestimation of in-hospital bleeding rates. Additionally, the present analysis assumes that the difference between Model 2 and Model 3 in each group is attributed to only differences in bleeding due to antithrombotic strategies; it assumes that all comorbidities that may influence choice of medication have been captured in the regression model. As with any retrospective analysis, there is selection bias in the use of specific therapies; therefore, even with our robust statistical methods, there are likely unmeasured confounders. The dataset version used for the present analysis did not capture the use of prasugrel. Future analysis to determine the influence of prasugrel on bleeding risk in a “real-world” population is needed. Finally, we could not account for the size of sheaths used, timing of sheath removal, or duration of manual compression—all of which have the potential to also reduce procedural bleeding. We could not determine the effect of excess dosing of anticoagulants, nor antiplatelets, on bleeding outcomes with this registry.
To partially address these limitations, we accounted for clustering of events (i.e., patients who underwent PCI at specific centers were likely to be treated similarly) in our analysis. Although the analysis demonstrated that the changes in antithrombotic strategy over time helped explain the temporal reduction in bleeding events, we were unable to determine which specific change in antithrombotic therapy (i.e., bivalirudin increase or heparin + GPI decrease) accounted for the change in bleeding events.
The present analysis is the first paper to demonstrate a significant temporal reduction in major bleeding over time in the elective PCI and UA/NSTEMI population. This temporal reduction was largely associated with changes in procedural antithrombotic strategy—either an increase in bivalirudin use or a decrease in heparin + GPI. Meanwhile, changes in vascular access strategies had a smaller impact on the change in bleeding risk, likely due to the extremely low prevalence of the radial approach and a small increase in the use of vascular closure device utilization during the study. Temporal bleeding in STEMI did not change significantly. It remains to be seen whether further changes in vascular strategies—particularly increased adoption of transradial PCI, newer antithrombotics, or both—will further reduce bleeding rates, particularly in the STEMI population. Future studies should continue to evaluate bleeding risk, as the landscape of antithrombotic therapies and use of bleeding avoidance strategies continues to evolve.
The authors thank Erin LoFrese for her editorial contributions to this paper.
For the elaborated statistical analysis, please see the online version of this article.
Temporal Trends in and Factors Associated With Bleeding Complications Among Patients Undergoing Percutaneous Coronary Intervention: A Report from the National Cardiovascular Data CathPCI Registry
The CathPCI Registry is an initiative of the American College of Cardiology Foundation and the Society for Cardiovascular Angiography and Interventions. The study was conducted under an unrestricted grant from The Medicines Company. The study design, analysis, and abstract and manuscript writing were independent of sponsors. Dr. Peterson has received research grants from Bristol-Myers Squibb/Sanofi-Aventis, Merck, Eli Lilly, Johnson and Johnson, and the American Heart Association. Dr. Feldman has consulted for Maquet and has received speaker's fees from Eli Lilly, Daiichi Sankyo, Abbott Vascular, and The Medicines Company. Dr. Marso has consulted for The Medicines Company, Novo Nordisk, Abbott Vascular, Amylin Pharmaceuticals, Boston Scientific, Volcano Corporation, and Terumo Medical. Dr. Roe has received research grants from AstraZeneca, Bristol-Myers Squibb, Eli Lilly and Company, GlaxoSmithKline, KAI Pharmaceuticals, Novartis, sanofi-aventis, Schering-Plough Corporation, and Orexigen. Dr. Rao has consulted for The Medicines Company, AstraZeneca, BMS/Sanofi-Aventis joint venture, Terumo Medical, and Daiichi Sankyo Lilly; has taken part in the Speaker's Bureau for The Medicines Company and BMS/Sanofi joint venture; and has received research funding from the Ikaria, Cordis Corporation, and sanofi-aventis. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- congestive heart failure
- glycoprotein IIb/IIIa inhibitor
- non–ST-segment elevation myocardial infarction
- odds ratio
- percutaneous coronary intervention
- ST-segment elevation myocardial infarction
- unstable angina
- Received July 20, 2011.
- Revision received December 14, 2011.
- Accepted December 15, 2011.
- American College of Cardiology Foundation
- Eikelboom J.W.,
- Mehta S.R.,
- Anand S.S.,
- Xie C.,
- Fox K.A.,
- Yusuf S.
- Mehta S.K.,
- Frutkin A.D.,
- Lindsey J.B.,
- et al.
- Rao S.V.,
- Eikelboom J.A.,
- Granger C.B.,
- Harrington R.A.,
- Califf R.M.,
- Bassand J.P.
- Pristipino C.,
- Trani C.,
- Nazzaro M.S.,
- et al.
- Sanborn T.A.,
- Ebrahimi R.,
- Manoukian S.V.,
- et al.
- Roe M.T.,
- Messenger J.C.,
- Weintraub W.S.,
- et al.
- Brindis R.G.,
- Fitzgerald S.,
- Anderson H.V.,
- Shaw R.E.,
- Weintraub W.S.,
- Williams J.F.
- National Cardiovascular Data Registry
- Mudrick D.W.,
- Chen A.Y.,
- Roe M.T.,
- et al.
- Moscucci M.,
- Fox K.A.,
- Cannon C.P.,
- et al.
- Spencer F.A.,
- Moscucci M.,
- Granger C.B.,
- et al.