Author + information
- Received November 16, 2017
- Revision received February 1, 2018
- Accepted February 2, 2018
- Published online April 9, 2018.
- aDepartment of Cardiology, Icahn School of Medicine at Mount Sinai University, New York, New York
- bWest Virginia University Heart and Vascular Institute, West Virginia University, Morgantown, West Virginia
- cM&H Research, LLC, San Antonio, Texas
- ↵∗Address for correspondence:
Dr. Partho P. Sengupta, Heart and Vascular Institute, West Virginia University, 1 Medical Center Drive, Morgantown, West Virginia 26506-8059.
Background Myocardial relaxation is impaired in almost all cases with left ventricular diastolic dysfunction (LVDD) and is a strong predictor of cardiovascular and all-cause mortality.
Objectives This study investigated the feasibility of signal-processed surface electrocardiography (spECG) as a diagnostic tool for predicting the presence of abnormal cardiac muscle relaxation.
Methods A total of 188 outpatients referred for coronary computed tomography (CT) angiography underwent an echocardiogram for assessment of LVDD. The use of 12-lead spECG for predicting myocardial relaxation abnormalities as identified using tissue Doppler echocardiography was validated with machine-learning approaches.
Results A total of 188 subjects underwent diagnostic testing, with 133 (70%) showing abnormal myocardial relaxation on tissue Doppler imaging. A 12-lead spECG showed an area under the curve of 91% (95% confidence interval: 86% to 95%) for prediction of abnormal myocardial mechanical relaxation with a sensitivity and specificity of 80% and 84%, respectively. The spECG demonstrated more accurate diagnostic performance in individuals age ≥60 years as well as those with obesity or hypertension, compared with their respective counterparts. Prediction of low early diastolic relaxation velocity (e′) also correctly identified concomitant significant underlying coronary artery disease in 23 of 28 cases (82%). Furthermore, a superior integrated discrimination and net reclassification improvement was observed for spECG over clinical features and traditional ECG.
Conclusions The spECG provides a robust prediction of abnormal myocardial relaxation. These data suggest a potential role for spECG as a novel screening strategy for identifying patients at risk for LVDD who would benefit undergoing echocardiographic evaluations.
Left ventricular diastolic dysfunction (LVDD) appears early during any cardiovascular disease and is recognized in approximately 20% to 30% of the general adult population (1). Clinical studies demonstrate that early stages of LVDD can progress to heart failure and are predictive of all-cause mortality, even after controlling for other comorbidities (1,2). The current guidelines, therefore, recommend stratifying heart failure clinically into 4 stages (stages A to D) (3). Stage A refers to patients with risk factors with no structural or functional cardiac changes, whereas stage B includes patients with structural heart disease with no current or prior symptoms of heart failure. The largest proportion of patients with stage B that were evaluated in clinical trials have an ischemic origin or subclinical myocardial dysfunction related to conditions such as diabetes and hypertension. Nearly all conditions with associated structural and functional cardiac changes have concurrent impairment of LVDD that can be readily identified using echocardiography (4). However, echocardiography remains expensive, and routine clinical screening of asymptomatic stage A/B patients is not currently recommended (5).
The electrical and mechanical functions of cardiac performance are closely coupled. A positive T-wave that represents LV base-to-apex and transmural advancement of repolarization from the epicardium to the endocardium is vital for normal LV mechanical relaxation (6,7). Subtle changes in the myocardial electrical condition that may lead to myocardial relaxation abnormalities, however, are not readily discerned on the surface electrocardiogram (ECG); therefore, routine application of ECG as a diagnostic tool to assess LVDD is not commonly recognized. As electrical activity of the heart is highly dynamic, small changes in the surface ECG frequency spectrum are better discriminated by using signal-processing techniques, and unique patterns can be extracted using machine learning techniques (8–12). Such a development may have a substantial effect on early detection of LVDD. Therefore, we explored the role of a novel signal-processed surface ECG algorithm to extract electrophysiological signal patterns uniquely associated with abnormal myocardial relaxation.
We performed a prospective, cross-sectional study at the Icahn School of Medicine at Mount Sinai (New York, New York). We initially recruited 200 unselected consecutive subjects in sinus rhythm, who were referred from outpatient clinics for computed tomography (CT) coronary angiography. Subjects underwent 12-lead ECG, CT coronary angiography, and comprehensive 2-dimensional echocardiography (including tissue Doppler) in the same visit. Subjects with arrhythmias, unstable angina, previous cardiac surgery, a pacemaker, chest deformity, or an inability to express well-defined mitral annular velocities due to severe mitral annular calcifications were excluded. Of the 200 subjects enrolled, we excluded 4 due to inadequate echocardiographic image quality, 3 due to suboptimal electrocardiograms, and 5 others who were unable to undergo CT scans. The resulting 188 subjects included in the study had clinical, echocardiographic, CT, ECG, and transformed ECG data available. We also validated the normal ECG repolarization patterns with a comparison control cohort of young sex-matched individuals with no known cardiac illness who had a normal echocardiogram and ECG evaluation at the University of West Virginia (Morgantown, West Virginia). This comparison cohort was recruited from an ongoing prospective investigation that is evaluating the utility of signal-processed surface ECG in detecting LVDD in population with a high prevalence of cardiac risk factors. The institutional review board approved the study protocol, and all study participants provided written informed consent.
We collected and analyzed the following clinical characteristics of the study subjects: demographics, comorbidities, medications, body mass indexes, and laboratory data (including serum potassium and renal function).
Hypertension was defined by systolic blood pressure >140 mm Hg or diastolic blood pressure >90 mm Hg, physician-documented history of hypertension, or by the use of antihypertensive medications. Diabetes mellitus was defined by the presence of physician-documented history of diabetes or use of oral hypoglycemic agents or insulin for the treatment of hyperglycemia. Coronary artery disease (CAD) was defined by the presence of coronary stenosis of >50% on a coronary CT angiogram, history of myocardial infarction, or percutaneous intervention. Obesity was defined as a body mass indexes >30 kg/m2.
Signal-processed surface ECG
All subjects underwent baseline 12-lead surface ECG recording at the time of the baseline echocardiogram. We used the University of Glasgow Interpretive Analysis program (release 28.5, January 2014) for automated interpretation of ECGs. This program has been extensively evaluated and provides standard amplitude, duration, and axes measurements, as well as a rhythm analysis and diagnostic interpretation, and is well-suited for diagnostic studies (13–15). The quantitative description includes average heart rate; the P, Q, R, and S waves (QRS); T axes; P and QRS durations; PR and QT intervals; and corrected QT (QTc) intervals. In addition, for qualitative summary descriptions (normal, borderline, abnormal), the software used the quantitative description to automatically classify the electrocardiographic abnormalities according to the Minnesota Code Manual.
Signal processing was performed using continuous wavelet transform mathematics (MyoVista hsECG Informatics, HeartSciences, Southlake, Texas). Using wavelets for medical diagnostic purposes is a recent development, although the mathematical theory is not new (8,16–18). The principles are similar to those of Fourier analysis, which were developed in the early part of the 19th century. This processing converts an ECG signal into a normalized energy distribution in which frequency is shown on the y-axis (3 to 200 Hz), time on the x-axis, and a colored spectrum representing myocardial energy (colors scale 0 to 255), with blue indicating the lowest energy and red the highest energy (Figure 1). The continuous wavelet transform is time-aligned with the traditional ECG signal for calculating multiple indices. For example, at a pre-defined time period prior to the T-wave peak, the maximum value of the MyoVista Color Waveform (normalized, unsigned continuous wavelet transform) across all frequencies (scales) is selected. This value is termed the ventricular index early measure (VIEM). Similarly, at a pre-defined time period after the T-wave peak, the maximum value of the MyoVista Color Waveform across all frequencies is also selected. This value is termed the ventricular index late measure (VILM). The MyoVista energy algorithm called an Icon also classifies the patients into 3 categories of energies (taking into account factors such as sex and age) with grade 3 being the lowest and grade 1 referring to the highest energy levels. A list of different indices used in the analysis is available in the Online Supplementary Data File.
All subjects underwent a complete 2-dimensional Doppler echocardiogram and tissue Doppler echocardiographic examination using an iE33 system (Philips Medical Systems, Andover, Massachusetts), with additional dedicated imaging of mitral inflow using pulsed-wave Doppler echocardiography and pulsed-wave tissue Doppler echocardiography of the septal and lateral mitral annulus according to guidelines published by ASE (4). A trained single reader, blinded to the subject’s ECG, Glasgow Interpretive Analysis, and clinical data, reviewed the echocardiograms. Utilizing the apical 2- and 4-chamber loops, the LV end-diastolic volume, end-systolic volume, and ejection fraction were calculated using the biplane Simpson’s method of discs, and the left atrial (LA) maximum volume was calculated using the biplane area length method. All measurements were made in ≥3 consecutive cardiac cycles, and average values were used for final analyses. The pulsed-wave Doppler-derived transmitral velocity and digital color tissue Doppler-derived mitral annular velocities were obtained from the apical 4-chamber view. The early diastolic wave velocity (E) and late diastolic atrial contraction wave velocity (A) were measured using a pulsed-wave Doppler recording. Continuous-wave Doppler was applied on the tricuspid valve in different windows (apical 4-chamber view, parasternal right ventricular inflow view, and parasternal short-axis view). The tricuspid regurgitation signal was recorded, and the tricuspid regurgitation maximum velocity was measured as the highest value recorded from all views. Spectral pulsed-wave tissue Doppler-derived early diastolic relaxation velocity (eʹ) were also obtained from the septal and lateral mitral annular position. Finally, the E/eʹ and E/A ratios were calculated as a Doppler echocardiographic estimate of the LV filling pressure. All measurements were made in ≥3 consecutive cardiac cycles and average values were used for final analyses. The following echocardiographic parameters were defined as normal or abnormal, with the given cutoffs for abnormality: 1) abnormal e′ (septal eʹ velocity <7 cm/s and/or lateral e′ velocity <10 cm/s); 2) abnormal E/e′ (averaged from septal and lateral wall E/e′ >14); 3) abnormal TR velocity (>2.8 m/s); and 4) abnormal left atrial volume index (>34 ml/m2) (4).
Machine learning–based classification
Our analytical, unsupervised approach to machine learning (ML)–based classification included the first step of choosing the most appropriate combination of classifier and sample-splitting methods. We used the R package chromosomal microarray (R Foundation for Statistical Computing, Vienna, Austria) (19), which permitted a direct comparison of 8 different classifiers (component-wise boosting, diagonal discriminant analysis, partial least squares-linear discriminant analysis, shrinkage discriminant analysis, feed-forward neural networks, probabilistic neural networks, random forest, and support vector machine) and 3 methods of sample-splitting (10-fold cross-validation, Monte Carlo cross-validation, and bootstrapping) with a fixed random seed. For each of these combinations, we estimated the misclassification rates, Brier scores, and average estimated probability of low e′. Based on these parameters, we chose a classifier and splitting method that best captured the data structure.
We used the random forest ensemble classifier (20) with a Monte Carlo cross-validation procedure to classify the study subjects. The random forest classifier was performed with the following specifications: input number of variables, 370; type of outcome, dichotomous classification; number of trees, 500; available number of variables for splitting at each node, 19; and minimum node size, 5. Error rates were estimated from the out-of-bag trees. Variable importance was measured using a permutation-based metric (21) that captured the mean improvement in prediction attributable to a given variable. The novel procedure described by Janitza et al. (22) was used to test the significance of variable importance. We performed a total of 20 iterations of the model, with each splitting the sample into a training (67%) and a testing (33%) subset. The multiple iterations were designed to accumulate at least 2 occurrences of each study participant in the testing subset. Predicted probabilities of low e′, variable importance, as well as out-of-bag error rates were aggregated and averaged over the 20 iterations of random forests.
Between-group comparisons were conducted using the Pearson’s chi-square test (for goodness of fit) or Fisher exact test (for categorical variables) and Student’s t-test (for continuous variables) after testing for normal distribution using the Kolmogorov-Smirnov test. Predictive accuracy and screening performance was assessed by estimating the area under the receiver-operating characteristic curve. Differences between potential moderator effects of clinical and ECG covariates on the predictive accuracy were evaluated using Cochrane’s Q test for heterogeneity of the area under the curve across categories of a moderating variable. Incremental value of the random forest-based classifier over other clinical predictors of low e′ was determined using the incremental discrimination improvement and continuous version of the net reclassification index (23). We used Stata 14.0 (Stata Corp, College Station, Texas) for all statistical analyses. The integrated discrimination index and net reclassification index were estimated using the Stata package IDI (Michael Lunt, University of Manchester, Manchester, United Kingdom). Statistical significance was tested at global type I error rate of 0.05.
Clinical characteristics of the 188 study subjects are shown in Table 1. The prevalence of low e′ was 70% in the study sample, and the risk of low e′ was significantly higher in individuals age ≥60 years and in those who were obese, hypertensive, or had a CT-confirmed stenosis (Table 1). Several echocardiographic features coexisted with low e′ status, including LV wall thickness, concentric remodeling or hypertrophy, as well as a number of variables that are used clinically for defining the presence of DD (Table 2).
Predictive performance of the signal-processed surface ECG–based random forest classifier
Of the 8 ML classifiers used to identify low e′ based on 370 features from the signal-processed surface electrocardiography (spECG), consistency and the least error-prone performance was provided by a combination of random forest classification and Monte Carlo cross-validation splitting strategy (Online Table 1). The area under the curve for prediction of low e′ using this random forest-based classifier was 91% (95% confidence interval [CI]: 86% to 95%), with a sensitivity and specificity of 80% and 84%, respectively (Figure 2). The Janitza test indicated that a total of 257 features (of the 370 detailed in the Online Supplementary Data File) were significantly important. The top 25 features are shown in Figure 3A. The variable that was most vital in building the classifier was ICON, which represents a derived energy measure that captures overall categories of the energy kinetics of the myocardium. Detailed analyses revealed that the ICON variable alone had a significantly independent and additive use in the prediction of low e′ (Online Table 2). To validate the distribution of ICON was not age- and population-specific, we analyzed the distribution of ICON repolarization energy index in a young (45 ± 8.7 years, 49% male) cohort from West Virginia with normal echocardiograms. The ICON energy grade distribution (Grade 1: normal, Grade 2: intermediate, and Grade 3: low) of the young cohort with normal eʹ closely mirrored the subject from New York with normal eʹ (Online Table 3). The error rates of the random forest model from multiple iterations are shown as mean (dark red curve) and confidence bands (pink curves) in Figure 3B. It is noticeable that the error rates converged after nearly 150 trees in all iterations.
Coronary artery stenosis and the presence of 2 or more variables of DD significantly coexisted with low e′ (Tables 1 and 2). The spECG-based random forest classifier could also detect 23 of the 28 (82%) cases with significant CAD (>50% stenosis) and 54 of 70 (77%) cases with ≥2 abnormal echocardiographic features of DD. Therefore, when a low e′ was predicted, the post-test probability of stenosis rose to 64% from a pre-test probability of 55% (likelihood ratio: 1.42; 95% CI: 1.11 to 1.82). Similarly, the post-test probability of DD increased to 48% from a pre-test probability of 38% (likelihood ratio: 1.50; 95% CI: 1.21 to 1.87).
Putative moderation of the predictive performance of the spECG-based random forest classifier
We investigated whether important clinical and ECG covariates are likely to influence the predictive performance of the spECG-based random forest classifier. Our results indicated that individuals ≥60 years of age, those who were obese, and those with hypertension had a better predictive performance compared with their respective counterparts (Table 3). Similarly, individuals with a “borderline” or “abnormal” rating on the Glasgow Interpretive Analysis were associated with a marginally better prediction of low e′ by the random forest classifier. However, this gradient across the ECG diagnosis criteria was not statistically heterogeneous (Table 3). No other variables demonstrated a statistically significant moderator-type association with the predictive performance of the random forest classifier, including CT-confirmed stenosis.
Incremental value of the spECG-based random forest classifier to detect low e′
We next determined whether addition of the random forest classifier-based prediction of low e′ had incremental value above and beyond the three clinical features associated with a high risk of low e′, age ≥60 years, obesity, and hypertension. We found a sharp improvement in prediction due to the classifier such that it improved the area under the curve by 19% (p < 0.001), integrated discrimination index by 0.42 (p < 0.001), and correctly reclassified over 80% of low e′ and normal individuals (p < 0.001) (Table 4, Model 1). Furthermore, upon addition of Glasgow risk categorization for surface ECG to this model, the spECG-based random forest classifier continued to significantly improve the area under the curve (by 16%), integrated discrimination index (by 0.40), and reclassification (correctly reclassifying over 80% of individuals) (Table 4, Model 2).
The high prevalence of LVDD in the general community, the inability of a physical examination to reliably detect LV dysfunction, and the limited utility of a standard ECG mandate physician discernment to identify those individuals who should undergo an echocardiogram. Although ECG is a widely used technical procedure for the evaluation of cardiovascular function, there is no single or explicit ECG pattern that is predictive of the presence of LVDD. In this investigation, we combined measurements from wavelet-transformed ECG with ML techniques to extract time-frequency indexes and features of the signal-processed ECG. This was done with the overarching aim of developing an algorithm for automated diagnosis of abnormal myocardial relaxation. The ML method aided in circumventing the extremely tedious task of manually recognizing features from the wavelet-transformed ECG signal, where patterns convey significant information regarding development of DD (Central Illustration). The spECG demonstrates a robust prediction of myocardial relaxation abnormalities as seen on echocardiography. Moreover, the prediction of abnormal relaxation also allowed ready recognition of subjects with more advanced stages of DD and concurrent CAD with significantly more incremental value compared with clinical variables and surface ECG.
With the advent of computerized techniques, there has been a resurgence of interest in correlating surface ECG with features of LVDD. There has also been a growing interest in different linear and nonlinear techniques for identifying subjects with relaxation abnormalities and for determining risks for heart failure. The present study extends these observations and provides strong evidence that use of ML techniques can help isolate discernible features from ECG wavelets for robust estimation of LV relaxation abnormalities. Specifically, the wavelet transform technique in this investigation utilized the wavelet energy in the peak of QRS to scale the appearance of the energy in the T-wave display. This relative distribution of energy colors improved the signal-to-noise ratio and magnified the repolarization energy signals, which allowed for extraction of features associated with reduced early diastolic myocardial relaxation velocity.
Cellular work, animal models, and human cohorts have demonstrated that electrocardiographic repolarization changes in the T-wave are accompanied by echocardiographic signs of DD (24–28). The link between electrical repolarization and diastolic mechanics may be mediated by changes in calcium handling. Indeed, myocardial ischemia, hypertrophy, aging, and risk factors such as hypertension, diabetes, or smoking are associated with sarcoplasmic reticulum calcium uptake inhibition (25). Delayed uptake of calcium is pathophysiologically associated with a prolongation of action potential and QT interval (26). A prolonged QT interval has been further correlated with progressive reduction of tissue Doppler-derived relaxation velocities in several clinical studies (27–29). In addition to the QT interval, animal and cellular experiments have also investigated a specific relationship between transmural heterogeneity of relaxation and the time interval between peak and end of the T-wave (30). A recent study indicated an inverse, linear correlation between the time interval between peak and end of the T-wave observed on surface ECG and tissue Doppler-derived early diastolic LV longitudinal myocardial relaxation velocities (31). Another study used unsupervised machine learning to divide heart failure with preserved ejection fraction into 3 phenotypes with different risk profiles. The T-wave peak to end duration was associated with higher B-type natriuretic peptide level, lower eʹ velocity, and a high-risk phenotype classification (32). In all of these studies, however, the correlations were modest, and the ECG indexes were not robust as diagnostic tests, most likely due to simplification by using only 1 ECG feature. We therefore selected a multifeature extraction method using machine learning to assess the capability of spECG to predict abnormal myocardial relaxation in patients with risk factors who were otherwise scheduled for CT coronary angiogram. The spECG not only revealed an ability to predict cardiac relaxation abnormalities, but 23 of the 28 cases (82%) with significant CAD (defined as >50% stenosis of the epicardial coronary arteries) were identified using spECG. A technique similar to spECG may, therefore, be a useful tool in daily clinical cardiology practices where screening strategies to identify subclinical LVDD and CAD risks are often clustered together.
Some limitations of this study should be considered before the results can be generalized. First, this was a feasibility study with a limited number of patients. We determined the post hoc statistical power of this dataset to detect observed differences in the area under the curve consequent to inclusion of the random forest classifier into the discriminating model using the pROC R package (33). Analyses revealed that the estimated post hoc statistical power of Models 1 and 2 were 0.9788 and 0.9562, respectively (Table 3). These data combined with the knowledge that the RF-based classifier somewhat obviates the need for multiple comparisons, because it demonstrates that these estimates had adequate statistical power to address the study question. Second, the trade-off between a simpler predictor (e.g., based on the ICON variable only as shown in Online Table 3) or a more comprehensive model such as the one used in this study remains to be assessed in future studies. Third, the sensitivity and specificity were 80% and 84%, respectively, and the reason for false positive or negative results may be related to a relatively fixed definition of abnormal eʹ without clinical adjustments for age- and sex-related variations in eʹ. For example, a value of 7 cm/s for septal eʹ may be extremely low velocity for defining abnormal relaxation in a younger population. Future studies would need to address the distribution of spECG in comparison with age- and sex-related ranges in eʹ and diastolic dysfunction parameters. Fourth, wavelet transform analysis over time has been applied to a variety of biomedical signals with predictable accuracy; however, the assessment of LVDD is a new application, and the reproducibility over time during serial visits would require evaluation in future investigations. Fifth, the use of spECG was evaluated for detecting abnormal LV relaxation, which refers to early stages of diastolic dysfunction where many patients may not have yet advanced features, including left atrial enlargement and elevation of left atrial and LV filling pressures. Future studies would be required to understand the value of spECG in predicting progressive grades of LVDD. Similarly, it would be valuable to investigate the role of spECG in assessing LV strain abnormalities, which are more sensitive markers of cardiac dysfunction. Finally, etiology-specific differences in diagnostic performance of spECG were not addressed in the present investigation and would require future in-depth exploration.
ECG remains one of the widely used diagnostic screening test procedures in cardiology. Although LVDD occurs early in cardiovascular disease, the use of ECG as a screening technique for detecting LVDD remains unrecognized. The present investigation used signal-processing techniques to magnify small changes on the surface ECG frequency spectrum associated with development of myocardial relaxation abnormalities and showed incremental value of spECG over conventional ECG for robust prediction of myocardial relaxation abnormalities. These data suggest a potential role for spECG as a novel screening mechanism for LVDD that can be utilized in population-based studies to identify patients that would benefit from echocardiographic evaluations.
COMPETENCY IN PATIENT CARE AND PROCEDURAL SKILLS: Early repolarization abnormalities identified by spECG are associated with abnormal myocardial relaxation detected by tissue Doppler echocardiography. These findings are incremental to clinical symptoms and signs and information obtained from the surface ECG.
TRANSLATIONAL OUTLOOK: Population-based studies are needed to evaluate the diagnostic use of spECG for detection of the earliest stages of diastolic dysfunction in various types of cardiac disease and monitoring the myocardial response to therapy.
The authors thank Drs. Alaa Omar, Allen Weiss, and Ahmad Mahmoud for their help in compiling the echocardiography database.
This research was funded by an investigator-initiated grant from HeartSciences. The sponsors had no roles in designing, acquisition or interpretation of data. Dr. Sengupta has served as a consultant for HeartSciences and Hitachi Aloka Ltd. Dr. Kulkarni has served as a statistical consultant to HeartSciences. Dr. Narula has reported that he has no relationships relevant to the contents of this paper to disclose. Bijoy Khanderia, MD, served as Guest Editor for this paper.
- Abbreviations and Acronyms
- late diastolic filling (atrial contraction wave) velocity
- body mass index
- coronary artery disease
- computed tomography
- peak early filling (diastolic wave) velocity
- early diastolic relaxation velocity
- left ventricular diastolic dysfunction
- machine learning
- signal-processed surface electrocardiography
- Received November 16, 2017.
- Revision received February 1, 2018.
- Accepted February 2, 2018.
- 2018 American College of Cardiology Foundation
- Wan S.H.,
- Vogel M.W.,
- Chen H.H.
- Hunt S.A.,
- Abraham W.T.,
- Chin M.H.,
- et al.
- Nagueh S.F.,
- Smiseth O.A.,
- Appleton C.P.,
- et al.
- Manning W.J.,
- Patel A.R.,
- Picard M.H.,
- et al.
- Brado J.,
- Dechant M.J.,
- Menza M.,
- et al.
- Sengupta P.P.,
- Khandheria B.K.,
- Korinek J.,
- et al.
- Shekatkar S.M.,
- Kotriwar Y.,
- Harikrishnan K.P.,
- Ambika G.
- Li Q.,
- Rajagopalan C.,
- Clifford G.D.
- Sudarshan V.K.,
- Acharya U.R.,
- Oh S.L.,
- et al.
- Fayn J.,
- Rubel P.,
- Macfarlane P.W.
- Murugappan M.,
- Murugappan S.,
- Zheng B.S.
- ↵Slawski M, Boulesteix A, Bernau C. CMA: synthesis of microarray-based classification. Availabe at: https://www.bioconductor.org/packages/release/bioc/html/CMA.html. Accessed December 2017.
- Janitza S.,
- Celik E.,
- Boulesteix A.-L.
- Prenner S.B.,
- Shah S.J.,
- Goldberger J.J.,
- Sauer A.J.
- Nador F.,
- Beria G.,
- De Ferrari G.M.,
- Stramba-Badiale M.,
- Locati E.H.,
- Lotto A.,
- Schwartz P.J.
- Sauer A.,
- Wilcox J.E.,
- Andrei A.C.,
- Passman R.,
- Goldberger J.J.,
- Shah S.J.
- Oskouie S.K.,
- Prenner S.B.,
- Shah S.J.,
- Sauer A.J.