A Fundamental Overview of SOTA-Ensemble Learning Methods for Deep Learning: A Systematic Literature Review

(1) * Marco Klaiber Mail (Aalen University of Applied Sciences, 73430 Aalen, Germany)
*corresponding author


The rapid growth in popularity of Deep Learning (DL) continues to bring more use cases and opportunities, with methods rapidly evolving and new fields developing from the convergence of different algorithms. For this systematic literature review, we considered the most relevant peer-reviewed journals and conference papers on the state of the art of various Ensemble Learning (EL) methods for application in DL, which are also expected to give rise to new ones in combination. The EL methods relevant to this work are described in detail and the respective popular combination strategies as well as the individual tuning and averaging procedures are presented. A comprehensive overview of the various limitations of EL is then provided, culminating in the final formulation of research gaps for future scholarly work on the results, which is the goal of this thesis. This work fills the research gap for upcoming work in EL for by proving in detail and making accessible the fundamental properties of the chosen methods, which will further deepen the understanding of the complex topic in the future and, following the maxim of ensemble learning, should enable better results through an ensemble of knowledge in the future.


Ensemble Learning; Bagging; Boosting; Deep Learning; Machine Learning; Predictive Performance; CNN




Article metrics

10.31763/sitech.v2i2.549 Abstract views : 929





Y. Chen, Y. Wang, Y. Gu, X. He, P. Ghamisi, and X. Jia, “Deep learning ensemble for hyperspectral image classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 6, pp. 1882–1897, 2019.

H. Greenspan, B. van Ginneken, and R. M. Summers, “Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique,” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1153–1159, 2016.

W. Liu, M. Zhang, Z. Luo, and Y. Cai, “An ensemble deep learning method for vehicle type classification on visual traffic surveillance sensors,” IEEE Access, vol. 5, pp. 24 417–24 425, 2017.

I. Kononenko, “Machine learning for medical diagnosis: history, state of the art and perspective,” Artificial Intelligence in Medicine, vol. 23, no. 1, pp. 89–109, 2001.

J. Latif, C. Xiao, A. Imran, and S. Tu, “Medical imaging using machine learning and deep learning algorithms: A review,” in 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), pp. 1-5, 2019.

I. E. Naqa and M. J. Murphy, “What is machine learning?” in Machine Learning in Radiation Oncology. Springer International Publishing, pp. 3–11, 2015.

C.-B. Ha and H.-K. Song, “Signal detection scheme based on adaptive ensemble deep learning model,” IEEE Access, vol. 6, pp. 21 342–21 349, 2018.

Y. Ren, L. Zhang, and P. Suganthan, “Ensemble classification and regression-recent developments, applications and future directions,” IEEE Computational Intelligence Magazine, vol. 11, no. 1, pp. 41–53, 2016.

S.-J. Lee, T. Chen, L. Yu, and C.-H. Lai, “Image classification based on the boost convolutional neural network,” IEEE Access, vol. 6, pp. 12 755–12 768, 2018.

Z. Zhou, “Ensemble learning,” Encyclopedia of Biometrics, pp. 270–273, 2009. [11] L. Wen, L. Gao, and X. Li, “A new snapshot ensemble convolutional neural network for fault diagnosis,” IEEE Access, vol. 7, pp. 32 037–32 047, 2019.

Wen, L. Gao, and X. Li, “A new snapshot ensemble convolutional neural network for fault diagnosis,” IEEEAccess, vol. 7, pp. 32037–32047, 2019.

Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.

X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma, “A survey on ensemble learning,” Frontiers of Computer Science, vol. 14, no. 2, pp. 241–258, 2019.

Y. Wang, Y. Yang, Y.-X. Liu, and A. A. Bharath, “A recursive ensemble learning approach with noisy labels or unlabeled data,” IEEE Access, vol. 7, pp. 36 459– 36 470, 2019.

F. Huang, J. Lu, J. Tao, L. Li, X. Tan, and P. Liu, “Research on optimization methods of ELM classification algorithm for hyperspectral remote sensing images,” IEEE Access, vol. 7, pp. 108070–108089, 2019.

O. Sagi and L. Rokach, “Ensemble learning: A survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 8, no. 4, pp. 1-18, 2018.

T. G. Dietterich, “Machine-learning research,” AI Magazine, vol. 18, no. 4, pp. 1-40, 1997.

S. Wan and H. Yang, “Comparison among methods of ensemble learning,” in 2013 International Symposium on Biometrics and Security Technologies, IEEE, pp. 286-290, 2013.

F. Huang, G. Xie, and R. Xiao, “Research on ensemble learning,” in 2009 International Conference on Artificial Intelligence and Computational Intelligence, IEEE, pp. 249-252, 2009.

L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, no. 2, pp. 123–140, 1996.

Y.-H. Na, H. Jo, and J.-B. Song, “Learning to grasp objects based on ensemble learning combining simulation data and real data,” in 2017 17th International Conference on Control, Automation and Systems (ICCAS), pp. 1030-1034, 2017.

M. A. Dede, E. Aptoula, and Y. Genc, “Deep network ensembles for aerial scene classification,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 5, pp. 732–735, 2019.

G. Webb and Z. Zheng, “Multistrategy ensemble learning: reducing error by combining ensemble learning techniques,” IEEE Transactions on Knowledge and Data Engineering, vol. 16, no. 8, pp. 980–991, 2004.

H. Guan and X. Xue, “Robust online visual tracking via a temporal ensemble framework,” in 2016 IEEE International Conference on Multimedia and Expo (ICME), IEEE, pp. 1-6, 2016.

N. Yu, L. Qian, Y. Huang, and Y. Wu, “Ensemble learning for facial age estimation within non-ideal facial imagery,” IEEE Access, vol. 7, pp. 97 938–97 948, 2019.

J. Zilly, J. M. Buhmann, and D. Mahapatra, “Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation,” Computerized Medical Imaging and Graphics, vol. 55, pp. 28–41, 2017.

S. A. Gyamerah, P. Ngare, and D. Ikpe, “On stock market movement prediction via stacking ensemble learning method,” in 2019 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr), pp. 1-8, 2019.

S. Dzeroski and B. ˇ Zenko, “Is combining classifiers with ˇ stacking better than selecting the best one?” Machine Learning, vol. 54, no. 3, pp. 255–273, 2004.

L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.

R. Buettner, S. Sauer, C. Maier, and A. Eckhardt, “Towards ex ante prediction of user performance: A novel NeuroIS methodology based on real-time measurement of mental effort,” in 2015 48th Hawaii International Conference on System Sciences, pp. 532-544, 2015.

H. Liang, L. Song, and X. Li, “The rotate stress of steam turbine prediction method based on stacking ensemble learning,” in 2019 IEEE 19th International Symposium on High Assurance Systems Engineering (HASE), pp. 146-149, 2019.

A. P. Piotrowski and J. J. Napiorkowski, “A comparison of methods to avoid overfitting in neural networks training in the case of catchment runoff modelling,” Journal of Hydrology, vol. 476, pp. 97–111, 2013.

T. G. Dietterich, “Ensemble methods in machine learning,” in Multiple Classifier Systems. Springer Berlin Heidelberg, pp. 1–15, 2000.

Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of Computer and System Sciences, vol. 55, no. 1, pp. 119–139, 1997.

B. Zhang, Y. Yang, C. Chen, L. Yang, J. Han, and L. Shao, “Action recognition using 3d histograms of texture and a multi-class boosting classifier,” IEEE Transactions on Image Processing, vol. 26, no. 10, pp. 4648– 4660, 2017.

Z.-H. Zhou, J. Wu, and W. Tang, “Ensembling neural networks: Many could be better than all,” Artificial Intelligence, vol. 137, no. 1-2, pp. 239–263, 2002.

Y. Zhao, J. Li, and L. Yu, “A deep learning ensemble approach for crude oil price forecasting,” Energy Economics, vol. 66, pp. 9–16, 2017.

G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, “Snapshot ensembles: Train 1, get m for free,” arXiv preprint arXiv:1704.00109, pp. 1–14, 2017.

Y. Freund, “Boosting a weak learning algorithm by majority,” Information and Computation, vol. 121, no. 2, pp. 256–285, 1995.

E. Bauer and R. Kohavi, “An empirical comparison of voting classification algorithms: Bagging, boosting, and variants,” Machine Learning, vol. 36, no. 1, pp. 105– 139, 1999.

J. R. Quinlan, “Bagging, boosting, and c4.5,” in In Proceedings of the Thirteenth National Conference on Artificial Intelligence. AAAI Press, pp. 725–730, 1996.

R. J. Tibshirani and B. Efron, “An introduction to the bootstrap,” Monographs on statistics and applied probability, vol. 57, pp. 1–436, 1993.

A. Kabir, C. Ruiz, and S. A. Alvarez, “Mixed bagging: A novel ensemble learning framework for supervised classification based on instance hardness,” in 2018 IEEE International Conference on Data Mining (ICDM), pp. 1073-1078, 2018.

B. Krawczyk and M. Wozniak, “Wagging for combining ´ weighted one-class support vector machines,” Procedia Computer Science, vol. 51, pp. 1565–1573, 2015.

G. I. Webb, “Multiboosting: A technique for combining boosting and wagging,” Machine Learning, vol. 40, no. 2, pp. 159–196, 2000.

D. H. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, no. 2, pp. 241–259, 1992.

T. K. Ho, “The random subspace method for constructing decision forests,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832–844, 1998.

Y. Lin and Y. Jeon, “Random forests and adaptive nearest neighbors,” Journal of the American Statistical Association, vol. 101, no. 474, pp. 578–590, 2006.

I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” pp. 1–16, 2016.

P. K. Chan and S. J. Stolfo, “A comparative evaluation of voting and meta-learning on partitioned data,” in Machine Learning Proceedings 1995. Elsevier, pp. 90–98, 1995.

J. Zheng, X. Cao, B. Zhang, X. Zhen, and X. Su, “Deep ensemble machine for video classification,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 2, pp. 553–565, 2019.


  • There are currently no refbacks.

Copyright (c) 2021 Marco Klaiber

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Science in Information Technology Letters
ISSN 2722-4139
Published by Association for Scientific Computing Electrical and Engineering (ASCEE)
W : http://pubs2.ascee.org/index.php/sitech
E : andri@ascee.org, andri.pranolo.id@ieee.org

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0

View My Stats