Pattern Classification Using Ensemble Methods (Series in Machine Perception and Artificial Intelligence)

Researchers from various disciplines such as pattern recognition, statistics, and machine learning have explored the use of ensemble methodology since the late seventies. Thus, they are faced with a wide variety of methods, given the growing interest in the field. This book aims to impose a degree of order upon this diversity by presenting a coherent and unified repository of ensemble methods, theories, trends, challenges and applications. The book describes in detail the classical methods, as well as the extensions and novel approaches developed recently. Along with algorithmic descriptions of each method, it also explains the circumstances in which this method is applicable and the consequences and the trade-offs incurred by using the method. Preface......Page 8 Contents......Page 12 1 Introduction to Pattern Classification......Page 17 1.1 Pattern Classification......Page 18 1.2 Induction Algorithms......Page 20 1.4 Decision Trees......Page 21 1.5 Bayesian Methods......Page 24 1.6 Other Induction Methods......Page 30 2 Introduction to Ensemble Learning......Page 35 2.1 Back to the Roots......Page 36 2.3 The Bagging Algorithm......Page 38 2.5 The AdaBoost Algorithm......Page 44 2.6 No Free Lunch Theorem and Ensemble Learning......Page 52 2.7 Bias-Variance Decomposition and Ensemble Learning......Page 54 2.8 Occam’s Razor and Ensemble Learning......Page 56 2.9 Classifier Dependency......Page 57 2.10 Ensemble Methods for Advanced Classification Tasks......Page 77 3.1 Fusions Methods......Page 81 3.2 Selecting Classification......Page 87 3.3 Mixture of Experts and Meta Learning......Page 98 4.1 Overview......Page 109 4.2 Manipulating the Inducer......Page 110 4.3 Manipulating the Training Samples......Page 112 4.4 Manipulating the Target Attribute Representation......Page 117 4.5 Partitioning the Search Space......Page 119 4.6 Multi-Inducers......Page 128 4.7 Measuring the Diversity......Page 130 5.1 Ensemble Selection......Page 135 5.3 Selection of the Ensemble Size While Training......Page 136 5.4 Pruning - Post Selection of the Ensemble Size......Page 137 6 Error Correcting Output Codes......Page 149 6.1 Code-matrix Decomposition of Multiclass Problems......Page 151 6.2 Type I - Training an Ensemble Given a Code-Matrix......Page 152 Problems......Page 165 7.1 Generalization Error......Page 169 7.2 Computational Complexity......Page 192 7.3 Interpretability of the Resulting Ensemble......Page 193 7.4 Scalability to Large Datasets......Page 194 7.5 Robustness......Page 195 7.9 Software Availability......Page 196 7.10 Which Ensemble Method Should be Used?......Page 197 Bibliography......Page 201 Index......Page 239 Preface 8 Contents 12 1 Introduction to Pattern Classification 17 1.1 Pattern Classification 18 1.2 Induction Algorithms 20 1.3 Rule Induction 21 1.4 Decision Trees 21 1.5 Bayesian Methods 24 1.6 Other Induction Methods 30 2 Introduction to Ensemble Learning 35 2.1 Back to the Roots 36 2.2 The Wisdom of Crowds 38 2.3 The Bagging Algorithm 38 2.4 The Boosting Algorithm 44 2.5 The AdaBoost Algorithm 44 2.6 No Free Lunch Theorem and Ensemble Learning 52 2.7 Bias-Variance Decomposition and Ensemble Learning 54 2.8 Occam’s Razor and Ensemble Learning 56 2.9 Classifier Dependency 57 2.10 Ensemble Methods for Advanced Classification Tasks 77 3 Ensemble Classification 81 3.1 Fusions Methods 81 3.2 Selecting Classification 87 3.3 Mixture of Experts and Meta Learning 98 4 Ensemble Diversity 109 4.1 Overview 109 4.2 Manipulating the Inducer 110 4.3 Manipulating the Training Samples 112 4.4 Manipulating the Target Attribute Representation 117 4.5 Partitioning the Search Space 119 4.6 Multi-Inducers 128 4.7 Measuring the Diversity 130 5 Ensemble Selection 135 5.1 Ensemble Selection 135 5.2 Pre Selection of the Ensemble Size 136 5.3 Selection of the Ensemble Size While Training 136 5.4 Pruning - Post Selection of the Ensemble Size 137 6 Error Correcting Output Codes 149 6.1 Code-matrix Decomposition of Multiclass Problems 151 6.2 Type I - Training an Ensemble Given a Code-Matrix 152 6.3 Type II - Adapting Code-matrices to the Multiclass 165 Problems 165 7 Evaluating Ensembles of Classifiers 169 7.1 Generalization Error 169 7.2 Computational Complexity 192 7.3 Interpretability of the Resulting Ensemble 193 7.4 Scalability to Large Datasets 194 7.5 Robustness 195 7.6 Stability 196 7.7 Flexibility 196 7.8 Usability 196 7.9 Software Availability 196 7.10 Which Ensemble Method Should be Used? 197 Bibliography 201 Index 239 "Researchers from various disciplines such as pattern recognition, statistics, and machine learning have explored the use of ensemble methodology since the late seventies. Thus, they are faced with a wide variety of methods, given the growing interest in the field. This book aims to impose a degree of order upon this diversity by presenting a coherent and unified repository of ensemble methods, theories, trends, challenges and applications." "The book describes in detail the classical methods, as well as the extensions and novel approaches developed recently. Along with algorithmic descriptions of each method, it also explains the circumstances in which this method is applicable and the consequences and the trade-offs incurred by using the method."--Jacket