Enhancing Surrogate-Based Optimization Through Parallelization

This book presents a solution to the challenging issue of optimizing expensive-to-evaluate industrial problems such as the hyperparameter tuning of machine learning models. The approach combines two well-established concepts, Surrogate-Based Optimization (SBO) and parallelization, to efficiently search for optimal parameter setups with as few function evaluations as possible. Through in-depth analysis, the need for parallel SBO solvers is emphasized, and it is demonstrated that they outperform model-free algorithms in scenarios with a low evaluation budget. The SBO approach helps practitioners save significant amounts of time and resources in hyperparameter tuning as well as other optimization projects. As a highlight, a novel framework for objectively comparing the efficiency of parallel SBO algorithms is introduced, enabling practitioners to evaluate and select the most effective approach for their specific use case. Based on practical examples, decision support is delivered, detailing which parts of industrial optimization projects can be parallelized and how to prioritize which parts to parallelize first. By following the framework, practitioners can make informed decisions about how to allocate resources and optimize their models efficiently. Acknowledgments 6 Contents 7 Symbols 9 1 Introduction 11 1.1 Motivation 11 1.2 Research Questions 12 1.3 Outline 13 1.4 Publications 14 References 16 2 Background 18 2.1 Surrogate-Based Optimization 18 2.2 Evolutionary Algorithms 20 2.3 A Taxonomy for Parallel SBO 22 2.3.1 Parallel Objective Function Evaluation (L1) 23 2.3.2 Parallel Model Building (L2) 27 2.3.3 Parallel Evaluation Proposals (L3) 28 2.3.4 Multi-algorithm Approaches (L4) 30 2.3.5 Recommendations for Practitioners 31 2.4 Parallel SBO—A Literature Review 32 References 36 3 Methods/Contributions 38 3.1 Benchmarking Parallel SBO Algorithms 39 3.1.1 A Framework for Parallel SBO Algorithms 41 3.1.2 Conclusions 43 3.2 Test Problems 44 3.2.1 Simulation Based Functions 46 3.2.2 Experiments 48 3.2.3 Results 50 3.2.4 Conclusions 53 3.3 Why Not Other Parallel Algorithms? 54 3.3.1 A Hybrid SBO Baseline 55 3.3.2 Experiments 56 3.3.3 Results 58 3.3.4 Conclusions 61 3.4 Parallelization as Hyper-Parameter Selection 62 3.4.1 Multi-config SBO 62 3.4.2 Experiments 63 3.4.3 Results 65 3.4.4 Conclusions 67 3.5 Exploration Versus Exploitation 68 3.5.1 Experiments 70 3.5.2 Results 71 3.5.3 Conclusions 76 3.6 Multi-local Expected Improvement 78 3.6.1 Batched Multi-local Expected Improvement 79 3.6.2 Experiments 82 3.6.3 Results 83 3.6.4 Conclusions 88 3.7 Adaptive Parameter Selection 89 3.7.1 Benchmark Based Algorithm Configuration 91 3.7.2 Experiments 94 3.7.3 Results 96 3.7.4 Conclusions 98 References 99 4 Application 104 4.1 Electrostatic Precipitators 104 4.1.1 Problem Description 104 4.1.2 Methods 107 4.1.3 Results 109 4.1.4 Conclusions 113 4.2 Application Case Studies 113 References 115 5 Final Evaluation 117 5.1 Define 117 5.2 Analyze 119 5.3 Enhance 120 5.4 Final Evaluation 121 References 121 Appendix Glossary 123