Hyperparameter Tuning Cookbook: A guide for scikit-learn, PyTorch, river, and spotPython

This document provides a comprehensive guide to hyperparameter tuning using spotPython for scikit-learn, PyTorch, and river. The first part introduces spotPython's surrogate model-based optimization process, while the second part focuses on hyperparameter tuning. Several case studies are presented, including hyperparameter tuning for sklearn models such as Support Vector Classification, Random Forests, Gradient Boosting (XGB), and K-nearest neighbors (KNN), as well as a Hoeffding Adaptive Tree Regressor from river. The integration of spotPython into the PyTorch and PyTorch Lightning training workflow is also discussed. With a hands-on approach and step-by-step explanations, this cookbook serves as a practical starting point for anyone interested in hyperparameter tuning with Python. Highlights include the interplay between Tensorboard, PyTorch Lightning, spotPython, and river. This publication is under development, with updates available on the corresponding webpage. Preface: Optimization and Hyperparameter Tuning Book Structure Software Used in this Book Spot as an Optimizer Introduction to spotPython Example: Spot and the Sphere Function The Objective Function: Sphere Spot Parameters: fun_evals, init_size and show_models Print the Results Show the Progress Visualizing the Optimization and Hyperparameter Tuning Process with TensorBoard Multi-dimensional Functions Example: Spot and the 3-dim Sphere Function The Objective Function: 3-dim Sphere Results A Contour Plot TensorBoard Conclusion Exercises The Three Dimensional fun_cubed The Ten Dimensional fun_wing_wt The Three Dimensional fun_runge The Three Dimensional fun_linear Isotropic and Anisotropic Kriging Example: Isotropic Spot Surrogate and the 2-dim Sphere Function The Objective Function: 2-dim Sphere Results Example With Anisotropic Kriging Taking a Look at the theta Values Exercises fun_branin fun_sin_cos fun_runge fun_wingwt Using sklearn Surrogates in spotPython Example: Branin Function with spotPython's Internal Kriging Surrogate The Objective Function Branin Running the surrogate model based optimizer Spot: TensorBoard Print the Results Show the Progress and the Surrogate Example: Using Surrogates From scikit-learn GaussianProcessRegressor as a Surrogate Example: One-dimensional Sphere Function With spotPython's Kriging Results Example: Sklearn Model GaussianProcess Exercises DecisionTreeRegressor RandomForestRegressor linear_model.LinearRegression linear_model.Ridge Exercise 2 Sequential Parameter Optimization: Using scipy Optimizers The Objective Function Branin The Optimizer TensorBoard Print the Results Show the Progress Exercises dual_annealing direct shgo basinhopping Performance Comparison Sequential Parameter Optimization: Gaussian Process Models Gaussian Processes Regression: Basic Introductory scikit-learn Example Train and Test Data Building the Surrogate With Sklearn Plotting the SklearnModel The spotPython Version Visualizing the Differences Between the spotPython and the sklearn Model Fits Exercises Schonlau Example Function Forrester Example Function fun_runge Function (1-dim) fun_cubed (1-dim) The Effect of Noise Expected Improvement Example: Spot and the 1-dim Sphere Function The Objective Function: 1-dim Sphere Results Same, but with EI as infill_criterion Non-isotropic Kriging Using sklearn Surrogates The spot Loop spot: The Initial Model Init: Build Initial Design Evaluate Build Surrogate A Simple Predictor Gaussian Processes regression: basic introductory example The Surrogate: Using scikit-learn models Additional Examples Optimize on Surrogate Evaluate on Real Objective Impute / Infill new Points Tests EI: The Famous Schonlau Example EI: The Forrester Example Noise Cubic Function Factors Hyperparameter Tuning and Noise Example: Spot and the Noisy Sphere Function The Objective Function: Noisy Sphere Print the Results Noise and Surrogates: The Nugget Effect The Noisy Sphere Exercises Noisy fun_cubed fun_runge fun_forrester fun_xsin Handling Noise: Optimal Computational Budget Allocation in Spot Example: Spot, OCBA, and the Noisy Sphere Function The Objective Function: Noisy Sphere Print the Results Noise and Surrogates: The Nugget Effect The Noisy Sphere Exercises Noisy fun_cubed fun_runge fun_forrester fun_xsin Hyperparameter Tuning HPT: sklearn SVC on Moons Data Step 1: Setup Step 2: Initialization of the Empty fun_control Dictionary Step 3: SKlearn Load Data (Classification) Step 4: Specification of the Preprocessing Model Step 5: Select Model (algorithm) and core_model_hyper_dict Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Modify hyperparameter of type numeric and integer (boolean) Modify hyperparameter of type factor Optimizers Step 7: Selection of the Objective (Loss) Function Predict Classes or Class Probabilities Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function Run the Spot Optimizer Starting the Hyperparameter Tuning Step 9: Results Show variable importance Get Default Hyperparameters Get SPOT Results Plot: Compare Predictions Detailed Hyperparameter Plots Parallel Coordinates Plot Plot all Combinations of Hyperparameters river Hyperparameter Tuning: Hoeffding Adaptive Tree Regressor with Friedman Drift Data Setup Initialization of the fun_control Dictionary Load Data: The Friedman Drift Data Specification of the Preprocessing Model SelectSelect Model (algorithm) and core_model_hyper_dict Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Selection of the Objective Function Calling the SPOT Function Prepare the SPOT Parameters The Objective Function Run the Spot Optimizer TensorBoard Results The Larger Data Set Get Default Hyperparameters Show Predictions Get SPOT Results Visualize Regression Trees Spot Model Detailed Hyperparameter Plots Parallel Coordinates Plots Plot all Combinations of Hyperparameters HPT: PyTorch With spotPython and Ray Tune on CIFAR10 Step 1: Setup Step 2: Initialization of the fun_control Dictionary Step 3: PyTorch Data Loading Step 4: Specification of the Preprocessing Model Step 5: Select Model (algorithm) and core_model_hyper_dict The Net_Core class Comparison of the Approach Described in the PyTorch Tutorial With spotPython The Search Space: Hyperparameters Configuring the Search Space With Ray Tune Configuring the Search Space With spotPython Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Optimizers Step 7: Selection of the Objective (Loss) Function Evaluation: Data Splitting Hold-out Data Split Cross-Validation Overview of the Evaluation Settings Evaluation: Loss Functions and Metrics Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function fun_torch Using Default Hyperparameters or Results from Previous Runs Starting the Hyperparameter Tuning Step 9: Tensorboard Tensorboard: Start Tensorboard Saving the State of the Notebook Step 10: Results Get the Tuned Architecture (SPOT Results) Get Default Hyperparameters Evaluation of the Default Architecture Evaluation of the Tuned Architecture Detailed Hyperparameter Plots Summary and Outlook Appendix Sample Output From Ray Tune's Run HPT: sklearn RandomForestClassifier VBDP Data Step 1: Setup Step 2: Initialization of the Empty fun_control Dictionary Step 3: PyTorch Data Loading Load Data: Classification VBDP Holdout Train and Test Data Step 4: Specification of the Preprocessing Model Step 5: Select Model (algorithm) and core_model_hyper_dict Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Modify hyperparameter of type numeric and integer (boolean) Modify hyperparameter of type factor Optimizers Selection of the Objective: Metric and Loss Functions Step 7: Selection of the Objective (Loss) Function Metric Function Evaluation on Hold-out Data OOB Score Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function Run the Spot Optimizer Step 9: Tensorboard Step 10: Results Show variable importance Get Default Hyperparameters Get SPOT Results Evaluate SPOT Results Handling Non-deterministic Results Evalution of the Default Hyperparameters Plot: Compare Predictions Cross-validated Evaluations Detailed Hyperparameter Plots Parallel Coordinates Plot Plot all Combinations of Hyperparameters HPT: sklearn XGB Classifier VBDP Data Step 1: Setup Step 2: Initialization of the Empty fun_control Dictionary Step 3: PyTorch Data Loading 1. Load Data: Classification VBDP Holdout Train and Test Data Step 4: Specification of the Preprocessing Model Step 5: Select Model (algorithm) and core_model_hyper_dict Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Modify hyperparameter of type numeric and integer (boolean) Modify hyperparameter of type factor Optimizers Step 7: Selection of the Objective (Loss) Function Evaluation Selection of the Objective: Metric and Loss Functions Loss Function Metric Function Evaluation on Hold-out Data Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function Run the Spot Optimizer Step 9: Tensorboard Step 10: Results Show variable importance Get Default Hyperparameters Get SPOT Results Evaluate SPOT Results Handling Non-deterministic Results Evalution of the Default Hyperparameters Plot: Compare Predictions Cross-validated Evaluations Detailed Hyperparameter Plots Parallel Coordinates Plot Plot all Combinations of Hyperparameters HPT: sklearn SVC VBDP Data Step 1: Setup Step 2: Initialization of the Empty fun_control Dictionary Step 3: PyTorch Data Loading 1. Load Data: Classification VBDP Holdout Train and Test Data Step 4: Specification of the Preprocessing Model Step 5: Select Model (algorithm) and core_model_hyper_dict Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Modify hyperparameter of type numeric and integer (boolean) Modify hyperparameter of type factor Optimizers Selection of the Objective: Metric and Loss Functions Step 7: Selection of the Objective (Loss) Function Metric Function Evaluation on Hold-out Data Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function Run the Spot Optimizer Step 9: Tensorboard Step 10: Results Show variable importance Get Default Hyperparameters Get SPOT Results Evaluate SPOT Results Handling Non-deterministic Results Evalution of the Default Hyperparameters Plot: Compare Predictions Cross-validated Evaluations Detailed Hyperparameter Plots Parallel Coordinates Plot Plot all Combinations of Hyperparameters HPT: sklearn KNN Classifier VBDP Data Step 1: Setup Step 2: Initialization of the Empty fun_control Dictionary Load Data: Classification VBDP Holdout Train and Test Data Step 4: Specification of the Preprocessing Model Step 5: Select Model (algorithm) and core_model_hyper_dict Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Modify hyperparameter of type numeric and integer (boolean) Modify hyperparameter of type factor Optimizers Selection of the Objective: Metric and Loss Functions Step 7: Selection of the Objective (Loss) Function Metric Function Evaluation on Hold-out Data Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function Run the Spot Optimizer Step 9: Tensorboard Step 10: Results Show variable importance Get Default Hyperparameters Get SPOT Results Evaluate SPOT Results Handling Non-deterministic Results Evalution of the Default Hyperparameters Plot: Compare Predictions Cross-validated Evaluations Detailed Hyperparameter Plots Parallel Coordinates Plot Plot all Combinations of Hyperparameters HPT PyTorch Lightning: VBDP Step 1: Setup Step 2: Initialization of the fun_control Dictionary Step 3: PyTorch Data Loading Lightning Dataset and DataModule Step 4: Preprocessing Step 5: Select the NN Model (algorithm) and core_model_hyper_dict Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model Step 7: Data Splitting, the Objective (Loss) Function and the Metric Evaluation Loss Functions and Metrics Metric Step 8: Calling the SPOT Function Preparing the SPOT Call The Objective Function fun Starting the Hyperparameter Tuning Step 9: Tensorboard Step 10: Results Get the Tuned Architecture Cross Validation With Lightning Detailed Hyperparameter Plots Parallel Coordinates Plot Plot all Combinations of Hyperparameters Visualizing the Activation Distribution Submission Appendix Differences to the spotPython Approaches for torch, sklearn and river Taking a Look at the Data The MAPK Metric Appendices Documentation of the Sequential Parameter Optimization Example: spot The Objective Function External Parameters The fun_control Dictionary The design_control Dictionary The surrogate_control Dictionary The optimizer_control Dictionary Run Print the Results Show the Progress Visualize the Surrogate Init: Build Initial Design Replicability Surrogates A Simple Predictor Demo/Test: Objective Function Fails PyTorch: Detailed Description of the Data Splitting Description of the "train_hold_out" Setting References