MODERN DEEP LEARNING FOR TABULAR DATA : novel approaches to common modeling problems

Deep learning is one of the most powerful tools in the modern artificial intelligence landscape. While having been predominantly applied to highly specialized image, text, and signal datasets, this book synthesizes and presents novel deep learning approaches to a seemingly unlikely domain – tabular data. Whether for finance, business, security, medicine, or countless other domain, deep learning can help mine and model complex patterns in tabular data – an incredibly ubiquitous form of structured data. Part I of the book offers a rigorous overview of machine learning principles, algorithms, and implementation skills relevant to holistically modeling and manipulating tabular data. Part II studies five dominant deep learning model designs – Artificial Neural Networks, Convolutional Neural Networks, Recurrent Neural Networks, Attention and Transformers, and Tree-Rooted Networks – through both their ‘default’ usage and their application to tabular data. Part III compounds the power of the previously covered methods by surveying strategies and techniques to supercharge deep learning systems: autoencoders, deep data generation, meta-optimization, multi-model arrangement, and neural network interpretability. Each chapter comes with extensive visualization, code, and relevant research coverage. Modern Deep Learning for Tabular Data is one of the first of its kind – a wide exploration of deep learning theory and applications to tabular data, integrating and documenting novel methods and techniques in the field. This book provides a strong conceptual and theoretical toolkit to approach challenging tabular data problems. What You Will Learn Important concepts and developments in modern machine learning and deep learning, with a strong emphasis on tabular data applications. Understand the promising links between deep learning and tabular data, and when a deep learning approach is or isn’t appropriate. Apply promising research and unique modeling approaches in real-world data contexts. Explore and engage with modern, research-backed theoretical advances on deep tabular modeling Utilize unique and successful preprocessing methods to prepare tabular data for successful modelling. Who This Book Is For Data scientists and researchers of all levels from beginner to advanced looking to level up results on tabular data with deep learning or to understand the theoretical and practical aspects of deep tabular modeling research. Applicable to readers seeking to apply deep learning to all sorts of complex tabular data contexts, including business, finance, medicine, education, and security. Table of Contents 5 About the Authors 12 About the Technical Reviewer 13 Acknowledgments 14 Foreword 1 15 Foreword 2 16 Introduction 17 Chapter 1: Classical Machine Learning Principles and Methods 23 Fundamental Principles of Modeling 24 What Is Modeling? 24 Modes of Learning 25 Quantitative Representations of Data: Regression and Classification 28 The Machine Learning Data Cycle: Training, Validation, and Test Sets 29 Bias-Variance Trade-Off 39 Feature Space and the Curse of Dimensionality 42 Optimization and Gradient Descent 51 Metrics and Evaluation 58 Mean Absolute Error 58 Mean Squared Error (MSE) 60 Confusion Matrix 62 Accuracy 63 Precision 63 Recall 64 F1 Score 65 Area Under the Receiver Operating Characteristics Curve (ROC-AUC) 66 Algorithms 69 K-Nearest Neighbors 70 Theory and Intuition 70 Implementation and Usage 74 Linear Regression 77 Theory and Intuition 78 Implementation and Usage 81 Other Variations on Simple Linear Regression 83 Logistic Regression 86 Theory and Intuition 86 Implementation and Usage 90 Other Variations on Logistic Regression 92 Decision Trees 93 Theory and Intuition 93 Implementation and Usage 96 Random Forest 100 Gradient Boosting 103 Theory and Intuition 103 AdaBoost 105 XGBoost 106 LightGBM 108 Summary of Algorithms 110 Thinking Past Classical Machine Learning 110 Key Points 113 Chapter 2: Data Preparation and Engineering 114 Data Storage and Manipulation 115 TensorFlow Datasets 115 Creating a TensorFlow Dataset 116 TensorFlow Sequence Datasets 117 Handling Large Datasets 119 Datasets That Fit in Memory 120 Pickle 120 SciPy and TensorFlow Sparse Matrices 120 Datasets That Do Not Fit in Memory 121 Pandas Chunker 121 h5py 121 NumPy Memory Map 123 Data Encoding 123 Discrete Data 123 Label Encoding 124 One-Hot Encoding 126 Binary Encoding 128 Frequency Encoding 130 Target Encoding 132 Leave-One-Out Encoding 134 James-Stein Encoding 135 Weight of Evidence 137 Continuous Data 138 Min-Max Scaling 139 Robust Scaling 140 Standardization 144 Text Data 145 Keyword Search 146 Raw Vectorization 147 Bag of Words 149 N-Grams 151 TF-IDF 152 Sentiment Extraction 153 Word2Vec 157 Time Data 160 Geographical Data 163 Feature Extraction 164 Single- and Multi-feature Transformations 164 Principal Component Analysis 170 t-SNE 175 Linear Discriminant Analysis 178 Statistics-Based Engineering 180 Feature Selection 182 Information Gain 182 Variance Threshold 184 High-Correlation Method 186 Recursive Feature Elimination 189 Permutation Importance 192 LASSO Coefficient Selection 194 Key Points 196 Chapter 3: Neural Networks and Tabular Data 199 What Exactly Are Neural Networks? 199 Neural Network Theory 201 Starting with a Single Neuron 201 Feed-Forward Operation 202 Introduction to Keras 205 Modeling with Keras 205 Defining the Architecture 208 Compiling the Model 211 Training and Evaluation 212 Loss Functions 213 Math Behind Feed-Forward Operation 217 Activation Functions 219 Sigmoid and Hyperbolic Tangent 220 Rectified Linear Unit 221 LeakyReLU 221 Swish 222 The Nonlinearity and Variability of Activation Functions 224 The Math Behind Neural Network Learning 228 Gradient Descent in Neural Networks 228 The Backpropagation Algorithm 229 Optimizers 232 Mini-batch Stochastic Gradient Descent (SGD) and Momentum 232 Nesterov Accelerated Gradient (NAG) 234 Adaptive Moment Estimation (Adam) 235 A Deeper Dive into Keras 237 Training Callbacks and Validation 238 Batch Normalization and Dropout 241 The Keras Functional API 246 Nonlinear Topologies 247 Multi-input and Multi-output Models 251 Embeddings 255 Model Weight Sharing 255 The Universal Approximation Theorem 258 Selected Research 262 Simple Modifications to Improve Tabular Neural Networks 262 Ghost Batch Normalization 262 Leaky Gates 264 Wide and Deep Learning 266 Self-Normalizing Neural Networks 269 Regularization Learning Networks 270 Key Points 273 Chapter 4: Applying Convolutional Structures to Tabular Data 275 Convolutional Neural Network Theory 276 Why Do We Need Convolutions? 276 The Convolution Operation 284 The Pooling Operation 308 Base CNN Architectures 327 ResNet 327 Inception v3 334 EfficientNet 339 Multimodal Image and Tabular Models 342 1D Convolutions for Tabular Data 355 2D Convolutions for Tabular Data 372 DeepInsight 374 IGTD (Image Generation for Tabular Data) 383 Key Points 394 Chapter 5: Applying Recurrent Structures to Tabular Data 395 Recurrent Models Theory 395 Why Are Recurrent Models Necessary? 395 Recurrent Neurons and Memory Cells 397 Backpropagation Through Time (BPTT) and Vanishing Gradients 400 LSTMs and Exploding Gradients 404 Gated Recurrent Units (GRUs) 408 Bidirectionality 412 Introduction to Recurrent Layers in Keras 413 Return Sequences and Return State 416 Standard Recurrent Model Applications 419 Natural Language 419 Time Series 425 Multimodal Recurrent Modeling 432 Direct Tabular Recurrent Modeling 440 A Novel Modeling Paradigm 440 Optimizing the Sequence 441 Optimizing the Initial Memory State(s) 452 Further Resources 464 Key Points 464 Chapter 6: Applying Attention to Tabular Data 467 Attention Mechanism Theory 467 The Attention Mechanism 468 The Transformer Architecture 473 BERT and Pretraining Language Models 477 Taking a Step Back 480 Working with Attention 483 Simple Custom Bahdanau Attention 483 Native Keras Attention 488 Attention in Sequence-to-Sequence Tasks 497 Improving Natural Language Models with Attention 501 Direct Tabular Attention Modeling 512 Attention-Based Tabular Modeling Research 516 TabTransformer 516 TabNet 530 SAINT 547 ARM-Net 558 Key Points 562 Chapter 7: Tree-Based Deep Learning Approaches 565 Tree-Structured Neural Networks 565 Deep Neural Decision Trees 566 Soft Decision Tree Regressors 572 NODE 577 Tree-Based Neural Network Initialization 580 Net-DNF 590 Boosting and Stacking Neural Networks 595 GrowNet 595 XBNet 600 Distillation 607 DeepGBM 607 Key Points 613 Chapter 8: Autoencoders 615 The Concept of the Autoencoder 615 Vanilla Autoencoders 620 Autoencoders for Pretraining 645 Multitask Autoencoders 654 Sparse Autoencoders 667 Denoising and Reparative Autoencoders 678 Key Points 694 Chapter 9: Data Generation 695 Variational Autoencoders 695 Theory 696 Implementation 701 Generative Adversarial Networks 711 Theory 711 Simple GAN in TensorFlow 716 CTGAN 720 Key Points 724 Chapter 10: Meta-optimization 725 Meta-optimization: Concepts and Motivations 725 No-Gradient Optimization 727 Optimizing Model Meta-parameters 739 Optimizing Data Pipelines 752 Neural Architecture Search 761 Key Points 766 Chapter 11: Multi-model Arrangement 767 Average Weighting 767 Input-Informed Weighting 779 Meta-evaluation 781 Key Points 784 Chapter 12: Neural Network Interpretability 785 SHAP 785 LIME 798 Activation Maximization 802 Key Points 805 Closing Remarks 806 Appendix NumPy and Pandas 807 NumPy Arrays 807 NumPy Array Construction 807 Simple NumPy Indexing 809 Quantitative Manipulation 811 Advanced NumPy Indexing 812 NumPy Data Types 813 Function Application and Vectorization 814 NumPy Array Application: Image Manipulation 815 Pandas DataFrames 826 Constructing Pandas DataFrames 826 Simple Pandas Mechanics 828 Advanced Pandas Mechanics 836 Pivot 837 Melt 838 Explode 839 Stack 840 Unstack 841 Conclusion 841 Index 842