Applied Deep Learning : A Case-Based Approach to Understanding Deep Neural Networks

Work with advanced topics in deep learning, such as optimization algorithms, hyper-parameter tuning, dropout, and error analysis as well as strategies to address typical problems encountered when training deep neural networks. You'll begin by studying the activation functions mostly with a single neuron (ReLu, sigmoid, and Swish), seeing how to perform linear and logistic regression using TensorFlow, and choosing the right cost function. The next section talks about more complicated neural network architectures with several layers and neurons and explores the problem of random initialization of weights. An entire chapter is dedicated to a complete overview of neural network error analysis, giving examples of solving problems originating from variance, bias, overfitting, and datasets coming from different distributions. __Applied Deep Learning__ also discusses how to implement logistic regression completely from scratch without using any Python library except NumPy, to let you appreciate how libraries such as TensorFlow allow quick and efficient experiments. Case studies for each method are included to put into practice all theoretical information. You'll discover tips and tricks for writing optimized Python code (for example vectorizing loops with NumPy). **What You Will Learn** Implement advanced techniques in the right way in Python and TensorFlow Debug and optimize advanced methods (such as dropout and regularization) Carry out error analysis (to realize if one has a bias problem, a variance problem, a data offset problem, and so on) Set up a machine learning project focused on deep learning on a complex dataset**Who This Book Is For**Readers with a medium understanding of machine learning, linear algebra, calculus, and basic Python programming. Table of Contents......Page 5 About the Author......Page 11 About the Technical Reviewer......Page 12 Acknowledgments......Page 14 Introduction......Page 15 How to Set Up Your Python Environment......Page 20 Creating an Environment......Page 22 Installing TensorFlow......Page 28 Jupyter Notebooks......Page 30 Computational Graphs......Page 33 Tensors......Page 36 Computational Graph with tf.constant......Page 38 Computational Graph with tf.Variable......Page 39 Computational Graph with tf.placeholder......Page 41 Differences Between run and eval......Page 44 Dependencies Between Nodes......Page 45 Tips on How to Create and Close a Session......Page 46 The Structure of a Neuron......Page 49 Matrix Notation......Page 53 Python Implementation Tip: Loops and NumPy......Page 54 Identity Function......Page 56 Sigmoid Function......Page 57 Tanh (Hyperbolic Tangent Activation) Function......Page 59 ReLU (Rectified Linear Unit) Activation Function......Page 60 Leaky ReLU......Page 63 Swish Activation Function......Page 64 Cost Function and Gradient Descent: The Quirks of the Learning Rate......Page 65 Learning Rate in a Practical Example......Page 68 Example of Linear Regression in tensorflow......Page 75 Dataset for Our Linear Regression Model......Page 77 Neuron and Cost Function for Linear Regression......Page 80 Satisficing and Optimizing a Metric......Page 86 Cost Function......Page 88 The Dataset......Page 89 tensorflow Implementation......Page 93 References......Page 98 Chapter 3: Feedforward Neural Networks......Page 100 Network Architecture......Page 101 Output of Neurons......Page 104 Example: Equations for a Network with Three Layers......Page 105 sof tmax Function for Multiclass Classification......Page 107 A Brief Digression: Overfitting......Page 108 A Practical Example of Overfitting......Page 109 Basic Error Analysis......Page 116 The Zalando Dataset......Page 117 Building a Model with tensorflow......Page 122 Network Architecture......Page 123 Modifying Labels for the softmax Function—One-Hot Encoding......Page 125 The tensor flow Model......Page 127 Batch Gradient Descent......Page 131 Stochastic Gradient Descent......Page 133 Mini-Batch Gradient Descent......Page 134 Comparison of the Variations......Page 136 Examples of Wrong Predictions......Page 140 Weight Initialization......Page 142 Adding Many Layers Efficiently......Page 144 Advantages of Additional Hidden Layers......Page 147 Comparing Different Networks......Page 148 Tips for Choosing the Right Network......Page 152 Dynamic Learning Rate Decay......Page 154 Iterations or Epochs?......Page 156 Staircase Decay......Page 157 Step Decay......Page 159 Inverse Time Decay......Page 162 Exponential Decay......Page 165 Natural Exponential Decay......Page 167 tensorflow Implementation......Page 175 Applying the Methods to the Zalando Dataset......Page 179 Exponentially Weighted Averages......Page 180 Momentum......Page 184 RMSProp......Page 189 Adam......Page 192 Which Optimizer Should I Use?......Page 194 Example of Self-Developed Optimizer......Page 196 Complex Networks and Overfitting......Page 202 What Is Regularization?......Page 207 About Network Complexity......Page 208 Theory of l2 Regularization......Page 209 tensorflow Implementation......Page 211 l1 Regularization......Page 222 Theory of l1 Regularization and tensorflow Implementation......Page 223 Are Weights Really Going to Zero?......Page 225 Dropout......Page 228 Early Stopping......Page 232 Additional Methods......Page 233 Chapter 6: Metric Analysis......Page 234 Human-Level Performance and Bayes Error......Page 235 A Short Story About Human-Level Performance......Page 238 Bias......Page 240 Training Set Overfitting......Page 242 Test Set......Page 245 How to Split Your Dataset......Page 247 Unbalanced Class Distribution: What Can Happen......Page 251 Precision, Recall, and F1 Metrics......Page 256 Datasets with Different Distributions......Page 262 K-Fold Cross-Validation......Page 270 Manual Metric Analysis: An Example......Page 280 Black-Box Optimization......Page 288 Notes on Black-Box Functions......Page 290 The Problem of Hyperparameter Tuning......Page 291 Sample Black-Box Problem......Page 292 Grid Search......Page 294 Random Search......Page 299 Coarse-to-Fine Optimization......Page 302 Bayesian Optimization......Page 306 Nadaraya-Watson Regression......Page 307 Gaussian Process......Page 308 Prediction with Gaussian Processes......Page 309 Acquisition Function......Page 315 Upper Confidence Bound (UCB)......Page 316 Example......Page 317 Sampling on a Logarithmic Scale......Page 327 Hyperparameter Tuning with the Zalando Dataset......Page 329 A Quick Note on the Radial Basis Function......Page 338 Kernels and Filters......Page 340 Convolution......Page 342 Examples of Convolution......Page 351 Pooling......Page 359 Padding......Page 362 Building Blocks of a CNN......Page 363 Convolutional Layers......Page 364 Stacking Layers Together......Page 366 Example of a CNN......Page 367 Introduction to RNNs......Page 372 Notation......Page 374 Basic Idea of RNNs......Page 375 Learning to Count......Page 376 The Problem Description......Page 382 Regression Problem......Page 386 Dataset Preparation......Page 392 Model Training......Page 401 Chapter 10: Logistic Regression from Scratch......Page 407 Mathematics Behind Logistic Regression......Page 408 Python Implementation......Page 411 Dataset Preparation......Page 414 Running the Test......Page 416 Conclusion......Page 417 Index......Page 418 Work with advanced topics in deep learning, such as optimization algorithms, hyper-parameter tuning, dropout, and error analysis as well as strategies to address typical problems encountered when training deep neural networks. You’ll begin by studying the activation functions mostly with a single neuron (ReLu, sigmoid, and Swish), seeing how to perform linear and logistic regression using TensorFlow, and choosing the right cost function. The next section talks about more complicated neural network architectures with several layers and neurons and explores the problem of random initialization of weights. An entire chapter is dedicated to a complete overview of neural network error analysis, giving examples of solving problems originating from variance, bias, overfitting, and datasets coming from different distributions. Applied Deep Learning also discusses how to implement logistic regression completely from scratch without using any Python library except NumPy, to let you appreciate how libraries such as TensorFlow allow quick and efficient experiments. Case studies for each method are included to put into practice all theoretical information. You’ll discover tips and tricks for writing optimized Python code (for example vectorizing loops with NumPy). What You Will Learn Implement advanced techniques in the right way in Python and TensorFlow Debug and optimize advanced methods (such as dropout and regularization) Carry out error analysis (to realize if one has a bias problem, a variance problem, a data offset problem, and so on) Set up a machine learning project focused on deep learning on a complex dataset Who This Book Is For Readers with a medium understanding of machine learning, linear algebra, calculus, and basic Python programming.