Deep Learning From Scratch : Building with Python From First Principles

With the resurgence of neural networks in the 2010s, deep learning has become essential for machine learning practitioners and even many software engineers. This book provides a comprehensive introduction for data scientists and software engineers with machine learning experience. You’ll start with deep learning basics and move quickly to the details of important advanced architectures, implementing everything from scratch along the way. Author Seth Weidman shows you how neural networks work using a first principles approach. You’ll learn how to apply multilayer neural networks, convolutional neural networks, and recurrent neural networks from the ground up. With a thorough understanding of how neural networks work mathematically, computationally, and conceptually, you’ll be set up for success on all future deep learning projects. This book provides: * Extremely clear and thorough mental models—accompanied by working code examples and mathematical explanations—for understanding neural networks * Methods for implementing multilayer neural networks from scratch, using an easy-to-understand object-oriented framework * Working implementations and clear-cut explanations of convolutional and recurrent neural networks * Implementation of these neural network concepts using the popular PyTorch framework Copyright Table of Contents Preface Understanding Neural Networks Requires Multiple Mental Models Chapter Outlines Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Chapter 1. Foundations Functions Math Diagrams Code Derivatives Math Diagrams Code Nested Functions Diagram Math Code Another Diagram The Chain Rule Math Code A Slightly Longer Example Math Diagram Code Functions with Multiple Inputs Math Diagram Code Derivatives of Functions with Multiple Inputs Diagram Math Code Functions with Multiple Vector Inputs Math Creating New Features from Existing Features Math Diagram Code Derivatives of Functions with Multiple Vector Inputs Diagram Math Code Vector Functions and Their Derivatives: One Step Further Diagram Math Code Vector Functions and Their Derivatives: The Backward Pass Computational Graph with Two 2D Matrix Inputs Math Diagram Code The Fun Part: The Backward Pass Diagram Math Code Conclusion Chapter 2. Fundamentals Supervised Learning Overview Supervised Learning Models Linear Regression Linear Regression: A Diagram Linear Regression: A More Helpful Diagram (and the Math) Adding in the Intercept Linear Regression: The Code Training the Model Calculating the Gradients: A Diagram Calculating the Gradients: The Math (and Some Code) Calculating the Gradients: The (Full) Code Using These Gradients to Train the Model Assessing Our Model: Training Set Versus Testing Set Assessing Our Model: The Code Analyzing the Most Important Feature Neural Networks from Scratch Step 1: A Bunch of Linear Regressions Step 2: A Nonlinear Function Step 3: Another Linear Regression Diagrams Code Neural Networks: The Backward Pass Training and Assessing Our First Neural Network Two Reasons Why This Is Happening Conclusion Chapter 3. Deep Learning from Scratch Deep Learning Definition: A First Pass The Building Blocks of Neural Networks: Operations Diagram Code The Building Blocks of Neural Networks: Layers Diagrams Building Blocks on Building Blocks The Layer Blueprint The Dense Layer The NeuralNetwork Class, and Maybe Others Diagram Code Loss Class Deep Learning from Scratch Implementing Batch Training NeuralNetwork: Code Trainer and Optimizer Optimizer Trainer Putting Everything Together Our First Deep Learning Model (from Scratch) Conclusion and Next Steps Chapter 4. Extensions Some Intuition About Neural Networks The Softmax Cross Entropy Loss Function Component #1: The Softmax Function Component #2: The Cross Entropy Loss A Note on Activation Functions Experiments Data Preprocessing Model Experiment: Softmax Cross Entropy Loss Momentum Intuition for Momentum Implementing Momentum in the Optimizer Class Experiment: Stochastic Gradient Descent with Momentum Learning Rate Decay Types of Learning Rate Decay Experiments: Learning Rate Decay Weight Initialization Math and Code Experiments: Weight Initialization Dropout Definition Implementation Experiments: Dropout Conclusion Chapter 5. Convolutional Neural Networks Neural Networks and Representation Learning A Different Architecture for Image Data The Convolution Operation The Multichannel Convolution Operation Convolutional Layers Implementation Implications The Differences Between Convolutional and Fully Connected Layers Making Predictions with Convolutional Layers: The Flatten Layer Pooling Layers Implementing the Multichannel Convolution Operation The Forward Pass Convolutions: The Backward Pass Batches, 2D Convolutions, and Multiple Channels 2D Convolutions The Last Element: Adding “Channels” Using This Operation to Train a CNN The Flatten Operation The Full Conv2D Layer Experiments Conclusion Chapter 6. Recurrent Neural Networks The Key Limitation: Handling Branching Automatic Differentiation Coding Up Gradient Accumulation Motivation for Recurrent Neural Networks Introduction to Recurrent Neural Networks The First Class for RNNs: RNNLayer The Second Class for RNNs: RNNNode Putting These Two Classes Together The Backward Pass RNNs: The Code The RNNLayer Class The Essential Elements of RNNNodes “Vanilla” RNNNodes Limitations of “Vanilla” RNNNodes One Solution: GRUNodes LSTMNodes Data Representation for a Character-Level RNN-Based Language Model Other Language Modeling Tasks Combining RNNLayer Variants Putting This All Together Conclusion Chapter 7. PyTorch PyTorch Tensors Deep Learning with PyTorch PyTorch Elements: Model, Layer, Optimizer, and Loss Implementing Neural Network Building Blocks Using PyTorch: DenseLayer Example: Boston Housing Prices Model in PyTorch PyTorch Elements: Optimizer and Loss PyTorch Elements: Trainer Tricks to Optimize Learning in PyTorch Convolutional Neural Networks in PyTorch DataLoader and Transforms LSTMs in PyTorch Postscript: Unsupervised Learning via Autoencoders Representation Learning An Approach for Situations with No Labels Whatsoever Implementing an Autoencoder in PyTorch A Stronger Test for Unsupervised Learning, and a Solution Conclusion Appendix A. Deep Dives Matrix Chain Rule Gradient of the Loss with Respect to the Bias Terms Convolutions via Matrix Multiplication Index About the Author Colophon With the resurgence of neural networks in the 2010s, understanding deep learning has become essential for machine learning practitioners and even many software engineers. This practical book provides a thorough introduction for data scientists and software engineers with previous exposure to machine learning. You'll start with deep learning basics and move quickly to the details of important advanced architectures, implementing everything from scratch along the way. Author Seth Weidman shows you how neural networks function using a first principles approach. You'll learn how to apply multilayer neural networks, convolutional neural networks, and recurrent neural networks from the ground up. With a detailed understanding of how these networks work mathematically, computationally, and conceptually, you'll be set up for success on future deep learning projects With the reinvigoration of neural networks in the 2000s, deep learning is now paving the way for modern machine learning. This practical book provides a solid foundation in how deep learning works for data scientists and software engineers with a background in machine learning. Author Seth Weidman shows you how to implement multilayer neural networks, convolutional neural networks, and recurrent neural networks from scratch. Using these networks as building blocks, you'll learn how to build advanced architectures such as image captioning and Neural Turing machines (NTMs). You'll also explore the math behind the theories.