All My Sons

A comprehensive guide to getting well-versed with the mathematical techniques for building modern deep learning architectures Key Features Understand linear algebra, calculus, gradient algorithms, and other concepts essential for training deep neural networks Learn the mathematical concepts needed to understand how deep learning models function Use deep learning for solving problems related to vision, image, text, and sequence applications Book Description Most programmers and data scientists struggle with mathematics, having either overlooked or forgotten core mathematical concepts. This book uses Python libraries to help you understand the math required to build deep learning (DL) models. You'll begin by learning about core mathematical and modern computational techniques used to design and implement DL algorithms. This book will cover essential topics, such as linear algebra, eigenvalues and eigenvectors, the singular value decomposition concept, and gradient algorithms, to help you understand how to train deep neural networks. Later chapters focus on important neural networks, such as the linear neural network and multilayer perceptrons, with a primary focus on helping you learn how each model works. As you advance, you will delve into the math used for regularization, multi-layered DL, forward propagation, optimization, and backpropagation techniques to understand what it takes to build full-fledged DL models. Finally, you'll explore CNN, recurrent neural network (RNN), and GAN models and their application. By the end of this book, you'll have built a strong foundation in neural networks and DL mathematical concepts, which will help you to confidently research and build custom models in DL. What you will learn Understand the key mathematical concepts for building neural network models Discover core multivariable calculus concepts Improve the performance of deep learning models using optimization techniques Cover optimization algorithms, from basic stochastic gradient descent (SGD) to the advanced Adam optimizer Understand computational graphs and their importance in DL Explore the backpropagation algorithm to reduce output error Cover DL algorithms such as convolutional neural networks (CNNs), sequence models, and generative adversarial networks (GANs) Who this book is for This book is for data scientists, machine learning developers, aspiring deep learning developers, or anyone who wants to understand the foundation of deep learning by learning the math behind it. Working knowledge of the Python programming language and machine learning basics is required. Table of Contents Linear Algebra Vector Calculus Probability and Statistics Optimization Graph Theory Linear Neural Networks Feedforward Neural Networks Regularization Convolutional Neural Networks Recurrent Neural Networks Attention Mechanisms Generative Models Transfer and Meta Learning Geometric Deep Learning Title Page Copyright and Credits About Packt Contributors Table of Contents Preface Section 1: Essential Mathematics for Deep Learning Linear Algebra Comparing scalars and vectors Linear equations Solving linear equations in n-dimensions Solving linear equations using elimination Matrix operations Adding matrices Multiplying matrices Inverse matrices Matrix transpose Permutations Vector spaces and subspaces Spaces Subspaces Linear maps Image and kernel Metric space and normed space Inner product space Matrix decompositions Determinant Eigenvalues and eigenvectors Trace Orthogonal matrices Diagonalization and symmetric matrices Singular value decomposition Cholesky decomposition Summary Vector Calculus Single variable calculus Derivatives Sum rule Power rule Trigonometric functions First and second derivatives Product rule Quotient rule Chain rule Antiderivative Integrals The fundamental theorem of calculus Substitution rule Areas between curves Integration by parts Multivariable calculus Partial derivatives Chain rule Integrals Vector calculus Derivatives Vector fields Inverse functions Summary Probability and Statistics Understanding the concepts in probability Classical probability Sampling with or without replacement Multinomial coefficient Stirling's formula Independence Discrete distributions Conditional probability Random variables Variance Multiple random variables Continuous random variables Joint distributions More probability distributions Normal distribution Multivariate normal distribution Bivariate normal distribution Gamma distribution Essential concepts in statistics Estimation Mean squared error Sufficiency Likelihood Confidence intervals Bayesian estimation Hypothesis testing Simple hypotheses Composite hypothesis The multivariate normal theory Linear models Hypothesis testing Summary Optimization Understanding optimization and it's different types Constrained optimization Unconstrained optimization Convex optimization Convex sets Affine sets Convex functions Optimization problems Non-convex optimization Exploring the various optimization methods Least squares Lagrange multipliers Newton's method The secant method The quasi-Newton method Game theory Descent methods Gradient descent Stochastic gradient descent Loss functions Gradient descent with momentum The Nesterov's accelerated gradient Adaptive gradient descent Simulated annealing Natural evolution Exploring population methods Genetic algorithms Particle swarm optimization Summary Graph Theory Understanding the basic concepts and terminology Adjacency matrix Types of graphs Weighted graphs Directed graphs Directed acyclic graphs Multilayer and dynamic graphs Tree graphs Graph Laplacian Summary Section 2: Essential Neural Networks Linear Neural Networks Linear regression Polynomial regression Logistic regression Summary Feedforward Neural Networks Understanding biological neural networks Comparing the perceptron and the McCulloch-Pitts neuron The MP neuron Perceptron Pros and cons of the MP neuron and perceptron MLPs Layers Activation functions Sigmoid Hyperbolic tangent Softmax Rectified linear unit Leaky ReLU Parametric ReLU Exponential linear unit The loss function Mean absolute error Mean squared error Root mean squared error The Huber loss Cross entropy Kullback-Leibler divergence Jensen-Shannon divergence Backpropagation Training neural networks Parameter initialization All zeros Random initialization Xavier initialization The data Deep neural networks Summary Regularization The need for regularization Norm penalties L2 regularization L1 regularization Early stopping Parameter tying and sharing Dataset augmentation Dropout Adversarial training Summary Convolutional Neural Networks The inspiration behind ConvNets Types of data used in ConvNets Convolutions and pooling Two-dimensional convolutions One-dimensional convolutions 1 × 1 convolutions Three-dimensional convolutions Separable convolutions Transposed convolutions Pooling Global average pooling Convolution and pooling size Working with the ConvNet architecture Training and optimization Exploring popular ConvNet architectures VGG-16 Inception-v1 Summary Recurrent Neural Networks The need for RNNs The types of data used in RNNs Understanding RNNs Vanilla RNNs Bidirectional RNNs Long short-term memory Gated recurrent units Deep RNNs Training and optimization Popular architecture Clockwork RNNs Summary Section 3: Advanced Deep Learning Concepts Simplified Attention Mechanisms Overview of attention Understanding neural Turing machines Reading Writing Addressing mechanisms Content-based addressing mechanism Location-based address mechanism Exploring the types of attention Self-attention Comparing hard and soft attention Comparing global and local attention Transformers Summary Generative Models Why we need generative models Autoencoders The denoising autoencoder The variational autoencoder Generative adversarial networks Wasserstein GANs Flow-based networks Normalizing flows Real-valued non-volume preserving Summary Transfer and Meta Learning Transfer learning Meta learning Approaches to meta learning Model-based meta learning Memory-augmented neural networks Meta Networks Metric-based meta learning Prototypical networks Siamese neural networks Optimization-based meta learning Long Short-Term Memory meta learners Model-agnostic meta learning Summary Geometric Deep Learning Comparing Euclidean and non-Euclidean data Manifolds Discrete manifolds Spectral decomposition Graph neural networks Spectral graph CNNs Mixture model networks Facial recognition in 3D Summary Other Books You May Enjoy Index **A comprehensive guide to getting well-versed with the mathematical techniques for building modern deep learning architectures** ## Key Features * Understand linear algebra, calculus, gradient algorithms, and other concepts essential for training deep neural networks * Learn the mathematical concepts needed to understand how deep learning models function * Use deep learning for solving problems related to vision, image, text, and sequence applications Most programmers and data scientists struggle with mathematics, having either overlooked or forgotten core mathematical concepts. This book uses Python libraries to help you understand the math required to build deep learning (DL) models. You'll begin by learning about core mathematical and modern computational techniques used to design and implement DL algorithms. This book will cover essential topics, such as linear algebra, eigenvalues and eigenvectors, the singular value decomposition concept, and gradient algorithms, to help you understand how to train deep neural networks. Later chapters focus on important neural networks, such as the linear neural network and multilayer perceptrons, with a primary focus on helping you learn how each model works. As you advance, you will delve into the math used for regularization, multi-layered DL, forward propagation, optimization, and backpropagation techniques to understand what it takes to build full-fledged DL models. Finally, you'll explore CNN, recurrent neural network (RNN), and GAN models and their application. By the end of this book, you'll have built a strong foundation in neural networks and DL mathematical concepts, which will help you to confidently research and build custom models in DL. ## What you will learn * Understand the key mathematical concepts for building neural network models * Discover core multivariable calculus concepts * Improve the performance of deep learning models using optimization techniques * Cover optimization algorithms, from basic stochastic gradient descent (SGD) to the advanced Adam optimizer * Understand computational graphs and their importance in DL * Explore the backpropagation algorithm to reduce output error * Cover DL algorithms such as convolutional neural networks (CNNs), sequence models, and generative adversarial networks (GANs) This book is for data scientists, machine learning developers, aspiring deep learning developers, or anyone who wants to understand the foundation of deep learning by learning the math behind it. Working knowledge of the Python programming language and machine learning basics is required. 1. Linear Algebra 2. Vector Calculus 3. Probability and Statistics 4. Optimization 5. Graph Theory 6. Linear Neural Networks 7. Feedforward Neural Networks 8. Regularization 9. Convolutional Neural Networks 10. Recurrent Neural Networks 11. Attention Mechanisms 12. Generative Models 13. Transfer and Meta Learning 14. Geometric Deep Learning The main aim of this book is to make the advanced mathematical background accessible to someone with a programming background. This book will equip the readers with not only deep learning architectures but the mathematics behind them. With this book, you will understand the relevant mathematics that goes behind building deep learning models