Beginning Deep Learning with TensorFlow : Work with Keras, MNIST Data Sets, and Advanced Neural Networks, 1st Edition

Incorporate deep learning into your development projects through hands-on coding and the latest versions of deep learning software, such as TensorFlow 2 and Keras. The materials used in this book are based on years of successful online education experience and feedback from thousands of online learners. You’ll start with an introduction to AI, where you’ll learn the history of neural networks and what sets deep learning apart from other varieties of machine learning. Discovery the variety of deep learning frameworks and set-up a deep learning development environment. Next, you’ll jump into simple classification programs for hand-writing analysis. Once you’ve tackled the basics of deep learning, you move on to TensorFlow 2 specifically. Find out what exactly a Tensor is and how to work with MNIST datasets. Finally, you’ll get into the heavy lifting of programming neural networks and working with a wide variety of neural network types such as GANs and RNNs. Deep Learning is a new area of Machine Learning research widely used in popular applications, such as voice assistant and self-driving cars. Work through the hands-on material in this book and become a TensorFlow programmer! What You'll Learn Develop using deep learning algorithms Build deep learning models using TensorFlow 2 Create classification systems and other, practical deep learning applications Who This Book Is For Students, programmers, and researchers with no experience in deep learning who want to build up their basic skillsets. Experienced machine learning programmers and engineers might also find value in updating their skills. Table of Contents About the Authors About the Technical Reviewer Acknowledgments Chapter 1: Introduction to Artificial Intelligence 1.1 Artificial Intelligence in Action 1.1.1 Artificial Intelligence Explained 1.1.2 Machine Learning 1.1.3 Neural Networks and Deep Learning 1.2 The History of Neural Networks 1.2.1 Shallow Neural Networks 1.2.2 Deep Learning 1.3 Deep Learning Characteristics 1.3.1 Data Volume 1.3.2 Computing Power 1.3.3 Network Scale 1.3.4 General Intelligence 1.4 Deep Learning Applications 1.4.1 Computer Vision 1.4.2 Natural Language Processing 1.4.3 Reinforcement Learning 1.5 Deep Learning Framework 1.5.1 Major Frameworks 1.5.2 TensorFlow 2 and 1.x 1.5.3 Demo 1.6 Development Environment Installation 1.6.1 Anaconda Installation 1.6.2 CUDA Installation 1.6.3 TensorFlow Installation 1.6.4 Common Editor Installation 1.7 Summary 1.8 Reference Chapter 2: Regression 2.1 Neuron Model 2.2 Optimization Method 2.3 Linear Model in Action 2.4 Summary 2.5 References Chapter 3: Classification 3.1 Handwritten Digital Picture Dataset 3.2 Build a Model 3.3 Error Calculation 3.4 Do We Really Solve the Problem? 3.5 Nonlinear Model 3.6 Model Complexity 3.7 Optimization Method 3.8 Hands-On Handwritten Digital Image Recognition 3.8.1 Build the Network 3.8.2 Model Training 3.9 Summary 3.10 Reference Chapter 4: Basic TensorFlow 4.1 Data Types 4.1.1 Numeric 4.1.2 String 4.1.3 Boolean 4.2 Numerical Precision 4.3 Tensors to Be Optimized 4.4 Create Tensors 4.4.1 Create Tensors from Arrays and Lists 4.4.2 Create All-0 or All-1 Tensors 4.4.3 Create a Customized Numeric Tensor 4.4.4 Create a Tensor from a Known Distribution 4.4.5 Create a Sequence 4.5 Typical Applications of Tensors 4.5.1 Scalar 4.5.2 Vector 4.5.3 Matrix 4.5.4 Three-Dimensional Tensor 4.5.5 Four-Dimensional Tensor 4.6 Indexing and Slicing 4.6.1 Indexing 4.6.2 Slicing 4.6.3 Slicing Summary 4.7 Dimensional Transformation 4.7.1 Reshape 4.7.2 Add and Delete Dimensions 4.7.3 Swap Dimensions 4.7.4 Copy Data 4.8 Broadcasting 4.9 Mathematical Operations 4.9.1 Addition, Subtraction, Multiplication and Division 4.9.2 Power Operations 4.9.3 Exponential and Logarithmic Operations 4.9.4 Matrix Multiplication 4.10 Hands-On Forward Propagation Chapter 5: Advanced TensorFlow 5.1 Merge and Split 5.1.1 Merge 5.1.2 Split 5.2 Common Statistics 5.2.1 Norm 5.2.2 Max, Min, Mean, and Sum 5.3 Tensor Comparison 5.4 Fill and Copy 5.4.1 Fill 5.4.2 Copy 5.5 Data Limiting 5.6 Advanced Operations 5.6.1 tf.gather 5.6.2 tf.gather_nd 5.6.3 tf.boolean_mask 5.6.4 tf.where 5.6.5 tf.scatter_nd 5.6.6 tf.meshgrid 5.7 Load Classic Datasets 5.7.1 Shuffling 5.7.2 Batch Training 5.7.3 Preprocessing 5.7.4 Epoch Training 5.8 Hands-On MNIST Dataset Chapter 6: Neural Networks 6.1 Perceptron 6.2 Fully Connected Layer 6.2.1 Tensor Mode Implementation 6.2.2 Layer Implementation 6.3 Neural Network 6.3.1 Tensor Mode Implementation 6.3.2 Layer Mode Implementation 6.3.3 Optimization 6.4 Activation function 6.4.1 Sigmoid 6.4.2 ReLU 6.4.3 LeakyReLU 6.4.4 Tanh 6.5 Design of Output Layer 6.5.1 Common Real Number Space 6.5.2 [0, 1] Interval 6.5.3 [0,1] Interval with Sum 1 6.5.4 (-1, 1) Interval 6.6 Error Calculation 6.6.1 Mean Square Error Function 6.6.2 Cross-Entropy Error Function 6.7 Types of Neural Networks 6.7.1 Convolutional Neural Network 6.7.2 Recurrent Neural Network 6.7.3 Attention Mechanism Network 6.7.4 Graph Convolutional Neural Network 6.8 Hands-On of Automobile Fuel Consumption Prediction 6.8.1 Dataset 6.8.2 Create a Network 6.8.3 Training and Testing 6.9 References Chapter 7: Backward Propagation Algorithm 7.1 Derivatives and Gradients 7.2 Common Properties of Derivatives 7.2.1 Common Derivatives 7.2.2 Common Property of Derivatives 7.2.3 Hands-On Derivative Finding 7.3 Derivative of Activation Function 7.3.1 Derivative of Sigmoid Function 7.3.2 Derivative of ReLU Function 7.3.3 Derivative of LeakyReLU Function 7.3.4 Derivative of Tanh Function 7.4 Gradient of Loss Function 7.4.1 Gradient of Mean Square Error Function 7.4.2 Gradient of Cross-Entropy Function 7.5 Gradient of Fully Connected Layer 7.5.1 Gradient of a Single Neuron 7.5.2 Gradient of Fully Connected Layer 7.6 Chain Rule 7.7 Back Propagation Algorithm 7.8 Hands-On Optimization of Himmelblau 7.9 Hands-On Back Propagation Algorithm 7.9.1 Dataset 7.9.2 Network Layer 7.9.3 Network model 7.9.4 Network Training 7.9.5 Network Performance 7.10 References Chapter 8: Keras Advanced API 8.1 Common Functional Modules 8.1.1 Common Network Layer Classes 8.1.2 Network Container 8.2 Model Configuration, Training, and Testing 8.2.1 Model Configuration 8.2.2 Model Training 8.2.3 Model Testing 8.3 Model Saving and Loading 8.3.1 Tensor Method 8.3.2 Network Method 8.3.3 SavedModel method 8.4 Custom Network 8.4.1 Custom Network Layer 8.4.2 Customized Network 8.5 Model Zoo 8.5.1 Load Model 8.6 Metrics 8.6.1 Create a Metrics Container 8.6.2 Write Data 8.6.3 Read Statistical Data 8.6.4 Clear the Container 8.6.5 Hands-On Accuracy Metric 8.7 Visualization 8.7.1 Model Side 8.7.2 Browser Side 8.8 Summary Chapter 9: Overfitting 9.1 Model Capacity 9.2 Overfitting and Underfitting 9.2.1 Underfitting 9.2.2 Overfitting 9.3 Dataset Division 9.3.1 Validation Set and Hyperparameters 9.3.2 Early Stopping 9.4 Model Design 9.5 Regularization 9.5.1 L0 Regularization 9.5.2 L1 Regularization 9.5.3 L2 Regularization 9.5.4 Regularization Effect 9.6 Dropout 9.7 Data Augmentation 9.7.1 Rotation 9.7.2 Flip 9.7.3 Cropping 9.7.4 Generate Data 9.7.5 Other Methods 9.8 Hands-On Overfitting 9.8.1 Build the Dataset 9.8.2 Influence of the Number of Network Layers 9.8.3 Impact of Dropout 9.8.4 Impact of Regularization 9.9 References Chapter 10: Convolutional Neural Networks 10.1 Problems with Fully Connected N 10.1.1 Local Correlation 10.1.2 Weight Sharing 10.1.3 Convolution Operation 10.2 Convolutional Neural Network 10.2.1 Single-Channel Input and Single Convolution Kernel 10.2.2 Multi-channel Input and Single Convolution Kernel 10.2.3 Multi-channel Input and Multi-convolution Kernel 10.2.4 Stride Size 10.2.5 Padding 10.3 Convolutional Layer Implementation 10.3.1 Custom Weights 10.3.2 Convolutional Layer Classes 10.4 Hands-On LeNet-5 10.5 Representation Learning 10.6 Gradient Propagation 10.7 Pooling Layer 10.8 BatchNorm Layer 10.8.1 Forward Propagation 10.8.2 Backward Propagation 10.8.3 Implementation of BatchNormalization layer 10.9 Classical Convolutional Network 10.9.1 AlexNet 10.9.2 VGG Series 10.9.3 GoogLeNet 10.10 Hands-On CIFAR10 and VGG13 10.11 Convolutional Layer Variants 10.11.1 Dilated/Atrous Convolution 10.11.2 Transposed Convolution o + 2p − k = n * s o + 2p − k ≠n * s Matrix Transposition Transposed Convolution Implementation 10.11.3 Separate Convolution 10.12 Deep Residual Network 10.12.1 ResNet Principle 10.12.2 ResBlock Implementation 10.13 DenseNet 10.14 Hands-On CIFAR10 and ResNet18 10.15 References Chapter 11: Recurrent Neural Network 11.1 Sequence Representation Method 11.1.1 Embedding Layer 11.1.2 Pre-trained Word Vectors 11.2 Recurrent Neural Network 11.2.1 Is a Fully Connected Layer Feasible? 11.2.2 Shared Weight 11.2.3 Global Semantics 11.2.4 Recurrent Neural Network 11.3 Gradient Propagation 11.4 How to Use RNN Layers 11.4.1 SimpleRNNCell 11.4.2 Multilayer SimpleRNNCell Network 11.4.3 SimpleRNN Layer 11.5 Hands-On RNN Sentiment Classification 11.5.1 Dataset 11.5.2 Network Model 11.5.3 Training and Testing 11.6 Gradient Vanishing and Gradient Exploding 11.6.1 Gradient Clipping 11.6.2 Gradient Vanishing 11.7 RNN Short-Term Memory 11.8 LSTM Principle 11.8.1 Forget Gate 11.8.2 Input Gate 11.8.3 Update Memory 11.8.4 Output Gate 11.8.5 Summary 11.9 How to Use the LSTM Layer 11.9.1 LSTMCell 11.9.2 LSTM layer 11.10 GRU Introduction 11.10.1 Reset Door 11.10.2 Update Gate 11.10.3 How to Use GRU 11.11 Hands-On LSTM/GRU Sentiment Classification 11.11.1 LSTM Model 11.11.2 GRU model 11.12 Pre-trained Word Vectors 11.13 Pre-trained Word Vectors 11.14 References Chapter 12: Autoencoder 12.1 Principle of Autoencoder 12.2 Hands-On Fashion MNIST Image Reconstruction 12.2.1 Fashion MNIST Dataset 12.2.2 Encoder 12.2.3 Decoder 12.2.4 Autoencoder 12.2.5 Network Training 12.2.6 Image Reconstruction 12.3 Autoencoder Variants 12.3.1 Dropout Autoencoder 12.3.2 Adversarial Autoencoder 12.4 Variational Autoencoder 12.4.1 Principle of VAE 12.4.2 Reparameterization Trick 12.5 Hands-On VAE Image Reconstruction 12.5.1 VAE model 12.5.2 Reparameterization Trick 12.5.3 Network Training 12.5.4 Image Generation 12.6 Summary 12.7 References Chapter 13: Generative Adversarial Networks 13.1 Examples of Game Learning 13.2 GAN Principle 13.2.1 Network Structure 13.2.2 Network Training 13.2.3 Unified Objective Function 13.3 Hands-On DCGAN 13.3.1 Cartoon Avatar Dataset 13.3.2 Generator 13.3.3 Discriminator 13.3.4 Training and Visualization 13.4 GAN Variants 13.4.1 DCGAN 13.4.2 InfoGAN 13.4.3 CycleGAN 13.4.4 WGAN 13.4.5 Equal GAN 13.4.6 Self-Attention GAN 13.4.7 BigGAN 13.5 Nash Equilibrium 13.5.1 Discriminator State 13.5.2 Generator State 13.5.3 Nash Equilibrium Point 13.6 GAN Training Difficulty 13.6.1 Hyperparameter Sensitivity 13.6.2 Model Collapse 13.7 WGAN Principle 13.7.1 JS Divergence Disadvantage 13.7.2 EM Distance 13.8 Hands-On WGAN-GP 13.9 References Chapter 14: Reinforcement Learning 14.1 See It Soon 14.1.1 Balance Bar Game 14.1.2 Gym Platform 14.1.3 Policy Network 14.1.4 Gradient Update 14.1.5 Hands-On Balance Bar Game 14.2 Reinforcement Learning Problems 14.2.1 Markov Decision Process 14.2.2 Objective Function 14.3 Policy Gradient Method 14.3.1 Reinforce Algorithm 14.3.2 Improvement of the Original Policy Gradient Method 14.3.3 REINFORCE Algorithm with Bias 14.3.4 Importance Sampling 14.3.5 PPO Algorithm 14.3.6 Hands-On PPO 14.4 Value Function Method 14.4.1 Value Function 14.4.2 Value Function Estimation 14.4.3 Policy Improvement 14.4.4 SARSA Algorithm 14.4.5 DQN Algorithm 14.4.6 DQN Variants 14.4.7 Hands-On DQN 14.5 Actor-Critic Method 14.5.1 Advantage AC Algorithm 14.5.2 A3C Algorithm 14.5.3 Hands-On A3C 14.6 Summary 14.7 References Chapter 15: Customized Dataset 15.1 Pokémon Go Dataset 15.2 Customized Dataset Loading 15.2.1 Create Code Table 15.2.2 Create Sample and Label Form 15.2.3 Dataset Division 15.3 Hands-On Pokémon Dataset 15.3.1 Create Dataset Object 15.3.2 Data Preprocessing 15.3.3 Create Model 15.3.4 Network Training and Testing 15.4 Transfer Learning 15.4.1 Principles of Transfer Learning 15.4.2 Hands-On Transfer Learning 15.5 Summary Index Incorporate deep learning into your development projects through hands-on coding and the latest versions of deep learning software, such as TensorFlow 2 and Keras. The materials used in this book are based on years of successful online education experience and feedback from thousands of online learners. You'll start with an introduction to AI, where you'll learn the history of neural networks and what sets deep learning apart from other varieties of machine learning. Discovery the variety of deep learning frameworks and set-up a deep learning development environment. Next, you'll jump into simple classification programs for hand-writing analysis. Once you've tackled the basics of deep learning, you move on to TensorFlow 2 specifically. Find out what exactly a Tensor is and how to work with MNIST datasets. Finally, you'll get into the heavy lifting of programming neural networks and working with a wide variety of neural network types such as GANs and RNNs. Deep Learning is a new area of Machine Learning research widely used in popular applications, such as voice assistant and self-driving cars. Work through the hands-on material in this book and become a TensorFlow programmer! You will: Develop using deep learning algorithms Build deep learning models using TensorFlow 2 Create classification systems and other, practical deep learning applications