Deep Learning Foundations

This book provides a conceptual understanding of Deep Learning algorithms. The book consists of the four parts: foundations, deep machine learning, deep neural networks, and textual deep learning. The first part provides traditional supervised learning, traditional unsupervised learning, and ensemble learning, as the preparation for studying deep learning algorithms. The second part deals with modification of existing Machine Learning algorithms into Deep Learning algorithms. The book’s third part deals with deep neural networks, such as Multiple Perceptron, Recurrent Networks, Restricted Boltzmann Machine, and Convolutionary Neural Networks. The last part provides deep learning techniques that are specialized for text mining tasks. The book is relevant for researchers, academics, students, and professionals in Machine Learning. Readers need the basic level of knowledge about linear algebra, vector calculus, and traditional Machine Learning algorithms for understanding the Deep Learning algorithms. The first part of this book is concerned with the foundation for studying the deep learning algorithms. Because the deep learning is the area, which is expanded from the machine learning, we need to study the machine learning algorithms before doing the deep learning algorithms. There are four machine learning types: supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but we will cover the supervised learning and the unsupervised learning. The ensemble learning is viewed as the learning type where multiple machine learning algorithms are combined for solving their own demerits as the alternative advanced learning to the deep learning. This part is intended to review the supervised learning algorithms, the unsupervised ones, and the ensemble learning, for providing the foundation for understanding the deep learning. Chapter 1 is concerned with the overview of deep learning. Before discussing the deep learning, let us discuss the swallow learning, which is opposite to the deep learning, and in the swallow learning, the output value is computed directly from the input value. The deep learning is characterized as the process of computing intermediate values between input values and output values, and the input encoding, the output decoding, the convolution, and the unsupervised layer between the input layer and the output layer become the components for implementing the deep learning. There are other advanced learning types than the deep learning: the kernel-based learning, the ensemble learning, and the semi-supervised learning. This chapter is intended to describe the deep learning conceptually for providing the introduction. Chapter 2 is concerned with the supervised learning as a kind of swallow learning. The supervised learning refers to the leaning type where its parameters are optimized for minimizing the error between the desired output and the computed one. Preface Part I: Foundation Part II: Deep Machine Learning Part III: Deep Neural Networks Part IV: Textual Deep Learning Contents Part I Foundation 1 Introduction 1.1 Definition of Deep Learning 1.2 Swallow Learning 1.2.1 Supervised Learning 1.2.2 Unsupervised Learning 1.2.3 Semi-supervised Learning 1.2.4 Reinforcement Learning 1.3 Deep Supervised Learning 1.3.1 Input Encoding 1.3.2 Output Encoding 1.3.3 Unsupervised Layer 1.3.4 Convolution 1.4 Advanced Learning Types 1.4.1 Ensemble Learning 1.4.2 Local Learning 1.4.3 Kernel-Based Learning 1.4.4 Incremental Learning 1.5 Summary and Further Discussions References 2 Supervised Learning 2.1 Introduction 2.2 Simple Supervised Learning Algorithms 2.2.1 Rule-Based Approach 2.2.2 Naive Retrieval 2.2.3 Data Similarity 2.2.4 One Nearest Neighbor 2.3 Neural Networks 2.3.1 Artificial Neuron 2.3.2 Activation Functions 2.3.3 Neural Connection 2.3.4 Perceptron 2.4 Advanced Supervised Learning Algorithms 2.4.1 Naive Bayes 2.4.2 Decision Tree 2.4.3 Random Forest 2.4.4 Support Vector Machine 2.5 Summary and Further Discussions References 3 Unsupervised Learning 3.1 Introduction 3.2 Simple Unsupervised Learning Algorithms 3.2.1 AHC Algorithm 3.2.2 Divisive Algorithm 3.2.3 Online Linear Clustering Algorithm 3.2.4 K Means Algorithm 3.3 Kohonen Networks 3.3.1 Initial Version 3.3.2 Learning Vector Quantization 3.3.3 Semi-supervised Model 3.3.4 Self-Organizing Map 3.4 EM Algorithm 3.4.1 Cluster Distributions 3.4.2 Notations 3.4.3 E-Step 3.4.4 M-Step 3.5 Summary and Further Discussions Reference 4 Ensemble Learning 4.1 Introduction 4.2 Partition 4.2.1 Training Set 4.2.2 Attribute Set 4.2.3 Array Partition 4.2.4 Partition Schemes 4.3 Supervised Combination Schemes 4.3.1 Voting 4.3.2 Expert Gate 4.3.3 Cascading 4.3.4 Cellular Learning 4.4 Multiple Viewed Learning 4.4.1 Views 4.4.2 Multiple Encodings 4.4.3 Multiple Viewed Supervised Learning 4.4.4 Multiple Viewed Unsupervised Learning 4.5 Summary and Further Discussions Part II Deep Machine Learning 5 Deep KNN Algorithm 5.1 Introduction 5.2 Swallow Version 5.2.1 KNN Algorithm 5.2.2 KNN Variants 5.2.3 Trainable KNN Algorithm 5.2.4 Radius Nearest Neighbor 5.3 Basic Deep Versions 5.3.1 Feature Reduction 5.3.2 Kernel-Based KNN Algorithm 5.3.3 Output Decoded KNN 5.3.4 Pooled KNN 5.4 Advanced Deep Versions 5.4.1 Unsupervised Layer 5.4.2 Unsupervised KNN 5.4.3 Stacked KNN 5.4.4 Convolutional KNN Algorithm 5.5 Summary and Further Discussions Reference 6 Deep Probabilistic Learning 6.1 Introduction 6.2 Swallow Version 6.2.1 Normal Distribution 6.2.2 Bayes Classifier 6.2.3 Naive Bayes 6.2.4 Bayesian Networks 6.3 Basic Deep Versions 6.3.1 Kernel-Based Bayes Classifier 6.3.2 Pooling-Based Bayes Classifier 6.3.3 Output Decoded Naive Bayes 6.3.4 Pooled Naive Bayes 6.4 Advanced Deep Versions 6.4.1 Unsupervised Bayes Classifier 6.4.2 Unsupervised Naive Bayes 6.4.3 Stacked Bayes Classifier 6.4.4 Stacked Bayes Classifier 6.5 Summary and Further Discussions Reference 7 Deep Decision Tree 7.1 Introduction 7.2 Swallow Version 7.2.1 Graphical View 7.2.2 Classification Process 7.2.3 Root Node Selection 7.2.4 Learning Process 7.3 Basic Deep Versions 7.3.1 Random Forest 7.3.2 Clustering-Based Multiple Decision Trees 7.3.3 Output-Decoded Decision Tree 7.3.4 Pooled Naive Bayes 7.4 Advanced Deep Versions 7.4.1 Unsupervised Decision Tree 7.4.2 Stacked Decision Tree 7.4.3 Unsupervised Random Forest 7.4.4 Stacked Random Forest 7.5 Summary and Further Discussions Reference 8 Deep Linear Classifier 8.1 Introduction 8.2 Support Vector Machine 8.2.1 Linear Classifier 8.2.2 Kernel Function 8.2.3 SVM Classifier 8.2.4 Dual Constraints 8.3 Basic Deep Versions 8.3.1 SVM as Deep Learning Algorithm 8.3.2 Multiple Kernel-Based SVM 8.3.3 SVM for Multiple Classification 8.3.4 Pooled SVM 8.4 Advanced Deep Versions 8.4.1 Unsupervised Linear Classifier 8.4.2 Stacked Linear Classifier 8.4.3 Unsupervised SVM 8.4.4 Stacked SVM 8.5 Summary and Further Discussions Part III Deep Neural Networks 9 Multiple Layer Perceptron 9.1 Introduction 9.2 Perceptron 9.2.1 Architecture 9.2.2 Classification Process 9.2.3 Learning Process 9.2.4 Perceptron for Regression 9.3 Multiple Layer Perceptrons 9.3.1 Architecture 9.3.2 Input Layer 9.3.3 Hidden Layer 9.3.4 Output Layer 9.4 Learning Process 9.4.1 Weight Update Between Hidden Layer and Output Layer 9.4.2 Weight Update Between Input Layer and Hidden Layer 9.4.3 Entire Learning Process 9.4.4 Stochastic Gradient Descent 9.5 Summary and Further Discussions Reference 10 Recurrent Neural Networks 10.1 Introduction 10.2 Recurrent Architecture 10.2.1 Forward Connection 10.2.2 Recurrent Connection 10.2.3 Hybrid Architecture 10.2.4 Hidden Recurrency 10.3 Recurrent Neural Networks 10.3.1 Basic Recurrent Neural Networks 10.3.2 RNN Variants 10.3.3 LSTM (Long Short-Term Memory) 10.3.4 LSTM Variants 10.4 Applications 10.4.1 Time Series Prediction 10.4.2 Sentimental Analysis 10.4.3 Entire Learning Process 10.4.4 Machine Translation 10.5 Summary and Further Discussions Reference 11 Restricted Boltzmann Machine 11.1 Introduction 11.2 Associative Memory 11.2.1 Input Restoration 11.2.2 Associative MLP 11.2.3 Hopfield Networks 11.2.4 Boltzmann Machine 11.3 Single RBM 11.3.1 Architecture 11.3.2 Input Layer 11.3.3 Learning Process 11.3.4 Classification Model 11.4 Stacked RBM 11.4.1 Multiple Stacked RBM 11.4.2 Input Encoding 11.4.3 Output Decoding 11.4.4 Evolutionary RBM 11.5 Summary and Further Discussions 12 Convolutional Neural Networks 12.1 Introduction 12.2 Pooling 12.2.1 Pooling Concepts 12.2.2 Pooling Types 12.2.3 Dimensionality Downsizing 12.2.4 Pooling for Ensemble Learning 12.3 Convolution 12.3.1 Tensor 12.3.2 Single-Channeled Convolution 12.3.3 Tensor Convolution 12.3.4 Convolution Variants 12.4 CNN Design 12.4.1 ReLU 12.4.2 Pooling + ReLU 12.4.3 Convolution + ReLU 12.4.4 Pooling + Convolution + ReLU 12.5 Summary and Further Discussions Reference Part IV Textual Deep Learning 13 Index Expansion 13.1 Introduction 13.2 Text Indexing 13.2.1 Tokenization 13.2.2 Pooling Types 13.2.3 Stop Word Removal 13.2.4 Additional Filtering 13.3 Semantic Similarity 13.3.1 Word Representation 13.3.2 Cosine Similarity 13.3.3 Euclidean Distances 13.3.4 Table Similarity 13.4 Expansion Schemes 13.4.1 Associated Words 13.4.2 Associated Text 13.4.3 Information Retrieval-Based Scheme 13.4.4 Index Optimization 13.5 Summary and Further Discussions References 14 Text Summarization 14.1 Introduction 14.2 Abstracting 14.2.1 Phrase-Based Abstracting 14.2.2 Keyword-Based Abstracting 14.2.3 Mapping Abstracting into Binary Classification 14.2.4 Machine Learning-Based Abstracting 14.3 Query-Based Text Summarization 14.3.1 Query 14.3.2 Word-Based Summarization 14.3.3 Sentence-Based Summarization 14.3.4 ML-Based Text Summarization 14.4 Multiple Text Summarization 14.4.1 Group Cohesion 14.4.2 Keyword-Based Summarization 14.4.3 Machine Learning-Based Text Summarization 14.4.4 Textual Cluster Prototype 14.5 Summary and Further Discussions References 15 Textual Deep Operations 15.1 Introduction 15.2 Numerical Deep Operations 15.2.1 Text Encoding 15.2.2 Convolution 15.2.3 Pooling 15.2.4 Virtual Examples 15.3 Textual Convolution 15.3.1 Raw Text Structure 15.3.2 Random Part Selection 15.3.3 Hierarchical Indexing 15.3.4 Temporal Topic Analysis 15.4 Textual Pooling 15.4.1 Text Partition 15.4.2 Sub-dimensional Down-sampling 15.4.3 Keyword Extraction 15.4.4 Text Summarization 15.5 Summary and Further Discussions References 16 Text Classification System 16.1 Introduction 16.2 System Architecture 16.2.1 Input Layer 16.2.2 Convolution Layer 16.2.3 Pooling Layer 16.2.4 Design 16.3 Text Classification Process 16.3.1 Convolutional KNN 16.3.2 Convolutional Naive Bayes 16.3.3 Restricted Boltzmann Machine 16.3.4 Convolutional Neural Networks 16.4 Learning Process 16.4.1 Convolutional KNN 16.4.2 Convolutional Naive Bayes 16.4.3 Restricted Boltzmann Machine 16.4.4 Convolutional Neural Networks 16.5 Summary and Further Discussions Reference Index