Representation Learning for Natural Language Processing, 2nd

This book provides an overview of the recent advances in representation learning theory, algorithms, and applications for natural language processing (NLP), ranging from word embeddings to pre-trained language models. It is divided into four parts. Part I presents the representation learning techniques for multiple language entries, including words, sentences and documents, as well as pre-training techniques. Part II then introduces the related representation techniques to NLP, including graphs, cross-modal entries, and robustness. Part III then introduces the representation techniques for the knowledge that are closely related to NLP, including entity-based world knowledge, sememe-based linguistic knowledge, legal domain knowledge and biomedical domain knowledge. Lastly, Part IV discusses the remaining challenges and future research directions. The theories and algorithms of representation learning presented can also benefit other related domains such as machine learning, social network analysis, semantic Web, information retrieval, data mining and computational biology. This book is intended for advanced undergraduate and graduate students, post-doctoral fellows, researchers, lecturers, and industrial engineers, as well as anyone interested in representation learning and natural language processing. As compared to the first edition, the second edition (1) provides a more detailed introduction to representation learning in Chapter 1; (2) adds four new chapters to introduce pre-trained language models, robust representation learning, legal knowledge representation learning and biomedical knowledge representation learning; (3) updates recent advances in representation learning in all chapters; and (4) corrects some errors in the first edition. The new contents will be approximately 50%+ compared to the first edition. This is an open access book. Preface 5 Acknowledgements 11 Contents 13 Acronyms 18 Symbols and Notations 22 1 Representation Learning and NLP 24 1.1 Motivation 24 1.2 Why Representation Learning Is Important for NLP 26 1.3 Basic Ideas of Representation Learning 26 1.4 Development of Representation Learning for NLP 27 1.5 Learning Approaches to Representation Learning for NLP 30 1.6 Applications of Representation Learning for NLP 31 1.7 The Organization of This Book 32 References 33 2 Word Representation 35 2.1 Introduction 35 2.2 One-Hot Word Representation 36 2.3 Distributed Word Representation 36 2.3.1 Brown Cluster 37 2.3.2 Latent Semantic Analysis 39 2.3.3 Word2vec 40 2.3.4 GloVe 43 2.4 Contextualized Word Representation 44 2.5 Extensions 45 2.5.1 Word Representation Theories 46 2.5.2 Multi-prototype Word Representation 47 2.5.3 Multisource Word Representation 48 2.5.4 Multilingual Word Representation 52 2.5.5 Task-Specific Word Representation 54 2.5.6 Time-Specific Word Representation 55 2.6 Evaluation 55 2.6.1 Word Similarity/Relatedness 56 2.6.2 Word Analogy 58 2.7 Summary 58 References 60 3 Compositional Semantics 64 3.1 Introduction 64 3.2 Semantic Space 66 3.2.1 Vector Space 66 3.2.2 Matrix-Vector Space 66 3.3 Binary Composition 67 3.3.1 Additive Model 68 3.3.2 Multiplicative Model 70 3.4 N-Ary Composition 72 3.4.1 Recurrent Neural Network 73 3.4.2 Recursive Neural Network 74 3.4.3 Convolutional Neural Network 76 3.5 Summary 76 References 77 4 Sentence Representation 79 4.1 Introduction 79 4.2 One-Hot Sentence Representation 80 4.3 Probabilistic Language Model 81 4.4 Neural Language Model 82 4.4.1 Feedforward Neural Network Language Model 82 4.4.2 Convolutional Neural Network Language Model 83 4.4.3 Recurrent Neural Network Language Model 83 4.4.4 Transformer Language Model 84 4.4.5 Extensions 89 4.5 Applications 92 4.5.1 Text Classification 93 4.5.2 Relation Extraction 94 4.6 Summary 104 References 105 5 RETRACTED CHAPTER: Document Representation 110 6 Sememe Knowledge Representation 111 6.1 Introduction 111 6.1.1 Linguistic Knowledge Graphs 112 6.2 Sememe Knowledge Representation 114 6.2.1 Simple Sememe Aggregation Model 115 6.2.2 Sememe Attention over Context Model 115 6.2.3 Sememe Attention over Target Model 117 6.3 Applications 118 6.3.1 Sememe-Guided Word Representation 118 6.3.2 Sememe-Guided Semantic Compositionality Modeling 120 6.3.3 Sememe-Guided Language Modeling 125 6.3.4 Sememe Prediction 128 6.3.5 Other Sememe-Guided Applications 139 6.4 Summary 143 References 144 7 World Knowledge Representation 148 7.1 Introduction 148 7.1.1 World Knowledge Graphs 149 7.2 Knowledge Graph Representation 151 7.2.1 Notations 153 7.2.2 TransE 153 7.2.3 Extensions of TransE 157 7.2.4 Other Models 167 7.3 Multisource Knowledge Graph Representation 174 7.3.1 Knowledge Graph Representation with Texts 175 7.3.2 Knowledge Graph Representation with Types 177 7.3.3 Knowledge Graph Representation with Images 179 7.3.4 Knowledge Graph Representation with Logic Rules 180 7.4 Applications 181 7.4.1 Knowledge Graph Completion 182 7.4.2 Knowledge-Guided Entity Typing 184 7.4.3 Knowledge-Guided Information Retrieval 186 7.4.4 Knowledge-Guided Language Models 190 7.4.5 Other Knowledge-Guided Applications 193 7.5 Summary 195 References 196 8 Network Representation 202 8.1 Introduction 202 8.2 Network Representation 204 8.2.1 Spectral Clustering Based Methods 204 8.2.2 DeepWalk 208 8.2.3 Matrix Factorization Based Methods 215 8.2.4 Structural Deep Network Methods 217 8.2.5 Extensions 219 8.2.6 Applications 232 8.3 Graph Neural Networks 237 8.3.1 Motivations 238 8.3.2 Graph Convolutional Networks 239 8.3.3 Graph Attention Networks 244 8.3.4 Graph Recurrent Networks 245 8.3.5 Extensions 247 8.3.6 Applications 251 8.4 Summary 260 References 262 9 Cross-Modal Representation 270 9.1 Introduction 270 9.2 Cross-Modal Representation 271 9.2.1 Visual Word2vec 271 9.2.2 Cross-Modal Representation for Zero-Shot Recognition 273 9.2.3 Cross-Modal Representation for Cross-Media Retrieval 277 9.3 Image Captioning 279 9.3.1 Retrieval Models for Image Captioning 279 9.3.2 Generation Models for Image Captioning 280 9.3.3 Neural Models for Image Captioning 281 9.4 Visual Relationship Detection 286 9.4.1 Visual Relationship Detection with Language Priors 286 9.4.2 Visual Translation Embedding Network 288 9.4.3 Scene Graph Generation 288 9.5 Visual Question Answering 292 9.5.1 VQA and VQA Datasets 292 9.5.2 VQA Models 293 9.6 Summary 296 References 299 10 Resources 303 10.1 Open-Source Frameworks for Deep Learning 303 10.1.1 Caffe 303 10.1.2 Theano 304 10.1.3 TensorFlow 305 10.1.4 Torch 305 10.1.5 PyTorch 306 10.1.6 Keras 307 10.1.7 MXNet 307 10.2 Open Resources for Word Representation 308 10.2.1 Word2Vec 308 10.2.2 GloVe 308 10.3 Open Resources for Knowledge Graph Representation 309 10.3.1 OpenKE 309 10.3.2 Scikit-Kge 310 10.4 Open Resources for Network Representation 310 10.4.1 OpenNE 310 10.4.2 GEM 310 10.4.3 GraphVite 311 10.4.4 CogDL 311 10.5 Open Resources for Relation Extraction 311 10.5.1 OpenNRE 311 References 312 11 Outlook 313 11.1 Introduction 313 11.2 Using More Unsupervised Data 314 11.3 Utilizing Fewer Labeled Data 314 11.4 Employing Deeper Neural Architectures 315 11.5 Improving Model Interpretability 316 11.6 Fusing the Advances from Other Areas 317 References 317 Correction to: Z. Liu et al., Representation Learning for Natural Language Processing, https://doi.org/10.1007/978-981-15-5573-2 319 This open access book provides an overview of the recent advances in representation learning theory, algorithms and applications for natural language processing (NLP). It is divided into three parts. Part I presents the representation learning techniques for multiple language entries, including words, phrases, sentences and documents. Part II then introduces the representation techniques for those objects that are closely related to NLP, including entity-based world knowledge, sememe-based linguistic knowledge, networks, and cross-modal entries. Lastly, Part III provides open resource tools for representation learning techniques, and discusses the remaining challenges and future research directions. The theories and algorithms of representation learning presented can also benefit other related domains such as machine learning, social network analysis, semantic Web, information retrieval, data mining and computational biology. This book is intended for advanced undergraduate and graduate students, post-doctoral fellows, researchers, lecturers, and industrial engineers, as well as anyone interested in representation learning and natural language processing.