Dual Learning

Many AI (and machine learning) tasks present in dual forms, e.g., English-to-Chinese translation vs. Chinese-to-English translation, speech recognition vs. speech synthesis,question answering vs. question generation, and image classification vs. image generation. Dual learning is a new learning framework that leverages the primal-dual structure of AI tasks to obtain effective feedback or regularization signals in order to enhance the learning/inference process. Since it was first introduced four years ago, the concept has attracted considerable attention in multiple fields, and been proven effective in numerous applications, such as machine translation, image-to-image translation, speech synthesis and recognition, (visual) question answering and generation, image captioning and generation, and code summarization and generation. Offering a systematic and comprehensive overview of dual learning, this book enables interested researchers (both established and newcomers) and practitioners to gain a better understanding of the state of the art in the field. It also provides suggestions for further reading and tools to help readers advance the area. The book is divided into five parts. The first part gives a brief introduction to machine learning and deep learning. The second part introduces the algorithms based on the dual reconstruction principle using machine translation, image translation, speech processing and other NLP/CV tasks as the demo applications. It covers algorithms, such as dual semi-supervised learning, dual unsupervised learning and multi-agent dual learning. In the context of image translation, it introduces algorithms including CycleGAN, DualGAN, DiscoGAN cdGAN and more recent techniques/applications. The third part presents various work based on the probability principle, including dual supervised learning and dual inference based on the joint-probability principle and dual semi-supervised learning based on the marginal-probability principle. The fourth part reviews various theoretical studies on dual learning and discusses its connections to other learning paradigms. The fifth part provides a summary and suggests future research directions. Preface Acknowledgments Contents About the Author 1 Introduction 1.1 Motivation 1.2 Structure Duality in AI Tasks 1.3 Categorization of Dual Learning 1.3.1 Classified by Data Settings 1.3.2 Classified by Principles 1.4 Book Overview References Part I Preparations 2 Machine Learning Basics 2.1 Machine Learning Paradigms 2.1.1 Supervised Learning 2.1.2 Unsupervised Learning 2.1.3 Reinforcement Learning 2.1.4 More Learning Paradigms 2.2 Key Components of a Learning Algorithm 2.2.1 An Example: Multiclass Logistic Regression 2.3 Generalization and Regularization 2.4 Building a Machine Learning Model 2.4.1 Data Collection and Feature Engineering 2.4.2 Algorithm Selection, Model Training, and Hyper-parameter Tuning References 3 Deep Learning Basics 3.1 Neural Networks 3.2 Convolutional Neural Networks 3.3 Sequence Modeling 3.3.1 Recurrent Neural Network and Its Variants 3.3.2 Encoder-Decoder Architecture 3.3.3 Transformer Networks 3.4 Training Deep Models 3.4.1 Stochastic Gradient Descent 3.4.2 Regularization 3.5 Why Deep Networks? References Part II The Dual Reconstruction Principle 4 Dual Learning for Machine Translation and Beyond 4.1 Introduction to Machine Translation 4.1.1 Neural Machine Translation 4.1.2 Back Translation 4.2 The Principle of Dual Reconstruction 4.3 Dual Semi-supervised Learning 4.3.1 Zero-Shot Dual Machine Translation 4.4 Dual Unsupervised Learning 4.4.1 Basic Ideas 4.4.2 System Architectures and Training Algorithms 4.5 Multi-Agent Dual Learning 4.5.1 The Framework 4.5.2 Extensions and Comparisons 4.5.3 Multi-Agent Dual Machine Translation 4.6 Beyond Machine Translation 4.6.1 Semantic Parsing 4.6.2 Text Style Transfer 4.6.3 Conversations References 5 Dual Learning for Image Translation and Beyond 5.1 Introduction 5.1.1 Generative Adversarial Networks 5.2 Basic Idea of Unsupervised Image Translation 5.3 Image to Image Translation 5.3.1 DualGAN 5.3.2 CycleGAN 5.3.3 DiscoGAN 5.4 Fine-Grained Image to Image Translation 5.4.1 The Problem of Fine-Grained Image Translation 5.4.2 Conditional DualGAN 5.4.3 Discussions 5.5 Multi-Domain Image Translation with Multi-Path Consistency 5.6 Beyond Image Translation 5.6.1 Face Related Tasks 5.6.2 Visual-Linguistic Tasks 5.6.3 Other Image Related Tasks References 6 Dual Learning for Speech Processing and Beyond 6.1 Neural Speech Synthesis and Recognition 6.2 Speech Chain with Dual Learning 6.3 Dual Learning for Low-Resource Speech Processing 6.3.1 Denoising Auto-Encoding with Bidirectional Sequence Modeling 6.3.2 Dual Reconstruction with Bidirectional Sequence Modeling 6.3.3 Model Training 6.4 Dual Learning for Extremely Low-Resource Speech Processing 6.4.1 Pre-training and Fine-Tuning 6.4.2 Dual Reconstruction 6.4.3 Knowledge Distillation 6.4.4 Performance of LRSpeech 6.5 Dual Learning for Non-native Speech Recognition 6.5.1 The Problem of Non-native Speech Recognition 6.5.2 The Method Based on the Dual Reconstruction Principle 6.6 Beyond Speech Processing References Part III The Probabilistic Principle 7 Dual Supervised Learning 7.1 The Joint-Probability Principle 7.2 The Algorithm of Dual Supervised Learning 7.3 Applications 7.3.1 Neural Machine Translation 7.3.2 Images Classification and Generation 7.3.3 Sentiment Analysis 7.3.4 Question Answering and Generation 7.3.5 Code Summarization and Generation 7.3.5.1 Augmentation with Attention Duality 7.3.5.2 Code Retrieval, Summarization, and Generation 7.3.6 Natural Language Understanding and Generation 7.4 Theoretical Analysis References 8 Dual Inference 8.1 General Formulation 8.2 Applications 8.3 Theoretical Analysis References 9 Marginal Probability Based Dual Semi-Supervised Learning 9.1 Efficient Approximation of Marginal Probability 9.2 Marginal Probability as a Constraint 9.3 Likelihood Maximization for Unlabeled Data 9.4 Discussions References Part IV Advanced Topics 10 Understanding Dual Reconstruction 10.1 Overview 10.2 Understanding Dual Reconstruction in Unsupervised Settings 10.2.1 A Formulation of Dual Unsupervised Mapping 10.2.2 Issues and the Simplicity Hypothesis 10.2.3 Minimal Complexity 10.3 Understanding Dual Reconstruction in Semi-Supervised Settings 10.3.1 Algorithm and Notations 10.3.2 Translation Between Two Languages 10.3.3 Extension: Multi-domain Dual Learning References 11 Connections to Other Learning Paradigms 11.1 Dual Semi-Supervised Learning and Co-training 11.2 Dual Learning and Multitask Learning 11.3 Dual Learning, GANs and Autoencoder 11.4 Dual Supervised Learning and Bayesian Ying-Yang Learning 11.5 Dual Reconstruction and Related Concepts References Part V Summary and Outlook 12 Summary and Outlook 12.1 Summary 12.2 Future Directions 12.2.1 More Learning Settings and Applications 12.2.2 Efficient Training 12.2.3 Theoretical Understanding References Many AI (and machine learning) tasks present in dual forms, e.g., English-to-Chinese translation vs. Chinese-to-English translation, speech recognition vs. speech synthesis, question answering vs. question generation, and image classification vs. image generation. Dual learning is a new learning framework that leverages the primal-dual structure of AI tasks to obtain effective feedback or regularization signals in order to enhance the learning/inference process. Since it was first introduced four years ago, the concept has attracted considerable attention in multiple fields, and been proven effective in numerous applications, such as machine translation, image-to-image translation, speech synthesis and recognition, (visual) question answering and generation, image captioning and generation, and code summarization and generation. Offering a systematic and comprehensive overview of dual learning, this book enables interested researchers (both established and newcomers) and practitioners to gain a better understanding of the state of the art in the field. It also provides suggestions for further reading and tools to help readers advance the area. The book is divided into five parts. The first part gives a brief introduction to machine learning and deep learning. The second part introduces the algorithms based on the dual reconstruction principle using machine translation, image translation, speech processing and other NLP/CV tasks as the demo applications. It covers algorithms, such as dual semi-supervised learning, dual unsupervised learning and multi-agent dual learning. In the context of image translation, it introduces algorithms including CycleGAN, DualGAN, DiscoGAN cdGAN and more recent techniques/applications. The third part presents various work based on the probability principle, including dual supervised learning and dual inference based on the joint-probability principle and dual semi-supervised learning based on the marginal-probability principle. The fourth part reviews various theoretical studies on dual learning and discusses its connections to other learning paradigms. The fifth part provides a summary and suggests future research directions