A Malice Love 2 (The Bailey Family)

This first comprehensive book on models behind Generative AI has been thoroughly revised to cover all major classes of deep generative models: mixture models, Probabilistic Circuits, Autoregressive Models, Flow-based Models, Latent Variable Models, GANs, Hybrid Models, Score-based Generative Models, Energy-based Models, and Large Language Models. In addition, Generative AI Systems are discussed, demonstrating how deep generative models can be used for neural compression, among others. Deep Generative Modeling is designed to appeal to curious students, engineers, and researchers with a modest mathematical background in undergraduate calculus, linear algebra, probability theory, and the basics of machine learning, deep learning, and programming in Python and PyTorch (or other deep learning libraries). It should find interest among students and researchers from a variety of backgrounds, including computer science, engineering, data science, physics, and bioinformatics who wish to get familiar with deep generative modeling. In order to engage with a reader, the book introduces fundamental concepts with specific examples and code snippets. The full code accompanying the book is available on the author's GitHub site: github.com/jmtomczak/intro_dgm The ultimate aim of the book is to outline the most important techniques in deep generative modeling and, eventually, enable readers to formulate new models and implement them. Foreword Preface Acknowledgments Contents 1 Why Deep Generative Modeling? 1.1 AI Is Not Only About Decision Making 1.2 Where Can We Use (Deep) Generative Modeling? 1.3 How to Formulate (Deep) Generative Modeling? 1.3.1 Autoregressive Models 1.3.2 Flow-Based Models 1.3.3 Latent Variable Models 1.3.4 Energy-Based Models 1.3.5 Score-Based Generative Models 1.3.6 Overview 1.4 Purpose and Content of This Book References 2 Probabilistic Modeling: From Mixture Models to Probabilistic Circuits 2.1 A Probabilistic Perspective on Modeling: Random Variables, Learning, Generalization and Inference 2.1.1 Random Variables and Probability Distributions 2.1.2 Modeling 2.1.3 Learning 2.1.4 Generalization 2.1.5 Inference 2.2 Interlude: Probabilistic Graphical Models 2.3 Mixture Models 2.3.1 Modeling with Mixture of Gaussians (MoG) 2.3.2 Training: The Log-Likelihood Function 2.3.3 Training: Algorithms 2.3.4 Other Mixture Models 2.3.5 Coding Mixture of Gaussians (MoG) 2.4 Probabilistic Circuits 2.4.1 Fully Factorized Models 2.4.2 Hierarchical Mixture Models a.k.a Probabilistic Circuits 2.4.2.1 Building Blocks 2.4.2.2 Building Probabilistic Circuits 2.4.2.3 Final Comments on PCs References 3 Autoregressive Models 3.1 Introduction 3.2 Autoregressive Models Parameterized by Neural Networks 3.2.1 Finite Memory 3.2.2 Long-Range Memory Through RNNs 3.2.3 Long-Range Memory Through Convolutional Nets 3.2.4 Deep Generative Autoregressive Model in Action! 3.2.5 Code 3.2.6 Is It All? No! 3.3 Autoregressive Models with Transformers 3.3.1 Introduction 3.3.2 Transformers, Because Attention Is All You Need! 3.3.2.1 Self-Attention 3.3.2.2 Toward Implementing Transformers 3.3.2.3 Implementing ARMs with Transformers 3.3.2.4 Transformers Constitute Their Own Field(Almost) References 4 Flow-Based Models 4.1 Flows for Continuous Random Variables 4.1.1 Introduction 4.1.2 Change of Variables for Deep Generative Modeling 4.1.3 Building Blocks of RealNVP 4.1.3.1 Coupling Layers 4.1.3.2 Permutation Layers 4.1.3.3 Dequantization 4.1.4 Flows in Action! 4.1.5 Code 4.1.6 Is It All? Really? 4.1.7 ResNet Flows and DenseNet Flows 4.1.7.1 ResNet Flows ch4:behrmann2019invertible, ch4:chen2019residual 4.1.7.2 DenseNet Flows ch4:perugachi2021invertible 4.2 Flows for Discrete Random Variables 4.2.1 Introduction 4.2.2 Flows in R or Maybe Rather in Z? 4.2.3 Integer Discrete Flows 4.2.4 Code 4.2.5 What's Next? References 5 Latent Variable Models 5.1 Introduction 5.2 Probabilistic Principal Component Analysis 5.3 Variational Auto-encoders: Variational Inference for Nonlinear Latent Variable Models 5.3.1 The Model and the Objective 5.3.2 A Different Perspective on the ELBO 5.3.3 Components of VAEs 5.3.3.1 Parameterization of Distributions 5.3.3.2 Reparameterization Trick 5.3.4 VAE in Action! 5.3.5 Code 5.3.6 Typical Issues with VAEs 5.3.7 There Is More! 5.4 Improving Variational Auto-encoders 5.4.1 Priors 5.4.1.1 Insights from Rewriting the ELBO 5.4.1.2 What Does the ELBO Tell Us About the Prior? 5.4.1.3 Standard Gaussian 5.4.1.4 Mixture of Gaussians 5.4.1.5 VampPrior: Variational Mixture of PosteriorsPrior 5.4.1.6 GTM: Generative Topographic Mapping 5.4.1.7 GTM-VampPrior 5.4.1.8 Flow-Based Prior 5.4.1.9 Remarks 5.4.2 Variational Posteriors 5.4.2.1 Variational Posteriors with HouseholderFlows ch5:tomczak2016improving 5.4.2.2 Variational Posteriors with Sylvester Flows ch5:van2018sylvester 5.4.2.3 Hyperspherical Latent Space 5.5 Hierarchical Latent Variable Models 5.5.1 Introduction 5.5.2 Hierarchical VAEs 5.5.2.1 Two-Level VAEs 5.5.2.2 Top-Down VAEs 5.5.2.3 Code 5.5.2.4 Further Reading 5.5.3 Diffusion-Based Deep Generative Models 5.5.3.1 Introduction 5.5.3.2 Model Formulation 5.5.3.3 Code 5.5.3.4 Discussion 5.5.3.5 Further Discussion References 6 Hybrid Modeling 6.1 Introduction 6.1.1 Approach 1: Let's Be Naive! 6.1.2 Approach 2: Shared Parameterization! 6.2 Hybrid Modeling 6.3 Let's Implement It! 6.4 Code 6.5 What's Next? References 7 Energy-Based Models 7.1 Introduction 7.2 Model Formulation 7.3 Training 7.4 Code 7.5 Restricted Boltzmann Machines 7.5.1 Restricting BMs 7.5.2 Learning RBMs 7.5.3 Defining Higher-Order Relationships Through the Energy Function 7.6 Final Remarks 7.7 Are EBMs the Future? References 8 Generative Adversarial Networks 8.1 Introduction 8.2 Implicit Modeling with Generative Adversarial Networks (GANs) 8.2.1 Getting Rid of Kullback-Leibler 8.2.2 Getting Rid of Prescribed Distributions 8.2.3 Adversarial Loss 8.2.4 GANs 8.3 Implementing GANs 8.3.1 Generator 8.3.2 Discriminator 8.3.3 GAN 8.3.4 Training 8.3.5 Results and Comments 8.4 There Are Many GANs Out There! References 9 Score-Based Generative Models 9.1 Introduction 9.2 Score Matching 9.2.1 Modeling and the Objective 9.2.2 Training 9.2.3 Sampling (Generation) 9.2.4 Score Matching and Diffusion-Based Models 9.2.5 Coding Score Matching 9.2.6 What Can We Do with Score Matching? 9.3 Generative Models as Stochastic/Ordinary Differential Equations 9.3.1 A Reminder on Diffusion-Based Models 9.3.2 In the Pursuit of a General Framework 9.3.2.1 ODEs and Numerical Methods 9.3.2.2 SDEs and Probability Flow ODEs 9.3.2.3 PF-ODEs as Score-Based Generative Models 9.3.3 An Example of Score-Based Generative Models: Variance Exploding PF-ODE 9.3.3.1 Model Formulation 9.3.3.2 The Choice of λt 9.3.3.3 Training 9.3.3.4 Sampling 9.3.4 Finally Some Code! 9.3.5 There Is a Fantastic World of Score-Based Generative Models Out There! 9.4 Flow Matching 9.4.1 A Different Perspective on Generative Models with ODEs: Continuous Normalizing Flows (CNFs) 9.4.1.1 About ODEs, Again 9.4.1.2 From the Continuity Equation (Conservation of Mass) to the Instantaneous Change of Variables 9.4.1.3 Calculating the Log-Likelihood for CNFs 9.4.1.4 Hutchinson's Trace Estimator 9.4.2 Going with the Flow: Flow Matching 9.4.2.1 The Idea 9.4.2.2 Conditional Flow Matching 9.4.2.3 Conditional Probability Paths 9.4.2.4 Training 9.4.2.5 Sampling from FM 9.4.2.6 Calculating the Log-Likelihood Function 9.4.3 Calculating the Log-Likelihood Function 9.4.4 What Is the Future of Flow Matching? References 10 Deep Generative Modeling for Neural Compression 10.1 Introduction 10.2 General Compression Scheme 10.2.1 Encoder 10.2.2 Decoder 10.2.3 The Full Scheme 10.2.4 The Objective 10.3 A Short Detour: JPEG 10.4 Neural Compression: Components 10.4.1 Encoders and Decoders 10.4.2 Differentiable Quantization 10.4.3 Adaptive Entropy Coding Model 10.4.4 A Neural Compression System 10.4.5 Example 10.5 What's Next? References 11 From Large Language Models to Generative AI Systems 11.1 Introduction 11.2 Large Language Models 11.2.1 What Are Large Language Models (LLMs)? 11.2.1.1 Natural Language Processing and Deep Learning 11.2.1.2 General Architectures of LLMs 11.2.1.3 Parameterizations 11.2.2 Learning LLMs 11.2.3 Famous LLMs 11.2.4 Coding Up Our teenyGPT 11.2.5 Other (Selected) Topics on LLMs 11.3 Generative AI Systems 11.3.1 Introduction 11.3.2 GenAISys: A General Architecture 11.3.3 Training 11.3.4 Examples of GenAISys 11.3.4.1 RAGs 11.3.4.2 Speech2Txt 11.3.4.3 Large Vision Models (LVMs) 11.3.5 The Future of AI Is GenAISys References Useful Facts from Algebra and Calculus A.1 Norms and Inner Products A.1.1 Norm Definition A.1.2 Inner Product Definition A.1.3 Chosen Properties of Norm and Inner Product A.2 Matrix Calculus A.2.1 Liner Dependency A.2.2 Orthogonal and Orthonormal Vectors A.2.3 Chosen Properties of Matrix Calculus A.2.4 Special Cases of Invertible Matrices Useful Facts from Probability Theory and Statistics B.1 Commonly Used Probability Distributions B.1.1 Bernoulli Distribution B.1.2 Categorical (Multinoulli) Distribution B.1.3 Normal Distribution B.1.4 Multivariate Normal Distribution B.1.5 Beta Distribution B.1.6 Marginal Distribution B.1.7 Conditional Distribution B.1.8 Marginal Distribution and Conditional Distribution for Multivariate Normal Distribution B.1.9 Sum Rule B.1.10 Product Rule B.1.11 Bayes' Rule B.2 Statistics B.2.1 Maximum Likelihood Estimator B.2.2 Maximum A Posteriori Estimator B.2.3 Risk in Decision-Making Index