Normalization Techniques in Deep Learning

translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Preface 6 Reference 7 Acknowledgements 8 References 8 Contents 10 About the Author 12 1 Introduction 13 [DELETE] 13 1.1 Denotations and Definitions 15 1.1.1 Optimization Objective 16 1.1.2 Neural Networks 16 1.1.3 Training DNNs 17 1.1.4 Normalization 18 2 Motivation and Overview of Normalization in DNNs 22 [DELETE] 22 2.1 Theory of Normalizing Input 22 2.2 Towards Normalizing Activations 25 2.2.1 Proximal Back-Propagation Framework 25 2.2.2 K-FAC Approximation 26 2.2.3 Highlights of Motivation 27 3 A General View of Normalizing Activations 30 [DELETE] 30 3.1 Normalizing Activations by Population Statistics 30 3.2 Local Statistics in a Sample 31 3.3 Batch Normalization 32 4 A Framework for Normalizing Activations as Functions 37 [DELETE] 37 4.1 Normalization Area Partitioning 39 4.2 Normalization Operation 40 4.2.1 Beyond Standardization Towards Whitening 40 4.2.2 Variations of Standardization 44 4.2.3 Reduced Standardization 45 4.3 Normalization Representation Recovery 46 5 Multi-mode and Combinational Normalization 52 [DELETE] 52 5.1 Multiple Modes 52 5.2 Combination 53 6 BN for More Robust Estimation 57 [DELETE] 57 6.1 Normalization as Functions Combining Population Statistics 57 6.2 Robust Inference Methods for BN 58 7 Normalizing Weights 61 [DELETE] 61 7.1 Constraints on Weights 62 7.2 Training with Constraints 64 8 Normalizing Gradients 69 [DELETE] 69 9 Analysis of Normalization 73 [DELETE] 73 9.1 Scale Invariance in Stabilizing Training 73 9.1.1 Auto-Tuning on Learning Rate 78 9.2 Improved Conditioning in Optimization 78 9.3 Stochasticity for Generalization 80 9.3.1 Theoretical Model for Stochasticity 81 9.3.2 Empirical Analyses for Stochasticity 83 9.4 Effects on Representation 87 9.4.1 Constraint on Feature Representation 88 9.4.2 Effect on Representational Capacity of Model 90 10 Normalization in Task-Specific Applications 95 [DELETE] 95 10.1 Domain Adaptation 96 10.1.1 Domain Generalization 98 10.1.2 Robust Deep Learning Under Covariate Shift 98 10.1.3 Learning Universal Representations 100 10.2 Image Style Transfer 100 10.2.1 Image Translation 101 10.3 Training GANs 102 10.4 Efficient Deep Models 103 11 Summary and Discussion 109 A Appendix 112 A.1 Back-Propagation Through Eigenvalue Decomposition 112 A.2 Derivation of Constraint Number of Normalization Methods 114 A.3 Proofs of Theorems 115 ​This book presents and surveys normalization techniques with a deep analysis in training deep neural networks. In addition, the author provides technical details in designing new normalization methods and network architectures tailored to specific tasks. Normalization methods can improve the training stability, optimization efficiency, and generalization ability of deep neural networks (DNNs) and have become basic components in most state-of-the-art DNN architectures. The author provides guidelines for elaborating, understanding, and applying normalization methods. This book is ideal for readers working on the development of novel deep learning algorithms and/or their applications to solve practical problems in computer vision and machine learning tasks. The book also serves as a resource researchers, engineers, and students who are new to the field and need to understand and train DNNs.