Evolutionary deep neural architecture search : fundamentals, methods, and recent advances / by Yunan Sun, Gary G. Yen, and Mengjie Zhang

This book systematically narrates the fundamentals, methods, and recent advances of evolutionary deep neural architecture search chapter by chapter. This will provide the target readers with sufficient details learning from scratch. In particular, the method parts are devoted to the architecture search of unsupervised and supervised deep neural networks. The people, who would like to use deep neural networks but have no/limited expertise in manually designing the optimal deep architectures, will be the main audience. This may include the researchers who focus on developing novel evolutionary deep architecture search methods for general tasks, the students who would like to study the knowledge related to evolutionary deep neural architecture search and perform related research in the future, and the practitioners from the fields of computer vision, natural language processing, and others where the deep neural networks have been successfully and largely used in their respective fields. Preface References Contents Acronyms Part I Fundamentals and Backgrounds 1 Evolutionary Computation 1.1 Genetic Algorithms (GAs) 1.2 Particle Swarm Optimization (PSO) 1.3 Differential Evolution (DE) 1.4 Genetic Programming (GP) 1.5 Chapter Summary References 2 Deep Neural Networks 2.1 Deep Belief Networks 2.2 Stacked Auto-Encoders 2.2.1 Sparse Auto-Encoders 2.2.2 Weight Decay Auto-Encoders 2.2.3 Denoising Auto-Encoders (DAEs) 2.2.4 Contractive Auto-Encoders 2.2.5 Convolutional Auto-Encoders (CAEs) 2.2.6 Variational Auto-Encoders (VAEs) 2.3 Convolutional Neural Networks (CNNs) 2.3.1 CNN Skeleton 2.3.2 Convolution 2.3.3 Pooling 2.3.4 Reflect Padding 2.3.5 Batch Normalization (BN) 2.3.6 ResNet Blocks (RBs) and DenseNet Blocks (DBs) 2.4 Benchmarks for Deep Neural Networks 2.5 Chapter Summary References Part II Evolutionary Deep Neural Architecture Search for Unsupervised DNNs 3 Architecture Design for Stacked AEs and DBNs 3.1 Introduction 3.2 Related Work and Motivations 3.2.1 Unsupervised Deep Learning 3.2.2 Evolutionary Algorithms for Evolving Neural Networks 3.3 Algorithm Details 3.3.1 Framework of EUDNN 3.3.2 Evolving Connection Weights and Activation Functions 3.3.3 Fine-Tuning Connection Weights 3.3.4 Discussion 3.4 Experimental Design 3.4.1 Performance Metric 3.4.2 Peer Competitors 3.4.3 Parameter Settings 3.5 Experimental Results and Analysis 3.5.1 Performance of EUDNN 3.5.2 Analysis on Pre-training of EUDNN 3.5.3 Analysis on Fine-Tuning of EUDNN 3.5.4 Representation Visualizations 3.6 Chapter Summary References 4 Architecture Design for Convolutional Auto-Encoders 4.1 Introduction 4.2 Motivation of FCAE 4.3 Algorithm Details 4.3.1 Algorithm Overview 4.3.2 Encoding Strategy 4.3.3 Particle Initialization 4.3.4 Fitness Evaluation 4.3.5 Velocity Calculation and Position Update 4.3.6 Deep Training on Global Best Particle 4.4 Experimental Design 4.4.1 Peer Competitors 4.4.2 Parameter Settings 4.5 Experimental Results and Analysis 4.5.1 Overview Performance 4.5.2 Evolution Trajectory of PSOAO 4.5.3 Performance on Different Numbers of Training Examples 4.5.4 Investigation on x-Reference Velocity Calculation 4.6 Chapter Summary References 5 Architecture Design for Variational Auto-Encoders 5.1 Introduction 5.2 Algorithm Details 5.2.1 Algorithm Overview 5.2.2 Strategy of Gene Encoding 5.2.3 Initialization of Population 5.2.4 Evaluation 5.2.5 Crossover Operator and Mutation Operator 5.2.6 Environmental Selection 5.3 Experimental Design 5.3.1 Parameter Setting 5.3.2 Peer Competitors 5.3.3 Performance Evaluation 5.4 Experimental Results and Analysis 5.4.1 Overall Performance 5.4.2 Evolution Trajectory of EvoVAE 5.4.3 Running Time 5.4.4 The Obtained Architecture 5.4.5 Ablation Experiments 5.5 Chapter Summary References Part III Evolutionary Deep Neural Architecture Search for Supervised DNNs 6 Architecture Design for Plain CNNs 6.1 Introduction 6.2 Algorithm Details 6.2.1 Algorithm Overview 6.2.2 Strategy of Gene Encoding 6.2.3 Initialization of Population 6.2.4 Evaluation of Fitness 6.2.5 Slack Binary Tournament Selection 6.2.6 Offspring Generation 6.2.7 Environmental Selection 6.2.8 Select and Decode Best Individual 6.3 Experimental Design 6.3.1 Peer Competitors 6.3.2 Parameter Settings 6.4 Experimental Results and Discussion 6.4.1 Overall Results 6.4.2 Performance of Weight Initialization 6.4.3 Discussion 6.5 Chapter Summary References 7 Architecture Design for RBs and DBs Based CNNs 7.1 Introduction 7.2 Algorithm Details 7.2.1 Algorithm Overview 7.2.2 Population Initialization 7.2.3 Fitness Evaluation 7.2.4 Offspring Generation 7.2.5 Environmental Selection 7.3 Experimental Design 7.3.1 Peer Competitors 7.3.2 Parameter Settings 7.4 Experimental Results and Analysis 7.4.1 Performance Overview 7.4.2 Evolution Trajectory 7.4.3 Designed CNN Architectures 7.5 Chapter Summary References 8 Architecture Design for Skip-Connection Based CNNs 8.1 Introduction 8.2 Algorithm Details 8.2.1 Algorithm Overview 8.2.2 Population Initialization 8.2.3 Fitness Evaluation 8.2.4 Offspring Generating 8.2.5 Environmental Selection 8.3 Experimental Design 8.3.1 Peer Competitors 8.3.2 Parameter Settings 8.4 Experimental Results and Analysis 8.4.1 Overall Results 8.4.2 Transferable Performance on ImageNet 8.4.3 Performance of Crossover Operator 8.4.4 Performance of Acceleration Components 8.4.5 Evolution Trajectory 8.5 Chapter Summary References 9 Hybrid GA and PSO for Architecture Design 9.1 Introduction 9.2 Algorithm Details 9.2.1 Overall Structure of the System 9.2.2 The Evolved CNN Architecture-DynamicNet 9.2.3 HGAPSO Encoding Strategy 9.2.4 HGAPSO Search 9.2.5 HGAPSO Fitness Evaluations 9.3 Experimental Studies 9.3.1 Parameter Settings 9.3.2 State-of-the-Art Methods Versus HGAPSO 9.3.3 Evolved CNN Architecture 9.3.4 One-Run Result on CIFAR-10 Dataset 9.4 Chapter Summary References 10 Internet Protocol Based Architecture Design 10.1 Introduction 10.2 Algorithm Details 10.2.1 Algorithm Overview 10.2.2 Encoding Strategy of Particle 10.2.3 Initialization of Population 10.2.4 Evaluation of Fitness 10.2.5 Update Particle with Velocity Clamping 10.2.6 Selection and Decoding of Best Individual 10.3 Experimental Design 10.3.1 Peer Competitors 10.3.2 Parameter Settings 10.4 Experimental Results and Analysis 10.4.1 Overall Performance 10.4.2 Evolved CNN Architectures 10.4.3 Trajectory Visualization 10.5 Chapter Summary References 11 Differential Evolution for Architecture Design 11.1 Introduction 11.2 Algorithm Details 11.2.1 DECNN Algorithm Overview 11.2.2 Adjusted IP-Based Encoding Strategy 11.2.3 Population Initialization 11.2.4 Fitness Evaluation 11.2.5 DECNN DE Mutation and Crossover 11.2.6 DECNN Second Crossover 11.3 Experimental Design 11.3.1 State-of-the-Art Competitors 11.3.2 Parameter Settings 11.4 Experimental Results and Analysis 11.4.1 DECNN Versus State-of-the-Art Methods 11.4.2 DECNN Versus IPPSO 11.4.3 Evolved CNN Architecture 11.5 Chapter Summary References 12 Architecture Design for Analyzing Hyperspectral Images 12.1 Introduction 12.2 Algorithm Details 12.2.1 Algorithm Overview 12.2.2 Gene Encoding Strategy 12.2.3 Offspring Generation 12.2.4 Environmental Selection 12.3 Experimental Design 12.3.1 Benchmark Dataset 12.3.2 Peer Competitors 12.3.3 Parameter Settings 12.3.4 Training Details 12.4 Experimental Results and Analysis 12.4.1 Overall Results 12.4.2 Comparisons with Artificial-CNN 12.5 Chapter Summary References Part IV Recent Advances in Evolutionary Deep Neural Architecture Search 13 Encoding Space Based on Directed Acyclic Graphs 13.1 Introduction 13.2 Algorithm Details 13.2.1 Encoding Strategy Overview 13.2.2 Representation and Decoding Details 13.2.3 Initialization Algorithm Overview 13.2.4 Initialization Algorithm Details 13.3 Experimental Studies 13.3.1 Overview 13.3.2 Parameter Settings 13.3.3 Experimental Results 13.4 Chapter Summary References 14 End-to-End Performance Predictors 14.1 Introduction 14.2 Related Work 14.3 Algorithm Details 14.3.1 Encoding 14.3.2 Training of the Random Forest 14.3.3 Performance Prediction 14.3.4 Strength and Weakness of E2EPP 14.4 Experimental Design 14.4.1 Peer Competitors 14.4.2 Parameter Settings 14.5 Experimental Results 14.5.1 Overall Results 14.5.2 Efficiency of E2EPP 14.5.3 Effectiveness of E2EPP 14.5.4 Comparison to Radial Basis Network 14.6 Chapter Summary References 15 Deep Neural Architecture Pruning 15.1 Introduction 15.2 Background 15.3 Algorithm Details 15.3.1 Genetic Representation of an Individual 15.3.2 Population Initialization 15.3.3 Individual Evaluation 15.3.4 Selection of Knee and Boundary 15.3.5 Offspring Generation 15.3.6 Fine Tuning 15.4 Experimental Design 15.4.1 Chosen CNN Architectures for Pruning 15.4.2 Algorithm Parameters 15.5 Experimental Results and Discussion 15.5.1 Experimental Results 15.5.2 Result Discussion 15.6 Chapter Summary References 16 Deep Neural Architecture Compression 16.1 Introduction 16.2 Related Work and Motivation 16.2.1 Convolutional Neural Network Compression 16.2.2 Evolutionary Algorithms and MMD 16.3 KGEA for Compressing DNNs 16.3.1 Convolutional Filter Pruning 16.3.2 Multi-objevtive Modeling for CNN Compression 16.3.3 KGEA 16.3.4 Encoding Scheme and Genetic Operators 16.3.5 Discussion 16.4 Experimental Studies 16.4.1 Experimental Settings 16.4.2 Experiments on Fully Convolutional LeNet 16.4.3 Experiments on VGG-19 16.5 Chapter Summary References 17 Distribution Training Framework for Architecture Design 17.1 Introduction 17.2 Distributed Deep Learning 17.3 The Distributed Framework 17.3.1 Motivation 17.3.2 Framework Overview 17.3.3 Definition of the Data Packet 17.3.4 Server Node 17.3.5 Computing Node 17.4 Experimental Studies 17.4.1 Evolutionary Pelee 17.4.2 Speedup Analysis 17.4.3 Inconsistent Performance Node Analysis 17.4.4 Communication Analysis 17.4.5 Efficiency Analysis 17.5 Chapter Summary References Appendix Book Conclusions