Engineering Design Optimization

Based on course-tested material, this rigorous yet accessible graduate textbook covers both fundamental and advanced optimization theory and algorithms. It covers a wide range of numerical methods and topics, including both gradient-based and gradient-free algorithms, multidisciplinary design optimization, and uncertainty, with instruction on how to determine which algorithm should be used for a given application. It also provides an overview of models and how to prepare them for use with numerical optimization, including derivative computation. Over 400 high-quality visualizations and numerous examples facilitate understanding of the theory, and practical tips address common issues encountered in practical engineering design optimization and how to address them. Numerous end-of-chapter homework problems, progressing in difficulty, help put knowledge into practice. Accompanied online by a solutions manual for instructors and source code for problems, this is ideal for a one- or two-semester graduate course on optimization in aerospace, civil, mechanical, electrical, and chemical engineering departments. Contents Preface Acknowledgements 1 Introduction 1.1 Design Optimization Process 1.2 Optimization Problem Formulation 1.2.1 Design Variables 1.2.2 Objective Function 1.2.3 Constraints 1.2.4 Optimization Problem Statement 1.3 Optimization Problem Classification 1.3.1 Smoothness 1.3.2 Linearity 1.3.3 Multimodality and Convexity 1.3.4 Deterministic versus Stochastic 1.4 Optimization Algorithms 1.4.1 Order of Information 1.4.2 Local versus Global Search 1.4.3 Mathematical versus Heuristic 1.4.4 Function Evaluation 1.4.5 Stochasticity 1.4.6 Time Dependence 1.5 Selecting an Optimization Approach 1.6 Notation 1.7 Summary Problems 2 A Short History of Optimization 2.1 The First Problems: Optimizing Length and Area 2.2 Optimization Revolution: Derivatives and Calculus 2.3 The Birth of Optimization Algorithms 2.4 The Last Decades 2.5 Toward a Diverse Future 2.6 Summary 3 Numerical Models and Solvers 3.1 Model Development for Analysis versus Optimization 3.2 Modeling Process and Types of Errors 3.3 Numerical Models as Residual Equations 3.4 Discretization of Differential Equations 3.5 Numerical Errors 3.5.1 Roundoff Errors 3.5.2 Truncation Errors 3.5.3 Iterative Solver Tolerance Error 3.5.4 Programming Errors 3.6 Overview of Solvers 3.7 Rate of Convergence 3.8 Newton-Based Solvers 3.9 Models and the Optimization Problem 3.10 Summary Problems 4 Unconstrained Gradient-Based Optimization 4.1 Fundamentals 4.1.1 Derivatives and Gradients 4.1.2 Curvature and Hessians 4.1.3 Taylor Series 4.1.4 Optimality Conditions 4.2 Two Overall Approaches to Finding an Optimum 4.3 Line Search 4.3.1 Sufficient Decrease and Backtracking 4.3.2 Strong Wolfe Conditions 4.3.3 Interpolation for Pinpointing 4.4 Search Direction 4.4.1 Steepest Descent 4.4.2 Conjugate Gradient 4.4.3 Newton's Method 4.4.4 Quasi-Newton Methods 4.4.5 Limited-Memory Quasi-Newton Methods 4.5 Trust-Region Methods 4.5.1 Quadratic Model with Spherical Trust Region 4.5.2 Trust-Region Sizing Strategy 4.5.3 Comparison with Line Search Methods 4.6 Summary Problems 5 Constrained Gradient-Based Optimization 5.1 Constrained Problem Formulation 5.2 Understanding n-Dimensional Space 5.3 Optimality Conditions 5.3.1 Equality Constraints 5.3.2 Inequality Constraints 5.3.3 Meaning of the Lagrange Multipliers 5.3.4 Post-Optimality Sensitivities 5.4 Penalty Methods 5.4.1 Exterior Penalty Methods 5.4.2 Interior Penalty Methods 5.5 Sequential Quadratic Programming 5.5.1 Equality Constrained SQP 5.5.2 Inequality Constraints 5.5.3 Merit Functions and Filters 5.5.4 Quasi-Newton SQP 5.5.5 Algorithm Overview 5.6 Interior-Point Methods 5.6.1 Modifications to the Basic Algorithm 5.6.2 SQP Comparisons and Examples 5.7 Constraint Aggregation 5.8 Summary Problems 6 Computing Derivatives 6.1 Derivatives, Gradients, and Jacobians 6.2 Overview of Methods for Computing Derivatives 6.3 Symbolic Differentiation 6.4 Finite Differences 6.4.1 Finite-Difference Formulas 6.4.2 The Step-Size Dilemma 6.4.3 Practical Implementation 6.5 Complex Step 6.5.1 Theory 6.5.2 Complex-Step Implementation 6.6 Algorithmic Differentiation 6.6.1 Variables and Functions as Lines of Code 6.6.2 Forward-Mode AD 6.6.3 Reverse-Mode AD 6.6.4 Forward Mode or Reverse Mode? 6.6.5 AD Implementation 6.6.6 AD Shortcuts for Matrix Operations 6.7 Implicit Analytic Methods—Direct and Adjoint 6.7.1 Residuals and Functions 6.7.2 Direct and Adjoint Derivative Equations 6.7.3 Direct or Adjoint? 6.7.4 Adjoint Method with AD Partial Derivatives 6.8 Sparse Jacobians and Graph Coloring 6.9 Unified Derivatives Equation 6.9.1 UDE Derivation 6.9.2 UDE for Mixed Implicit and Explicit Components 6.9.3 Recovering AD 6.10 Summary Problems 7 Gradient-Free Optimization 7.1 When to Use Gradient-Free Algorithms 7.2 Classification of Gradient-Free Algorithms 7.3 Nelder–Mead Algorithm 7.4 Generalized Pattern Search 7.5 DIRECT Algorithm 7.6 Genetic Algorithms 7.6.1 Binary-Encoded Genetic Algorithms 7.6.2 Real-Encoded Genetic Algorithms 7.6.3 Constraint Handling 7.6.4 Convergence 7.7 Particle Swarm Optimization 7.8 Summary Problems 8 Discrete Optimization 8.1 Binary, Integer, and Discrete Variables 8.2 Avoiding Discrete Variables 8.3 Branch and Bound 8.3.1 Binary Variables 8.3.2 Integer Variables 8.4 Greedy Algorithms 8.5 Dynamic Programming 8.6 Simulated Annealing 8.7 Binary Genetic Algorithms 8.8 Summary Problems 9 Multiobjective Optimization 9.1 Multiple Objectives 9.2 Pareto Optimality 9.3 Solution Methods 9.3.1 Weighted Sum 9.3.2 Epsilon-Constraint Method 9.3.3 Normal Boundary Intersection 9.3.4 Evolutionary Algorithms 9.4 Summary Problems 10 Surrogate-Based Optimization 10.1 When to Use a Surrogate Model 10.2 Sampling 10.2.1 Latin Hypercube Sampling 10.2.2 Low-Discrepancy Sequences 10.3 Constructing a Surrogate 10.3.1 Linear Least Squares Regression 10.3.2 Maximum Likelihood Interpretation 10.3.3 Nonlinear Least Squares Regression 10.3.4 Cross Validation 10.3.5 Common Basis Functions 10.4 Kriging 10.5 Deep Neural Networks 10.6 Optimization and Infill 10.6.1 Exploitation 10.6.2 Efficient Global Optimization 10.7 Summary Problems 11 Convex Optimization 11.1 Introduction 11.2 Linear Programming 11.3 Quadratic Programming 11.4 Second-Order Cone Programming 11.5 Disciplined Convex Optimization 11.6 Geometric Programming 11.7 Summary Problems 12 Optimization Under Uncertainty 12.1 Robust Design 12.2 Reliable Design 12.3 Forward Propagation 12.3.1 First-Order Perturbation Method 12.3.2 Direct Quadrature 12.3.3 Monte Carlo Simulation 12.3.4 Polynomial Chaos 12.4 Summary Problems 13 Multidisciplinary Design Optimization 13.1 The Need for MDO 13.2 Coupled Models 13.2.1 Components 13.2.2 Models and Coupling Variables 13.2.3 Residual and Functional Forms 13.2.4 Coupled System Structure 13.2.5 Solving Coupled Numerical Models 13.2.6 Hierarchical Solvers for Coupled Systems 13.3 Coupled Derivatives Computation 13.3.1 Finite Differences 13.3.2 Complex Step and AD 13.3.3 Implicit Analytic Methods 13.4 Monolithic MDO Architectures 13.4.1 Multidisciplinary Feasible 13.4.2 Individual Discipline Feasible 13.4.3 Simultaneous Analysis and Design 13.5 Distributed MDO Architectures 13.5.1 Collaborative Optimization 13.5.2 Analytical Target Cascading 13.5.3 Bilevel Integrated System Synthesis 13.5.4 Asymmetric Subspace Optimization 13.5.5 Other Distributed Architectures 13.6 Summary Problems A Mathematics Background A.1 Taylor Series Expansion A.2 Chain Rule, Total Derivatives, and Differentials A.3 Matrix Multiplication A.3.1 Vector-Vector Products A.3.2 Matrix-Vector Products A.3.3 Quadratic Form (Vector-Matrix-Vector Product) A.4 Four Fundamental Subspaces in Linear Algebra A.5 Vector and Matrix Norms A.6 Matrix Types A.7 Matrix Derivatives A.8 Eigenvalues and Eigenvectors A.9 Random Variables B Linear Solvers B.1 Systems of Linear Equations B.2 Conditioning B.3 Direct Methods B.4 Iterative Methods B.4.1 Jacobi, Gauss–Seidel, and SOR B.4.2 Conjugate Gradient Method B.4.3 Krylov Subspace Methods C Quasi-Newton Methods C.1 Broyden's Method C.2 Additional Quasi-Newton Approximations C.2.1 Davidon–Fletcher–Powell Update C.2.2 BFGS C.2.3 Symmetric Rank 1 Update C.2.4 Unification of SR1, DFP, and BFGS C.3 Sherman–Morrison–Woodbury Formula D Test Problems D.1 Unconstrained Problems D.1.1 Slanted Quadratic Function D.1.2 Rosenbrock Function D.1.3 Bean Function D.1.4 Jones Function D.1.5 Hartmann Function D.1.6 Aircraft Wing Design D.1.7 Brachistochrone Problem D.1.8 Spring System D.2 Constrained Problems D.2.1 Barnes Problem D.2.2 Ten-Bar Truss Bibliography Index "Based on course-tested material, this rigorous yet accessible graduate textbook covers both fundamental and advanced optimization theory and algorithms. It covers a wide range of numerical methods and topics, including both gradient-based and gradient-free algorithms, multidisciplinary design optimization, and uncertainty, with instruction on how to determine which algorithm should be used for a given application. It also provides an overview of models and how to prepare them for use with numerical optimization, including derivative computation. Over 200 high-quality visualizations and numerous examples facilitate understanding of the theory, and practical tips address common issues encountered in practical engineering design optimization and how to address them. Numerous end-of-chapter homework problems, progressing in difficulty, help put knowledge into practice"-- Provided by publisher