Engineering dependable and secure machine learning systems : third international workshop, EDSMLS 2020, New York City, NY, USA, February 7, 2020 : revised selected papers

This book constitutes the revised selected papers of the Third International Workshop on Engineering Dependable and Secure Machine Learning Systems, EDSMLS 2020, held in New York City, NY, USA, in February 2020. The 7 full papers and 3 short papers were thoroughly reviewed and selected from 16 submissions. The volume presents original research on dependability and quality assurance of ML software systems, adversarial attacks on ML software systems, adversarial ML and software engineering, etc. Preface 6 Organization 8 Contents 9 Quality Management of Machine Learning Systems 10 1 Introduction 10 2 AI Is Software 11 2.1 Traditional Software Quality Management 12 2.2 Machine Learning Systems 13 3 A Quality Management Framework for ML Systems 14 3.1 Where Are the Bugs? 15 3.2 Quality Improvement Tasks for ML Systems 16 3.3 AI Trust Assessment 18 3.4 Quality Metrics 19 4 Conclusions 20 References 20 Can Attention Masks Improve Adversarial Robustness? 23 1 Introduction 23 2 Background 25 3 Our Approach 26 3.1 Dataset Creation with Foreground Masks 27 4 Results 29 5 Conclusion 29 References 30 Learner-Independent Targeted Data Omission Attacks 32 1 Related Work 33 2 Problem and Solution Approach 34 2.1 Formalization of the Problem 35 2.2 Methodology and Experimental Environment 35 3 Attack Methods 39 3.1 The KNN Attack 39 3.2 The Greedy Attack 40 3.3 The Genetic Attack 41 4 Results and Comparison 42 4.1 Analysis and Comparison of Attack Methods 43 4.2 Attack on Various Learners 44 4.3 Attack on MNIST Experiment Analysis 47 5 Conclusion 48 References 48 Extraction of Complex DNN Models: Real Threat or Boogeyman? 51 1 Introduction 51 2 Background 53 2.1 Deep Neural Networks 53 2.2 Model Extraction Attacks 53 3 Knockoff Nets Model Extraction Attack 54 3.1 Attack Description 54 3.2 Knockoff Nets: Evaluation 54 4 Detection of Knockoff Nets Attack 56 4.1 Goals and Overview 56 4.2 Training Setup 57 4.3 Experimental Results 58 5 Revisiting the Adversary Model 60 5.1 Victim Model Architecture 60 5.2 Granularity of Prediction API Output 60 5.3 Transfer Set Construction 61 5.4 Access to In-Distribution Data 62 6 Related Work 63 7 Conclusion 64 References 64 Principal Component Properties of Adversarial Samples 67 1 Introduction 67 2 Background and Prior Work 68 2.1 Adversarial Samples 68 2.2 Prior Work 69 3 Methodology 69 3.1 Threat Model 69 3.2 Defensive PCA 69 3.3 Detecting Dominant Classes 70 4 Experiments 72 4.1 Experimental Setup 72 4.2 Results 72 5 Discussion 73 6 Conclusion 74 References 74 FreaAI: Automated Extraction of Data Slices to Test Machine Learning Models 76 1 Introduction 76 2 Related Work and Background 77 3 Methodology 79 3.1 Single Feature Analysis 79 3.2 Feature Interactions Analysis 79 3.3 Heuristics for Feature and Interaction Analysis 80 3.4 Defining Data Slices Requirements 82 4 Experimental Results 83 4.1 Input 83 4.2 Finding Under-Performing Data Slices 83 4.3 Slice Significance Testing 86 5 Discussion 90 6 Conclusions 91 References 92 Density Estimation in Representation Space to Predict Model Uncertainty 93 1 Introduction 93 2 Representation Space in Classification Models 94 3 Method 97 4 Experiments 98 4.1 In-Distribution Uncertainty Detection Results 99 4.2 Out-of-Distribution Detection Results 100 5 Related Work 101 6 Conclusions 102 A Appendix 103 References 103 Automated Detection of Drift in Deep Learning Based Classifiers Performance Using Network Embeddings 106 1 Introduction 107 2 Methodology 108 2.1 The Data Drift Detection Algorithm 109 2.2 Data and Classifiers 111 2.3 Experiment Design 111 3 Experimental Results 112 4 Related Work 115 5 Conclusion 116 References 117 Quality of Syntactic Implication of RL-Based Sentence Summarization 119 1 Introduction 119 2 Related Work 120 3 Models 121 3.1 Baseline 121 3.2 Integrating Syntax 122 3.3 RL Learning 123 4 Experiments 125 5 Results and Analysis 126 6 Conclusion 131 References 132 Dependable Neural Networks for Safety Critical Tasks 135 1 Introduction 135 1.1 Training Robust Networks 136 1.2 Software Dependability 136 1.3 Adaptive Network Testing 137 1.4 Our Contributions 137 2 Methods 138 2.1 Machine Learning Dependability 138 2.2 Derivation 139 3 Experiments 141 3.1 Performance During Testing 142 3.2 Predicting Model Performance in Novel Operating Conditions 143 3.3 Performance with a Safety Function 144 4 Discussion 145 4.1 Robot Manipulation Task 145 4.2 Dependable Networks in Practical Applications 147 4.3 Future Work 147 5 Conclusions 148 References 148 Author Index 150 Front Matter ....Pages i-ix Quality Management of Machine Learning Systems (P. Santhanam)....Pages 1-13 Can Attention Masks Improve Adversarial Robustness? (Pratik Vaishnavi, Tianji Cong, Kevin Eykholt, Atul Prakash, Amir Rahmati)....Pages 14-22 Learner-Independent Targeted Data Omission Attacks (Guy Barash, Onn Shehory, Sarit Kraus, Eitan Farchi)....Pages 23-41 Extraction of Complex DNN Models: Real Threat or Boogeyman? (Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, N. Asokan)....Pages 42-57 Principal Component Properties of Adversarial Samples (Malhar Jere, Sandro Herbig, Christine Lind, Farinaz Koushanfar)....Pages 58-66 FreaAI: Automated Extraction of Data Slices to Test Machine Learning Models (Samuel Ackerman, Orna Raz, Marcel Zalmanovici)....Pages 67-83 Density Estimation in Representation Space to Predict Model Uncertainty (Tiago Ramalho, Miguel Miranda)....Pages 84-96 Automated Detection of Drift in Deep Learning Based Classifiers Performance Using Network Embeddings (Parijat Dube, Eitan Farchi)....Pages 97-109 Quality of Syntactic Implication of RL-Based Sentence Summarization (Hoa T. Le, Christophe Cerisara, Claire Gardent)....Pages 110-125 Dependable Neural Networks for Safety Critical Tasks (Molly O’Brien, William Goble, Greg Hager, Julia Bukowski)....Pages 126-140 Back Matter ....Pages 141-141