Benchmarking, Measuring, and Optimizing : 14th BenchCouncil International Symposium, Bench 2022, Virtual Event, November 7-9, 2022, Revised Selected Papers

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Preface Organization Invited Talks BenchCouncil Achievement Award Lecture: Benchmarking: An Incomparable Science? BenchCouncil Rising Star Award Lecture: Rethinking Benchmarking in AI: Evaluation-as-a-Service and Dynamic Adversarial Data Collection Towards Fair Machine Learning with Imperfect Information Contents Architecture and System A Quantitative Analysis of OpenMP Task Runtime Systems 1 Introduction 2 Related Work 3 OMPTB: The OpenMP Task Benchmark 4 Experimental Setup 5 Experimental Results 5.1 General Experimental Factors 5.2 Homogeneous Workloads 5.3 Homogeneous Case: Correlation Analysis 5.4 Heterogeneous Workloads 6 Conclusions References EAIBench: An Energy Efficiency Benchmark for AI Training 1 Introduction 2 Related Work 2.1 Benchmark 2.2 Metrics 3 Methodology and Implementation 3.1 Metric and Measurement 3.2 Benchmarks and Configuration Space 3.3 Monitor and Analyzer 3.4 System Under Test 4 Experiment and Result 4.1 Experimental Methodology 4.2 Stability of Epochs 4.3 Model Parameters Analysis 4.4 GPU Comparison 4.5 Different Layers Analysis 5 Conclusion References MSDBench: Understanding the Performance Impact of Isolation Domains on Microservice-Based IoT Deployments 1 Introduction 2 Related Work 3 Benchmark Design 4 Empirical Evaluation 4.1 Microbenchmark Results 4.2 End to End Benchmark Results 4.3 Throughput Results 5 Conclusion References Algorithm and Dataset ShoeMaster: A Benchmark for Sketch2Image Translation of Shoes 1 Introduction 2 Related Work 2.1 Fashion Datasets 2.2 Sketch2Image Translation 3 The ShoeMaster Benchmark 3.1 Shoes Category Knowledge Hierarchy 3.2 Dataset Construction 3.3 Evaluation Metric 4 Experiments 4.1 Approaches 4.2 Methodology 4.3 Results and Analysis 5 Conclusion References Open Source Software Supply Chain Recommendation Based on Heterogeneous Information Network 1 Related Work 1.1 GitHub Open Source Data Collection and Mining 1.2 GitHub Project Recommendation 1.3 Recommendation Methods Based on Heterogeneous Information Networks 2 Proposed Framework 2.1 Data Preprocessing 2.2 Oswhin Construction 2.3 Metapath Selection 2.4 Oswhin-Based Embedding 2.5 Personalized Recommendation 3 Experiment 3.1 Dataset 3.2 Evaluation Metrics 3.3 Experimental Results and Analysis 4 Conclusion References BasicTS: An Open Source Fair Multivariate Time Series Prediction Benchmark 1 Introduction 2 Related Works 3 Benchmark Building 3.1 Design Thoughts 3.2 Implementation of BasicTS 4 Evaluation 4.1 Experimental Setup 4.2 Experimental Results 4.3 Review 5 Conclusion 6 Future Works References Benchmarking Object Detection Models with Mummy Nuts Datasets 1 Introduction 2 Background and Motivation 3 Requirements of Detecting Mummy Nuts with CNNs 3.1 Overview of Object Detection Models 3.2 Requirements of Benchmarking Models 4 Benchmarking Methodology 4.1 Proposed Datasets: Original and Augmented 4.2 Metrics 4.3 Proposed Tools 5 Experiments 5.1 Platform Selection 5.2 Results 5.3 Observations and Discussion 6 Related Work 7 Conclusion and Future Work References Network and Memory An Analysis of Long-Tailed Network Latency Distribution and Background Traffic on Dragonfly+ 1 Introduction 1.1 Motivations 1.2 Contributions 2 Related Work 3 Experimental Setup 3.1 Computing 3.2 Network 3.3 Microbenchmarks and Applications 4 Network Latency Distribution Analysis 4.1 Job Placement Locality and Performance Variability 5 Background Traffic Analysis 5.1 Background Traffic Heuristic 5.2 Correlation Analysis 5.3 The Impact of Background Traffic on Long-Tail 5.4 Application Analysis 6 Discussion 7 Conclusion References MCCBench: A C10M Benchmark Oriented to Interactive Network Services 1 Introduction 2 Motivation 2.1 Benchmark Design 2.2 Benchmark Implementation 3 MCCBench Design 3.1 Workload Definition 3.2 Service Framework 3.3 Performance Evaluation Criteria 4 MCCBench Implementation 4.1 Workloads 4.2 Architecture 4.3 Case Study 5 Evaluation 5.1 Experiment Setup 5.2 Results 5.3 Expansion Experiment 6 Conclusion References STAMP-Rust: Language and Performance Comparison to C on Transactional Benchmarks 1 Introduction 2 Background and Related Work 2.1 The STAMP Benchmark Suite 2.2 Software Transactional Memory in Rust 3 STAMP on Safe Rust 3.1 Type-Level Safety 3.2 Composable Transactions 3.3 The Overhead of Safety 3.4 The Complexity of Using Associative Arrays 4 Analysis of Automatically Generated Benchmarks 5 Evaluation 5.1 Methodology 5.2 Performance Comparison 5.3 Qualitative Analysis 6 Conclusion References Author Index This book constitutes the refereed post-conference proceedings of the 14th BenchCouncil International Symposium on Benchmarking, Measuring, and Optimization, Bench 2022, held virtually in November 2022. The 10 revised full papers presented were carefully reviewed and selected from 20 submissions. The papers are organized in topical sections named: Architecture and System, Algorithm and Dataset, Network and Memory.