Benchmarking, Measuring, and Optimizing: 15th BenchCouncil International Symposium, Bench 2023, Sanya, China, December 3–5, 2023, Revised Selected Papers (Lecture Notes in Computer Science, 14521)

This book constitutes the refereed proceedings of the 14th BenchCouncil International Symposium on Benchmarking, Measuring, and Optimizing, Bench 2023, held in Sanya, China, during December 3–5, 2023. The 11 full papers included in this book were carefully reviewed and selected from 20 submissions. The Bench symposium invites papers that exhibit three defining characteristics: (1) It provides a high-quality, single-track forum for presenting results and discussing ideas that further the knowledge and understanding of the benchmark community; (2) It is a multi-disciplinary conference, attracting researchers and practitioners from different communities, including architecture, systems, algorithms, and applications; (3) The program features both invited and contributed talks. Preface Organization Invited Talks BenchCouncil Achievement Award Lecture: Essentially, All Models Are Wrong, but Some Are Useful Designing High-Performance and Scalable Middleware and Benchmarks for HPC, AI, and Data Sciences Contents ICBench: Benchmarking Knowledge Mastery in Introductory Computer Science Education 1 Introduction 2 Overview 2.1 The RPCC Model for Knowledge Points 2.2 Metrics 2.3 The Process of the Knuth Test 3 Methodology 3.1 Encoding Questions 3.2 Constructing Seed Question Set 3.3 Personalizing Question Set 4 Implementation 5 Evaluation 5.1 Student Performance 5.2 System Overhead 6 Related Work 6.1 The Personalization of Question Set 6.2 The Coverage of Question Set 7 Conclusion References Generating High Dimensional Test Data for Topological Data Analysis 1 Introduction 2 Background 3 Related Work 4 Overview of the Approach 4.1 HMC Sampling of Algebraic Varieties 4.2 Sensitivity of HMC Parameters 4.3 Combine, Rotate, and Embed Constructions 5 Ground Truth Validation 6 Examples of Data Generation 6.1 Data Generation in R2 and R3 6.2 Data Generation in Rn for n>3 7 Limitations and Challenges 8 Conclusions References Does AI for Science Need Another ImageNet or Totally Different Benchmarks? A Case Study of Machine Learning Force Fields 1 Introduction 2 Preliminaries 3 Related Works 4 Benchmarking MLFF 4.1 Sample Efficiency 4.2 Time-Series Extrapolation 4.3 Cross-Molecule Generalization Benchmarks 5 Conclusion References MolBench: A Benchmark of AI Models for Molecular Property Prediction 1 Introduction 2 Related Work 3 Methodology 3.1 Datasets 3.2 Models 3.3 Metrics 4 Experiments 4.1 Results 4.2 Stability 4.3 Task Coverage 4.4 API 5 Concluding Remarks References Cross-Layer Profiling of IoTBench 1 Introduction 2 Background and Related Work 2.1 IoT Benchmarks 2.2 Workload Characterizations 3 Evaluation Methodology 3.1 Methodology 3.2 Metric 3.3 Tool 4 Experiment and Result 4.1 Experiment on ARM Platform 4.2 Comparative Experiment of ARM and X86 4.3 Experiment Summary 5 Conclusions References MMDBench: A Benchmark for Hybrid Query in Multimodal Database 1 Introduction 2 Related Work 2.1 Single Model Benchmark Programs 2.2 Multi-model Benchmark Programs 3 Data Modalities 4 Data Generator 4.1 Constructing Data 4.2 Scaling Data from Different Modalities 5 Workload 5.1 Framework of Benchmark Program 5.2 Multimodal Data Schema in Social Network 5.3 Hybrid Query in Social Network 6 Evaluation 6.1 Polyglot Persistence System for Evaluation 6.2 Data Generation 6.3 Baseline Evaluation 6.4 Latency of Polyglot Persistence 6.5 Scaling Data Evaluation 6.6 Summary of Evaluation 7 Conclusion References Benchmarking Modern Databases for Storing and Profiling Very Large Scale HPC Communication Data 1 Introduction and Motivation 1.1 Problem Statement 1.2 Contributions 2 Methodology for Realistic Benchmarking of Large-Scale HPC Profiling Data 2.1 Data Schema and Table Design 2.2 Data Querying Methodology 2.3 Data Insertion Methodology 3 Performance Evaluation Methodology 3.1 Evaluation Considerations 4 Database Performance Evaluation 4.1 Experimental Setup 4.2 Impact of Parallelism on Data Insertion 4.3 Impact of Batching Rows on Data Insertion Performance 4.4 Evaluation of Scaling Users Querying Data 4.5 Evaluation of Scaling Insertion Processes 4.6 Scaling Simultaneous Insertion and Querying Processes 5 HPC Tool Integration and Evaluation 5.1 In-Production Performance Evaluation of Database Options 5.2 Evaluation of Disk Space Usage for Each Table 6 Conclusion and Future Work References A Linear Combination-Based Method to Construct Proxy Benchmarks for Big Data Workloads 1 Introduction 2 Proxy Benchmark Generation Methodology 2.1 Problem Description 2.2 Basic Block 2.3 Linear Combination Method 2.4 Algorithm Flow 3 Evaluation 3.1 Experiment Setups 3.2 Accuracy 3.3 Running Time 3.4 Summary 4 Case Studies 4.1 Prefetch Strategy Setting 4.2 Hyper-Threading Technology 4.3 Summary 5 Related Work 6 Conclusion References AGIBench: A Multi-granularity, Multimodal, Human-Referenced, Auto-Scoring Benchmark for Large Language Models 1 Introduction 2 Related Work 3 The Design and Implementation 3.1 Methodology 3.2 AGIBench Design and Implementation 4 Evaluation 4.1 Evaluation Methodology 4.2 Experiment Setup 4.3 Evaluation Results 5 Conclusion References Automated HPC Workload Generation Combining Statistical Modeling and Autoregressive Analysis 1 Introduction 2 Preliminary 3 Modeling Methodology 3.1 Job Arrival Model 3.2 Job Attribute Model 4 Evaluation 4.1 Evaluation of Job Arrival Generation 4.2 Evaluation of Workload Generation 4.3 Use Case of Workload Generation 5 Related Work 5.1 Workload Modeling in HPC 5.2 Workload Modeling in Cloud Computing 5.3 Resource Management in HPC 6 Conclusion References Hmem: A Holistic Memory Performance Metric for Cloud Computing 1 Introduction 2 Related Work 2.1 Cloud Memory Performance Evaluation 2.2 Memory Performance Benchmarking 2.3 Dimensional Analysis 3 Cloud Server Selection Framework 4 Motivating Example 5 Holistic Memory Performance Metric 5.1 MLP Metric 5.2 Power Metric 5.3 Hmem Metric 6 Evaluation 6.1 Experimental Setup 6.2 Physical Meaning Evaluation 6.3 Proximity of Holistic Metrics and Workload Performance 7 Threats to Validity 8 Practical Experience 9 Conclusion and Future Work References Author Index