Performance Evaluation and Benchmarking: 15th TPC Technology Conference, TPCTC 2023, Vancouver, BC, Canada, August 28 – September 1, 2023, Revised Selected ... Notes in Computer Science Book 14247)

This book constitutes the refereed post-conference proceedings from the 15th TPC Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2023, held in Vancouver, British Columbia, Canada, during August 28 – September 1, 2023. The 9 full papers included in this book were carefully reviewed and selected from 17 submissions. These papers focus on various novel ideas and methodologies for Performance evaluation and Benchmarking in emerging technology areas. Preface Redefining Performance Evaluation and Benchmarking in the Era of Artificial Intelligence TPCTC 2023 Organization Contents Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology 1 Introduction 1.1 MVTS Anomaly Detection: a Hot Research Topic 2 Related Work 3 Evaluation Protocols for MVTS Anomaly Detection 3.1 Point-Adjust: A Non-protocol for Time Series Anomaly Detection 3.2 Point-Wise: A Good Protocol but Not for All Situations 3.3 Alternative Evaluation Protocols 4 Benchmark Datasets, Experiment Design and Algorithms Comparison 5 Algorithms 6 Discussion: Towards Better Practices for MVTS Anomaly Detection 6.1 Evaluation Protocols and Metrics 6.2 Datasets and Experiment Design 6.3 Algorithms References A Comprehensive Study on Benchmarking Permissioned Blockchains 1 Introduction 2 Permissioned Blockchains 2.1 Hyperledger Fabric 2.2 Corda 2.3 Multichain 2.4 Quorum 2.5 Diem 3 Benchmarking Guidelines 3.1 System Configuration 3.2 Parameter Tuning 3.3 Workloads and Use Cases 3.4 Performance Metrics 4 Case Study 5 Related Work 6 Conclusion References Benchmarking Generative AI Performance Requires a Holistic Approach 1 Introduction 2 Benchmarking Approaches for AI Systems 3 LLMs Used for Benchmarking and Associated Responsible AI Concerns 4 Holistic Approach for Benchmarking Generative AI System 5 Applying LLMs to DBMSs 6 Summary and Conclusions References Graph Stores with Application-Level Query Result Caches 1 Introduction 2 Caching and Strong Consistency 2.1 Write-Around and Strong Consistency 3 eBay's Graph Store and Query Result Cache 3.1 Caching Query Results 4 An Evaluation 4.1 Results 5 Conclusions and Future Research Directions References Chaosity: Understanding Contemporary NUMA-Architectures 1 Introduction 2 Motivation: Rising Entropy in the Hardware Landscape 2.1 Hierarchical NUMA 2.2 On-Chip Heterogeneity 2.3 Data Highways: Interconnect 2.4 Systems with Complex Data Access 3 Chaosity Framework Understand Thy Hardware 4 Heterogeneous Compute Units Apple M1 Pro Silicon 4.1 Interference in Unified Memory 5 Chiplet-Based Server AMD EPYC 5.1 Hierarchical NUMA 5.2 Bandwidth Interference – Interconnect & Memory 6 Discussion 7 Conclusion References Benchmarking Large Language Models: Opportunities and Challenges 1 Introduction 2 Difficulties in LLM Benchmarking 3 LLM Models 3.1 BERT 3.2 GPT-3 3.3 PaLM 3.4 LLaMA 3.5 GPT-J 3.6 BLOOM 4 LLM Datasets 5 LLM Training: Benchmarking Considerations 6 LLM Inference: Benchmarking Considerations 7 LLM Performance Evaluation 8 Current Benchmarking Efforts 8.1 MLPerf Training V3.0 LLM Benchmark Design 8.2 MLPerf Training V3.0 LLM Benchmark Results 8.3 MLPerf Inference 8.4 Other AI Benchmarking Suites 9 Summary and Conclusions References The Linked Data Benchmark Council (LDBC): Driving Competition and Collaboration in the Graph Data Management Space 1 Introduction 2 The LDBC Organization 2.1 History of the Organization 2.2 Organizational Structure and Operations 2.3 Liaison with ISO on Standard Query Languages (GQL, SQL/PGQ) 2.4 Technical User Community (TUC) Meetings 3 Benchmarks 3.1 Benchmark Terminology 3.2 Social Network Benchmark (SNB) Suite 3.3 Semantic Publishing Benchmark (SPB) 3.4 Graphalytics 3.5 FinBench 4 Benchmark Processes 4.1 Defining New LDBC Benchmarks 4.2 Auditing Process 4.3 Trademark 5 Benchmarking Lessons Learnt 6 Working Groups 6.1 Graph Query Languages Working Group 6.2 Formal Semantics Working Group (FSWG) 6.3 Property Graph Schema Working Group (PGSWG) 6.4 LDBC Extended GQL Schema (LEX) 7 Conclusion and Future Outlook References The LDBC Social Network Benchmark Interactive Workload v2: A Transactional Graph Query Benchmark with Deep Delete Operations 1 Introduction 2 Design Principles 2.1 Relevance: Choke Point-Based Design Process and Domain 2.2 Portability: Implementation Rules 2.3 Scalability: Scalable Data Generator and Driver 2.4 Simplicity: Stable Query Runtimes, Single Output Metric 3 Data Sets 3.1 Graph Schema 3.2 Distribution and Correlations 3.3 Graph Generation Stages 3.4 Scale Factors 4 Operations 4.1 Complex Read Queries () 4.2 Short Read Queries () 4.3 Insert Operations () 4.4 Delete Operations () 5 Workload Scheduling and Benchmark Driver 5.1 Scheduling Operations 5.2 Driver 6 Parameter Curation 6.1 Building Blocks for Parameter Curation 6.2 Parameter Curation for Relational Queries 6.3 Parameter Curation for Path-Finding Queries 6.4 Query Variants 6.5 Parameter Generator Implementation 7 Using the SNB Interactive v2 Workload 7.1 Implementations 7.2 Auditing 8 Related Work: LDBC FinBench 9 Conclusion References A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and Tools 1 Introduction 1.1 Contribution 1.2 Related Work 1.3 Motivation 2 Solution Concept 3 Experiments 3.1 YCSB: Threads vs Processes 3.2 HammerDB's TPC-C 3.3 Benchbase's TPC-C 3.4 TPC-H: Throughput of Loading and Execution 4 Discussion 5 Conclusion References Author Index