Performance Evaluation and Benchmarking : 14th TPC Technology Conference, TPCTC 2022, Sydney, NSW, Australia, September 5, 2022, Revised Selected Papers

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Preface 6 TPCTC 2022 Organization 8 Contents 10 Pick & Mix Isolation Levels: Mixed Serialization Graph Testing 11 1 Introduction 11 2 Serialization Graph Testing 13 2.1 Protocol Description 13 2.2 Algorithmic Adjustments 14 2.3 Many-Core Optimizations 15 3 Mixing in the Wild 16 4 Mixing Theory 16 4.1 System Model 16 4.2 Weak Isolation Levels 18 4.3 Mixing of Isolation Levels 18 5 Mixed Serialization Graph Testing 19 5.1 Protocol Design 19 5.2 Discussion 20 5.3 Implementation Details 20 6 Evaluation 21 6.1 Isolation 22 6.2 Contention 22 6.3 Update Rate 24 6.4 Scalability 25 7 Conclusion 25 References 26 BoDS: A Benchmark on Data Sortedness 27 1 Introduction 27 2 Data Sortedness Metrics 29 3 Generating (K,L)-Sorted Data 30 4 The Benchmark on Data Sortedness 34 5 BoDS in Action 35 5.1 Raw Ingestion Performance 36 5.2 Mixed Workload Performance 38 6 Toward Sortedness Awareness 39 7 Conclusion 40 References 41 Disaggregated Database Management Systems 43 1 Introduction 43 2 Hardware Disaggregation 45 2.1 Fungible's DPU-Based Disaggregation 46 2.2 Liqid's Composable Disaggregated Infrastructure (CDI) 47 3 Memory Disaggregation 49 4 Disaggregated Database Management Systems 50 4.1 AlloyDB 50 4.2 Rockset 52 4.3 Nova-LSM 56 5 Future Research Directions 57 References 57 TPCx-AI on NVIDIA Jetsons 59 1 Introduction 59 2 Background 61 2.1 Resource-Aware Machine Learning 61 2.2 System-on-Chip Devices 61 2.3 TPCx-AI Benchmark 62 3 Related Work 64 4 Experimental Methodology and Setup 64 4.1 Systems 64 4.2 Metrics 66 4.3 Benchmark Suite Modifications 66 5 Results 67 5.1 Whole Benchmark Run 67 5.2 Time-Breakdown per Use Case 69 5.3 Use Case 8 72 6 Discussion 72 6.1 Machine Learning on Jetsons 72 6.2 TPCx-AI Benchmark for Edge Devices 73 7 Conclusion 74 References 75 More the Merrier: Comparative Evaluation of TPCx-AI and MLPerf Benchmarks for AI 77 1 Introduction 77 2 AI Benchmarking Tools 78 3 MLPerf AI Benchmarks 79 3.1 MLPerf Training 80 3.2 MLPerf Inference 81 4 TPCx-AI Benchmark 81 4.1 TPCx-AI Metric 82 4.2 TPCx-AI Metric Analysis 83 5 MLPerf vs TPCx-AI 84 5.1 Scope and Scoring 84 5.2 Results Review 85 5.3 Code License 85 5.4 Cost 85 5.5 Efficiency Scores 86 5.6 Accelerators 86 5.7 When to Use Each Tool 86 6 Summary and Conclusions 86 References 87 Preliminary Scaling Characterization of TPCx-AI 88 1 Introduction 88 2 Related Work 91 3 TPCx-AI Kit 91 3.1 Licensing and Setup 93 3.2 Configuration 94 3.3 Benchmark Execution 94 4 Performance Results 95 4.1 System Under Test (SUT) 95 4.2 Single-node Implementation 96 4.3 Multinode Implementation 98 5 Conclusions and Future Work 101 References 101 4mbench: Performance Benchmark of Manufacturing Business Database 104 1 Introduction 104 2 Description of the Manufacturing Business Based on the 4m Model 105 3 4mbench 106 3.1 Database Schema 107 3.2 Business Case and Dataset Generation 107 3.3 Test Queries 109 4 Experimental Study 110 4.1 Dataset Generation and Loading 110 4.2 Test Queries 112 4.3 4mQ.3 on Different Settings 112 5 Related Work 115 6 Conclusion 116 A Experiment Configuration 116 A.1 SQL Description of Test Query (4mQ.3) 116 A.2 [DATE] Variable Substitution 118 A.3 Configuration Parameters of PostgreSQL 118 References 118 Benchmarking Considerations for Trustworthy and Responsible AI (Panel) 120 1 Introduction 120 2 Current State of AI Benchmarking 121 3 Deconstructing Trust and Responsibility in AI 122 4 Metrics for Trust and Responsibility 125 5 Challenges and Opportunities for Benchmarking Trust and Interpretability 126 6 Summary and Conclusions 127 References 128 TPCx-AI: First Adopter’s Experience Report 130 1 TTA 130 1.1 Background of TTA 130 1.2 Test Description 130 References 136 New Initiatives in the TPC 137 1 Introduction 137 1.1 Venturing into New Benchmark Domains 138 1.2 TPC's Benchmark Development Model Until 2013 140 1.3 Express BenchmarksTM, a Benchmark Model for Rapid Benchmark Development 143 2 TPC Express BenchmarksTM Becoming Reality 144 3 Big Data Benchmarks 144 3.1 Virtualization Benchmarks, TPCx-V and TPCx-HCI 147 3.2 TPCx-IoT 148 3.3 Enterprise and Express Class Publications 149 4 TPC Derived Benchmarks 150 4.1 Initiative to Allow the Use of TPC Benchmark Material in Non-TPC Benchmarks 151 4.2 TPC's Open Source Initiative 152 5 TPCx-AITM, First End-To-End AI benchmark Standard 153 5.1 Allowing Cloud Based Benchmark Publications 156 6 Conclusion 156 References 156 Author Index 159 This book constitutes the refereed post-conference proceedings the 14th TPC Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2022, which was held in Sydney, NSW, Australia, on September 5, 2022. The 5 revised full papers presented were carefully selected from 12 submissions. The conference focuses on Pick and Mix Isolation Levels; Benchmarking considerations for Trustworthy and Responsible AI (Panel); Preliminary Scaling Characterization with TPCx-AI and New Initiatives.