Performance Evaluation and Benchmarking for the Era of Cloud(s) : 11th TPC Technology Conference, TPCTC 2019 : Los Angeles, CA, USA, August 26, 2019 : revised selected papers

This book constitutes the refereed post-conference proceedings of the 11th TPC Technology Conference on Performance Evaluation and Benchmarking, TPCTC 2019, held in conjunction with the 45th International Conference on Very Large Databases (VLDB 2019) in August 2019. The 11 papers presented were carefully reviewed and focus on topics such as blockchain; big data and analytics; complex event processing; database Optimizations; data Integration; disaster tolerance and recovery; artificial Intelligence; emerging storage technologies (NVMe, 3D XPoint Memory etc.); hybrid workloads; energy and space efficiency; in-memory databases; internet of things; virtualization; enhancements to TPC workloads; lessons learned in practice using TPC workloads; collection and interpretation of performance data in public cloud environments. Preface 6 Organization 7 About the TPC 8 TPC 2019 Organization 10 Contents 12 Benchmarking Elastic Cloud Big Data Services Under SLA Constraints 14 1 Introduction 14 2 Workload Characterization and Generation 15 2.1 Analyzing Cloud Services Workloads 15 2.2 Workload Generation for Cloud Services 18 3 New SLA-Aware Metric for Big Data Systems 20 4 Experimental Evaluation 22 4.1 Data Loading Test 23 4.2 Elasticity Test 23 4.3 Power Test 24 4.4 Throughput Test 26 5 Results Analysis 27 5.1 BBQ++ Metric 27 5.2 Costs 28 5.3 Price/Performance Score 29 6 Related Work 30 7 Conclusions 30 References 30 Efficient Multiway Hash Join on Reconfigurable Hardware 32 1 Motivation 32 1.1 Contributions 33 1.2 Simplifying Assumptions 34 2 Background and Related Work 35 2.1 Multiway Joins 35 2.2 Hash-Join Acceleration 35 2.3 Spatially Reconfigurable Architectures 35 3 Accelerating Multiway Joins 36 3.1 Notations 36 4 Linear 3-Way Join 36 4.1 Joining Relations on Plasticine-Like Accelerator 37 4.2 Analysis of the Linear 3-Way Join 39 5 Cyclic 3-Way Join 40 5.1 Joining Relations on Plasticine-Like Accelerator 41 5.2 Analysis of Cyclic Three-Way Join 41 6 Performance Evaluation 42 6.1 Target Systems 42 6.2 Accelerator's Performance Model 43 6.3 Performance Analysis of Cascaded Binary Join 43 6.4 Performance Analysis of Linear Self Join 45 6.5 Performance Analysis of Linear Star 3-Way Join 46 7 Conclusions 47 A Performance Model of Plasticine 47 References 49 Challenges in Distributed MLPerf 52 1 Introduction 52 2 Related Work 53 3 Distributed MLPerf Analysis 54 4 Distributed Tensorflow in MLPerf 55 4.1 General Considerations 55 4.2 Generating Unofficial MLPerf Scores Using Distributed Tensorflow 55 5 Towards Improving MLPerf 57 6 Conclusions and Future Works 59 References 59 ADABench - Towards an Industry Standard Benchmark for Advanced Analytics 60 1 Introduction 60 2 Machine Learning Benchmark 62 2.1 Use Cases 62 2.2 Benchmark Details 66 3 Evaluation 68 4 Related Work 72 5 Conclusion 74 References 74 TPCx-BB (Big Bench) in a Single-Node Environment 77 1 Introduction 77 2 Background Concepts 80 3 Experimental Setup 82 4 Results 83 5 Related Work 87 6 Conclusions and Future Work 89 References 94 CBench-Dynamo: A Consistency Benchmark for NoSQL Database Systems 97 1 Introduction 97 2 Related Work 98 3 Dynamo 99 4 Consistency in Dynamo-Based Databases 99 5 CBench-Dynamo 102 5.1 CBench-Dynamo Properties 102 5.2 Architecture Specification 103 5.3 Workload 105 6 Benchmark Testing 106 6.1 Testing Architecture 106 6.2 Experiment 107 7 Conclusions and Future Work 110 References 110 Benchmarking Pocket-Scale Databases 112 1 Introduction 112 2 The Need for Mobile Benchmarking 114 3 PocketData 116 3.1 Benchmark Overview 116 3.2 Benchmark Harness 117 3.3 The PocketData Benchmark 118 4 Benchmark Results 118 4.1 Results Obtained and Analysis Method 118 4.2 Read Heavy Workloads 119 4.3 Write Heavy Workloads 121 4.4 Scan Heavy Workloads 122 4.5 Sources of Measurement Variance 124 5 Related Work 125 6 Conclusions 126 References 126 End-to-End Benchmarking of Deep Learning Platforms 129 1 Introduction 129 2 Background 130 2.1 Software Platforms 131 2.2 Hardware Environments 132 3 The Rysia Benchmarking Framework 133 3.1 Design Principles 133 3.2 Architecture 134 4 Experiment Setup 135 4.1 Workloads and Deep Learning Model Architectures 136 4.2 Training Runtime Performance 136 4.3 Inference Throughput Performance 137 5 Experiment Results 138 5.1 Training Performance 138 5.2 Inference Throughput Performance 140 6 Related Work 141 7 Conclusion 143 7.1 Summary 143 7.2 Future Work 144 References 144 Role of the TPC in the Cloud Age 146 1 Introduction 146 2 State of the Union of the TPC and TPCTC 147 3 Panel Discussions 149 4 Conclusion 151 References 151 Benchmarking Database Cloud Services 152 1 Requirements to Benchmark Software 152 2 Key Performance Indicators 153 3 The Architecture of Peakmarks Benchmark Software 154 4 Simple and Complex Workloads 155 4.1 Server Workloads 155 4.2 Storage Workloads 156 4.3 Data Load Workloads 157 4.4 Data Analytic Workloads 158 4.5 Transaction Processing Workloads 160 4.6 PL/SQL Application Performance 161 4.7 Database Service Processes 162 4.8 Order in Which Workloads Are Executed 164 5 Case Study 164 References 166 Use Machine Learning Methods to Predict Lifetimes of Storage Devices 167 1 Introduction 167 2 Methods 169 2.1 Determination of Disk Type 170 2.2 Prediction of Lifetime 171 2.3 Change of Disk Type 172 3 Implementations 172 4 Discussions and Future Work 174 A Estimation of Gaussian Parameters 174 References 176 Author Index 177 Front Matter ....Pages i-xiv Benchmarking Elastic Cloud Big Data Services Under SLA Constraints (Nicolas Poggi, Víctor Cuevas-Vicenttín, Josep Lluis Berral, Thomas Fenech, Gonzalo Gómez, Davide Brini et al.)....Pages 1-18 Efficient Multiway Hash Join on Reconfigurable Hardware (Rekha Singhal, Yaqi Zhang, Jeffrey D. Ullman, Raghu Prabhakar, Kunle Olukotun)....Pages 19-38 Challenges in Distributed MLPerf (Miro Hodak, Ajay Dholakia)....Pages 39-46 ADABench - Towards an Industry Standard Benchmark for Advanced Analytics (Tilmann Rabl, Christoph Brücke, Philipp Härtling, Stella Stars, Rodrigo Escobar Palacios, Hamesh Patel et al.)....Pages 47-63 TPCx-BB (Big Bench) in a Single-Node Environment (Dippy Aggarwal, Shreyas Shekhar, Chris Elford, Umachandar Jayachandran, Sadashivan Krishnamurthy, Jamie Reding et al.)....Pages 64-83 CBench-Dynamo: A Consistency Benchmark for NoSQL Database Systems (Miguel Diogo, Bruno Cabral, Jorge Bernardino)....Pages 84-98 Benchmarking Pocket-Scale Databases (Carl Nuessle, Oliver Kennedy, Lukasz Ziarek)....Pages 99-115 End-to-End Benchmarking of Deep Learning Platforms (Vincent Deuschle, Alexander Alexandrov, Tim Januschowski, Volker Markl)....Pages 116-132 Role of the TPC in the Cloud Age (Alain Crolotte, Feifei Li, Meikel Poess, Peter Boncz, Raghunath Nambiar)....Pages 133-138 Benchmarking Database Cloud Services (Manfred Drozd)....Pages 139-153 Use Machine Learning Methods to Predict Lifetimes of Storage Devices (Yingxuan Zhu, Bowen Jiang, Yong Wang, Tim Tingqiu Yuan, Jian Li)....Pages 154-163 Back Matter ....Pages 165-165