Database Benchmarking for Performance Masterclass: Session 1 - Benchmarking Fundamentals
0 likes79 views
Why and how to run performance benchmarks.
![Database Performance Metrics: Throughput
■ Throughput [ops/s]: average over the runtime, time-series over the
runtime
24](https://image.slidesharecdn.com/part1seybolddatabasebenchmarkingforpeformancemasterclass-250619170827-e1230d5d/85/Database-Benchmarking-for-Performance-Masterclass-Session-1-Benchmarking-Fundamentals-24-320.jpg)
![Database Performance Metrics: Latency
■ Remember of client- vs server-side latency
■ Latency [ms]: P95, P99, time-series over the runtime
■ See also:
https://bravenewgeek.com/everything-you-know-about-latency-is
-wrong/
■ "How NOT to Measure Latency" by Gil Tene
https://www.youtube.com/watch?v=lJ8ydIuPFeU
25](https://image.slidesharecdn.com/part1seybolddatabasebenchmarkingforpeformancemasterclass-250619170827-e1230d5d/85/Database-Benchmarking-for-Performance-Masterclass-Session-1-Benchmarking-Fundamentals-25-320.jpg)
Database Benchmarking for Performance Masterclass: Session 1 - Benchmarking Fundamentals
- 4. What is database benchmarking? 4 vs. “There are lies, damn lies, and then there are performance measures”, Jim Gray
- 5. What is database benchmarking? Database benchmarking is the systematic measurement and evaluation of database management systems (DBMS) under controlled workloads to assess their performance in terms of throughput, latency, scalability, and resource usage. ■ Jim Gray (ed.): The Benchmark Handbook for Database and Transaction Processing Systems. Morgan Kaufmann, 1991 5
- 6. What is database benchmarking? Technical benchmarking vs. feature benchmarking ■ Technical benchmarking: executing actual measurements against different database systems to compare technical features such as performance ■ Feature benchmarking: comparing database system features based on existing documentation and specifications ■ For a DBaaS feature benchmark see: https://www.scylladb.com/tech-talk/a-benchmark-of-the-dbaas-m arket/ 6
- 9. Why Database Benchmarking? The database provider perspective 9
- 10. Why Database Benchmarking? The performance engineering perspective ■ “… we recommend measuring the performance of applications to identify appropriate instance types … we also recommend rigorous load/scale testing …“ – AWS ■ “… measure everything, assume nothing …“ – MongoDB ■ “Benchmarking will help you to fail fast and recover fast before it’s too late.” – ScyllaDB ■ “… approach performance systematically and do not settle for random Googling …“ – Peter Zaitsev (Percona) 10
- 11. Why Database Benchmarking? The research perspective ■ “benchmarks tremendously helped move forward the database industry and the database research community” — Seattle Report on Database Research, 2022 ■ “An important area of future research will be creating synthetic benchmarks that accurately mirror real-world use cases” — Cambridge Report on Database Research, 2025 ■ Dedicated (database) performance benchmarking conferences such as TPCTC and DBTEST as well as dedicated performance tracks on the major database conferences such as VLDB and SIGMOD 11
- 12. Why Database Benchmarking? ■ Comparing the performance of different database systems ■ Comparing the performance of different physical/cloud resources for a specific database system ■ Comparing the performance of a database systems for different workloads ■ Optimizing the performance of an existing database deployment by database configuration/OS/hardware tuning 12
- 14. How to design a Database Benchmark Pillars of a meaningful database benchmark ■ Workload ■ Metrics ■ Database deployment ■ Benchmark execution 14
- 15. Benchmarking Suites & Workloads A benchmarking suite generates the target database workload(s) and adds additional features such as metric reporting, metric processing and scalability of the workload execution. A workload emulates an application specific database workload as realistically as possible. 15
- 16. Database Workloads What is your application workload? ■ Online Transaction Processing (OLTP) ■ transactional workloads on operational data (~GBs) ■ typical queries include primary key reads, simple joins, inserts, updates ■ high number of concurrent requests ■ exemplary benchmark suites: TPC-C, YCSB 16
- 17. Database Workloads What is your application workload? ■ Online Analytical Processing (OLAP) ■ transactional workloads on analytical data (~TBs) ■ typical queries include complex joins with aggregates and extensive secondary index reads, no updates and periodic batch inserts ■ Low number of concurrent requests ■ exemplary benchmark suites: TPC-DS, TPC-H 17
- 18. Database Workloads What is your application workload? ■ Hybrid Transactional Analytical Processing (HTAP) ■ operational and transactional workloads on operational data (~GBs to TBs) ■ typical queries are a combination of OLTP and OLAP ■ Low number of analytical and high number of operational requests in parallel ■ exemplary benchmark suites: HTAPBench, Web3Bench 18
- 19. Selecting a Database Benchmarking Suites ■ Database benchmarking landscape evolves with the database system landscape ■ Research driven benchmarking suites: YCSB, BenchBase ■ Consortium driven benchmarking suites: TPC, LDBC ■ Vendor driven benchmarking suites: TSBS, ClickBench, VectorDBBench 19
- 20. Selecting a Database Benchmarking Suites ■ benchANT maintains a list of database benchmarking suites ■ 46 database benchmarking suites ■ OLTP, OLAP, HTAP, NoSQL, Time-Series, Vector ■ https://benchant.com/blog/benchmarking-suites 20
- 21. Database Benchmarking Suites Do I need to create a custom benchmarking suite for my application? ■ Greenfield application: start with an existing benchmarking suite ■ Brownfield application: select an existing benchmarking suite that is highly configurable or easily extensible ■ Legacy application: select an easily extensible benchmark suite as starting point 21
- 22. Creating a Custom Database Workload What do I need to know to create a custom workload? ■ Data model ■ Common queries and their data access pattern (new vs. old vs. random, hot data vs. cold data, …) ■ Data set size ■ Number of concurrent users ■ Workload intensity pattern (constant, spikey, growing, …) ■ Extensible benchmark suites: e.g. YCSB ■ Frameworks to start from scratch: e.g. Locust 22
- 23. Database Performance Metrics Client-side performance metrics are reported by the benchmarking suite while server-side performance metrics can be exposed on the database side. Performance metrics can be reported per query and/or combined over all queries 23
- 24. Database Performance Metrics: Throughput ■ Throughput [ops/s]: average over the runtime, time-series over the runtime 24
- 25. Database Performance Metrics: Latency ■ Remember of client- vs server-side latency ■ Latency [ms]: P95, P99, time-series over the runtime ■ See also: https://bravenewgeek.com/everything-you-know-about-latency-is -wrong/ ■ "How NOT to Measure Latency" by Gil Tene https://www.youtube.com/watch?v=lJ8ydIuPFeU 25
- 26. Database Performance Metrics In addition to the core database performance metrics, the following metrics are also relevant to every database benchmark: ■ Price-Performance ratio: Throughput in relation to the operational database costs ■ System utilization: CPU, memory, network, IOPS There are also benchmarking suite specific performance metrics such as tpm-C of the TPC-C or recall for VectorDBBench 26
- 27. Benchmarking Beyond Performance: Scalability Scalability: benchmarking the performance of a database system for growing workloads and growing database resources (i.e. scaling the database vertically and/or horizontally) ■ Good horizontal scalability is indicated by a constant latency as well as throughput increasing proportionally with the cluster sizes 27
- 28. Benchmarking Beyond Performance: Scalability Scalability: benchmarking the performance of a database system for growing workloads and growing database resources (i.e. scaling the database vertically and/or horizontally) ■ Good horizontal scalability is indicated by a constant latency as well as throughput increasing proportionally with the cluster sizes ■ Scalability does not consider the transition from cluster size A to B 28
- 29. Benchmarking Beyond Performance: Elasticity In contrast to scalability benchmarks, elasticity benchmarks measure the ability to handle growing/fluctuating workloads at runtime by adapting the database cluster size accordingly. Elasticity metrics are: ■ Elastic speedup ■ Data distribution impact on performance ■ Data distribution time ■ More details: Seybold, Daniel, et al. "Kaa: Evaluating elasticity of cloud-hosted dbms." 2019 IEEE International Conference on Cloud Computing Technology and Science (CloudCom). IEEE, 2019. 29
- 30. Benchmarking Beyond Performance: Availability Availability benchmarking measures the impact of failures in a database cluster. Availability metrics are: ■ Failure performance impact ■ Accessibility ■ Request error rate ■ More details: Seybold, Daniel, Stefan Wesner, and Jörg Domaschka. "King Louie: reproducible availability benchmarking of cloud-hosted DBMS." Proceedings of the 35th Annual ACM Symposium on Applied Computing. 2020. 30
- 31. Benchmarking Beyond Performance: Consistency Consistency benchmarking measures the staleness and request execution order of data items under specific workloads or changing database deployments such as scaling or failures. ■ More details: Bermbach, David. Benchmarking eventually consistent distributed storage systems. KIT Scientific Publishing, 2014. ■ https://jepsen.io/ 31
- 32. Benchmarking Process 32 Allocate database and benchmark resources Deploy & configure database cluster Deploy benchmark Deploy monitoring Load data benchmark Execute benchmark Process results
- 34. Benchmarking Process — Automation ■ ScyllaDB Cluster Tests: https://github.com/scylladb/scylla-cluster-tests ■ Flexi-Bench:https://github.com/AgentEnder/flexi-bench ■ Mowgli: https://research.spec.org/tools/overview/mowgli/ ■ benchANT (closed source)
- 35. Benchmarking Guidelines There are numerous database benchmarking guidelines available, see below a recommended selection of the research domain: ■ Gray, Jim. "Database and transaction processing performance handbook." (1993). ■ Raasveldt, Mark, et al. "Fair benchmarking considered difficult: Common pitfalls in database performance testing." Proceedings of the Workshop on Testing Database Systems. 2018. ■ Papadopoulos, Alessandro Vittorio, et al. "Methodological principles for reproducible performance evaluation in cloud computing." IEEE Transactions on Software Engineering 47.8 (2019) 35
- 36. Where to find (reliable) Database Performance Results? 36
- 37. Blog Posts & Whitepaper ■ Numerous studies are available, from in-depth to high-level ■ Studies are mostly database/cloud vendor driven benchmarking studies ■ Reliable studies should reference or include the applied benchmarking suite, applied commands and raw results besides high-level charts ■ Ideally, the full database deployment details and configurations are also publicly available 37
- 38. Scientific Publications ■ Conferences such as VLDB, SIGMOD, ICPE, SIGMETRICS, TPCTC or DBTEST include database benchmark papers ■ In-depth studies targeting specific aspects of selected database systems or present new database workloads and/or benchmarking suites ■ A growing number of scientific conferences emphasize reproducibility, i.e. the papers have associated software artifacts to reproduce the results 38
- 39. Public Rankings ■ TPC rankings per TPC benchmark: https://www.tpc.org/information/benchmarks5.asp ■ ClickBench by ClickHouse: https://benchmark.clickhouse.com/ ■ RTABench by Timescale: https://rtabench.com/ ■ VectorDBBench by Zilliz: https://zilliz.com/vector-database-benchmark-tool ■ Open Database Performance Ranking by benchANT: https://benchant.com/ranking/database-ranking 39
- 40. Your Own Benchmarks It is quite likely that existing benchmark results do not match your benchmarking objective 1:1 but might provide a decent starting point. For designing and executing your own benchmarking study, some general advice: ■ Allocate a decent amount of resources and time to ensure reliable results (1-2 benchmark runs will never be enough) ■ Expect unexpected problems, especially at scale ■ Once you started, you will always want to measure another detail 40
- 41. Keep in touch! Dr. Daniel Seybold Co-Founder benchANT [email protected]