Tuning Linux for your database FLOSSUK 2016
7 likes1,931 views
The presentation discusses optimizing Linux for database performance, highlighting key areas such as hardware, I/O, memory, CPU, and network configurations. It emphasizes the importance of selecting appropriate storage, filesystems, and schedulers, as well as various tuning techniques to improve database efficiency. Additional topics include resource limits, virtualization considerations, and useful benchmarking tools for evaluating performance.
Tuning Linux for your database FLOSSUK 2016
- 1. Tuning Linux for your database Colin Charles,Team MariaDB, MariaDB Corporation [email protected] / [email protected] http://bytebot.net/blog/ | @bytebot on Twitter FLOSSUK 2016, London 16 March 2016 1
- 2. whoami • Work on MariaDB at MariaDB Corporation (SkySQL Ab) • Merged with Monty Program Ab, makers of MariaDB • Formerly MySQL AB (exit: Sun Microsystems) • Past lives include Fedora Project (FESCO), OpenOffice.org • MySQL Community Contributor of theYear Award winner 2014 2
- 3. Agenda • Hardware or cloud • I/O • Filesystems • Memory • CPU • Network • Resources 3
- 4. Operating System • Focus of this talk & tools: Linux • But distributions make a difference too • like RHEL6 has tuned but Ubuntu doesn’t • Distribution versions can make a difference too • RHEL5: ktune; RHEL6: tuned; RHEL7: tuned throughput-performance 4
- 5. Servers • When you buy servers... do you think in terms of CPU, then memory, then I/O? • Database servers clearly love I/O more than memory more than CPU • queries! scans of large tables? index scans with random I/O? BI? • Durability requires each transaction commit to be flushed to disk - i.e. call fsync() 5
- 6. I/O • What kind of storage do you use? • SAS or SATA? • Ethernet (NAS, DRBD) • SSD • Fusion IO (NVMFS) 6
- 7. Schedulers (I/O Elevators) • cfq: default, great for slower storage (SATA) • noop: low latency storage (SSD) • deadline: multi-process apps • Most database workloads benefit from the deadline scheduler (goal is after all to minimise seeks, prioritise process I/O) 7
- 8. Elevators continued • Boot time: elevator=deadline •echo “deadline” > /sys/class/ block/sdaN/queue/scheduler • http://dimitrik.free.fr/blog/archives/2012/01/ mysql-performance-linux-io.html • http://www.mysqlperformanceblog.com/ 2009/01/30/linux-schedulers-in-tpcc-like- benchmark/ 8
- 9. Sample elevator results 9 1 thread tps 1 thread ioutil% 8 threads tps 8 threads ioutil% deadline* 15000 <1% 19100 <1% deadline 6850 32% 15250 45% cfq 5880 45% 1240 94%
- 10. File systems • Choices: ext4, XFS • mount options: • ext4: rw, nosuid, noatime, data=ordered, nobarrier • xfs: nobarrier • Turn off I/O barriers • http://lwn.net/Articles/ 283161/ • Separate files (data/logs/ undo/etc.) or not? • Don’t forget flash backed write caches or battery backed write cache on RAID cards (way more fsync/s) • iostat -dmx <interval> is your friend 10
- 11. SSD & Flash • SSD • much more IOPS than traditional disks • lower latency • make sure you have a new RAID controller • Flash • more IOPS than SSD • very low latency • InnoDB doublewrite buffer -- MariaDB 10.0, Percona Server 5.6 (Fusion IO), MySQL 5.7 11
- 12. RAID • RAID0 - fast writes, reads, no redundancy • RAID1 - slower writes, fast reads • RAID5 - slow for random writes, fast for sequential writes, fast reads too but slow recovery • RAID10 - fast reads & writes 12
- 13. LVM • Why? • easily expand disk • snapshot backups • Why not? • 2-3% performance penalty • Snapshot penalties 13
- 14. Memory • The more RAM the better -> reduces I/O requirements • Use ECC RAM • NUMA - “swap insanity” • http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the- numa-architecture/ •numactl --interleave=all mysqld & • Twitter patch/Percona Server 5.6 • numa_interleave option • innodb_buffer_pool_populate - NUMA decisions to be made when buffer cache is clean 14
- 15. Why does MySQL need memory? • Per-session buffers • sort buffer, temp tables • Metadata/locking • index statistics, table definitions • Caching data • decrease reads: faster response time • decrease random writes: queues write in cache; perform more sequential writes 15
- 16. Where is memory used? • filesystem cache • binary / relay logs • MyISAM data relies on filesystem cache • MySQL cache: • innodb_buffer_pool_size (pages), MyISAM’s key_buffer_size (indexes) • Per session buffers •sort_buffer_size, join_buffer_size, read_buffer_size, tmp_table_size •free -m / SHOW ENGINE INNODB STATUS 16
- 17. Huge pages • 2M (4M even) pages vs 4K standard Linux page •vm.nr_hugepages=8192 • MySQL benefits from hugepages, MongoDB doesn’t 17
- 18. Swappiness • Controls how aggressively the system reclaims “mapped” memory • default: 60, vm.swappiness=0 is ok (but not for newer kernels!) • decrease: aggressive reclaiming of unmapped pagecache memory • increase: aggressive swapping of mapped memory • 3.5 kernels change behaviour (and this was backported to RHEL 2.6.32-303) 18
- 19. CPU • default is ondemand, you should turn it to performance generally •echo "performance" > /sys/devices/ system/cpu/cpu0/cpufreq/ scaling_governor • disable powersave mode • cpufrequtils is where it’s at • look at BIOS: turbo mode/C-state/OS power control 19
- 20. Network • Tools: tc, dropwatch • Use gigabit Ethernet (more bandwidth/ throughput, less latency) • Can also trunk/bond for HA 20
- 21. Network II • ARP filter - prevents ARP flux •echo 1 > /proc/sys/net/ipv4/ conf/all/arp_filter • MTU size at 9,000 - “jumbo frames” • sar -n DEV 21
- 22. Network III • allow more connections to queue up: echo 4096 > / proc/sys/net/ipv4/tcp_max_syn_backlog • /etc/sysctl.conf - net.ipv4.tcp_max_syn_backlog = 4096 • increase kernel packet buffer: net.core.netdev_max_backlog = 4096 • increase socket connection wait queue: net.core.somaxconn = 4096 • Reduce TCP timeout that comes after closing socket (default 1m): net.ipv4.tcp_fin_timeout = 30 22
- 23. Network IV • Fully synchronous replication, ala, Galera Cluster? (NDBCluster too) • Fastest as your slowest node • evs.send_window + evs.user_send_window (something large e.g. 512) 23
- 24. Resource Limits • ulimits / /etc/security/limits.conf - they need to be high/sensible over the defaults • -f (file size): unlimited • -t (cpu time): unlimited • -v (virtmem): unlimited • -n (open files): 64000 • -m (memsize): unlimited • -u (processes/threads): 32000 24
- 25. Tune that database • Focus on database design - find slow, crappy queries and fix them (eg. mysql: pt- query-digest) • Reduce locking or time waiting - always benchmark your application continually • Don’t forget /etc/my.cnf (and appropriate friends) 25
- 26. KVM/Xen/Virtualization • Most of your raw database performance comes from being hardware-centric • KVM at device level you can turn off caching • Native AIO used over threaded nowadays • VMWare: http://kb.vmware.com/selfservice/ microsites/search.do? language=en_US&cmd=displayKC&externalI d=1008542 26
- 27. Containers • Kubernetes - vitess.io rolls out w/o issue, CoreOS, Docker • Remember when benchmarking, you have to look at local vs remote benchmarking (networking can add overhead) • 1 thread (up to 25% difference), but average seems to be 15% loss for r/w workloads; remotely? 32% is possible 27
- 28. EC2 • EC2 - instance limitations • EBS - unpredictable I/O performance, use RAID10 or RAID0 • RDS - similar performance between EBS RAID10 MySQL & RDS • choice of MySQL 5.7 (and older) or MariaDB Server 10.0 or PostgreSQL 28
- 29. Why MariaDB/Percona Server • Threadpool • NUMA interleaving • Numerous performance fixes in XtraDB like fast InnoDB restarts, buffer pool warming, etc. • Can run (supported) MariaDB Galera Cluster/Percona XtraDB Cluster 29
- 30. Hardware overall • Test everything • Sometimes NIC’s are a bad batch or the driver is buggy (RHEL5 + some Broadcom NICs drop packets under high load - bnx2 before 2.6.33) • Sometimes you get a bad batch of disk 30
- 31. Hadoop/Hbase • running this in EC2 is problematic • make sure name resolution works • enable bonding • networking: hadoop uses big buffers (64MB blocks by default) • net.ipv4.tcp_rmem = 32768 436600 4194304 • net.ipv4.tcp_wmem = 32768 436600 4194304 • http://docs.hortonworks.com/HDPDocuments/HDP2/ HDP-2.1.3/bk_cluster-planning-guide/content/ch_hardware- recommendations.html 31
- 32. Other databases • MongoDB: http://docs.mongodb.org/manual/ administration/production-notes/ • PostgreSQL: http://wiki.postgresql.org/wiki/ Tuning_Your_PostgreSQL_Server • Neo4j: http://docs.neo4j.org/chunked/stable/ configuration-linux-notes.html • Riak: http://docs.basho.com/riak/1.4.0/ cookbooks/Linux-Performance-Tuning/ 32
- 33. SELinux, firewalls, security • The easy way is to “disable SELinux” • The reality is you should learn it • Its just like a firewall that you really should configure 33
- 34. Tools • ps • vmstat 1 • iostat • top • free • sar (sar -n DEV 1) • gdb • tcpdump • strace • oprofile • htop • MySQL only: Percona Toolkit (pt-diskstats, pt- summary, pt-query-digest, etc.) 34
- 35. Benchmarking • sysbench • OLTP test, use tables with 20M rows and 20M transactions, check 1-128 threads/run • LinkBench • Yahoo! Cloud Serving Benchmark • https://github.com/ brianfrankcooper/ YCSB • Google’s PerfKit Benchmarker • https://github.com/ GoogleCloudPlatfor m/ PerfKitBenchmarke r 35
- 36. Books 36
- 37. Resources • http://doc.opensuse.org/products/draft/SLES/ SLES-tuning_sd_draft/book.sle.tuning.html • https://access.redhat.com/site/documentation/ en-US/Red_Hat_Enterprise_Linux/6/html-single/ Performance_Tuning_Guide/index.html • https://access.redhat.com/site/documentation/ en-US/Red_Hat_Enterprise_Linux/6/html-single/ Virtualization_Tuning_and_Optimization_Guide /index.html 37
- 38. Thanks/Q&A Colin Charles, [email protected] | [email protected] http://bytebot.net/blog/ | @bytebot slides: slideshare.net/bytebot/ Download MariaDB and give it a try: http://mariadb.org/ 38