diff --git a/content/learning-paths/servers-and-cloud-computing/mysql_tune/_review.md b/content/learning-paths/servers-and-cloud-computing/mysql_tune/_review.md index ee0568b79..e9fc873d2 100644 --- a/content/learning-paths/servers-and-cloud-computing/mysql_tune/_review.md +++ b/content/learning-paths/servers-and-cloud-computing/mysql_tune/_review.md @@ -13,25 +13,25 @@ review: - questions: question: > - When tuning a thread count configuration parameter like innodb_read_io_threads or innodb_write_io_threads. What can often be a good starting value? + How do you select the number of huge pages that should be used? answers: - - "One" - - "Total number of CPUs on the system" - - "Half the number of CPUs on the system" + - "Set to the size of the redo log file" + - "Divide the buffer pool size by the huge page size" + - "Set to the buffer pool size" correct_answer: 2 explanation: > - The total number of CPUs is a good starting point because it can ensure you are using all compute resources on the system. + We divide by the bugger pool size because we want as much huge page space as there is buffer pool space. - questions: question: > What is the recommended size for the MySQL buffer pool? answers: - - "Up to 40% of system memory" - - "Up to 20% of system memory" - - "Up to 80% of system memory" + - "Up to 30-40% of system memory" + - "Up to 10-20% of system memory" + - "Up to 70-80% of system memory" correct_answer: 3 explanation: > - The MySQL documentation suggests up to 80% of system memory. Depending on the use case, it's also possible that a much smaller percentage performs just as well as 80%. + The MySQL documentation suggests up to 80% of system memory. Depending on the use case, it's also possible that a much smaller percentage performs just as well as 80%. Buffer pool size is also automatically set to 75% of system memory if you use the innodb_dedicated_server option (See MySQL docs). - questions: question: > diff --git a/content/learning-paths/servers-and-cloud-computing/mysql_tune/before_and_after.md b/content/learning-paths/servers-and-cloud-computing/mysql_tune/before_and_after.md index f6ddc8b94..b5fd336d9 100644 --- a/content/learning-paths/servers-and-cloud-computing/mysql_tune/before_and_after.md +++ b/content/learning-paths/servers-and-cloud-computing/mysql_tune/before_and_after.md @@ -6,8 +6,12 @@ layout: "learningpathall" ## About database performance tuning -Deployment configurations and the profile of SQL requests made by clients will differ based on your use case. This means there is no one size fits all set of tuning parameters for `MySQL`. Use the information in this learning path to help you tune `MySQL` for your use case. +The configuration of a database and the types of requests made by clients to it will differ from use case to use case. This means there is no one size fits all set of tuning parameters for `MySQL`. Use the information in this learning path as general guidance to help with tuning `MySQL`. ## Importance of tuning -Application tuning allows you to gain performance without scaling your deployment up (bigger machines) or out (more machines). You have the option to use the gained performance or trade it for cost savings by reducing the total compute resources provisioned. Requirements vary based on the use case. \ No newline at end of file +Application tuning allows you to gain performance without scaling a deployment up (bigger machines) or out (more machines). This gives the option to use the gained performance, or to trade it for cost savings by reducing the total compute resources provisioned. + +## Note on MySQL Documentation + +All links to [MySQL documentation](https://dev.mysql.com/doc/refman/en/) in this learning path point to the latest version of the documentation. If you are using an older version of MySQL, be sure to change the documentation version after clicking the links. \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/mysql_tune/kernel_comp_lib.md b/content/learning-paths/servers-and-cloud-computing/mysql_tune/kernel_comp_lib.md index 449f9e31f..95d35ff73 100644 --- a/content/learning-paths/servers-and-cloud-computing/mysql_tune/kernel_comp_lib.md +++ b/content/learning-paths/servers-and-cloud-computing/mysql_tune/kernel_comp_lib.md @@ -1,6 +1,6 @@ --- # User change -title: "System, Kernel, compiler, and Libraries" +title: "System, Kernel, Compiler, and Libraries" weight: 3 # 1 is first, 2 is second, etc. @@ -8,68 +8,45 @@ weight: 3 # 1 is first, 2 is second, etc. layout: "learningpathall" --- -## Storage technology and file system format +## Storage technology, file system format, and disk scheduling -The underlying storage technology and the file system format can impact performance significantly. In general, locally attached SSD storage will perform best. However, network based storage systems can perform well. As always, performance is dependent on the request profile coming from clients. You should spend some time studying and experimenting with different storage technologies and configuration options. +The underlying storage technology and the file system format can impact performance. In general, locally attached SSD storage will perform best. However, network based storage systems can perform well. You should spend some time studying and experimenting with different storage technologies and configuration options. -Aside from the storage technology, the file system format used with `MySQL` can impact performance. The `xfs` file system is a good starting point. The `ext4` file system is another good alternative. +Aside from the storage technology, the file system format used with `MySQL` can impact performance. The `xfs` file system is a good starting point. The `ext4` file system is another good alternative. Last, it is recommended to use storage drives that are dedicated to the database (i.e. not shared with the OS or other applications). + +When running in the cloud, the disk scheduling algorithm is typically set to `noop` or a similar "dumb" algorithm. This is typically optimal for `MySQL` in the cloud, so no adjustment is needed. However, if running `MySQL` on an on-prem server, it's a good idea to double check what the disk scheduling algorithm is, and possibly change it. According to the [Optimizing InnoDB Disk I/O documentation]https://dev.mysql.com/doc/refman/en/optimizing-innodb-diskio.html), `noop` or `deadline` might be better options. It's worth testing this with on-prem systems. ## MySQL storage engines -There are different storage engines available for `MySQL`. The default storage engine is `InnoDB`. `InnoDB` is good for performance testing and tuning. +There are different storage engines available for `MySQL`. The default storage engine is `InnoDB`. `InnoDB` is the default storage engine because it performs the best in the broadest set of use cases. -Information on alternative storage engines can be found in the [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/storage-engines.html). +Information on alternative storage engines can be found in the [MySQL documentation](https://dev.mysql.com/doc/refman/en/storage-engines.html). ## Kernel configuration `MySQL` can benefit from adjustments to kernel parameters. Below is a list of kernel related settings that can have a positive impact on performance. -### Linux-PAM limits - -Linux-PAM limits can be changed in the `/etc/security/limits.conf` file, or by using the `ulimit` command. - -If you want more information about how to display and modify parameters check the documentation of the `ulimit` command. - -To display all limits: -```bash -ulimit -a -``` - -To display the `memlock` (Max locked-in-memory address space) limit only: -```bash -ulimit -l -``` - -`memlock` is the only PAM limit which is useful to adjust for `MySQL`. - -The suggested value for `memlock` is `unlimited` when using huge pages with `MySQL`. - -Enabling huge pages can result in significant performance gains (discussed below). - -The suggestion to set `memlock` when huge pages are enabled can be found in the [MySQL documentation](https://dev.mysql.com/doc/refman/8.1/en/large-page-support.html). - - ### Linux virtual memory subsystem -Making changes to the Linux Virtual Memory subsystem can also improve performance. +Making changes to the Linux Virtual Memory subsystem can improve performance. These settings can be changed in the `/etc/sysctl.conf` file, or by using the `sysctl` command. -If you want more information about how to display and modify virtual memory parameters check the documentation of the `sysctl` command. - -Documentation on each of these parameters can be found in the [admin-guide for sysctl in the Linux source code](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/sysctl/vm.rst). +Documentation on the virtual memory subsystem parameters can be found in the [admin-guide for sysctl in the Linux source code](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/sysctl/vm.rst). -To list all kernel parameters available: +To list all sysctl parameters available: ```bash sudo sysctl -a ``` +See the `sysctl` command documentation for more. + ### Huge memory pages `MySQL` benefits from using huge memory pages. Huge pages reduce how often virtual memory pages are mapped to physical memory. -To see the current memory page configuration, run the following command on the host: +To see the current huge memory page configuration, run the following command on the host: ```bash cat /proc/meminfo | grep ^Huge @@ -86,11 +63,11 @@ Hugepagesize: 2048 kB Hugetlb: 0 kB ``` -Huge pages are not being used if `HugePages_Total` is 0 (this is the default). +Huge pages are not being used if `HugePages_Total` is 0 (this is typically the default). -Also note that `Hugepagesize` is 2MB which is the typical default for huge pages on Linux. +Also note that `Hugepagesize` is 2MiB which is the typical default for huge pages on Linux. -The kernel parameter that enables huge pages is shown below: +The sysctl parameter that enables huge pages is shown below: ```output vm.nr_hugepages @@ -114,11 +91,9 @@ sudo sh -c 'echo "vm.nr_hugepages=500" >> /etc/sysctl.conf' ### Selecting the number of huge pages to use -You should set `vm.nr_hugepages` to a value that gives a total huge page space slightly larger than the `MySQL` buffer pool size (discussed later). - -It should be slightly larger than the buffer pool because `MySQL` will use additional memory for things like connection management. +You should set `vm.nr_hugepages` to a value that gives a total huge page space equal to or slightly larger than the `MySQL` buffer pool size. Selecting the buffer pool size is discussed in the [Tuning MySQL](/learning-paths/servers-and-cloud-computing/mysql_tune/tuning) section. -More information on the different parameters that affect the configuration of huge pages can be found in the [admin-guide for hugetlbpage in the Linux source code](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/mm/hugetlbpage.rst). +Typically, only the number of huge pages needs to be configured. However, for more information on the different parameters that affect the configuration of huge pages, review the [admin-guide for hugetlbpage in the Linux source code](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/mm/hugetlbpage.rst). ## Compiler Considerations diff --git a/content/learning-paths/servers-and-cloud-computing/mysql_tune/tuning.md b/content/learning-paths/servers-and-cloud-computing/mysql_tune/tuning.md index 5da4be4c9..df793d42a 100644 --- a/content/learning-paths/servers-and-cloud-computing/mysql_tune/tuning.md +++ b/content/learning-paths/servers-and-cloud-computing/mysql_tune/tuning.md @@ -12,16 +12,13 @@ layout: "learningpathall" There are different ways to set configuration parameters for `MySQL`. -This is discussed in the [MySQL Programs documentation](https://dev.mysql.com/doc/refman/8.0/en/programs.html). +This is discussed in the [Specifying Program Options](https://dev.mysql.com/doc/refman/en/program-options.html) section of the MySQL documentation. The configurations below can be directly pasted into a `MySQL` configuration file under the group `mysqld`. -It's also possible to specify these configurations on the `mysqld` command line (typically within a linux service file). +It's also possible to specify these configurations on the `mysqld` command line. However, it's better to use configuration files since these can be version controlled more easily. -To display configuration settings: -```bash { pre_cmd="sudo apt install -y mysql-server" } -mysqld --verbose --help -``` +Last, keep in mind, that in general, it's usually best to leave most configs at default, and only change them if there is a suspected/known issue. ### Connections and prepared transactions @@ -30,54 +27,68 @@ max_connections=100000 # Default 151 max_prepared_stmt_count=4194304 # Default 16382 ``` -`max_connections` doesn't impact performance, but if a high client connection count is expected or required, it's a good idea to raise this in order to not reject request from clients. +`max_connections` doesn't impact performance, but if a high connection count is expected or required, this should be raised in order to not reject requests from clients. -Keep in mind that more client connections means more resources will be consumed (especially memory). Setting this to something higher is completely dependent on use case and requirements. +Keep in mind that more connections means more resources will be consumed, especially memory. Setting this to something higher is completely dependent on use case and requirements. `max_prepared_stmt_count` is 16382 by default. It's a good idea to set this as small as possible in order to help prevent denial of service attacks. You can make it very large in a test environment that uses many prepared statements. -### Memory related configuration +### Dedicated Server Configuration +```output +innodb_dedicated_server=ON +``` +If the node will only run `MySQL` and no other application. One of the easiest ways to gain performance is by setting `innodb_dedicated_server=ON`. This setting does different things depending on the version of `MySQL`, so it's important to check the [documentation](https://dev.mysql.com/doc/refman/en/innodb-dedicated-server.html) for the specific version of `MySQL` being deployed. As of version 8.4, this setting will automatically set both `innodb_buffer_pool_size` and `innodb_redo_log_capacity`. Two configurations that impact performance. + +On systems with a large amount of RAM (greater than 4GB), the `innodb_buffer_pool_size` will be set to 75% total system memory. `innodb_buffer_pool_size` is one of the most important configuration parameters that can be set. It determines how much memory can be used to store indexes and table data. It's a cache that improves read/write latency by relieving pressure on storage. If `innodb_dedicated_server` is not used, then this parameter should be set. The [MySQL documentation](https://dev.mysql.com/doc/refman/en/innodb-buffer-pool.html) suggests this be set to up to 80% of total system memory. The default of 128MiB is probably going to be far less than 80% of total system memory. + +`innodb_redo_log_capacity` was introduced in MySQL 8.0.30. It controls the amount of disk space used for redo log files. In earlier versions of `MySQL`, the parameters `innodb_log_file_size` and `innodb_log_files_in_group` influence redo log size and behavior. `innodb_redo_log_capacity` simplifies things. When `innodb_dedicated_server` is enabled, `innodb_redo_log_capacity` is set to (number of available logical processors / 2)GB with a max of 16GB. If `innodb_dedicated_server` is not used, then `innodb_redo_log_capacity` should be set. If the redo log is too small, the status variable [`Innodb_log_waits`](https://dev.mysql.com/doc/refman/en/server-status-variables.html#statvar_Innodb_log_waits) will be large. This indicates that the redo log buffer is waiting for the log to be flushed to storage before continuing. Ideally, this status variable is 0 at all times. + +### Huge Pages ```output large_pages=ON # default is OFF -innodb_buffer_pool_size= # Default is 128MB ``` Turning on `large_pages` can result in significant performance gains. Using larger pages helps to reduce how often physical memory has to get mapped to virtual memory. Note that huge pages needs to be [turned on at the kernel level](/learning-paths/servers-and-cloud-computing/mysql_tune/kernel_comp_lib) for this to work. -`innodb_buffer_pool_size` is one of the most important configuration parameters that can be set. It determines how much memory can be used to store indexes and table data. It's a cache that improves read/write latency and relieves pressure on storage. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-buffer-pool.html) suggests this be set to up to 80% of total system memory. Setting this value significantly larger than the default of 128MB is a good idea. +If `innodb_dedicated_server` is set to `ON`, then `innodb_buffer_pool_size` will automatically be set. The value of `innodb_buffer_pool_size` is needed to calculate how many huge pages will need to be allocated. The value of `innodb_buffer_pool_size` can be confirmed by checking the variable in the `mysql` cli. -Other memory related configurations that could be worth exploring are the [Buffer Pool Prefetching](https://dev.mysql.com/doc/refman/8.0/en/innodb-performance-read_ahead.html) configurations. You may see modest performance gains by decreasing the `innodb_read_ahead_threshold` from the default. The default is very conservative and will result in very little to no prefetching. Some workloads may benefit from being less conservative with prefetching. Turning on random prefetch (`innodb_random_read_ahead`) seems to hurt performance but could benefit some user cases. +```sql +SHOW VARIABLES LIKE 'innodb_buffer_pool_size'; +``` + +> NOTE: The innodb_buffer_pool_size unit is bytes. +> There is no need to confirm the value of `innodb_buffer_pool_size` if it is set manually (i.e. you are not using `innodb_dedicated_server=ON`). +> Alternatively, you can estimate the value of `innodb_buffer_pool_size` since the calculation when `innodb_dedicated_server=ON` is in the documentation. Use the command "`free --bytes`" to get the total system memory in bytes. + +Divide this number by the huge page size (convert to bytes first) to get the number of huge pages that should be allocated. Instructions on how to get the huge page size and to set the number of pages is in the [System, Kernel, Compiler, and Libraries](/learning-paths/servers-and-cloud-computing/mysql_tune/kernel_comp_lib) section. -### Logging and disk flush behavior +In general, there's no need to adjust other memory parameters unless an issue is suspected/found. That said, other memory related configurations that could be worth exploring are the [Buffer Pool Prefetching](https://dev.mysql.com/doc/refman/en/innodb-performance-read_ahead.html) configurations (take this as an FYI). You may see modest performance gains by decreasing the `innodb_read_ahead_threshold` from the default. The default is very conservative and will result in very little to no prefetching. Some workloads may benefit from being less conservative with prefetching. Turning on random prefetch (`innodb_random_read_ahead`) seems to hurt performance but could benefit some user cases. The affects of these settings will be use case dependent. + +### Disk flush behavior ```output -innodb_use_fdatasync=ON # Default is OFF -innodb_log_file_size=20GB # Default is 48MB +innodb_use_fdatasync=ON # Default is OFF prior to MySQL 8.4 ``` -Setting `innodb_use_fdatasync` to ON helps reduce the number of system calls that occur when flushing data to disk. Using `fdatasync` reduces flushing by not updating the meta data associated with files when those files are written to. For most use cases, this is acceptable. - -Setting `innodb_log_file_size` to much larger than the default (48MB) helps reduce how much flushing and check pointing occurs. See the [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_file_size) for more information. Also note, there is another parameter called `innodb_log_buffer_size` that may be worth experimenting with as well. [Documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_buffer_size) on this parameter is also available. +Setting `innodb_use_fdatasync` to ON helps reduce the number of system calls that occur when flushing data to disk. Using `fdatasync` reduces flushing by not updating the meta data associated with files when those files are written to. For most use cases, this is acceptable. As of MySQL 8.4, this is now set to `ON` by default. If running an older version of `MySQL`, it is generally recommended to set this to `ON`. In fact, this is why it has been defaulted to `ON` in newer version of `MySQL`. ### Concurrency configuration Increasing parallelism uses available resources more efficiently. It's always a good idea to look at parameters related to parallel execution. ```output -innodb_io_capacity=1000 # Default is 200 -innodb_io_capacity_max=2000 # Default is 2x innodb_io_capacity +innodb_io_capacity=10000 # Default is 200 prior to MySQL 8.4 and 10000 from 8.4 +innodb_io_capacity_max=20000 # Default is 2x innodb_io_capacity innodb_read_io_threads= # Default is 4 innodb_write_io_threads= # Default is 4 ``` -`innodb_io_capacity` tells the `InnoDB` storage engine how many IOPS it can issue to storage. The default of 200 is quite low and more appropriate for rotational storage. Modern SSD storage and even cloud based storage can benefit greatly from increasing this value. The [MySQL InnoDB I/O Capacity documentation](https://dev.mysql.com/doc/refman/8.1/en/innodb-configuring-io-capacity.html) suggests this be set to around 1000 for higher performing storage. In some cases, it might be worth setting this higher than 1000. - -`innodb_io_capacity_max` defaults to 2x of `innodb_io_capacity`. It is worth experimenting with this value in use cases that experience heavy disk usage. +`innodb_io_capacity` tells the `InnoDB` storage engine how many IOPS it can issue to storage. Prior to `MySQL` 8.4, the default was 200 which is very low and more appropriate for rotational storage. Modern SSD storage and even cloud based storage can benefit greatly from increasing this value. As of `MySQL` 8.4, the default value has been increased to 10000. If you are using an older version of `MySQL` where this is defaulted to 200, it might benefit you to set this to 10000. See the [MySQL InnoDB I/O Capacity documentation](https://dev.mysql.com/doc/refman/en/innodb-configuring-io-capacity.html) for more. -`innodb_read_io_threads` and `innodb_write_io_threads` sets the number of threads used for IO disk operations. Setting this to the number of CPUs in the system can provide performance benefits. Experiment with these parameters since a value that is smaller then the total number of CPUs in the system may be sufficient. +`innodb_io_capacity_max` defaults to 2x of `innodb_io_capacity`. This should be ok for most use cases. ### Spin lock configuration @@ -87,4 +98,4 @@ innodb_sync_spin_loops=120 # Default is 30 You should experiment with the `innodb_sync_spin_loops` parameter. This sets the number of times a thread checks for an `InnoDB` lock to be free before yielding execution to another thread. -Profiling `MySQL` under heavy load with Linux `perf` shows that `MySQL` spends a lot of time waiting for locks to be freed. Experimenting with tuning parameters around locks might help. Increasing the number of times a lock is checked before the thread yields can reduce context switching. This reduction in context switching tends to increase performance. Start with a value of 120 for `innodb_sync_spin_loops`, but you can also try values such as 30, 60, 180, and 240. \ No newline at end of file +Profiling `MySQL` under heavy load on Arm with Linux `perf` shows that `MySQL` spends a lot of time waiting for locks to be freed. Experimenting with tuning parameters around locks might help. Increasing the number of times a lock is checked before the thread yields can reduce context switching. This reduction in context switching tends to increase performance. Start with a value of 120 for `innodb_sync_spin_loops`, but you can also try values such as 30, 60, 180, and 240. See the [Configuring Spin Lock Polling](https://dev.mysql.com/doc/refman/en/innodb-performance-spin_lock_polling.html) for more. \ No newline at end of file