Percona QA is a suite of scripts and utilities that assists in building, continuous integration, automated testing & bug reporting for Percona Server, Percona XtraDB Cluster, and Percona Server for MongoDB.
For a video introduction to some of the scripts, see: https://www.percona.com/blog/2015/03/17/free-mysql-qa-and-bash-linux-training-series/
It is a requirement for many scripts to work that sudo is enabled and working (and should not request a password) for the user who is using these scripts.
This script iterates through MySQL crash core files and runs analyze_crash.sh on them.
See Also: analyze_crash.sh (calls)
This script takes a MySQL crash core dump as an argument. It creates a an output file with a full backtrace of all threads associated with the core and the time it them backtrace was generated.
See Also: analyze_crashes.sh (called from)
This script will iterate through all core files starting with the current working directory and creates report containing full backtrace of all threads for each in the file gdb__STD/_FULL.txt
Ansible installation/setup information.
Download and build mysql from bzr url.
Example: ./build_mysql.sh lp:mysql-server/5.6
Note: only current cmake builds will work, various compile options can be set by editing the script.
download and build percona server from bzr url. Note: script is outdated and it is preferred to use get_percona.sh (see description)
This script will download and build xtrabackup from a bzr url
Example: ./build_xtrabackup.sh lp:percona-xtrabackup/2.2
Deletes ALARM status RQG trials for which there are no errors of any importance in the server log It does so by checking for which trials ALARM.sh does not return any important failures (Whether this is the case or not is determined by the $FINDS and $IGNOR variables in rqg_results.sh)
See Also: rqg_failure_counts.sh, rqg_results.sh
For use with lp:randgen combinations.pl results
Use with caution. Removes all established known issues for a given combinations.pl run based on a search string No valid parameters were passed. Need a status (1) and a search string to cleanup on (2). Retry. Example use: $cleanup_failures.sh SERVER_CRASHED 'srv_log_block_size' Note; if you want to delete all items for a given status, use 'a' (or shortly: a) as the search string Note; this script relies on the STATUS ./{status passed as first option} scripts to be present Note; this script relies on the delete_single_trial.sh script to be present in the same directory as this script
See Also: delete_single_trial.sh (calls)
Build full and package Percona-XtraDB-Cluster-codership.tar.gz binary including the galera engine. Version 5.6
Build full and package Percona-XtraDB-Cluster-codership.tar.gz binary including the galera engine. Version 5.5
License (GPL 2)
For a given trial, this script quickly produces a tarball containing: the core file generated by that trial, the mysqld used by that trial, and the ldd dependency files needed by that This can be used to quickly get a tarball which is ready to be added to a bug report for analysis by developers.
Takes the RQG trial number as a parameter.
See Also: crash_bug_files.sh (called from), ldd_files.sh (calls)
This script quickly gathers all files required for logging a crash bug report.
Takes the RQG trial number as a parameter.
See Also: startup.sh (calls), analyze_crash.sh (calls), core_retrieve.sh (calls)
For use with lp:randgen combinations.pl results
This script deletes a given trial (from a combinations.pl run) completely. Execute from the combinations.pl workdir"
Example: to delete trial 1000, execute as:
./delete_single_trial.sh 1000"
See Also: cleanup_ALARMs.sh (called from), cleanup_failures.sh (called from), rqg_results.sh (called from)
This script executes startup.sh in manual RQG mode (without extracting mysql build and randgen), then runs the resulting cmdtrace script and finally prepare_reducer.sh. It expects one parameter: the trial number to analyze
See Also: startup.sh (calls), prepare_reducer.sh (calls)
Previously in directory ./docker, Percona QA PXC Docker (multi-node Docker configurations PXC testing) has been moved to https://github.com/percona/pxc-docker
Docker installation/setup information. Includes information on day-to-day tasks, helpful resources etc.
This bash script will download a single file from a launchpad project. See the usage in the script for more information. There is a section at the bottom that can be embedded in another script or Jenkins job if it is known that the target machine has 'wget' installed.
This script will download all of the rpm/deb packages from the given web page link. It is designed to easily download Jenkin's package builds for testing. (Requires lynx and wget to be installed)
Example:
download_rpmdeb.sh http://jenkins.percona.com/view/Percona/job/pxc56/999/label_exp=rhel6
This script will load and test the MySQL stored function check_ip() into a mysql database. The command line for running the mysql client is defined in the environment variable CLI at the top of the script.
check_ip( char(255) ) is an IP address validator.
The function returns:
OK (IPV4) - Good IPV4 address
OK (IPV6) - Good IPV6 address
>!FAIL!(x)< - Fail where 'x' is a-e designating the type of failure
(see function for more info)
NOT_AN_IP - Input does not resemble an IPV4 or IPV6 address.
The stored function uses the underlying IS_IPV4(), INET_ATON(), INET_NTOA(), INET6_ATON() and INET6_NTOA() functions in order to validate the given ip address.
This is a gdb script which is called from prepare_reducer.sh and helps extract queries that were running from a mysqld core dump.
See Also: prepare_reducer.sh (called from)
This script downloads recent binaries from the Jenkins CI server (optimized, with debug symbols and valgrind) and extracts them onto the local system for testing. This avoids the time required for local builds and issues with toolchain variations. Examine the script to determine or change which OS/target binaries are being downloaded (i.e. centos6-64).
Various shell and command line tips and tricks.
This small gdb command script allows you to analyze the state of a currently running but hung process. It will dump all thread backtrace call stacks for the given process id.
Example:
gdb mysqld {pid} < hung_mysqld.gdb
This script keeps a given RQG trial (from a combinations.pl run) completely (i.e. all files included) in the KEEP directory within a given combinations.pl workdir. Execute this script directly from within the combinations.pl workdir. Example: to keep trial 1000, execute as:
./keep_single_trial.sh 1000
This script quickly gathers all shared library dependencies for mysqld and xtrabackup in the current working directory the script is run from. The shared libraries are copied into a lib64 subdirectory of the current working directory.
See Also: core_retrieve.sh (called from)
This script will add a attached file to an existing launchpad bug report.
This script needs three parameters; the Launchpad bug number, the file to upload and the file's description.
Usage: ./lp-bug-logger.py -b <bugno> -f <file> -d '<description>'
This script is usually not executed directly, but is instead used by lp-bug-logger.sh However, if you use it directly, a 'None' output means there were no errors/that the upload was successful.
You will need to install python-launchpadlib for the proper operation of this script.
Ubuntu:
$sudo apt-get install python-launchpadlib
Others:
$sudo pip install launchpadlib
Some important information on interaction with Launchpad via the Python Launchpad API used by this script: The first time you run this script, you will see something like this (after quite some time):
The authorization page:
(https://launchpad.net/+authorize-token?oauth_token=xxxxxxxxxxxxxx&allow_permission=DESKTOP_INTEGRATION)
should be opening in your browser. Use your browser to authorize this
program to access LP on your behalf Waiting to hear from Launchpad about
your decision...
When you see this (and assuming you're working in a text-only ssh connection; otherwise a browser may have already automatically opened for you), copy and paste the URL shown into any browser and authorize the app (for example, until revoked) after logging into LP. This only has to be done once (if you chose an indefinite until revoked authorization), and it can be done from any machine (provided you login to LP). If you run into any issues, also review Unrecoverable error when authorization fails if applicable. And, authorized applications can be viewed (and revoked) at: https://launchpad.net/~<your_lp_user_id>/+oauth-tokens
See Also: lp-bug-logger.sh (called from)
This script uploads, for a given trial, all files in the directory ./BUNDLE_/* to a given Launchpad bug report. It expects two parameters; the trial number & the bug number. For Example, to action trial 1000, and upload to bug report 1259034 use:
$./lp-bug-logger.sh 1000 1259034
Note: to generate a BUNDLE_ directory, use crash_bug_files.sh
Note: This script calls lp-bug-add-file.py, please review the description of that script for other important information.
See Also: lp-bug-add-file.py (calls)
Scans all MTR testcase commands & generates bughunt RQG grammar/SQL
Run multiple mysql clients executing the given input SQL multiple times each.
This script expects exactly 5 options. Execute as follows:
$ multirun_cli.sh threads repetitions input.sql cli_binary socket
| Where | Description |
|---|---|
| threads | is the number of simultaneously started cli threads |
| repetitions | the number of repetitions per individual thread |
| input.sql | is the input SQL file executed by the clients |
| cli_binary | is the location of the mysql binary |
| socket | is the location of the socket file for connection to mysqld |
Example:
$ multirun_cli.sh 10 5 bug.sql /ssd/pxc-5.5/bin/mysql /ssd/test/mysql.sock
Cautionary Notes:
- Script does not check yet if options passed are all valid/present, make sure you get it right
- If root user uses a password, a script hack is suggested ftm
- This script may cause very significant server load if used with many threads
- This script expects write permissions in the current directory (ex. /home/{user}/percona-qa)
- Output files for each thread are written as: multi.. (e.g. multi.1.1 etc.)
This script will execute a given saved RQG trial against multiple instances of a given mysql installation.
This script expects 3 parameters to test server crash with multiple mysqld processes.
Parameters:
- mysql base directory
- RQG trial number
- crashed SQL file name.
Example:
$ multirun_mysqld.sh /ssd/ramesh/pxc-5.6.20-x64 36 in.sql
*See Also:$ myextra.sh (calls)
This script outputs the extra mysqld options used in a given RQG trial. It expects one parameter: the trial number to analyze. Ensure that startup.sh has already been executed for this trial.
See Also: multirun_mysqld.sh (called from), prepare_reducer.sh (called from)
The scripts in this directory will test packaging across supported OS distributions.
This script can be used to extract SQL from a MySQL general-log.
Usage:
[perl] ./parse_general_log.pl -i infile -o outfile
Where:
- infile is a standard mysqld general query log
- outfile is the SQL output extracted from the log.
General mysqld logs can be generated with these options to MTR (or RQG):
--mysqld=--log-output=FILE --mysqld=--general_log --mysqld=--general_log_file=out.sql
WARNING: by default this script eliminates a number of statements which may cause replay issues (KILL, RELEASE) This may lead to non-reproducibility for certain bugs. Check the script's source for specifics
This script will enable the Percona apt/yum repository on a RHEL,Centos, Debian or Ubuntu server. The script must be run with root permissions.
If the script is hosted at a public URL then it can install the Percona apt/yum repo with a single command:
# curl -s {url}/percona_repo_install.sh | bash /dev/stdin
Note: Jenkins CI script
This is a performance testing script that is designed to be run from the Jenkins Continuous Integration system. It will run a series of tests with the MySQL Test Suite and Sysbench both with default and tuned settings.
It requires two parameters.
- relative workdir
- relative basedir
Usage example:
$./performance.sh 100 Percona-Server-5.5.28-rel29.3-435.Linux.x86_64
This would lead to /data/bench/qa/100 being created, in which testing takes /place and data/bench/qa/Percona-Server-5.5.28-rel29.3-435.Linux.x86_64 would /be used to test. The fixed root workdir to which all options are relative is /set to /data/bench/qa in the script.
Note: Jenkins CI script
This is an older version of the performance.sh script. It takes the same parameters.
See: performance.pl
A multi-threaded stress test for mysqld. This utility will read SQL queries from a file (one per line) and spawn a given number of threads that will execute the queries from the list in random order. The number of queries each thread will execute is equal to the SQL input line count but it is not guaranteed that all queries will run as they are selected at random, some queries may be run multiple times and others not at all.
Also included is a generator script which maybe used to generate queries for pquery. Other tools such as randgen can also be used to generate random query data for pquery input.
See Also: pquery-sb/
pquery-sb is a lua script to be used with the sysbench application. It performs the same function as the C++ based pquery utility.
See Also: pquery/
A wrapper for /util/reducer/reducer.sh in the lp:randgen source tree.
This script creates a local run script for reducer. It expects one parameter: the trial number to analyze.
To aid with correct bug to testcase generation, this script creates a local run script for reducer and sets #VARMOD#. This handles crashes/asserts for the moment only. Could be expanded later for other cases, and to handle more unforeseen situations.
script which displays process info for pid along with information on which signals are pending, blocked, ignored and caught by the process.
This script is designed to run from the Percona Jenkins job named 'pxc56-mtr'. It runs selected MTR tests on a recent PXC 5.6 build.
Note: Jenkins CI script
2 node cluster testing script, also tests upgrade from 5.5 to 5.6
This script tests a two node PXC 5.5 cluster, runs the mysql test suite and sysbench against it, then upgrades the cluster to 5.6 and re-runs the MySQL test suite with and without compatibility options, runs sysbench under 5.6 and then outputs the versions and row count of the test table on both nodes.
Note: Jenkins CI script
tests cluster combining 5.6 release with 5.6 latest.
Tests cluster SST and IST using Mysql test suite and multi-table sysbench from both nodes, performs an mysql_upgrade retests. Reports versions and row counts at the end.
Alternative version of pxc56to56.sh testing from one release version to another automatically downloads and builds sysbench if not found, wsrep debugging disabled, common parameter ordering changed.
Note: Jenkins CI script
This is a benchmark script for Percona XtraDB Cluster 5.6. It benchmarks using MySQL Test Suite, sysbench and has support for dual node cluster testing.
Note: Jenkins CI script
This is an automated randgen testing script for PXC 5.6. Resulting randgen trials are stored for later analysis.
Note: Jenkins CI script
This script tests Galera snapshot transfer between nodes in a cluster configuration. For PXC 5.6.
Note: Jenkins CI script
Earlier version of the PXC benchmarking script.
See: pxc6-bench.sh
This script is for PXC Testing ChaosMonkey Style Please go through the tutorial to install PXC on your machine and execute this script from node1 Analyze script_out.log file from workdir to get the PXC testcase output
This script is designed to monitor IST replication state between PXC wsrep nodes for various SQL instructions. An example of a bug this will test is for is lp1421360. For a reliable test please make sure there is no other traffic on the test cluster.
Note: Jenkins CI script
Earlier version of the pxc6-rqg.sh randgen script.
See: pxc6-rqg.sh
Note: Jenkins CI script
Earlier version of the pxc6-sst=test.sh
See: pxc6-sst-test.sh
reducer.sh - the best mysql testcase reduction program
For more info, search Google for: reducer.sh
Note: Jenkins CI script
Jenkins CI script to check randgen failure counts.
This script handles various trial outcomes for a given RQG run (run from the RQG combinations.pl workdir containing all trialx.log files)
Note: this script relies on the delete_single_trial.sh script to be present in the same directory as this script
If you start rqg_results.sh with WIPE as the first option ($./rqg_results.sh WIPE) it will wipe all STATUS_ENVIRONMENT_FAILURE, all STATUS_INTERNAL_ERROR, and all STATUS_PERL_FAILURE failed trials by default
parse_times.sh allows one to check the start and end times for all trials within a RQG combinations.pl run direct This allows for quick analysis to see if any of the trials stalled/hanged/locked up by reviewing the durations
Deprecated
This script does not work, leaving here for future work or future deletion
This script makes major changes to any machine it is executed on (installs packages, enables core files, changes significant security settings etc.). Please review the planned changes by reviewing the script contents. All of the changes in the script prepare this machine to be a proper QA server. This script is meant for use on Centos (6.x mainly). Make sure that you want to continue.
Note: this script removes pulseaudio drivers. If you are using a desktop/would like to keep pulseaudio, please remark the line in the script first. Finally, this script can be executed multiple times without doubling up configurations/without messing up anything. It was specifically coded for this. This script requires sudo and su commands to be available already. Use visudo as described above (in 'Sudo:') to add yourself to sudo list (faster run). This script is provided AS-IS with no warranty of any kind. Do NOT run on production machines. This script lowers the servers security. Use at own risk.
| Reminder | Notes |
|---|---|
| Syntax: | {variable} needs replacing with your specific configuration, for |
| example {your_port} becomes 22
SSHD: | follow http://www.cyberciti.biz/faq/centos-ssh/ except;
| Set port to new number in sshd_config and add this line instead in
| iptables (adjust port, and adjust your subnet if needed); | -A INPUT
| -s 192.168.0.0/24 -m state --state NEW -m tcp -p tcp --dport
| {your_port} -j ACCEPT |Note there is likely a similar line there
| already that needs removing
User: | useradd {username} -m -U; su passwd {username}
Disk: | use parted for partition management, fdisk -l for partition list
Net: | set ONBOOT=yes and BOOTPROTO=dhcp in
| /etc/sysconfig/network-scripts/ifcfg-eth0 if using Centos minimal
| install CD
| also set NETWORKING=yes /etc/sysconfig/network and set a
| HOSTNAME={your_host_name}.{your_domain_name} in
| /etc/sysconfig/network
Sudo: | # visudo (or sudo visudo), then add this line:
| {username} ALL=(ALL) NOPASSWD: ALL
fstab: | /dev/sdd /ssd /ext4 noatime,nodiratime,discard,errors=remount-ro 0 2 # use fdisk -l | mkfs.ext4
Bzr: | to get this script to continue setting up a server once this initial setup is done, use:
| sudo yum install bzr; cd ~; bzr branch lp:percona-qa; cd percona-qa; ./setup_server.sh
Bzr setup:| [Optional] to configure bzr for uploading code, follow this;
| bzr whoami '{yourname} <{your.email@somedomain.com}>'
| bzr whoami # verify it worked
| bzr launchpad-login {your_launchpad_id}
| vi ~/.ssh/id_rsa.pub # copy this onto one line and paste in
| at: https://launchpad.net/~{your_launchpad_id}/+editsshkeys
| Alternatively (alternative to the last line), copy in the
| existing & correct ssh key into ~/.ssh
| To fix privileges use; cd ~; chmod 755 .ssh; cd .ssh;
| chmod 600 id_rsa; chmod 644 id_rsa.pub; chmod 644 known_hosts
| Some launchpad push examples:
| bzr push bzr+ssh://bazaar.launchpad.net/~randgen/randgen/rqg2/ --remember # for RQG + save location
| bzr push lp:percona-qa --remember # for percona-qa + save location
| bzr push :parent # for any downloaded R/W branch (handy shortcut), may fail
Bzr repo: | Setting up a local repo ensures that broken downloads (removed with rm -Rf {download_dir_name}) can be resumed (a must for PS)
| cd /;sudo mkdir bzr;sudo chown -R {username}:{username} bzr;sudo chmod -R 777 bzr;cd bzr;bzr init-repo .
This script will search through the standard input for a sequence of lines that match a sequence of regular expressions and report when a match is found.
This can be used to find blocks of matching code within a project.
Generate cmd startup scripts used in an RQG trial.
This script expects the trial number to be passed - e.g. 'startup 10' for actioning trial 10 (trial10.log) Other uses: 'startup 10 man': actions trial 10 as found in the RQG run without attempting to extract any tars Other uses: 'startup 0': local build mode (create scripts for a standard server setup without reference to RQG)
In other words, use:
$ startup.sh <trialno> When analyzing automated Percona Jenkins RQG runs
$ startup.sh <trialno> man When analyzing manually executed RQG runs
$ startup.sh <trialno> man2 Like man option above, but do not attempt to untar tarball again
$ startup.sh 0 When having extracted a binary tar build with ./build/build-binary.sh (no RQG)
This script quickly tests all mysqld options. Note it expects cores to be written to /cores/core.pid. Location can be changed in-script, but not filename To set your server up in this way (in terms of corefile generation), see core file setting part of setup_server.sh, available at lp:percona-qa