Solr Bench

A comprehensive Solr performance benchmark and stress test framework.

Benchmarking & stress test for standard operations (indexing, querying, collection operations, restarting nodes) or advanced tests (e.g. simulating GC pauses etc.) for a specified Solr build.

Benchmarking

Prerequisites

GNU/Linux

Local Mode

Ubuntu/Debian:

# Install JDK 11, make sure it is the default JDK. Following is a potential way to install the right JDK:
sudo apt install openjdk-11-jdk

sudo apt install wget unzip zip ant ivy lsof git netcat make maven jq

Fedora/RHEL:

# Install JDK 11, make sure it is the default JDK. Following is a potential way to install the right JDK:
sudo yum install  java-11-openjdk-devel

sudo yum install wget unzip zip ant ivy lsof git nc make maven jq

GCP Mode

If running on GCP, spin up a coordinator VM where this suite will run from. Make sure to use a service account for that VM that has permissions to create other VMs.

The VM should have the following:

• Maven and other tools for building apt install wget unzip zip ant ivy lsof git netcat make openjdk-11-jdk maven jq (for Ubuntu/Debian) or sudo yum install wget unzip zip ant ivy lsof git nc make java-11-openjdk-devel maven jq (for Redhat/CentOS/Fedora)
• Terraform. wget https://releases.hashicorp.com/terraform/0.12.26/terraform_0.12.26_linux_amd64.zip; sudo unzip terraform_0.12.26_linux_amd64.zip -d /usr/local/bin

Mac OS

TBD (PRs welcome!)

Running the suite

In the coordinator VM, check out this solr-bench repository.

1. mvn clean compile assembly:single
2. ./cleanup.sh && ./stress.sh -c <commit> <config-file>

Example: ./cleanup.sh ./stress.sh -c dfde16a004206cc92e21cc5a6cad9030fbe13c20 suites/stress-facets-local.json

Available tests

Note: This is subject to change

1. cluster-test.json : Creates an 8 node cluster and create 1000 collections of various numShards and measure shutdown & restart performance
2. stress-facets-local.json : Indexes 20 million documents from an ecommerce events dataset, issues 5k facet queries against it.
3. prs-vs-nonprs.json : Creates 1k collections using non-PRS mode (default), then restarts 7 nodes, then cleans up, and repeats the same for PRS.

Results

• Results are available after the benchmark in ./suites/results/<testname (without the .json suffix)>/<commit-id>/results.json file.

Datasets

TBD

Visualization

To select/process the test results of specific branch or branch comparisons:

• python3 graph-scripts/generate_graph_json.py -r suites/results/<test name> -b main (only select tests with commits that belongs to main branch)
• python3 graph-scripts/generate_graph_json.py -r suites/results/<test name> -b main...my-branch (compare tests between main and my-branch)

Note: The `<test name>` currently is the test config file name without the `.json` file extension.

Note: Many result directories can be specified at once, in case multiple tests have been performed. Example: `python3 graph-scripts/generate_graph_json.py -r suites/results/stress-facets-local -r suites/results/cluster-test -b branch_9x...branch_9_1`

The script should generate graph/graph-data.js. Open graph/graph.html, it should show graphs grouped by branches and test tasks

Dependencies on mac

Mac OS requires a few tools to run this script. Install the following:

1. brew install coreutils

Prometheus Exporter

Currently, query and index benchmark metrics can also be exported via a Prometheus Exporter endpoint (default at port 11100 at /metrics).

To enable this, simply set prometheus-export node in the root level of the configuration, there are 2 optional fields under such node:

• port : the port to run the http server which exposes the metrics, default 11100 (to avoid conflicts with node-exporter etc at 9100)
• type-label : an extra label value with key "type" will be sent along the metrics, by default it has value "all"

for example:

{
  ...
  "prometheus-export": { "port": 1234, "type-label": "facet" }
}

To enable Prometheus with defaults, simply add "prometheus-export": {}

Label override can also defined as prometheus-type-label under query-benchmark

External mode with SSL

For testing when external mode (ie cluster.provisioning-method as "external"), which benchmarks against an external (existing) Solr cluster, we might want to set the solr-bench driver with extra JVM args for certs/keys location.

Set at the root of the json config with extra-jvm-args, for example : "extra-jvm-args": "-Djavax.net.ssl.trustStore=<truststore path> -Djavax.net.ssl.trustStorePassword=<secret> -Djavax.net.ssl.keyStore=<keystore path> -Djavax.net.ssl.keyStorePassword=<secret>”

Acknowledgement

This started as a project funded by Google Summer of Code (SOLR-10317), later supported by FullStory.