Skip to content

ThreeDotsLabs/watermill-benchmark

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Watermill Benchmark

This is an early set of tools for benchmarking watermill.

Warning: This tool is meant to provide a rough estimate on how fast each Pub/Sub can process messages. It uses very simplified infrastructure to set things up and default configurations.

Keep in mind that final performance depends on multiple factors.

How it works

  • All tests are run on a single 16 CPU GCloud compute instance (n1-highcpu-16).
  • Docker Compose is used to run Pub/Sub infrastructure and benchmark code (except for Google Cloud Pub/Sub).
  • The tool will first produce a big number of messages on a generated topic.
  • Then it will subscribe to the topic and consume all of the messages.
  • Multiple message sizes can be chosen (by default: 16, 64 and 256 bytes).

Results

This is an early version of benchmark results. Expect it to be updated and grow.

Kafka (one node)

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 44,090 108.285
64 41,371 108,848
256 41,497 111,756

NATS Jetstream (16 Subscribers)

Message size (bytes) Publish (messages / s) Subscribe (messages / s) Subscribe (messages / s - async ack)
16 49,255 33,009 63,065
64 49,296 33,016 62,667
256 48.488 32,573 62,745

NATS Jetstream (48 Subscribers)

Message size (bytes) Publish (messages / s) Subscribe (messages / s) Subscribe (messages / s - async ack)
16 48,882 45,275 90,917
64 48,681 44,746 89,527
256 48,097 44,487 90,510

Redis

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 61,642 11,213
64 58,554 11,252
256 58,906 11,408

SQL (MySQL)

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 5,599 167
64 5,625 168
256 5,381 164

SQL (PostgreSQL)

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 3,834 455
64 3,923 468
256 3,855 472

Google Cloud Pub/Sub (16 subscribers)

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 3,689 30,229
64 3,408 26,448
256 6,967 30,123

AMQP (RabbitMQ, 16 subscribers)

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 2,702 13,192
64 2,712 12,980
256 2,692 8,027

GoChannel

Message size (bytes) Publish (messages / s) Subscribe (messages / s)
16 331,882 118,943
64 298,847 123,499
256 373,053 130,940

VM Setup

The project includes Terraform definition for setting up an instance on Google Cloud Platform.

It will spin up a fresh Ubuntu 19.04 instance, install docker with dependencies and clone this repository.

Set environment variables:

# path to GCP credentials file
TF_VAR_credentials_path=
# project name on GCP
TF_VAR_project=
# public part of the key that you will use to access SSH
TF_VAR_pub_key_path=

Create the VM:

cd setup
terraform apply

The command will output the public IP address of the server. Use ssh with user benchmark to access it.

After running all benchmarks, destroy the VM:

terraform destroy

Configuration

Google Pub/Sub

Set environment variables in compose/.env:

# path to json file within the project with GCP credentials
GOOGLE_APPLICATION_CREDENTIALS=compose/key.json
# project name on GCP
GOOGLE_CLOUD_PROJECT=

Running

Run benchmarks with:

./run.sh <pubsub>