WarpStream Fly.io Template

This repository contains a template for deploying WarpStream Agents to fly.io.

Instructions

Technical Note: In this documentation, the terms fly and flyctl are used interchangeably, consistent with the Fly.io documentation. During the following steps, you may encounter errors due to the latency of various processes starting up. If you successfully complete the process, these errors can generally be disregarded. For further guidance, please refer to the video walkthrough available here.

1. Sign up for a WarpStream account.
2. Sign up for a fly.io account and download flyctl.
3. Clone this repository and cd into it.
4. Review and edit fly.toml to fill in your specific details. At the very least, you should:
   1. Pick a name for your WarpStream cluster and put it in the value of app.
   2. Determine which fly.io region you want to deploy to and set that in primary_region.
   3. Edit WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE to use the same value as app. For example, if your app name is warpstream_fly_123 then the value of WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE should be warpstream_fly_123.fly.dev.
   4. Fill in the value of WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID from the WarpStream Admin Console. This ID will begin with vci_
   5. Fill in the value of WARPSTREAM_BUCKET_URL. If you don't have an object store/bucket selected already, we've found that the fly.io integration of Tigris is seamless. Their documentation can be found here With fly.io installed, the command is fly storage create --public. Alternatively, Cloudflare R2 also works well with fly.io. We have more documentation on how to set that up here. The string will look like the this WARPSTREAM_BUCKET_URL = "s3://FILL_ME_IN_BUCKET_NAME?s3ForcePathStyle=true&region=auto&endpoint=https://FILL_ME_IN_ENDPOINT" where FILL_ME_IN_BUCKET_NAME is equal to your app name, and FILLE_ME_IN_ENDPOINT is the URL of your object store.
5. Run flyctl launch to create your WarpStream cluster.
6. Run flyctl deploy to make sure your cluster is deployed.
7. Run flyctl ips allocate-v4 to create a public IP4 address at which your WarpStream cluster can be reached.
8. Run fly secrets set AWS_ACCESS_KEY_ID=XXXXXXXX with whatever the value of your object store access key is.
9. Run fly secrets set AWS_SECRET_ACCESS_KEY=XXXXXXXX with whatever the value of your object store secret access key is.
10. Run fly secrets set WARPSTREAM_API_KEY=XXXXXXXX with your WarpStream Application Key from the admin console.
11. Run fly scale show to see what machines your WarpStream Agents are running on.
12. The flyctl default machines are a bit too underpowered for the WarpStream Agents, so run fly scale vm performance-2x to get a bit more CPU and RAM.
13. If you want high availability, run fly scale count 2 or fly scale count 3 to run more than one instance of the WarpStream Agents.

Congrats! You have a fully functional WarpStream cluster running on fly.io now backed by either a Tigris S3-compatible, or Cloudflare R2 object storage.

Before you attempt to use the cluster, make sure that authentication is properly enabled. Run the following command:

warpstream kcmd -type diagnose-connection -bootstrap-host $YOUR_APP_NAME.fly.dev -tls

You should see an error about how SASL authentication failed, which means your cluster is properly enforcing authentication. If you see success, DO NOT USE THIS CLUSTER AS IT IS NOT PROPERLY ENFORCING AUTHENTICATION.

Now that we've confirmed authentication is enforced, navigate to the WarpStream Admin Console and create a new set of credentials for this cluster.

Once you have the credentials, try the diagnose-connection command again, but this time with credentials:

warpstream kcmd -type diagnose-connection -bootstrap-host warpstream-agent-demo.fly.dev -tls -username ccun_XXXXXXXX -password ccp_XXXXXXXX

This time you should get a successful response. That means your WarpStream cluster is ready to go!