The modern, distributed and scalable implementation of a game inspired by Connect Four.
- There are two main factions: Yellow vs Red
- There's only one big playing field for all players
- Each player can make a move after a certain timeout
- Other players cannot place a chip close to another chip for a certain timeout
- If more people join, the timeouts are altered, and the playing field scales automatically
- More people => make playing field wider
- Fewer people => Mark (now superfluous) rows and columns as soon to be deleted, delete after a certain time passed
- If a disk you placed is part of a 4 disk long line you get points, and the associated columns are cleared
- You get more points if you're on the losing faction for balance purposes
- You can exchange your points for perks and skins
- To ensure fairness, some bots might be added to balance the teams
The service names are all a reference to the popular anime JoJo's Bizarre Adventure.
| Name | Role | Description |
|---|---|---|
| Doppio | Frontend | Svelte-based web client |
| Speedwagon | Load balancer | Envoy-based load balancing reverse proxy |
| Joestar | Scaling backend | Micronaut server for user authentication and caching |
| Rohan | Central backend | Micronaut server for central storage and processing of the game and its users |
The frontend is written in Svelte. As it's main purpose is to display the current state of the board, we decided that frameworks such as Angular are overkill.
We use Envoy as a reverse proxy to handle load balancing.
We had to use Envoy as it's the only reverse proxy which currently supports grpc-web.
The backend is split into two parts:
- Scaling: Communicates directly with our Frontend, issues and validates JWT, caches game state and sends game updates to all clients
- Central: Manages the actual game state, synchronizes requests and handles game logic, stores persistent information, such as user credentials and scores, in a json document
Both backends use Micronaut with Kotlin.
We decided not to use a database but instead store the (very minimalistic) data in a JSON document.
Bidirectional communication is enabled through gRPC.
For example, this allows Rohan to send a game update event to all Joestar instances, which then forward them realtime to all Doppio clients.
Because we use gRPC for communication, which is based on HTTP, we send the authentication token in the Authorization Header. We use a custom solution based on symmetric JWT for authentication, as we did not want to commit to a vendor-specific or work-in-progress solution (see Symmetric JWT for details).
We use streaming to propagate game state updates from Rohan to Joestar servers and from Joestar servers to Doppio clients. All other communication is request-based.
The streaming API offers hooks for various events. The request-based API is asynchronous by nature as well, but allows for convenient programming styles that are similar to those common in synchronous contexts.
| Control Flow | Kotlin | TypeScript |
|---|---|---|
| Streams | kotlinx.coroutines.flow.Flow + callback methods |
ClientReadableStream + callback methods |
| 'Synchronous' Requests | suspend fun + kotlinx.coroutines.BuildersKt.runBlocking |
Promise + async + await |
JWTs are issued and terminated by Joestar servers.
We use Micronaut Application Configuration to make some parts of the application configurable.
These configurations can be set through the application.yml file at compile time or through an environment variable at runtime.
The following configurations are probably the most interesting ones to configure, for a full list see the source code:
- Joestar
JOESTAR_PORT: The port at which Rohan is listeningROHAN_SERVER: The hostname or ip of the Rohan serverROHAN_PORT: The port at which Rohan is listeningJWT_SECRET: The base-64 encoded 512-bit private key used for signing the JWT
- Rohan
ROHAN_PORT: The port at which Rohan is listening
We use state-of-the-art technology and are using the latest version to take advantage of the latest language and framework features. For example, many Kotlin coroutine features we rely upon to easily implement real time updates were released as stable May 14th, just in time for the v1 release.
We try to avoid unnecessary code cohesion and favor testability instead. Currently, we have more than 200 unit tests.
In order to enable easy deployment every service is dockerized.
We do not use docker containers for developing, but you can easily build the containers locally to test their cross-container communication.
We also use docker to run gRPC code generation for grpc-web to ensure the code gets generated with the same compiler version on every device.
The docker images are generally optimized for file size and try to use the smallest available base image.
Our images also implement Docker Healthchecks which can be used to determine if a server is irrecoverably broken.
We use a dockerized Envoy reverse proxy server as a load balancer (see Envoy for details).
We use GitHub Actions to test and build ChaosConnect automatically. Every push is tested and built and Pull Requests can only be merged if all tests succeed. Creating a new Release automatically builds and publishes all required Packages such that users can run ChaosConnect with a specific version without needing to build the images themselves.
At the time of writing this, the latest stable version of ChaosConnect is deployed at chaos.honegger.dev. We decided to use the cloud provider Linode because they provide fair pricing and a good free initial credit.
Setting up hosting was very easy, you just copy the docker-compose.yml file, replace the placeholder jwt-secret with a generated one and you're almost ready to start. If you don't already have a valid certificate around, you can easily generate one using Certbot using a command similar to the following and mounting them to their corresponding location within the Speedwagon container:
sudo docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" -p 80:80 certbot/certbot certonly --standalone
We wanted real time updates and gRPC provides a way to stream real time updates. Implementing real time update from server-to-server and server-to-browser is no easy task and implementing a type safe variant through websockets would have taken a lot longer than just using a gRPC library.
Micronaut offers a couple of benefits:
- It supports gRPC out-of-the-box
- It offers Configuration Management
- It provides fast startup and performance by using build-time dependency injection
Kotlin is a modern, concise, and null safe programming language, which offer great language and ecosystem support for gRPC.
For example, implementing parallel code using Kotlin coroutines is vastly easier than using plain old java.
Svelte is a modern, light-weight SPA framework that meets the needs of small projects like ours.
Envoy is the only reverse proxy with official and native grpc-web support known to us.
Envoy uses round-robin for load balancing and only considers servers for load balancing which fulfill a readiness check. For example, a Joestar instance without a valid rohan connection is not considered ready and will not be load-balanced.
We couldn't easily use the micronaut-security package because the feature is still WIP for gRPC (see micronaut-grpc issue #164 for details).
The official token-based authentication works by using Google as a token provider, but we didn't want to have a vendor lock-in.
In the end, we decided to use simple symmetric tokens because of the project scope.
The client does use the metadata to send the token which was recommended back in 2018 by the grpc-web team.
In order to preserve network bandwidth, which can be pricey depending on the hosting provider, we use real-time updates with light-weight change events instead of complete states.
The easiest way to get ChaosConnect running is using docker-compose. We do not provide support for running the software components otherwise.
The docker-compose.yml file contains a placeholder for the JWT signing key.
This base64-encoded 512 bit secret is crucial for verifying user authenticity and must be configured before starting.
Another crucial point are HTTPS certificates, which have to be mounted to /etc/envoy/certs/cc.key and /etc/envoy/certs/cc.cert within the container.
If you're running on localhost, see the development guide below on how to generate self-signed certificates.
# Run services under https://localhost/ using images published to GitHub Packages
# Valid certificates are expected to be placed under ./certs/cc.key and ./certs/cc.cert
# Valid environment variables have to be configured
docker-compose up --scale joestar=5 -dFor local development JDK 16, Node 16 and a modern version of docker need to be installed. Our project also contains run configurations for IntelliJ, which is our IDE of choice.
In order to run components natively in development mode, the following commands are good to get started:
# Required once at the beginning and afterwards once the protocol buffer contract changes
docker-compose -f docker-compose.gen.yml up gen_grpc_joestar_client
# Run proxy (proxies http://localhost:5001 => http://localhost:5000 and http://localhost:5001/api => http://localhost:8080/)
# (Docker for Windows / Docker for Mac)
docker-compose -f docker-compose.proxy.yml up -d
# (Docker on Linux)
docker-compose -f docker-compose.proxy-linux.yml up -d
# Svelte Frontend
cd frontend
npm run dev
# backend
cd backend
# run joestar
./gradlew joestar:run
# run rohan
./gradlew rohan:runOr if you want to test the release configuration locally, you can build and run them locally:
# Generate self-signed certs, puts them in the ./proxy/certs directory
# Ensure the permissions are set such that the envoy docker user can read the certificate
docker-compose -f docker-compose.gen.yml up gen_self_signed_cert
# Run services hosted under http://localhost:5001, built locally, run 2 joestar instances
docker-compose -f docker-compose.dev.yml up --build --scale joestar=2