Description

This is the HTTPS reverse proxy configuration for https://tradingstrategy.ai for Caddy web server.

We use special Dockerimage to build support for

• Cloudflare plugin

• Datadog plugin

We source the website frontend from multiple servers and make them available in the same domain, same path structure, using Caddy.

The purpose of this setup is search engine optimisations and better UX.

• Proxy web frontend to SvelteKit

• Proxy API access to the oracle server

• Proxy documentation to the statically hosted documentation on Netlify

• HTTPS certificates are issued by Cloudflare

• The web server is configured to ignore traffic that is not from Cloudflare edge servers, to make it easier to manage malicious traffic

• Caddy OpenMetrics endpoint

Development flow

• Git checkout
• Edit Caddyfile config file locally
• Check syntax with caddy validate --config Caddyfile
• Sync Caddyfile to server
• Restart the production Docker

Running

AT this will bind all 80 on the Docker host. HTTPS traffic is terminated by Cloudflare.

# Start the service
docker-compose up -d

This will a start container caddy that terminates all tradingstrategy.ai web traffic.

This setup does not have any development testing - all changes to tweak URL configuration must be done directly on the production server. You need to have tradingstrategy.ai override in /etc/hosts to test.

Restarting Caddy on production and making config changes

Get an updated Caddyfile and then run

# Use caddy validate from Docker image to validate our config file
docker-compose run -v ${PWD}/Caddyfile:/tmp/Caddyfile caddy caddy validate --config /tmp/Caddyfile

docker-compose up --force-recreate -d

Caddy and caddy-logstash should be running

Caddy-logstash should have the following ENV variables set:

  ECS_SERVER: ${ECS_SERVER}
  ECS_USER: ${ECS_USER}
  ECS_PASSWORD: ${ECS_PASSWORD}

# Testing

Check URls:

- https://tradingstrategy.ai
- https://tradingstrategy.ai/api/explorer/
- https://tradingstrategy.ai/docs

# Diagnosing

Manually checking error counts:

```shell
curl http://127.0.0.1:6000/metrics|grep -i error

Ready Caddy logs real time:

tail -f logs/access.log

Reading logs using jq (See blog post). This will show status code, URL, IP address as a tail follow.

tail -f logs/access.log | jq '[.status, .request.remote_addr, .request.uri] | join(" ")'

"200 x:61170 /api/xyliquidity?pair_id=60282&time_bucket=1d"
"200 x:22642 /api/pair-details?chain_slug=ethereum&exchange_slug=uniswap-v2&pair_slug=akeno-eth"
"200 x:20432 /api/xyliquidity?pair_id=60291&time_bucket=1d"
"200 x:45076 /api/pair-details?chain_

With CloudFlare country and Ray ID information:

tail -f logs/access.log | jq '[.status, .request.headers["Cf-Connecting-Ip"][0], .request.headers["Cf-Ipcountry"][0], .request.headers["Cf-Ray"][0], .request.uri ] | join(" ")'

Note that jq matching is case-sensitive.

Then tailing page load requests only (no backend /api calls):

tail -f logs/access.log | jq --unbuffered -r '[.status, .request.headers["Cf-Connecting-Ip"][0], .request.headers["Cf-Ipcountry"][0], .request.headers["Cf-Ray"][0], .request.uri ] | join(" ")' | grep -v api

Showing error URLs (HTTP 500+):

cat logs/access.log | jq 'select(.status >= 500) | [.status, .request.host, .request.uri ] | join(" ")' 

Metrics

Metrics are expored on http://127.0.0.1:6000/metrics. Note that the port binding works on Linux only.

To test the metrics endpoint:

curl http://127.0.0.1:6000/metrics

See example of available metrics OpenMetrics

Datadog integration

You can make Datadog agent to read OpenMetrics from Caddy endpoint.

On the reverse proxy host, Go to DataDog OpenMetrics plugin directory and enable OpenMetrics:

/etc/datadog-agent/conf.d/openmetrics.d
mv conf.yaml.example conf.yaml

Edit conf.yaml:

instances:

  - openmetrics_endpoint: http://127.0.0.1:6000/metrics

    namespace: caddy

    metrics: ["caddy*"]

Restart agent:

sudo service datadog-agent restart

Give it 1 minute. Then check the OpenMetrics plugin is running:

datadog-agent status|grep -C 30 openmetrics

It should be running:

    openmetrics (1.15.2)
    --------------------
      Instance ID: openmetrics:caddy:17dd45bf18763b62 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
      Total Runs: 41
      Metric Samples: Last Run: 657, Total: 26,937
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 41
      Average Execution Time : 27ms
      Last Execution Date : 2022-01-09 21:11:18 CET / 2022-01-09 20:11:18 UTC (1641759078000)
      Last Successful Execution Date : 2022-01-09 21:11:18 CET / 2022-01-09 20:11:18 UTC (1641759078000)

Building and development

Build Caddy inside Docker with:

docker build

Write some crap to the port and see that it is not being closed by nc.

Logs

Stdout logs:

docker-compose logs caddy

File logs (needs a big screen or small font):

docker-compose exec -it caddy tail -f /var/log/caddy/access.log | jq .

Checking for specific string in logs

docker-compose exec -it caddy tail -f /var/log/caddy/access.log | grep por-que | jq .

Displaying custom fields:

docker-compose exec -it caddy tail -f /var/log/caddy/access.log | jq '[.status, .request.remote_addr, .request.uri] | join(" ")'

Checking 500s:

docker-compose exec -it caddy tail -f /var/log/caddy/access.log | | jq 'select(.status >= 500) | [.status, .request.host, .request.uri ] | join(" ")'

Notes

• Docker network_mode: host containers do not work on macOS

• DataDog OpenMetrcis checks source code