Skip to content

Commit

Permalink
Start to send data to Snowflake too (#20698)
Browse files Browse the repository at this point in the history
This PR adds support for sending telemetry events to AWS Kinesis.

In our AWS account we now have three new things:
* The [Kinesis data
stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring)
that we will actually write to.
* A [Firehose for
Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring)
that sends events from that stream to Axiom for ad-hoc queries over
recent data.
* A [Firehose for
Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring)
that sends events from that stream to Snowflake for long-term retention.
This Firehose also backs up data into an S3 bucket in case we want to
change how the system works in the future.

In a follow-up PR, we'll add support for ad-hoc telemetry events; and
slowly move away from the current Clickhouse defined schemas; though we
won't move off click house until we have what we need in Snowflake.

Co-Authored-By: Nathan <nathan@zed.dev>

Release Notes:

- N/A
  • Loading branch information
ConradIrwin authored Nov 15, 2024
1 parent f449e8d commit 6ff69fa
Show file tree
Hide file tree
Showing 7 changed files with 305 additions and 28 deletions.
59 changes: 41 additions & 18 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions crates/collab/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ async-stripe.workspace = true
async-tungstenite.workspace = true
aws-config = { version = "1.1.5" }
aws-sdk-s3 = { version = "1.15.0" }
aws-sdk-kinesis = "1.51.0"
axum = { version = "0.6", features = ["json", "headers", "ws"] }
axum-extra = { version = "0.4", features = ["erased-json"] }
base64.workspace = true
Expand Down
25 changes: 25 additions & 0 deletions crates/collab/k8s/collab.template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,31 @@ spec:
secretKeyRef:
name: blob-store
key: bucket
- name: KINESIS_ACCESS_KEY
valueFrom:
secretKeyRef:
name: kinesis
key: access_key
- name: KINESIS_SECRET_KEY
valueFrom:
secretKeyRef:
name: kinesis
key: secret_key
- name: KINESIS_STREAM
valueFrom:
secretKeyRef:
name: kinesis
key: stream
- name: KINESIS_REGION
valueFrom:
secretKeyRef:
name: kinesis
key: region
- name: BLOB_STORE_BUCKET
valueFrom:
secretKeyRef:
name: blob-store
key: bucket
- name: CLICKHOUSE_URL
valueFrom:
secretKeyRef:
Expand Down
Loading

0 comments on commit 6ff69fa

Please sign in to comment.