Skip to content

Commit

Permalink
Jun's edits to Get started page.
Browse files Browse the repository at this point in the history
  • Loading branch information
Oxyjun committed Sep 16, 2024
1 parent 07f0c6a commit 7158748
Showing 1 changed file with 40 additions and 21 deletions.
61 changes: 40 additions & 21 deletions src/content/docs/pipelines/get-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,7 @@ head:

import { Render, PackageManagers } from "~/components";

:::note

Pipelines is in **public beta**, and any developer with a [paid Workers plan](/workers/platform/pricing/#workers) can start using Pipelines immediately.

Pipelines lets you ingest real time data streams, such as click events on a website or logs from a service. You can send data to a Pipeline from a Worker, or via HTTP. Pipelines handles batching requests, and scales in response to your workload. Finally, Pipelines delivers the output into R2, as JSON files, automatically handling partitioning and compression for efficient querying.
Pipelines let you ingest real-time data streams, such as click events on a website, or logs from a service. You can send data to a Pipeline from a Worker, or via HTTP. Pipelines handle batching requests and scales in response to your workload. Finally, Pipelines deliver the output into R2 as JSON files, automatically handling partitioning and compression for efficient querying.

By following this guide, you will:

Expand All @@ -23,6 +19,10 @@ By following this guide, you will:
3. Post data to it via HTTP.
4. Verify the output file written to R2.

:::note

Pipelines is in **public beta**, and any developer with a [paid Workers plan](/workers/platform/pricing/#workers) can start using Pipelines immediately.

:::

## Prerequisites
Expand All @@ -34,46 +34,65 @@ To use Pipelines, you will need:
### 1. Enable Pipelines
TODO

## 1. Setup an R2 bucket to use as a destination
Pipelines is built to ingest data, and store it in an R2 bucket. Create a bucket, following our [Get Started Guide for R2](r2/get-started/) if you need to. Save the bucket name; you'll need it for the next sep.
## 1. Set up an R2 bucket to use as a destination

Pipelines are built to ingest data and store it in an R2 bucket. Create a bucket by following the [get started guide for R2](r2/get-started/).

Save the bucket name for the next step.

## 2. Create a Pipeline
To create a pipeline using Wrangler, run this command in a shell. Specify the name of your pipeline, as well as the name of the R2 bucket you created in Step 1.

To create a Pipeline using Wrangler, run the following command in a the terminal, and specify:

- The name of your Pipeline
- The name of the R2 bucket you created in step 1

```sh
npx wrangler pipelines create <PIPELINE-NAME> --r2 <R2-BUCKET-NAME>
npx wrangler pipelines create [PIPELINE-NAME] --r2 [R2-BUCKET-NAME]
```

Choose a descrpitive name for your Pipeline, related to the type of events you intend to ingest. You cannot change the Pipeline name after you have set it.
When choosing a name for your Pipeline:

Pipeline names must be 1 to 63 characters long. Queue names cannot contain special characters outside dashes (`-`), and must start and end with a letter or number.
1. Ensure it is descriptive and relevant to the type of events you intend to ingest. You cannot change the name of the Pipeline after creating it.
2. Pipeline names must be between 1 and 63 characters long.
3. The name cannot contain special characters outside dashes (`-`).
4. The name must start and end with a letter or a number.

Once your pipeline is created, you'll receive an HTTP endpoint which you can post data to. You should see output which resembles the below:
Once you create your Pipeline, you will receive a HTTP endpoint which you can post data to. You should see output as shown below:

```sh
🌀 Authorizing R2 bucket "<R2-BUCKET-NAME>"
🌀 Creating pipeline named "<PIPELINE-NAME>"
✅ Successfully created pipeline <PIPELINE-NAME> with ID <PIPELINE-ID>
```sh output
🌀 Authorizing R2 bucket "[R2-BUCKET-NAME]"
🌀 Creating pipeline named "[PIPELINE-NAME]"
✅ Successfully created pipeline [PIPELINE-NAME] with ID [PIPELINE-ID]

You can now send data to your pipeline with:
curl "https://<PIPELINE-ID>.pipelines.cloudflare.com/" -d '[{ ...JSON_DATA... }]'
```

## 3. Post data to your pipeline

Use a curl command in your terminal to post an array of JSON objects to the endpoint you received in Step 1.
Use a curl command in your terminal to post an array of JSON objects to the endpoint you received in step 1.

```sh
curl -H "Content-Type:application/json" \
-d '[{"account_id":"test", "other_data": "test"},{"account_id":"test","other_data": "test2"}]' \
<HTTP-endpoint>
```

Once the data has been successfully accepted by the Pipeline, you'll receive a success message.
Once the Pipeline successfully accepts the data, you will receive a success message.

Pipelines handles batching the data, so you can continue posting data to the Pipeline. Once a batch is filled up, the data will be partitioned by date, and written to your R2 bucket.
Pipelines handle batching the data, so you can continue posting data to the Pipeline. Once a batch is filled up, the data will be partitioned by date, and written to your R2 bucket.

## 4. Verify in R2
Navigate to the dashboard for the R2 bucket that you created in step 1. You should see a prefix for today's date. Click through, and you'll see a file created containing the JSON data you sent in Step 3.

By completing this guide, you've got a pipeline, with an HTTP endpoint as a source, and an R2 bucket as a destination.
Go to the R2 bucket you created in step 1 via [Cloudflare dashboard](https://dash.cloudflare.com/). You should see a prefix for today's date. Click through, and you will see a file created containing the JSON data you posted in step 3.

## Summary

By completing this guide, you have:

- Created a Pipeline
- Connected the Pipeline with an R2 bucket as destination.
- Posted data to the R2 bucket via HTTP.
- Verified the output in the R2 bucket.

0 comments on commit 7158748

Please sign in to comment.