Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Groovy container w/ examples and Redpanda #4673

Merged
merged 4 commits into from
Jan 17, 2024
Merged

Add Groovy container w/ examples and Redpanda #4673

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b7b57ee
Groovy container w/ redpanda and examples. Also README.
jjbrosnan Oct 19, 2023
2caf1e7
Updates from Devin's review - remove apicurio from docker-compose fil…
jjbrosnan Nov 9, 2023
b1b7e2a
Updates from Devin's review
jjbrosnan Dec 12, 2023
aa459a9
Make consistent with Redpanda
jjbrosnan Dec 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions containers/groovy-examples-redpanda/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Overview

A Docker Compose deployment for [Deephaven](https://deephaven.io).

## Features

- [Deephaven](https://deephaven.io)
- [Groovy](https://groovy-lang.org/) scripting
- [Deephaven Examples](https://github.com/deephaven/examples)
- [Redpanda](https://vectorized.io/), a Kakfa-compatible event streaming platform.

## Launch Deephaven

For instructions on launching and using Kafka in Deephaven, see[How to connect to a Kafka stream](https://deephaven.io/core/groovy/docs/how-to-guides/data-import-export/kafka-stream/). For instructions on getting started with Deephaven, see the [Quickstart](https://deephaven.io/core/groovy/docs/tutorials/quickstart).

To launch Deephaven, execute the following in your deployment directory:

```bash
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples-redpanda/docker-compose.yml
curl -O "${compose_file}"
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

docker compose pull
docker compose up -d
```
38 changes: 38 additions & 0 deletions containers/groovy-examples-redpanda/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
version: "3.4"

services:
deephaven:
image: ghcr.io/deephaven/server-slim:${VERSION:-latest}
ports:
- "${DEEPHAVEN_PORT:-10000}:10000"
volumes:
- ./data:/data
environment:
- START_OPTS=-Xmx4g

examples:
image: ghcr.io/deephaven/examples
volumes:
- ./data:/data
command: initialize

redpanda:
command:
- redpanda
- start
- --smp
- '1'
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- '0'
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
image: docker.redpanda.com/vectorized/redpanda:v22.2.2
ports:
- 9092:9092
- 29092:29092

4 changes: 2 additions & 2 deletions containers/groovy-examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
docker compose up -d
```
21 changes: 21 additions & 0 deletions containers/groovy-redpanda/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Overview

## Features

- [Deephaven](https://deephaven.io)
- [Groovy](https://groovy-lang.org/) scripting
- [Redpanda](https://vectorized.io/), a Kafka-compatible event streaming platform

## Launch Deephaven

For instructions on launching and using Kafka in Deephaven, see[How to connect to a Kafka stream](https://deephaven.io/core/groovy/docs/how-to-guides/data-import-export/kafka-stream/). For instructions on getting started with Deephaven, see the [Quickstart](https://deephaven.io/core/groovy/docs/tutorials/quickstart).

To launch Deephaven, execute the following in your deployment directory:

```sh
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-redpanda/docker-compose.yml
curl -O "${compose_file}"
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

docker compose pull
docker compose up -d
```
31 changes: 31 additions & 0 deletions containers/groovy-redpanda/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
version: "3.4"

services:
deephaven:
image: ghcr.io/deephaven/server-slim:${VERSION:-latest}
ports:
- "${DEEPHAVEN_PORT:-10000}:10000"
volumes:
- ./data:/data
environment:
- START_OPTS=-Xmx4g

redpanda:
command:
- redpanda
- start
- --smp
- '1'
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- '0'
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
image: docker.redpanda.com/vectorized/redpanda:v22.2.2
ports:
- 9092:9092
- 29092:29092
6 changes: 3 additions & 3 deletions containers/groovy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ A Docker Compose deployment for [Deephaven](https://deephaven.io).

## Launch Deephaven

For launch instructions, see the [README](https://github.com/deephaven/deephaven-core/#launch-groovy--java). For full instructions to work with Deephaven, see the [Quick start](https://deephaven.io/core/docs/tutorials/quickstart).
For launch instructions, see the [README](https://github.com/deephaven/deephaven-core/#launch-groovy--java). For full instructions to work with Deephaven, see the [Quick start](https://deephaven.io/core/groovy/docs/tutorials/quickstart).

To launch Deephaven, execute the following in your deployment directory:

```bash
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
docker compose up -d
```
8 changes: 4 additions & 4 deletions containers/python-examples-redpanda/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,18 @@ A Docker Compose deployment for [Deephaven](https://deephaven.io).
- [Deephaven](https://deephaven.io)
- [Python](https://python.org/) scripting
- [Deephaven Examples](https://github.com/deephaven/examples)
- [Redpanda](https://vectorized.io/), a Kakfa-compatible event streaming platform.
- [Redpanda](https://vectorized.io/), a Kakfa-compatible event streaming platform

## Launch Deephaven

For instructions on launching and using Kafka, see [Kafka in Deephaven: an introduction](https://deephaven.io/core/docs/conceptual/kafka-in-deephaven/) and [Simple Kafka import](https://deephaven.io/core/docs/how-to-guides/kafka-simple/). For full instructions to work with Deephaven, see the [Quick start](https://deephaven.io/core/docs/tutorials/quickstart).
For instructions on launching and using Kafka in Deephaven, see[How to connect to a Kafka stream](https://deephaven.io/core/groovy/docs/how-to-guides/data-import-export/kafka-stream/). For instructions on getting started with Deephaven, see the [Quickstart](https://deephaven.io/core/docs/tutorials/quickstart).

To launch Deephaven, execute the following in your deployment directory:

```bash
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples-redpanda/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
docker compose up -d
```
12 changes: 0 additions & 12 deletions containers/python-examples-redpanda/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,3 @@ services:
ports:
- 9092:9092
- 29092:29092

registry:
image: apicurio/apicurio-registry-mem:1.2.2.Final
expose:
- 8080
ports:
- "8081:8080"
environment:
QUARKUS_PROFILE: prod
KAFKA_BOOTSTRAP_SERVERS: redpanda:9092
APPLICATION_ID: registry_id
APPLICATION_SERVER: localhost:9000
29 changes: 0 additions & 29 deletions containers/python-examples/All-AI/README.md
View file
Open in desktop

This file was deleted.

17 changes: 0 additions & 17 deletions containers/python-examples/All-AI/docker-compose.yml
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions containers/python-examples/NLTK/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
4 changes: 2 additions & 2 deletions containers/python-examples/PyTorch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
4 changes: 2 additions & 2 deletions containers/python-examples/SciKit-Learn/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
4 changes: 2 additions & 2 deletions containers/python-examples/TensorFlow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
4 changes: 2 additions & 2 deletions containers/python-examples/base/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
23 changes: 23 additions & 0 deletions containers/python-redpanda/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Overview

A Docker Compose deployment for [Deephaven](https://deephaven.io).

## Features

- [Deephaven](https://deephaven.io)
- [Python](https://python.org) scripting
- [Redpanda](https://vectorized.io/), a Kafka-compatible event streaming platform

## Launch Deephaven

For instructions on launching and using Kafka, see [How to connect to a Kafka stream](https://deephaven.io/core/docs/how-to-guides/data-import-export/kafka-stream/). For instructions on getting started with Deephaven, see the [Quickstart](https://deephaven.io/core/docs/tutorials/quickstart).

To launch Deephaven, execute the following in your deployment directory:

```sh
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-redpanda/docker-compose.yml
curl -O "${compose_file}"

docker compose pull
docker compose up -d
```
31 changes: 31 additions & 0 deletions containers/python-redpanda/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
version: "3.4"

services:
deephaven:
image: ghcr.io/deephaven/server:${VERSION:-latest}
ports:
- "${DEEPHAVEN_PORT:-10000}:10000"
volumes:
- ./data:/data
environment:
- START_OPTS=-Xmx4g

redpanda:
command:
- redpanda
- start
- --smp
- '1'
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- '0'
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should adopt the formatting style redpanda uses for this:

    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
      - --smp 1
      - --memory 1G
      - --mode dev-container

I've modified this from https://docs.redpanda.com/current/get-started/quick-start/; we should test it in all of the different compose files to make sure it works. Do we use the schema registry for any of our examples?

Copy link
Contributor Author

@jjbrosnan jjbrosnan Dec 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we do. I've made updates that make our syntax consistent with Redpanda and tested it to make sure it works with the examples in our Kafka documentation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we're consistent w/ the redpanda quickstart - they use internal and external, and use dev-container.

image: docker.redpanda.com/vectorized/redpanda:v22.2.2
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
ports:
- 9092:9092
- 29092:29092
28 changes: 0 additions & 28 deletions containers/python/All-AI/README.md
View file
Open in desktop

This file was deleted.

11 changes: 0 additions & 11 deletions containers/python/All-AI/docker-compose.yml
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions containers/python/NLTK/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
4 changes: 2 additions & 2 deletions containers/python/PyTorch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ To launch Deephaven, execute the following in your deployment directory:
compose_file=https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml
curl -O "${compose_file}"

docker-compose pull
docker-compose up -d
docker compose pull
docker compose up -d
```
Loading
Loading