Skip to content

Commit

Permalink
Big updates
Browse files Browse the repository at this point in the history
  • Loading branch information
@jjbrosnan
jjbrosnan committed Jul 25, 2023
1 parent 9530d7d commit 6748dfc
Showing 1 changed file with 82 additions and 138 deletions.
220 changes: 82 additions & 138 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,9 @@ Deephaven includes an intuitive user experience and visualization tools.
It can ingest data from a variety of sources, apply computation and analysis algorithms
to that data, and build rich queries, dashboards, and representations with the results.

Deephaven Community Core is an open version of [Deephaven Enterprise](https://deephaven.io),
Deephaven Community Core is the open version of [Deephaven Enterprise](https://deephaven.io),
which functions as the data backbone for prominent hedge funds, banks, and financial exchanges.

[![Join the chat at https://gitter.im/deephaven/deephaven](https://badges.gitter.im/deephaven/deephaven.svg)](https://gitter.im/deephaven/deephaven?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
![Build CI](https://github.com/deephaven/deephaven-core/actions/workflows/build-ci.yml/badge.svg?branch=main)
![Quick CI](https://github.com/deephaven/deephaven-core/actions/workflows/quick-ci.yml/badge.svg?branch=main)
![Docs CI](https://github.com/deephaven/deephaven-core/actions/workflows/docs-ci.yml/badge.svg?branch=main)
![Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/check-ci.yml/badge.svg?branch=main)
![Nightly Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-check-ci.yml/badge.svg?branch=main)
![Nightly Benchmarks](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-benchmarks.yml/badge.svg?branch=main)

## Supported Languages

| Language | Server Application | Client Application |
Expand All @@ -30,200 +22,148 @@ which functions as the data backbone for prominent hedge funds, banks, and finan
| JavaScript | No | Yes |
| Go | No | Yes |
| R | No | Yes |
| gRPC | - | Yes |

## Run Deephaven

There are both server- and client-side applications available, as seen in the table above.

### Server-side
## Installation

Deephaven's server-side APIs allow you to connect directly to the Deephaven server and execute commands, write queries, and much more.
Deephaven Community Core can be used from both server- and client-side APIs. Server-side APIs allow users to connect directly to the Deephaven server and execute commands, write queries, and much more. Client-side APIs allow users to connect to a Deephaven server through a client application.

Most users will want to run Deephaven from pre-built images. It's the easiest way to configure, deploy, and use Deephaven. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart).
See the following documentation links for installation instructions and more:

Some Python users will want to use Deephaven entirely from Python. For instructions on starting and using Deephaven from Python, see [Quick start for pip-installed Deephaven](https://deephaven.io/core/docs/tutorials/quickstart-pip/).
- Python
- [Run from Docker](https://deephaven.io/core/docs/tutorials/quickstart/)
- [pip-installed](https://deephaven.io/core/docs/tutorials/quickstart/)
- [Groovy]
- [Run from Docker](https://deephaven.io/core/groovy/docs/tutorials/quickstart/)
- [Python client](https://pypi.org/project/pydeephaven/)
- [Java client](https://deephaven.io/core/docs/how-to-guides/java-client/)
- [JS client](https://deephaven.io/core/docs/reference/js-api/documentation/)
- [Go client](https://pkg.go.dev/github.com/deephaven/deephaven-core/go)
- [R client](https://github.com/deephaven/deephaven-core/blob/main/R/rdeephaven/README.md)

Developers interested in tinkering with and modifying source code should build from source. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build).
Deephaven's client APIs use [gRPC](https://grpc.io/), [protobuf](https://github.com/deephaven/deephaven-core/tree/main/proto/proto-backplane-grpc/src/main/proto/deephaven/proto), [Apache Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), and [Barrage](https://github.com/deephaven/barrage) to handle ticking data. Users who wish to build their own client APIs can use these tools to do so.

If you are not sure which of the two is right for you, use the pre-built images.
## Build from source

### Client-side
Users who wish to modify or contribute to this project should [build from source](https://deephaven.io/core/docs/how-to-guides/launch-build/) to run either the Python or Groovy server-side API.

Deephaven's client APIs allow you to connect to a Deephaven server and execute code. Documentation links are below:
Start by cloning this repository, and `cd` into the cloned repo.

- [Python](https://deephaven.io/core/client-api/python/)
- [Java/Groovy](https://deephaven.io/core/javadoc/)
- [C++](https://deephaven.io/core/client-api/cpp/)
- [JavaScript](https://deephaven.io/core/docs/reference/js-api/documentation/)
- [Go](https://pkg.go.dev/github.com/deephaven/deephaven-core/go)
- Documentation for R can be found with the `?` operator in R.
```bash
git clone https://github.com/deephaven/deephaven-core.git
cd deephaven-core
```

### Docker Dependencies
### Required dependencies

Running Deephaven from pre-built Docker images requires a few software packages.
Building and running Deephaven requires a few software packages.

| Package | Version | OS |
| -------------- | ----------------------------- | ------------ |
| git | ^2.25.0 | All |
| java | >=11, <20 | All |
| docker | ^20.10.8 | All |
| docker-compose | ^1.29.0 | All |
| Windows | 10 (OS build 20262 or higher) | Only Windows |
| WSL | 2 | Only Windows |

You can check if these packages are installed and functioning by running:
```

```bash
git version
java -version
docker version
docker-compose version
docker run hello-world
```

> :warning: **On Windows, all commands must be run inside a WSL 2 terminal.**
:::note

If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions.
Internally, the Java build process will use [Gradle Auto Provisioning](https://docs.gradle.org/current/userguide/toolchains.html#sec:provisioning)
to download and use the appropriate Java version for building and testing.

On Windows, all commands must be run inside a WSL 2 terminal.

### Create deployment
:::

A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory.
### Python

You will need to `cd` into the deployment directory to launch or interact with the deployment.
Set up a virtual environment.

```bash
mkdir deephaven-deployment
cd deephaven-deployment
python -m venv /tmp/my-dh-venv
source /tmp/my-dh-venv/bin/activate
```

> :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.**
### Launch options

Deephaven offers a variety of pre-made Docker Compose YAML files for different deployments. The following commands will launch Deephaven. Replace `<URL>` with that of the deployment from the table below that best suits your needs.
Then, build and install the wheel.

```bash
curl <URL> -O
docker compose pull
docker compose up -d
./gradlew py-server:assemble
pip install "py/server/build/wheel/deephaven_core-<version>-py3-non-any.whl[autocomplete]
```
| Language | Example data | Additional packages | URL |
| Python | No | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml |
| Python | No | SciKit-Learn | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml |
| Python | No | PyTorch | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml |
| Python | No | TensorFlow | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml |
| Python | No | NLTK | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml |
| Python | Yes | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml |
| Python | Yes | SciKit-Learn | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml |
| Python | Yes | PyTorch | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml |
| Python | Yes | TensorFlow | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml |
| Python | Yes | NLTK | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml |
| Groovy | No | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml |
| Groovy | Yes | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml |

Where:
### Monitor logs
- `<version>` is replaced by a deephaven-core [release](https://github.com/deephaven/deephaven-core/releases).
- Users who wish to use Deephaven without autocomplete should remove `[autocomplete]` from the `pip install` command.
The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers.

Since the container is running detached, you will not see any logs. However, you can follow the logs by running:
Lastly, build.
```bash
docker-compose logs -f
./gradlew server-jetty-app:run
```
Use CTRL+C to stop monitoring the logs and return to a prompt.

### Authentication

Deephaven, by default, comes stock with pre-shared key to authenticate any users trying to access it. If no key is set, a randomly generated key is printed to the Docker logs. The key can be found by monitoring the logs. It will look like this:
### Groovy/Java
```
================================================================================
Superuser access through pre-shared key is enabled - use 1c3opvgzl5scm to connect
Connect automatically to Web UI with http://localhost:10000/?psk=1c3opvgzl5scm
================================================================================
```

You can also `grep` for the key in the logs:
This single command will build and run the Deephaven Groovy server-side API from source.
```bash
docker compose logs -f | grep "access through pre-shared key"
./gradlew server-jetty-app:run
```
For more information on setting your own pre-shared key, see [How to configure and use pre-shared key authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-psk/).

For information on enabling anonymous authentication, see [How to enable anonymous authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-anon/).

### Shutdown

The deployment can be brought down by running:

```bash
docker-compose down
```

### Manage example data

[Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning
to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them.

If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets.

### Authentication
To upgrade a deployment to the latest example data, run:
Deephaven, by default, uses pre-shared key authentication to authenticate against unauthorized access. Users will be prompted for a key when connecting to an instance of the server. Unless otherwise specified, a new randomly generated key will be used each time the server is started. The key is printed to the Docker logs. To search the logs for the key, run:
```bash
docker-compose run examples download
docker compose logs -f | grep "access through pre-shared key"
```
To see what other example data management commands are available, run:
To set the key, add `-Ppsk` to the build command:
```bash
docker-compose run examples
./gradlew server-jetty-app:run -Ppsk=My-Password
```
If your deployment does not have example data, these commands will fail with `ERROR: No such service`.

### Build from source

If you are interested in tinkering and modifying source code for your own Deephaven deployment, or contributing to the project, see this [README](https://github.com/deephaven/deephaven-core/blob/main/server/jetty-app/README.md) for build instructions. There are several [required dependencies](https://deephaven.io/core/docs/how-to-guides/launch-build/#required-dependencies) to build and launch from source code.
### Connect to the server
Additionally, see our [code of conduct](https://github.com/deephaven/deephaven-core/blob/main/CODE_OF_CONDUCT.md) and [contributing guide](https://github.com/deephaven/deephaven-core/blob/main/CONTRIBUTING.md).

## Run Deephaven IDE

Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data.

- If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/).
- If Deephaven is running remotely, navigate to `http://<hostname>:10000/ide/`, where `<hostname>` is the address of the machine Deephaven is running on.

![alt_text](docs/images/ide_startup.png "Deephaven IDE")
Deephaven is run from a web browser, and can be connected to via `http://localhost:10000/ide`.
## First query
From the Deephaven IDE, you can perform your first query.
This script creates two small tables: one for employees and one for departments.
This Python script creates two small tables: one for employees and one for departments.
It joins the two tables on the DeptID column to show the name of the department
where each employee works.
```python

from deephaven.TableTools import newTable, stringCol, intCol
from deephaven.conversion_utils import NULL_INT

left = newTable(
stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"),
intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT),
stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None)
)

right = newTable(
intCol("DeptID", 31, 33, 34, 35),
stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"),
stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111")
)

t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone")
from deephaven import new_table
from deephaven.column import string_col, int_col
from deephaven.constants import NULL_INT
left = new_table([
string_col("LastName", ["Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"]),
int_col("DeptID", [31, 33, 33, 34, 34, NULL_INT]),
string_col("Telephone", ["(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None])
])
right = new_table([
int_col("DeptID", [31, 33, 34, 35]),
string_col("DeptName", ["Sales", "Engineering", "Clerical", "Marketing"]),
string_col("Telephone", ["(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111"])
])
t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone")
```
![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query")
Expand All @@ -237,15 +177,19 @@ t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone")
* [Java API docs](https://deephaven.io/core/javadoc/)
* [Python API docs](https://deephaven.io/core/pydoc/)
## Code Of Conduct
## Contributing
See [CONTRIBUTING](./CONTRIBUTING.md) for full instructions on how to contribute to this project.
### Code Of Conduct
This project has adopted the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/).
For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [opencode@deephaven.io](mailto:opencode@deephaven.io)
with any additional questions or comments.
## License
### License
Copyright (c) Deephaven Data Labs. All rights reserved.
Provided under the [Deephaven Community License](LICENSE.md).
Provided under the [Deephaven Community License](LICENSE.md).

0 comments on commit 6748dfc

Please sign in to comment.