Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update readme #3722

Closed
wants to merge 4 commits into from
Closed

Update readme #3722

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3dc9bf7
Update readme
mattrunyon Apr 24, 2023
df603ca
Add proto generation info
mattrunyon Apr 24, 2023
f63930c
Apply suggestions from code review
mattrunyon Apr 24, 2023
fc7c880
Update logo
mattrunyon May 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ devopsLocal/
pyclient/venv
pyclient/dist
pyclient/*.egg-info
venv
.venv
*.egg-info
Comment on lines +50 to +52
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why were these removed? I think the venv is there to handle venvs that pycharm creates, and the egg-info is for python files that are not checked in.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These were added. And anything in .gitignore without a leading or middle slash will match at any level. So it is effectively the same as **/venv/** or "ignore any folder at any depth called venv". I could actually remove the pyclient/venv since it will match under this new item


jenkins/failures

Expand Down
266 changes: 25 additions & 241 deletions README.md
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely agree that the readmes are due to be reworked. Looking at this has pointed out a number of problems. Overall, I think there are a number of things that need to be addressed:

  1. These documents are ultra stripped down. They do not provide enough context or meat to be good tools for outside contributors.
  2. The documents are very stripped down "core dumps", rather than high quality external-facing documentation for all of our user types. Basically, if you are already familiar with everything and know where to look, you can probably find what you are looking for. The documents need more high level storyline and context to be generally useful. For example, I don't think someone from DHE could use these to be quickly productive.
  3. In becoming super stripped down, these readmes have lost all material that would sell someone on trying the project. We need something.
  4. We are not effectively directing users to the website for tutorials and other key user-facing stuff.
  5. We aren't providing clear storylines that new users can follow. e.g. python pip is not present. Everything is way too focused only on developers of the DHC codebase without enough focused on the users.
  6. There are inconsistencies between this readme and the deephaven.io docs. We need a consistent message.

I'm going to ask @margaretkennedy to also get involved since I think there is a legitimate amount of work between the website and the readme to end up with something everyone will be happy with.

Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

![Deephaven Data Labs Logo](docs/images/Deephaven_GH_Logo.svg)

[Quick start for Docker user guide](https://deephaven.io/core/docs/tutorials/quickstart)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also reference the quick start for pip.
https://deephaven.io/core/docs/tutorials/quickstart-pip/


Deephaven Community Core is a real-time, time-series, column-oriented analytics engine
with relational database features.
Queries can seamlessly operate upon both historical and real-time data.
Expand Down Expand Up @@ -32,269 +34,51 @@ which functions as the data backbone for prominent hedge funds, banks, and finan

## Run Deephaven

This section is a quick start guide for running Deephaven from pre-built images. Almost all users will want to run Deephaven using pre-built images. It is the easiest way to deploy. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart).
If you want to use Deephaven, see our [Quick start for Docker guide](https://deephaven.io/core/docs/tutorials/quickstart).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also be directing people to https://deephaven.io/core/docs/tutorials/quickstart-pip/. We need to provide very obvious flight paths for docker vs pip users. This document is so lean that there is one flight path for end users.


Developers interested in tinkering with and modifying source code should build from the source code. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build).
Developers interested in modifying source code should read on for development instructions.
Comment on lines -37 to +39
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link to the build docs on the website should not have been lost.
https://deephaven.io/core/docs/how-to-guides/launch-build/


If you are not sure which of the two is right for you, use the pre-built images.
## Development Instructions

### Required Dependencies

Running Deephaven requires a few software packages.

| Package | Version | OS |
| -------------- | ----------------------------- | ------------ |
| docker | ^20.10.8 | All |
| docker-compose | ^1.29.0 | All |
| Windows | 10 (OS build 20262 or higher) | Only Windows |
| WSL | 2 | Only Windows |
Developing in Deephaven requires a few software packages.

You can check if these packages are installed and functioning by running:
```
docker version
docker-compose version
docker run hello-world
```
| Package | Version | OS |
| ------- | ----------------------------- | ------------ |
| Java | >=11, <20 | All |
| Python | >= 3.8 | All |
| Windows | 10 (OS build 20262 or higher) | Only Windows |
| WSL | 2 | Only Windows |
Comment on lines +47 to +52
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dependency list does not match what is on the website.
https://deephaven.io/core/docs/how-to-guides/launch-build/
Is python a hard requirement? If not, that is not represented in either place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python isn't a hard requirement. Docker isn't either technically I think.

I'm also not sure the Java version is entirely correct here or the website. I think gradle requires a more specific version to run, but not 100% sure (I had some issues in intellij b/c I had my gradle version set too new)


> :warning: **On Windows, all commands must be run inside a WSL 2 terminal.**

If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions.


### Create deployment

A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory.

You will need to `cd` into the deployment directory to launch or interact with the deployment.

```bash
mkdir deephaven-deployment
cd deephaven-deployment
```

> :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.**

### Launch: Python

Run the following commands to launch Deephaven for Python server applications.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with NLTK

Run the following commands to launch Deephaven for Python server applications with the [NLTK](https://nltk.org/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with PyTorch

Run the following commands to launch Deephaven for Python server applications with the [PyTorch](https://pytorch.org/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with SciKit-Learn

Run the following commands to launch Deephaven for Python server applications with the [SciKit-Learn](https://scikit-learn.org/stable/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with TensorFlow

Run the following commands to launch Deephaven for Python server applications with the [TensorFlow](https://www.tensorflow.org/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with example data

Run the following commands to launch Deephaven for Python server applications, with example data.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with example data and NLTK

Run the following commands to launch Deephaven for Python server applications, with example data and [NLTK](https://nltk.org/).

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with example data and PyTorch

Run the following commands to launch Deephaven for Python server applications, with example data and [PyTorch](https://pytorch.org/).

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with example data and SciKit-Learn

Run the following commands to launch Deephaven for Python server applications, with example data and [SciKit-Learn](https://scikit-learn.org/stable/).

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with example data and TensorFlow

Run the following commands to launch Deephaven for Python server applications, with example data and [TensorFlow](https://www.tensorflow.org/).

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```
### Local Development

### Launch: Groovy / Java

Run the following commands to launch Deephaven for Groovy / Java server applications.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Groovy / Java with example data

Run the following commands to launch Deephaven for Groovy / Java server applications, with example data.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Monitor logs

The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers.

Since the container is running detached, you will not see any logs. However, you can follow the logs by running:

```bash
docker-compose logs -f
```

Use CTRL+C to stop monitoring the logs and return to a prompt.

### Shutdown

The deployment can be brought down by running:

```bash
docker-compose down
```

### Manage example data

[Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning
to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them.

If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets.


To upgrade a deployment to the latest example data, run:

```bash
docker-compose run examples download
```

To see what other example data management commands are available, run:

```bash
docker-compose run examples
```

If your deployment does not have example data, these commands will fail with `ERROR: No such service`.


## Run Deephaven IDE

Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data.

- If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/).
- If Deephaven is running remotely, navigate to `http://<hostname>:10000/ide/`, where `<hostname>` is the address of the machine Deephaven is running on.

![alt_text](docs/images/ide_startup.png "Deephaven IDE")

# First query

From the Deephaven IDE, you can perform your first query.

This script creates two small tables: one for employees and one for departments.
It joins the two tables on the DeptID column to show the name of the department
where each employee works.

```python

from deephaven.TableTools import newTable, stringCol, intCol
from deephaven.conversion_utils import NULL_INT

left = newTable(
stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"),
intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT),
stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None)
)

right = newTable(
intCol("DeptID", 31, 33, 34, 35),
stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"),
stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111")
)

t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone")
```

![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query")
Comment on lines -240 to -277
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an example of material to get new users to consider the product -- which should not have been deleted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should put just an image (or gif/mp4 if it plays in github)? The code adds another place to maintain and this code is actually invalid by quite a few releases. The docs has the up to date version here

See the following READMEs for info about developing in various environments within the Deephaven core repository.

- [Java](./server/jetty-app/README.md)
- [Python](./py/README.md)
- [Web](./web/client-ui/README.md)
- [Go](./go/README.md)
- [C++](./cpp-client/README.md)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs an R link.


## Resources

* [Help!](https://github.com/deephaven/deephaven-core/discussions/969)
* [Deephaven Community Slack](https://deephaven.io/slack)
* [Discussions](https://github.com/deephaven/deephaven-core/discussions)
* [Deephaven Community Core docs](https://deephaven.io/core/docs/)
* [Java API docs](https://deephaven.io/core/javadoc/)
* [Python API docs](https://deephaven.io/core/pydoc/)
- [Deephaven Community Slack](https://deephaven.io/slack)
- [GitHub Discussions](https://github.com/deephaven/deephaven-core/discussions)
- [Deephaven Community Core docs](https://deephaven.io/core/docs/)
- [Java API docs](https://deephaven.io/core/javadoc/)
- [Python API docs](https://deephaven.io/core/pydoc/)

## Code Of Conduct

This project has adopted the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/).
For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [opencode@deephaven.io](mailto:opencode@deephaven.io)
with any additional questions or comments.


## License

Copyright (c) Deephaven Data Labs. All rights reserved.
Copyright &copy; Deephaven Data Labs. All rights reserved.

Provided under the [Deephaven Community License](LICENSE.md).
Loading
Loading