diff --git a/.gitignore b/.gitignore index 8c1c4d98f1c..d773f5dae3a 100644 --- a/.gitignore +++ b/.gitignore @@ -47,6 +47,9 @@ devopsLocal/ pyclient/venv pyclient/dist pyclient/*.egg-info +venv +.venv +*.egg-info jenkins/failures diff --git a/README.md b/README.md index e1bf0f0b96b..333d9e97b74 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ ![Deephaven Data Labs Logo](docs/images/Deephaven_GH_Logo.svg) +[Quick start for Docker user guide](https://deephaven.io/core/docs/tutorials/quickstart) + Deephaven Community Core is a real-time, time-series, column-oriented analytics engine with relational database features. Queries can seamlessly operate upon both historical and real-time data. @@ -32,259 +34,42 @@ which functions as the data backbone for prominent hedge funds, banks, and finan ## Run Deephaven -This section is a quick start guide for running Deephaven from pre-built images. Almost all users will want to run Deephaven using pre-built images. It is the easiest way to deploy. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart). +If you want to use Deephaven, see our [Quick start for Docker guide](https://deephaven.io/core/docs/tutorials/quickstart). -Developers interested in tinkering with and modifying source code should build from the source code. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build). +Developers interested in modifying source code should read on for development instructions. -If you are not sure which of the two is right for you, use the pre-built images. +## Development Instructions ### Required Dependencies -Running Deephaven requires a few software packages. - -| Package | Version | OS | -| -------------- | ----------------------------- | ------------ | -| docker | ^20.10.8 | All | -| docker-compose | ^1.29.0 | All | -| Windows | 10 (OS build 20262 or higher) | Only Windows | -| WSL | 2 | Only Windows | +Developing in Deephaven requires a few software packages. -You can check if these packages are installed and functioning by running: -``` -docker version -docker-compose version -docker run hello-world -``` +| Package | Version | OS | +| ------- | ----------------------------- | ------------ | +| Java | >=11, <20 | All | +| Python | >= 3.8 | All | +| Windows | 10 (OS build 20262 or higher) | Only Windows | +| WSL | 2 | Only Windows | > :warning: **On Windows, all commands must be run inside a WSL 2 terminal.** -If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions. - - -### Create deployment - -A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory. - -You will need to `cd` into the deployment directory to launch or interact with the deployment. - -```bash -mkdir deephaven-deployment -cd deephaven-deployment -``` - -> :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.** - -### Launch: Python - -Run the following commands to launch Deephaven for Python server applications. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with NLTK - -Run the following commands to launch Deephaven for Python server applications with the [NLTK](https://nltk.org/) module pre-installed. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with PyTorch - -Run the following commands to launch Deephaven for Python server applications with the [PyTorch](https://pytorch.org/) module pre-installed. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with SciKit-Learn - -Run the following commands to launch Deephaven for Python server applications with the [SciKit-Learn](https://scikit-learn.org/stable/) module pre-installed. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with TensorFlow - -Run the following commands to launch Deephaven for Python server applications with the [TensorFlow](https://www.tensorflow.org/) module pre-installed. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with example data - -Run the following commands to launch Deephaven for Python server applications, with example data. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with example data and NLTK - -Run the following commands to launch Deephaven for Python server applications, with example data and [NLTK](https://nltk.org/). - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with example data and PyTorch - -Run the following commands to launch Deephaven for Python server applications, with example data and [PyTorch](https://pytorch.org/). - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with example data and SciKit-Learn - -Run the following commands to launch Deephaven for Python server applications, with example data and [SciKit-Learn](https://scikit-learn.org/stable/). - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with example data and TensorFlow - -Run the following commands to launch Deephaven for Python server applications, with example data and [TensorFlow](https://www.tensorflow.org/). - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` +### Local Development -### Launch: Groovy / Java - -Run the following commands to launch Deephaven for Groovy / Java server applications. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Groovy / Java with example data - -Run the following commands to launch Deephaven for Groovy / Java server applications, with example data. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Monitor logs - -The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers. - -Since the container is running detached, you will not see any logs. However, you can follow the logs by running: - -```bash -docker-compose logs -f -``` - -Use CTRL+C to stop monitoring the logs and return to a prompt. - -### Shutdown - -The deployment can be brought down by running: - -```bash -docker-compose down -``` - -### Manage example data - -[Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning -to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them. - -If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets. - - -To upgrade a deployment to the latest example data, run: - -```bash -docker-compose run examples download -``` - -To see what other example data management commands are available, run: - -```bash -docker-compose run examples -``` - -If your deployment does not have example data, these commands will fail with `ERROR: No such service`. - - -## Run Deephaven IDE - -Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data. - -- If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/). -- If Deephaven is running remotely, navigate to `http://:10000/ide/`, where `` is the address of the machine Deephaven is running on. - -![alt_text](docs/images/ide_startup.png "Deephaven IDE") - -# First query - -From the Deephaven IDE, you can perform your first query. - -This script creates two small tables: one for employees and one for departments. -It joins the two tables on the DeptID column to show the name of the department -where each employee works. - -```python - -from deephaven.TableTools import newTable, stringCol, intCol -from deephaven.conversion_utils import NULL_INT - -left = newTable( - stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"), - intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT), - stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None) - ) - -right = newTable( - intCol("DeptID", 31, 33, 34, 35), - stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"), - stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111") - ) - -t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") -``` - -![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query") +See the following READMEs for info about developing in various environments within the Deephaven core repository. +- [Java](./server/jetty-app/README.md) +- [Python](./py/README.md) +- [Web](./web/client-ui/README.md) +- [Go](./go/README.md) +- [C++](./cpp-client/README.md) ## Resources -* [Help!](https://github.com/deephaven/deephaven-core/discussions/969) -* [Deephaven Community Slack](https://deephaven.io/slack) -* [Discussions](https://github.com/deephaven/deephaven-core/discussions) -* [Deephaven Community Core docs](https://deephaven.io/core/docs/) -* [Java API docs](https://deephaven.io/core/javadoc/) -* [Python API docs](https://deephaven.io/core/pydoc/) +- [Deephaven Community Slack](https://deephaven.io/slack) +- [GitHub Discussions](https://github.com/deephaven/deephaven-core/discussions) +- [Deephaven Community Core docs](https://deephaven.io/core/docs/) +- [Java API docs](https://deephaven.io/core/javadoc/) +- [Python API docs](https://deephaven.io/core/pydoc/) ## Code Of Conduct @@ -292,9 +77,8 @@ This project has adopted the [Contributor Covenant Code of Conduct](https://www. For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [opencode@deephaven.io](mailto:opencode@deephaven.io) with any additional questions or comments. - ## License -Copyright (c) Deephaven Data Labs. All rights reserved. +Copyright © Deephaven Data Labs. All rights reserved. Provided under the [Deephaven Community License](LICENSE.md). diff --git a/docs/images/Deephaven_GH_Logo.svg b/docs/images/Deephaven_GH_Logo.svg index 7bb82b30714..5e7c25fc714 100644 --- a/docs/images/Deephaven_GH_Logo.svg +++ b/docs/images/Deephaven_GH_Logo.svg @@ -1,58 +1,104 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/proto/README.md b/proto/README.md new file mode 100644 index 00000000000..91b1ca78f03 --- /dev/null +++ b/proto/README.md @@ -0,0 +1,17 @@ +# Deephaven Protobuf + +For Java, proto changes will be automatically reflected when building/running the Java server app. Sections below list files that must be manually generated and committed to source control when proto files change. + +## Python/Go/C++ API + +```sh +./gradlew :py-client:updateProtobuf :go:updateProtobuf :cpp-client:updateProtobuf +``` + +## Authorization + +See [authorization readme](../authorization/README.md) + +## JS API + +Needs info on how to generate diff --git a/py/README.md b/py/README.md new file mode 100644 index 00000000000..925de4ab634 --- /dev/null +++ b/py/README.md @@ -0,0 +1,76 @@ +# Deephaven Python Development + +## Requirements + +- Python >=3.8 + +## Setting up Virtual Environment + +For any of the options below, you should first set up a virtual environment in Python. Once created, you must activate the virtual environment every time you open a shell. + +```sh +python3 -m venv .venv # Create. Only needed once +source .venv/bin/activate # Activate the virtual environment +deactivate # Exit virtual environment when done +``` + +**Note:** + +- This is not an exhaustive guide to managing Python environments. +- Depending on your OS and how your PATH is set up, you may need to use `python3`, or a path to the explicit Python version you want to use. +- You may set up a "permanent" virtual environment location. +- You'll need to re-install the wheel whenever you make Python code changes that affect the wheel. +- `pip` can be a pain if you are trying to (re-)install a wheel with the same version number as before. + - A `pip install --force-reinstall --no-deps "py/server/build/wheel/deephaven_core--py3-none-any.whl[autocomplete]"` may do the trick. +- You can install other Python packages in your venv using `pip install `. +- You can set up multiple virtual environments and switch between them as necessary using `source /path/to/other-venv/bin/activate`. +- You can de-activate the virtual environment by running `deactivate`. +- You can use the `VIRTUAL_ENV` environment variable instead of sourcing/activating virtual environments: `VIRTUAL_ENV=/my/venv ./gradlew server-jetty-app:run`. + +## Local Development + +Using a `pip` editable install is recommended so that Python changes are reflected when your server is restarted. These instructions are for the Python server package in `py/server` but should apply to any other Python packages. + +Install the Python server package. This will install the dependencies as well as the autocomplete dependencies. + +```sh +DEEPHAVEN_VERSION=0.23.0 pip install -e "py/server[autocomplete]" +``` + +**Note:** Pip may report an error that the version is not what is expected if you have installed the locally built version before. Adjust the version to match what `pip` expects. + +After installing, you can run `pip list` and you should see the package `deephaven-core` with an editable project location listed. + +Now you can start the Java server by following the instructions [here](../server/jetty-app/README.md). Any time you make Python changes, restart the server to apply them. + +## Local Build + +You can use these instructions if you do not plan on making frequent changes to your local Python environment or need to try things in a more production-like environment locally. + +First, run the following command to build the server and embedded-server if needed. The embedded-server is used to start [Deephaven from Python](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/#python-embedded-server). + +```sh +./gradlew :py-server:assemble + +# If using embedded-server +# You can run both tasks w/ ./gradlew :task1 :task2 if you need both +./gradlew :py-embedded-server:assemble +``` + +Then run the following commands to install the server package. Note that the `wheel` folder should only contain 1 file, so you should be able to tab-complete the wheel file. + +**Note:** Add the `--no-deps` flag on subsequent installs to skip re-installing other dependencies. + +```sh +pip install --force py/server/build/wheel/deephaven_core--py3-none-any.whl + +# If using embedded-server +# You can combine as pip install core.whl server.whl if you need both +pip install --force py/embedded-server/build/wheel/deephaven_server--py3-none-any.whl +``` + +Start the Java server by following the instructions [here](../server/jetty-app/README.md). + +## Tests + +This section needs info about how to run tests. diff --git a/py/server/README.md b/py/server/README.md index b2a77ae785f..0ae0373ed59 100644 --- a/py/server/README.md +++ b/py/server/README.md @@ -1,4 +1,3 @@ - # Deephaven Python Integration Package Deephaven Python Integration Package is created by Deephaven Data Labs. It allows Python developers, including data @@ -7,13 +6,14 @@ maximum performance. By taking advantage of the unique streaming table capabilit facilities (Kafka, Parquet, CSV, SQL, etc.), Python developers can quickly put together a real-time data processing pipeline that is high performing and easy to consume. - ## Install -Because this package depends on the Deephaven server, it comes preinstalled with Deephaven Docker images and is made + +Because this package depends on the Deephaven server, it comes preinstalled with Deephaven Docker images and is made available at runtime in the Python console in the Deephaven Web UI. ## Quick start -``` python + +```python from deephaven import read_csv from deephaven.stream.kafka.consumer import kafka_consumer, TableType from deephaven.plot import Figure, PlotStyle @@ -26,8 +26,6 @@ plot = Figure() \ .show() ``` -## Related documentation -* https://deephaven.io/ - ## API Reference -[start here] https://deephaven.io/core/pydoc/ + +[Pydoc](https://deephaven.io/core/pydoc/) diff --git a/server/jetty-app/README.md b/server/jetty-app/README.md index c148c094d78..b76ab30d200 100644 --- a/server/jetty-app/README.md +++ b/server/jetty-app/README.md @@ -1,70 +1,29 @@ # server-jetty-app -## Native development - This document is oriented towards getting a server up for local development. -If you are interested in running a native production release, please see +If you are interested in running a native production release or to see other options for starting the server, please see [https://deephaven.io/core/docs/how-to-guides/configuration/native-application/](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/). -### Groovy Quickstart - -```shell -./gradlew server-jetty-app:run -Pgroovy -``` - -### Python Quickstart - -1. Setup virtual environment: - - ```shell - python -m venv /tmp/my-dh-venv - source /tmp/my-dh-venv/bin/activate - ``` - -1. Build and install wheel - - ```shell - ./gradlew py-server:assemble +If using Python, see [Python development guide](../../py/README.md) for setting up the Python environment. - # replace with the appropriate - pip install "py/server/build/wheel/deephaven_core--py3-none-any.whl[autocomplete]" +## Local Development - # To install without the optional `autocomplete` feature, run: - # pip install "py/server/build/wheel/deephaven_core--py3-none-any.whl" - ``` +`server-jetty-app:run` will incorporate local Java changes on each start. If you are not frequently changing Java code, see the next section. -1. Run - - ```shell - ./gradlew server-jetty-app:run - ``` - -**Note:** - -* This is not an exhaustive guide to managing python environments -* Depending on your OS and how your PATH is setup, you may need to use `python3`, or a path to the explicit python version you want to use -* You may choose to setup a "permanent" virtual environment location -* You'll need to re-install the wheel anytime you are making python code changes that affect the wheel -* `pip` can be a pain if you are trying to (re-)install a wheel with the same version number as before - * A `pip install --force-reinstall --no-deps "py/server/build/wheel/deephaven_core--py3-none-any.whl[autocomplete]"` may do the trick -* You can install other python packages in your venv using `pip install ` -* You can setup multiple virtual environments, and switch between them as necessary using `source /path/to/other-venv/bin/activate` -* You can de-activate the virtual environment by running `deactivate` -* You can use the `VIRTUAL_ENV` environment variable instead of sourcing / activating virtual environments: `VIRTUAL_ENV=/my/venv ./gradlew server-jetty-app:run` +```shell +./gradlew server-jetty-app:run # Python enigne by default +./gradlew server-jetty-app:run -Pgroovy # Groovy engine +``` -### Start script +## Development with Infrequent Changes -To create a more production-like environment, you can create and invoke the start script instead of running via gradle: +To create a more production-like environment, you can create and invoke the start script instead of running via gradle. This is faster if you need to often restart the server without making any changes to Java code (such as Python server development). ```shell -./gradlew server-jetty-app:installDist +./gradlew server-jetty-app:installDist # Run after any Java changes ./server/jetty-app/build/install/server-jetty/bin/start ``` - -See [https://deephaven.io/core/docs/how-to-guides/configuration/native-application/](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/) -for options when invoking the start script. - ### Configuration The `START_OPTS` environment variable is used to set JVM arguments. For example: @@ -88,6 +47,7 @@ By default, the server starts up on all interfaces with plaintext port 10000 (po expiration duration of 5 minutes, a scheduler pool size of 4, and a max inbound message size of 100 MiB. To bring up a SSL-enabled server on port 8443 with a development key and certificate, you can run: + ```shell ./gradlew server-jetty-app:run -Pgroovy -PdevCerts ```