diff --git a/proto/README.md b/proto/README.md new file mode 100644 index 00000000000..abbcf927348 --- /dev/null +++ b/proto/README.md @@ -0,0 +1,18 @@ +# Deephaven Protobuf + +Deephaven's [protobuf](https://protobuf.dev/) files are what drives its client APIs such as [Python](https://deephaven.io/core/client-api/python/), [C++](https://deephaven.io/core/client-api/cpp-examples/getting_started.html), and more using [gRPC](https://grpc.io/). If you wish to build your own client API or change how all client APIs interact with the Deephaven server, this is the place to start. + +For Java, proto changes will be automatically reflected when building/running the Java server app. Sections below list files that must be manually generated and committed to source control when proto files change. + +## Python/Go/C++ API + +```sh +./gradlew :py-client:updateProtobuf :go:updateProtobuf :cpp-client:updateProtobuf +``` + +## Authorization + +See [authorization readme](../authorization/README.md) + +## JS API + diff --git a/py/README.md b/py/README.md new file mode 100644 index 00000000000..5e4ff2ef9d0 --- /dev/null +++ b/py/README.md @@ -0,0 +1,79 @@ +# Deephaven Python Source + +This folder contains Deephaven Community Core's Python source code, tests, and more. Source code can be found in `server/deephaven` and unit tests can be found in `server/tests`. Other content includes `jpy`, the Python client, wheels, and more. + +## Requirements + +Development and contribution requires Python 3.8+. We recommend always staying up-to-date on the latest Python version. + +## Virtual environments + +Deephaven Python development should always be done from a [virtual environment](https://docs.python.org/3/library/venv.html). See [PEP 405](https://peps.python.org/pep-0405/) for the motivation behind virtual environments (venvs). Creating and activating a venv takes two shell commands. + +From the root directory of your `deephaven-core` clone: + +```sh +python3 -m venv .venv # Create a venv. Only needed once. +source .venv/bin/activate # Activate the virtual environment. +``` + +A virtual environment only exists in the terminal it's run from. New or other terminal windows or tabs will need to activate the virtual environment again. + +Once finished with a virtual environment, exiting is done with a single command. + +```sh +deactivate # Exit the venv when finished +``` + +## Local Development + +Using a `pip` editable install is recommended so that Python changes are reflected when your server is restarted. These instructions are for the Python server package in `py/server` but should apply to any other Python packages. + +Install the Python server package. This will install the dependencies as well as the autocomplete dependencies. + +```sh +DEEPHAVEN_VERSION=0.dev pip install -e "py/server[autocomplete]" +``` + +After installing, a `pip list` should include `deephaven-core` with an editable project location listed. + +Now you can start the Java server by following the instructions [here](../server/jetty-app/README.md). Any time you make Python changes, restart the server to apply them. + +## Local Build + +You can use these instructions if you do not plan on making frequent changes to your local Python environment or need to try things in a more production-like environment locally. + +First, run the following command to build the server and embedded-server if needed. The embedded-server is used to start [pip-installed Deephaven](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/#python-embedded-server). + +```sh +./gradlew :py-server:assemble + +# If using embedded-server +# You can run both tasks w/ ./gradlew :task1 :task2 if you need both +./gradlew :py-embedded-server:assemble +``` + +Then run the following commands to install the server package. Note that the `wheel` folder should only contain 1 file, so you should be able to tab-complete the wheel file. + +**Note:** Adding the `--no-deps` flag on subsequent installs will skip re-installing other dependencies. + +```sh +pip install --force py/server/build/wheel/deephaven_core--py3-none-any.whl + +# If using embedded-server +# You can combine as pip install core.whl server.whl if you need both +pip install --force py/embedded-server/build/wheel/deephaven_server--py3-none-any.whl +``` + +**If you make Python server code changes, you will need to re-build and re-install the wheel.** + +Start the Java server by following the instructions [here](../server/jetty-app/README.md). + +## Tests + +Python unit tests must be run from the root directory of the cloned repository. + +```sh +cd .. +./gradlew integrations:test-py-deephaven +``` \ No newline at end of file diff --git a/server/jetty-app/README.md b/server/jetty-app/README.md index c148c094d78..590ede89d15 100644 --- a/server/jetty-app/README.md +++ b/server/jetty-app/README.md @@ -1,76 +1,40 @@ # server-jetty-app -## Native development +This README is oriented towards getting a server up for local development in a development environment. From a development environment, users can modify source code towards contributing to the project, creating custom capabilities, and more. -This document is oriented towards getting a server up for local development. -If you are interested in running a native production release, please see -[https://deephaven.io/core/docs/how-to-guides/configuration/native-application/](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/). +If you wish to use Deephaven from a production environment, which is simpler but source code cannot be modified, see one of the following documents: -### Groovy Quickstart +- [Quickstart for Docker](https://deephaven.io/core/docs/tutorials/quickstart/) +- [How to configure the Deephaven native application](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/) -```shell -./gradlew server-jetty-app:run -Pgroovy -``` - -### Python Quickstart - -1. Setup virtual environment: - - ```shell - python -m venv /tmp/my-dh-venv - source /tmp/my-dh-venv/bin/activate - ``` - -1. Build and install wheel - - ```shell - ./gradlew py-server:assemble - - # replace with the appropriate - pip install "py/server/build/wheel/deephaven_core--py3-none-any.whl[autocomplete]" - - # To install without the optional `autocomplete` feature, run: - # pip install "py/server/build/wheel/deephaven_core--py3-none-any.whl" - ``` - -1. Run +This README deals with general development for either the Python or Groovy server-side API. For Python-specific development instructions, see the [Python development README](../../py/README.md). - ```shell - ./gradlew server-jetty-app:run - ``` +## Local development -**Note:** +`./gradlew server-jetty-app:run` will incorporate local Java changes on each start. If you are not frequently changing Java code, see the next section. -* This is not an exhaustive guide to managing python environments -* Depending on your OS and how your PATH is setup, you may need to use `python3`, or a path to the explicit python version you want to use -* You may choose to setup a "permanent" virtual environment location -* You'll need to re-install the wheel anytime you are making python code changes that affect the wheel -* `pip` can be a pain if you are trying to (re-)install a wheel with the same version number as before - * A `pip install --force-reinstall --no-deps "py/server/build/wheel/deephaven_core--py3-none-any.whl[autocomplete]"` may do the trick -* You can install other python packages in your venv using `pip install ` -* You can setup multiple virtual environments, and switch between them as necessary using `source /path/to/other-venv/bin/activate` -* You can de-activate the virtual environment by running `deactivate` -* You can use the `VIRTUAL_ENV` environment variable instead of sourcing / activating virtual environments: `VIRTUAL_ENV=/my/venv ./gradlew server-jetty-app:run` +```shell +./gradlew server-jetty-app:run # Python session (default) +./gradlew server-jetty-app:run -Pdebug # Attach a Java debugger to the Python session on port 5005 +./gradlew server-jetty-app:run -Pgroovy # Groovy session +./gradlew server-jetty-app:run -Pgroovy -Pdebug # Attach a Java debugger to the Groovy session on port 5005 +``` -### Start script +## Development with infrequent changes -To create a more production-like environment, you can create and invoke the start script instead of running via gradle: +To create a more production-like environment, you can create and invoke the start script instead of running via gradle. This is faster if you need to often restart the server without making any changes to Java code (such as Python server development). ```shell -./gradlew server-jetty-app:installDist +./gradlew server-jetty-app:installDist # Run after any Java changes ./server/jetty-app/build/install/server-jetty/bin/start ``` - -See [https://deephaven.io/core/docs/how-to-guides/configuration/native-application/](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/) -for options when invoking the start script. - ### Configuration The `START_OPTS` environment variable is used to set JVM arguments. For example: ```shell -START_OPTS="-Xmx12g" ./gradlew server-jetty-app:run +START_OPTS="-Xmx12g" ./gradlew server-jetty-app:run # Starts Deephaven with 12gb of heap memory ``` While configuration properties can be inherited via JVM system properties (`-Dmy.property=my.value`), you may prefer to @@ -82,12 +46,21 @@ See [config-dir](https://deephaven.io/core/docs/how-to-guides/configuration/nati See [config-file](https://deephaven.io/core/docs/how-to-guides/configuration/config-file/) for more information on the configuration file format. +### Shutdown + +There are multiple ways to shut down the Deephaven server. The easiest is to `ctrl+C` the process. If it's being run in background mode, you can kill it with a `SIGINT`. + +```sh +kill -2 +``` + ### SSL By default, the server starts up on all interfaces with plaintext port 10000 (port 443 when SSL is enabled), a token expiration duration of 5 minutes, a scheduler pool size of 4, and a max inbound message size of 100 MiB. To bring up a SSL-enabled server on port 8443 with a development key and certificate, you can run: + ```shell ./gradlew server-jetty-app:run -Pgroovy -PdevCerts ``` @@ -155,4 +128,4 @@ outbound.ssl.trust.path=outbound-ca.crt ``` In all of the above cases, the outbound Deephaven-to-Deephaven connections can be configured separately from the -server's inbound configuration. +server's inbound configuration. \ No newline at end of file