Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FSTORE-1281] Add method to stop execution + type hinting login+get_feature_store #203

Merged
merged 4 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,24 @@
pip install -e ".[dev]"
```

- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The library uses pre-commit to ensure code-style and code formatting through [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8). Run the following commands from the `python` directory:
- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The library uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory:

```bash
cd python
pip install --user pre-commit
pre-commit install
```
```bash
cd python
pip install --user pre-commit
pre-commit install
```

Afterwards, pre-commit will run whenever you commit.

- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use black and flake8, or run them via the command line:
- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use `ruff`, or run it via the command line:

```bash
cd python
flake8 hopsworks
black hopsworks
```
```bash
# linting
ruff check python --fix
# formatting
ruff format python
```

### Python documentation

Expand Down
25 changes: 13 additions & 12 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,24 @@
pip install -e ".[dev]"
```

- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The library uses pre-commit to ensure code-style and code formatting through [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8). Run the following commands from the `python` directory:
- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The Feature Store uses pre-commit to ensure code-style and code formatting through [ruff](https://docs.astral.sh/ruff/). Run the following commands from the `python` directory:
vatj marked this conversation as resolved.
Show resolved Hide resolved

```bash
cd python
pip install --user pre-commit
pre-commit install
```
```bash
cd python
pip install --user pre-commit
pre-commit install
```

Afterwards, pre-commit will run whenever you commit.

- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use black and flake8, or run them via the command line:
- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use `ruff`, or run it via the command line:

```bash
cd python
flake8 hopsworks
black hopsworks
```
```bash
# linting
ruff check python --fix
# formatting
ruff format python
```

### Python documentation

Expand Down
15 changes: 9 additions & 6 deletions python/hopsworks/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,27 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
from __future__ import annotations

import warnings
import getpass
import logging
import os
import sys
import getpass
import tempfile
import warnings
from pathlib import Path

from hopsworks.client.exceptions import RestAPIError, ProjectException
from hopsworks import version, constants, client
from hopsworks import client, constants, project, version
from hopsworks.client.exceptions import ProjectException, RestAPIError
from hopsworks.connection import Connection


# Needs to run before import of hsml and hsfs
warnings.filterwarnings(action="ignore", category=UserWarning, module=r".*psycopg2")

import hsml # noqa: F401, E402
import hsfs # noqa: F401, E402
import hsml # noqa: F401, E402


__version__ = version.__version__

Expand Down Expand Up @@ -62,7 +65,7 @@ def login(
project: str = None,
api_key_value: str = None,
api_key_file: str = None,
):
) -> project.Project:
"""Connect to [Serverless Hopsworks](https://app.hopsworks.ai) by calling the `hopsworks.login()` function with no arguments.

```python
Expand Down
18 changes: 18 additions & 0 deletions python/hopsworks/core/execution_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,21 @@ def _delete(self, job_name, id):
id,
]
_client._send_request("DELETE", path_params)

def _stop(self, job_name: str, id: int) -> None:
_client = client.get_instance()
path_params = [
"project",
self._project_id,
"jobs",
job_name,
"executions",
id,
"status",
]
_client._send_request(
"PUT",
path_params=path_params,
data={"state": "stopped"},
headers={"Content-Type": "application/json"},
)
16 changes: 13 additions & 3 deletions python/hopsworks/execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@
# limitations under the License.
#

import humps
import json
from hopsworks.engine import execution_engine
from hopsworks.core import execution_api

import humps
from hopsworks import constants, util
from hopsworks.core import execution_api
from hopsworks.engine import execution_engine


class Execution:
Expand Down Expand Up @@ -212,6 +213,15 @@ def delete(self):
"""
self._execution_api._delete(self._job.name, self.id)

def stop(self):
"""Stop the execution
!!! danger "Potentially dangerous operation"
This operation stops the execution.
# Raises
`RestAPIError`.
"""
self._execution_api._stop(self.job_name, self.id)

def await_termination(self):
"""Wait until execution reaches terminal state
# Raises
Expand Down
16 changes: 9 additions & 7 deletions python/hopsworks/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,23 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
from __future__ import annotations

import humps
import json

from hopsworks import util, client, constants
import humps
from hopsworks import client, constants, util
from hopsworks.client.external import Client
from hopsworks.core import (
job_api,
git_api,
dataset_api,
kafka_api,
opensearch_api,
environment_api,
flink_cluster_api,
git_api,
job_api,
kafka_api,
opensearch_api,
)
from hsfs import feature_store


class Project:
Expand Down Expand Up @@ -101,7 +103,7 @@ def created(self):
"""Timestamp when the project was created"""
return self._created

def get_feature_store(self, name: str = None):
def get_feature_store(self, name: str = None) -> feature_store.FeatureStore:
robzor92 marked this conversation as resolved.
Show resolved Hide resolved
"""Connect to Project's Feature Store.

Defaulting to the project name of default feature store. To get a
Expand Down