Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SHARK user guide to root of docs directory (archived) #524

Closed
wants to merge 1 commit into from
Closed

Add SHARK user guide to root of docs directory (archived) #524

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 4 additions & 61 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,68 +61,11 @@ Model name | Model recipes | Serving apps
SDXL | [`sharktank/sharktank/models/punet/`](https://github.com/nod-ai/SHARK-Platform/tree/main/sharktank/sharktank/models/punet) | [`shortfin/python/shortfin_apps/sd/`](https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin/python/shortfin_apps/sd)
llama | [`sharktank/sharktank/models/llama/`](https://github.com/nod-ai/SHARK-Platform/tree/main/sharktank/sharktank/models/llama) | [`shortfin/python/shortfin_apps/llm/`](https://github.com/nod-ai/SHARK-Platform/tree/main/shortfin/python/shortfin_apps/llm)

## Development tips

Each sub-project has its own developer guide. If you would like to work across
projects, these instructions should help you get started:
## SHARK Users

### Setup a venv
If you're looking to use SHARK check out our [User Guide](docs/README.md).

We recommend setting up a Python
[virtual environment (venv)](https://docs.python.org/3/library/venv.html).
The project is configured to ignore `.venv` directories, and editors like
VSCode pick them up by default.
## SHARK Developers

```bash
python -m venv .venv
source .venv/bin/activate
```

### Install PyTorch for your system

If no explicit action is taken, the default PyTorch version will be installed.
This will give you a current CUDA-based version, which takes longer to download
and includes other dependencies that SHARK does not require. To install a
different variant, run one of these commands first:

* *CPU:*

```bash
pip install -r pytorch-cpu-requirements.txt
```

* *ROCM:*

```bash
pip install -r pytorch-rocm-requirements.txt
```

* *Other:* see instructions at <https://pytorch.org/get-started/locally/>.

### Install development packages

```bash
# Install editable local projects.
pip install -r requirements.txt -e sharktank/ shortfin/

# Optionally clone and install the latest editable iree-turbine dep in deps/,
# along with nightly versions of iree-base-compiler and iree-base-runtime.
pip install -f https://iree.dev/pip-release-links.html --upgrade --pre \
iree-base-compiler iree-base-runtime --src deps \
-e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine"
```

See also: [`docs/nightly_releases.md`](./docs/nightly_releases.md).

### Running tests

```bash
pytest sharktank
pytest shortfin
```

### Optional: pre-commits and developer settings

This project is set up to use the `pre-commit` tooling. To install it in
your local repo, run: `pre-commit install`. After this point, when making
commits locally, hooks will run. See https://pre-commit.com/
If you're looking to develop SHARK, check out our [Developer Guide](docs/developer.md).
118 changes: 118 additions & 0 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# SHARK User Guide
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd personally put this at /docs/user-guide.md, but I could also see a case for /docs/users/README.md or where you have it now at docs/README.md.

Organizing into docs/users/ and docs/developers/ gives us a nice way to direct users away from guides that are less actively maintained.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do an incremental step. I'll go with user-guide.md since that accomplishes the goal and is to understand. We can further restructure later as we get more developer-only and user-only content.


> [!WARNING]
> This is still pre-release so the artifacts listed here may be broken
>

These instructions cover the usage of the latest stable release of SHARK. For a more bleeding edge release please install the [nightly releases](nightly_releases.md).

## Prerequisites

Our current user guide requires that you have:
- Access to a computer with an installed AMD Instinct™ MI300x Series Accelerator
- Installed a compatible version of Linux and ROCm on the computer (see the [ROCm compatability matrix](https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html))


## Set up Environment

You will need a recent version of Python. We recommend also setting up a Python environment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Python environment" isn't very specific. Maybe suggest venv?

Suggested change
You will need a recent version of Python. We recommend also setting up a Python environment.
You will need a recent version of Python. We recommend also setting up a [Python virtual environment](https://docs.python.org/3/library/venv.html).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleaning up language


Officially we support Python versions: 3.11, 3.12, 3.13, 3.13t

The rest of this guide assumes you are using Python 3.11.

### Install Python
To install Python 3.11 on Ubuntu:

```bash
sudo apt install python3.11 python3.11-dev python3.11-venv

which python3.11
# /usr/bin/python3.11
```

### Create a Python Environment

This guide assumes you'll be using pyenv. Setup your pyenv with the following commands:

```bash
# Set up a virtual environment to isolate packages from other envs.
python3.11 -m venv 3.11.venv
source 3.11.venv/bin/activate
```
Comment on lines +36 to +42
Copy link
Member

@ScottTodd ScottTodd Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah. Python.

pyenv and Python venv are two different things. They can work together though.

https://github.com/pyenv/pyenv

If you use pyenv, you can get behavior like this:

python --version
# 3.10

pyenv install 3.11
pyenv global 3.11
python --version
# 3.11

Then when you create a venv:

python -m venv .venv
source .venv/bin/activate
python --version
# 3.11

Without venv pyenv (edit: ugh, now I'm confusing myself), you would have the code you have here:

python --version
# 3.10
python -m venv .venv
source .venv/bin/activate
python --version
# 3.10
deactivate

python3.11 -m venv 3.11.venv
source 3.11.venv/bin/activate
python --version
# 3.11

So pyenv lets you just call python as if it was python3.11, but venv is what actually isolates installed packages into a virtual environment


## Install SHARK and its dependencies

```bash
pip install transformers
pip install dataclasses-json
pip install pillow
pip install shark-ai
Comment on lines +47 to +50
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be simplified now

Suggested change
pip install transformers
pip install dataclasses-json
pip install pillow
pip install shark-ai
pip install shark-ai[apps]

```

Temporarily, you may need an update to your `shortfin` install.
Install the latest pre-release with:
```
pip install shortfin --upgrade --pre -f https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels
```

### Test the installation.

```
python -m shortfin_apps.sd.server --help
```

## Quickstart

### Run the SD Server

Run the [SD Server](../shortfin/python/shortfin_apps/sd/README.md#Start SD Server)

### Run the SD Client

```
python -m shortfin_apps.sd.simple_client --interactive
```

Congratulations!!! At this point you can play around with the server and client based on your usage.
Comment on lines +65 to +77
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put some context here (and/or in https://github.com/nod-ai/SHARK-Platform/blob/main/shortfin/python/shortfin_apps/sd/README.md) about what "SD" is.

The guide here should have enough information so users know what the server will do: text prompt(s) in, images out

Once we have official support for more models, we'll want to structure the user guide so there is a branch: general setup -> choose your model/app. Fine to start with the SDXL focus though.


### Update flags

Please see --help for both the server and client for usage instructions. Here's a quick snapshot.

#### Update server options:

| Flags | options |
|---|---|
|--host HOST |
|--port PORT | server port |
|--root-path ROOT_PATH |
|--timeout-keep-alive |
|--device | local-task,hip,amdgpu | amdgpu only supported in this release
|--target | gfx942,gfx1100 | gfx942 only supported in this release
|--device_ids |
|--tokenizers |
|--model_config |
| --workers_per_device |
| --fibers_per_device |
| --isolation | per_fiber, per_call, none |
| --show_progress |
| --trace_execution |
| --amdgpu_async_allocations |
| --splat |
| --build_preference | compile,precompiled |
| --compile_flags |
| --flagfile FLAGFILE |
| --artifacts_dir ARTIFACTS_DIR | Where to store cached artifacts from the Cloud |

#### Update client with different options:

| Flags |options|
|---|---
|--file |
|--reps |
|--save | Whether to save image generated by the server |
|--outputdir| output directory to store images generated by SDXL |
|--steps |
|--interactive |
|--port| port to interact with server |
65 changes: 65 additions & 0 deletions docs/developer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# SHARK Developer Guide

Each sub-project has its own developer guide. If you would like to work across
projects, these instructions should help you get started:

### Setup a venv

We recommend setting up a Python
[virtual environment (venv)](https://docs.python.org/3/library/venv.html).
The project is configured to ignore `.venv` directories, and editors like
VSCode pick them up by default.

```bash
python -m venv .venv
source .venv/bin/activate
```

### Install PyTorch for your system

If no explicit action is taken, the default PyTorch version will be installed.
This will give you a current CUDA-based version, which takes longer to download
and includes other dependencies that SHARK does not require. To install a
different variant, run one of these commands first:

* *CPU:*

```bash
pip install -r pytorch-cpu-requirements.txt
```

* *ROCM:*

```bash
pip install -r pytorch-rocm-requirements.txt
```

* *Other:* see instructions at <https://pytorch.org/get-started/locally/>.

### Install development packages

```bash
# Install editable local projects.
pip install -r requirements.txt -e sharktank/ shortfin/

# Optionally clone and install the latest editable iree-turbine dep in deps/,
# along with nightly versions of iree-base-compiler and iree-base-runtime.
pip install -f https://iree.dev/pip-release-links.html --upgrade --pre \
iree-base-compiler iree-base-runtime --src deps \
-e "git+https://github.com/iree-org/iree-turbine.git#egg=iree-turbine"
```

See also: [nightly_releases.md](nightly_releases.md).

### Running tests

```bash
pytest sharktank
pytest shortfin
```

### Optional: pre-commits and developer settings

This project is set up to use the `pre-commit` tooling. To install it in
your local repo, run: `pre-commit install`. After this point, when making
commits locally, hooks will run. See https://pre-commit.com/
26 changes: 6 additions & 20 deletions shortfin/python/shortfin_apps/sd/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,12 @@
This directory contains a SD inference server, CLI and support components.


## Quick start
## Install

In your shortfin environment,
```
pip install transformers
pip install dataclasses-json
pip install pillow
pip install shark-ai

```

Temporarily, you may need an update to your `shortfin` install.
Install the latest pre-release with:
```
pip install shortfin --upgrade --pre -f https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels
```
For [nightly releases](../../../../docs/nightly_releases.md)
For our [stable release](../../../../docs/README.md)

```
python -m shortfin_apps.sd.server --help
```

# Run on MI300x
## Start SD Servier
The server will prepare runtime artifacts for you.

By default, the port is set to 8000. If you would like to change this, use `--port` in each of the following commands.
Expand All @@ -39,6 +23,8 @@ python -m shortfin_apps.sd.server --device=amdgpu --device_ids=0 --build_prefere
INFO - Application startup complete.
INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```
## Run the SD Client

- Run a CLI client in a separate shell:
```
python -m shortfin_apps.sd.simple_client --interactive
Expand Down
Loading