Skip to content

Latest commit

 

History

History
120 lines (80 loc) · 5.48 KB

developer.md

File metadata and controls

120 lines (80 loc) · 5.48 KB
Raw

Developer Guide

This doc explains how to setup a development environment so you can get started contributing.

Prerequisites

Follow the instructions below to set up your development environment.

Install requirements

You must install these tools:

  1. Python: The project needs Python 3.10+ for development.
  2. Ray: The project depends on Ray cluster.

Setup your environment

Create Python virtual environments.

$ python -m venv ./venv

Activate the virtual environment

$ source venv/bin/activate

Checkout your fork

To check out this repository:

  1. Create your own fork of this repo
  2. Clone it to your machine:
git clone https://github.com/${YOUR_GITHUB_USERNAME}/llm-inference.git
cd llm-inference
git remote add upstream https://github.com/OpenCSGs/llm-inference.git

Adding the upstream remote sets you up nicely for regularly syncing your fork.

Once you reach this point you are ready to do a full build and deploy as described below.

Deploy llm-inference

See the guide in Readme to deploy llm-inference.

Note that you need to start Ray cluster under llm-inference directory.

Add your model

LLMServe allows you to easily add new models by adding a single configuration file. To learn more about how to customize or add new models, see the LLMServe Model Registry.

Then run following command to deploy your model serving.

llm-serve start serving-rest --model=<your_model_yaml_path>

Quickly test after deployment

If changes related with CLI llm-serve, quickly testing with below:

python launch.py ...

For example:

python launch.py --help

 Usage: launch.py [OPTIONS] COMMAND [ARGS]...

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ list        List available model(s) and deployed serving etc.                                                                                                      │
│ predict     Predict one or several models with one or multiple prompts, optionally read from file, and save the results to a file.                                 │
│ start       Start application(s) for LLM serving, API server, experimention, fine-tuning and comparation.                                                          │
│ stop        Stop application(s) for LLM serving and API server.                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

If your changes is related with API Server, need to restart API to active that.

llm-serve stop apiserver
llm-serve start apiserver

If you would like to use the Restful API to manage your model serving, the API serve is needed.

llm-serve start apiserver

Running unit/integration tests

Before commit your changes, ensure the pylint test is passed.

$ pylint --rcfile=tests/pylint.conf $(git ls-files '*.py')

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 6.02/10, +3.98)

Contribute to the code

Welcome send a PR for your changes, or create a issue for question, problem, or feature requests.

Feedback

The best place to provide feedback about the code is via a Github issue.