From 0efb86a08681af4ca90e43b80af455d3ad068a48 Mon Sep 17 00:00:00 2001 From: Yu Fan Date: Sat, 17 Aug 2024 01:15:40 +0800 Subject: [PATCH] add: README.md for cloud-edge joint inference example Signed-off-by: Yu Fan --- .../README.md | 100 ++++++++++++++---- .../requirements.txt | 4 + requirements.txt | 5 +- 3 files changed, 89 insertions(+), 20 deletions(-) create mode 100644 examples/cloud-edge-collaborative-inference-for-llm/requirements.txt diff --git a/examples/cloud-edge-collaborative-inference-for-llm/README.md b/examples/cloud-edge-collaborative-inference-for-llm/README.md index 934bd79a..f19ed674 100644 --- a/examples/cloud-edge-collaborative-inference-for-llm/README.md +++ b/examples/cloud-edge-collaborative-inference-for-llm/README.md @@ -1,20 +1,65 @@ -# README +# Quick Start -## Simple QA +Before using this example, you need to have the device ready: -### Prepare Data +One machine is all you need, i.e., a laptop or a virtual machine is sufficient and a cluster is not necessary -The data of simple-qa example structure is: +- 2 CPUs or more + +- 1 GPU with at least 4GB of memory, depends on the tested model + +- 4GB+ free memory, depends on algorithm and simulation setting + +- 10GB+ free disk space + +- Internet connection for GitHub and pip, etc + +- Python 3.8+ environment + +## Step 1. Ianvs Preparation +```bash +# Create a new conda environment with Python>=3.8 (venv users can do it in their own way). +conda create -n ianvs-experiment python=3.8 + +# Activate our environment +conda activate ianvs-experiment + +# Clone Ianvs Repo +git clone https://github.com/kubeedge/ianvs.git +cd ianvs + +# Install a modified sedna wheel (a small bug and dependencies was fixed) +wget https://github.com/FuryMartin/sedna/releases/download/v0.4.1.1/sedna-0.4.1.1-py3-none-any.whl +pip install sedna-0.4.1.1-py3-none-any.whl + +# Install dependencies for this example. +pip install examples/cloud-edge-collaborative-inference-for-llm/requirements.txt + +# Install dependencies for Ianvs Core. +pip install requirements.txt + +# Install ianvs +python setup.py install +``` + +## Step 2. Dataset and Model Preparation + +### Dataset Configuration + +You need to create a dataset folder in`ianvs/` in the following structure. ``` . -├── test_data -│ └── data.jsonl -└── train_data - └── data.jsonl +├── dataset + ├── test_data + │ └── data.jsonl + └── train_data + └── data.jsonl ``` -`train_data/data.jsonl` is empty, and the `test_data/data.jsonl` is as follows: +Leave `train_data/data.jsonl` as empty. + +Fill the `test_data/data.jsonl` as follows: ``` {"question": "如果小明有5个苹果，他给了小华3个，那么小明还剩下多少个苹果？\nA. 2个\nB. 3个\nC. 4个\nD. 5个", "answer": "A"} @@ -29,26 +74,43 @@ The data of simple-qa example structure is: {"question": "下列哪个图形的周长最长？\nA. 正方形\nB. 长方形\nC. 圆形\nD. 三角形", "answer": "C"} ``` -### Prepare Environment +Then, check the path of `train_data` and `test_dat` in +`examples/cloud-edge-collaborative-inference-for-llm/testenv/testenv.yaml`. + +- If you created the `dataset` folder inside `ianvs/` as mentioned earlier, then the relative path is correct and does not need to be modified. + +- If your `dataset` is created in a different location, please use an absolute path, and using `~` to represent the home directory is not supported. -You need to install the changed-sedna package, which added `JsonlDataParse` in `sedna.datasources` +### Model Configuration -Replace the file in `yourpath/anaconda3/envs/ianvs/lib/python3.x/site-packages/sedna` with `examples/resources/sedna-with-jsonl.zip` +The models are configured in `examples/cloud-edge-collaborative-inference-for-llm/testalgorithms/query-routing/test_queryrouting.yaml`. +If you just want to run this example quickly, you can skip the configuration. -### Run Ianvs +The default model is `Qwen/Qwen2-1.5B-Instruct`, which is configured by `model_name`. We use transformers to load the model, which is completely consistent with most LLMs you can see. If you don't have this model, HuggingFace will download it automatically. + +This example support two LLM serving framework: HuggingFace and vllm. You can choose one and change the value of `backend`. + +Besides, this example support different quatization methods. You can pass "full", "4-bit", "8-bit" to the value of `quantization` + + +## Step 3. Run Ianvs Run the following command: `ianvs -f examples/llm/singletask_learning_bench/simple_qa/benchmarkingjob.yaml` -## OpenCompass Evaluation - -### Prepare Environment +After several seconds, you will see the following output: -`pip install examples/resources/opencompass-0.2.5-py3-none-any.whl` +```bash ++------+---------------+-----+-----------+----------------+-----------+------------+---------------------+--------------------------+-------------------+------------------------+---------------------+---------------------+--------------------------------------------------------------------------------+ +| rank | algorithm | acc | edge-rate | paradigm | basemodel | apimodel | hard_example_mining | basemodel-model_name | basemodel-backend | basemodel-quantization | apimodel-model_name | time | url | ++------+---------------+-----+-----------+----------------+-----------+------------+---------------------+--------------------------+-------------------+------------------------+---------------------+---------------------+--------------------------------------------------------------------------------+ +| 1 | query-routing | 1.0 | 0.4 | jointinference | EdgeModel | CloudModel | BERT | Qwen/Qwen2-1.5B-Instruct | huggingface | full | gpt-4o-mini | 2024-08-17 01:02:45 | ./workspace/benchmarkingjob/query-routing/493d14ea-5bf1-11ef-bf9b-755996a48c84 | ++------+---------------+-----+-----------+----------------+-----------+------------+---------------------+--------------------------+-------------------+------------------------+---------------------+---------------------+--------------------------------------------------------------------------------+ +``` -### Run Evaluation +Ianvs will output a `rank.csv` and `selected_rank.csv` in `ianvs/workspace`, which will record the test results of each test. -`python run_op.py examples/llm/singletask_learning_bench/simple_qa/testalgorithms/gen/op_eval.py` +You can modify the relevant model parameters in `examples/cloud-edge-collaborative-inference-for-llm/testalgorithms/query-routing/test_queryrouting.yaml`, conduct multiple tests, and compare the results of different configurations. diff --git a/examples/cloud-edge-collaborative-inference-for-llm/requirements.txt b/examples/cloud-edge-collaborative-inference-for-llm/requirements.txt new file mode 100644 index 00000000..57f1fe78 --- /dev/null +++ b/examples/cloud-edge-collaborative-inference-for-llm/requirements.txt @@ -0,0 +1,4 @@ +vllm +transformers +openai +accelerate \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 9f7cec56..2e8d4ff6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,4 @@ -prettytable~=2.5.0 # BSD \ No newline at end of file +prettytable~=2.5.0 # BSD +scikit-learn +numpy +pandas \ No newline at end of file