Add vllm deploy (#45)

* support qlora

* upload dummy conversation data

* delete doc and docker

* update pyproject pip install package

* continue cleaning

* delete more files

* delete a format

* add llm_deploy

* add testing scripts

* update deployment readme

* update readme and fix some bug

* finalize the inference and deployment based on vllm

README.md

@@ -4,5 +4,6 @@ We split our overall framework into multiple parts
 
 1. Data Processing --> Output general form of sotopia train and test data
 2. Together AI Finetuning --> Input the train and test data / Output model checkpoint
-3. LLM Finetuning --> Input the train and test data / Output model checkpoint + deployed model API form
-4. Eval --> Input model checkpoint / Output evaluation scores
+3. LLM Finetuning --> Input the train and test data / Output model checkpoint
+4. LLM Deplyment --> Input LLM Finetuned model checkpoint / Output Deployable OpenAI type API
+5. Eval --> Input model checkpoint / Output evaluation scores

llm_deploy/README.md

@@ -0,0 +1,5 @@
+We need to use an unmerged branch to support deploying lora-finetuned model. (the forked repo is https://github.com/troph-team/vllm.git)
+
+Go to the vllm dir and pip install -e .
+
+To notice https://github.com/vllm-project/vllm/issues/1283, need to modify the config file to "== 2.0.1" and the pytorch version if facing with CUDA version error.

llm_deploy/requirements.txt

@@ -0,0 +1 @@
+vllm

llm_deploy/vllm_deploy.sh

@@ -0,0 +1 @@
+python -m vllm.entrypoints.openai.api_server --model ../llm_ft/vicuna-7b-1.5

llm_deploy/vllm_inference_test.py

@@ -0,0 +1,23 @@
+from vllm import LLM, SamplingParams
+from vllm.model_executor.adapters import lora
+
+# Create an LLM, need to change gpu memory utilization based on our need
+llm = LLM(model="../llm_ft/vicuna-7b-1.5", gpu_memory_utilization=0.5)
+
+# Add LoRA adapter
+lora.LoRAModel.from_pretrained(llm.llm_engine.workers[0].model, "../llm_ft/vicuna_checkpoints/checkpoint-1200")
+
+prompts = [
+    "Hello, my name is",
+    "The capital of France is",
+    "The future of AI is",
+]
+
+sampling_params = SamplingParams(temperature=0, top_k=-1)
+
+outputs = llm.generate(prompts, sampling_params)
+
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")