add llm_deploy

README.md

@@ -4,5 +4,6 @@ We split our overall framework into multiple parts
 
 1. Data Processing --> Output general form of sotopia train and test data
 2. Together AI Finetuning --> Input the train and test data / Output model checkpoint
-3. LLM Finetuning --> Input the train and test data / Output model checkpoint + deployed model API form
-4. Eval --> Input model checkpoint / Output evaluation scores
+3. LLM Finetuning --> Input the train and test data / Output model checkpoint
+4. LLM Deplyment --> Input LLM Finetuned model checkpoint / Output Deployable OpenAI type API
+5. Eval --> Input model checkpoint / Output evaluation scores

llm_deploy/deploy.sh

@@ -0,0 +1 @@
+python -m vllm.entrypoints.openai.api_server --model ../llm_ft/vicuna-7b-1.5

llm_deploy/requirements.txt

@@ -0,0 +1 @@
+vllm

llm_deploy/vllm_test.py

@@ -0,0 +1,6 @@
+from vllm import LLM
+
+prompts = ["Hello, my name is", "The capital of France is"]  # Sample prompts.
+llm = LLM(model="../llm_ft/vicuna-7b-1.5")  # Create an LLM.
+outputs = llm.generate(prompts)  # Generate texts from the prompts.
+print(outputs)