From 3986c4fe297d00c22f78d071ee7a5062d3cc22c1 Mon Sep 17 00:00:00 2001 From: Yao Qing Date: Thu, 30 May 2024 09:06:26 +0800 Subject: [PATCH] Llama3 and phi3 validation result update. (#79) * Llama3 and Phi3 validation results update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- comps/llms/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/comps/llms/README.md b/comps/llms/README.md index 95c5fac2f..96a69ebf0 100644 --- a/comps/llms/README.md +++ b/comps/llms/README.md @@ -115,3 +115,14 @@ curl http://${your_ip}:9000/v1/chat/completions \ -d '{"query":"What is Deep Learning?","max_new_tokens":17,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"streaming":true}' \ -H 'Content-Type: application/json' ``` + +## Validated Model + +| Model | TGI-Gaudi | vLLM-CPU | Ray | +| ------------------------- | --------- | -------- | --- | +| Intel/neural-chat-7b-v3-3 | ✓ | ✓ | ✓ | +| Llama-2-7b-chat-hf | ✓ | ✓ | ✓ | +| Llama-2-70b-chat-hf | ✓ | - | x | +| Meta-Llama-3-8B-Instruct | ✓ | ✓ | ✓ | +| Meta-Llama-3-70B-Instruct | ✓ | - | x | +| Phi-3 | x | Limit 4K | ✓ |