New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

djl lmi images with vllm and hf quantizaton support #1188

Open

Nagarajj opened this issue Oct 16, 2023 · 0 comments

Labels

Nagarajj commented Oct 16, 2023 •

edited

Loading

Description

Trying to use the below sample examples notebooks from dj-demo results in error as 0.24.0 image of djl-deepspeed isnt available? .

https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/sample-llm/vllm_deploy_llama_13b.ipynb

https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/sample-llm/rollingbatch_deploy_llama2-13b-gptq.ipynb

@lanking520 and others whats the plan to release 0.24.0 images ? Or is there a way to get the latest images with the above support. Thanks

Expected Behavior

(what's the expected behavior?)
Ideal would be djl-deepspeed image which supports vllm and quantization schemes.

Error Message

image 0.24.0 isnt available

How to Reproduce?

https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/sample-llm/vllm_deploy_llama_13b.ipynb

https://github.com/deepjavalibrary/djl-demo/blob/master/aws/sagemaker/large-model-inference/sample-llm/rollingbatch_deploy_llama2-13b-gptq.ipynb

Steps to reproduce

(Paste the commands you ran that produced the error.)

What have you tried to solve it?

Nagarajj added the bug label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment