Unable to load llama model on GPU (Kubernetes + AWS) #1431

lili-wan · 2023-12-12T23:06:33Z

lili-wan
Dec 12, 2023

Hi everyone, could I get some help to load llama model on GPU? I am using Kubernetes to deploy local-ai and p2.x8large AWS instance for GPU. Here is the error message I got from the log when I try to load llama models (https://huggingface.co/TheBloke/Llama-2-7B-GGUF/blob/main/llama-2-7b.Q4_0.gguf)

From backend llama

10:26PM DBG [llama] Attempting to load
10:26PM DBG Loading model llama from llama-2-7b.Q4_0.gguf
10:26PM DBG Loading model in memory from file: /models/llama-2-7b.Q4_0.gguf
10:26PM DBG Loading Model llama-2-7b.Q4_0.gguf with gRPC (file: /models/llama-2-7b.Q4_0.gguf) (backend: llama): {backendString:llama model:llama-2-7b.Q4_0.gguf threads:1 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0002ce780 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh petals:/build/backend/python/petals/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
10:26PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/llama
10:26PM DBG GRPC Service for llama-2-7b.Q4_0.gguf will be running at: '127.0.0.1:37579'
10:26PM DBG GRPC Service state dir: /tmp/go-processmanager1384024355
10:26PM DBG GRPC Service Started
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:37579: connect: connection refused"
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr 2023/12/12 22:26:18 gRPC Server listening at 127.0.0.1:37579
10:26PM DBG GRPC Service Ready
10:26PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:llama-2-7b.Q4_0.gguf ContextSize:2048 Seed:0 NBatch:512 F16Memory:true MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:1 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/llama-2-7b.Q4_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0}
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr create_gpt_params_cuda: loading model /models/llama-2-7b.Q4_0.gguf
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr ggml_init_cublas: found 1 CUDA devices:
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr   Device 0: Tesla K80, compute capability 3.7
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr gguf_init_from_file: invalid magic number 4f44213c
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr error loading model: llama_model_loader: failed to load model from /models/llama-2-7b.Q4_0.gguf
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr llama_load_model_from_file: failed to load model
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr llama_init_from_gpt_params: error: failed to load model '/models/llama-2-7b.Q4_0.gguf'
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:37579): stderr load_binding_model: error: unable to load model
10:26PM DBG [llama] Fails: could not load model: rpc error: code = Unknown desc = failed loading model

From backend llama-cpp

10:26PM DBG [llama-cpp] Attempting to load
10:26PM DBG Loading model llama-cpp from llama-2-7b.Q4_0.gguf
10:26PM DBG Loading model in memory from file: /models/llama-2-7b.Q4_0.gguf
10:26PM DBG Loading Model llama-2-7b.Q4_0.gguf with gRPC (file: /models/llama-2-7b.Q4_0.gguf) (backend: llama-cpp): {backendString:llama-cpp model:llama-2-7b.Q4_0.gguf threads:1 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0002ce780 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh petals:/build/backend/python/petals/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
10:26PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/llama-cpp
10:26PM DBG GRPC Service for llama-2-7b.Q4_0.gguf will be running at: '127.0.0.1:40303'
10:26PM DBG GRPC Service state dir: /tmp/go-processmanager2160931682
10:26PM DBG GRPC Service Started
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:40303: connect: connection refused"
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stdout Server listening on 127.0.0.1:40303
10:26PM DBG GRPC Service Ready
10:26PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:llama-2-7b.Q4_0.gguf ContextSize:2048 Seed:0 NBatch:512 F16Memory:true MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:1 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/llama-2-7b.Q4_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0}
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr ggml_init_cublas: found 1 CUDA devices:
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr   Device 0: Tesla K80, compute capability 3.7
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr gguf_init_from_file: invalid magic characters <!DO.
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr error loading model: llama_model_loader: failed to load model from /models/llama-2-7b.Q4_0.gguf
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr llama_load_model_from_file: failed to load model
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stderr llama_init_from_gpt_params: error: failed to load model '/models/llama-2-7b.Q4_0.gguf'
10:26PM DBG GRPC(llama-2-7b.Q4_0.gguf-127.0.0.1:40303): stdout {"timestamp":1702419976,"level":"ERROR","function":"load_model","line":562,"message":"unable to load model","model":"/models/llama-2-7b.Q4_0.gguf"}
10:26PM DBG [llama-cpp] Fails: could not load model: rpc error: code = Canceled desc =

Here is my configurations:

replicaCount: 1

deployment:
  image: quay.io/go-skynet/local-ai:v2.0.0-cublas-cuda12-core
  env:
    threads: 1
    context_size: 2048
    debug: "true"
    f16: true # enable with GPU acceleration
    gpu_layers: 22 # GPU Layers (only used when built with cublas)
  modelsPath: "/models"
  download_model:
    # To use cloud provided (eg AWS) image, provide it like: 1234356789.dkr.ecr.us-REGION-X.amazonaws.com/busybox
    image: busybox:latest
  prompt_templates:
    # To use cloud provided (eg AWS) image, provide it like: 1234356789.dkr.ecr.us-REGION-X.amazonaws.com/busybox
    image: busybox:latest
  pullPolicy: IfNotPresent
  imagePullSecrets: []
    # - name: secret-names

resources:
  limits:
    memory: 40Gi
    nvidia.com/gpu: 1

# Prompt templates to include
# Note: the keys of this map will be the names of the prompt template files
promptTemplates:
  # {}
   ggml-gpt4all-j.tmpl: |
     ### Prompt:
     {{.Input}}

# Models to download at runtime
models:
  # Whether to force download models even if they already exist
  forceDownload: false

  # The list of URLs to download models from
  # Note: the name of the file will be the name of the loaded model
  list:
    - url: "https://huggingface.co/TheBloke/Llama-2-7B-GGUF/blob/main/llama-2-7b.Q4_0.gguf"

  # Persistent storage for models and prompt templates.
  # PVC and HostPath are mutually exclusive. If both are enabled,
  # PVC configuration takes precedence. If neither are enabled, ephemeral
  # storage is used.
  persistence:
    pvc:
      enabled: false
      size: 15Gi
      accessModes:
        - ReadWriteOnce

      annotations: {}

      # Optional
      storageClass: ~

    hostPath:
      enabled: false
      path: "/models"

service:
  type: ClusterIP
  # If deferring to an internal only load balancer
  # externalTrafficPolicy: Local
  port: 8080
  annotations: {}
  # If using an AWS load balancer, you'll need to override the default 60s load balancer idle timeout
  # service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "1200"

nodeSelector: {}

tolerations: []

affinity: {}

image:
  pullPolicy: IfNotPresent

Could you please help to recommend which llama model would work for cuda? Thanks and really appreciated