alta3/llm-the-alta3-way

Learn Generative AI - 4 Days Training

├── ansible      # common playbooks and roles
├── doc          # troubleshooting and other misc documentation
│   └── provider # cloud gpu provider details and fix-up scripts
├── model        # models with supporting fetch/clean/unsplit scripts
└── prompt       # prompts for testing/demonstration

Assumptions

• Ubuntu 20.04 or 22.04 target host
• Tested with Nvidia H100, A100, A40

Quickstart

1. Every cloud service PROVIDER requires specific configuration in order to complete this quickstart. As soon as you have connected to the provider's host, find your CLOUD PROVIDER on this list, read the README found there and run the tasks specified in that README. Skip this step at your peril.

2. Exit the provider's machine.

3. ssh back into the provider as ubuntu.

4. Setup ansible and clone this repo

   {
     git clone https://github.com/alta3/llm-the-alta3-way.git
     cd llm-the-alta3-way
     bash ansible/nvidia.sh
     python3 -m pip install --upgrade --user pip
     python3 -m pip install --user ansible
   }

5. System reboot is required because cuda-drivers are reinstalled.

   sudo systemctl reboot

6. Run nvcc --version and nvidia-smi to verify versions.

   nvcc --version
   nvidia-smi

7. Select a model and Run (see models section for specific playbooks)

   cd ~/llm-the-alta3-way/
   ansible-playbook model/{{ model }}/install.yml
   bash ~/llm/model/{{ model }}/test.sh

8. Want to try again? This directory structure is created to make that action really easy. rm the following directories to reset your machine:

   rm -r ~/llm
   rm -r ~/llm-the-alta3-way
   

Models

• Llama2 70B Orca 200k GGUF

  ansible-playbook model/Llama-2-70B-Orca-200k/install.yml

• Orca Mini v3 13B GGML

  ansible-playbook model/orca_mini_v3_13b/install.yml

• Falcon 40B Instruct:

  ansible-playbook model/falcon-40b-instruct/install.yml

• CodeLlama

llm directory

Deployed by this repo's base role, this directory structure is a non-git directory named llm for development and work with deployed models.

├── llm                # llm working directory
│   ├── bin            # installed binaries (e.g. hfdownloader)
│   ├── dataset        # <future use>
│   ├── git            # installed git repos
│   │   └── llama.cpp  # inference of LLaMA model in pure C/C++
│   ├── model          # deployed models
│   └── prompt         # prompts for testing/demonstration      
└── llm-the-alta3-way  # this repo checked out

Model Loaders Frameworks

• ggerganov/llama.cpp
• marella/ctransformers
• turboderp/exllama

Tools

• simonw/llm

  llm --help; llm --version

• simonw/ttok

  ttok --help; ttok --version

• aristocratos/bpytop

  bpytop

• wookayin/gpustat

  gpustat --interval 1 --show-all

• bodaay/HuggingFaceModelDownloader

  hfdownloader --help

• golang 1.20.5

  go help; go version; which go

• oobabooga/text-generation-webui

  cd ~/llm/git/webui
  source venv/bin/activate
  python3 server.py

Additional dependencies

• jllllll/llama-cpp-python-cuBLAS-wheels

Deprecated

• cmp-nct/ggllm.cpp - Falcon support added to llama.cpp