Skip to content

Latest commit

 

History

History
44 lines (32 loc) · 1.78 KB

README.md

File metadata and controls

44 lines (32 loc) · 1.78 KB
Raw

JetsonLab

BLADE for Nvidia Jetson devices.

Devices

  • Jetson Orin AGX
  • Jetson Orin Nano

Structure

├── scripts/
│   ├── check_model_existence.sh  # Script that checks that all models exist locally
│   ├── jetson-docker.sh          # Launch dustynv's docker images for MLC execution
│   ├── llamacpp_models.txt       # llama.cpp models to run
│   ├── mlc_models.txt            # MLC models to run
│   ├── run_llamacpp.sh           # Script for automated llama.cpp execution.
│   └── run_mlc.sh                # Script for automated MLC execution.
└── src/
    ├── edit_configs.py           # Args conversion based on model config. It is invoked by scripts/run_{llamacpp,mlc}.sh
    ├── report_performance.py     # Output parsing. It is invoked by parse_jetson_runs.sh

How to run experiments

  1. Checkout the MELT repo and its on jetson device.
git clone git@github.com/brave-experiments/melt --recursive
  1. For MLC, launch dustynv's docker container from ./scripts/jetson-docker.sh. For llama.cpp, skip this step (or pass the respective library paths).
  2. Build on device, based on instructions from frameworks/*/*/build_scripts/
  3. Edit the models you want to execute in scripts/{llamacpp,mlc}_models.txt.
  4. Execute by running the following commands:
REPETITIONS=3 CONVERSATION_FROM=0 CONVERSATION_TO=5 MEASURE_ENERGY=1 DEVICE="orin_xx" ./run_mlc.sh
REPETITIONS=3 CONVERSATION_FROM=0 CONVERSATION_TO=5 MEASURE_ENERGY=1 DEVICE="orin_xx" ./run_llamacp.sh
  1. After the experiments are done, you can run the notebook notebooks/parse_jetson_runs.ipynb to plot the results.