Jetson (Xavier series) compatible LLM based assistant and robot control package
Explore the docs »
Report Bug
.
Request Feature
- Jetson Xavier NX Development Kit
- Seed studio Respeaker Microphone Array
- Generic USB stereo speaker
This project came into fruition through an excited bit of research into the increasing ability to deploy LLMs on edge devices. It owes its inspiration and initial structure to a number of initiatives - most notably ollama reference above but also:
A number of changes have been made or are in progress that diverge from these original projects:
-
pyttsx3 has been swapped out for Piper TTS - while nsss and Sapi5 have great voice support for mac and Windows respectively, espeak leaves much to be desired for Linux. Piper tts is not only fast but has outsanding pronounciation and quality in general - you'll find all dependencies and a sample voice is installed by this project
-
I'm using gemma 2b rather than llama here with great results on the board. Both latency and mobile battery power
-
Finally you'll notice that support is slowly coming to a halt for Xavier devices with Orin in the picture. This project is both an attenpt to keep the series alive as they're still incredibly powerful - quantised ollama models run with impressive low latency - but also an alternative approach to a dockerised environment for those who are both new to docker and docker as a tool to solve Jetson compatibility issues.
Given the last point the project assumes that you've made it as far as Jetpack 5.1.4 - although Jetpack 6 support would be great once I can get my hands on an Orin device!
-
We're assuming here that you've have a board ready to go - check out the hardware link above for the Xavier NX I'm using
-
If you're not set up yet you'll need to to install jetpack 5.1.4 with an SD card or a NVMe SSD by folling the official guide here
-
If you're headed down the NVMe road you can also use the the following bug fix branch from the fantastic Jetson Hacks in order to expand the partition size of your root on installation
-
All installation scripts are self contained and debugged for this specific setup but note that you'll need to ensure you're in a Python 3.9 virtual environment rather than the stock Python 3.8 - this is due to dependency conflicts at Piper-TTS. All dependencies at install.sh include any relevant 3.9 wheels as a result
-
Download the pre-built torch 2.1.0 wheel for Python 3.9 at the following Google Drive link and add to your project root
-
Then simply run
sh install.sh
- You'll notice a number of customisable options at the installation script
- The whisper model downloaded to the
whispersubfolder is the stockbase.pt - The onnx voice available via the script is
en_GB-alba-medium.onnxthis is also easily changed by dropping a new voice intovoices
- The whisper model downloaded to the
- assistant.yaml also points to a number of project specific parrameters:
- Notice that my ollama model is mini-b named after a robot I'm bulding - check out ollama's customize prompt section for more on this
- Coming soon
- coming soon
- Jetpack 5.1.4 - Xavier NX Development Kit - passing
- Likely appicable to an AGX Xavier too - I have one available - testing results shortly
coming soon
coming soon
I went down a pretty lengthy rabbit hole for this one and for those of you who don't wish to stick exclusively to Python there's a numbe of great projects out there - these have not been tested with Jetson boards however:
- June 3rd 2023 - Swapping out pygame for a terminal based space bar execution with sshkeyboard
- Incoming - Whisper tensorrt for better performance
- Wake word based activation
- command execution
- automatic image live camera retrieval and analysis
- optimised RAG
- pip package support
- Project link: https://github.com/rhysdg/ollama-voice-jetson
- Email: Rhys