Skip to content

nod-ai/shark-ai

Repository files navigation

shark-ai: SHARK Modeling and Serving Libraries

Important

Development is still in progress for several project components. See the notes below for which workflows are best supported.

GitHub License pre-commit

Sub-projects

PyPI version CI - shortfin

The shortfin sub-project is SHARK's high performance inference library and serving engine.

  • API documentation for shortfin is available on readthedocs.

PyPI version CI - sharktank

The SHARK Tank sub-project contains a collection of model recipes and conversion tools to produce inference-optimized programs.

Warning

SHARK Tank is still under development. Experienced users may want to try it out, but we currently recommend most users download pre-exported or pre-compiled model files for serving with shortfin.

CI - Tuner

The Tuner sub-project assists with tuning program performance by searching for optimal parameter configurations to use during model compilation.

Warning

SHARK Tuner is still in early development. Interested users may want to try it out, but the tuner is not ready for general use yet. Check out the readme for more details.

Support matrix

Models

Model name Model recipes Serving apps
SDXL sharktank/sharktank/models/punet/ shortfin/python/shortfin_apps/sd/
llama sharktank/sharktank/models/llama/ shortfin/python/shortfin_apps/llm/

SHARK Users

If you're looking to use SHARK check out our User Guide.

SHARK Developers

If you're looking to develop SHARK, check out our Developer Guide.