[draft] add llamafile 🦙📁 #866
Conversation
not-lain
requested review from
osanseviero,
SBrandeis,
gary149,
Wauplin,
julien-c and
pcuenca
as code owners
There was a problem hiding this comment.
Thanks for the PR @not-lain, copying our message from slack here:
A good idea would be to add llamafile as a LocalApp instead of a library.
(that's also how llama.cpp/ llama-cpp-python) are added as well.
So what I'd recommend is that you open a PR to LocalApps: https://github.com/huggingface/huggingface.js/blob/1de39598a231afab805e252d2b27e0ec56a1897a/packages/tasks/src/local-apps.ts#L142
This way we can add support for both GGUFs + llamafiles on the same PR and people would be able to opt-in to the local app as well. Makes it more use-ful for the people.
Would you be keen on opening a PR for that?
| const command = (binary: string) => | ||
| [ | ||
| "# Load and run the model :", | ||
| `wget https://huggingface.co/${model.id}/resolve/main/`.concat(`${filepath ?? "{{GGUF_FILE}}"}`), // could not figure out how to do it without concat |
There was a problem hiding this comment.
| `wget https://huggingface.co/${model.id}/resolve/main/`.concat(`${filepath ?? "{{GGUF_FILE}}"}`), // could not figure out how to do it without concat | |
| `wget https://huggingface.co/${model.id}/resolve/main/${filepath ?? '{{GGUF_FILE}}'}`, |
Would something like that work? (not sure)
| "# Load and run the model :", | ||
| `wget https://huggingface.co/${model.id}/resolve/main/`.concat(`${filepath ?? "{{GGUF_FILE}}"}`), // could not figure out how to do it without concat | ||
| `chmod +x ${binary}`, | ||
| `${binary} -m ${filepath?? "{{GGUF_FILE}}"} -p 'You are a helpful assistant' `, // will this create a second dropdown ? |
There was a problem hiding this comment.
| `${binary} -m ${filepath?? "{{GGUF_FILE}}"} -p 'You are a helpful assistant' `, // will this create a second dropdown ? | |
| `${binary} -m ${filepath ?? "{{GGUF_FILE}}"} -p 'You are a helpful assistant'`, // will this create a second dropdown ? |
| docsUrl : "https://github.com/Mozilla-Ocho/llamafile", | ||
| mainTask : "text-generation", | ||
| displayOnModelPage : isLlamaCppGgufModel, // update this later to include .llamafile | ||
| snippet: snippetLlamafileGGUF , |
There was a problem hiding this comment.
| snippet: snippetLlamafileGGUF , | |
| snippet: snippetLlamafileGGUF, |
| ].join("\n"); | ||
| return [ | ||
| { | ||
| title: "Use pre-built binary", |
There was a problem hiding this comment.
I don't think this step is correct in llamafile, because the downloaded model file is already a pre-built binary.
The release binary from https://github.com/Mozilla-Ocho/llamafile/releases is used in Creating llamafiles section of the README. So this step should be named Create your own llamafile (or we could also remove this section, and just leave a link to README)
|
cc @pcuenca and @ngxson |
llamafile is a local application for running distributed LLM using a single file, developed by jart in collaboration with the mozilla team.
you can already filter by llamafile under hf.co/models under the
librariessection, so I thought this is a good time to add the code snippets for the library.the app handles
.gguffiles.llamafilefiles 🦙📁This pr tackles only
.gguffiles, leaving.llamafilefor another pr because the current huggingface.js does not handle{{LLAMAFILE_FILE}}yet (I think)I barely know any JS, so I might need some expert help with this PR.
an example of a code snippet as provided by the maintainer of the library could be used as such :
As for the
.llamafilesnippet I have reported it internally to our chief 🦙 officer @osanseviero in here (private dm)