Fast Llama Inference with SambaNova

https://www.unremarkable.ai/exploring-accelerated-compound-ai-systems-with-sambanova-llama-3-3/

Setup

Run npm install and build.

npm install
npm run build

Make sure you have the following environment variables needed:

• SAMBANOVA_API_KEY - Your SambaNova API Key.

All demos & experiments leverage the following tools:

1. Use of Inquirer.js with the CLI to prompt for user questions.
2. The Vercel AI SDK is used to invoke and stream model output to the CLI.
3. Using the Meta-Llama-3.3-70B-Instruct model via hyper fast inference thanks to SambaNova.

npm run demo
npm run evalVerifyClaims

If you are testing other providers such as OpenAI or Bedrock you will need to make sure you have supporting API keys in your environment. For example OPENAI_API_KEY and standard AWS environment variables.

MODEL=bedrock npm run evalVerifyClaims

Notes

The bedrock-cross-region.md file contains the following claims which the demo verifies.

✅ Amazon Bedrock launched cross-region inference on August 27, 2024
   Summary: The claim is correct as Amazon Bedrock did launch cross-region inference on August 27, 2024, as stated in the provided sources.
✅ Cross-region inference provides up to 2x the allocated in-region quotas
   Summary: The claim is correct as cross-region inference provides up to 2x the allocated in-region quotas according to the sources.
✅ Cross-region inference support was extended to Knowledge Bases on September 13, 2024
   Summary: The claim is correct as cross-region inference support was indeed extended to Knowledge Bases on September 13, 2024, as stated in the provided sources.

But if you change the first claim in that file to August 25, 2024, you will see the following output.

❌ Amazon Bedrock launched cross-region inference on August 25, 2024
   Summary: The claim is incorrect because Amazon Bedrock launched cross-region inference on August 27, 2024, not August 25, 2024.
   Fixed Claim: Amazon Bedrock launched cross-region inference on August 27, 2024