NVIDIA NIM LLM Hosting Pattern

[hustshawn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

NVIDIA NIM pvovides an easy way to self-host LLMs with containers. AWS can provide series of solutions to help customer to deploy the NIM provided LLMs in a performant and cost optimized way with EKS, Karpenter, Spot etc.
Would like to have a pattern for the customer to easily deploy the NIM LLMs in AWS.

Describe the solution you would like

Provide a IaC pattern in Data on EKS to showcase deploy NVIDIA NIM provided LLMs. Will use Karpenter, Spot, and Bottlerocket with image caching techniques which I introduced in this AWS blog.

Describe alternatives you have considered

Additional context

NVIDIA NIM: https://www.nvidia.com/en-us/ai/
Features and architecture: https://docs.nvidia.com/nim/large-language-models/latest/introduction.html

[vara-bonthu]

This will be a great addition.

Step1: Leverage Existing Terraform Blueprint:
Utilize the existing Terraform blueprint to set up infrastructure components like EKS Cluster, Karpenter, and other addons. Update this template if required specific to Bottlerocket implementation etc.

Step2: Add a New Folder:
Create a new folder under gen-ai named nvidia-nims.
Place all the necessary code for your blueprint in this folder.

Step3: Create Documentation:
Add a new documentation file under the Website docs with the naming convention nvidia-nim-<modelname>.

[hustshawn]

Thank you @vara-bonthu . I am gonna implement it, please feel free to assign this to me. I raise a PR once it is finished.