Enhance pull speed for Large ML container Images with Bottlerocket

[ratnopamc]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

Currently the inference container images run into multiple GBs in size. This negatively impacts the start up time for Ray Pods. We should look for ways to reduce the startup time of the Ray head and worker pods.

Describe the solution you would like

Validate if pre-fetching of images can be used leveraging EKS bottle rocket data volume in the inference blueprints to achieve this.

Describe alternatives you have considered

Additional context

https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/

[lindarr915]

I would like to work on this

[vara-bonthu]

@lindarr915 Please let us know if you need any guidance on the repo. This will be a great addition. You can also write a blog doc under the resources section to discuss the performance improvements with Bottlerocket.

[hustshawn]

Hi @vara-bonthu Darren works with me in a same team. One thing we have discussed, do you want to make this solution as a parallel pattern as others? or make it kind of a "shared pattern" (eg. a TF module or a separate stack), as some of the existing patterns may all need it.
What do you think?