Advantage of NEMO models & Toolkit

[karrtikiyer]

Can someone direct me (if it exists) any documentation around the advantages of converting HF models to .NEMO? I could not find out. Thanks in advance.
Especially I am trying to answer for myself:

• Is there some hardware specific optimisation done because of which say inference is faster?
• how does NEMO toolkit offerings compare to say Hugging Face library for model training?
• How does it compare to https://github.com/microsoft/DeepSpeed? or for that matter https://www.mosaicml.com/
• What makes training speeds to reduce upto 30%?
• What are some of the other alternatives to NeMo?

[hijkzzz]

I can answer part of the questions，

• Is there some hardware specific optimisation done because of which say inference is faster?Is there some hardware specific optimisation done because of which say inference is faster?
  NeMo is fast in training stage. Because it's based on Megatron and Transformer Engine。
  For inference, it's slow.

• How does NEMO toolkit offerings compare to say Hugging Face library for model training?
  It doesn't to be as easy to use as HuggingFace(HF).

• How does it compare to https://github.com/microsoft/DeepSpeed?
  DeepSpeed supports ZeRO PP
  NeMo supports 3D parallelism natively
  btw, DeepSpeed+HF is easy to use

• What are some of the other alternatives to NeMo?
  Megatron LM or Megatron-DeepSpeed

If you find any mistakes, please correct me

[karrtikiyer]

Thanks a lot @hijkzzz
For Inference, Nvidia has mentioned that their Triton Serving layer/sw speeds up the inference. Any details or thoughts around it? How does it compare to say vLLM

[joaocp98662]

Can someone point me in the direction of converting HuggingFace models to .NEMO? I've searched but i can't find any relevant information/documentation. If someone that have done it before or know how to do it could help i would appreciate. Thank you