HuggingFace models in NeMo

[joaocp98662]

I would like to use several Transformer models provided by HuggingFace (e.g. tiiuae/falcon-7b) for inference using NeMo, but I cannot understand how to do it. I found a method get_huggingface_lm_model() from the nemo.collections.nlp.modules.common.huggingface.huggingface_utils, that imports the HF's model using the AutoModel class from Transformers. If the model is a BERT it works fine, but the Falcon LLM is detected as a RW model and fails to match any of the model types defined in the HUGGINGFACE_MODELS (BertModel, DistilBertModel, CamembertModel, RobertaModel, AlbertModel, GPT2Model) in the huggingface_utils.py, and raises an exception saying "Use HuggingFace API directly in NeMo".

How can i convert HF models to NeMo for inference and trainning?

I've searched on the NeMo documentation, and through the internet, but I'm not finding any relevant information nor examples. Can someone help me or point me in the right direction please?

Thank you very much!

[blahBlahhhJ]

We're working to add more community models to Nemo. Currently, falcon is not supported because of its special architecture. We do support llama2 recently.

[blahBlahhhJ]

in the config, enable megatron_amp_O2 for whatever example you are running (I assume you're doing megatron_gpt_eval)

[joaocp98662]

@blahBlahhhJ thank you very much, it solved the problem. If you could give same help in one more issue I am encountering now please, I would appreciate it very much.

Now, I am trying to generate some responses.

from pytorch_lightning.trainer.trainer import Trainer
from nemo.collections.nlp.models.language_modeling.megatron_gpt_model import MegatronGPTModel

trainer = Trainer(accelerator='gpu', devices=1)
# load Megatron GPT model from the .nemo file from which model should be instantiated
infer_model = MegatronGPTModel.restore_from(restore_path=args.restore_path, trainer=trainer)

length_params = {"max_length": 100, "min_length": 50}
result = infer_model.generate(inputs=["Hey, are you conscious? Can you talk to me?"], length_params=length_params)

It breaks with the following error: RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Note: I am using a single GPU.

Traceback (most recent call last):
  File "/home/ubuntu/hoopie/model_inference_nemo.py", line 58, in <module>
    result = infer_model.generate(inputs=["Hey, are you conscious? Can you talk to me?"], length_params=length_params)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py", line 1137, in generate
    return megatron_gpt_generate(self.cuda(), inputs, self.tokenizer, length_params, sampling_params)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/text_generation_utils.py", line 116, in megatron_gpt_generate
    output = generate(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/text_generation_utils.py", line 512, in generate
    if torch.distributed.get_rank() == get_model_parallel_src_rank():
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1158, in get_rank
    default_pg = _get_default_group()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 697, in _get_default_group
    raise RuntimeError(
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

In the text_generation_utils.py, it breaks in the generate method at torch.distributed.get_rank().

Thank you again

[blahBlahhhJ]

Not sure about the details but you probably need to add this argument to the trainer strategy=NLPDDPStrategy().

Basically copying what the example script is doing https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_eval.py

[joaocp98662]

Thanks you so much @blahBlahhhJ, again it solve the problem.

[Prannesh45]

Hi @joaocp98662, would like to know how you resolved the error what you mentioned earlier. I witnessed the argument is already available in the trainer.