How can I specify temperature for model generation?

[aalyousfi]

Hi,

I'm wondering how I can specify generation parameters like temperature for Transformsers or AWQ models?

I've searched the documentation and the repo but couldn't find anything about that.

[rlouf]

Currently not, but we could introduce it after #366. Is that a critical feature for you?

[aalyousfi]

@rlouf I think so, I want to set temperature to 0 to make the model more adherent to instructions. I'm wondering what's the default temperature?

After I posted my question, I was going through the repo code. I found here that you can pass model_kwargs to AWQ models. So if I pass something like temperature=0.2, do_sample=True, does that work? Or it has no effect currently too?

[rlouf]

This has to be passed to the sampler, so I doubt that model_kwargs would change this. This change is planned and tracked by #335