You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm new to using fastchat, I've been able to load my models using exllamav2 and I see it mentioned that xformers can be used for training models, but it was my understanding (forwarned - can be incorrect) that xformers could be used in loading/using the model with exllamav2 reducing (slightly?) the size and speed of the model. So few questions:
Can xformers be used in conjunction with exllamav2? Yes, it works in oobabooga and models seem to load faster/respond faster...
Can xformers or is xformers already used in fastchat when loading the model with exllamav2? Sorry in advance for asking here they don't seem to have a friendly discussion channel there... just issues
My desire to use fastchat stems from the MS blog for the local LLM example for autogen. Thanks in advance for any responses.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I'm new to using fastchat, I've been able to load my models using exllamav2 and I see it mentioned that xformers can be used for training models, but it was my understanding (forwarned - can be incorrect) that xformers could be used in loading/using the model with exllamav2 reducing (slightly?) the size and speed of the model. So few questions:
Can xformers be used in conjunction with exllamav2? Yes, it works in oobabooga and models seem to load faster/respond faster...
Can xformers or is xformers already used in fastchat when loading the model with exllamav2? Sorry in advance for asking here they don't seem to have a friendly discussion channel there... just issues
My desire to use fastchat stems from the MS blog for the local LLM example for autogen. Thanks in advance for any responses.
Beta Was this translation helpful? Give feedback.
All reactions