i always get short outputs of < 1600 tokens, how to increase more?

[sprappcom]

python3 examples/chat.py -m ../llamamodels/WestLake-7B-v2-laser-truthy-dpo-4.0bpw-h6-exl2 -mode llama -l 8192 -fst -maxr 8192 -amnesia --low_mem

i've done this but still getting lesser than 1600 tokens. how to increase? anyone knows?

[bjj]

You can always ban out the eos token with settings.disallow_tokens(tokenizer, [tokenizer.eos_token_id]) and remove any stop strings (in the chat prompt config) and it will go until the context fills. Actually chat.py might keep going until it throws away the entire conversation prior to the generation. Or possibly forever, I'd have to go reread the code.

[kolinfluence]

@bjj i want it to write an encyclopedic article, but it always stops at around 1500 tokens, (the article generated is good) but i dont need more "nonsense". i need it more comprehensive. how can i get it done that way?

write 8000 words of... (it does not respect this command) it doesnt respect any word counter command type. does anyone have any ideas on how to get this fix properly for good? even chatgpt doesnt respect this

[bjj]

It's not going to do thing like plan out 8000 words to write. If you want to try to prompt engineer your way to it, try things it would know, like "Write a Wikipedia article on..."

[ouvaa]

@bjj can you write the prompt with the encyclopedia article that make sense with a 8000 words output? i'll use whatever model that can achieve this