Bedrock, output of Llama 3.1 405B model just stops

[dirkpetersen]

What happened?

When I ask the Llama 3.1 405B model on Bedrock to return a page, it will just stop after a while without completing. This does not happen with the Llama 3.1 70B model or any other models. It also does not happen when I query bedrock on the CLI or via Python. My current streamRate is 75 but i slowed it down to as much as 250 with no effect

Steps to Reproduce

• Select Bedrock
• Select meta.llama3-1-405b-instruct-v1:0
• Enter Prompt, e.g. "Give me a full page on little known facts about Mark Twain"

What browsers are you seeing the problem on?

Firefox

Relevant log output

No log output

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

The same is observed on Amazon Bedrock Chat playground:

It seems that response length is lower for 405b than the other llama models

70b:

I will have to add an inference parameter for llama models for this

[dirkpetersen]

ah that makes sense ..... for my testing with Python I had already globally increased that length