Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add additional token calculation information to the Limit Azure OpenAI API token usage API Management policy #123618

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add additional token calculation information to the Limit Azure OpenAI API token usage API Management policy #123618

wants to merge 1 commit into from
+1 −1

Commits on Jul 3, 2024

  1. Add additional token calculation information to the Limit Azure OpenA… 

    …I API token usage API Management policy

How does the new APIM policy “Limit Azure OpenAI API token usage” `estimate-prompt-tokens` feature work? How does it estimate the tokens? Only of the prompt or also of the response? What changes, when I activate/deactivate it? The documentation was way too minimal on this.

This change adds some additional information about the impact of setting `estimate-prompt-tokens` to false.

This is important, because it might cause the probably unexpected behavior of sending requests to the LLM backend, which should have been filtered out by the policy.

Please check for correctness by the Product Team.
    @robinmanuelthiel
    robinmanuelthiel committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    4c9b755 View commit details
    Browse the repository at this point in the history