Creating a langflow with Litellm with authentification

[flefevre]

Dear all
I would like to create a langflow ith Litellm as LLM model provider.
I do not know how to pass the authetification Bearer token?
Where should i do it?
Thanks
François

[dosubot[bot]]

Bonjour @flefevre,

Je suis là pour vous aider avec votre problème. N'hésitez pas à poser des questions ou à signaler tout autre problème.

Dosu

To pass the authentication Bearer token when creating a Langflow with Litellm as the LLM model provider, you should set the appropriate API key in the configuration. You need to specify the api_key field in the configuration for the ChatLiteLLMModelComponent.

Here is how you can set it:

from langflow.base.models.model import LCModelComponent

class ChatLiteLLMModelComponent(LCModelComponent):
    # Other configurations...
    def build(
        self,
        input_value: Text,
        model: str,
        provider: str,
        api_key: Optional[str] = None,
        stream: bool = False,
        temperature: Optional[float] = 0.7,
        model_kwargs: Optional[Dict[str, Any]] = {},
        top_p: Optional[float] = None,
        top_k: Optional[int] = None,
        n: int = 1,
        max_tokens: int = 256,
        max_retries: int = 6,
        verbose: bool = False,
        system_message: Optional[str] = None,
    ) -> Text:
        try:
            import litellm  # type: ignore

            litellm.drop_params = True
            litellm.set_verbose = verbose
        except ImportError:
            raise ChatLiteLLMException(
                "Could not import litellm python package. " "Please install it with `pip install litellm`"
            )
        provider_map = {
            "OpenAI": "openai_api_key",
            "Azure": "azure_api_key",
            "Anthropic": "anthropic_api_key",
            "Replicate": "replicate_api_key",
            "Cohere": "cohere_api_key",
            "OpenRouter": "openrouter_api_key",
        }
        # Set the API key based on the provider
        api_keys: dict[str, Optional[str]] = {v: None for v in provider_map.values()}

        if variable_name := provider_map.get(provider):
            api_keys[variable_name] = api_key
        else:
            raise ChatLiteLLMException(
                f"Provider {provider} is not supported. Supported providers are: {', '.join(provider_map.keys())}"
            )
        output = ChatLiteLLM(
            model=model,
            client=None,
            streaming=stream,
            temperature=temperature,
            model_kwargs=model_kwargs if model_kwargs is not None else {},
            top_p=top_p,
            top_k=top_k,
            n=n,
            max_tokens=max_tokens,
            max_retries=max_retries,
            openai_api_key=api_keys["openai_api_key"],
            azure_api_key=api_keys["azure_api_key"],
            anthropic_api_key=api_keys["anthropic_api_key"],
            replicate_api_key=api_keys["replicate_api_key"],
            cohere_api_key=api_keys["cohere_api_key"],
            openrouter_api_key=api_keys["openrouter_api_key"],
        )
        return self.get_chat_result(output, stream, input_value, system_message)

Replace "your_bearer_token_here" with your actual Bearer token and set the api_key field accordingly. This configuration will be picked up by Langflow when it initializes the LLM model provider ^[1][2].

To continue talking to Dosu, mention @dosu.

[flefevre]

Dear all, I have tried to create a workflow in order to connect to my Litellm but i do not find any LiteLLM where i can provide the base url. What should I use? I did'nt find any example in the store. Thanks

…

On Tue, Jun 11, 2024 at 10:25 AM dosubot[bot] ***@***.***> wrote: Bonjour @flefevre <https://github.com/flefevre>, Je suis là pour vous aider avec votre problème. N'hésitez pas à poser des questions ou à signaler tout autre problème. Dosu <https://dosu.dev> To pass the authentication Bearer token when creating a Langflow with Litellm as the LLM model provider, you should set the appropriate API key in the configuration. You need to specify the api_key field in the configuration for the ChatLiteLLMModelComponent. Here is how you can set it: from langflow.base.models.model import LCModelComponent class ChatLiteLLMModelComponent(LCModelComponent): # Other configurations... def build( self, input_value: Text, model: str, provider: str, api_key: Optional[str] = None, stream: bool = False, temperature: Optional[float] = 0.7, model_kwargs: Optional[Dict[str, Any]] = {}, top_p: Optional[float] = None, top_k: Optional[int] = None, n: int = 1, max_tokens: int = 256, max_retries: int = 6, verbose: bool = False, system_message: Optional[str] = None, ) -> Text: try: import litellm # type: ignore litellm.drop_params = True litellm.set_verbose = verbose except ImportError: raise ChatLiteLLMException( "Could not import litellm python package. " "Please install it with `pip install litellm`" ) provider_map = { "OpenAI": "openai_api_key", "Azure": "azure_api_key", "Anthropic": "anthropic_api_key", "Replicate": "replicate_api_key", "Cohere": "cohere_api_key", "OpenRouter": "openrouter_api_key", } # Set the API key based on the provider api_keys: dict[str, Optional[str]] = {v: None for v in provider_map.values()} if variable_name := provider_map.get(provider): api_keys[variable_name] = api_key else: raise ChatLiteLLMException( f"Provider {provider} is not supported. Supported providers are: {', '.join(provider_map.keys())}" ) output = ChatLiteLLM( model=model, client=None, streaming=stream, temperature=temperature, model_kwargs=model_kwargs if model_kwargs is not None else {}, top_p=top_p, top_k=top_k, n=n, max_tokens=max_tokens, max_retries=max_retries, openai_api_key=api_keys["openai_api_key"], azure_api_key=api_keys["azure_api_key"], anthropic_api_key=api_keys["anthropic_api_key"], replicate_api_key=api_keys["replicate_api_key"], cohere_api_key=api_keys["cohere_api_key"], openrouter_api_key=api_keys["openrouter_api_key"], ) return self.get_chat_result(output, stream, input_value, system_message) Replace "your_bearer_token_here" with your actual Bearer token and set the api_key field accordingly. This configuration will be picked up by Langflow when it initializes the LLM model provider [1] <https://github.com/logspace-ai/langflow/blob/dev/src/backend/base/langflow/components/models/ChatLiteLLMModel.py> [2] <https://github.com/langflow-ai/langflow/blob/dev/src/backend/base/langflow/components/models/ChatLiteLLMModel.py> . *To continue talking to Dosu <https://dosu.dev>, mention @dosu <https://github.com/dosu>.* — Reply to this email directly, view it on GitHub <#2137 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZRFACS2DRA2SRXCOJYQDZG2YALAVCNFSM6AAAAABJDWXOZOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMZWGA3DK> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- ----------------------------------------- François Le Fèvre 36 rue Jean Poulmarch 91190 Gif-sur-Yvette 0665604928

[dosubot[bot]]

To provide the base URL for Litellm in your workflow, you should use the OllamaLLM class. You can set the base_url parameter when building the configuration for the OllamaLLM component. If the base_url is not specified, it defaults to 'http://localhost:11434'.

Here is an example of how you can provide the base URL:

ollama_llm = OllamaLLM()
config = ollama_llm.build_config()
config['base_url']['value'] = 'http://your-custom-url:port'

llm_instance = ollama_llm.build(
    base_url=config['base_url']['value'],
    model=config['model']['value'],
    temperature=config['temperature']['value'],
    mirostat=config['mirostat']['value'],
    mirostat_eta=config.get('mirostat_eta', {}).get('value'),
    mirostat_tau=config.get('mirostat_tau', {}).get('value'),
    num_ctx=config.get('num_ctx', {}).get('value'),
    num_gpu=config.get('num_gpu', {}).get('value'),
    num_thread=config.get('num_thread', {}).get('value'),
    repeat_last_n=config.get('repeat_last_n', {}).get('value'),
    repeat_penalty=config.get('repeat_penalty', {}).get('value'),
    stop=config.get('stop', {}).get('value'),
    tfs_z=config.get('tfs_z', {}).get('value'),
    top_k=config.get('top_k', {}).get('value'),
    top_p=config.get('top_p', {}).get('value')
)

Replace 'http://your-custom-url:port' with the actual base URL you want to use. The OllamaLLM class provides a method build_config to set up the configuration, and the build method to create an instance of the language model with the specified parameters ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[flefevre]

Thanks for the tips.
But I would like to do it through the GUI directly in Langflow application.
As you can see in the image, there is no parameter to enter the base url in Litellm widget.

Does anyone know how to use Litellm rather Ollama?

Thanks

[dosubot[bot]]

To set the base URL for LiteLLM directly through the Langflow GUI, you need to add a custom field for the base URL in the LiteLLM configuration. Currently, the LiteLLM configuration does not include a base URL parameter by default. However, you can extend the configuration to include this parameter.

Here is how you can modify the ChatLiteLLMModelComponent class to include a base URL parameter:

1. Modify the build_config Method:
   Add the base_url field to the build_config method in the ChatLiteLLMModelComponent class.

def build_config(self):
    return {
        "model": {
            "display_name": "Model name",
            "field_type": "str",
            "advanced": False,
            "required": True,
            "info": "The name of the model to use. For example, `gpt-3.5-turbo`.",
        },
        "api_key": {
            "display_name": "API key",
            "field_type": "str",
            "advanced": False,
            "required": False,
            "password": True,
        },
        "provider": {
            "display_name": "Provider",
            "info": "The provider of the API key.",
            "options": [
                "OpenAI",
                "Azure",
                "Anthropic",
                "Replicate",
                "Cohere",
                "OpenRouter",
            ],
        },
        "temperature": {
            "display_name": "Temperature",
            "field_type": "float",
            "advanced": False,
            "required": False,
            "default": 0.7,
        },
        "model_kwargs": {
            "display_name": "Model kwargs",
            "field_type": "dict",
            "advanced": True,
            "required": False,
            "default": {},
        },
        "top_p": {
            "display_name": "Top p",
            "field_type": "float",
            "advanced": True,
            "required": False,
        },
        "top_k": {
            "display_name": "Top k",
            "field_type": "int",
            "advanced": True,
            "required": False,
        },
        "n": {
            "display_name": "N",
            "field_type": "int",
            "advanced": True,
            "required": False,
            "info": "Number of chat completions to generate for each prompt. "
            "Note that the API may not return the full n completions if duplicates are generated.",
            "default": 1,
        },
        "max_tokens": {
            "display_name": "Max tokens",
            "field_type": "int",
            "advanced": False,
            "required": False,
            "default": 256,
            "info": "The maximum number of tokens to generate for each chat completion.",
        },
        "max_retries": {
            "display_name": "Max retries",
            "field_type": "int",
            "advanced": True,
            "required": False,
            "default": 6,
        },
        "verbose": {
            "display_name": "Verbose",
            "field_type": "bool",
            "advanced": True,
            "required": False,
            "default": False,
        },
        "base_url": {
            "display_name": "Base URL",
            "field_type": "str",
            "advanced": False,
            "required": True,
            "info": "The base URL for the API.",
        },
    }

2. Update the build Method:
   Ensure the build method uses the base_url parameter.

def build(
    self,
    input_value: Text,
    model: str,
    provider: str,
    api_key: Optional[str] = None,
    stream: bool = False,
    temperature: Optional[float] = 0.7,
    model_kwargs: Optional[Dict[str, Any]] = {},
    top_p: Optional[float] = None,
    top_k: Optional[int] = None,
    n: int = 1,
    max_tokens: int = 256,
    max_retries: int = 6,
    verbose: bool = False,
    system_message: Optional[str] = None,
    base_url: str = "http://localhost:11434"
) -> Text:
    try:
        import litellm  # type: ignore

        litellm.drop_params = True
        litellm.set_verbose = verbose
    except ImportError:
        raise ChatLiteLLMException(
            "Could not import litellm python package. " "Please install it with `pip install litellm`"
        )
    provider_map = {
        "OpenAI": "openai_api_key",
        "Azure": "azure_api_key",
        "Anthropic": "anthropic_api_key",
        "Replicate": "replicate_api_key",
        "Cohere": "cohere_api_key",
        "OpenRouter": "openrouter_api_key",
    }
    # Set the API key based on the provider
    api_keys: dict[str, Optional[str]] = {v: None for v in provider_map.values()}

    if variable_name := provider_map.get(provider):
        api_keys[variable_name] = api_key
    else:
        raise ChatLiteLLMException(
            f"Provider {provider} is not supported. Supported providers are: {', '.join(provider_map.keys())}"
        )
    output = ChatLiteLLM(
        model=model,
        client=None,
        streaming=stream,
        temperature=temperature,
        model_kwargs=model_kwargs if model_kwargs is not None else {},
        top_p=top_p,
        top_k=top_k,
        n=n,
        max_tokens=max_tokens,
        max_retries=max_retries,
        openai_api_key=api_keys["openai_api_key"],
        azure_api_key=api_keys["azure_api_key"],
        anthropic_api_key=api_keys["anthropic_api_key"],
        replicate_api_key=api_keys["replicate_api_key"],
        cohere_api_key=api_keys["cohere_api_key"],
        openrouter_api_key=api_keys["openrouter_api_key"],
        base_url=base_url
    )
    return self.get_chat_result(output, stream, input_value, system_message)

By adding the base_url field to the build_config method and ensuring it is used in the build method, you can configure the base URL directly through the Langflow GUI for the LiteLLM model provider ^[1][2].

To continue talking to Dosu, mention @dosu.

[galaplexus]

Any updates on this?

The documentation on https://docs.langflow.org/components/models indicates that LiteLLM should be a standard model compoenent. But on Langchain version 1.0.5 the component is nowhere to be found.
Thanks for the help

[luizhsalazar]

Following this too. Looking for a way to interact with LiteLLM in Langflow.

[flefevre]

hello, i have create a new ticket here: #4741