Plugins with litellm/ollama/mistral?

[shuther]

Contact Details

No response

What happened?

Maybe it is expected and only works with openAI? I could have missed it from the documentation.
Either the output in the chat window seems to be the call to the function:

{"func": "searchAbstracts", "intent": "Find the best place to rent an apartment in France based on scientific research."}

Or I do see:

User Instruction: Please use the 'google' tool to retrieve internet search results relevant to finding the best place to rent an apartment in France. The user is looking for information on the best location for renting an apartment in France.

Steps to Reproduce

1. Setup litellm, ollama, mistral with:
   litellm --model ollama/mistral --add_function_to_prompt --telemetry=false --debug --drop_params
2. Try with different plugings (I tried google and ScholarAI)

What browsers are you seeing the problem on?

No response

Relevant log output

LibreChat         | [chain/error] [1:chain:AgentExecutor] [3.88s] Chain run errored with error: "AbortError"
LibreChat         | Error: AbortError
LibreChat         |     at EventTarget.<anonymous> (/app/api/node_modules/langchain/dist/chains/base.cjs:109:36)
LibreChat         |     at [nodejs.internal.kHybridDispatch] (node:internal/event_target:761:20)
LibreChat         |     at EventTarget.dispatchEvent (node:internal/event_target:703:26)
LibreChat         |     at abortSignal (node:internal/abort_controller:314:10)
LibreChat         |     at AbortController.abort (node:internal/abort_controller:332:5)
LibreChat         |     at abortController.abortCompletion (/app/api/server/middleware/abortMiddleware.js:48:21)
LibreChat         |     at abortMessage (/app/api/server/middleware/abortMiddleware.js:15:37)
LibreChat         |     at /app/api/server/middleware/abortMiddleware.js:27:20
LibreChat         |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
LibreChat         | [handleResponseMessage] Output: {
LibreChat         |   output: 'Encountered an error while attempting to respond. Error: AbortError',
LibreChat         |   errorMessage: 'AbortError',
LibreChat         |   intermediateSteps: []
LibreChat         | }
LibreChat         | [llm/error] [2:llm:ChatOpenAI] [3.88s] LLM run errored with error: "Request was aborted."
LibreChat         | handleLLMError: {"context":"plugins","conversationId":"efc7605e-3e14-40f5-abd4-ecd7af549f7f","initialMessageCount":6}
LibreChat         | Error [AbortError]: Request was aborted.
LibreChat         |     at wrapOpenAIClientError (/app/api/node_modules/langchain/dist/util/openai.cjs:13:17)
LibreChat         |     at /app/api/node_modules/langchain/dist/chat_models/openai.cjs:621:69
LibreChat         |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
LibreChat         |     at async RetryOperation._fn (/app/node_modules/p-retry/index.js:50:12) {
LibreChat         |   attemptNumber: 1,
LibreChat         |   retriesLeft: 6
LibreChat         | }
LibreChat         | [llm/error] [1:llm:ChatOpenAI] [3.87s] LLM run errored with error: "Request was aborted."
LibreChat         | Error in handler Handler, handleLLMError: Error: AbortError: Request was aborted.
chat-meilisearch  | [2023-12-07T13:33:27Z INFO  actix_web::middleware::logger] 172.26.0.5 "GET /indexes/convos/documents/efc7605e-3e14-40f5-abd4-ecd7af549f7f HTTP/1.1" 200 168 "-" "undici" 0.000154
LibreChat         | CLIENT RESPONSE
LibreChat         | {
LibreChat         |   messageId: '37dc646f-9607-427e-86a3-195b8df5eb5b',
LibreChat         |   conversationId: 'efc7605e-3e14-40f5-abd4-ecd7af549f7f',
LibreChat         |   parentMessageId: 'c9b451f6-e952-413b-b4cc-b7add554bdd6',
LibreChat         |   isCreatedByUser: false,
LibreChat         |   isEdited: undefined,
LibreChat         |   model: 'ollama/mistral',
LibreChat         |   sender: 'OpenAI',
LibreChat         |   promptTokens: 133,
LibreChat         |   error: true,
LibreChat         |   text: 'Encountered an error while attempting to respond. Error: AbortError',
LibreChat         |   intermediateSteps: []
LibreChat         | }

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Maybe it is expected and only works with openAI?

Yes. This is not optimized for non-OpenAI and relies heavily on OpenAI functions. In the future, this will be different and working on the groundwork to make this all possible, but for now, compatibility cannot be expected.

[ogencoglu]

Does librechat support liteLLM in general?

[danny-avila]

yes it does!

https://docs.librechat.ai/install/configuration/litellm.html

[whalygood]

Hi Danny,

Are you able to clarify whether plugins such as Google or Azure AI Search can now be used with other models than OpenAI and Azure OpenAI? It looks like we are running into LangChain errors like the following when we use an Anthropic model such as Claude 3 Sonnet exposed through a LiteLLM proxy container.

Also, the responses don't appear to take into account retrieved data beyond the knowledge cut off of the model in March 2023.

error: [OpenAIClient] There was an issue generating the title with the completion method Error: 400 {"error":"Invalid model name passed in"}

[danny-avila]

Are you able to clarify whether plugins such as Google or Azure AI Search can now be used with other models than OpenAI and Azure OpenAI

Yes I can clarify. You cannot use other models because the LangChain handling is not generic and using methods specific for OpenAI. Only OpenAI-compatible API in every way including function calling could work.

This way of using tools is dated and a complete overhaul that can allow any endpoint to be used will eventually come to the project.

[whalygood]

Appreciate the speedy response. So just to make sure my understanding is correct, a non-OpenAI model exposed through an OpenAI compatible API using LiteLLM will not work with plugins. But if I were instead to implement using the Azure OpenAI Service, and set the PLUGINS_USE_AZURE="true" env variable, it should work, correct?

[danny-avila]

Azure does work, however doing that way is considered a legacy configuration.

You can try it first but here is the updated way (with librechat.yaml config file): https://docs.librechat.ai/install/configuration/azure_openai.html#using-plugins-with-azure