[Question]: How do I add a custom endpoint?

[jerowe]

What is your question?

Hello,

Thanks for the great project! I'm trying to add a custom endpoint from a python flask application. I want to integrate AWS's bedrock and RAG workflows into a custom app.

I'm able to add the flask application like so:

# docker-compose.yml
services:
  flask-api:
    restart: always
    ports:
      - 3000:3000
    build:
      context: ../kb-bedrock-flask
    volumes:
      - ../kb-bedrock-flask:/root/app
  api:
    container_name: LibreChat

The flask app:

# app.py
from flask import Flask

import boto3
import pprint
from botocore.client import Config
from langchain.llms.bedrock import Bedrock
from langchain.retrievers.bedrock import AmazonKnowledgeBasesRetriever
from langchain.prompts import PromptTemplate
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
from langchain_community.chat_models import BedrockChat
from langchain_core.messages import HumanMessage
from langchain.chains import RetrievalQA
import json
import os
import logging
from rich.logging import RichHandler
from flask import stream_with_context, request
import json

FORMAT = "%(message)s"
logging.basicConfig(
    level="INFO", format=FORMAT, datefmt="[%X]", handlers=[RichHandler()]
)
app = Flask(__name__)


@app.route("/v1/chat/completions", methods=["GET", "POST"])
def v1_chat_completions():
    data = request.data.decode("utf-8")
    logging.info(data)

    def generate():
        r_data = {
            "id": "chatcmpl-948NlyY8WiorHEGSuWbJJEF1HECHD",
            "object": "chat.completion",
            "created": 1710772561,
            "model": "gpt-3.5-turbo-0125",
            "choices": [
                {
                    "index": 0,
                    "message": {
                        "role": "assistant",
                        "content": "Hello! How can I assist you today?"
                    },
                    "logprobs": None,
                    "finish_reason": "stop"
                }
            ],
            "usage": {
                "prompt_tokens": 19,
                "completion_tokens": 9,
                "total_tokens": 28
            },
            "system_fingerprint": "fp_4f2ebda25a"
        }
        yield json.dumps(r_data)
        # return r_data

    # return generate()
    return app.response_class(stream_with_context(generate()))


if __name__ == "__main__":
    app.run("0.0.0.0", port=3000, debug=True)

More Details

Here are some logs:

chat-meilisearch  | [2024-03-18T14:46:10Z INFO  index_scheduler::batch] document addition done: DocumentAdditionResult { indexed_documents: 1, number_of_documents: 23 }
flask-api_1       | [14:46:10] INFO     {                                                  app.py:31
flask-api_1       |                       "model": "kb-1",                                          
flask-api_1       |                       "temperature": 1,                                         
flask-api_1       |                       "top_p": 1,                                               
flask-api_1       |                       "presence_penalty": 0,                                    
flask-api_1       |                       "frequency_penalty": 0,                                   
flask-api_1       |                       "stop": [                                                 
flask-api_1       |                         "||>",                                                  
flask-api_1       |                         "\nUser:",                                              
flask-api_1       |                         "<|diff_marker|>"                                       
flask-api_1       |                       ],                                                        
flask-api_1       |                       "user": "65f7fe1d5ec4e7470770b272",                       
flask-api_1       |                       "stream": true,                                           
flask-api_1       |                       "messages": [                                             
flask-api_1       |                         {                                                       
flask-api_1       |                           "role": "user",                                       
flask-api_1       |                           "content": "HELLO"                                    
flask-api_1       |                         }                                                       
flask-api_1       |                       ],                                                        
flask-api_1       |                       "safe_prompt": true                                       
flask-api_1       |                     }                                                           
flask-api_1       |            INFO     172.18.0.5 - - [18/Mar/2024 14:46:10] "POST  _internal.py:96
flask-api_1       |                     /v1/chat/completions HTTP/1.1" 200 -                        
LibreChat         | 2024-03-18 14:46:10 warn: [OpenAIClient.chatCompletion][stream] Unhandled error type
LibreChat         | 2024-03-18 14:46:10 warn: [OpenAIClient.chatCompletion][finalChatCompletion] Unhandled error type
chat-meilisearch  | [2024-03-18T14:46:10Z INFO  actix_web::middleware::logger] 172.18.0.5 "POST /indexes/convos/documents HTTP/1.1" 202 133 "-" "node" 0.044223
LibreChat         | 2024-03-18 14:46:10 error: [OpenAIClient.chatCompletion] Unhandled error type Error: request ended without sending any chunks
LibreChat         | 2024-03-18 14:46:10 error: [handleAbortError] AI response error; aborting request: Error: request ended without sending any chunks

What is the main subject of your question?

Documentation, Endpoints

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Thanks for checking out the project!

Your custom endpoint has to be OpenAI compatible as much as possible. This is no small feat but there are a few projects on github that attempt to do this.

As for RAG, I hope to add my own implementation today, although it won't use bedrock.

[danny-avila]

You can test with your endpoint with the openai SDK from node or python to test compatibility.

[jerowe]

I tried this. I used the openai model to generate the response like so:

from openai import OpenAI
client = OpenAI()

completion = client.chat.completions.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)

And then returned it as a flask response. You are right though and it must be something to do with the way the server is sending events, because I get an error about not returning any chunks.

I was hoping to avoid stepping through code with the debugger, but if that's what I have to do we'll do it. ;-)

[jerowe]

Your custom endpoint has to be OpenAI compatible as much as possible. This is no small feat but there are a few projects on github that attempt to do this.

Could you point me towards these projects? It doesn't really matter to me how I format the response, I just want them to come up in the UI.

[danny-avila]

It involves not only formatting but transmitting server sent events, unless you are calling a non-chat endpoint. This is one such project: https://github.com/songquanpeng/one-api

[jerowe]

For example if I look at the openai docs - https://platform.openai.com/docs/api-reference/chat/create

This is an example response:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-3.5-turbo-0125",
  "system_fingerprint": "fp_44709d6fcb",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "\n\nHello there, how may I assist you today?",
    },
    "logprobs": null,
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
  }
}

I checked it with curl and python.

[jerowe]

I think the answer is here -

https://github.com/openai/openai-python/blob/main/src/openai/_streaming.py

And then to use an SSE library specific to your framework (so flask for me) https://flask-sse.readthedocs.io/en/latest/quickstart.html

Thanks for the tips! When I get it figured out I'll post back with the solution!

[jerowe]

Hello!

So I haven't been able to figure out the streaming response, but since I don't need to stream it doesn't really matter to me. (Personally. I'm sure its great and matters to others!)

Flask App

from flask import Flask, Response
from flask import stream_with_context, request
# from flask_sse import sse

import boto3
import pprint
from botocore.client import Config
from langchain.llms.bedrock import Bedrock
from langchain.retrievers.bedrock import AmazonKnowledgeBasesRetriever
from langchain.prompts import PromptTemplate
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
from langchain_community.chat_models import BedrockChat
from langchain_core.messages import HumanMessage
from langchain.chains import RetrievalQA
import json
import os
import logging
from rich.logging import RichHandler
import json
import time

FORMAT = "%(message)s"
logging.basicConfig(
    level="INFO", format=FORMAT, datefmt="[%X]", handlers=[RichHandler()]
)
app = Flask(__name__)
# app.config["REDIS_URL"] = "redis://flask-api-redis"

CHAT_GPT_RESPONSE = {
    "id": "chatcmpl-948NlyY8WiorHEGSuWbJJEF1HECHD",
    "object": "chat.completion",
    "created": 1710772561,
    "model": "gpt-3.5-turbo-0125",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello! How can I assist you today?",
            },
            "logprobs": None,
            "finish_reason": "stop",
        }
    ],
    "usage": {"prompt_tokens": 19, "completion_tokens": 9, "total_tokens": 28},
    "system_fingerprint": "fp_4f2ebda25a",
}


@app.route("/v1/chat/completions", methods=["GET", "POST"])
def v1_chat_completions():
    # logging.info(request)
    # logging.info(request.headers)
    data = request.data.decode("utf-8")
    logging.info(data)

    return CHAT_GPT_RESPONSE


if __name__ == "__main__":
    app.run("0.0.0.0", port=3000, debug=True)

Librechat config

# For more information, see the Configuration Guide:
# https://docs.librechat.ai/install/configuration/custom_config.html

# Configuration version (required)
version: 1.0.3

# Cache settings: Set to true to enable caching
cache: true

# Definition of custom endpoints
endpoints:
  custom:
    - name: "KB"  # Unique name for the endpoint"
    # Mistral AI API
    - name: "My Knowledge Base"  # Unique name for the endpoint
      # For `apiKey` and `baseURL`, you can use environment variables that you define.
      # recommended environment variables:
      apiKey: "API_KEY"
      baseURL: "http://flask-api:3000/v1/"
      # Models configuration
      models:
        # List of default models to use. At least one value is required.
        default: [ "kb-1" ]
        # Fetch option: Set to true to fetch models from API.
        fetch: false  # Defaults to false.

      # Optional configurations
      
      # Title Conversation setting
      titleConvo: true  # Set to true to enable title conversation

      # Title Method: Choose between "completion" or "functions".
      #      titleMethod: "completion"  # Defaults to "completion" if omitted.
      titleMethod: "functions"  # Defaults to "completion" if omitted.

      # Title Model: Specify the model to use for titles.
      titleModel: "My KnowledgeBase"  # Defaults to "gpt-3.5-turbo" if omitted.

      # Summarize setting: Set to true to enable summarization.
      summarize: false

      # Summary Model: Specify the model to use if summarization is enabled.
      summaryModel: "kb-1"  # Defaults to "gpt-3.5-turbo" if omitted.

      # Force Prompt setting: If true, sends a `prompt` parameter instead of `messages`.
      forcePrompt: false

      # The label displayed for the AI model in messages.
      modelDisplayLabel: "My Knowledge Base"  # Default is "AI" when not set.

      # Add additional parameters to the request. Default params will be overwritten.
      addParams:
        safe_prompt: true # This field is specific to Mistral AI: https://docs.mistral.ai/api/
        stream: false

I think the important part is to set stream: false, and then I can just return a regular json object.

Thanks again! Its a great project and I'm excited to get started on presets and assistants next!

[danny-avila]

thanks for sharing! some great updates coming just this week so stay tuned