Implementing OAuth Plugins, particularly Noteable

[rgbkrk]

Hey! 👋🏻 I'm Kyle from Noteable, along with @shouples and @kafonek.

Recently @glowforgedan told me about LibreChat's ability to work with ChatGPT Plugins and the desire to support OAuth, particularly Noteable. I'm coming to you with a few options that I'll also post over to LangChain's repo.

Note: I realize that LibreChat is written in JavaScript so these are mostly here to get discussions going.

ChatGPT Plugin API Using RequestsWrapper and LangChain OpenAPI Schema

Since we're not seeing a way to inject headers with AIPluginTool.from_plugin_url @shouples went with the following approach

1. Pull the OpenAPI Spec pulled from the plugin (e.g. chat.noteable.io)
2. Create a RequestsWrapper to incorporate the Noteable Token
3. Set up an openapi_agent

Pull the OpenAPI Spec

import requests
from langchain.agents.agent_toolkits.openapi.spec import reduce_openapi_spec

noteable_api_spec = requests.get("https://chat.noteable.io/api/origami/openapi.json").json()
noteable_api_spec = reduce_openapi_spec(noteable_api_spec)

Set up a RequestsWrapper with a Noteable API Token

Caveat: our system currently only accepts JWTs. We're working on releasing a version that will accept your API Token as well as the OAuth token.

import os
from langchain.requests import RequestsWrapper
app_token = os.environ['NOTEABLE_TOKEN']
requests_wrapper = RequestsWrapper(headers={"Authorization": f"Bearer {app_token}"})

Create an agent using the spec and wrapper

from langchain.agents.agent_toolkits.openapi import planner
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-4", temperature=0.0)

origamist_agent = planner.create_openapi_agent(
    api_spec=noteable_api_spec,
    requests_wrapper=requests_wrapper,
    llm=llm,
)

result = origamist_agent.run("Create a streamplot")

Some major issues with this approach is that it doesn't include the model_for_description from Noteable's AI Plugin JSON. This does lead the model towards some eh paths instead of diving in like your personal data science buddy.

Directly using origami, Noteable's API

Any approach using origami directly will be faster to work with (no round trip to the plugin, full notebook document synced in realtime).

Example Client

I've created a simplified client for LLMs around origami's API and RTU Client (Real time updates) https://github.com/rgbkrk/chatlab/blob/main/chatlab/builtins/noteable.py#L24. Since it was built purposefully for use with OpenAI Function Calling, I'm using the related langchain modules.

pip install chatlab[noteable] --pre -q

import os
from chatlab.builtins.noteable import NotebookClient

# Create a new notebook that will be given to the agent
nc = await NotebookClient.create("Demo for LibreChat.ipynb")
print(nc.notebook_url)

from langchain.chains.openai_functions import (
    create_openai_fn_chain,
    create_structured_output_chain,
)
from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate, HumanMessagePromptTemplate
from langchain.schema import HumanMessage, SystemMessage

from pydantic import BaseModel, Field
from typing import Optional, Literal

class CreateCell(BaseModel):
    """Create a cell. To run it as part of this call, set `and_run` to true."""
    source: str = Field(..., description="The source code for the cell")
    cell_type: Literal['code', 'markdown', 'sql'] = Field(..., description="cell type")
    cell_id: Optional[str] = Field(..., description="The ID of the cell to create. Optional.")

llm = ChatOpenAI(model="gpt-4", temperature=0)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a world class data scientist."),
        ("human", "Make calls to the relevant functions to work in a notebook using the following input: {input}"),
        ("human", "Tip: Make sure to answer in the correct format"),
    ]
)

chain = create_openai_fn_chain(
    [CreateCell], llm, prompt, verbose=True
)
op = chain.run("Make me a nice cell to show a streamplot")

if isinstance(op, CreateCell):
    display("Running", op)
    await nc.create_cell(**op.dict())

This works, though there are several ways I wish we could improve this:

• Pass the functions directly to create_openai_fn_chain. Right now it doesn't like methods, which is odd to me because I setup something similar in chatlab to accept bound functions.
• Instead of just creating the pydantic model, I want to let the agent call the function directly.

In chatlab I would just run:

from chatlab import Chat

chat = Chat(
  chat_functions=nc.chat_functions
)
await chat("Make me a nice cell to show a streamplot")

[danny-avila]

Hey @rgbkrk @shouples @kafonek! 👋

First off, amazing work with Noteable. It was actually the first plugin I attempted to get working with this project, and I'm honored to have LibreChat looked at by you and your team!

Some major issues with this approach is that it doesn't include the model_for_description from Noteable's AI Plugin JSON

My LangChainJS method of using the OpenAPI plugins gets over this issue as well as makes all the OpenAPI spec methods known in the system message, but I lack a way to authenticate.

In short, I basically create an OpenAPI chain as a tool for a functions agent.

We're working on releasing a version that will accept your API Token as well as the OAuth token.

This would make the OpenAPI method possible, and would be really welcome!

---

As for the second method, I agree it would be easier to work with the noteable SDK.

Pass the functions directly to create_openai_fn_chain. Right now it doesn't like methods, which is odd to me because I setup something similar in chatlab to accept bound functions.
Instead of just creating the pydantic model, I want to let the agent call the function directly.

For both points, the functions agent sounds like it will work better for you as it can accept not just LangChain Tools but OpenAI functions as Tools.
https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent

Since noteable SDK is in python, this project would still benefit from your development on the client, and while not as proficient as you in Python, I would be excited to help as well. For LibreChat, I can proxy from LangChainJS to a python micro-service.

[danny-avila]

In short, I basically create an OpenAPI chain as a tool for a functions agent.

This is my "second" iteration and I have a better way still in the works, as I learn more of how langchain handles this under the hood.

[shouples]

Hello @danny-avila! 👋

I'm definitely still learning more about how langchain handles this workflow as well, so you're not alone! As a quick update from the original post @rgbkrk made; we were able to use the OpenAPIToolkit and AIPluginTool, albeit with some hiccups hitting the token limit (likely from OpenAPI spec duplication happening between the JsonSpec and the AIPluginTool). Here's our latest example in python:

from langchain.agents import AgentType, load_tools, initialize_agent
from langchain.agents.agent_toolkits import OpenAPIToolkit
# from langchain.agents.agent_toolkits.openapi import planner
# from langchain.agents.agent_toolkits.openapi.spec import reduce_openapi_spec
from langchain.chat_models import ChatOpenAI
from langchain.requests import RequestsWrapper
from langchain.tools import AIPluginTool
from langchain.tools.json.tool import JsonSpec
​
import httpx


client = httpx.Client()
openapi_spec_resp = client.get("https://chat.noteable.io/api/origami/openapi.json")
​
llm = ChatOpenAI(model_name="gpt-4", temperature=0.0)
# wrap requests using token from https://app.noteable.io/api/token
requests_wrapper = RequestsWrapper(headers={"Authorization": f"Bearer {app_token}"})
​
# old agent using just the request wrapper and the reduced OpenAPI spec
# origamist_agent = planner.create_openapi_agent(
#     api_spec=reduce_openapi_spec(openapi_spec_resp.json()),
#     requests_wrapper=requests_wrapper,
#     llm=llm,
# )
​
# new agent using the OpenAPIToolkit with the Noteable ChatGPT plugin manifest
openai_toolkit = OpenAPIToolkit.from_llm(
    llm=llm,
    json_spec=JsonSpec(dict_=openapi_spec_resp.json()),
    requests_wrapper=requests_wrapper,
    verbose=True,
)
tools = [
    AIPluginTool.from_plugin_url("https://chat.noteable.io/.well-known/ai-plugin.json")
] + openai_toolkit.get_tools()
​
origamist_agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
    requests_wrapper=requests_wrapper,
)
​
result = origamist_agent.run( ... )

[rgbkrk]

This is my "second" iteration and I have a better way still in the works, as I learn more of how langchain handles this under the hood.

Happy to learn from you! I'd love to make this much easier to work with. We have a lot of learning lessons from the plugin we'd love to bring into the LangChain universe.

[danny-avila]

@shouples the official ChatGPT plugins implementation seem to run openapi routes/methods as OpenAI functions, which get interpreted by some backend service to do the heavy lifting of the actual API call.

I think going for this direct format could be more token efficient than the AIPlugin + OpenAPIToolkit approach. I'm not sure if langchain Python has the openApiChain that langchainJS has, but the latter does a lot of algorithmic work to the end I outlined, converting the spec endpoints to OpenAI functions.

I benefit off this + using a dynamicTool to create an "AIPlugin" dynamically. I'm working on improving it by utilizing a lot of the algorithmic logic from createOpenAPIChain without actually using it, as it's double work for the LLM to decide on a method, and then rely on the chain to call it.

[danshapiro]

Is there anything I or anyone can do to help you with this?

[rgbkrk]

The Noteable ChatGPT Plugin (chat.noteable.io) now works with API Keys in addition to OAuth. Generate an API key in the settings page on app.noteable.io

Then reach out to chat.noteable.io with a Bearer token.

➜ curl -H "Authorization: Bearer $NOTEABLE_TOKEN" https://chat.noteable.io/api/origami/u/me
{"id":"781e7113-59de-474f-a88f-79977f705770","email":"rgbkrk@gmail.com","auth_type":"pat:40cd170759b647c3a2d60ba78bccb5e2","default_project":{"id":"3b179d03-9efb-4f6c-ad7a-fa0a478d45e4","name":"IMDb ETL","description":""}

[danny-avila]

Amazing! Will try to set this up for LibreChat

[glowforgedan]

Y'all are amazing. I'm terribly excited for this.

…

On Thu, Oct 12, 2023 at 11:46 AM Danny Avila ***@***.***> wrote: Amazing! Will try to set this up for LibreChat — Reply to this email directly, view it on GitHub <#929 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIPO5QNSCEEXSPJMBUBDFTX7A3G7ANCNFSM6AAAAAA4ZJ23GE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shouples]

@danny-avila it took longer than I'd like to get back into this thread, but I'm ready to jump in and help get this hammered out!

Looking at the langchain Python docs, there's a get_openapi_chain function we can use that accepts custom request headers. I think this is probably the closest available option to the JS createOpenAPIChain function you mentioned. Setup is pretty straightforward (using the same custom headers @rgbkrk mentioned in his last post):

import requests

from langchain.chains.openai_functions.openapi import get_openapi_chain
from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain.utilities.openapi import OpenAPISpec


MANIFEST_URL = "https://chat.noteable.io/.well-known/ai-plugin.json"
OPENAPI_SPEC_URL = "https://chat.noteable.io/api/origami/openapi.json"

manifest = requests.get(MANIFEST_URL).json()
openapi_spec = requests.get(OPENAPI_SPEC_URL).json()

# should we try adding the `description_for_model` from https://chat.noteable.io/.well-known/ai-plugin.json in here?
prompt_template = ChatPromptTemplate.from_template(
    "Use the provided APIs to respond to this user query:\n\n{query}"
)

llm = ChatOpenAI(
    model_name="gpt-4",
    temperature=0.0,
)

# load Noteable API key from env/file/etc
# noteable_token = os.environ['NOTEABLE_TOKEN']
custom_headers = {"Authorization": f"Bearer {noteable_token}"}

noteable_chain = get_openapi_chain(
    spec=OpenAPISpec.parse_obj(openapi_spec),
    llm=llm,
    verbose=True,
    prompt=prompt_template,
    headers=custom_headers,
)

noteable_chain.run("what's my default project in Noteable?")

> Entering new SequentialChain chain...

> Entering new LLMChain chain...
Prompt after formatting:
Human: Use the provided APIs to respond to this user query:

what's my default project in Noteable?

> Finished chain.


> Entering new SimpleRequestChain chain...
Calling endpoint get_user_info with arguments:
{}
> Finished chain.

> Finished chain.

{'id': '266b1248-528a-48d0-b880-d012d29924ce',
 'email': 'david.shoup@noteable.io',
 'auth_type': 'pat:1aaa384b7c8c456aad132f7446c07094',
 'default_project': {'id': '91300166-e481-4d40-b700-b7f012e33d8f',
  'name': 'CHATGPT PLUGIN TESTING',
  'description': ''}}

This is where it gets a little fuzzy for me based on my limited exposure to langchain chains and tools -- what are your thoughts on how this could/should be implemented as a tool (or toolkit) for an agent?

One thing I like about the OpenAPIToolkit is it comes with the following tools for exploring the OpenAPI spec and making the requests (with a wrapper):

[RequestsGetTool(requests_wrapper=TextRequestsWrapper(headers={'Authorization': 'Bearer XXXXXXXXXXXXXXXXXXXX'}, aiosession=None, auth=None)),
 RequestsPostTool(requests_wrapper=TextRequestsWrapper(headers={'Authorization': 'Bearer XXXXXXXXXXXXXXXXXXXX'}, aiosession=None, auth=None)),
 RequestsPatchTool(requests_wrapper=TextRequestsWrapper(headers={'Authorization': 'Bearer XXXXXXXXXXXXXXXXXXXX'}, aiosession=None, auth=None)),
 RequestsPutTool(requests_wrapper=TextRequestsWrapper(headers={'Authorization': 'Bearer XXXXXXXXXXXXXXXXXXXX'}, aiosession=None, auth=None)),
 RequestsDeleteTool(requests_wrapper=TextRequestsWrapper(headers={'Authorization': 'Bearer XXXXXXXXXXXXXXXXXXXX'}, aiosession=None, auth=None)),
 Tool(name='json_explorer', description="Can be used to answer questions about the openapi spec for the API. Always use this tool before trying to make a request. ... )

...but doesn't seem to take advantage of the ReducedOpenAPISpec. And we'd either have to add in some extra context around what the plugin can do, or (on the Noteable side) provide our manifest's description_for_model in the OpenAPI spec description.

[danny-avila]

@rgbkrk @shouples for what it's worth, I came up with a much better way of handling OpenAPI specs for the upcoming Assistants update.

What I implemented for Assistants will work with regular "plugins" just not at first. Wanted to try noteable next but noticed I couldn't access app.noteable.io

[glowforgedan]

Sadly Noteable ceased operating. Disappointing; there's nothing quite as good to replace it.

…

--dan

On Mon, Jan 29, 2024 at 10:41 AM Danny Avila ***@***.***> wrote: @rgbkrk <https://github.com/rgbkrk> @shouples <https://github.com/shouples> for what it's worth, I came up with a much better way of handling OpenAPI specs for the upcoming Assistants update. What I implemented for Assistants will work with regular "plugins" just not at first. Wanted to try noteable next but noticed I couldn't access app.noteable.io image.png (view on web) <https://github.com/danny-avila/LibreChat/assets/110412045/ab136af5-b419-4824-bb51-d4d86a77dd1e> — Reply to this email directly, view it on GitHub <#929 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIPO5UZI4X2VQBD47UZIR3YQ7UO5AVCNFSM6AAAAAA4ZJ23GGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DEOBYGY2TS> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[danny-avila]

Ah I see, that is sad. One of these days I will get around to figuring out how to setup JupyterHub for LibreChat. A friend has figured out an open source, scalable "code interpreter" through this. It's easy enough to get a local instance going but need to go into the weeds of documentation to set this up right.

[glowforgedan]

If memory serves, JupyterHub was built by one of the Noteable cofounders before they founded the company, so it's probably the next-best thing! It would be really valuable to have something to replace Code Interpreter.

…

On Mon, Jan 29, 2024 at 10:50 AM Danny Avila ***@***.***> wrote: Ah I see, that is sad. One of these days I will get around to figuring out how to setup JupyterHub for LibreChat. A friend has figured out an open source, scalable "code interpreter" through this. It's easy enough to get a local instance going but need to go into the weeds of documentation to set this up right. — Reply to this email directly, view it on GitHub <#929 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIPO5Q7NHSSHTTZL77G4U3YQ7VOFAVCNFSM6AAAAAA4ZJ23GGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DEOBYG43TO> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rgbkrk]

I'm happy to comment on this as a fully independent person that worked at Noteable. I didn't found Noteable. Just to clear things up, none of the founders created JupyterHub.

Early days on with JupyterHub I created tmpnb which became the Dockerspawner that folded into JupyterHub. That work helped form much of the original JupyterHub and Binder setup. Many more maintainers took it much further after. I went on to Netflix where we had our own notebook orchestration system directly built on that original tmpnb code and plugging in with our container orchestration system.

Coming up to the future here, I am currently evaluating how we can enable LLM-assisted compute on existing JupyterHub deployments. I've been talking to Jupyter folks about it recently while building prototypes. I'm currently seeking grant funding to make it happen. I'd love to talk to folks to make connections and help re-enable the amazing experience of notebooks + LLMs.