product/101_portkey_gateway_configs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMhtu16mQSfX2yZ7d7tTWqx",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Portkey-AI/portkey-cookbook/blob/main/product/101_portkey_gateway_configs.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 101 on Portkey's Gateway Configs\n",
        "You are likely familiar with how to make an API call to GPT4 for chat completions. However, did you know you can set up automatic retries for requests that might fail on OpenAI’s end using Portkey?\n",
        "\n",
        "The Portkey AI gateway provides several useful features that you can use to enhance your requests. In this cookbook, we will start by making an API call to LLM and explore how Gateway Configs can be utilized to optimize these API calls\n",
        "\n",
        "![](https://github.com/Portkey-AI/portkey-cookbook/blob/main/product/images/101-portkey-gateway-configs/1-gateway-config-builder.png?raw=true)"
      ],
      "metadata": {
        "id": "PL39um8eAiRL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. API calls to LLMs with Portkey\n",
        "\n",
        "Consider a typical API call to GPT4 to get chat completions using OpenAI SDK. It takes `messages` and `model` arguments to get us a response. If you have tried one before, the following code snippet should look familiar. That’s because Portkey Client SDK follows the same signature as OpenAI’s."
      ],
      "metadata": {
        "id": "htopmHfoczP-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install portkey-ai openai"
      ],
      "metadata": {
        "id": "I1Sr07ohc5Vk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from portkey_ai import Portkey\n",
        "from google.colab import userdata\n",
        "\n",
        "PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "\n",
        "portkey = Portkey(\n",
        "    api_key=PORTKEYAI_API_KEY,\n",
        "    virtual_key=OPENAI_VIRTUAL_KEY\n",
        ")\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "response = portkey.chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-4'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u1LUpGnPd1DY",
        "outputId": "36a8e6e3-9935-42f5-b8ec-af0d84e8b952"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1. The Great Wall of China (China)\n",
            "2. Petra (Jordan)\n",
            "3. Christ the Redeemer (Brazil)\n",
            "4. Machu Picchu (Peru)\n",
            "5. The Colosseum (Italy)\n",
            "6. Chichen Itza (Mexico)\n",
            "7. The Taj Mahal (India)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Along with Portkey API Key ([get one](https://portkey.ai/docs/api-reference/authentication#obtaining-your-api-key)), you might’ve noticed a new parameter while instantiating the `portkey` variable — `virtualKey`. Portkey securely stores API keys of LLM providers in a vault and substitutes them at runtime in your requests. These unique identifiers to your API keys are called Virtual Keys. For more information, see the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys#creating-virtual-keys).\n",
        "\n",
        "With basics out of our way, let’s jump into applying what we set out to do in the first place with the AI gateway — To automatically retry our request when we hit rate-limits (429 status codes)."
      ],
      "metadata": {
        "id": "qHgbQ3OtjJMg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Apply Gateway Configs\n",
        "\n",
        "The AI gateway requires instructions to automatically retry requests. This involves providing Gateway Configs, which are essentially JSON objects that orchestrate the AI gateway. In our current scenario, we are targeting GPT4 with requests that have automatic retries on 429 status codes.\n",
        "\n",
        "```json\n",
        "{\n",
        "  \"retry\": {\n",
        "    \"attempts\": 3,\n",
        "    \"on_status_codes\": [429]\n",
        "  }\n",
        "}\n",
        "```\n",
        "We now have our Gateway Configs sorted. But how do we instruct our AI gateway?\n",
        "\n",
        "You guessed it, on the request headers. The next section will explore two ways to create and reference Gateway Configs."
      ],
      "metadata": {
        "id": "JC10TCn0jMF5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### a. Reference Gateway Configs from the UI\n",
        "\n",
        "Just as the title says — you create them on the UI and use an ID to have Portkey automatically apply in the request headers to instruct the AI gateway. UI builder features lint suggestions, makes it easy to reference (through config ID), eliminates manual management, and allows you to view version history.\n",
        "\n",
        "To create Gateway Configs,\n",
        "\n",
        "1. Go to **portkey.ai** and\n",
        "2. Click on **Configs**\n",
        "   1. Select **Create**\n",
        "   2. Choose any name (such as request_retries)\n",
        "\n",
        "Write the configs in the playground and click **Save Config**:\n",
        "\n",
        "\n",
        "![](https://raw.githubusercontent.com/Saif-Shines/pk-cookbook/next/product/images/101-portkey-gateway-configs/1-gateway-config-builder.png)\n",
        "\n",
        "See the saved configs in the list along with the `ID`:\n",
        "\n",
        "![Config ID](https://raw.githubusercontent.com/Saif-Shines/pk-cookbook/next/product/images/101-portkey-gateway-configs/2-list-config-id.png)\n",
        "\n",
        "Try it out now!\n",
        "\n",
        "The Configs saved will appear as a row item on the Configs page. The `ID` is important as it is referenced in our calls through the AI gateway.\n",
        "\n"
      ],
      "metadata": {
        "id": "_37iYjvTjUxK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Portkey SDK\n",
        "\n",
        "The Portkey SDK accepts the config parameter that takes the created config ID as it’s argument. To ensure all requests have automatic retries enabled on them, pass the config ID as argument when `portkey` is instantiated.\n",
        "\n",
        "That’s right! One line of code, and all the request from your apps now inherit Gateway Configs and demonstrate automatic retries.\n",
        "\n",
        "Let’s take a look at the code snippet:"
      ],
      "metadata": {
        "id": "251K-ff7nuPs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from portkey_ai import Portkey\n",
        "from google.colab import userdata\n",
        "\n",
        "PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "CONFIG_ID=\"<input_config_id>\"\n",
        "\n",
        "portkey = Portkey(\n",
        "    api_key=PORTKEYAI_API_KEY,\n",
        "    virtual_key=OPENAI_VIRTUAL_KEY,\n",
        "    config=CONFIG_ID\n",
        ")\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "response = portkey.chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-4'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content)"
      ],
      "metadata": {
        "id": "5C7AudqIoCax"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Requests\n",
        "\n",
        "In the cases, where you are not able to use an SDK, you can pass the same configs as headers with the key `x-portkey-config` ."
      ],
      "metadata": {
        "id": "kx0VoahUMCoD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests, json\n",
        "\n",
        "CONFIG_ID = 'pc-reques-edf21c'\n",
        "PORTKEY_API_KEY =userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_API_KEY = userdata.get('OPENAI_API_KEY')\n",
        "\n",
        "data = {\n",
        "    \"model\": \"gpt-4\",\n",
        "    \"messages\": [{\"role\": \"user\",\"content\": \"What are the 7 wonders of the world?\"}]\n",
        "}\n",
        "\n",
        "headers = {\n",
        "    'Content-Type': 'application/json',\n",
        "    'Authorization': f'Bearer {OPENAI_API_KEY}',\n",
        "    'x-portkey-api-key': PORTKEY_API_KEY,\n",
        "    'x-portkey-provider': 'openai',\n",
        "    'x-portkey-config': CONFIG_ID\n",
        "}\n",
        "\n",
        "response = requests.post('https://api.portkey.ai/v1/chat/completions', headers=headers, data=json.dumps(data))\n",
        "\n",
        "print(response.json())"
      ],
      "metadata": {
        "id": "pUNxvEjfMLhC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### OpenAI SDK\n",
        "\n",
        "Portkey can be used with OpenAI SDK.\n",
        "\n",
        "To send a request with using OpenAI SDK client and apply gateway configs to the request pass a `baseURL` and necessary headers as follows:"
      ],
      "metadata": {
        "id": "CaCzitoGP698"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from openai import OpenAI\n",
        "from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
        "\n",
        "PORTKEY_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "CONFIG_ID='pc-reques-edf21c'\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key=\"<uses_virtual_key>\", # Not respected since Virtual Key is used\n",
        "    base_url=PORTKEY_GATEWAY_URL,\n",
        "    default_headers=createHeaders(\n",
        "        provider=\"openai\",\n",
        "        api_key=PORTKEY_API_KEY,\n",
        "        virtual_key=OPENAI_VIRTUAL_KEY,\n",
        "        config=CONFIG_ID\n",
        "    )\n",
        ")\n",
        "\n",
        "chat_complete = client.chat.completions.create(\n",
        "    model=\"gpt-4\",\n",
        "    messages=[{\"role\": \"user\", \"content\": \"What are the 7 wonders of the world?\"}],\n",
        ")\n",
        "\n",
        "\n",
        "print(chat_complete.choices[0].message.content)"
      ],
      "metadata": {
        "id": "viUPhL33P_cr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The approach to declare the Gateway Configs in the UI and reference them in the code is recommended since it keeps the Configs atomic and decoupled from the business logic and can be upgraded to add more features. What if you want to enable caching for all your thousands of requests? Just update the Configs from the UI. No commits. No redeploys."
      ],
      "metadata": {
        "id": "9Z_Jt3thShGc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### b. Reference Gateway Configs in the Code\n",
        "\n",
        "Depending on the dynamics of your app, you might want to construct the Gateway Configs at the runtime. All you need to do is to pass the Gateway Configs directly to the `config` parameter as an argument."
      ],
      "metadata": {
        "id": "x1EJ_cAHSq1K"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Portkey SDK"
      ],
      "metadata": {
        "id": "o5krybQZSyOn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from portkey_ai import Portkey\n",
        "from google.colab import userdata\n",
        "import json\n",
        "\n",
        "PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "config_data = {\n",
        "    \"retry\": {\n",
        "        \"attempts\": 3,\n",
        "        \"on_status_codes\": [429]\n",
        "    }\n",
        "}\n",
        "\n",
        "\n",
        "portkey = Portkey(\n",
        "    api_key=PORTKEYAI_API_KEY,\n",
        "    virtual_key=OPENAI_VIRTUAL_KEY,\n",
        "    config=json.dumps(config_data)\n",
        ")\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "response = portkey.chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-4'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content)"
      ],
      "metadata": {
        "id": "2YxHjOUZSy9f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Requests"
      ],
      "metadata": {
        "id": "RnyHRTL1U975"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests, json\n",
        "\n",
        "config_data = {\n",
        "    \"retry\": {\n",
        "        \"attempts\": 3,\n",
        "        \"on_status_codes\": [429]\n",
        "    }\n",
        "}\n",
        "\n",
        "CONFIG_ID = json.dumps(config_data)\n",
        "PORTKEY_API_KEY =userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_API_KEY = userdata.get('OPENAI_API_KEY')\n",
        "\n",
        "data = {\n",
        "    \"model\": \"gpt-4\",\n",
        "    \"messages\": [{\"role\": \"user\",\"content\": \"What are the 7 wonders of the world?\"}]\n",
        "}\n",
        "\n",
        "headers = {\n",
        "    'Content-Type': 'application/json',\n",
        "    'Authorization': f'Bearer {OPENAI_API_KEY}',\n",
        "    'x-portkey-api-key': PORTKEY_API_KEY,\n",
        "    'x-portkey-provider': 'openai',\n",
        "    'x-portkey-config': CONFIG_ID\n",
        "}\n",
        "\n",
        "response = requests.post(\n",
        "    'https://api.portkey.ai/v1/chat/completions',\n",
        "    headers=headers,\n",
        "    data=json.dumps(data)\n",
        "  )\n",
        "\n",
        "print(response.json())"
      ],
      "metadata": {
        "id": "zJYKXVglVAy9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### OpenAI SDK"
      ],
      "metadata": {
        "id": "da26Xr1WVds2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from openai import OpenAI\n",
        "from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
        "\n",
        "config_data = {\n",
        "    \"retry\": {\n",
        "        \"attempts\": 3,\n",
        "        \"on_status_codes\": [429]\n",
        "    }\n",
        "}\n",
        "\n",
        "CONFIG_ID = json.dumps(config_data)\n",
        "PORTKEY_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
        "OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
        "\n",
        "messages = [\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What are the 7 wonders of the world?\"\n",
        "    }\n",
        "]\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key=\"<uses_virtual_key>\", # Not respected since Virtual Key is used\n",
        "    base_url=PORTKEY_GATEWAY_URL,\n",
        "    default_headers=createHeaders(\n",
        "        provider=\"openai\",\n",
        "        api_key=PORTKEY_API_KEY,\n",
        "        virtual_key=OPENAI_VIRTUAL_KEY,\n",
        "        config=CONFIG_ID\n",
        "    )\n",
        ")\n",
        "\n",
        "chat_complete = client.chat.completions.create(\n",
        "    model=\"gpt-4\",\n",
        "    messages=[{\"role\": \"user\", \"content\": \"What are the 7 wonders of the world?\"}],\n",
        ")\n",
        "\n",
        "\n",
        "print(chat_complete.choices[0].message.content)"
      ],
      "metadata": {
        "id": "Jggs-elAVmpk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Those are three ways to use Gateway Configs in your requests.\n",
        "\n",
        "In the cases where you want to specifically add a config for a specific request instead of all, Portkey allows you to pass `config` argument as seperate objects right at the time of chat completions call instead of `Portkey({..})` instantiation."
      ],
      "metadata": {
        "id": "ymVFZkf4VxfE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = portkey.with_options(config=\"<config_id>\").chat.completions.create(\n",
        "    messages = messages,\n",
        "    model = 'gpt-4'\n",
        ")\n",
        "\n",
        "print(response.choices[0].message.content)"
      ],
      "metadata": {
        "id": "UaNtJAavWZs6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Applying retry super power to your requests is that easy!"
      ],
      "metadata": {
        "id": "WHC1H4ZUWowO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Next Steps: Dive into features of AI gateway\n",
        "\n",
        "Great job on implementing the retry behavior for your LLM calls to OpenAI!\n",
        "\n",
        "Gateway Configs is a tool that can help you manage fallbacks, request timeouts, load balancing, caching, and more. With Portkey's support for over 100+ LLMs, it is a powerful tool for managing complex use cases that involve multiple target configurations. A Gateway Config that encompasses such complexity may look like:\n",
        "\n",
        "```\n",
        "TARGET 1 (root):\n",
        "  OpenAI GPT4\n",
        "  Simple Cache\n",
        "  On 429:\n",
        "    TARGET 2 (loadbalance):\n",
        "      Anthropic Claude3\n",
        "      Semantic Cache\n",
        "      On 5XX\n",
        "    TARGET 3 (loadbalance):\n",
        "      Anyscale Mixtral 7B\n",
        "      On 4XX, 5XX\n",
        "        TARGET 4 (fallback):\n",
        "          Llama Models\n",
        "          Automatic Retries\n",
        "          Request Timeouts\n",
        "```\n",
        "\n",
        "For complete reference, refer to the _[Config Object](https://portkey.ai/docs/api-reference/config-object)_.\n",
        "\n",
        "It's exciting to see all the AI gateway features available for your requests. Feel free to experiment and make the most of them. Keep up the great work!"
      ],
      "metadata": {
        "id": "0upDoU4rWp66"
      }
    }
  ]
}