mistralEval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "faa969bc",
   "metadata": {},
   "source": [
    "# Mistral Benchmarking & Prompt Completion evaluation using SageMaker Jumpstart\n",
    "\n",
    "---\n",
    "Let's do some text generation, benchmarking and evaluation for Mistral models on sagemaker jumpstart. You can easily deploy the model on jumpstart.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34b9a1f1",
   "metadata": {},
   "source": [
    "## Setup\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "15510777",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdOnly"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id = \"huggingface-llm-mistral-7b-instruct\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "85a2a8e5-789f-4041-9927-221257126653",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using model 'huggingface-llm-mistral-7b-instruct' with wildcard version identifier '*'. You can pin to version '2.0.0' for more stable results. Note that models may have different input/output signatures after a major version upgrade.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------!"
     ]
    }
   ],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "model = JumpStartModel(model_id=model_id)\n",
    "predictor = model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7910793e",
   "metadata": {},
   "source": [
    "### Supported parameters\n",
    "\n",
    "***\n",
    "This model supports many parameters while performing inference. They include:\n",
    "\n",
    "* **max_length:** Model generates text until the output length (which includes the input context length) reaches `max_length`. If specified, it must be a positive integer.\n",
    "* **max_new_tokens:** Model generates text until the output length (excluding the input context length) reaches `max_new_tokens`. If specified, it must be a positive integer.\n",
    "* **num_beams:** Number of beams used in the greedy search. If specified, it must be integer greater than or equal to `num_return_sequences`.\n",
    "* **no_repeat_ngram_size:** Model ensures that a sequence of words of `no_repeat_ngram_size` is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **early_stopping:** If True, text generation is finished when all beam hypotheses reach the end of sentence token. If specified, it must be boolean.\n",
    "* **do_sample:** If True, sample the next word as per the likelihood. If specified, it must be boolean.\n",
    "* **top_k:** In each step of text generation, sample from only the `top_k` most likely words. If specified, it must be a positive integer.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "* **return_full_text:** If True, input text will be part of the output generated text. If specified, it must be boolean. The default value for it is False.\n",
    "* **stop**: If specified, it must a list of strings. Text generation stops if any one of the specified strings is generated.\n",
    "\n",
    "We may specify any subset of the parameters mentioned above while invoking an endpoint. Next, we show an example of how to invoke endpoint with these arguments\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306389a4",
   "metadata": {},
   "source": [
    "## Instruction prompts\n",
    "***\n",
    "The examples in this section demonstrate queries to the Mistral 7B Instruct model. This involves special token formatting within the prompt input. The base, pre-trained Mistral model is not fine-tuned to perform this instruction task -- please use the prompts in the next section instead.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2c6a501",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from typing import Dict, List\n",
    "\n",
    "\n",
    "def format_instructions(instructions: List[Dict[str, str]]) -> List[str]:\n",
    "    \"\"\"Format instructions where conversation roles must alternate user/assistant/user/assistant/...\"\"\"\n",
    "    prompt: List[str] = []\n",
    "    for user, answer in zip(instructions[::2], instructions[1::2]):\n",
    "        prompt.extend([\"<s>\", \"[INST] \", (user[\"content\"]).strip(), \" [/INST] \", (answer[\"content\"]).strip(), \"</s>\"])\n",
    "    prompt.extend([\"<s>\", \"[INST] \", (instructions[-1][\"content\"]).strip(), \" [/INST] \"])\n",
    "    return \"\".join(prompt)\n",
    "\n",
    "\n",
    "def print_prompt_and_response(prompt: str, response: str) -> None:\n",
    "    bold, unbold = '\\033[1m', '\\033[0m'\n",
    "    print(f\"{bold}> Input{unbold}\\n{prompt}\\n\\n{bold}> Output{unbold}\\n{response[0]['generated_text']}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "456d35f3",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m> Input\u001b[0m\n",
      "<s>[INST] what is the recipe of mayonnaise? [/INST] \n",
      "\n",
      "\u001b[1m> Output\u001b[0m\n",
      "1. 1/2 cup egg yolks\n",
      "2. 1 tablespoon Dijon mustard\n",
      "3. 2 tablespoons lemon juice\n",
      "4. 1/2 teaspoon salt\n",
      "5. 3/4 cup vegetable oil or canola oil\n",
      "6. Equipment: whisk, cutting board, measure, mixing bowl.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "instructions = [{\"role\": \"user\", \"content\": \"what is hindi?\"}]\n",
    "prompt = format_instructions(instructions)\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\"max_new_tokens\": 256, \"do_sample\": True}\n",
    "}\n",
    "response = predictor.predict(payload)\n",
    "print_prompt_and_response(prompt, response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e8aa0242",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m> Input\u001b[0m\n",
      "<s>[INST] I am going to Paris, what should I see? [/INST] Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n",
      "\n",
      "1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n",
      "2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n",
      "3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n",
      "\n",
      "These are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world.</s><s>[INST] What is so great about #1? [/INST] \n",
      "\n",
      "\u001b[1m> Output\u001b[0m\n",
      "1. The Eiffel Tower: The Eiffel Tower is one of the most famous and iconic landmarks in the world. Standing at 330 meters tall, it offers stunning views of the city from its three observation decks. Visitors can either take the stairs or elevator to the top and enjoy panoramic views of Paris. The Eiffel Tower is also a symbol of French culture and is an important part of the country's history and identity.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "instructions = [\n",
    "    {\"role\": \"user\", \"content\": \"I am going to Paris, what should I see?\"},\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": \"\"\"\\\n",
    "Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n",
    "\n",
    "1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n",
    "2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n",
    "3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n",
    "\n",
    "These are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world.\"\"\",\n",
    "    },\n",
    "    {\"role\": \"user\", \"content\": \"What is so great about #1?\"},\n",
    "]\n",
    "prompt = format_instructions(instructions)\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\"max_new_tokens\": 256, \"do_sample\": True}\n",
    "}\n",
    "response = predictor.predict(payload)\n",
    "print_prompt_and_response(prompt, response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c16697f",
   "metadata": {},
   "source": [
    "## Pre-trained model prompts\n",
    "***\n",
    "The examples in this section demonstrate how to perform text generation on the base, pre-trained Mistral model. If you have deployed the instruction-tuned model, please use prompt formatting in the previous section instead.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ffbd1391",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m> Input\u001b[0m\n",
      "Write a program to compute factorial in python:\n",
      "\n",
      "\u001b[1m> Output\u001b[0m\n",
      "\n",
      "\n",
      "```python\n",
      "def factorial(n):\n",
      "    if n == 0:\n",
      "        return 1\n",
      "    else:\n",
      "        return n * factorial(n-1)\n",
      "\n",
      "n = int(input(\"Enter a number: \"))\n",
      "result = factorial(n)\n",
      "print(\"Factorial of\", n, \"is\", result)\n",
      "```\n",
      "\n",
      "In this program, we define a function named `factorial` that takes an integer `n` as an argument. Inside the function, we use recursion to compute the factorial of the given number. If the value of `n` is `0`, then the function returns `1`. Otherwise, it returns the product of `n` and the factorial of `n-1`.\n",
      "\n",
      "We then take an input from the user to compute the factorial of a given number. We call the `factorial` function with the input value and store the result in a\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Write a program to compute factorial in python:\"\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"max_new_tokens\": 200,\n",
    "    },\n",
    "}\n",
    "response = predictor.predict(payload)\n",
    "print_prompt_and_response(prompt, response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "368c27c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m> Input\u001b[0m\n",
      "Building a website can be done in 10 simple steps:\n",
      "\n",
      "\u001b[1m> Output\u001b[0m\n",
      "\n",
      "\n",
      "1. Choose a domain name: This is the first step in creating a website. Choose a name that represents your brand or business.\n",
      "\n",
      "2. Choose a hosting provider: A hosting provider is a company that provides the server space where your website will be stored.\n",
      "\n",
      "3. Choose a website builder: A website builder is a tool that allows you to create a website without any coding knowledge.\n",
      "\n",
      "4. Choose a website template: A website template is a pre-designed layout that you can customize to fit your\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Building a website can be done in 10 simple steps:\"\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"max_new_tokens\": 110,\n",
    "        \"no_repeat_ngram_size\": 3,\n",
    "    },\n",
    "}\n",
    "response = predictor.predict(payload)\n",
    "print_prompt_and_response(prompt, response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09c2843a-f5ee-4f9e-b199-7044e8e80802",
   "metadata": {},
   "source": [
    "## Now - Let's deploy Mistral 7b itself and test it on some questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "31efaefb-09e9-4b31-a398-f8a5ecf2c748",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id_noinstruct = \"huggingface-llm-mistral-7b\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "240a1663-6099-4755-a947-f7c1768bd14e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------!"
     ]
    }
   ],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "model_new = JumpStartModel(model_id=model_id)\n",
    "predictor_new = model_new.deploy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c4825cdc-088d-419d-947e-81633500d562",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m> Input\u001b[0m\n",
      "Building a website can be done in 10 simple steps:\n",
      "\n",
      "\u001b[1m> Output\u001b[0m\n",
      "\n",
      "\n",
      "1. Choose a domain name: This is the first step in creating a website. Choose a name that represents your brand or business.\n",
      "\n",
      "2. Choose a hosting provider: A hosting provider is a company that provides the server space where your website will be stored.\n",
      "\n",
      "3. Choose a website builder: A website builder is a tool that allows you to create a website without any coding knowledge.\n",
      "\n",
      "4. Choose a website template: A website template is a pre-designed layout that you can customize to fit your\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Building a website can be done in 10 simple steps:\"\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"max_new_tokens\": 110,\n",
    "        \"no_repeat_ngram_size\": 3,\n",
    "    },\n",
    "}\n",
    "response = predictor_new.predict(payload)\n",
    "print_prompt_and_response(prompt, response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6002318-e329-496b-ab14-4ba4cc11f3b2",
   "metadata": {},
   "source": [
    "## Now, let's load the aisquared/dais-question-answers dataset to test for some latency :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3c83918b-bc25-490c-845b-1ef030173f0a",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: sagemaker in /opt/conda/lib/python3.8/site-packages (2.199.0)\n",
      "Collecting sagemaker\n",
      "  Downloading sagemaker-2.202.1-py2.py3-none-any.whl.metadata (13 kB)\n",
      "Collecting datasets\n",
      "  Downloading datasets-2.16.0-py3-none-any.whl.metadata (20 kB)\n",
      "Requirement already satisfied: attrs<24,>=23.1.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (23.1.0)\n",
      "Requirement already satisfied: boto3<2.0,>=1.33.3 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (1.33.9)\n",
      "Requirement already satisfied: cloudpickle==2.2.1 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (2.2.1)\n",
      "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.8/site-packages (from sagemaker) (0.2.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (1.24.4)\n",
      "Requirement already satisfied: protobuf<5.0,>=3.12 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (4.25.1)\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (1.0.1)\n",
      "Requirement already satisfied: importlib-metadata<7.0,>=1.4.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (4.8.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (21.0)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.8/site-packages (from sagemaker) (1.3.4)\n",
      "Requirement already satisfied: pathos in /opt/conda/lib/python3.8/site-packages (from sagemaker) (0.3.1)\n",
      "Requirement already satisfied: schema in /opt/conda/lib/python3.8/site-packages (from sagemaker) (0.7.5)\n",
      "Requirement already satisfied: PyYAML~=6.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (6.0)\n",
      "Requirement already satisfied: jsonschema in /opt/conda/lib/python3.8/site-packages (from sagemaker) (4.20.0)\n",
      "Requirement already satisfied: platformdirs in /opt/conda/lib/python3.8/site-packages (from sagemaker) (4.1.0)\n",
      "Requirement already satisfied: tblib<3,>=1.7.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (1.7.0)\n",
      "Collecting urllib3<1.27 (from sagemaker)\n",
      "  Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m48.9/48.9 kB\u001b[0m \u001b[31m195.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: uvicorn==0.22.0 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (0.22.0)\n",
      "Requirement already satisfied: fastapi==0.95.2 in /opt/conda/lib/python3.8/site-packages (from sagemaker) (0.95.2)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from sagemaker) (2.31.0)\n",
      "Requirement already satisfied: docker in /opt/conda/lib/python3.8/site-packages (from sagemaker) (6.1.3)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.8/site-packages (from sagemaker) (4.62.3)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.8/site-packages (from sagemaker) (5.8.0)\n",
      "Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /opt/conda/lib/python3.8/site-packages (from fastapi==0.95.2->sagemaker) (1.10.13)\n",
      "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /opt/conda/lib/python3.8/site-packages (from fastapi==0.95.2->sagemaker) (0.27.0)\n",
      "Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.8/site-packages (from uvicorn==0.22.0->sagemaker) (8.1.7)\n",
      "Requirement already satisfied: h11>=0.8 in /opt/conda/lib/python3.8/site-packages (from uvicorn==0.22.0->sagemaker) (0.14.0)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from datasets) (3.3.1)\n",
      "Requirement already satisfied: pyarrow>=8.0.0 in /opt/conda/lib/python3.8/site-packages (from datasets) (14.0.1)\n",
      "Collecting pyarrow-hotfix (from datasets)\n",
      "  Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)\n",
      "Requirement already satisfied: dill<0.3.8,>=0.3.0 in /opt/conda/lib/python3.8/site-packages (from datasets) (0.3.7)\n",
      "Collecting xxhash (from datasets)\n",
      "  Downloading xxhash-3.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Requirement already satisfied: multiprocess in /opt/conda/lib/python3.8/site-packages (from datasets) (0.70.15)\n",
      "Collecting fsspec<=2023.10.0,>=2023.1.0 (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets)\n",
      "  Downloading fsspec-2023.10.0-py3-none-any.whl.metadata (6.8 kB)\n",
      "Collecting aiohttp (from datasets)\n",
      "  Downloading aiohttp-3.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.4 kB)\n",
      "Collecting huggingface-hub>=0.19.4 (from datasets)\n",
      "  Downloading huggingface_hub-0.20.1-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: botocore<1.34.0,>=1.33.9 in /opt/conda/lib/python3.8/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (1.33.9)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.8/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (1.0.1)\n",
      "Requirement already satisfied: s3transfer<0.9.0,>=0.8.2 in /opt/conda/lib/python3.8/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (0.8.2)\n",
      "Collecting multidict<7.0,>=4.5 (from aiohttp->datasets)\n",
      "  Downloading multidict-6.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (121 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.3/121.3 kB\u001b[0m \u001b[31m246.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting yarl<2.0,>=1.0 (from aiohttp->datasets)\n",
      "  Downloading yarl-1.9.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)\n",
      "Collecting frozenlist>=1.1.1 (from aiohttp->datasets)\n",
      "  Downloading frozenlist-1.4.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Collecting aiosignal>=1.1.2 (from aiohttp->datasets)\n",
      "  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets)\n",
      "  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.8/site-packages (from huggingface-hub>=0.19.4->datasets) (4.8.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.8/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (3.6.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging>=20.0->sagemaker) (3.0.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.8/site-packages (from requests->sagemaker) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->sagemaker) (3.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->sagemaker) (2023.11.17)\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.8/site-packages (from docker->sagemaker) (1.7.0)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from google-pasta->sagemaker) (1.16.0)\n",
      "Requirement already satisfied: importlib-resources>=1.4.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema->sagemaker) (6.1.1)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/conda/lib/python3.8/site-packages (from jsonschema->sagemaker) (2023.11.2)\n",
      "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /opt/conda/lib/python3.8/site-packages (from jsonschema->sagemaker) (1.3.10)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /opt/conda/lib/python3.8/site-packages (from jsonschema->sagemaker) (0.31.1)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /opt/conda/lib/python3.8/site-packages (from jsonschema->sagemaker) (0.13.2)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.8/site-packages (from pandas->sagemaker) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas->sagemaker) (2021.3)\n",
      "Requirement already satisfied: ppft>=1.7.6.7 in /opt/conda/lib/python3.8/site-packages (from pathos->sagemaker) (1.7.6.7)\n",
      "Requirement already satisfied: pox>=0.3.3 in /opt/conda/lib/python3.8/site-packages (from pathos->sagemaker) (0.3.3)\n",
      "Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.8/site-packages (from schema->sagemaker) (0.6.0.post1)\n",
      "Requirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/lib/python3.8/site-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (4.1.0)\n",
      "Requirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.8/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.2.0)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in /opt/conda/lib/python3.8/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.2.0)\n",
      "Downloading sagemaker-2.202.1-py2.py3-none-any.whl (1.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m241.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading datasets-2.16.0-py3-none-any.whl (507 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m507.1/507.1 kB\u001b[0m \u001b[31m222.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading fsspec-2023.10.0-py3-none-any.whl (166 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.4/166.4 kB\u001b[0m \u001b[31m230.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading aiohttp-3.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m282.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading huggingface_hub-0.20.1-py3-none-any.whl (330 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m321.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.8/143.8 kB\u001b[0m \u001b[31m294.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
      "Downloading xxhash-3.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.6/194.6 kB\u001b[0m \u001b[31m203.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Downloading frozenlist-1.4.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (240 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m240.9/240.9 kB\u001b[0m \u001b[31m314.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading yarl-1.9.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (308 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m308.8/308.8 kB\u001b[0m \u001b[31m165.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0mInstalling collected packages: xxhash, urllib3, pyarrow-hotfix, multidict, fsspec, frozenlist, async-timeout, yarl, aiosignal, huggingface-hub, aiohttp, sagemaker, datasets\n",
      "  Attempting uninstall: urllib3\n",
      "    Found existing installation: urllib3 2.1.0\n",
      "    Uninstalling urllib3-2.1.0:\n",
      "      Successfully uninstalled urllib3-2.1.0\n",
      "  Attempting uninstall: fsspec\n",
      "    Found existing installation: fsspec 2021.8.1\n",
      "    Uninstalling fsspec-2021.8.1:\n",
      "      Successfully uninstalled fsspec-2021.8.1\n",
      "  Attempting uninstall: sagemaker\n",
      "    Found existing installation: sagemaker 2.199.0\n",
      "    Uninstalling sagemaker-2.199.0:\n",
      "      Successfully uninstalled sagemaker-2.199.0\n",
      "Successfully installed aiohttp-3.9.1 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.16.0 frozenlist-1.4.1 fsspec-2023.10.0 huggingface-hub-0.20.1 multidict-6.0.4 pyarrow-hotfix-0.6 sagemaker-2.202.1 urllib3-1.26.18 xxhash-3.4.1 yarl-1.9.4\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade sagemaker datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a859d46a-4360-47f3-895c-fddbd5b12b83",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7695d2ec9ae64744a3d72403e93480e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "840825"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dais = load_dataset(\"aisquared/dais-question-answers\", split=\"train\")\n",
    "\n",
    "# We split the dataset into two where test data is used to evaluate at the end.\n",
    "train_and_test_dataset = dais.train_test_split(test_size=0.1)\n",
    "\n",
    "# Dumping the training data to a local file to be used for training.\n",
    "train_and_test_dataset[\"train\"].to_json(\"train.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "015b0350-79be-4852-a4d3-7aa9ef57b3c1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompt_inference = (\"\"\"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "dfedb672-7ce5-46c0-9b67-eff0d28316bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results from predictor_new:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Ground Truth Answer</th>\n",
       "      <th>Model Prediction</th>\n",
       "      <th>Inference Latency (seconds)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the Centro de demonstração de produtos e parcerias da Databricks and what services does it offer?</td>\n",
       "      <td>The Centro de demonstração de produtos e parcerias da Databricks is a platform that offers demonstrations of solution accelerators. It also offers services such as documentation, training and certification, resources, online community, alliance with universities, events, and partnership programs for cloud partners and technology and data partners. Additionally, it provides solutions for partners to connect with validated partner solutions.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The Centro de demonstração de produtos e parcerias da Databricks is a platform that allows users to test and evaluate Databricks products and services. It offers a range of features and services, including:\n",
       "\n",
       "1. Data Science Workspace: A cloud-based platform that provides a complete environment for data scientists to develop, test, and deploy machine learning models.\n",
       "\n",
       "2. Apache Spark: An open-source distributed computing engine that'}]</td>\n",
       "      <td>3.368706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of the Data and AI Summit 2023?</td>\n",
       "      <td>The purpose of the Data and AI Summit 2023 is to bring together experts, researchers, contributors, and professionals from the global data community to understand the potential of Large Language Models (LLMs) and shape the future of their industries with data and AI. Attendees can expect to hear from top speakers in the field, including those from Databricks and across the data and AI community, and learn about building, training, and deploying LLMs, as well as other relevant topics in data and AI.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The purpose of the Data and AI Summit 2023 is to bring together industry leaders, experts, and professionals to discuss the latest trends, innovations, and challenges in the field of data and artificial intelligence. The summit aims to provide a platform for knowledge sharing, networking, and collaboration, and to inspire attendees to drive the adoption and implementation of data and AI technologies in their organizations.'}]</td>\n",
       "      <td>2.803292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What AWS competencies did Databricks showcase at AWS re:Invent?</td>\n",
       "      <td>Databricks showcased AWS competencies in data engineering, data warehousing, data streaming, and machine learning at AWS re:Invent.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "At AWS re:Invent, Databricks showcased several AWS competencies, including:\n",
       "\n",
       "1. Big Data: Databricks is a leading big data platform that leverages AWS services such as Amazon S3, Amazon EMR, and Amazon Redshift to process and analyze large volumes of data.\n",
       "2. Machine Learning: Databricks provides a comprehensive machine learning platform that integrates with AWS services such as Amazon SageMaker, Amazon K'}]</td>\n",
       "      <td>3.293204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is the date and time of the upcoming webinar about transitioning from data warehouse to data lakehouse?</td>\n",
       "      <td>The upcoming webinar about transitioning from data warehouse to data lakehouse is on May 18 at 8 AM PT.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The upcoming webinar about transitioning from data warehouse to data lakehouse is on [insert date and time here].'}]</td>\n",
       "      <td>0.863199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What are Databricks Solution Accelerators and how do they help deliver data and AI value faster?</td>\n",
       "      <td>Databricks Solution Accelerators are tools developed by Databricks that help deliver data and AI value faster. They save time in the discovery, design, development, and testing phases by providing pre-built solutions for common use cases in industries such as Financial Services, Healthcare, Manufacturing, and more. These accelerators allow organizations to quickly implement data and AI solutions, enabling them to achieve their business objectives faster.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Databricks Solution Accelerators are pre-built, reusable components that help organizations deliver data and AI value faster. These accelerators are designed to address common data and AI challenges and provide a quick and easy way to get started with Databricks.\n",
       "\n",
       "There are several types of Solution Accelerators available, including:\n",
       "\n",
       "1. Data Pipelines: These accelerators provide pre-built data pipelines that can be used to move data between different'}]</td>\n",
       "      <td>3.295417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>What is the purpose of Databricks Notebooks and how can it benefit different teams in an organization?</td>\n",
       "      <td>The purpose of Databricks Notebooks is to provide a collaborative workspace for teams working on data science, engineering, and machine learning projects. It allows teams to work with familiar languages and tools, use built-in data visualizations, and have automatic versioning within the notebooks. Databricks Notebooks can benefit different teams in an organization by providing a centralized platform for data collaboration, improving productivity, and promoting efficient communication between different teams. It can also help with data analysis, exploration, and modeling, leading to faster and more accurate insights.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Databricks Notebooks is a powerful tool that allows users to create, share, and collaborate on data science projects. It provides a user-friendly interface for creating and executing code, visualizations, and other data-related tasks.\n",
       "\n",
       "The purpose of Databricks Notebooks is to provide a centralized platform for data scientists, analysts, and other teams to work together on data-related projects. It enables teams to easily share code, data'}]</td>\n",
       "      <td>3.294186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>What is the Databricks Lakehouse Platform and how does it fit within a modern data stack?</td>\n",
       "      <td>The Databricks Lakehouse Platform is a data management platform that encompasses a wide range of data technologies, including Delta Lake, data governance, data engineering, data streaming, data warehousing, data sharing, machine learning, and data science. As a \"lakehouse,\" it provides a unified, scalable, and secure platform for managing and processing both structured and unstructured data, bridging the gap between traditional data warehouses and data lakes. It fits within a modern data stack by providing a comprehensive solution for data management, processing, and analysis, and can be integrated with other technologies and tools as needed to meet specific business needs.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The Databricks Lakehouse Platform is a cloud-based data platform that provides a unified environment for data storage, processing, and analytics. It is designed to be a centralized hub for all data-related activities, including data ingestion, ETL, data warehousing, machine learning, and data governance.\n",
       "\n",
       "The Lakehouse Platform is built on top of Apache Spark, an open-source data processing engine that is widely used for big data processing'}]</td>\n",
       "      <td>3.297496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Who is Ruifeng Zheng and what is his role at Databricks?</td>\n",
       "      <td>Ruifeng Zheng is a Senior Software Engineer at Databricks and an Apache Spark committer. He works on various modules in Apache Spark including Spark Connect, Pandas API on Spark, PySpark, MLlib, Spark SQL, SparkR, etc. Prior to Databricks, he worked on applied machine learning for over 10 years. He holds a Master degree in Electronics Engineering from Peking University.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Ruifeng Zheng is a Senior Manager of Product Marketing at Databricks. In this role, he leads the product marketing team and is responsible for developing and executing the product marketing strategy for Databricks' cloud-based big data platform. He works closely with the product development team to understand customer needs and develops marketing materials that effectively communicate the value of Databricks' platform to potential customers.'}]</td>\n",
       "      <td>2.966294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>What are the benefits of using Databricks and how can I try it for free?</td>\n",
       "      <td>The benefits of using Databricks include simplifying data ingestion, automating ETL processes, collaborating in multiple programming languages such as Python, R, Scala, and SQL, and a price performance up to 12x better than cloud data warehouses. To try Databricks for free, one can sign up for a 14-day trial on their AWS, Microsoft Azure, or Google Cloud platform. To do so, visit the Databricks website and fill out the registration form with your name, email, company, and country.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Databricks is a cloud-based data science platform that provides a unified environment for data engineering, machine learning, and analytics. It offers several benefits, including:\n",
       "\n",
       "1. Scalability: Databricks can handle large-scale data processing and analytics tasks with ease. It can scale up or down based on the workload, ensuring that you only pay for what you use.\n",
       "\n",
       "2. Integration: Databricks integrates with a wide range'}]</td>\n",
       "      <td>3.295242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Who is Pieter Noordhuis and what topic will he be speaking about at the Data + AI Summit 2023?</td>\n",
       "      <td>Pieter Noordhuis is a Senior Staff Software Engineer at Databricks and will be speaking about developer tooling at the Data + AI Summit 2023.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Pieter Noordhuis is a data scientist and researcher with a focus on machine learning and deep learning. He has worked on a variety of projects in fields such as healthcare, finance, and energy. At the Data + AI Summit 2023, he will be speaking about the potential and limitations of deep learning in various industries.'}]</td>\n",
       "      <td>2.475078</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results from predictor:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Ground Truth Answer</th>\n",
       "      <th>Model Prediction</th>\n",
       "      <th>Inference Latency (seconds)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the Centro de demonstração de produtos e parcerias da Databricks and what services does it offer?</td>\n",
       "      <td>The Centro de demonstração de produtos e parcerias da Databricks is a platform that offers demonstrations of solution accelerators. It also offers services such as documentation, training and certification, resources, online community, alliance with universities, events, and partnership programs for cloud partners and technology and data partners. Additionally, it provides solutions for partners to connect with validated partner solutions.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The Centro de demonstração de produtos e parcerias da Databricks is a platform that allows users to test and evaluate Databricks products and services. It offers a range of features and services, including:\n",
       "\n",
       "1. Data Science Workspace: A cloud-based platform that provides a complete environment for data scientists to develop, test, and deploy machine learning models.\n",
       "\n",
       "2. Apache Spark: An open-source distributed computing engine that'}]</td>\n",
       "      <td>3.315304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of the Data and AI Summit 2023?</td>\n",
       "      <td>The purpose of the Data and AI Summit 2023 is to bring together experts, researchers, contributors, and professionals from the global data community to understand the potential of Large Language Models (LLMs) and shape the future of their industries with data and AI. Attendees can expect to hear from top speakers in the field, including those from Databricks and across the data and AI community, and learn about building, training, and deploying LLMs, as well as other relevant topics in data and AI.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The purpose of the Data and AI Summit 2023 is to bring together industry leaders, experts, and professionals to discuss the latest trends, innovations, and challenges in the field of data and artificial intelligence. The summit aims to provide a platform for knowledge sharing, networking, and collaboration, and to inspire attendees to drive the adoption and implementation of data and AI technologies in their organizations.'}]</td>\n",
       "      <td>2.876742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What AWS competencies did Databricks showcase at AWS re:Invent?</td>\n",
       "      <td>Databricks showcased AWS competencies in data engineering, data warehousing, data streaming, and machine learning at AWS re:Invent.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "At AWS re:Invent, Databricks showcased several AWS competencies, including:\n",
       "\n",
       "1. Big Data: Databricks is a leading big data platform that leverages AWS services such as Amazon S3, Amazon EMR, and Amazon Redshift to process and analyze large volumes of data.\n",
       "2. Machine Learning: Databricks provides a comprehensive machine learning platform that integrates with AWS services such as Amazon SageMaker, Amazon K'}]</td>\n",
       "      <td>3.294858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is the date and time of the upcoming webinar about transitioning from data warehouse to data lakehouse?</td>\n",
       "      <td>The upcoming webinar about transitioning from data warehouse to data lakehouse is on May 18 at 8 AM PT.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The upcoming webinar about transitioning from data warehouse to data lakehouse is on [insert date and time here].'}]</td>\n",
       "      <td>0.864549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What are Databricks Solution Accelerators and how do they help deliver data and AI value faster?</td>\n",
       "      <td>Databricks Solution Accelerators are tools developed by Databricks that help deliver data and AI value faster. They save time in the discovery, design, development, and testing phases by providing pre-built solutions for common use cases in industries such as Financial Services, Healthcare, Manufacturing, and more. These accelerators allow organizations to quickly implement data and AI solutions, enabling them to achieve their business objectives faster.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Databricks Solution Accelerators are pre-built, reusable components that help organizations deliver data and AI value faster. These accelerators are designed to address common data and AI challenges and provide a quick and easy way to get started with Databricks.\n",
       "\n",
       "There are several types of Solution Accelerators available, including:\n",
       "\n",
       "1. Data Pipelines: These accelerators provide pre-built data pipelines that can be used to move data between different'}]</td>\n",
       "      <td>3.295434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>What is the purpose of Databricks Notebooks and how can it benefit different teams in an organization?</td>\n",
       "      <td>The purpose of Databricks Notebooks is to provide a collaborative workspace for teams working on data science, engineering, and machine learning projects. It allows teams to work with familiar languages and tools, use built-in data visualizations, and have automatic versioning within the notebooks. Databricks Notebooks can benefit different teams in an organization by providing a centralized platform for data collaboration, improving productivity, and promoting efficient communication between different teams. It can also help with data analysis, exploration, and modeling, leading to faster and more accurate insights.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Databricks Notebooks is a powerful tool that allows users to create, share, and collaborate on data science projects. It provides a user-friendly interface for creating and executing code, visualizations, and other data-related tasks.\n",
       "\n",
       "The purpose of Databricks Notebooks is to provide a centralized platform for data scientists, analysts, and other teams to work together on data-related projects. It enables teams to easily share code, data'}]</td>\n",
       "      <td>3.301518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>What is the Databricks Lakehouse Platform and how does it fit within a modern data stack?</td>\n",
       "      <td>The Databricks Lakehouse Platform is a data management platform that encompasses a wide range of data technologies, including Delta Lake, data governance, data engineering, data streaming, data warehousing, data sharing, machine learning, and data science. As a \"lakehouse,\" it provides a unified, scalable, and secure platform for managing and processing both structured and unstructured data, bridging the gap between traditional data warehouses and data lakes. It fits within a modern data stack by providing a comprehensive solution for data management, processing, and analysis, and can be integrated with other technologies and tools as needed to meet specific business needs.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "The Databricks Lakehouse Platform is a cloud-based data platform that provides a unified environment for data storage, processing, and analytics. It is designed to be a centralized hub for all data-related activities, including data ingestion, ETL, data warehousing, machine learning, and data governance.\n",
       "\n",
       "The Lakehouse Platform is built on top of Apache Spark, an open-source data processing engine that is widely used for big data processing'}]</td>\n",
       "      <td>3.293112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Who is Ruifeng Zheng and what is his role at Databricks?</td>\n",
       "      <td>Ruifeng Zheng is a Senior Software Engineer at Databricks and an Apache Spark committer. He works on various modules in Apache Spark including Spark Connect, Pandas API on Spark, PySpark, MLlib, Spark SQL, SparkR, etc. Prior to Databricks, he worked on applied machine learning for over 10 years. He holds a Master degree in Electronics Engineering from Peking University.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Ruifeng Zheng is a Senior Manager of Product Marketing at Databricks. In this role, he leads the product marketing team and is responsible for developing and executing the product marketing strategy for Databricks' cloud-based big data platform. He works closely with the product development team to understand customer needs and develops marketing materials that effectively communicate the value of Databricks' platform to potential customers.'}]</td>\n",
       "      <td>2.965049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>What are the benefits of using Databricks and how can I try it for free?</td>\n",
       "      <td>The benefits of using Databricks include simplifying data ingestion, automating ETL processes, collaborating in multiple programming languages such as Python, R, Scala, and SQL, and a price performance up to 12x better than cloud data warehouses. To try Databricks for free, one can sign up for a 14-day trial on their AWS, Microsoft Azure, or Google Cloud platform. To do so, visit the Databricks website and fill out the registration form with your name, email, company, and country.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Databricks is a cloud-based data science platform that provides a unified environment for data engineering, machine learning, and analytics. It offers several benefits, including:\n",
       "\n",
       "1. Scalability: Databricks can handle large-scale data processing and analytics tasks with ease. It can scale up or down based on the workload, ensuring that you only pay for what you use.\n",
       "\n",
       "2. Integration: Databricks integrates with a wide range'}]</td>\n",
       "      <td>3.295256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Who is Pieter Noordhuis and what topic will he be speaking about at the Data + AI Summit 2023?</td>\n",
       "      <td>Pieter Noordhuis is a Senior Staff Software Engineer at Databricks and will be speaking about developer tooling at the Data + AI Summit 2023.</td>\n",
       "      <td>[{'generated_text': '\n",
       "\n",
       "Pieter Noordhuis is a data scientist and researcher with a focus on machine learning and deep learning. He has worked on a variety of projects in fields such as healthcare, finance, and energy. At the Data + AI Summit 2023, he will be speaking about the potential and limitations of deep learning in various industries.'}]</td>\n",
       "      <td>2.477939</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import time\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "# Assuming test_dataset is already defined\n",
    "test_dataset = train_and_test_dataset[\"test\"]\n",
    "\n",
    "def predict_and_print(datapoint, predictor):\n",
    "    payload = {\n",
    "        \"inputs\": datapoint[\"question\"],\n",
    "        \"parameters\": {\"max_new_tokens\": 100},\n",
    "    }\n",
    "\n",
    "    start_time = time.time()\n",
    "    response = predictor.predict(payload)\n",
    "    end_time = time.time()\n",
    "\n",
    "    latency = end_time - start_time\n",
    "    response_text = response[\"generated_text\"] if isinstance(response, dict) else response\n",
    "\n",
    "    return {\n",
    "        \"Question\": datapoint[\"question\"],\n",
    "        \"Ground Truth Answer\": datapoint[\"answer\"],\n",
    "        \"Model Prediction\": response_text,\n",
    "        \"Inference Latency (seconds)\": latency\n",
    "    }\n",
    "\n",
    "results_new = []\n",
    "results_old = []\n",
    "\n",
    "try:\n",
    "    for i, datapoint in enumerate(test_dataset.select(range(10))):\n",
    "        result_new = predict_and_print(datapoint, predictor_new)\n",
    "        result_old = predict_and_print(datapoint, predictor)\n",
    "        results_new.append(result_new)\n",
    "        results_old.append(result_old)\n",
    "\n",
    "    df_new = pd.DataFrame(results_new)\n",
    "    df_old = pd.DataFrame(results_old)\n",
    "\n",
    "    print(\"Results from predictor_new:\")\n",
    "    display(HTML(df_new.to_html()))\n",
    "\n",
    "    print(\"Results from predictor:\")\n",
    "    display(HTML(df_old.to_html()))\n",
    "\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9feee71-4c5b-4570-9363-4d3b8d4033ad",
   "metadata": {},
   "source": [
    "## Model evaluation with OS eval frameworks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7b69436b-4212-4313-a6a3-5b27afc74acf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting uptrain\n",
      "  Downloading uptrain-0.4.8-py3-none-any.whl.metadata (19 kB)\n",
      "Collecting pydantic<1.10.10 (from uptrain)\n",
      "  Downloading pydantic-1.10.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (147 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.8/147.8 kB\u001b[0m \u001b[31m111.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting loguru (from uptrain)\n",
      "  Downloading loguru-0.7.2-py3-none-any.whl.metadata (23 kB)\n",
      "Collecting lazy-loader (from uptrain)\n",
      "  Downloading lazy_loader-0.3-py3-none-any.whl.metadata (4.3 kB)\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from uptrain) (2.6.3)\n",
      "Collecting polars>=0.18 (from uptrain)\n",
      "  Downloading polars-0.20.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.8/site-packages (from uptrain) (1.3.4)\n",
      "Requirement already satisfied: numpy>=1.23.0 in /opt/conda/lib/python3.8/site-packages (from uptrain) (1.24.4)\n",
      "Collecting httpx>=0.24.1 (from uptrain)\n",
      "  Downloading httpx-0.26.0-py3-none-any.whl.metadata (7.6 kB)\n",
      "Requirement already satisfied: plotly>=5.0.0 in /opt/conda/lib/python3.8/site-packages (from uptrain) (5.18.0)\n",
      "Requirement already satisfied: anyio in /opt/conda/lib/python3.8/site-packages (from httpx>=0.24.1->uptrain) (4.1.0)\n",
      "Requirement already satisfied: certifi in /opt/conda/lib/python3.8/site-packages (from httpx>=0.24.1->uptrain) (2023.11.17)\n",
      "Collecting httpcore==1.* (from httpx>=0.24.1->uptrain)\n",
      "  Downloading httpcore-1.0.2-py3-none-any.whl.metadata (20 kB)\n",
      "Requirement already satisfied: idna in /opt/conda/lib/python3.8/site-packages (from httpx>=0.24.1->uptrain) (3.2)\n",
      "Requirement already satisfied: sniffio in /opt/conda/lib/python3.8/site-packages (from httpx>=0.24.1->uptrain) (1.2.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /opt/conda/lib/python3.8/site-packages (from httpcore==1.*->httpx>=0.24.1->uptrain) (0.14.0)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in /opt/conda/lib/python3.8/site-packages (from plotly>=5.0.0->uptrain) (8.2.3)\n",
      "Requirement already satisfied: packaging in /opt/conda/lib/python3.8/site-packages (from plotly>=5.0.0->uptrain) (21.0)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in /opt/conda/lib/python3.8/site-packages (from pydantic<1.10.10->uptrain) (4.8.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.8/site-packages (from pandas->uptrain) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas->uptrain) (2021.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas->uptrain) (1.16.0)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in /opt/conda/lib/python3.8/site-packages (from anyio->httpx>=0.24.1->uptrain) (1.2.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging->plotly>=5.0.0->uptrain) (3.0.4)\n",
      "Downloading uptrain-0.4.8-py3-none-any.whl (228 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m228.8/228.8 kB\u001b[0m \u001b[31m210.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading httpx-0.26.0-py3-none-any.whl (75 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.9/75.9 kB\u001b[0m \u001b[31m225.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m193.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading polars-0.20.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m28.4/28.4 MB\u001b[0m \u001b[31m243.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading pydantic-1.10.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.2/3.2 MB\u001b[0m \u001b[31m234.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading lazy_loader-0.3-py3-none-any.whl (9.1 kB)\n",
      "Downloading loguru-0.7.2-py3-none-any.whl (62 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.5/62.5 kB\u001b[0m \u001b[31m205.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0mInstalling collected packages: pydantic, polars, loguru, lazy-loader, httpcore, httpx, uptrain\n",
      "  Attempting uninstall: pydantic\n",
      "    Found existing installation: pydantic 1.10.13\n",
      "    Uninstalling pydantic-1.10.13:\n",
      "      Successfully uninstalled pydantic-1.10.13\n",
      "Successfully installed httpcore-1.0.2 httpx-0.26.0 lazy-loader-0.3 loguru-0.7.2 polars-0.20.2 pydantic-1.10.9 uptrain-0.4.8\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install uptrain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f1b8f0d-edb7-4a8b-8eb0-092637b46099",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6141f43d",
   "metadata": {},
   "source": [
    "## Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5553726e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# predictor.delete_model()\n",
    "# predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fec22b92-4246-4194-8d62-f7c3f2608ceb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}