llama-2-hf-tgi/llama-2-70b-chat-hf/1-deploy-llama-2-70b-chat-hf-tgi-sagemaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6926777f-823c-4c78-8222-c2c7e9ba3037",
   "metadata": {},
   "source": [
    "# Deploy Llama 2 70B Chat HF TGI via Amazon SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5da7cdf2-6bf2-458b-9d9b-b42098ae08dc",
   "metadata": {},
   "source": [
    "Llama 2 is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. The fine-tuned LLMs, called Llama-2-Chat, are optimized for dialogue use cases. Llama 2 outperforms other open source language models on many external benchmarks, including reasoning, coding, proficiency, and knowledge tests.\n",
    "\n",
    "\n",
    "Amazon Web Services (AWS) and Hugging Face have released a new Hugging Face Deep Learning Container (DLC) for inference with Large Language Models (LLMs). This new DLC is powered by Text Generation Inference (TGI), an open source solution for deploying and serving LLMs. TGI enables high-performance text generation using Tensor Parallelism and dynamic batching for popular LLMs like StarCoder, BLOOM, GPT-NeoX, StableLM, Llama, and T5.\n",
    "\n",
    "In this notebook, we will use Amazon SageMaker Hosting instances Real-Time Inference to deploy `meta-llama/Llama-2-70b-chat-hf` model powered by Text Generation Inference (TGI) on Hugging Face LLM DLC."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7b4827-b257-41ba-b454-74cd67cdfe56",
   "metadata": {},
   "source": [
    "## Prerequisite\n",
    "\n",
    "### Hugging Face Account\n",
    "\n",
    "You need to have Hugging Face account. Sign Up here https://huggingface.co/join with your email if you do not already have account.\n",
    "\n",
    "- For seamless access of the models avaialble on Hugging Face especially gated models such as Llama, for fine-tuning and inferencing purposes, you need to have Hugging Face Account to obtain read Access Token.\n",
    "- After signup, login to visit https://huggingface.co/settings/tokens to create read Access token.\n",
    "\n",
    "### Request access to the next version of Llama\n",
    "\n",
    "Use the same email id to obtain permission from meta by visiting this link: https://ai.meta.com/resources/models-and-libraries/llama-downloads/\n",
    "\n",
    "- The Llama models available via Hugging Face are gated models. The use of Llama model is governed by the Meta license. In order to download the model weights and tokenizer, please visit https://ai.meta.com/resources/models-and-libraries/llama-downloads/ and accept their License before requesting access.\n",
    "- Within 2 days you might be granted access to use Llama models via a confirmation email with subject: **[Access granted] Your request to access model meta-llama/Llama-2-70b-chat-hf has been accepted.** Though the model id is Llama-2-70b-chat-hf, you should be able to access other variants too.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec6d468-ca22-4958-ba46-723001f17178",
   "metadata": {},
   "source": [
    "## Setup Development Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011f0f1e-49cd-4220-9b3f-c9dfb353ca33",
   "metadata": {},
   "source": [
    "Upgrade `pip` and install the latest version of `sagemaker` and `boto3` packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a710943d-f305-4fea-b3d0-1e90c896748d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -Uq pip\n",
    "!pip install -Uq boto3 sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a002800-ec13-467c-9ccf-eeedd7060555",
   "metadata": {},
   "source": [
    "Obtain the sagemaker execution role. We need it to create `HuggingFaceModel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "655f2ec4-f2b8-48b1-8c6b-85d823bc909a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except ValueError:\n",
    "    iam = boto3.client('iam')\n",
    "    role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "589b1b23-3a09-418e-b185-5d836ece61f4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Get the latest Hugging Face LLM Container Image URI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e72556-eb2f-4e61-a2d6-339d49c5892c",
   "metadata": {},
   "source": [
    "Obtain the latest Hugging Face LLM DLC powered by Text Generation Inference (TGI) available on SageMaker. We will use this image to deploy `meta-llama/Llama-2-70b-chat-hf` model on SageMaker, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75426efc-7dc5-4dba-8896-413fd894c556",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import get_huggingface_llm_image_uri\n",
    "\n",
    "# retrieve the llm image uri\n",
    "llm_image = get_huggingface_llm_image_uri(\n",
    "  \"huggingface\",\n",
    "  version=\"2.0.2\"\n",
    ")\n",
    "\n",
    "# print ecr image uri\n",
    "print(f\"llm image uri: {llm_image}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b21f40a-ee3a-40d4-aef1-6f890d86fd5d",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Llama 2 70B Chat HuggingFace Model\n",
    "\n",
    "Here is the default recommendation of sagemaker instances when deploying Llama 2 models for inferencing purposes.\n",
    "\n",
    "| Model Name | Model ID | Minimum Recommended Instance for inference |\n",
    "|---|---|---|\n",
    "| Llama-2-7b | meta-llama/Llama-2-7b | ml.g5.2xlarge |\n",
    "| Llama-2-7b-hf | meta-llama/Llama-2-7b-hf | ml.g5.2xlarge |\n",
    "| Llama-2-7b-chat | meta-llama/Llama-2-7b-chat-hf | ml.g5.2xlarge |\n",
    "| Llama-2-13b | meta-llama/Llama-2-13b | ml.g5.12xlarge |\n",
    "| Llama-2-13b-hf | meta-llama/Llama-2-13b-hf | ml.g5.12xlarge |\n",
    "| Llama-2-13b-chat | meta-llama/Llama-2-13b-chat-hf | ml.g5.12xlarge |\n",
    "| Llama-2-70b | meta-llama/Llama-2-70b | ml.g5.48xlarge |\n",
    "| Llama-2-70b-hf | meta-llama/Llama-2-70b-hf | ml.g5.48xlarge |\n",
    "| Llama-2-70b-chat | meta-llama/Llama-2-70b-chat-hf | ml.g5.48xlarge |\n",
    "\n",
    "Reference: [Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart](https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/)\n",
    "\n",
    "We will proceed with deploying `meta-llama/Llama-2-70b-chat-hf` model on `ml.g5.48xlarge`. Also notice that the config for `SM_NUM_GPUS` is 8 for `meta-llama/Llama-2-70b-chat-hf` model.\n",
    "\n",
    "https://huggingface.co/docs/text-generation-inference/main/en/basic_tutorials/launcher#maxbatchtotaltokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f40aeb7e-d242-43ba-865b-46f4099d4ab6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "\n",
    "# sagemaker config\n",
    "instance_type = \"ml.g5.48xlarge\"\n",
    "number_of_gpu = 8\n",
    "health_check_timeout = 300\n",
    "\n",
    "# Define Model and Endpoint configuration parameters\n",
    "config = {\n",
    "    'HF_MODEL_ID': \"meta-llama/Llama-2-70b-chat-hf\", # model_id from hf.co/models\n",
    "    'SM_NUM_GPUS': json.dumps(number_of_gpu), # Number of GPU used per replica\n",
    "    'MAX_INPUT_LENGTH': json.dumps(2048),  # Max length of input text\n",
    "    'MAX_TOTAL_TOKENS': json.dumps(4096),  # Max length of the generation (including input text)\n",
    "    'MAX_BATCH_TOTAL_TOKENS': json.dumps(32768),  # Limits the number of tokens that can be processed in parallel during the generation\n",
    "    'HUGGING_FACE_HUB_TOKEN': \"<YOUR_HUGGING_FACE_READ_ACCESS_TOKEN>\" # Read Access token of your HuggingFace profile https://huggingface.co/settings/tokens\n",
    "}\n",
    "\n",
    "# create HuggingFaceModel with the image uri\n",
    "llm_model = HuggingFaceModel(\n",
    "  role=role,\n",
    "  image_uri=llm_image,\n",
    "  env=config\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "729f634f-1bbe-4f7e-86b2-61480837b2d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "endpoint_name = name_from_base(f\"{config['HF_MODEL_ID'].split('/')[1]}-tgi-streaming\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e9847e0-5cde-4630-a334-b13447df07f8",
   "metadata": {},
   "source": [
    "Deploy the model to the SageMaker Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cff1879-02bf-4840-a6e6-b9f6073b9cd6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "# https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy\n",
    "\n",
    "llm = llm_model.deploy(\n",
    "    endpoint_name=endpoint_name,\n",
    "    initial_instance_count=1,\n",
    "    instance_type=instance_type,\n",
    "    container_startup_health_check_timeout=health_check_timeout,\n",
    "    routing_config = {\n",
    "        \"RoutingStrategy\":  sagemaker.enums.RoutingStrategy.LEAST_OUTSTANDING_REQUESTS\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea9f120f-0db4-4883-af35-1037fa13097b",
   "metadata": {},
   "source": [
    "We will store the value of the variable `endpoint_name` to use it in inference notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71d53918-f35b-40a5-b2d9-71868f2ee32d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%store \\\n",
    "endpoint_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "080200a2-bd73-4973-a7d0-81eb4f90aee3",
   "metadata": {},
   "source": [
    "## References:\n",
    "1. [Announcing the launch of new Hugging Face LLM Inference containers on Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/)\n",
    "2. [Sagemaker Real-time Inference now supports response streaming](https://aws.amazon.com/about-aws/whats-new/2023/09/sagemaker-real-time-inference-response-streaming/)\n",
    "3. [Deep learning containers for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-dlc.html)\n",
    "4. [Hugging Face Gated Models](https://huggingface.co/docs/hub/models-gated)\n",
    "5. [Hugging Face TGI Open API specification](https://huggingface.github.io/text-generation-inference/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "807b9017-2761-4db5-b53a-13005a05f87e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}