2_save_model-s3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save the Model and the Evaluation Data\n",
    "\n",
    "To save this model and the evaluation data so that you can use it from various locations, including other notebooks or the model server, upload it to s3-compatible storage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Install the required packages and define a function for the upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: boto3 in /opt/conda/lib/python3.10/site-packages (1.35.59)\n",
      "Requirement already satisfied: botocore in /opt/conda/lib/python3.10/site-packages (1.35.59)\n",
      "Requirement already satisfied: python-dotenv in /opt/conda/lib/python3.10/site-packages (1.0.1)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from boto3) (1.0.1)\n",
      "Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /opt/conda/lib/python3.10/site-packages (from boto3) (0.10.3)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.10/site-packages (from botocore) (2.9.0.post0)\n",
      "Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /opt/conda/lib/python3.10/site-packages (from botocore) (1.26.20)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore) (1.16.0)\n",
      "Name: boto3\n",
      "Version: 1.35.59\n",
      "Summary: The AWS SDK for Python\n",
      "Home-page: https://github.com/boto/boto3\n",
      "Author: Amazon Web Services\n",
      "Author-email: \n",
      "License: Apache License 2.0\n",
      "Location: /opt/conda/lib/python3.10/site-packages\n",
      "Requires: botocore, jmespath, s3transfer\n",
      "Required-by: \n",
      "---\n",
      "Name: botocore\n",
      "Version: 1.35.59\n",
      "Summary: Low-level, data-driven core of boto 3.\n",
      "Home-page: https://github.com/boto/botocore\n",
      "Author: Amazon Web Services\n",
      "Author-email: \n",
      "License: Apache License 2.0\n",
      "Location: /opt/conda/lib/python3.10/site-packages\n",
      "Requires: jmespath, python-dateutil, urllib3\n",
      "Required-by: boto3, s3transfer\n",
      "---\n",
      "Name: python-dotenv\n",
      "Version: 1.0.1\n",
      "Summary: Read key-value pairs from a .env file and set them as environment variables\n",
      "Home-page: https://github.com/theskumar/python-dotenv\n",
      "Author: Saurabh Kumar\n",
      "Author-email: me+github@saurabh-kumar.com\n",
      "License: BSD-3-Clause\n",
      "Location: /opt/conda/lib/python3.10/site-packages\n",
      "Requires: \n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip install boto3==1.35.59 botocore==1.35.59 python-dotenv==1.0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import botocore\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Load the .env file\n",
    "load_dotenv()\n",
    "\n",
    "aws_access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')\n",
    "aws_secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')\n",
    "region_name = os.environ.get('AWS_REGION')\n",
    "bucket_name = os.environ.get('AWS_S3_BUCKET')\n",
    "\n",
    "if not all([aws_access_key_id, aws_secret_access_key, region_name, bucket_name]):\n",
    "    raise ValueError(\"One or more data connection variables are empty.  \"\n",
    "                     \"Please check your data connection to an S3 bucket.\")\n",
    "\n",
    "session = boto3.session.Session(aws_access_key_id=aws_access_key_id,\n",
    "                                aws_secret_access_key=aws_secret_access_key,\n",
    "                                region_name=region_name)\n",
    "\n",
    "s3_resource = session.resource(\n",
    "    's3',\n",
    "    config=botocore.client.Config(signature_version='s3v4')\n",
    ")\n",
    "\n",
    "bucket = s3_resource.Bucket(bucket_name)\n",
    "\n",
    "\n",
    "def upload_file_to_s3(local_file, s3_key):\n",
    "    print(f\"{local_file} -> {s3_key}\")\n",
    "    bucket.upload_file(local_file, s3_key)\n",
    "\n",
    "def upload_directory_to_s3(local_directory, s3_prefix):\n",
    "    num_files = 0\n",
    "    for root, dirs, files in os.walk(local_directory):\n",
    "        for filename in files:\n",
    "            file_path = os.path.join(root, filename)\n",
    "            relative_path = os.path.relpath(file_path, local_directory)\n",
    "            s3_key = os.path.join(s3_prefix, relative_path)\n",
    "            print(f\"{file_path} -> {s3_key}\")\n",
    "            bucket.upload_file(file_path, s3_key)\n",
    "            num_files += 1\n",
    "    return num_files\n",
    "\n",
    "\n",
    "def list_objects(prefix):\n",
    "    filter = bucket.objects.filter(Prefix=prefix)\n",
    "    for obj in filter.all():\n",
    "        print(obj.key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Verify the upload\n",
    "\n",
    "In your S3 bucket, under the `models` upload prefix, run the `list_object` command. As best practice, to avoid mixing up model files, keep only one model and its required files in a given prefix or directory. This practice allows you to download and serve a directory with all the files that a model requires. \n",
    "\n",
    "If this is the first time running the code, this cell will have no output.\n",
    "\n",
    "If you've already uploaded your model, you should see this output: `models/fraud/1/model.onnx`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_objects(\"models\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you've already uploaded your model, you should see this output: `scaler.pkl` and `test_data.pkl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_objects(\"artifact\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Upload and check again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compress files: models/fraud/1/model.onnx, artifact/scaler.pkl and artifact/scaler.pkl using python's zipfile module\n",
    "import zipfile\n",
    "import os\n",
    "\n",
    "# Define the files to be compressed\n",
    "files_to_compress = [\n",
    "    \"models/fraud/1/model.onnx\",\n",
    "    \"artifact/scaler.pkl\",\n",
    "    \"artifact/test_data.pkl\"\n",
    "]\n",
    "\n",
    "# Define the name of the compressed file\n",
    "compressed_file_name = \"models/evaluation_kit.zip\"\n",
    "\n",
    "# Create a zip file and add the files to it\n",
    "with zipfile.ZipFile(compressed_file_name, 'w') as zipf:\n",
    "    for file in files_to_compress:\n",
    "        zipf.write(file)\n",
    "\n",
    "# Verify the compressed file\n",
    "if os.path.exists(compressed_file_name):\n",
    "    print(f\"Files compressed successfully. Compressed file: {compressed_file_name}\")\n",
    "    # Upload the compressed file to S3\n",
    "    upload_file_to_s3(compressed_file_name, compressed_file_name)\n",
    "else:\n",
    "    print(\"Failed to compress files.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the function to upload the `models` and `artifact` folders in a rescursive fashion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_models_directory = \"models\"\n",
    "\n",
    "if not os.path.isdir(local_models_directory):\n",
    "    raise ValueError(f\"The directory '{local_models_directory}' does not exist.  \"\n",
    "                     \"Did you finish training the model in the previous notebook?\")\n",
    "\n",
    "num_files = upload_directory_to_s3(\"models\", \"models\")\n",
    "\n",
    "if num_files == 0:\n",
    "    raise ValueError(\"No files uploaded.  Did you finish training and \"\n",
    "                     \"saving the model to the \\\"models\\\" directory?  \"\n",
    "                     \"Check for \\\"models/fraud/1/model.onnx\\\"\")\n",
    "\n",
    "local_artifacts_directory = \"artifact\"\n",
    "\n",
    "if not os.path.isdir(local_artifacts_directory):\n",
    "    raise ValueError(f\"The directory '{local_artifacts_directory}' does not exist.  \"\n",
    "                     \"Did you finish training the model in the previous notebook?\")\n",
    "\n",
    "num_files = upload_directory_to_s3(local_artifacts_directory, local_artifacts_directory)\n",
    "\n",
    "if num_files == 0:\n",
    "    raise ValueError(\"No files uploaded.  Did you finish training and \"\n",
    "                     \"saving the model to the \\\"artifacts\\\" directory?\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "To confirm this worked, run the `list_objects` function again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_objects(\"models\")\n",
    "list_objects(\"artifact\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Step\n",
    "\n",
    "Now that you've saved the model to s3 storage, you can refer to the model by using the same data connection to serve the model as an API.\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "environment": {
   "kernel": "conda-base-py",
   "name": "workbench-notebooks.m125",
   "type": "gcloud",
   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m125"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel) (Local)",
   "language": "python",
   "name": "conda-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}