sample_type_creation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "sample_type_creation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/seek4science/stress-testing/blob/main/sample_type_creation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install num2words\n",
        "import num2words"
      ],
      "metadata": {
        "id": "eE7L3Dvru5hX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHED0l52PClp"
      },
      "source": [
        "Import the libraries so that they can be used within the notebook\n",
        "\n",
        "* **requests** is used to make HTTP calls\n",
        "* **json** is used to encode and decode strings into JSON\n",
        "* **string** is used to perform text manipulation and checking\n",
        "* **getpass** is used to do non-echoing password input"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5EGmpx3Id7py"
      },
      "source": [
        "import requests\n",
        "import json\n",
        "import string\n",
        "import getpass"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9t7T9pmQJsn"
      },
      "source": [
        "The **base_url** holds the URL to the SEEK instance that will be used in the notebook\n",
        "\n",
        "**headers** holds the HTTP headers that will be sent with every HTTP call\n",
        "\n",
        "* **Content-type: application/vnd.api+json** - indicates that any data sent will be in JSON API format\n",
        "* **Accept: application/vnd.api+json** - indicates that the notebook expects any data returned to be in JSON API format\n",
        "* **Accept-Charset: ISO-8859-1** - indicates that the notebook expects any text returned to be in ISO-8859-1 character set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QKepIS_xd7qL"
      },
      "source": [
        "base_url = 'https://sandbox10.fairdomhub.org/'\n",
        "\n",
        "headers = {\"Content-type\": \"application/vnd.api+json\",\n",
        "           \"Accept\": \"application/vnd.api+json\",\n",
        "           \"Accept-Charset\": \"ISO-8859-1\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0U3A7RNQJsu"
      },
      "source": [
        "Create a **requests** HTTP **Session**. A **Session** has re-usable settings such as **headers**\n",
        "\n",
        "The **authorization** is username and password. The user is prompted for this information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UKqc5Sr6d7qS"
      },
      "source": [
        "session = requests.Session()\n",
        "session.headers.update(headers)\n",
        "session.auth = (input('Username: '), getpass.getpass('Password: '))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0xz097gufmP3"
      },
      "source": [
        "The sample types will be created within **Project** 3"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CPcirVGad7qa"
      },
      "source": [
        "containing_project_id = 3\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x = '''{\n",
        "  \"data\": {\n",
        "    \"type\": \"sample_types\",\n",
        "    \"attributes\": {\n",
        "      \"title\": \"Two columns API\",\n",
        "      \"description\": \"\",\n",
        "      \"sample_attributes\": [\n",
        "        {\n",
        "          \"title\": \"title\",\n",
        "          \"description\": \"\",\n",
        "          \"pid\": \"\",\n",
        "          \"sample_attribute_type\": {\n",
        "            \"id\": 8,\n",
        "            \"title\": \"String\",\n",
        "            \"base_type\": \"String\",\n",
        "            \"regexp\": \".*\"\n",
        "          },\n",
        "          \"required\": true,\n",
        "          \"pos\": \"1\",\n",
        "          \"unit\": null,\n",
        "          \"is_title\": true,\n",
        "          \"sample_controlled_vocab_id\": null,\n",
        "          \"linked_sample_type_id\": null\n",
        "        }\n",
        "      ],\n",
        "      \"tags\": []\n",
        "    },\n",
        "    \"relationships\": {\n",
        "      \"projects\": {\n",
        "        \"data\": [\n",
        "          {\n",
        "            \"id\": \"2\",\n",
        "            \"type\": \"projects\"\n",
        "          }\n",
        "        ]\n",
        "      },\n",
        "      \"submitter\": {\n",
        "        \"data\": [\n",
        "          {\n",
        "            \"id\": \"2\",\n",
        "            \"type\": \"people\"\n",
        "          }\n",
        "        ]\n",
        "      }\n",
        "    }\n",
        "  }\n",
        "}'''"
      ],
      "metadata": {
        "id": "weF1pHBhgCnb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sample_data = {\n",
        "    \"data\": {\n",
        "        \"type\": \"samples\",\n",
        "        \"attributes\": {\n",
        "            \"title\": \"t\",\n",
        "            \"attribute_map\": {\n",
        "                \"title\": \"t\"\n",
        "            }\n",
        "        },\n",
        "        \"relationships\": {\n",
        "            \"creators\": {\n",
        "                \"data\": [\n",
        "                    {\n",
        "                        \"id\": \"2\",\n",
        "                        \"type\": \"people\"\n",
        "                    }\n",
        "                ]\n",
        "            },\n",
        "            \"submitter\": {\n",
        "                \"data\": [\n",
        "                    {\n",
        "                        \"id\": \"2\",\n",
        "                        \"type\": \"people\"\n",
        "                    }\n",
        "                ]\n",
        "            },\n",
        "            \"projects\": {\n",
        "                \"data\": [\n",
        "                    {\n",
        "                        \"id\": \"2\",\n",
        "                        \"type\": \"projects\"\n",
        "                    }\n",
        "                ]\n",
        "            },\n",
        "            \"sample_type\": {\n",
        "                \"data\": {\n",
        "                    \"id\": \"st\",\n",
        "                    \"type\": \"sample_types\"\n",
        "                }\n",
        "            },\n",
        "            \"people\": {\n",
        "                \"data\": [\n",
        "                    {\n",
        "                        \"id\": \"2\",\n",
        "                        \"type\": \"people\"\n",
        "                    }\n",
        "                ]\n",
        "            }\n",
        "        }\n",
        "    }\n",
        "}"
      ],
      "metadata": {
        "id": "kmcA_pLvxpRF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from num2words import num2words\n"
      ],
      "metadata": {
        "id": "7sk6GleAmQm8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def create_sampletype(column_count) :\n",
        "  x2 = json.loads(x)\n",
        "  x2['data']['attributes']['title'] = num2words(column_count).capitalize() + ' columns API'\n",
        "\n",
        "  for i in range(1, column_count):\n",
        "    to_add =  {}\n",
        "    to_add['title'] = 'attr_' + str(i)\n",
        "    to_add['sample_attribute_type'] = {}\n",
        "    to_add['sample_attribute_type']['id'] = '8'\n",
        "    to_add['required'] = 'false'\n",
        "         \n",
        "    x2['data']['attributes']['sample_attributes'].append(to_add)\n",
        "  return(x2)"
      ],
      "metadata": {
        "id": "Et4k9B_LqHzg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from pprint import pprint"
      ],
      "metadata": {
        "id": "gxghQSKnpqY6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import copy\n"
      ],
      "metadata": {
        "id": "TpKDINiU73gK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def handle_sampletype(column_count, sample_count):\n",
        "  r = session.post(base_url + 'sample_types', json=create_sampletype(column_count))\n",
        "  r.raise_for_status()\n",
        "  j = r.json()\n",
        "  sampletype_id = j['data']['id']\n",
        "\n",
        "  singletons = []\n",
        "  for i in range(1, sample_count + 1):\n",
        "    s = sample_data.copy()\n",
        "    s['data']['attributes']['title'] = 't_' + str(i)\n",
        "    s['data']['attributes']['attribute_map']['title'] = 't_' + str(i)\n",
        "    s['data']['relationships']['sample_type']['data']['id'] = sampletype_id\n",
        "\n",
        "    for c in range(1, column_count):\n",
        "      s['data']['attributes']['attribute_map']['attr_' + str(c)] = 'v_' + str(c) + '_' + str(i)\n",
        "    \n",
        "    singletons.append (copy.deepcopy(s))\n",
        "\n",
        "  return (copy.deepcopy(singletons))"
      ],
      "metadata": {
        "id": "qFMxobBQhUn6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def post_samples(posts):\n",
        "  for s in posts:\n",
        "    r = session.post(base_url + 'samples', json=s)\n",
        "    r.raise_for_status()"
      ],
      "metadata": {
        "id": "9z0AL1yCNXY5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def post_batch_samples(post):\n",
        "  r = session.post(base_url + 'samples/batch_create', json=post)\n",
        "  r.raise_for_status()"
      ],
      "metadata": {
        "id": "rEeuBmj6wG_E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%time singletons = handle_sampletype(10, 2000)"
      ],
      "metadata": {
        "id": "rK--Jo9pr9Lc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%time post_samples(singletons)"
      ],
      "metadata": {
        "id": "ZAdgB05PNuke"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "batch_samples = {'data' : handle_sampletype(10,2000)}"
      ],
      "metadata": {
        "id": "mF2TkV_uphZA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The post_batch_samples is expected to fail with a Bad Gateway error. This is due to a timeout in the communication with the server."
      ],
      "metadata": {
        "id": "JsKyIiWzqNyu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%time post_batch_samples(batch_samples)"
      ],
      "metadata": {
        "id": "JoNXSCrfRA4Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "handle_sampletype(100, 1000)"
      ],
      "metadata": {
        "id": "-rZzp52V5jfu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def search_samples (s):\n",
        "  r = session.get(base_url + 'search?search_type=samples&q=' + s)\n",
        "  r.raise_for_status()"
      ],
      "metadata": {
        "id": "utiYjoXpKYsZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%time search_samples('v_6_')"
      ],
      "metadata": {
        "id": "wzl7Wn6MLQVx"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}