CourseEra_Courses_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CourseEra Courses Analysis.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "LJbRF53245ds",
        "zRed5EPQ4tjV"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **This notebook is breakdown of required web scraper and analysis code used in project.**\n",
        "\n",
        "\n",
        "> Scrapy is used to crawl web page data."
      ],
      "metadata": {
        "id": "EtfJaGBW5xJl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install crochet scrapy"
      ],
      "metadata": {
        "id": "1kR1RZDrEDJp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import crochet\n",
        "crochet.setup()     # initialize crochet\n",
        "\n",
        "#All imports used in project\n",
        "import json\n",
        "import scrapy\n",
        "from scrapy.crawler import CrawlerRunner\n",
        "from scrapy.utils.log import configure_logging\n",
        "from scrapy.crawler import CrawlerProcess\n",
        "from scrapy.utils.project import get_project_settings\n",
        "#Intilazations\n",
        "crawl_runner = CrawlerRunner()      # requires the Twisted reactor to run                  # store quotes"
      ],
      "metadata": {
        "id": "sS8GHs0S8OGU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Spider to extract basic course details**"
      ],
      "metadata": {
        "id": "LJbRF53245ds"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "urls=['https://www.coursera.org/learn/introduction-to-data-analytics']\n",
        "global final_data\n",
        "final_data = {}"
      ],
      "metadata": {
        "id": "cydXOrOm-ZW4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JYMmEehHRV5T"
      },
      "outputs": [],
      "source": [
        "class MySpider2(scrapy.Spider):\n",
        "    name=\"course_details\"\n",
        "    def start_requests(self):\n",
        "      for url in urls:\n",
        "          yield scrapy.Request(url,callback=self.parse,meta={'dont_merge_cookies': True})\n",
        "            \n",
        "    def parse(self,response):\n",
        "        course_details={}\n",
        "        name= response.css('h1.banner-title::text').get()\n",
        "        instructor_rating= response.css('span.avg-instructor-rating__total span::text').get()\n",
        "        skills=response.css('div.Skills span::text').getall()\n",
        "        rating_div=response.css('div.XDPRating span::text').getall()\n",
        "        content_rating=rating_div[0]\n",
        "        no_of_ratings=rating_div[1]\n",
        "        domains=response.css('a.color-white.font-weight-bold::text').getall()\n",
        "        domain=domains[1:]\n",
        "        no_of_reviews=response.xpath('.//div/div/div/div/span/strong/span/text()').getall()[0]\n",
        "        time= response.css('div._16ni8zai.m-b-0.m-t-1s span::text').get()\n",
        "        org=response.css('h3.headline-4-text.bold.rc-Partner__title::text').get()\n",
        "        instructor=response.css('h3.instructor-name.headline-3-text.bold::text').get()\n",
        "        course_details['name']=name\n",
        "        course_details['instructor_rating']=instructor_rating\n",
        "        course_details['no_of_reviews']=no_of_reviews.split(\" \")[0]\n",
        "        course_details['skills']=skills\n",
        "        course_details['content_rating']=content_rating\n",
        "        course_details['no_of_ratings']=no_of_ratings.split(\" \")[0]\n",
        "        course_details['domain']=domain\n",
        "        course_details['time']=time.split(\" \")[1]\n",
        "        course_details['org']=org\n",
        "        final_data[name]=course_details\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "@crochet.run_in_reactor\n",
        "def scrape_with_crochet(spider):\n",
        "    eventual=crawl_runner.crawl(spider)\n",
        "    eventual.addCallback(finished_scrape)"
      ],
      "metadata": {
        "id": "69GCpZV_8YkB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def finished_scrape(final_data):\n",
        "  print(final_data)"
      ],
      "metadata": {
        "id": "tp-DNpBBjK-Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})\n",
        "scrape_with_crochet(MySpider2)"
      ],
      "metadata": {
        "id": "6Q0gJ1Sl-7kn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c6fad59f-577b-4fb2-86e8-44935ec2901c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:scrapy.crawler:Overridden settings:\n",
            "{}\n",
            "INFO: Overridden settings:\n",
            "{}\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<crochet._eventloop.EventualResult at 0x7fb32b0cae10>"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "final_data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3fjqhi-vi39A",
        "outputId": "ebfc7200-716d-4379-fa47-c87991de7faa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{}"
            ]
          },
          "metadata": {},
          "execution_count": 26
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:scrapy.extensions.telnet:Telnet Password: 8850aca59d03a31d\n",
            "2022-05-20 20:32:13 [scrapy.extensions.telnet] INFO: Telnet Password: 8850aca59d03a31d\n",
            "INFO:scrapy.middleware:Enabled extensions:\n",
            "['scrapy.extensions.corestats.CoreStats',\n",
            " 'scrapy.extensions.telnet.TelnetConsole',\n",
            " 'scrapy.extensions.memusage.MemoryUsage',\n",
            " 'scrapy.extensions.logstats.LogStats']\n",
            "2022-05-20 20:32:14 [scrapy.middleware] INFO: Enabled extensions:\n",
            "['scrapy.extensions.corestats.CoreStats',\n",
            " 'scrapy.extensions.telnet.TelnetConsole',\n",
            " 'scrapy.extensions.memusage.MemoryUsage',\n",
            " 'scrapy.extensions.logstats.LogStats']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Spider to extract course reviews**\n",
        "\n",
        "\n",
        "> Will store it in a dataframe for this demostration\n",
        "\n"
      ],
      "metadata": {
        "id": "zRed5EPQ4tjV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# review_urls=['https://www.coursera.org/learn/introduction-to-data-analytics/reviews']\n",
        "review_urls=['https://www.coursera.org/learn/pythonforcybersecurity-introduction/reviews']"
      ],
      "metadata": {
        "id": "ksGLuqD_8HUe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "global df\n",
        "df = pd.DataFrame(columns=['review','star','upvote'])\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XndHY5Ms8fQ8",
        "outputId": "0ce7a0b3-1d43-4ca0-ccdf-074bf26f5ad9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [review, star, upvote]\n",
              "Index: []"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-46109187-181b-4239-a4d9-b2ae99649787\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>star</th>\n",
              "      <th>upvote</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-46109187-181b-4239-a4d9-b2ae99649787')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-46109187-181b-4239-a4d9-b2ae99649787 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-46109187-181b-4239-a4d9-b2ae99649787');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "global res\n",
        "res={}"
      ],
      "metadata": {
        "id": "bxX0JWAXJ_WE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class MySpider1(scrapy.Spider):\n",
        "    name=\"course_reviews\"\n",
        "    def start_requests(self):\n",
        "        for url in review_urls:\n",
        "            yield scrapy.Request(url,callback=self.parse,meta={'dont_merge_cookies': True})\n",
        "            \n",
        "    def parse(self,response):\n",
        "        review=response.css('div.rc-ReviewsList')\n",
        "        reviewBlock=review.css('div.review.review-text.review-page-review')\n",
        "        for ix in reviewBlock:\n",
        "            review=ix.css('div.reviewText p::text').get()\n",
        "            stars=ix.css('svg::attr(style)').getall()\n",
        "            star=len([i for i in stars if i[6:12]=='F2D049'])\n",
        "            help_button=ix.css('button.review-helpful-button')\n",
        "            upvote=help_button.css('span::text').get().split(\" \")[-1]\n",
        "            global df\n",
        "            df=df.append({'review':review,'star':star,'upvote':upvote},ignore_index=True)\n",
        "        next_page=response.css('ul.cui-buttonList a::attr(href)').getall()[-1]\n",
        "        if next_page is not None:\n",
        "          next_page=response.urljoin(next_page)\n",
        "          yield scrapy.Request(next_page,callback=self.parse)\n"
      ],
      "metadata": {
        "id": "F6_oNnMf5TDU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " ### Wait for few minutes or check res status to see which page number is being crawled"
      ],
      "metadata": {
        "id": "EG8WtGvVK7wJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})\n",
        "scrape_with_crochet(MySpider1)"
      ],
      "metadata": {
        "id": "44U0zdh-9L6s",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6d405b97-c9b6-4437-c7ef-7dc444e42d55"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:scrapy.crawler:Overridden settings:\n",
            "{}\n",
            "INFO: Overridden settings:\n",
            "{}\n",
            "INFO:scrapy.extensions.telnet:Telnet Password: 716ac697e9f865b0\n",
            "2022-05-20 20:32:20 [scrapy.extensions.telnet] INFO: Telnet Password: 716ac697e9f865b0\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<crochet._eventloop.EventualResult at 0x7fb163c74150>"
            ]
          },
          "metadata": {},
          "execution_count": 31
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:scrapy.middleware:Enabled extensions:\n",
            "['scrapy.extensions.corestats.CoreStats',\n",
            " 'scrapy.extensions.telnet.TelnetConsole',\n",
            " 'scrapy.extensions.memusage.MemoryUsage',\n",
            " 'scrapy.extensions.logstats.LogStats']\n",
            "2022-05-20 20:32:20 [scrapy.middleware] INFO: Enabled extensions:\n",
            "['scrapy.extensions.corestats.CoreStats',\n",
            " 'scrapy.extensions.telnet.TelnetConsole',\n",
            " 'scrapy.extensions.memusage.MemoryUsage',\n",
            " 'scrapy.extensions.logstats.LogStats']\n",
            "INFO:scrapy.middleware:Enabled downloader middlewares:\n",
            "['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\n",
            " 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\n",
            " 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\n",
            " 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\n",
            " 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\n",
            " 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\n",
            " 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\n",
            " 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\n",
            " 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\n",
            " 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\n",
            " 'scrapy.downloadermiddlewares.stats.DownloaderStats']\n",
            "2022-05-20 20:32:20 [scrapy.middleware] INFO: Enabled downloader middlewares:\n",
            "['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\n",
            " 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\n",
            " 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\n",
            " 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\n",
            " 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\n",
            " 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\n",
            " 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\n",
            " 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\n",
            " 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\n",
            " 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\n",
            " 'scrapy.downloadermiddlewares.stats.DownloaderStats']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "id": "WQxoT6Sb_pk8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fc139850-d06e-4c48-db12-afe509b23015"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:scrapy.middleware:Enabled spider middlewares:\n",
            "['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\n",
            " 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\n",
            " 'scrapy.spidermiddlewares.referer.RefererMiddleware',\n",
            " 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\n",
            " 'scrapy.spidermiddlewares.depth.DepthMiddleware']\n",
            "2022-05-20 20:32:20 [scrapy.middleware] INFO: Enabled spider middlewares:\n",
            "['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\n",
            " 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\n",
            " 'scrapy.spidermiddlewares.referer.RefererMiddleware',\n",
            " 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\n",
            " 'scrapy.spidermiddlewares.depth.DepthMiddleware']\n",
            "INFO:scrapy.middleware:Enabled item pipelines:\n",
            "[]\n",
            "2022-05-20 20:32:20 [scrapy.middleware] INFO: Enabled item pipelines:\n",
            "[]\n",
            "INFO:scrapy.core.engine:Spider opened\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [review, star, upvote]\n",
              "Index: []"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a5eef693-5d6f-45b2-b793-7d90a0c7e3a2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>star</th>\n",
              "      <th>upvote</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a5eef693-5d6f-45b2-b793-7d90a0c7e3a2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a5eef693-5d6f-45b2-b793-7d90a0c7e3a2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a5eef693-5d6f-45b2-b793-7d90a0c7e3a2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 32
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2022-05-20 20:32:20 [scrapy.core.engine] INFO: Spider opened\n",
            "INFO:scrapy.extensions.logstats:Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\n",
            "2022-05-20 20:32:20 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\n",
            "INFO:scrapy.extensions.telnet:Telnet console listening on 127.0.0.1:6023\n",
            "2022-05-20 20:32:20 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Using BERT for sentimental analysis**\n",
        "\n",
        "\n",
        "\n",
        "> After reading few research paper and various articles I found this one to be most helpful and with good explanation for beginners.\n",
        "\n",
        "\n",
        ">[Article Link](https://towardsdatascience.com/sentiment-analysis-in-10-minutes-with-bert-and-hugging-face-294e8a04b671)\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "PkigSOvvrqqE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers"
      ],
      "metadata": {
        "id": "UZ-dxSDcrqnk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import BertTokenizer, TFBertForSequenceClassification\n",
        "from transformers import InputExample, InputFeatures\n",
        "\n",
        "model = TFBertForSequenceClassification.from_pretrained(\"bert-base-uncased\")\n",
        "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")"
      ],
      "metadata": {
        "id": "7N9hVYstrqdt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LvLeNQdXyUkO",
        "outputId": "e0e9a4a4-ceee-451c-fcd9-9735029811c6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"tf_bert_for_sequence_classification\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " bert (TFBertMainLayer)      multiple                  109482240 \n",
            "                                                                 \n",
            " dropout_37 (Dropout)        multiple                  0         \n",
            "                                                                 \n",
            " classifier (Dense)          multiple                  1538      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 109,483,778\n",
            "Trainable params: 109,483,778\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf"
      ],
      "metadata": {
        "id": "AHgBkSZ_yUhV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "URL = \"https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n",
        "\n",
        "dataset = tf.keras.utils.get_file(fname=\"aclImdb_v1.tar.gz\", \n",
        "                                  origin=URL,\n",
        "                                  untar=True,\n",
        "                                  cache_dir='.',\n",
        "                                  cache_subdir='')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rr2uXZmCy2Ta",
        "outputId": "82179c9f-e1f1-4dbc-de65-937d829a6921"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
            "84131840/84125825 [==============================] - 4s 0us/step\n",
            "84140032/84125825 [==============================] - 4s 0us/step\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import shutil\n",
        "# Create main directory path (\"/aclImdb\")\n",
        "main_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')\n",
        "# Create sub directory path (\"/aclImdb/train\")\n",
        "train_dir = os.path.join(main_dir, 'train')\n",
        "# Remove unsup folder since this is a supervised learning task\n",
        "remove_dir = os.path.join(train_dir, 'unsup')\n",
        "shutil.rmtree(remove_dir)\n",
        "# View the final train folder\n",
        "print(os.listdir(train_dir))"
      ],
      "metadata": {
        "id": "qDziMqxiyUe1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "efae9f28-2336-40a6-fbd1-5c27f3fc39b8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['neg', 'urls_pos.txt', 'labeledBow.feat', 'urls_unsup.txt', 'urls_neg.txt', 'unsupBow.feat', 'pos']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# We create a training dataset and a validation \n",
        "# dataset from our \"aclImdb/train\" directory with a 80/20 split.\n",
        "train = tf.keras.preprocessing.text_dataset_from_directory(\n",
        "    'aclImdb/train', batch_size=30000, validation_split=0.2, \n",
        "    subset='training', seed=123)\n",
        "test = tf.keras.preprocessing.text_dataset_from_directory(\n",
        "    'aclImdb/train', batch_size=30000, validation_split=0.2, \n",
        "    subset='validation', seed=123)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jONX9Ey7yUcO",
        "outputId": "8ac12444-dd82-4e59-cab3-07649757fe3f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 25000 files belonging to 2 classes.\n",
            "Using 20000 files for training.\n",
            "Found 25000 files belonging to 2 classes.\n",
            "Using 5000 files for validation.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "ACzuxAv3_y61"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for i in train.take(1):\n",
        "  train_feat = i[0].numpy()\n",
        "  train_lab = i[1].numpy()\n",
        "\n",
        "train = pd.DataFrame([train_feat, train_lab]).T\n",
        "train.columns = ['DATA_COLUMN', 'LABEL_COLUMN']\n",
        "train['DATA_COLUMN'] = train['DATA_COLUMN'].str.decode(\"utf-8\")\n",
        "train.head()"
      ],
      "metadata": {
        "id": "d6mUjVKC11KG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "9f599b78-fbf5-413b-820a-136078ee013b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                         DATA_COLUMN LABEL_COLUMN\n",
              "0  I loved this movie. It is rare to get a glimps...            1\n",
              "1  How much could the general Hollywood director ...            1\n",
              "2  This is a film which should be seen by anybody...            1\n",
              "3  A mock documentary about a pair of Canadian pr...            1\n",
              "4  This movie changed it all for me...I heard of ...            1"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9530597a-4cb3-46a5-9808-fa6861cd5a6d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>DATA_COLUMN</th>\n",
              "      <th>LABEL_COLUMN</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I loved this movie. It is rare to get a glimps...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>How much could the general Hollywood director ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>This is a film which should be seen by anybody...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A mock documentary about a pair of Canadian pr...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>This movie changed it all for me...I heard of ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9530597a-4cb3-46a5-9808-fa6861cd5a6d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9530597a-4cb3-46a5-9808-fa6861cd5a6d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9530597a-4cb3-46a5-9808-fa6861cd5a6d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for j in test.take(1):\n",
        "  test_feat = j[0].numpy()\n",
        "  test_lab = j[1].numpy()\n",
        "\n",
        "test = pd.DataFrame([test_feat, test_lab]).T\n",
        "test.columns = ['DATA_COLUMN', 'LABEL_COLUMN']\n",
        "test['DATA_COLUMN'] = test['DATA_COLUMN'].str.decode(\"utf-8\")\n",
        "test.head()"
      ],
      "metadata": {
        "id": "ZwXGE0-n104e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "29829f4d-076f-4319-8eb5-422d886a69f8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                         DATA_COLUMN LABEL_COLUMN\n",
              "0  I can't believe that so much talent can be was...            0\n",
              "1  This movie blows - let's get that straight rig...            0\n",
              "2  The saddest thing about this \"tribute\" is that...            0\n",
              "3  I'm only rating this film as a 3 out of pity b...            0\n",
              "4  Something surprised me about this movie - it w...            1"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d75a2d07-9064-45f4-9108-6392933ce391\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>DATA_COLUMN</th>\n",
              "      <th>LABEL_COLUMN</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I can't believe that so much talent can be was...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>This movie blows - let's get that straight rig...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>The saddest thing about this \"tribute\" is that...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>I'm only rating this film as a 3 out of pity b...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Something surprised me about this movie - it w...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d75a2d07-9064-45f4-9108-6392933ce391')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d75a2d07-9064-45f4-9108-6392933ce391 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d75a2d07-9064-45f4-9108-6392933ce391');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def convert_data_to_examples(train, test, DATA_COLUMN, LABEL_COLUMN): \n",
        "  train_InputExamples = train.apply(lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this case\n",
        "                                                          text_a = x[DATA_COLUMN], \n",
        "                                                          text_b = None,\n",
        "                                                          label = x[LABEL_COLUMN]), axis = 1)\n",
        "\n",
        "  validation_InputExamples = test.apply(lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this case\n",
        "                                                          text_a = x[DATA_COLUMN], \n",
        "                                                          text_b = None,\n",
        "                                                          label = x[LABEL_COLUMN]), axis = 1)\n",
        "  \n",
        "  return train_InputExamples, validation_InputExamples\n",
        "\n",
        "  train_InputExamples, validation_InputExamples = convert_data_to_examples(train, \n",
        "                                                                           test, \n",
        "                                                                           'DATA_COLUMN', \n",
        "                                                                           'LABEL_COLUMN')\n",
        "  \n",
        "def convert_examples_to_tf_dataset(examples, tokenizer, max_length=128):\n",
        "    features = [] # -> will hold InputFeatures to be converted later\n",
        "\n",
        "    for e in examples:\n",
        "        # Documentation is really strong for this method, so please take a look at it\n",
        "        input_dict = tokenizer.encode_plus(\n",
        "            e.text_a,\n",
        "            add_special_tokens=True,\n",
        "            max_length=max_length, # truncates if len(s) > max_length\n",
        "            return_token_type_ids=True,\n",
        "            return_attention_mask=True,\n",
        "            pad_to_max_length=True, # pads to the right by default # CHECK THIS for pad_to_max_length\n",
        "            truncation=True\n",
        "        )\n",
        "\n",
        "        input_ids, token_type_ids, attention_mask = (input_dict[\"input_ids\"],\n",
        "            input_dict[\"token_type_ids\"], input_dict['attention_mask'])\n",
        "\n",
        "        features.append(\n",
        "            InputFeatures(\n",
        "                input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label=e.label\n",
        "            )\n",
        "        )\n",
        "\n",
        "    def gen():\n",
        "        for f in features:\n",
        "            yield (\n",
        "                {\n",
        "                    \"input_ids\": f.input_ids,\n",
        "                    \"attention_mask\": f.attention_mask,\n",
        "                    \"token_type_ids\": f.token_type_ids,\n",
        "                },\n",
        "                f.label,\n",
        "            )\n",
        "\n",
        "    return tf.data.Dataset.from_generator(\n",
        "        gen,\n",
        "        ({\"input_ids\": tf.int32, \"attention_mask\": tf.int32, \"token_type_ids\": tf.int32}, tf.int64),\n",
        "        (\n",
        "            {\n",
        "                \"input_ids\": tf.TensorShape([None]),\n",
        "                \"attention_mask\": tf.TensorShape([None]),\n",
        "                \"token_type_ids\": tf.TensorShape([None]),\n",
        "            },\n",
        "            tf.TensorShape([]),\n",
        "        ),\n",
        "    )\n",
        "\n",
        "\n",
        "DATA_COLUMN = 'DATA_COLUMN'\n",
        "LABEL_COLUMN = 'LABEL_COLUMN'"
      ],
      "metadata": {
        "id": "OBBuBqsr10gm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_InputExamples, validation_InputExamples = convert_data_to_examples(train, test, DATA_COLUMN, LABEL_COLUMN)\n",
        "\n",
        "train_data = convert_examples_to_tf_dataset(list(train_InputExamples), tokenizer)\n",
        "train_data = train_data.shuffle(100).batch(32).repeat(2)\n",
        "\n",
        "validation_data = convert_examples_to_tf_dataset(list(validation_InputExamples), tokenizer)\n",
        "validation_data = validation_data.batch(32)"
      ],
      "metadata": {
        "id": "v0SpONcz8Mbc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2eb9b208-91cc-49b6-f2c4-3bd95e037a3f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2291: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0), \n",
        "              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), \n",
        "              metrics=[tf.keras.metrics.SparseCategoricalAccuracy('accuracy')])\n",
        "\n",
        "model.fit(train_data, epochs=2, validation_data=validation_data)"
      ],
      "metadata": {
        "id": "zDqz3TAs8MY7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6837c02e-0273-4cac-898b-b2d67c2f8e8a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/2\n",
            "1250/1250 [==============================] - 1092s 856ms/step - loss: 0.2681 - accuracy: 0.8869 - val_loss: 0.3024 - val_accuracy: 0.8762\n",
            "Epoch 2/2\n",
            "1250/1250 [==============================] - 1073s 858ms/step - loss: 0.0743 - accuracy: 0.9750 - val_loss: 0.4634 - val_accuracy: 0.8764\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7fb3d620b2d0>"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.save('/content/drive/MyDrive/bert_model')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gW4ZbycfFVWL",
        "outputId": "c9c098f3-6d02-4a32-adf3-c9ea2e1dff2f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:Found untraced functions such as embeddings_layer_call_fn, embeddings_layer_call_and_return_conditional_losses, encoder_layer_call_fn, encoder_layer_call_and_return_conditional_losses, pooler_layer_call_fn while saving (showing 5 of 420). These functions will not be directly callable after loading.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:tensorflow:Assets written to: /content/drive/MyDrive/bert_model/assets\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:tensorflow:Assets written to: /content/drive/MyDrive/bert_model/assets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pred_sentences=df.review.to_list()\n",
        "pred_sentences"
      ],
      "metadata": {
        "id": "WV9yD-ZWPQeI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tf_batch = tokenizer(pred_sentences, max_length=128, padding=True, truncation=True, return_tensors='tf')\n",
        "tf_outputs = model(tf_batch)\n",
        "tf_predictions = tf.nn.softmax(tf_outputs[0], axis=-1)\n",
        "labels = ['Negative','Positive']\n",
        "label = tf.argmax(tf_predictions, axis=1)\n",
        "label = label.numpy()\n",
        "for i in range(len(pred_sentences)):\n",
        "  print(pred_sentences[i], \": \\n\", labels[label[i]])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3UrZA1c98MWR",
        "outputId": "03a8fc3f-650e-408c-e782-9e2cbf64f4ae"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I will be super honest for this course, I had to give up, I was really wasting time in boring explanations, a course that is not challenging at all and they did not have imagination to create a lab environment or recommend something nice or challenge (There are really a lot of option out there) : \n",
            " Negative\n",
            "This introduction is really cool for beginners. Thanks! : \n",
            " Positive\n",
            "There is no hands-on and step-by-step instruction. It's a joke. : \n",
            " Negative\n",
            "Good course material. Code files should be provided in addition to the course material. : \n",
            " Positive\n",
            "Well i know this course gonna be this fun i have completed it early. I learn a lot and in truth i always want a course like this. : \n",
            " Positive\n",
            "Wanted to walk away from this with more code. : \n",
            " Negative\n",
            "Maybe the instructor is an incredible e-hacker, but it is a terrible communicator, so an awful teacher: an absolutely linear voice, without a body expression ... it is a zombie or a robot. The subject is somehow challenging, and that is why he must stay focused ... but not boring. Teaching is not for everyone. Why people ar still thinking they can??? It is so difficult that it was even deciding to leave my cybersecurity training. : \n",
            " Negative\n",
            "Very good introduction to cybersec scripting. I really appreciate the diving into the MITRE ATT&CK : \n",
            " Positive\n",
            "Wonderful course and really help me a lot in understanding the python concepts in cyber security. : \n",
            " Positive\n",
            "The course is very neat and explained clearly.  : \n",
            " Positive\n",
            "I really learning very much from this course!  : \n",
            " Positive\n",
            "Great and amazing course, thank you. : \n",
            " Positive\n",
            "good for learners and for everyone : \n",
            " Positive\n",
            "Wonderful, a good instructor : \n",
            " Positive\n",
            "really good one : \n",
            " Positive\n",
            "Good hands on : \n",
            " Positive\n",
            "Fantástico : \n",
            " Positive\n",
            "e​xcellent  : \n",
            " Positive\n",
            "Excellent : \n",
            " Positive\n",
            "good : \n",
            " Positive\n",
            "good : \n",
            " Positive\n",
            "p : \n",
            " Positive\n",
            "Great examples of Python in action. The course provides a useful starting point for those already familiar with basic Python. This is not a \"zero to hero\" course. If you don't know the Python basics, you might get lost. Some of the videos are cropped badly, cutting off view of code. However, the content and instructions are clear and practical.  : \n",
            " Positive\n",
            "It is easy to understand  the mitre attack, dns exploration and the valid account : \n",
            " Positive\n",
            "Nice introduction course, i really like it  : \n",
            " Positive\n",
            "Good course ,   need more hands on work ! : \n",
            " Positive\n",
            "Good Course, Wish there was more. : \n",
            " Negative\n",
            "Good intro and simple to digest! : \n",
            " Positive\n",
            "Great introduction into Python : \n",
            " Positive\n",
            "awesome content : \n",
            " Positive\n",
            "Yes : \n",
            " Positive\n",
            "I​ could not find the technical details about the course, like the Python version, which caused trouble when I tried to write the code of the examples. : \n",
            " Negative\n",
            "it's only to get basic notions (come on... the word introduction is on the title) about cyber security : \n",
            " Positive\n",
            "Instructor was too fast and have only Few Assessment!! : \n",
            " Negative\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "KXuJLjGh8MTp"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}