Tokenization_Basically,Stemming&lemmatization,Stop_words.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tokenization Basically,Stemming&lemmatization,Stop words.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**NLP Basic**\n",
        "1.Tokenization\n",
        "2.Stemming and lemmatization\n",
        "3.stopping words\n",
        "4.vocabulary & matching\n",
        "5.parts of speech tagging\n",
        "6.named entity recognition\n",
        "7.Sentences segmentation"
      ],
      "metadata": {
        "id": "AjccdYqnvA3C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **1)Tokenization**"
      ],
      "metadata": {
        "id": "osVJrZhgv6Bz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "s1='Apple is looking at buying U.K. startup for $1 billion!'\n",
        "s2='Hello all,We are here to help you!email support @udemy.com or visit us at http://www.udemy.com!'\n",
        "s3='10km cab ride almost costs $20 in NYC'\n",
        "s4='Lets watch a movie together.'"
      ],
      "metadata": {
        "id": "QsevA1ubv_ZA"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import spacy"
      ],
      "metadata": {
        "id": "Rju1xQ3byD8c"
      },
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nlp=spacy.load(name='en_core_web_sm') #https://spacy.io/models  use for small model"
      ],
      "metadata": {
        "id": "0EqXkplWyO9C"
      },
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nlp_1=spacy.load(name='en_core_web_md') #https://spacy.io/models"
      ],
      "metadata": {
        "id": "fBtKRjoez315"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m spacy download en_core_web_md"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LkPrT4_TyhCZ",
        "outputId": "6067ce57-b866-45a4-aef6-3b95657ac78c"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-27 10:22:30.781362: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting en-core-web-md==3.4.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.4.0/en_core_web_md-3.4.0-py3-none-any.whl (42.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 42.8 MB 1.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy<3.5.0,>=3.4.0 in /usr/local/lib/python3.7/dist-packages (from en-core-web-md==3.4.0) (3.4.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (21.3)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (3.0.9)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (1.0.7)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.11.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.23.0)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (0.6.2)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (1.9.1)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (1.0.3)\n",
            "Requirement already satisfied: typing-extensions<4.2.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (4.1.1)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.0.6)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (3.3.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (57.4.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (1.21.6)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.0.7)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (4.64.0)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (0.4.2)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (0.9.1)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (3.0.6)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (8.1.0)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.7/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.4.4)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (3.8.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (3.0.9)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.2.1 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (5.2.1)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2022.6.15)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.10)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.7/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (0.7.8)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer<0.5.0,>=0.3.0->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.5.0,>=3.4.0->en-core-web-md==3.4.0) (2.0.1)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_md')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import en_core_web_md"
      ],
      "metadata": {
        "id": "jk10-nQX018m"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nlp_1=spacy.load(name='en_core_web_md') #https://spacy.io/models  use for middle model"
      ],
      "metadata": {
        "id": "VC9iriN21Q09"
      },
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "doc1=nlp(s1)"
      ],
      "metadata": {
        "id": "DmjazyjX06QR"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(s1)\n",
        "for token in doc1:\n",
        "  print(token)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fYyF5Gup5zIA",
        "outputId": "5240ac76-95e1-407b-e209-72da205ca9a9"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Apple is looking at buying U.K. startup for $1 billion!\n",
            "Apple\n",
            "is\n",
            "looking\n",
            "at\n",
            "buying\n",
            "U.K.\n",
            "startup\n",
            "for\n",
            "$\n",
            "1\n",
            "billion\n",
            "!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc2=nlp(s2)\n",
        "print(s2)\n",
        "for token in doc2:\n",
        "  print(token)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QPeiQVv05-_U",
        "outputId": "36ba96e7-8796-42d0-b5d1-3f0c919935e6"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hello all,We are here to help you!email support @udemy.com or visit us at http://www.udemy.com!\n",
            "Hello\n",
            "all\n",
            ",\n",
            "We\n",
            "are\n",
            "here\n",
            "to\n",
            "help\n",
            "you!email\n",
            "support\n",
            "@udemy.com\n",
            "or\n",
            "visit\n",
            "us\n",
            "at\n",
            "http://www.udemy.com\n",
            "!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc3=nlp(s3)\n",
        "print(s3)\n",
        "for token in doc3:\n",
        "  print(token)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6wlGD5LD6Vmc",
        "outputId": "9b9e12e2-c1ba-43dd-ce6b-2b0364d6483d"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "10km cab ride almost costs $20 in NYC\n",
            "10\n",
            "km\n",
            "cab\n",
            "ride\n",
            "almost\n",
            "costs\n",
            "$\n",
            "20\n",
            "in\n",
            "NYC\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc4=nlp(s4)\n",
        "print(s4)\n",
        "for token in doc4:\n",
        "  print(token)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ENyF_M8y64sG",
        "outputId": "204bd491-022f-4d04-8e51-59419b685ed2"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Lets watch a movie together.\n",
            "Lets\n",
            "watch\n",
            "a\n",
            "movie\n",
            "together\n",
            ".\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Slicing and Indexing"
      ],
      "metadata": {
        "id": "vnoMBtMa7lBf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "type(doc4)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "veb3enhP68Zd",
        "outputId": "dcbc37d9-bbc8-4cf9-f0fb-7d95c56ec838"
      },
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "spacy.tokens.doc.Doc"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(doc4) #how many tokens in the document"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zDlvst3Q75br",
        "outputId": "791d5609-572d-40c9-fed6-2443f40eaa28"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc4[0] #take the first token in the sentence"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QG3s-Db38LRk",
        "outputId": "c04c24ad-3576-4ba1-c01b-e4de89c0d852"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Lets"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc4[1:4]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sErpBkD98SCQ",
        "outputId": "70cf1dbb-dc63-4d68-c636-0aadded41fa2"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "watch a movie"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc4[-1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b-wtTK4q8iix",
        "outputId": "abace7d6-aa3e-4894-9ee6-81416bd39602"
      },
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "."
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **2)Stemming & Lemmatization**"
      ],
      "metadata": {
        "id": "wbtlKmj-88-y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   Stemming\n",
        "\n"
      ],
      "metadata": {
        "id": "IGaPYBJd_yMW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "words=['run','runner','running','ran','runs','easily','fairly']"
      ],
      "metadata": {
        "id": "KZfAyW0c8p69"
      },
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "playing\n",
        "play    \n",
        "played\n",
        "\n",
        "-->play(the stem)"
      ],
      "metadata": {
        "id": "I0u73MmP9LlK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from nltk.stem.porter import PorterStemmer\n",
        "from nltk.stem.snowball import SnowballStemmer"
      ],
      "metadata": {
        "id": "00imuPnC-BZL"
      },
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "p_stemmer=PorterStemmer()\n",
        "s_stemmer=SnowballStemmer(language='english')"
      ],
      "metadata": {
        "id": "K2DBPgKQ-e8a"
      },
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for word in words:\n",
        "  print(word + ' ---- ' +p_stemmer.stem(word))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8_xOcqoZ-2hv",
        "outputId": "0ae29e1c-e1cf-4d48-da65-5579c1183240"
      },
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "run ---- run\n",
            "runner ---- runner\n",
            "running ---- run\n",
            "ran ---- ran\n",
            "runs ---- run\n",
            "easily ---- easili\n",
            "fairly ---- fairli\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#the performance above is not very good"
      ],
      "metadata": {
        "id": "2Q_8D3N5_N5E"
      },
      "execution_count": 66,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for word in words:\n",
        "  print(word + ' ---- ' +s_stemmer.stem(word))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "84mLVatb_bdQ",
        "outputId": "8d976fc5-3734-4f1e-97be-9e04112b0bc8"
      },
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "run ---- run\n",
            "runner ---- runner\n",
            "running ---- run\n",
            "ran ---- ran\n",
            "runs ---- run\n",
            "easily ---- easili\n",
            "fairly ---- fair\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#this looks better than before"
      ],
      "metadata": {
        "id": "ie_6hliT_nkU"
      },
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   Lemmatization\n"
      ],
      "metadata": {
        "id": "e4ipbeKw_1uq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import spacy\n",
        "nlp=spacy.load('en_core_web_sm')"
      ],
      "metadata": {
        "id": "w-CyPbNi_qVg"
      },
      "execution_count": 69,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "doc1=nlp('The striped bats are hanging on their feet for best')"
      ],
      "metadata": {
        "id": "JSm10kAbAqzY"
      },
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for token in doc1:\n",
        "  print(token.text,'\\t', token.lemma_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "elswd2QBA-Kn",
        "outputId": "a6d802ba-a542-41c8-936e-5987cf4fb2bc"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The \t the\n",
            "striped \t stripe\n",
            "bats \t bat\n",
            "are \t be\n",
            "hanging \t hang\n",
            "on \t on\n",
            "their \t their\n",
            "feet \t foot\n",
            "for \t for\n",
            "best \t good\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "s1='The striped bats are hanging on their feet for best'\n",
        "for word in s1.split():\n",
        "  print(word + ' ---- ' +s_stemmer.stem(word))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g0Yveo7lBRdj",
        "outputId": "b985fc40-c566-43a1-9a64-3f01069ab3de"
      },
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The ---- the\n",
            "striped ---- stripe\n",
            "bats ---- bat\n",
            "are ---- are\n",
            "hanging ---- hang\n",
            "on ---- on\n",
            "their ---- their\n",
            "feet ---- feet\n",
            "for ---- for\n",
            "best ---- best\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **3)Stop Words**"
      ],
      "metadata": {
        "id": "_b_dbhugCB3E"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "a an the always"
      ],
      "metadata": {
        "id": "YXjTmT5NCMv7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(nlp.Defaults.stop_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FGTvbWKVCFI_",
        "outputId": "0473df3c-a406-4c15-ad73-20f4d760f7a5"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'nor', 'former', 'sometime', 'when', 'is', 'twenty', 'amount', 'again', 'side', '‘ve', 'my', \"'s\", 'cannot', 'sixty', 'none', 'fifteen', 'own', 'last', 'everyone', 'either', 'five', 'yet', 'first', 'everywhere', 'some', 'between', 'can', 'would', 'amongst', 'move', 'too', 'anywhere', 'at', 'off', 'elsewhere', 'no', 'while', 'that', 'any', 'ca', '‘ll', 'becomes', 'now', 'all', 'whereafter', 're', 'something', 'several', 'using', 'each', 'do', 'everything', 'say', 'thereupon', 'always', 'are', 'nowhere', 'unless', 'whereupon', 'further', 'across', 'an', 'its', 'what', 'quite', 'more', 'toward', 'before', 'namely', 'few', 'were', 'without', 'throughout', 'top', 'whatever', 'even', 'already', 'must', 'same', 'nothing', 'and', 'show', 'regarding', 'most', 'under', 'hers', 'whereby', 'whole', 'or', 'get', 'yours', 'they', 'against', 'once', 'herein', 'rather', 'due', 'how', 'many', 'least', '‘re', 'fifty', 'on', 'upon', 'but', 'within', 'over', 'eleven', 'in', 'anyhow', 'beside', 'take', 'empty', 'did', \"n't\", 'where', 'so', 'forty', 'six', 'his', 'full', 'it', 'therefore', 'as', 'indeed', 'back', 'also', 'every', 'our', 'for', 'besides', 'anyone', 'n’t', 'out', 'whither', 'down', 'whose', '’ve', 'done', 'bottom', 'about', 'ever', 'someone', 'often', 'than', 'hereby', 'was', 'wherever', 'which', '’d', 'becoming', 'via', 'this', 'has', 'thereby', 'above', 'anything', 'here', 'ourselves', 'such', 'should', 'sometimes', 'beyond', '’re', 'himself', 'never', 'them', 'there', 'less', 'thus', 'to', 'whether', '‘d', 'from', \"'re\", 'latter', 'four', 'these', 'into', 'around', 'hence', 'through', 'latterly', 'much', 'just', 'because', '’m', 'call', 'still', 'whom', 'others', 'though', 'behind', 'below', 'almost', 'have', \"'ll\", 'may', 'mostly', 'among', 'else', 'together', 'why', 'then', 'towards', 'if', 'am', 'be', 'had', 'yourselves', 'nine', 'seems', \"'ve\", 'hereafter', 'three', 'us', 'perhaps', 'hundred', 'onto', \"'d\", 'until', 'really', 'go', 'does', 'eight', 'give', 'whence', 'whoever', 'him', 'thereafter', 'hereupon', 'her', 'by', 'whereas', 'being', 'two', 'formerly', 'one', 'whenever', 'nevertheless', 'please', 'herself', 'their', 'wherein', '’ll', 'will', 'yourself', 'neither', 'thence', 'name', 'third', 'somehow', 'ten', 'those', 'might', 'except', 'he', 'themselves', 'both', 'of', 'noone', 'twelve', 'not', 'therein', 'beforehand', 'another', 'a', 'made', 'along', 'she', 'doing', 'enough', 'moreover', 'ours', 'only', 'otherwise', 'itself', 'mine', 'serious', 'next', 'seem', 'thru', '’s', 'you', 'i', 'during', 'put', 'we', 'somewhere', 'with', 'however', '‘m', 'part', 'me', 'per', 'myself', 'very', 'could', 'anyway', 'afterwards', 'meanwhile', 'up', 'the', 'alone', 'seemed', 'after', 'although', 'front', 'since', 'see', 'become', 'make', 'became', 'your', 'been', '‘s', 'who', 'well', \"'m\", 'n‘t', 'used', 'other', 'keep', 'nobody', 'various', 'seeming'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(nlp.Defaults.stop_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7NMbpC39CeWl",
        "outputId": "068b0734-0d37-4824-ab6e-1b2b45090962"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "326"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nlp.vocab['always'].is_stop"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u8vcZd7zLN9N",
        "outputId": "d0ac2f61-9cd5-4981-bf56-77c9f112ece9"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nlp.vocab['Finance'].is_stop"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YkoZq4Z0LX_r",
        "outputId": "7a20e7ac-a770-40b2-e718-5ee008a4fe91"
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#add 'asdf' to the default list\n",
        "nlp.Defaults.stop_words.add('asdf')"
      ],
      "metadata": {
        "id": "K3-IlUANLmo7"
      },
      "execution_count": 75,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(nlp.Defaults.stop_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ePDjc7anL6Ii",
        "outputId": "51d62c33-583b-4156-d098-7e00a6ff426b"
      },
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "327"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#remove stop word"
      ],
      "metadata": {
        "id": "R928zocXMGND"
      },
      "execution_count": 81,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(nlp.Defaults.stop_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AjdNH9akMW_n",
        "outputId": "394765d5-3a97-4e57-97e5-3a1af5a8891c"
      },
      "execution_count": 82,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'nor', 'former', 'sometime', 'when', 'is', 'twenty', 'amount', 'again', 'side', '‘ve', 'my', \"'s\", 'cannot', 'sixty', 'none', 'fifteen', 'own', 'last', 'everyone', 'either', 'five', 'yet', 'first', 'everywhere', 'some', 'between', 'can', 'would', 'amongst', 'move', 'too', 'anywhere', 'at', 'off', 'elsewhere', 'no', 'while', 'that', 'any', 'ca', '‘ll', 'becomes', 'now', 'all', 'whereafter', 're', 'something', 'several', 'using', 'each', 'do', 'everything', 'say', 'thereupon', 'always', 'are', 'nowhere', 'unless', 'whereupon', 'further', 'across', 'an', 'its', 'what', 'quite', 'more', 'toward', 'before', 'namely', 'few', 'were', 'without', 'throughout', 'top', 'whatever', 'even', 'already', 'must', 'same', 'nothing', 'and', 'show', 'regarding', 'most', 'under', 'hers', 'whereby', 'whole', 'or', 'get', 'yours', 'they', 'against', 'once', 'herein', 'rather', 'due', 'how', 'many', 'least', '‘re', 'fifty', 'on', 'upon', 'but', 'within', 'over', 'eleven', 'in', 'anyhow', 'beside', 'take', 'empty', 'did', \"n't\", 'where', 'so', 'forty', 'six', 'his', 'full', 'it', 'therefore', 'as', 'indeed', 'back', 'also', 'every', 'our', 'for', 'besides', 'anyone', 'n’t', 'out', 'whither', 'down', 'whose', '’ve', 'done', 'bottom', 'about', 'ever', 'someone', 'often', 'than', 'hereby', 'was', 'wherever', 'which', '’d', 'becoming', 'via', 'this', 'has', 'thereby', 'above', 'anything', 'here', 'ourselves', 'such', 'should', 'sometimes', 'beyond', '’re', 'himself', 'never', 'them', 'there', 'less', 'thus', 'to', 'whether', '‘d', 'from', \"'re\", 'latter', 'four', 'these', 'into', 'around', 'hence', 'through', 'latterly', 'much', 'just', 'because', '’m', 'call', 'still', 'whom', 'others', 'though', 'behind', 'below', 'almost', 'have', \"'ll\", 'may', 'mostly', 'among', 'else', 'together', 'why', 'then', 'towards', 'if', 'am', 'be', 'had', 'yourselves', 'nine', 'seems', \"'ve\", 'hereafter', 'three', 'us', 'perhaps', 'hundred', 'onto', \"'d\", 'until', 'really', 'go', 'does', 'eight', 'give', 'whence', 'whoever', 'him', 'thereafter', 'hereupon', 'her', 'by', 'whereas', 'being', 'two', 'formerly', 'one', 'whenever', 'nevertheless', 'please', 'herself', 'their', 'wherein', '’ll', 'will', 'yourself', 'neither', 'thence', 'name', 'third', 'somehow', 'ten', 'those', 'might', 'except', 'he', 'themselves', 'both', 'of', 'noone', 'twelve', 'not', 'therein', 'beforehand', 'another', 'a', 'made', 'along', 'she', 'doing', 'enough', 'moreover', 'ours', 'only', 'otherwise', 'itself', 'mine', 'asdf', 'serious', 'next', 'seem', 'thru', '’s', 'you', 'i', 'during', 'put', 'we', 'somewhere', 'with', 'however', '‘m', 'part', 'me', 'per', 'myself', 'very', 'could', 'anyway', 'afterwards', 'meanwhile', 'up', 'the', 'alone', 'seemed', 'after', 'although', 'front', 'since', 'see', 'become', 'make', 'became', 'your', 'been', '‘s', 'who', 'well', \"'m\", 'n‘t', 'used', 'other', 'keep', 'nobody', 'various', 'seeming'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nlp.Defaults.stop_words.remove('former')"
      ],
      "metadata": {
        "id": "EdPvSmDYMt3h"
      },
      "execution_count": 83,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        " nlp.vocab['former'].is_stop=False"
      ],
      "metadata": {
        "id": "uBW16oM3M026"
      },
      "execution_count": 84,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(nlp.Defaults.stop_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QFVtHlY1M6W6",
        "outputId": "ef51e56f-fe9f-4626-877a-23afa3c1870c"
      },
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'nor', 'sometime', 'when', 'is', 'twenty', 'amount', 'again', 'side', '‘ve', 'my', \"'s\", 'cannot', 'sixty', 'none', 'fifteen', 'own', 'last', 'everyone', 'either', 'five', 'yet', 'first', 'everywhere', 'some', 'between', 'can', 'would', 'amongst', 'move', 'too', 'anywhere', 'at', 'off', 'elsewhere', 'no', 'while', 'that', 'any', 'ca', '‘ll', 'becomes', 'now', 'all', 'whereafter', 're', 'something', 'several', 'using', 'each', 'do', 'everything', 'say', 'thereupon', 'always', 'are', 'nowhere', 'unless', 'whereupon', 'further', 'across', 'an', 'its', 'what', 'quite', 'more', 'toward', 'before', 'namely', 'few', 'were', 'without', 'throughout', 'top', 'whatever', 'even', 'already', 'must', 'same', 'nothing', 'and', 'show', 'regarding', 'most', 'under', 'hers', 'whereby', 'whole', 'or', 'get', 'yours', 'they', 'against', 'once', 'herein', 'rather', 'due', 'how', 'many', 'least', '‘re', 'fifty', 'on', 'upon', 'but', 'within', 'over', 'eleven', 'in', 'anyhow', 'beside', 'take', 'empty', 'did', \"n't\", 'where', 'so', 'forty', 'six', 'his', 'full', 'it', 'therefore', 'as', 'indeed', 'back', 'also', 'every', 'our', 'for', 'besides', 'anyone', 'n’t', 'out', 'whither', 'down', 'whose', '’ve', 'done', 'bottom', 'about', 'ever', 'someone', 'often', 'than', 'hereby', 'was', 'wherever', 'which', '’d', 'becoming', 'via', 'this', 'has', 'thereby', 'above', 'anything', 'here', 'ourselves', 'such', 'should', 'sometimes', 'beyond', '’re', 'himself', 'never', 'them', 'there', 'less', 'thus', 'to', 'whether', '‘d', 'from', \"'re\", 'latter', 'four', 'these', 'into', 'around', 'hence', 'through', 'latterly', 'much', 'just', 'because', '’m', 'call', 'still', 'whom', 'others', 'though', 'behind', 'below', 'almost', 'have', \"'ll\", 'may', 'mostly', 'among', 'else', 'together', 'why', 'then', 'towards', 'if', 'am', 'be', 'had', 'yourselves', 'nine', 'seems', \"'ve\", 'hereafter', 'three', 'us', 'perhaps', 'hundred', 'onto', \"'d\", 'until', 'really', 'go', 'does', 'eight', 'give', 'whence', 'whoever', 'him', 'thereafter', 'hereupon', 'her', 'by', 'whereas', 'being', 'two', 'formerly', 'one', 'whenever', 'nevertheless', 'please', 'herself', 'their', 'wherein', '’ll', 'will', 'yourself', 'neither', 'thence', 'name', 'third', 'somehow', 'ten', 'those', 'might', 'except', 'he', 'themselves', 'both', 'of', 'noone', 'twelve', 'not', 'therein', 'beforehand', 'another', 'a', 'made', 'along', 'she', 'doing', 'enough', 'moreover', 'ours', 'only', 'otherwise', 'itself', 'mine', 'asdf', 'serious', 'next', 'seem', 'thru', '’s', 'you', 'i', 'during', 'put', 'we', 'somewhere', 'with', 'however', '‘m', 'part', 'me', 'per', 'myself', 'very', 'could', 'anyway', 'afterwards', 'meanwhile', 'up', 'the', 'alone', 'seemed', 'after', 'although', 'front', 'since', 'see', 'become', 'make', 'became', 'your', 'been', '‘s', 'who', 'well', \"'m\", 'n‘t', 'used', 'other', 'keep', 'nobody', 'various', 'seeming'}\n"
          ]
        }
      ]
    }
  ]
}