03_Named_entity_recognition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03. Named entity recognition.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP09STdhiKKOryn4Xkn1Qk8",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tanaymukherjee/Natural-Language-Processing/blob/master/03_Named_entity_recognition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qV943AF27yww",
        "colab_type": "text"
      },
      "source": [
        "# Named Entity Recognition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xGVacYSP7vWh",
        "colab_type": "text"
      },
      "source": [
        "## NER with NLTK"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y3uIsqFm-_Wm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "bea1141a-7798-4515-b8da-40b8765607cc"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PVWlnCOA_KLq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import necessary modules\n",
        "from nltk.tokenize import sent_tokenize\n",
        "from nltk.tokenize import word_tokenize"
      ],
      "execution_count": 95,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vc07EOJv_OtR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "article = '\\ufeffThe taxi-hailing company Uber brings into very sharp focus the question of whether corporations can be said to have a moral character. If any human being were to behave with the single-minded and ruthless greed of the company, we would consider them sociopathic. Uber wanted to know as much as possible about the people who use its service, and those who don’t. It has an arrangement with unroll.me, a company which offered a free service for unsubscribing from junk mail, to buy the contacts unroll.me customers had had with rival taxi companies. Even if their email was notionally anonymised, this use of it was not something the users had bargained for. Beyond that, it keeps track of the phones that have been used to summon its services even after the original owner has sold them, attempting this with Apple’s phones even thought it is forbidden by the company.\\r\\n\\r\\n\\r\\nUber has also tweaked its software so that regulatory agencies that the company regarded as hostile would, when they tried to hire a driver, be given false reports about the location of its cars. Uber management booked and then cancelled rides with a rival taxi-hailing company which took their vehicles out of circulation. Uber deny this was the intention. The punishment for this behaviour was negligible. Uber promised not to use this “greyball” software against law enforcement – one wonders what would happen to someone carrying a knife who promised never to stab a policeman with it. Travis Kalanick of Uber got a personal dressing down from Tim Cook, who runs Apple, but the company did not prohibit the use of the app. Too much money was at stake for that.\\r\\n\\r\\n\\r\\nMillions of people around the world value the cheapness and convenience of Uber’s rides too much to care about the lack of drivers’ rights or pay. Many of the users themselves are not much richer than the drivers. The “sharing economy” encourages the insecure and exploited to exploit others equally insecure to the profit of a tiny clique of billionaires. Silicon Valley’s culture seems hostile to humane and democratic values. The outgoing CEO of Yahoo, Marissa Mayer, who is widely judged to have been a failure, is likely to get a $186m payout. This may not be a cause for panic, any more than the previous hero worship should have been a cause for euphoria. Yet there’s an urgent political task to tame these companies, to ensure they are punished when they break the law, that they pay their taxes fairly and that they behave responsibly.'"
      ],
      "execution_count": 96,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LiWueT1r_UsS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "0d31e236-b986-4a7f-a9c3-9620b6ae0ec2"
      },
      "source": [
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('maxent_ne_chunker')\n",
        "nltk.download('words')"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
            "[nltk_data]       date!\n",
            "[nltk_data] Downloading package maxent_ne_chunker to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Package words is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xqmnY6lU7PjK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 228
        },
        "outputId": "07f18c93-e60f-42a4-efc8-e591719474b9"
      },
      "source": [
        "# Tokenize the article into sentences: sentences\n",
        "sentences = nltk.sent_tokenize(article)\n",
        "\n",
        "# Tokenize each sentence into words: token_sentences\n",
        "token_sentences = [nltk.word_tokenize(sent) for sent in sentences]\n",
        "\n",
        "# Tag each tokenized sentence into parts of speech: pos_sentences\n",
        "pos_sentences = [nltk.pos_tag(sent) for sent in token_sentences] \n",
        "\n",
        "# Create the named entity chunks: chunked_sentences\n",
        "chunked_sentences = nltk.ne_chunk_sents(pos_sentences, binary=True)\n",
        "\n",
        "# Test for stems of the tree with 'NE' tags\n",
        "for sent in chunked_sentences:\n",
        "    for chunk in sent:\n",
        "        if hasattr(chunk, \"label\") and chunk.label() == \"NE\":\n",
        "            print(chunk)"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(NE Uber/NNP)\n",
            "(NE Beyond/NN)\n",
            "(NE Apple/NNP)\n",
            "(NE Uber/NNP)\n",
            "(NE Uber/NNP)\n",
            "(NE Travis/NNP Kalanick/NNP)\n",
            "(NE Tim/NNP Cook/NNP)\n",
            "(NE Apple/NNP)\n",
            "(NE Silicon/NNP Valley/NNP)\n",
            "(NE CEO/NNP)\n",
            "(NE Yahoo/NNP)\n",
            "(NE Marissa/NNP Mayer/NNP)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DRrQsC1LBNLc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from collections import defaultdict\n",
        "from matplotlib import pyplot as plt"
      ],
      "execution_count": 99,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tH1XwkFCBESX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create the defaultdict: ner_categories\n",
        "ner_categories = defaultdict(int)\n",
        "\n",
        "# Create the nested for loop\n",
        "for sent in chunked_sentences:\n",
        "    for chunk in sent:\n",
        "        if hasattr(chunk, 'label'):\n",
        "            ner_categories[chunk.label()] += 1\n",
        "            \n",
        "# Create a list from the dictionary keys for the chart labels: labels\n",
        "labels = list(ner_categories.keys())\n",
        "\n",
        "# Create a list of the values: values\n",
        "values = [ner_categories.get(v) for v in labels]\n",
        "\n",
        "# Create the pie chart\n",
        "plt.pie(values, labels=labels, autopct='%1.1f%%', startangle=140)\n",
        "\n",
        "# Display the chart\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OjY52SRGCkjJ",
        "colab_type": "text"
      },
      "source": [
        "## SpaCy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzRyC58HCmFA",
        "colab_type": "text"
      },
      "source": [
        "### Comparing NLTK with spaCy NER"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5XBwVtqtCnID",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        },
        "outputId": "960dad08-c1aa-4b46-bc45-24177ab5fddb"
      },
      "source": [
        "# Import spacy\n",
        "import spacy\n",
        "\n",
        "# Instantiate the English model: nlp\n",
        "nlp = spacy.load('en', tagger=False, parser=False, matcher=False)\n",
        "\n",
        "# Create a new document: doc\n",
        "doc = nlp(article)\n",
        "\n",
        "# Print all of the found entities and their labels\n",
        "for ent in doc.ents:\n",
        "    print(ent.label_, ent.text)"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ORG Uber\n",
            "PERSON Uber\n",
            "ORG unroll.me\n",
            "ORG Apple\n",
            "PERSON Travis Kalanick\n",
            "PERSON Uber\n",
            "PERSON Tim Cook\n",
            "ORG Apple\n",
            "CARDINAL Millions\n",
            "PERSON Uber\n",
            "LOC Silicon Valley\n",
            "ORG Yahoo\n",
            "PERSON Marissa Mayer\n",
            "MONEY $186m\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZbnG4LFD4Cm",
        "colab_type": "text"
      },
      "source": [
        "## Multilingual NER with polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iCt_TwivEw5o",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "ab111cfd-d2df-4d1c-e3d0-bf4e54572011"
      },
      "source": [
        "!pip install polyglot"
      ],
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: polyglot in /usr/local/lib/python3.6/dist-packages (16.7.4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M1BefLH6Fxjk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "fad63466-740c-4576-8e65-c4842faa61ea"
      },
      "source": [
        "!pip install pycld2"
      ],
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: pycld2 in /usr/local/lib/python3.6/dist-packages (0.41)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BmYyrCI3GM7X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "317d73ae-5c42-4a3e-9836-cf99cf6ebdde"
      },
      "source": [
        "!pip install morfessor"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: morfessor in /usr/local/lib/python3.6/dist-packages (2.0.6)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "70r2prvqGoik",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "37d1e30f-3787-453a-c131-0b8b782ffb0a"
      },
      "source": [
        "!pip install pyicu"
      ],
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: pyicu in /usr/local/lib/python3.6/dist-packages (2.5)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ddFb2ZWAFWr3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from icu import Locale\n",
        "import pycld2 as cld2"
      ],
      "execution_count": 106,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qz78umc0D6f7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from polyglot.text import Text"
      ],
      "execution_count": 107,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWXDQqB3HyzV",
        "colab_type": "text"
      },
      "source": [
        "### French NER with polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_wAgnKNHZNE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 651
        },
        "outputId": "0376b005-1fe8-4e48-8938-ca207365832b"
      },
      "source": [
        "!polyglot download LANG:fr"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Downloading collection 'LANG:fr'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sgns2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sgns2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package unipos.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package ner2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package ner2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package counts2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package counts2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package transliteration2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package transliteration2.fr is already up-to-\n",
            "[polyglot_data]    |       date!\n",
            "[polyglot_data]    | Downloading package embeddings2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package uniemb.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package pos2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package pos2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package sentiment2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.fr is already up-to-date!\n",
            "[polyglot_data]    | Downloading package morph2.fr to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package morph2.fr is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:fr\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ON9VaO8KHK4G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "article = \"\\ufeffédition abonné\\r\\n\\r\\n\\r\\nDans une tribune au « Monde », l’universitaire Charles Cuvelliez estime que le fantasme d’un remplacement de l’homme par l’algorithme et le robot repose sur un malentendu.\\r\\n\\r\\n\\r\\nLe Monde | 10.05.2017 à 06h44 • Mis à jour le 10.05.2017 à 09h47 | Par Charles Cuvelliez (Professeur à l’Ecole polytechnique de l'université libre de Bruxelles)\\r\\n\\r\\n\\r\\nTRIBUNE. L’usage morbide, par certains, de Facebook Live a amené son fondateur à annoncer précipitamment le recrutement de 3 000 modérateurs supplémentaires. Il est vrai que l’intelligence artificielle (IA) est bien en peine de reconnaître des contenus violents, surtout diffusés en direct.\\r\\n\\r\\n\\r\\nLe quotidien affreux de ces modérateurs, contraints de visionner des horreurs à longueur de journée, mériterait pourtant qu’on les remplace vite par des machines !\\r\\n\\r\\n\\r\\nL’IA ne peut pas tout, mais là où elle peut beaucoup, on la maudit, accusée de détruire nos emplois, de remplacer la convivialité humaine. Ce débat repose sur un malentendu.\\r\\n\\r\\n\\r\\nIl vient d’une définition de l’IA qui n’a, dans la réalité, jamais pu être mise en pratique : en 1955, elle était vue comme la création de programmes informatiques qui, quoi qu’on leur confie, le feraient un jour mieux que les humains. On pensait que toute caractéristique de l’intelligence humaine pourrait un jour être si précisément décrite qu’il suffirait d’une machine pour la simuler. Ce n’est pas vrai.\\r\\n\\r\\n\\r\\nAngoisses infondées\\r\\n\\r\\n\\r\\nComme le dit un récent Livre blanc sur la question (Pourquoi il ne faut pas avoir peur de l’Intelligence arti\\xadficielle, Julien Maldonato, Deloitte, mars 2017), rien ne pourra remplacer un humain dans sa globalité.\\r\\n\\r\\n\\r\\nL’IA, c’est de l’apprentissage automatique doté d’un processus d’ajustement de modèles statistiques à des masses de données, explique l’auteur. Il s’agit d’un apprentissage sur des paramètres pour lesquels une vision humaine n’explique pas pourquoi ils marchent si bien dans un contexte donné.\\r\\n\\r\\n\\r\\nC’est aussi ce que dit le rapport de l’Office parlementaire d’évaluation des choix scientifiques et technologiques (« Pour une intelligence artificielle maîtrisée, utile et démystifiée », 29 mars 2017), pour qui ce côté « boîte noire » explique des angoisses infondées. Ethiquement, se fonder sur l’IA pour des tâches critiques sans bien comprendre le comment...\""
      ],
      "execution_count": 109,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wwN5ljPFFeiz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "20ff782f-a437-47f7-c4f4-1907861dba5d"
      },
      "source": [
        "# Create a new text object using Polyglot's Text class: txt\n",
        "txt = Text(article)\n",
        "\n",
        "# Print each of the entities found\n",
        "for ent in txt.entities:\n",
        "    print(ent)\n",
        "    \n",
        "# Print the type of ent\n",
        "print(type(ent))"
      ],
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Charles', 'Cuvelliez']\n",
            "['Charles', 'Cuvelliez']\n",
            "['Bruxelles']\n",
            "['l’IA']\n",
            "['Julien', 'Maldonato']\n",
            "['Deloitte']\n",
            "['Ethiquement']\n",
            "['l’IA']\n",
            "['.']\n",
            "<class 'polyglot.text.Chunk'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gG070KftH32M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "8888ca38-9aeb-4f8f-9ad1-abf5564d03da"
      },
      "source": [
        "# Create the list of tuples: entities\n",
        "entities = [(ent.tag, ' '.join(ent)) for ent in txt.entities]\n",
        "\n",
        "# Print entities\n",
        "print(entities)"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('I-PER', 'Charles Cuvelliez'), ('I-PER', 'Charles Cuvelliez'), ('I-ORG', 'Bruxelles'), ('I-PER', 'l’IA'), ('I-PER', 'Julien Maldonato'), ('I-ORG', 'Deloitte'), ('I-PER', 'Ethiquement'), ('I-LOC', 'l’IA'), ('I-PER', '.')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eWPPzR9LJ5X8",
        "colab_type": "text"
      },
      "source": [
        "### All languages from polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sAjV7FEzJpaI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "16871b87-fa37-4a28-a542-c617dea1000e"
      },
      "source": [
        "from polyglot.downloader import downloader\n",
        "downloader.download(\"embeddings2.en\")"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Downloading package embeddings2.en to\n",
            "[polyglot_data]     /root/polyglot_data...\n",
            "[polyglot_data]   Package embeddings2.en is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nC-Iak_gJZec",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "3b9c2f46-c5c8-4171-b8c2-a7351f036ea5"
      },
      "source": [
        "downloader.download(\"TASK:transliteration2\", quiet=True)"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Error downloading 'transliteration2.pl' from <http://p\n",
            "[polyglot_data]     olyglot.cs.stonybrook.edu/~polyglot/transliteratio\n",
            "[polyglot_data]     n2/pl/transliteration.pl.tar.bz2>:   HTTP Error\n",
            "[polyglot_data]     403: Forbidden\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qGyCzdkoJxJE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "8f8d065a-af47-4d62-b40d-ea07f335ca45"
      },
      "source": [
        "downloader.list(show_packages=False)"
      ],
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using default data directory (/root/polyglot_data)\n",
            "======================================================================\n",
            " Data server index for <http://polyglot.cs.stonybrook.edu/~polyglot/>\n",
            "======================================================================\n",
            "Collections:\n",
            "  [P] LANG:af............. Afrikaans            packages and models\n",
            "  [ ] LANG:als............ Alemannic            packages and models\n",
            "  [P] LANG:am............. Amharic              packages and models\n",
            "  [ ] LANG:an............. Aragonese            packages and models\n",
            "  [P] LANG:ar............. Arabic               packages and models\n",
            "  [ ] LANG:arz............ Egyptian Arabic      packages and models\n",
            "  [ ] LANG:as............. Assamese             packages and models\n",
            "  [ ] LANG:ast............ Asturian             packages and models\n",
            "  [P] LANG:az............. Azerbaijani          packages and models\n",
            "  [ ] LANG:ba............. Bashkir              packages and models\n",
            "  [ ] LANG:bar............ Bavarian             packages and models\n",
            "  [P] LANG:be............. Belarusian           packages and models\n",
            "  [P] LANG:bg............. Bulgarian            packages and models\n",
            "  [P] LANG:bn............. Bangla               packages and models\n",
            "  [ ] LANG:bo............. Tibetan              packages and models\n",
            "  [ ] LANG:bpy............ Bishnupriya          packages and models\n",
            "  [ ] LANG:br............. Breton               packages and models\n",
            "  [ ] LANG:bs............. Bosnian              packages and models\n",
            "  [P] LANG:ca............. Catalan              packages and models\n",
            "  [ ] LANG:ce............. Chechen              packages and models\n",
            "  [ ] LANG:ceb............ Cebuano              packages and models\n",
            "  [ ] LANG:cs............. Czech                packages and models\n",
            "  [ ] LANG:cv............. Chuvash              packages and models\n",
            "  [P] LANG:cy............. Welsh                packages and models\n",
            "  [P] LANG:da............. Danish               packages and models\n",
            "  [ ] LANG:de............. German               packages and models\n",
            "  [ ] LANG:diq............ Zazaki               packages and models\n",
            "  [ ] LANG:dv............. Divehi               packages and models\n",
            "  [P] LANG:el............. Greek                packages and models\n",
            "  [P] LANG:en............. English              packages and models\n",
            "  [ ] LANG:eo............. Esperanto            packages and models\n",
            "  [-] LANG:es............. Spanish              packages and models\n",
            "  [P] LANG:et............. Estonian             packages and models\n",
            "  [P] LANG:eu............. Basque               packages and models\n",
            "  [P] LANG:fa............. Persian              packages and models\n",
            "  [P] LANG:fi............. Finnish              packages and models\n",
            "  [ ] LANG:fo............. Faroese              packages and models\n",
            "  [-] LANG:fr............. French               packages and models\n",
            "  [ ] LANG:fy............. Western Frisian      packages and models\n",
            "  [P] LANG:ga............. Irish                packages and models\n",
            "  [*] LANG:gan............ Gan Chinese          packages and models\n",
            "  [ ] LANG:gd............. Scottish Gaelic      packages and models\n",
            "  [P] LANG:gl............. Galician             packages and models\n",
            "  [P] LANG:gu............. Gujarati             packages and models\n",
            "  [ ] LANG:gv............. Manx                 packages and models\n",
            "  [ ] LANG:he............. Hebrew               packages and models\n",
            "  [-] LANG:hi............. Hindi                packages and models\n",
            "  [ ] LANG:hif............ Fiji Hindi           packages and models\n",
            "  [P] LANG:hr............. Croatian             packages and models\n",
            "  [ ] LANG:hsb............ Upper Sorbian        packages and models\n",
            "  [P] LANG:ht............. Haitian Creole       packages and models\n",
            "  [ ] LANG:hu............. Hungarian            packages and models\n",
            "  [P] LANG:hy............. Armenian             packages and models\n",
            "  [ ] LANG:ia............. Interlingua          packages and models\n",
            "  [ ] LANG:id............. Indonesian           packages and models\n",
            "  [ ] LANG:ilo............ Iloko                packages and models\n",
            "  [ ] LANG:io............. Ido                  packages and models\n",
            "  [P] LANG:is............. Icelandic            packages and models\n",
            "  [P] LANG:it............. Italian              packages and models\n",
            "  [P] LANG:ja............. Japanese             packages and models\n",
            "  [ ] LANG:jv............. Javanese             packages and models\n",
            "  [P] LANG:ka............. Georgian             packages and models\n",
            "  [ ] LANG:kk............. Kazakh               packages and models\n",
            "  [P] LANG:km............. Khmer                packages and models\n",
            "  [P] LANG:kn............. Kannada              packages and models\n",
            "  [P] LANG:ko............. Korean               packages and models\n",
            "  [ ] LANG:ku............. Kurdish              packages and models\n",
            "  [ ] LANG:ky............. Kyrgyz               packages and models\n",
            "  [P] LANG:la............. Latin                packages and models\n",
            "  [ ] LANG:lb............. Luxembourgish        packages and models\n",
            "  [ ] LANG:li............. Limburgish           packages and models\n",
            "  [ ] LANG:lmo............ Lombard              packages and models\n",
            "  [ ] LANG:lt............. Lithuanian           packages and models\n",
            "  [P] LANG:lv............. Latvian              packages and models\n",
            "  [ ] LANG:mg............. Malagasy             packages and models\n",
            "  [P] LANG:mk............. Macedonian           packages and models\n",
            "  [ ] LANG:ml............. Malayalam            packages and models\n",
            "  [ ] LANG:mn............. Mongolian            packages and models\n",
            "  [P] LANG:mr............. Marathi              packages and models\n",
            "  [P] LANG:ms............. Malay                packages and models\n",
            "  [P] LANG:mt............. Maltese              packages and models\n",
            "  [ ] LANG:my............. Burmese              packages and models\n",
            "  [ ] LANG:ne............. Nepali               packages and models\n",
            "  [P] LANG:nl............. Dutch                packages and models\n",
            "  [P] LANG:nn............. Norwegian Nynorsk    packages and models\n",
            "  [P] LANG:no............. Norwegian            packages and models\n",
            "  [ ] LANG:oc............. Occitan              packages and models\n",
            "  [ ] LANG:or............. Odia                 packages and models\n",
            "  [ ] LANG:os............. Ossetic              packages and models\n",
            "  [ ] LANG:pa............. Punjabi              packages and models\n",
            "  [ ] LANG:pam............ Pampanga             packages and models\n",
            "  [ ] LANG:pl............. Polish               packages and models\n",
            "  [ ] LANG:pms............ Piedmontese          packages and models\n",
            "  [ ] LANG:ps............. Pashto               packages and models\n",
            "  [P] LANG:pt............. Portuguese           packages and models\n",
            "  [ ] LANG:qu............. Quechua              packages and models\n",
            "  [ ] LANG:rm............. Romansh              packages and models\n",
            "  [P] LANG:ro............. Romanian             packages and models\n",
            "  [ ] LANG:ru............. Russian              packages and models\n",
            "  [ ] LANG:sa............. Sanskrit             packages and models\n",
            "  [ ] LANG:sah............ Sakha                packages and models\n",
            "  [ ] LANG:scn............ Sicilian             packages and models\n",
            "  [ ] LANG:sco............ Scots                packages and models\n",
            "  [ ] LANG:se............. Northern Sami        packages and models\n",
            "  [P] LANG:sh............. Serbo-Croatian       packages and models\n",
            "  [ ] LANG:si............. Sinhala              packages and models\n",
            "  [P] LANG:sk............. Slovak               packages and models\n",
            "  [P] LANG:sl............. Slovenian            packages and models\n",
            "  [P] LANG:sq............. Albanian             packages and models\n",
            "  [P] LANG:sr............. Serbian              packages and models\n",
            "  [ ] LANG:su............. Sundanese            packages and models\n",
            "  [P] LANG:sv............. Swedish              packages and models\n",
            "  [P] LANG:sw............. Swahili              packages and models\n",
            "  [ ] LANG:szl............ Silesian             packages and models\n",
            "  [P] LANG:ta............. Tamil                packages and models\n",
            "  [P] LANG:te............. Telugu               packages and models\n",
            "  [ ] LANG:tg............. Tajik                packages and models\n",
            "  [ ] LANG:th............. Thai                 packages and models\n",
            "  [ ] LANG:tk............. Turkmen              packages and models\n",
            "  [P] LANG:tl............. Tagalog              packages and models\n",
            "  [ ] LANG:tr............. Turkish              packages and models\n",
            "  [ ] LANG:tt............. Tatar                packages and models\n",
            "  [ ] LANG:ug............. Uyghur               packages and models\n",
            "  [P] LANG:uk............. Ukrainian            packages and models\n",
            "  [P] LANG:ur............. Urdu                 packages and models\n",
            "  [ ] LANG:uz............. Uzbek                packages and models\n",
            "  [ ] LANG:vec............ Venetian             packages and models\n",
            "  [-] LANG:vi............. Vietnamese           packages and models\n",
            "  [ ] LANG:vls............ West Flemish         packages and models\n",
            "  [ ] LANG:vo............. Volapük              packages and models\n",
            "  [ ] LANG:wa............. Walloon              packages and models\n",
            "  [ ] LANG:war............ Waray                packages and models\n",
            "  [P] LANG:yi............. Yiddish              packages and models\n",
            "  [ ] LANG:yo............. Yoruba               packages and models\n",
            "  [-] LANG:zh............. Chinese              packages and models\n",
            "  [*] LANG:zhc............ Chinese Character    packages and models\n",
            "  [*] LANG:zhw............ Chinese Word         packages and models\n",
            "  [P] TASK:counts2........ counts2\n",
            "  [P] TASK:embeddings2.... embeddings2\n",
            "  [P] TASK:morph2......... morph2\n",
            "  [P] TASK:ner2........... ner2\n",
            "  [P] TASK:pos2........... pos2\n",
            "  [P] TASK:sentiment2..... sentiment2\n",
            "  [P] TASK:sgns2.......... sgns2\n",
            "  [P] TASK:transliteration2 transliteration2\n",
            "  [P] TASK:tsne2.......... tsne2\n",
            "  [-] TASK:uniemb......... uniemb\n",
            "  [-] TASK:unipos......... unipos\n",
            "\n",
            "([*] marks installed packages; [-] marks out-of-date or corrupt packages;\n",
            " [P] marks partially installed collections)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KjPn4FpdLDok",
        "colab_type": "text"
      },
      "source": [
        "### Spanish NER with polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_kWbQhtYKBGq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 651
        },
        "outputId": "0b290cd1-8973-46f2-c4e4-3d5f8f89c6da"
      },
      "source": [
        "!polyglot download LANG:es"
      ],
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Downloading collection 'LANG:es'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sgns2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sgns2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package unipos.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package ner2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package ner2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package counts2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package counts2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package transliteration2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package transliteration2.es is already up-to-\n",
            "[polyglot_data]    |       date!\n",
            "[polyglot_data]    | Downloading package embeddings2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package uniemb.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package pos2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package pos2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package sentiment2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.es is already up-to-date!\n",
            "[polyglot_data]    | Downloading package morph2.es to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package morph2.es is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:es\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6MfuaO-SIdzu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "article = \"Lina del Castillo es profesora en el Instituto de Estudios Latinoamericanos Teresa Lozano Long (LLILAS) y el Departamento de Historia de la Universidad de Texas en Austin. Ella será la moderadora del panel “Los Mundos Políticos de Gabriel García Márquez” este viernes, Oct. 30, en el simposio Gabriel García Márquez: Vida y Legado.LIna del Castillo Actualmente, sus investigaciones abarcan la intersección de cartografía, disputas a las demandas de tierra y recursos, y la formación del n...el tren de medianoche que lleva a miles y miles de cadáveres uno encima del otro como tantos racimos del banano que acabarán tirados al mar. Ningún recuento periodístico podría provocar nuestra imaginación y nuestra memoria como este relato de García Márquez. Contenido Relacionado Lea más artículos sobre el archivo de Gabriel García Márquez Reciba mensualmente las últimas noticias e información del Harry Ransom Center con eNews, nuestro correo electrónico mensual. ¡Suscríbase hoy!\""
      ],
      "execution_count": 116,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fx05bznuNSsy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        },
        "outputId": "b2840f1d-68fd-4975-f53f-a82fbfe4138f"
      },
      "source": [
        "# Create a new text object using Polyglot's Text class: txt\n",
        "txt = Text(article)\n",
        "\n",
        "# Print each of the entities found\n",
        "for ent in txt.entities:\n",
        "    print(ent)\n",
        "    \n",
        "# Print the type of ent\n",
        "print(type(ent))"
      ],
      "execution_count": 117,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Lina']\n",
            "['Castillo']\n",
            "['Teresa', 'Lozano', 'Long']\n",
            "['Universidad', 'de', 'Texas']\n",
            "['Austin']\n",
            "['Austin', '.']\n",
            "['Austin', '.', 'Ella']\n",
            "['Gabriel', 'García', 'Márquez']\n",
            "['Gabriel', 'García', 'Márquez']\n",
            "['Legado.LIna']\n",
            "['Castillo']\n",
            "['García', 'Márquez']\n",
            "['Gabriel', 'García', 'Márquez']\n",
            "['Harry', 'Ransom']\n",
            "['Harry', 'Ransom', 'Center']\n",
            "<class 'polyglot.text.Chunk'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wKBBPYPfK6ny",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "59700bb8-acf5-4ad7-f108-1e7371d171cf"
      },
      "source": [
        "# Create the list of tuples: entities\n",
        "entities = [(ent.tag, ' '.join(ent)) for ent in txt.entities]\n",
        "\n",
        "# Print entities\n",
        "print(entities)"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('I-PER', 'Lina'), ('I-PER', 'Castillo'), ('I-PER', 'Teresa Lozano Long'), ('I-ORG', 'Universidad de Texas'), ('I-PER', 'Austin'), ('I-LOC', 'Austin .'), ('I-PER', 'Austin . Ella'), ('I-PER', 'Gabriel García Márquez'), ('I-PER', 'Gabriel García Márquez'), ('I-PER', 'Legado.LIna'), ('I-LOC', 'Castillo'), ('I-PER', 'García Márquez'), ('I-PER', 'Gabriel García Márquez'), ('I-PER', 'Harry Ransom'), ('I-ORG', 'Harry Ransom Center')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-SUFeHbI2aD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "46291c1d-0411-4e54-eaee-5f678f881939"
      },
      "source": [
        "# Create a new text object using Polyglot's Text class: txt\n",
        "txt = Text(article)\n",
        "\n",
        "# Initialize the count variable: count\n",
        "count = 0\n",
        "\n",
        "# Iterate over all the entities\n",
        "for ent in txt.entities:\n",
        "    # Check whether the entity contains 'Márquez' or 'Gabo'\n",
        "    if \"Márquez\" in ent or \"Gabo\" in ent:\n",
        "        # Increment count\n",
        "        count += 1\n",
        "\n",
        "# Print count\n",
        "print(count)\n",
        "\n",
        "# Calculate the percentage of entities that refer to \"Gabo\": percentage\n",
        "percentage = count / len(txt.entities)\n",
        "print(percentage)"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4\n",
            "0.26666666666666666\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1K-oDyFKOTy4",
        "colab_type": "text"
      },
      "source": [
        "### Hindi NER with polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V-KwAbxYMA_9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 598
        },
        "outputId": "da42f00c-4763-4aa1-98b3-2a7d3495f4ff"
      },
      "source": [
        "!polyglot download LANG:hi"
      ],
      "execution_count": 120,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Downloading collection 'LANG:hi'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sgns2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sgns2.hi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package unipos.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package ner2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package ner2.hi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package counts2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package counts2.hi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package transliteration2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package transliteration2.hi is already up-to-\n",
            "[polyglot_data]    |       date!\n",
            "[polyglot_data]    | Downloading package embeddings2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.hi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package uniemb.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package sentiment2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.hi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.hi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package morph2.hi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package morph2.hi is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:hi\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dg6bioc2X6Nf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "article = \"मेरा नाम तनय मुख़र्जी है | मैं इंदौर का रहने वाला हूँ | मैं रिलायंस कंपनी के लिए काम करता हूँ | भारत सरकार के तत्कालीन प्रधान मंत्री नरेंद्र मोदी है | भारत के राष्ट्र पिता का नाम महात्मा गाँधी है |\""
      ],
      "execution_count": 121,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TBY2UKS1BAI7",
        "colab_type": "text"
      },
      "source": [
        "**English translation:** \"My name is Tanay Mukherjee. I live in Indore. I work for this company called Reliance. The head of Indian Government is Narendra Modi. Mahatma Gandhi is the founding father of India.\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8351UElWMgM6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 140
        },
        "outputId": "0780497e-79ef-4cbc-ccc0-aa040d083de0"
      },
      "source": [
        "# Create a new text object using Polyglot's Text class: txt\n",
        "txt = Text(article)\n",
        "\n",
        "# Print each of the entities found\n",
        "for ent in txt.entities:\n",
        "    print(ent)\n",
        "    \n",
        "# Print the type of ent\n",
        "print(type(ent))"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['इंदौर']\n",
            "['रिलायंस']\n",
            "['भारत', 'सरकार']\n",
            "['नरेंद्र', 'मोदी']\n",
            "['भारत']\n",
            "['महात्मा', 'गाँधी']\n",
            "<class 'polyglot.text.Chunk'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TCoIPguBOPtS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "7f94dbf0-1dbb-4087-ddb2-a5402e5d4f1f"
      },
      "source": [
        "# Create the list of tuples: entities\n",
        "entities = [(ent.tag, ' '.join(ent)) for ent in txt.entities]\n",
        "\n",
        "# Print entities\n",
        "print(entities)"
      ],
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('I-LOC', 'इंदौर'), ('I-ORG', 'रिलायंस'), ('I-ORG', 'भारत सरकार'), ('I-PER', 'नरेंद्र मोदी'), ('I-LOC', 'भारत'), ('I-PER', 'महात्मा गाँधी')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CgQqBccX4u-V",
        "colab_type": "text"
      },
      "source": [
        "### Vietnamese NER with polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1pMCA1Jy40-a",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 598
        },
        "outputId": "3027639e-27da-492d-d85c-7be1c8274b95"
      },
      "source": [
        "!polyglot download LANG:vi"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Downloading collection 'LANG:vi'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sgns2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sgns2.vi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package unipos.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package ner2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package ner2.vi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package counts2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package counts2.vi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package transliteration2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package transliteration2.vi is already up-to-\n",
            "[polyglot_data]    |       date!\n",
            "[polyglot_data]    | Downloading package embeddings2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.vi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package uniemb.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package sentiment2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.vi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.vi is already up-to-date!\n",
            "[polyglot_data]    | Downloading package morph2.vi to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package morph2.vi is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:vi\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zOPa_u5Y45SS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "article = \"Tên tôi là Châu Hoàng. Tôi sống tại thành phố Hồ Chí Minh. Tôi làm việc cho công ty chứng khoán Bảo Việt. Người đứng đầu của chính phủ Việt Nam là Nguyễn Phú Trọng. Hồ Chí Minh là cha đẻ của Việt Nam.\""
      ],
      "execution_count": 125,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uaga8pV5Azkh",
        "colab_type": "text"
      },
      "source": [
        "**English translation:** \"My name is Chau Hoang. I live in Ho Chi Minh City. I work for this company called Bao Viet Holdings. The head of Vietnamese Government is Nguyễn Phú Trọng. Hồ Chí Minh is the founding father of Vietnam.\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8tU7alYv49zy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "outputId": "5cfb2b50-297b-4a76-81ef-4a95b20f6da3"
      },
      "source": [
        "# Create a new text object using Polyglot's Text class: txt\n",
        "txt = Text(article)\n",
        "\n",
        "# Print each of the entities found\n",
        "for ent in txt.entities:\n",
        "    print(ent)\n",
        "    \n",
        "# Print the type of ent\n",
        "print(type(ent))"
      ],
      "execution_count": 126,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Châu', 'Hoàng']\n",
            "['Hồ', 'Chí', 'Minh']\n",
            "['Bảo']\n",
            "['Việt', 'Nam']\n",
            "['Phú', 'Trọng']\n",
            "['Chí']\n",
            "['Chí', 'Minh']\n",
            "['Việt', 'Nam']\n",
            "<class 'polyglot.text.Chunk'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KEqcsfT049pK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "f236c6ce-9137-4bba-b669-6910b698a215"
      },
      "source": [
        "# Create the list of tuples: entities\n",
        "entities = [(ent.tag, ' '.join(ent)) for ent in txt.entities]\n",
        "\n",
        "# Print entities\n",
        "print(entities)"
      ],
      "execution_count": 127,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('I-LOC', 'Châu Hoàng'), ('I-LOC', 'Hồ Chí Minh'), ('I-LOC', 'Bảo'), ('I-LOC', 'Việt Nam'), ('I-PER', 'Phú Trọng'), ('I-PER', 'Chí'), ('I-LOC', 'Chí Minh'), ('I-LOC', 'Việt Nam')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbJARaA67r4N",
        "colab_type": "text"
      },
      "source": [
        "### Chinese (Mandarin) NER with polyglot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "54bhRZVp8ICX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7c8e02f6-ade7-4264-ad5a-14c84a865e71"
      },
      "source": [
        "!polyglot download LANG:gan\n",
        "!polyglot download LANG:zh\n",
        "!polyglot download LANG:zhc\n",
        "!polyglot download LANG:zhw"
      ],
      "execution_count": 128,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[polyglot_data] Downloading collection 'LANG:gan'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sgns2.gan to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sgns2.gan is already up-to-date!\n",
            "[polyglot_data]    | Downloading package counts2.gan to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package counts2.gan is already up-to-date!\n",
            "[polyglot_data]    | Downloading package embeddings2.gan to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.gan is already up-to-date!\n",
            "[polyglot_data]    | Downloading package sentiment2.gan to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.gan is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.gan to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.gan is already up-to-date!\n",
            "[polyglot_data]    | Downloading package morph2.gan to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package morph2.gan is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:gan\n",
            "[polyglot_data] Downloading collection 'LANG:zh'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sgns2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sgns2.zh is already up-to-date!\n",
            "[polyglot_data]    | Downloading package unipos.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package ner2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package ner2.zh is already up-to-date!\n",
            "[polyglot_data]    | Downloading package counts2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package counts2.zh is already up-to-date!\n",
            "[polyglot_data]    | Downloading package transliteration2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package transliteration2.zh is already up-to-\n",
            "[polyglot_data]    |       date!\n",
            "[polyglot_data]    | Downloading package embeddings2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.zh is already up-to-date!\n",
            "[polyglot_data]    | Downloading package uniemb.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    | Downloading package sentiment2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.zh is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.zh is already up-to-date!\n",
            "[polyglot_data]    | Downloading package morph2.zh to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package morph2.zh is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:zh\n",
            "[polyglot_data] Downloading collection 'LANG:zhc'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package embeddings2.zhc to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package embeddings2.zhc is already up-to-date!\n",
            "[polyglot_data]    | Downloading package tsne2.zhc to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package tsne2.zhc is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:zhc\n",
            "[polyglot_data] Downloading collection 'LANG:zhw'\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]    | Downloading package sentiment2.zhw to\n",
            "[polyglot_data]    |     /root/polyglot_data...\n",
            "[polyglot_data]    |   Package sentiment2.zhw is already up-to-date!\n",
            "[polyglot_data]    | \n",
            "[polyglot_data]  Done downloading collection LANG:zhw\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IemtRCSQ-GbW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "blob = u\"\"\"\n",
        "我叫祝优冉。我住在北京。在华为工作。中国的现任主席是习近平。毛泽东是中国的第一代最高领导人。\n",
        "\"\"\"\n",
        "text = Text(blob)"
      ],
      "execution_count": 129,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ehCWxU9Y-RJ6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "efad44ad-0671-467a-f4a6-85db1922625d"
      },
      "source": [
        "print(text.language)"
      ],
      "execution_count": 130,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "name: Chinese     code: zh       confidence:  99.0 read bytes:  1960\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CB1D7y-N7sNU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "article = \"我叫祝优冉。我住在北京。在华为工作。中国的现任主席是习近平。毛泽东是中国的第一代最高领导人\""
      ],
      "execution_count": 134,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_ssk3eJApdy",
        "colab_type": "text"
      },
      "source": [
        "**English translation:** \"My name is Yoran Zhu. I live in Beijing. I work for this company called Huawei. The head of Chinese Government is Xi Jinping. Mao Zedong is the founding father of China.\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "niuZF4W97skF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "5ee8f7c2-9d49-4e1e-a091-c6103f1c7691"
      },
      "source": [
        "# Create a new text object using Polyglot's Text class: txt\n",
        "txt = Text(article)\n",
        "\n",
        "# Print each of the entities found\n",
        "for ent in txt.entities:\n",
        "    print(ent)\n",
        "    \n",
        "# Print the type of ent\n",
        "print(type(ent))"
      ],
      "execution_count": 140,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'polyglot.text.Chunk'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Cu_jZtN8FHj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "bfc1cfef-aa0a-4d2f-e0cc-332a81d7888a"
      },
      "source": [
        "# Create the list of tuples: entities\n",
        "entities = [(ent.tag, ' '.join(ent)) for ent in txt.entities]\n",
        "\n",
        "# Print entities\n",
        "print(entities)"
      ],
      "execution_count": 141,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BapebfQDOVHn",
        "colab_type": "text"
      },
      "source": [
        "**NOTE:** For Chinese literals pyplot library successfully identifies the language but is unable to parse the identifiers."
      ]
    }
  ]
}