bag_of_words.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Bag of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebooks helps to understand the basics of processing text data and representation. \n",
    "\n",
    "The first part explores the naive way of processing a dataset without use of any pre-processing tools, techniques and libraries.\n",
    "`data` is a list containing three sentences (marked by commas). The words are identified by whitespace characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ['The boy has a green bird',\n",
    "              'The bird eats a duck',\n",
    "              'The duck eats a worm']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "string.split() function splits the string by the whitespace characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents = [sent.split() for sent in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extracting vocabularies from the sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vocabulary is the list of non-repeating words that are present in the whole dataset. In the below example, `allwords` has all the words that are present in the dataset, but we need to remove the repetition. `set()` obtains the unique words in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allwords = [word for sent in sents for word in sent]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = list(set(allwords))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets compute bag of words. i.e, a matrix between vocabulary and the sentences in the dataset, where the counts of the vocabulary present in each sentences are filled in the matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents_vec = [[sent.count(v) for v in vocab ] for sent in sents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents_vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vocab)\n",
    "print(\"S1 : \", sents_vec[0])\n",
    "print(\"S2 : \", sents_vec[1])\n",
    "print(\"S3 : \", sents_vec[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great!. But this is an inefficient method for computing vocabularies and occurrences. So, We could make use of the `CountVectorizer()` from `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = cv.fit_transform(data) #This command computes vocabulary and occurrences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv.get_feature_names() # vocabulary of the term-document matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i,x in enumerate(matrix):\n",
    "    print(f\"sentence {i}\")\n",
    "    print(x)\n",
    "    \n",
    "# It is important to note that the letter a is removed by the CountVectorizer because \n",
    "# by default minimum 2 character words are counted as words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practical cases, the real world data does not contain clear separation of words and sentences. There is complexity in grammar, writing styles, etc. Thus whitespaces are not good enough to separate words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "foo = \"Oh God! I haven't saved any of it's responses\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "[w for w in foo.split()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenization is a process of breaking down a text into smaller chunks be it word or sentence depending upon the level of tokenization.\n",
    "\n",
    "`nltk` is short for natural language toolkit. It is one of the classical libraries used for text processing. nltk provides functions for words and sentence tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "foo = \"Oh God!\\n I haven't saved any of it's responses\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(word_tokenize(foo))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apart from general word tokenizers, there are also specific tokenizers such as for tokenizing tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "footweet = \"@Tomato Wassssssssssup man, that's so cooool\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import TweetTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_tokenize(footweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TweetTokenizer(strip_handles=True, reduce_len=True).tokenize(footweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to tokenizing words, there are also sentence tokenizers, which helps us to slice sentences in a paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import sent_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bar = \"Sent tokenize knows that time period from 10 a.m. to 1 p.m. are not sentence boundaries. neither are the names G.H.Hardy and J.J.Thompson. you can even start the sentence without Caps\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent_tokenize(bar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text data has lots of information, but based on the use case one could simplify the complex information leading to reduced space and time complexity. For example words like bike, biking, biked provides the same meaning varied in tenses. When we don't need such information they could be stripped to its root word using two NLP techniques, Stemming and Lemmatization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming & Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import PorterStemmer, SnowballStemmer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "foo = \"My baby and all other babies in my buildings are crying laughing and playing all the time\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print([ps.stem(tok) for tok in foo.split()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could see that babies is shortened to `babi`, which is not really an english word, but as long as suffixes of `babi-` are grouped as a single token, it helps in easy processing. But what if we need to be still grammatically correct and trade off with the speed of calculation. In that case we have to use Lemmatization. It is grammatically accurate and it is computationally expensive, since it loads a lemmatizer model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wnl = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bar = \"Those people were crying and running every day\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[wnl.lemmatize(tok, pos=\"v\") for tok in bar.split()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Punctuation removal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eggs = '''Phew! After lots of <br>, I finally cleared all these \"English\" and \"Science\" exams'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "string.punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "punctuation_map = str.maketrans('', '', string.punctuation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "str.maketrans?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eggs.translate(punctuation_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!Exercise\n",
    "\n",
    "Try to apply all that we have learnt with the following paragraph. Tokenize the sentences, words, Remove punctuations, Apply stemming or lemmatization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "foobar = '''With a key sentence established, the next task is to shape the body of your paragraph to be both cohesive and coherent. As Williams and Bizup[3] explain, cohesion is about the “sense of flow” (how each sentence fits with the next), while coherence is about the “sense of the whole.”[4]\n",
    "To reiterate the initial point, it is useful to think of paragraphs as punctuation that organize your ideas in a readable way. Each paragraph should be an irreplaceable node within a coherent sequence of logic. Thinking of paragraphs as “building blocks” evokes the “five-paragraph theme” structure explained earlier: if you have identical stone blocks, it hardly matters what order they’re in.'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}