TMRetrieval-FULL-Optimised-MEMORY.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Translation Memory Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Preprocessing is a separate module and must be done before using this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import nltk\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/khannatanmai/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop_words = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I want to safely remove THE selected drive if it is POSSIBLE\n",
      "['want', 'safely', 'remove', 'selected', 'drive', 'possible']\n"
     ]
    }
   ],
   "source": [
    "input_line = input()\n",
    "\n",
    "#convert input to lowercase\n",
    "input_line = input_line.lower()\n",
    "\n",
    "#tokenise\n",
    "input_tokens = word_tokenize(input_line)\n",
    "\n",
    "content_words = [word for word in input_tokens if word not in stop_words] #Removing Stopwords\n",
    "\n",
    "print(content_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edit Distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load TM\n",
    "\n",
    "Now we are dealing with the whole file which has ~800000 sentences in the TM. \n",
    "\n",
    "Approach:\n",
    "We take a chunk of size N bytes and in that, We take each sentence in the TM and check if any of the content words are present in it. If they are, we then calculate edit-distance and store it. This way we save time as we don't have to calculate edit distance for each sentence in the TM.\n",
    "\n",
    "This helps with huge files and is a scalable option, unlike the last option which would fail above a certain memory.\n",
    "\n",
    "Once we have a list of edit distances, we take the N lowest, i.e. N best matches and print from the Target TM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_in_chunks(file_object, chunk_size):\n",
    "    \"\"\"Lazy function (generator) to read a file piece by piece.\"\"\"\n",
    "    while True:\n",
    "        data = file_object.read(chunk_size)\n",
    "        if not data:\n",
    "            break\n",
    "        yield data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Edit Distance on 8013 Candidates out of a possible 772820!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "N = 5 #Top N matches returned\n",
    "csize = 3000000 #Chunk Size (In Bytes) ~10000 sentences in a chunk(assuming 30 bytes per sentence)\n",
    "\n",
    "edit_distance_all = []\n",
    "indices_all = []\n",
    "\n",
    "add_to_next_chunk = ''\n",
    "\n",
    "with open('../tm_data/tm_src_pp.txt') as f: #f = sys.stdin\n",
    "    last_noun = ''\n",
    "    \n",
    "    count = 0\n",
    "    j = 0\n",
    "    \n",
    "    for piece in read_in_chunks(f, csize):\n",
    "\n",
    "        piece = add_to_next_chunk + piece #Adding cut info from last chunk\n",
    "\n",
    "        i = len(piece) - 1\n",
    "\n",
    "        while(i >= 0): #looking for the last endline before the chunk ends\n",
    "            if(piece[i] == '\\n'):\n",
    "                break\n",
    "\n",
    "            i -= 1\n",
    "\n",
    "        add_to_next_chunk = piece[i+1:] #Info for next chunk\n",
    "        \n",
    "        piece = piece[:i] #Remove part that has gone to next chunk\n",
    "        piece = piece.rstrip()\n",
    "        \n",
    "        #Now, analysing this piece to get content words in Source TM\n",
    "        \n",
    "        src_tm = piece.split('\\n')\n",
    "        \n",
    "        src_tm_words = [] #Content Words in Source TM\n",
    "        \n",
    "        for candidate in src_tm:\n",
    "            words = candidate.split('\\t') #Tab Separated TM\n",
    "            \n",
    "            src_tm_words.append(words)\n",
    "            \n",
    "        #Finding out which of these have the content words and calculating Edit Distance\n",
    "        \n",
    "        for candidate in src_tm_words:\n",
    "            \n",
    "            #Check if Content Words present in Candidate\n",
    "            for word in content_words:\n",
    "                if(word in candidate):\n",
    "                    count += 1\n",
    "                    #print(candidate)\n",
    "\n",
    "                    ed = nltk.edit_distance(content_words, candidate) #Calculate Edit Distance only if content words exist\n",
    "\n",
    "                    edit_distance_all.append(ed)\n",
    "                    indices_all.append(j)\n",
    "\n",
    "                    break\n",
    "\n",
    "            j += 1\n",
    "    \n",
    "    print('Running Edit Distance on ' + str(count) + ' Candidates out of a possible ' + str(j) + '!\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Matches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[207, 216241, 649639, 191950, 724125]\n"
     ]
    }
   ],
   "source": [
    "#Get top N results\n",
    "edit_distance_all = np.array(edit_distance_all)\n",
    "\n",
    "sorted_indices = np.argsort(edit_distance_all) #Sorts in ascending order and returns the indices of indices_all array\n",
    "least_N_indices = sorted_indices[:N] #We want least edit distance\n",
    "\n",
    "#print(least_N_indices)\n",
    "\n",
    "final_indices = []\n",
    "\n",
    "for i in least_N_indices:\n",
    "    final_indices.append(indices_all[i])\n",
    "    \n",
    "print(final_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval of Target from TM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "208 चयनित ड्राइव सुरक्षित रूप से निकालें\n",
      "\n",
      "216242 क्षैतिज विभाजन और स्तर विन्यास के वास्तविक अभिलोपन की लगभग सीमा तक दमन के ऐसे परिप्रेक्ष्य में शैली समाधान नहीं कर सकी .\n",
      "\n",
      "649640 पिछले साल इसने कुल 17.7 मिलियन का यात्री प्रवाह देखा और वर्ष 2017 के आरंभिक 6 महीनों में\n",
      "\n",
      "191951 वर्ष 1968 में तीन कॉलेजों का विलय किया गया था और इसे प्रौद्योगिकी संस्थान का नाम दिया गया ताकि शैक्षिक और प्रशासनिक निर्णयों को बेहतर परिदृश्य देने के लिए अधिक स्वायत्तता दी जा सके।\n",
      "\n",
      "724126 #rcorners7 {\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tgt_tm_array = []\n",
    "\n",
    "with open('../tm_data/tm_tgt.txt') as tgt_tm:\n",
    "    line = tgt_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        tgt_tm_array.append(line)\n",
    "        line = tgt_tm.readline()\n",
    "        \n",
    "for i in final_indices:\n",
    "    print(i+1, tgt_tm_array[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}