TMRetrieval-FULL-Optimised.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Translation Memory Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Preprocessing is a separate module and must be done before using this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import nltk\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/khannatanmai/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop_words = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I want to safely remove THE selected drive if it is POSSIBLE\n",
      "['want', 'safely', 'remove', 'selected', 'drive', 'possible']\n"
     ]
    }
   ],
   "source": [
    "input_line = input()\n",
    "\n",
    "#convert input to lowercase\n",
    "input_line = input_line.lower()\n",
    "\n",
    "#tokenise\n",
    "input_tokens = word_tokenize(input_line)\n",
    "\n",
    "content_words = [word for word in input_tokens if word not in stop_words] #Removing Stopwords\n",
    "\n",
    "print(content_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edit Distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load TM\n",
    "\n",
    "Now we are dealing with the whole file which has ~800000 sentences in the TM. \n",
    "\n",
    "Approach:\n",
    "We take each sentence in the TM and check if any of the content words are present in it. If they are, we then calculate edit-distance and store it. This way we save time as we don't have to calculate edit distance for each sentence in the TM.\n",
    "\n",
    "Once we have a list of edit distances, we take the N lowest, i.e. N best matches and print from the Target TM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "src_tm_words = [] #Content Words in Source TM\n",
    "\n",
    "with open('../tm_data/tm_src_pp.txt') as src_tm:\n",
    "    line = src_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        line = line.rstrip() #Removing Trailing Whitespace\n",
    "        \n",
    "        words = line.split('\\t')\n",
    "        src_tm_words.append(words)\n",
    "        \n",
    "        line = src_tm.readline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute Edit Distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Edit Distance on 8013 Candidates out of a possible 772820!\n",
      "\n",
      "208 ['safely', 'remove', 'selected', 'drive'] 2\n",
      "216242 ['safely', 'remove', 'drive'] 3\n",
      "83954 ['remove', 'selected', 'toolbar'] 4\n",
      "315127 ['disconnect', 'selected', 'drive'] 4\n",
      "385361 ['connect', 'selected', 'drive'] 4\n"
     ]
    }
   ],
   "source": [
    "N = 5 #Top N matches returned\n",
    "\n",
    "edit_distance_all = []\n",
    "indices_all = []\n",
    "\n",
    "i = 0\n",
    "count = 0\n",
    "\n",
    "for candidate in src_tm_words:\n",
    "    \n",
    "    #Check if Content Words present in Candidate\n",
    "    for word in content_words:\n",
    "        if(word in candidate):\n",
    "            count += 1\n",
    "            #print(candidate)\n",
    "            \n",
    "            ed = nltk.edit_distance(content_words, candidate) #Calculate Edit Distance only if content words exist\n",
    "            \n",
    "            edit_distance_all.append(ed)\n",
    "            indices_all.append(i)\n",
    "            \n",
    "            break\n",
    "    \n",
    "    i += 1\n",
    "    \n",
    "print('Running Edit Distance on ' + str(count) + ' Candidates out of a possible ' + str(i) + '!\\n')\n",
    "    \n",
    "#Get top N results\n",
    "edit_distance_all = np.array(edit_distance_all)\n",
    "\n",
    "sorted_indices = np.argsort(edit_distance_all) #Sorts in ascending order and returns the indices of indices_all array\n",
    "least_N_indices = sorted_indices[:N] #We want least edit distance\n",
    "\n",
    "#print(least_N_indices)\n",
    "\n",
    "for i in least_N_indices:\n",
    "    print(indices_all[i]+1, src_tm_words[indices_all[i]], edit_distance_all[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval of Target from TM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "208 चयनित ड्राइव सुरक्षित रूप से निकालें\n",
      "\n",
      "216242 क्षैतिज विभाजन और स्तर विन्यास के वास्तविक अभिलोपन की लगभग सीमा तक दमन के ऐसे परिप्रेक्ष्य में शैली समाधान नहीं कर सकी .\n",
      "\n",
      "83954 क्या गलत हो सकता है?\n",
      "\n",
      "315127 गुरुदेव को वे समझ सके स्वयं के अस्तित्व को गुरुदेव के स्वरूप में विलीन कर सके ।\n",
      "\n",
      "385361 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "tgt_tm_array = []\n",
    "\n",
    "with open('../tm_data/tm_tgt.txt') as tgt_tm:\n",
    "    line = tgt_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        tgt_tm_array.append(line)\n",
    "        line = tgt_tm.readline()\n",
    "        \n",
    "for i in least_N_indices:\n",
    "    print(indices_all[i]+1, tgt_tm_array[indices_all[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}