TMRetrieval-FULL-Optimised-TEST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Translation Memory Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Preprocessing is a separate module and must be done before using this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import nltk\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/khannatanmai/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop_words = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I want to safely remove THE selected drive if it is POSSIBLE\n",
      "['want', 'safely', 'remove', 'selected', 'drive', 'possible']\n"
     ]
    }
   ],
   "source": [
    "input_line = input()\n",
    "\n",
    "#convert input to lowercase\n",
    "input_line = input_line.lower()\n",
    "\n",
    "#tokenise\n",
    "input_tokens = word_tokenize(input_line)\n",
    "\n",
    "content_words = [word for word in input_tokens if word not in stop_words] #Removing Stopwords\n",
    "\n",
    "print(content_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edit Distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load TM\n",
    "\n",
    "Now we are dealing with the whole file which has ~800000 sentences in the TM. \n",
    "\n",
    "Approach:\n",
    "We take each sentence in the TM and check if any of the content words are present in it. If they are, we then calculate edit-distance and store it. This way we save time as we don't have to calculate edit distance for each sentence in the TM.\n",
    "\n",
    "Once we have a list of edit distances, we take the N lowest, i.e. N best matches and print from the Target TM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "src_tm_words = [] #Content Words in Source TM\n",
    "src_tm_tokenized = []\n",
    "\n",
    "with open('../tm_data/tm_src_pp.txt') as src_tm:\n",
    "    line = src_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        line = line.rstrip() #Removing Trailing Whitespace\n",
    "        \n",
    "        words = line.split('\\t')\n",
    "        src_tm_words.append(words)\n",
    "        \n",
    "        line = src_tm.readline()\n",
    "\n",
    "with open('../tm_data/tm_src_lower.txt') as org_src_tm:\n",
    "    line = org_src_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        tokens = word_tokenize(line)\n",
    "        src_tm_tokenized.append(tokens)\n",
    "        line = org_src_tm.readline()\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute Edit Distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 5 #Top N matches returned\n",
    "\n",
    "edit_distance_all = []\n",
    "indices_all = []\n",
    "\n",
    "i = 0\n",
    "count = 0\n",
    "\n",
    "for candidate in src_tm_words:\n",
    "    \n",
    "    #Check if Content Words present in Candidate\n",
    "    for word in content_words:\n",
    "        if(word in candidate):\n",
    "            count += 1\n",
    "            #print(candidate)\n",
    "            \n",
    "            ed = nltk.edit_distance(input_tokens, src_tm_tokenized[i]) #Calculate Edit Distance only if content words exist\n",
    "            \n",
    "            edit_distance_all.append(ed)\n",
    "            indices_all.append(i)\n",
    "            \n",
    "            break\n",
    "    \n",
    "    i += 1\n",
    "    \n",
    "print('Running Edit Distance on ' + str(count) + ' Candidates out of a possible ' + str(i) + '!\\n')\n",
    "    \n",
    "#Get top N results\n",
    "edit_distance_all = np.array(edit_distance_all)\n",
    "\n",
    "sorted_indices = np.argsort(edit_distance_all) #Sorts in ascending order and returns the indices of indices_all array\n",
    "least_N_indices = sorted_indices[:N] #We want least edit distance\n",
    "\n",
    "#print(least_N_indices)\n",
    "\n",
    "for i in least_N_indices:\n",
    "    print(indices_all[i]+1, src_tm_words[indices_all[i]], edit_distance_all[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval of Target from TM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tgt_tm_array = []\n",
    "\n",
    "with open('../tm_data/tm_tgt.txt') as tgt_tm:\n",
    "    line = tgt_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        tgt_tm_array.append(line)\n",
    "        line = tgt_tm.readline()\n",
    "        \n",
    "for i in least_N_indices:\n",
    "    print(indices_all[i]+1, tgt_tm_array[indices_all[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}