TMRetrieval-FULL-Optimised-Ranking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Translation Memory Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Preprocessing is a separate module and must be done before using this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import nltk\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/khannatanmai/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop_words = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I want to safely remove THE selected drive if it is POSSIBLE\n",
      "['want', 'safely', 'remove', 'selected', 'drive', 'possible']\n"
     ]
    }
   ],
   "source": [
    "input_line = input()\n",
    "\n",
    "#convert input to lowercase\n",
    "input_line = input_line.lower()\n",
    "\n",
    "#tokenise\n",
    "input_tokens = word_tokenize(input_line)\n",
    "\n",
    "content_words = [word for word in input_tokens if word not in stop_words] #Removing Stopwords\n",
    "\n",
    "print(content_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edit Distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load TM\n",
    "\n",
    "Now we are dealing with the whole file which has ~800000 sentences in the TM. \n",
    "\n",
    "Approach:\n",
    "We take each sentence in the TM and check if any of the content words are present in it. \n",
    "\n",
    "Then we rank these candidate sentences based on how many content words they have in common with the input sentence.\n",
    "\n",
    "We take the top 500 candidate sentences, calculate edit-distance and store it. This way we save time as we don't have to calculate edit distance for each sentence in the TM.\n",
    "\n",
    "Once we have a list of edit distances, we take the N lowest, i.e. N best matches and print from the Target TM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "src_tm_words = [] #Content Words in Source TM\n",
    "\n",
    "with open('../tm_data/tm_src_pp.txt') as src_tm:\n",
    "    line = src_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        line = line.rstrip() #Removing Trailing Whitespace\n",
    "        \n",
    "        words = line.split('\\t')\n",
    "        src_tm_words.append(words)\n",
    "        \n",
    "        line = src_tm.readline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ranking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[733266, 734127, 750193, 765606, 766613, 213568, 216242, 360814, 533620, 208]\n",
      "8013 got reduced to 500\n"
     ]
    }
   ],
   "source": [
    "match_counts = []\n",
    "match_indices = []\n",
    "\n",
    "i = 1\n",
    "for candidate in src_tm_words:\n",
    "    \n",
    "    count_for_candidate = 0\n",
    "    flag = 0 \n",
    "    \n",
    "    for word in content_words:\n",
    "        if(word in candidate):\n",
    "            count_for_candidate += 1 #adding count of input content words in candidate\n",
    "            flag = 1 #to avoid adding same index multiple times\n",
    "    \n",
    "    if (flag == 1): #only add for candidates which have at least one input content word\n",
    "        match_indices.append(i)\n",
    "        match_counts.append(count_for_candidate)\n",
    "    \n",
    "    i += 1\n",
    "    \n",
    "#print(len(match_counts))\n",
    "\n",
    "#Sorting match_indices based on match_counts\n",
    "sorted_indices = [x for _,x in sorted(zip(match_counts,match_indices))]\n",
    "\n",
    "#Now we want the top 500 ranks (or if less than 500 matches, then all of them)\n",
    "final_indices = []\n",
    "\n",
    "if(len(sorted_indices) <= 500):\n",
    "    final_indices = sorted_indices\n",
    "else:\n",
    "    final_indices = sorted_indices[-500:] #Only put top 500 ranks in final indices\n",
    "    \n",
    "print(final_indices[-10:]) #Show top 10 ranks\n",
    "print(str(len(match_counts)) + ' got reduced to ' + str(len(final_indices)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute Edit Distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Edit Distance on 500 Ranked Candidates out of a possible 772820!\n",
      "\n",
      "208 ['safely', 'remove', 'selected', 'drive'] 2\n",
      "216242 ['safely', 'remove', 'drive'] 3\n",
      "266397 ['detect', 'media', 'selected', 'drive'] 4\n",
      "255472 ['remove', 'selected', 'snippet'] 4\n",
      "500634 ['remove', 'selected', 'items'] 4\n"
     ]
    }
   ],
   "source": [
    "N = 5 #Top N matches returned\n",
    "\n",
    "edit_distance_all = []\n",
    "indices_all = []\n",
    "\n",
    "for index_tm in final_indices:\n",
    "    ed = nltk.edit_distance(content_words, src_tm_words[index_tm - 1]) #Calculate Edit Distance (src_tm_words is 0 indexed)\n",
    "            \n",
    "    edit_distance_all.append(ed)\n",
    "    indices_all.append(index_tm)\n",
    "    \n",
    "print('Running Edit Distance on ' + str(len(final_indices)) + ' Ranked Candidates out of a possible ' + str(len(src_tm_words)) + '!\\n')\n",
    "    \n",
    "#Get top N results\n",
    "edit_distance_all = np.array(edit_distance_all)\n",
    "\n",
    "sorted_indices = np.argsort(edit_distance_all) #Sorts in ascending order and returns the indices of indices_all array\n",
    "least_N_indices = sorted_indices[:N] #We want least edit distance\n",
    "\n",
    "#print(least_N_indices)\n",
    "\n",
    "for i in least_N_indices:\n",
    "    print(indices_all[i], src_tm_words[indices_all[i]-1], edit_distance_all[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval of Target from TM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "208 चयनित ड्राइव सुरक्षित रूप से निकालें\n",
      "\n",
      "216242 क्षैतिज विभाजन और स्तर विन्यास के वास्तविक अभिलोपन की लगभग सीमा तक दमन के ऐसे परिप्रेक्ष्य में शैली समाधान नहीं कर सकी .\n",
      "\n",
      "266397 प्रत्येक संचालक (कीपर) ने एक खास हाथी का पीछा किया।\n",
      "\n",
      "255472 क्या लोगों ने यह समझ रखा है कि वे इतना कह देने मात्र से छोड़ दिए जाएँगे कि हम ईमान लाए और उनकी परीक्षा न की जाएगी?\n",
      "\n",
      "500634 सामान्य तौर पर वायु गुणवत्ता मानक दो तरह के होते हैं. मानकों की प्रथम श्रेणी (जैसे अमेरिकन राष्ट्रीय परिवेश वायु गुणवत्ता मानक (National Ambient Air Quality Standards)) विशिष्ट प्रदूषकों के लिए अधिकतम सांद्रता निर्धारित करता है.) पर्यावरण एजेंसियां नियम अधिनियमित करती है जिनसे अपेक्षा होती है की इनसे लक्षित स्तर प्राप्त होंगे. दूसरी श्रेणी ( जैसे की उत्तर अमेरिका का वायु गुणवत्ता सूचकांक (Air Quality Index)) जो विभिन्न सीमाओं के साथ एक पैमाने का रूप ले लेता है जिसे जनता को बाहरी गतिविधि से सम्बद्ध जोखिमों से अवगत कराने के लिया उपयोग में लाया जाता है. यह पैमाना विभिन्न प्रदूषकों के बीच भेद कर भी सकता है और नहीं भी कर सकता है.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tgt_tm_array = []\n",
    "\n",
    "with open('../tm_data/tm_tgt.txt') as tgt_tm:\n",
    "    line = tgt_tm.readline()\n",
    "    \n",
    "    while line:\n",
    "        tgt_tm_array.append(line)\n",
    "        line = tgt_tm.readline()\n",
    "        \n",
    "for i in least_N_indices:\n",
    "    print(indices_all[i], tgt_tm_array[indices_all[i]-1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}