Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation \n",
    "\n",
    "This notebooks provides details on our two experiments as described in Section 5 in the paper, each illustrates one potential source of misalignement. More specifically, we simulate two realistic scenarios: \n",
    "\n",
    "- Experiment A (Translated communication): we consider the case of communication over an imperfect MT system \n",
    "- Experiment B (Misaligned perception): we consider cases when the sender and the listener have different perceptions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import spacy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from collections import defaultdict\n",
    "\n",
    "from perception_utils import estimate_perception\n",
    "\n",
    "from convokit import Corpus, Utterance, Speaker\n",
    "from convokit import TextParser, PolitenessStrategies\n",
    "\n",
    "from bleu import get_bleu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Preliminary \n",
    "\n",
    "We first load the two test corpuses, `mt-test-corpus` and `ind-test-corpus`, as well as the politeness perception models that we will use to estimate the perceived level of politeness. More details about them can be found in [Evaluation_Data.ipynb](Evaluation_Data.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "mt_corpus = Corpus(filename=\"data/test/mt-test-corpus\")\n",
    "ind_corpus = Corpus(filename=\"data/test/ind-test-corpus/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We consider the following \"individuals\", where 'average' represents a prototypical \"average person\", and the rest, prolific annotators in Stanford Politeness corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [],
   "source": [
    "individuals = ['average', 'A23', 'A2U', 'A1F', 'A3S', 'AYG']\n",
    "\n",
    "perception_models = defaultdict()\n",
    "\n",
    "for ind in individuals:\n",
    "    with open(\"data/perceptions/{}.json\".format(ind), 'r') as f:\n",
    "        perception_models[ind] = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We consider the following strategies as at-risk for Experiment A:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [],
   "source": [
    "at_risk_strategies = {'Subjunctive', \"Please\", 'Filler', \"Swearing\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Default perception gaps\n",
    "\n",
    "We first compute the default level of misalignement in perception, i.e., the difference between the (predicted) politeness level perceived by the _sender_ and the (predicted) politeness level preceived by the _receiver_, when no intervention is in place. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = PolitenessStrategies(strategy_attribute_name = \"strategies\", \\\n",
    "                          marker_attribute_name = \"markers\", \\\n",
    "                          strategy_collection=\"politeness_local\")\n",
    "\n",
    "spacy_nlp = spacy.load('en', disable=['ner'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first identify politeness strategies used in these utterances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get politeness strategies for the test utterances \n",
    "mt_corpus = ps.transform(mt_corpus, markers=True)\n",
    "\n",
    "# get politeness strategies for the back translations \n",
    "translated_strategies = {utt.id: ps.transform_utterance(utt.meta['back_translation'], \\\n",
    "                                                        spacy_nlp=spacy_nlp, markers=True) for utt in mt_corpus.iter_utterances()}\n",
    "\n",
    "ind_corpus = ps.transform(ind_corpus, markers=True)\n",
    "\n",
    "# for readability, we save the strategy set present as seperate meta data \n",
    "for corpus in [mt_corpus, ind_corpus]:\n",
    "    \n",
    "    for utt in corpus.iter_utterances():\n",
    "        \n",
    "        utt.meta['strategy_set'] = {k for k,v in utt.meta['strategies'].items() if v ==1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then compute the intended and perceived levels of politeness with the perception models used by the sender and the receiver. In the case of translated communication, we take the back-translation as an approximation for the \"no intervention\" scenario. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use the \"average person\" model as both the sender and the receiver\n",
    "average_model = perception_models['average']\n",
    "\n",
    "for utt in mt_corpus.iter_utterances():\n",
    "    \n",
    "    strategies_used = utt.meta['strategy_set']\n",
    "    strategies_preserved = {k for k, v in translated_strategies[utt.id].meta['strategies'].items() if v == 1}\n",
    "    \n",
    "    utt.meta['intended_politeness'] = estimate_perception(average_model, strategies_used)\n",
    "    utt.meta['perceived_politeness'] = estimate_perception(average_model, strategies_preserved)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "for utt in ind_corpus.iter_utterances():\n",
    "    \n",
    "    strategies_used = utt.meta['strategy_set']\n",
    "    \n",
    "    sender_model = perception_models[utt.meta['sender']]\n",
    "    receiver_model = perception_models[utt.meta['receiver']]\n",
    "    \n",
    "    utt.meta['intended_politeness'] = estimate_perception(sender_model, strategies_used)\n",
    "    utt.meta['perceived_politeness'] = estimate_perception(receiver_model, strategies_used)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By comparing these politeness perception estimates, we obtain the default perception gaps. Specifically, we compute the mean absolute difference/error (MAE) between the (estimated) intended and perceived politeness, which we aim to reduce with our paraphrasing approach. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE: No intervention\n",
      "\t Translated communication: 0.43\n",
      "\t Translated communication: 1.01\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE: No intervention\")\n",
    "\n",
    "print(\"\\t Translated communication: {:.2f}\".format(np.mean([abs(utt.meta['intended_politeness'] - utt.meta['perceived_politeness']) \\\n",
    "                                                            for utt in mt_corpus.iter_utterances()])))\n",
    "      \n",
    "print(\"\\t Translated communication: {:.2f}\".format(np.mean([abs(utt.meta['intended_politeness'] - utt.meta['perceived_politeness']) \\\n",
    "                                                            for utt in ind_corpus.iter_utterances()])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Effect of planning "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We propose to plan for an optimal strategy combination under the circumstance with Integer Linear Programming (ILP). This ILP-based planning is compared with two other baselines. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Planning with ILP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [],
   "source": [
    "import imp\n",
    "import plan_with_ilp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "from plan_with_ilp import get_ilp_solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "800it [01:40,  7.95it/s]\n"
     ]
    }
   ],
   "source": [
    "# For Experiment A, we consider average perception models, and account for the set of at-risk strategies\n",
    "mt_ilp_solutions = {utt.id: get_ilp_solution(utt.id, utt.meta['strategy_set'], \\\n",
    "                                          average_model, average_model, \\\n",
    "                                          at_risk_strategies) for utt in tqdm(mt_corpus.iter_utterances())}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2000it [01:16, 26.03it/s]\n"
     ]
    }
   ],
   "source": [
    "# For Experiment B, we consider individualized perception models, and all strategies are considered safe\n",
    "ind_ilp_solutions = {utt.id: get_ilp_solution(utt.id, utt.meta['strategy_set'], \\\n",
    "                                          perception_models[utt.meta['sender']], \\\n",
    "                                          perception_models[utt.meta['receiver']], set()) for utt in tqdm(ind_corpus.iter_utterances())}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Planning baselines \n",
    "\n",
    "We consider two baseline approaches in planning and compare them with our proposed ILP-based approach. We compare the effects of these different planning approaches by the extent to which they reduce the MAE from the No Intervention case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Greedy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The greedy approach seeks to substitute strategies with ones that are safe to transmit and has the closest strategy strength in the eye of the receiver (if a strategy itself is safe and do not result in different interpretations, it will be left as is). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [],
   "source": [
    "from plan_with_baselines import get_greedy_plan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [],
   "source": [
    "mt_greedy_solutions = {utt.id: get_greedy_plan(utt.meta['strategy_set'], average_model, average_model, \\\n",
    "                                              at_risk_strategies) for utt in mt_corpus.iter_utterances()}\n",
    "\n",
    "ind_greedy_solutions = {utt.id: get_greedy_plan(utt.meta['strategy_set'], \\\n",
    "                                                perception_models[utt.meta['sender']], \\\n",
    "                                                perception_models[utt.meta['receiver']], set()) for utt in ind_corpus.iter_utterances()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The retrieval approach takes the coarser grained view that aims to find, among utterances in the training corpus that exibhit the same poliarity as the input utterance, the most similar one in terms of content as the reference utterance, and adopt the strategy plan used in the reference as the retrieval solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [],
   "source": [
    "from plan_with_baselines import init_transformer, add_similarity_scores, get_retrieval_plan\n",
    "from strategy_manipulation import remove_strategies_from_utt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We only search from reference utterances that has the same polarity as the input\n",
    "# Thus we need information on the polarity of the texts \n",
    "\n",
    "def add_estimated_polarity(utt, perception_model, polarity_attribute_name):\n",
    "    \n",
    "    # 1 = polite, 0 = impolite\n",
    "    polarity = int(estimate_perception(perception_model, utt.meta['strategy_set']) >= 0)\n",
    "    \n",
    "    utt.meta[polarity_attribute_name] = polarity\n",
    "    \n",
    "    return polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading the training (reference) corpus\n",
    "train_corpus = Corpus(filename=\"data/train/training-corpus\")\n",
    "\n",
    "# obtain politeness annotations \n",
    "train_corpus = ps.transform(train_corpus, markers=True)\n",
    "\n",
    "for utt in train_corpus.iter_utterances():\n",
    "    utt.meta['strategy_set'] = {k for k,v in utt.meta['strategies'].items() if v ==1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preparing corpuses: adding polarity, and getting marker-removed contents\n",
    "\n",
    "# for the training corpus and experiment A, we use the average person model to judge polarity  \n",
    "for corpus in [train_corpus, mt_corpus]:\n",
    "    \n",
    "    for utt in corpus.iter_utterances():\n",
    "    \n",
    "        remove_strategies_from_utt(utt, utt.meta['strategy_set'], \\\n",
    "                                   removed_attribute_name=\"content\")\n",
    "\n",
    "        add_estimated_polarity(utt, average_model, \"polarity\")\n",
    "    \n",
    "# for experiment B, we judge polarity from the sender's perspective \n",
    "for utt in ind_corpus.iter_utterances():\n",
    "\n",
    "    remove_strategies_from_utt(utt, utt.meta['strategy_set'], \\\n",
    "                               removed_attribute_name=\"content\")\n",
    "    \n",
    "    add_estimated_polarity(utt, perception_models[utt.meta['sender']], \"polarity\")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[91mWARNING: \u001b[0mVector matrix \"tfidf\" already exists. Overwriting it with newly set vector matrix.\n",
      "\u001b[91mWARNING: \u001b[0mVector matrix \"tfidf\" already exists. Overwriting it with newly set vector matrix.\n"
     ]
    }
   ],
   "source": [
    "# we compare contents of utterances with their tfidf representation  \n",
    "tfidf = init_transformer(train_corpus)\n",
    "\n",
    "# for each utterance in test corpus, we obtain the cosine similarity scores with each utterance in the training corpus\n",
    "mt_corpus = add_similarity_scores(mt_corpus, train_corpus, tfidf)\n",
    "ind_corpus = add_similarity_scores(ind_corpus, train_corpus, tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_ids = train_corpus.get_vector_matrix('tfidf').ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "mt_retrieval_solutions = {utt.id: get_retrieval_plan(utt, train_corpus, train_ids, \\\n",
    "                                                     at_risk_strategies) for utt in mt_corpus.iter_utterances()}\n",
    "\n",
    "ind_retrieval_solutions = {utt.id: get_retrieval_plan(utt, train_corpus, train_ids, \\\n",
    "                                                      set()) for utt in ind_corpus.iter_utterances()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected perception gap with new plans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then add information of different plans into corpus, and compare the aggregated effects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_plan_to_corpus(corpus, solutions, solution_attribute_name):\n",
    "    \n",
    "    for idx, sol in solutions.items():\n",
    "        \n",
    "        corpus.get_utterance(idx).meta[solution_attribute_name] = sol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Experiment A\n",
    "mt_plans = {'ilp_plan': mt_ilp_solutions, 'greedy_plan': mt_greedy_solutions, 'retrieval_plan': mt_retrieval_solutions}\n",
    "\n",
    "for attribute_name, plan in mt_plans.items():\n",
    "    \n",
    "    add_plan_to_corpus(mt_corpus, plan, attribute_name)\n",
    "    \n",
    "# Experiment B\n",
    "ind_plans = {'ilp_plan': ind_ilp_solutions, 'greedy_plan': ind_greedy_solutions, 'retrieval_plan': ind_retrieval_solutions}\n",
    "\n",
    "for attribute_name, plan in ind_plans.items():\n",
    "    \n",
    "    add_plan_to_corpus(ind_corpus, plan, attribute_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment A\n",
      "------------\n",
      "MAE_plan\n",
      "\tRETRIEVAL_PLAN: 0.66\n",
      "\tGREEDY_PLAN: 0.35\n",
      "\tILP_PLAN: 0.14\n",
      "#-ADDED\n",
      "\tRETRIEVAL_PLAN: 1.09\n",
      "\tGREEDY_PLAN: 1.20\n",
      "\tILP_PLAN: 2.38\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment A\")\n",
    "print(\"------------\")\n",
    "\n",
    "print(\"MAE_plan\")\n",
    "for plan in ['retrieval_plan', 'greedy_plan', 'ilp_plan']:\n",
    "    \n",
    "    print('\\t{}: {:.2f}'.format(plan.upper(), np.mean([abs(estimate_perception(average_model, utt.meta[plan]) - \\\n",
    "             utt.meta['intended_politeness']) for utt in mt_corpus.iter_utterances()])))\n",
    "    \n",
    "print(\"#-ADDED\")\n",
    "for plan in ['retrieval_plan', 'greedy_plan', 'ilp_plan']:\n",
    "    \n",
    "    print('\\t{}: {:.2f}'.format(plan.upper(), \\\n",
    "                                np.mean([len(utt.meta[plan] - utt.meta['strategy_set']) for utt in mt_corpus.iter_utterances()])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment B\n",
      "------------\n",
      "MAE_plan\n",
      "\tRETRIEVAL_PLAN: 0.81\n",
      "\tGREEDY_PLAN: 0.48\n",
      "\tILP_PLAN: 0.03\n",
      "#-ADDED\n",
      "\tRETRIEVAL_PLAN: 1.07\n",
      "\tGREEDY_PLAN: 1.82\n",
      "\tILP_PLAN: 2.30\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment B\")\n",
    "print(\"------------\")\n",
    "\n",
    "print(\"MAE_plan\")\n",
    "for plan in ['retrieval_plan', 'greedy_plan', 'ilp_plan']:\n",
    "    print('\\t{}: {:.2f}'.format(plan.upper(), np.mean([abs(estimate_perception(perception_models[utt.meta['receiver']], \\\n",
    "                            utt.meta[plan]) - utt.meta['intended_politeness']) for utt in ind_corpus.iter_utterances()])))\n",
    "    \n",
    "print(\"#-ADDED\")\n",
    "for plan in ['retrieval_plan', 'greedy_plan', 'ilp_plan']:\n",
    "    print('\\t{}: {:.2f}'.format(plan.upper(), \\\n",
    "                                np.mean([len(utt.meta[plan] - utt.meta['strategy_set']) for utt in ind_corpus.iter_utterances()])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Generating paraphrases "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate paraphrases based on the strategy plans, we check strategies that needs to be incorporated, and remove existing strategies that are not part of the plan, forming the _post-deletion context_. \n",
    "\n",
    "For reference, the generation outputs are saved in `output/mt.tsv` and `output/ind.tsv`, which can be directly used for next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from strategy_manipulation import add_strategies_to_utt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# requires strategy marker annotations \n",
    "# and also an already computed plan, stored under \"<plan_prefix>_plan\"\n",
    "\n",
    "def prepare_corpus_for_generation(corpus, plan_prefix):\n",
    "    \n",
    "    for utt in corpus.iter_utterances():\n",
    "        \n",
    "        plan = utt.meta[\"{}_plan\".format(plan_prefix)]\n",
    "        \n",
    "        # strategies to remove\n",
    "        strategies_to_remove = utt.meta['strategy_set'] - plan\n",
    "        remove_strategies_from_utt(utt, strategies_to_remove, \\\n",
    "                                   removed_attribute_name=\"{}_context\".format(plan_prefix))\n",
    "        \n",
    "        # strategies to be added \n",
    "        utt.meta['{}_addition'.format(plan_prefix)] = plan - utt.meta['strategy_set'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "for mode in ['ilp', 'greedy', 'retrieval']:\n",
    "    \n",
    "    prepare_corpus_for_generation(mt_corpus, mode)\n",
    "    prepare_corpus_for_generation(ind_corpus, mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can save the processed corpuses for future reference \n",
    "# mt_corpus.dump('mt-processed-corpus', \\\n",
    "#                exclude_vectors=['tfidf', 'similarity'])\n",
    "\n",
    "# ind_corpus.dump('ind-processed-corpus', \\\n",
    "#                 exclude_vectors=['tfidf', 'similarity'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each test set, we generate the three sets of parapharases according to the corresponding plan:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for prefix in ['ilp', 'greedy', 'retrieval']: \n",
    "\n",
    "    for utt in tqdm(mt_corpus.iter_utterances()):\n",
    "        \n",
    "        add_strategies_to_utt(utt, utt.meta['{}_addition'.format(prefix)], ps, spacy_nlp, \\\n",
    "                          content_attribute_name='{}_context'.format(prefix), \\\n",
    "                          output_attribute_name=\"{}_paraphrase\".format(prefix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for prefix in ['ilp', 'greedy', 'retrieval']: \n",
    "\n",
    "    for utt in tqdm(ind_corpus.iter_utterances()):\n",
    "        \n",
    "        # receiver's perception model is used to select the output that minimizes the perception gap \n",
    "        add_strategies_to_utt(utt, utt.meta['{}_addition'.format(prefix)], ps, spacy_nlp, \\\n",
    "                              perception_model = perception_models[utt.meta['receiver']], \\\n",
    "                              content_attribute_name='{}_context'.format(prefix), \\\n",
    "                              output_attribute_name=\"{}_paraphrase\".format(prefix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "paraphrases_mt = mt_corpus.get_attribute_table(obj_type=\"utterance\", \\\n",
    "                                        attrs=['intended_politeness', 'ilp_paraphrase', 'greedy_paraphrase', 'retrieval_paraphrase'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "paraphrases_ind = ind_corpus.get_attribute_table(obj_type=\"utterance\", \\\n",
    "                                        attrs=['intended_politeness', 'ilp_paraphrase', 'greedy_paraphrase', 'retrieval_paraphrase'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Comparing paraphrases "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the generated outputs directly \n",
    "\n",
    "paraphrases_mt = pd.read_csv('output/mt.tsv', sep='\\t', index_col=0, na_filter=False)\n",
    "paraphrases_ind = pd.read_csv('output/ind.tsv', sep='\\t', index_col=0, na_filter=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first compare the effectiveness of different paraphrases in terms of their power in reducing misalignment between intentions and perceptions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_mae(df, col_name, perception_models, \\\n",
    "                politeness_transformer, spacy_nlp, \\\n",
    "                intended_col = 'intended_politeness', receiver_col = 'receiver'):\n",
    "    \n",
    "    # compute strategies present in each paraphrases\n",
    "    annotated_utts = [politeness_transformer.transform_utterance(text, spacy_nlp) for text in df[col_name]]\n",
    "    \n",
    "    strategy_sets = [{k for k,v in utt.meta['strategies'].items() if v == 1} for utt in annotated_utts]\n",
    "    receivers = df[receiver_col]\n",
    "    \n",
    "    # obtain perceived politeness with strategy set by the receiver \n",
    "    perceived_politeness = [estimate_perception(perception_models[receiver], strategies) \\\n",
    "                        for receiver, strategies in zip(receivers, strategy_sets)]\n",
    "    \n",
    "    return abs(df['intended_politeness'] - perceived_politeness)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment A\n",
      "------------\n",
      "MAE_gen\n",
      "\tRETRIEVAL_PARAPHRASE: 0.61\n",
      "\tGREEDY_PARAPHRASE: 0.35\n",
      "\tILP_PARAPHRASE: 0.21\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment A\")\n",
    "print(\"------------\")\n",
    "\n",
    "print(\"MAE_gen\")\n",
    "\n",
    "results_a = defaultdict()\n",
    "\n",
    "for col in [\"retrieval_paraphrase\", 'greedy_paraphrase', 'ilp_paraphrase']:\n",
    "    \n",
    "    results_a[col] = compute_mae(paraphrases_mt, col, perception_models, ps, spacy_nlp)\n",
    "    \n",
    "    print(\"\\t{}: {:.2f}\".format(col.upper(), np.mean(results_a[col])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment B\n",
      "------------\n",
      "MAE_gen\n",
      "\tRETRIEVAL_PARAPHRASE: 0.76\n",
      "\tGREEDY_PARAPHRASE: 0.47\n",
      "\tILP_PARAPHRASE: 0.12\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment B\")\n",
    "print(\"------------\")\n",
    "\n",
    "print(\"MAE_gen\")\n",
    "\n",
    "results_b = defaultdict()\n",
    "\n",
    "for col in [\"retrieval_paraphrase\", 'greedy_paraphrase', 'ilp_paraphrase']:\n",
    "    \n",
    "    results_b[col] =  compute_mae(paraphrases_ind, col, perception_models, ps, spacy_nlp)\n",
    "    \n",
    "    print(\"\\t{}: {:.2f}\".format(col.upper(), np.mean(results_b[col])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can verify if the difference is statistically significant: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import ttest_ind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment A\n",
      "\tpvalue for the diff between ilp and retrieval: 0.000\n",
      "\tpvalue for the diff between ilp and greedy: 0.000\n",
      "\n",
      "Experiment B\n",
      "\tpvalue for the diff between ilp and retrieval: 0.000\n",
      "\tpvalue for the diff between ilp and greedy: 0.000\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment A\")\n",
    "for baseline in [\"retrieval\", 'greedy']:\n",
    "    stats, pval = ttest_ind(results_a['ilp_paraphrase'], results_a[\"{}_paraphrase\".format(baseline)])\n",
    "    print(\"\\tpvalue for the diff between ilp and {}: {:.3f}\".format(baseline, pval))\n",
    "\n",
    "print()\n",
    "\n",
    "print(\"Experiment B\")\n",
    "for baseline in [\"retrieval\", 'greedy']:\n",
    "    stats, pval = ttest_ind(results_b['ilp_paraphrase'], results_b[\"{}_paraphrase\".format(baseline)])\n",
    "    print(\"\\tpvalue for the diff between ilp and {}: {:.3f}\".format(baseline, pval))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also check the degree to which the contents of the utterances are preserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add original texts to df \n",
    "paraphrases_mt['text'] = [mt_corpus.get_utterance(idx).text for idx in paraphrases_mt.index]\n",
    "paraphrases_mt['back_translation'] = [mt_corpus.get_utterance(idx).meta['back_translation'] for idx in paraphrases_mt.index]\n",
    "\n",
    "paraphrases_ind['text'] = [ind_corpus.get_utterance(idx).text for idx in paraphrases_ind.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bleu import get_bleu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first check how well back-translation is preserving contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64.17123305208725"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_bleu(paraphrases_mt['back_translation'], paraphrases_mt['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment A\n",
      "------------\n",
      "BLUE-s\n",
      "\tRETRIEVAL_PARAPHRASE: 74.7\n",
      "\tGREEDY_PARAPHRASE: 73.5\n",
      "\tILP_PARAPHRASE: 67.0\n",
      "\n",
      "Experiment B\n",
      "------------\n",
      "BLUE-s\n",
      "\tRETRIEVAL_PARAPHRASE: 72.0\n",
      "\tGREEDY_PARAPHRASE: 70.3\n",
      "\tILP_PARAPHRASE: 68.8\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment A\")\n",
    "print(\"------------\")\n",
    "\n",
    "print(\"BLUE-s\")\n",
    "\n",
    "for col in [\"retrieval_paraphrase\", 'greedy_paraphrase', 'ilp_paraphrase']:\n",
    "        \n",
    "    print(\"\\t{}: {:.1f}\".format(col.upper(), get_bleu(paraphrases_mt[col], paraphrases_mt['text'])))\n",
    "    \n",
    "print() \n",
    "\n",
    "print(\"Experiment B\")\n",
    "print(\"------------\")\n",
    "\n",
    "print(\"BLUE-s\")\n",
    "\n",
    "for col in [\"retrieval_paraphrase\", 'greedy_paraphrase', 'ilp_paraphrase']:\n",
    "        \n",
    "    print(\"\\t{}: {:.1f}\".format(col.upper(), get_bleu(paraphrases_ind[col], paraphrases_ind['text'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can also check the proportion of content words that remain in the paraphrases. The results suggest that majority of the changes indeed happen to strategy markers (and the content words are mostly left untouched). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment A\n",
      "------------\n",
      "\tRETRIEVAL: 96.45% of content tokens are retained.\n",
      "\tGREEDY: 95.62% of content tokens are retained.\n",
      "\tILP: 95.29% of content tokens are retained.\n",
      "\n",
      "Experiment B\n",
      "------------\n",
      "\tRETRIEVAL: 95.72% of content tokens are retained.\n",
      "\tGREEDY: 94.80% of content tokens are retained.\n",
      "\tILP: 93.77% of content tokens are retained.\n"
     ]
    }
   ],
   "source": [
    "print(\"Experiment A\")\n",
    "print(\"------------\")\n",
    "\n",
    "for prefix in [\"retrieval\", 'greedy', 'ilp']: \n",
    "\n",
    "    ret_rate = []\n",
    "\n",
    "    for idx, row in paraphrases_ind.iterrows():\n",
    "\n",
    "        toks_paraphrase = set(row['{}_paraphrase'.format(prefix)].split())\n",
    "        toks = set(ind_corpus.get_utterance(idx).meta['content'].split())\n",
    "\n",
    "        if len(toks) == 0: continue\n",
    "\n",
    "        ret_rate.append(len(toks.intersection(toks_paraphrase))/len(toks))\n",
    "    \n",
    "    print(\"\\t{}: {:.2f}% of content tokens are retained.\".format(prefix.upper(), 100*np.mean(ret_rate)))\n",
    "    \n",
    "print()\n",
    "\n",
    "print(\"Experiment B\")\n",
    "print(\"------------\")\n",
    "\n",
    "for prefix in [\"retrieval\", 'greedy', 'ilp']: \n",
    "\n",
    "    ret_rate = []\n",
    "\n",
    "    for idx, row in paraphrases_mt.iterrows():\n",
    "\n",
    "        toks_paraphrase = set(row['{}_paraphrase'.format(prefix)].split())\n",
    "        toks = set(mt_corpus.get_utterance(idx).meta['content'].split())\n",
    "\n",
    "        if len(toks) == 0: continue\n",
    "\n",
    "        ret_rate.append(len(toks.intersection(toks_paraphrase))/len(toks))\n",
    "    \n",
    "    print(\"\\t{}: {:.2f}% of content tokens are retained.\".format(prefix.upper(), 100*np.mean(ret_rate)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our human-judged naturalness ratings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_annotations = pd.read_csv('output/naturalness_annotations.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average rating for RETRIEVAL outputs: 4.5\n",
      "Average rating for GREEDY outputs: 4.2\n",
      "Average rating for ILP outputs: 4.2\n"
     ]
    }
   ],
   "source": [
    "for mode in ['retrieval', 'greedy', 'ilp']:\n",
    "    print(\"Average rating for {} outputs: {:.1f}\".format(mode.upper(), np.mean(df_annotations[df_annotations['mode'] == mode]['rating'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "convokit-test",
   "language": "python",
   "name": "convokit-test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}