notebooks/3.RAG_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9bc6b8a5-c02f-40ba-b4ae-ec8c14683e51",
   "metadata": {},
   "source": [
    "# RAG Evaluation\n",
    "\n",
    "In this notebook, we evaluate different RAG (Retrieval-Augmented Generation) approaches to determine the most effective method for our task.\n",
    "\n",
    "## Evaluation Process\n",
    "\n",
    "To conduct a comprehensive evaluation, we will:\n",
    "\n",
    "1. Use questions from ground-truth dataset\n",
    "2. Implement multiple RAG approaches\n",
    "3. Generate responses using each approach\n",
    "4. Measure the performance of each method using different techniques \n",
    "5. Compare the results and select the best-performing approach\n",
    "\n",
    "## RAG Approaches to Evaluate\n",
    "\n",
    "We will assess the following RAG approaches:\n",
    "\n",
    "1. Prompt with 5 full articles in context or 10 chunks\n",
    "3. More sophisticated prompt or more simplistic\n",
    "\n",
    "## Evaluation Metrics\n",
    "\n",
    "To measure the effectiveness of each approach, we'll use the following metrics:\n",
    "- LLM-as-a-judge (percent of non-relevant)\n",
    "- ROUGE score (mean, median)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af4744b9-ff2d-40b0-acd7-e25e414a01e0",
   "metadata": {},
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7a9ccfa-2992-47b5-8986-6bc1918f5f85",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "import re\n",
    "\n",
    "from tqdm import tqdm\n",
    "from anthropic import Anthropic\n",
    "from elasticsearch import Elasticsearch\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from rouge import Rouge\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30579eb1-4199-4fc1-b425-77add7ca7a16",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/ground-truth.json', 'r') as f_in:\n",
    "    ground_truth = json.load(f_in)\n",
    "\n",
    "with open('../data/site_content.json', 'r') as f_in:\n",
    "    raw_doc = json.load(f_in)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cde98b19-e40c-4468-bf81-95fc5b38d88a",
   "metadata": {},
   "source": [
    "first we need to start Elasticsearch locally, if it's not started yet\n",
    "\n",
    "```bash\n",
    "docker run -it \\\n",
    "    --rm \\\n",
    "    --name elasticsearch \\\n",
    "    -p 9200:9200 \\\n",
    "    -p 9300:9300 \\\n",
    "    -e \"discovery.type=single-node\" \\\n",
    "    -e \"xpack.security.enabled=false\" \\\n",
    "    docker.elastic.co/elasticsearch/elasticsearch:8.4.3\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "885287be-c6e5-4c6f-90f6-529f22532513",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"all-MiniLM-L12-v2\"\n",
    "model = SentenceTransformer(model_name)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f8531ca-85e9-49f8-889c-4f73fc7e1d84",
   "metadata": {},
   "outputs": [],
   "source": [
    "es_client = Elasticsearch('http://localhost:9200') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "534128c2-9cbf-4077-9568-195ea752c71e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchtext'})"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"}\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name = \"esearchtext\"\n",
    "\n",
    "es_client.indices.delete(index=index_name, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4120cead-f400-4a83-bf04-456111a14ca6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 1030/1030 [00:13<00:00, 77.41it/s]\n"
     ]
    }
   ],
   "source": [
    "data = [{'url': k, 'header':v['header'], 'main_content':v['main_content']} for k,v in raw_doc.items()]\n",
    "for doc in tqdm(data):\n",
    "    es_client.index(index=index_name, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d5dfa41-8fd5-47ab-bd61-973c133fa4bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_boost(query):\n",
    "    search_query = {\n",
    "        \"size\": 5,\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"must\": {\n",
    "                    \"multi_match\": {\n",
    "                        \"query\": query,\n",
    "                        \"fields\": [\"url\", \"header\", \"main_content^3\"],\n",
    "                        \"type\": \"best_fields\"\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "\n",
    "    response = es_client.search(index=index_name, body=search_query)\n",
    "    \n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in response['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "    \n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ce0f9fd-2d8d-4f1b-9f14-d86044cc7b70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchvector_chunks_2'})"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"},\n",
    "            \"main_content_vector\": {\n",
    "                \"type\": \"dense_vector\",\n",
    "                \"dims\": model.get_sentence_embedding_dimension(),\n",
    "                \"index\": True,\n",
    "                \"similarity\": \"cosine\"\n",
    "            },\n",
    "            \n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name_vector = \"esearchvector_chunks_2\"\n",
    "\n",
    "es_client.indices.delete(index=index_name_vector, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name_vector, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9d52d4f-f72b-49a3-b894-2dc6e2300bf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def chunk_data(raw_doc, chunk_size=1000, overlap=100):\n",
    "    def chunk_content(content, chunk_size=1000, overlap=100):\n",
    "        chunks = []\n",
    "        start = 0\n",
    "        while start < len(content):\n",
    "            end = start + chunk_size\n",
    "            chunk = content[start:end]\n",
    "            chunks.append(chunk)\n",
    "            start = end - overlap\n",
    "        return chunks\n",
    "\n",
    "    chunked_data = []\n",
    "    for k, v in raw_doc.items():\n",
    "        content_chunks = chunk_content(v['main_content'], chunk_size, overlap)\n",
    "        for i, chunk in enumerate(content_chunks):\n",
    "            chunked_data.append({\n",
    "                'url': k,\n",
    "                'header': v['header'],\n",
    "                'main_content': chunk,\n",
    "                'chunk_index': i\n",
    "            })\n",
    "    \n",
    "    return chunked_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7f1eef4-bd58-4c2c-bb2c-bc11e555d7a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_chunk = chunk_data(raw_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c0056af-541e-43fb-817d-17694da2e540",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 2834/2834 [06:32<00:00,  7.22it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    doc['header_vector'] = model.encode(doc['header'])\n",
    "    doc['main_content_vector'] = model.encode(doc['main_content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6358432-1064-4284-bced-1a0e4675c7e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 2834/2834 [00:24<00:00, 116.01it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    es_client.index(index=index_name_vector, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8624ccfe-44cb-4dc6-aeee-609d1c893558",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_combined_10(query):\n",
    "    vector = model.encode(query)\n",
    "    search_query = {\n",
    "        \"_source\": [\"url\", \"header\", \"main_content\", \"header_vector\", \"main_content_vector\"],\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"should\": [\n",
    "                    {\n",
    "                        \"multi_match\": {\n",
    "                            \"query\": query,\n",
    "                            \"fields\": [\"header\", \"main_content\"],\n",
    "                            \"type\": \"best_fields\",\n",
    "                            \"tie_breaker\": 0.3\n",
    "                        }\n",
    "                    },\n",
    "                    {\n",
    "                        \"script_score\": {\n",
    "                            \"query\": {\"match_all\": {}},\n",
    "                            \"script\": {\n",
    "                                \"source\": \"cosineSimilarity(params.query_vector, 'main_content_vector') + 1.0\",\n",
    "                                \"params\": {\"query_vector\": vector}\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        },\n",
    "        \"size\": 10\n",
    "    }\n",
    "    \n",
    "    es_results = es_client.search(\n",
    "        index=index_name_vector,\n",
    "        body=search_query\n",
    "    )\n",
    "    \n",
    "    result_docs = [hit['_source'] for hit in es_results['hits']['hits']]\n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e149b27-0884-47d4-9d6c-84b98e539062",
   "metadata": {},
   "outputs": [],
   "source": [
    "clientA = Anthropic()\n",
    "def llm(prompt):\n",
    "    response = clientA.messages.create(\n",
    "        model=\"claude-3-5-sonnet-20240620\",\n",
    "        max_tokens = 500,\n",
    "        messages=[\n",
    "            {\n",
    "              \"role\": \"user\",\n",
    "              \"content\": [\n",
    "                {\n",
    "                  \"type\": \"text\",\n",
    "                  \"text\": prompt\n",
    "                }\n",
    "              ]\n",
    "            }\n",
    "      ]\n",
    "    )\n",
    "    return response.content[0].text\n",
    "\n",
    "def llm_haiku(prompt):\n",
    "    response = clientA.messages.create(\n",
    "        model=\"claude-3-haiku-20240307\",\n",
    "        max_tokens = 500,\n",
    "        messages=[\n",
    "            {\n",
    "              \"role\": \"user\",\n",
    "              \"content\": [\n",
    "                {\n",
    "                  \"type\": \"text\",\n",
    "                  \"text\": prompt\n",
    "                }\n",
    "              ]\n",
    "            }\n",
    "      ]\n",
    "    )\n",
    "    return response.content[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "250333dc-2439-4b13-8a5f-8c8bbef1d9a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_search_results(search_results: list[dict]) -> str:\n",
    "    formatted_results = \"\"\n",
    "    for result in search_results:\n",
    "        formatted_results += f\"- **{result['header']}**\\n  {result['main_content']}\\n  URL: {result['url']}\\n\\n\"\n",
    "    return formatted_results.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29c2d99e-d9bf-469d-ae2f-91fb3ee1714d",
   "metadata": {},
   "source": [
    "## Building prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8514389e-6d8e-409a-afb6-3ab332ac56f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_basic_prompt(query: str, search_results: list[dict]) -> str:\n",
    "    prompt = f\"Question: {query}\\n\\nContext:\\n{format_search_results(search_results)}\"\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fb071e3-d40c-443e-a1a7-35dd2ef5adaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_structured_prompt(query: str, search_results: list[dict]) -> str:\n",
    "    prompt = f\"\"\"\n",
    "Question: {query}\n",
    "\n",
    "Context:\n",
    "\n",
    "\n",
    "Instructions:\n",
    "1. Analyze the question and identify key points related to New Zealand visas.\n",
    "2. Review the provided context for relevant information.\n",
    "3. Formulate a clear, concise answer based on official information.\n",
    "4. If the question cannot be fully answered with the given context, state this clearly.\n",
    "5. Use markdown syntax for formatting (**bold** for emphasis, *italics* for titles).\n",
    "6. Include at least one relevant URL as a reference at the end of your answer.\n",
    "\n",
    "Please provide your answer below:\n",
    "\"\"\"\n",
    "    return prompt.strip()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5188c9ea-d9cc-42eb-abe0-9563df3d6ab3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_big_prompt(query, search_results):\n",
    "    prompt_template = f\"\"\"\n",
    "You are an AI assistant specializing in answering questions about New Zealand visas. Your knowledge comes from official New Zealand immigration information. You will be provided with context from relevant articles and a specific question to answer.\n",
    "\n",
    "First, review the following context:\n",
    "\n",
    "<context>\n",
    "{format_search_results(search_results)}\n",
    "</context>\n",
    "\n",
    "Process this context carefully. Each item in the context contains a URL, a header, and main content. Use this information to inform your answers, ensuring you provide accurate and up-to-date information about New Zealand visas.\n",
    "\n",
    "Now, answer the following question:\n",
    "\n",
    "<question>\n",
    "{query}\n",
    "</question>\n",
    "\n",
    "To answer the question:\n",
    "1. Analyze the question and identify the key points related to New Zealand visas.\n",
    "2. Search through the provided context for relevant information.\n",
    "3. Formulate a clear, concise answer based on the official information.\n",
    "4. If the question cannot be fully answered with the given context, state this clearly and provide the most relevant information available.\n",
    "\n",
    "Write your answer using short markdown syntax, as it will be displayed in a Telegram chat. Use **bold** for emphasis and *italics* for titles or important terms.\n",
    "\n",
    "Always include at least one relevant URL from the context as a reference. Format the URL reference at the end of your answer like this:\n",
    "[Source](URL)\n",
    "\n",
    "If multiple sources are used, include them as separate reference links.\n",
    "\n",
    "Provide your answer within <answer> tags.\n",
    "\"\"\".strip()\n",
    "    return prompt_template\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa5ba2c3-87bc-4845-8457-887460d8496f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_basic_answer(llm_response: str) -> str:\n",
    "    return llm_response.strip()\n",
    "\n",
    "def extract_structured_answer(llm_response: str) -> str:\n",
    "    answer_start = llm_response.find(\"Please provide your answer below:\")\n",
    "    if answer_start != -1:\n",
    "        return llm_response[answer_start + len(\"Please provide your answer below:\"):].strip()\n",
    "    else:\n",
    "        return llm_response.strip()\n",
    "\n",
    "def extract_big_answer(llm_response: str) -> str:\n",
    "    match = re.search(r'<answer>(.*?)</answer>', llm_response, re.DOTALL)\n",
    "    if match:\n",
    "        return match.group(1).strip()\n",
    "    else:\n",
    "        return llm_response.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a734186e-e815-4652-879c-b730bc5606b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def rag(query, version):\n",
    "    if version == \"text+simple\":\n",
    "        search = elastic_search_boost\n",
    "        build_prompt = build_basic_prompt\n",
    "        extractor = extract_basic_answer\n",
    "    elif version == \"text+structured\":\n",
    "        search = elastic_search_boost\n",
    "        build_prompt = build_structured_prompt\n",
    "        extractor = extract_structured_answer\n",
    "    elif version == \"text+big\":\n",
    "        search = elastic_search_boost\n",
    "        build_prompt = build_big_prompt\n",
    "        extractor = extract_big_answer\n",
    "    elif version == \"vector+simple\":\n",
    "        search = elastic_search_combined_10\n",
    "        build_prompt = build_basic_prompt\n",
    "        extractor = extract_basic_answer\n",
    "    elif version == \"vector+structured\":\n",
    "        search = elastic_search_combined_10\n",
    "        build_prompt = build_structured_prompt\n",
    "        extractor = extract_structured_answer\n",
    "    elif version == \"vector+big\":\n",
    "        search = elastic_search_combined_10\n",
    "        build_prompt = build_big_prompt\n",
    "        extractor = extract_big_answer\n",
    "    \n",
    "    search_results = search(query)\n",
    "    prompt = build_prompt(query, search_results)\n",
    "    answer = llm(prompt)\n",
    "    return extractor(answer)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "221ae018-ef9a-47fe-86a8-8ce876f8d721",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Based on the information provided, since you are from Russia, you would need to apply for a visa before traveling to New Zealand. Russia is not on the list of visa waiver countries, so Russian citizens cannot use an NZeTA for entry and must obtain an appropriate visa in advance. You would need to explore the visa options and apply for the relevant visa type depending on your purpose of travel (e.g. visitor visa, work visa, student visa etc). An NZeTA or visa-free entry is not available for Russian passport holders.'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rag(\"I am from russia\", \"text+simple\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eded296-d1f6-4c75-ac7b-8ec152328543",
   "metadata": {},
   "source": [
    "## Evaluation relevance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "579e7762-7888-4d72-8742-9f47f0975401",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_relevance(question: str, answer_llm: str) -> dict:\n",
    "    prompt2_template = \"\"\"\n",
    "You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system.\n",
    "Your task is to analyze the relevance of the generated answer to the given question.\n",
    "Based on the relevance of the generated answer, you will classify it\n",
    "as \"NON_RELEVANT\", \"PARTLY_RELEVANT\", or \"RELEVANT\".\n",
    "\n",
    "Here is the data for evaluation:\n",
    "\n",
    "Question: {question}\n",
    "Generated Answer: {answer_llm}\n",
    "\n",
    "Please analyze the content and context of the generated answer in relation to the question\n",
    "and provide your evaluation in parsable JSON without using code blocks:\n",
    "\n",
    "{{\n",
    "  \"Relevance\": \"NON_RELEVANT\" | \"PARTLY_RELEVANT\" | \"RELEVANT\",\n",
    "  \"Explanation\": \"[Provide a brief explanation for your evaluation]\"\n",
    "}}\n",
    "\"\"\".strip()\n",
    "\n",
    "    # Format the prompt with the question and answer\n",
    "    evaluation_prompt = prompt2_template.format(question=question, answer_llm=answer_llm)\n",
    "\n",
    "    # Get the evaluation from the LLM\n",
    "    evaluation_response = llm_haiku(evaluation_prompt)\n",
    "\n",
    "    # Parse the JSON response\n",
    "    try:\n",
    "        evaluation_result = json.loads(evaluation_response)\n",
    "    except json.JSONDecodeError:\n",
    "        # If JSON parsing fails, return an error result\n",
    "        return {\n",
    "            \"Relevance\": \"ERROR\",\n",
    "            \"Explanation\": \"Failed to parse LLM response as JSON\"\n",
    "        }\n",
    "\n",
    "    # Validate the structure of the parsed result\n",
    "    if \"Relevance\" not in evaluation_result or \"Explanation\" not in evaluation_result:\n",
    "        return {\n",
    "            \"Relevance\": \"ERROR\",\n",
    "            \"Explanation\": \"LLM response does not contain expected fields\"\n",
    "        }\n",
    "\n",
    "    # Validate the Relevance value\n",
    "    if evaluation_result[\"Relevance\"] not in [\"NON_RELEVANT\", \"PARTLY_RELEVANT\", \"RELEVANT\"]:\n",
    "        evaluation_result[\"Relevance\"] = \"ERROR\"\n",
    "        evaluation_result[\"Explanation\"] += \" (Invalid Relevance value)\"\n",
    "\n",
    "    rouge = Rouge()\n",
    "    scores = rouge.get_scores(answer_llm, question)\n",
    "    # Add ROUGE scores to the evaluation result as plain text\n",
    "    evaluation_result[\"ROUGE_1\"] = scores[0]['rouge-1']['f']\n",
    "    evaluation_result[\"ROUGE_2\"] = scores[0]['rouge-2']['f']\n",
    "    evaluation_result[\"ROUGE_3\"] = scores[0]['rouge-l']['f']\n",
    "    \n",
    "    \n",
    "    \n",
    "    return evaluation_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85fa80c2-4f5e-4f8f-8a8c-bbf143203b23",
   "metadata": {},
   "outputs": [],
   "source": [
    "version = \"text+simple\"\n",
    "relevance_list = []\n",
    "for i in range(5,8):\n",
    "    question = ground_truth[i][\"question\"]\n",
    "    answer = rag(question, version)\n",
    "    score = evaluate_relevance(question, answer)\n",
    "    score[\"question\"] = question\n",
    "    score[\"answer\"] = answer\n",
    "    relevance_list.append(score)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b098d17-e99c-43e8-916d-d3437759db4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_3</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer does not directly suggest...</td>\n",
       "      <td>0.105263</td>\n",
       "      <td>0.011976</td>\n",
       "      <td>0.105263</td>\n",
       "      <td>What does the content suggest users to do first?</td>\n",
       "      <td>Based on the content provided, there is no cle...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.238095</td>\n",
       "      <td>Where can users navigate back to if needed?</td>\n",
       "      <td>Based on the context provided, users can navig...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer thoroughly addresses the ...</td>\n",
       "      <td>0.170732</td>\n",
       "      <td>0.104167</td>\n",
       "      <td>0.146341</td>\n",
       "      <td>How many ways of interacting with the page are...</td>\n",
       "      <td>Based on the context provided, there are 5 way...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Relevance                                        Explanation  \\\n",
       "0  PARTLY_RELEVANT  The generated answer does not directly suggest...   \n",
       "1         RELEVANT  The generated answer directly addresses the qu...   \n",
       "2         RELEVANT  The generated answer thoroughly addresses the ...   \n",
       "\n",
       "    ROUGE_1   ROUGE_2   ROUGE_3  \\\n",
       "0  0.105263  0.011976  0.105263   \n",
       "1  0.285714  0.083333  0.238095   \n",
       "2  0.170732  0.104167  0.146341   \n",
       "\n",
       "                                            question  \\\n",
       "0   What does the content suggest users to do first?   \n",
       "1        Where can users navigate back to if needed?   \n",
       "2  How many ways of interacting with the page are...   \n",
       "\n",
       "                                              answer  \n",
       "0  Based on the content provided, there is no cle...  \n",
       "1  Based on the context provided, users can navig...  \n",
       "2  Based on the context provided, there are 5 way...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relevance_df = pd.DataFrame(relevance_list)\n",
    "relevance_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47457864-8d26-4237-8598-6e027510f930",
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_stats(relevance_df):\n",
    "    print(\"RELEVANCE STATS\")\n",
    "    print(relevance_df.Relevance.value_counts())\n",
    "    print()\n",
    "    print(\"ROUGE\")\n",
    "    print(relevance_df[[\"ROUGE_1\", \"ROUGE_2\", \"ROUGE_3\"]].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5824237-95d6-4eab-8d00-668b83e8e2c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           2\n",
      "PARTLY_RELEVANT    1\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.187236\n",
      "ROUGE_2    0.066492\n",
      "ROUGE_3    0.163233\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(relevance_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "901aae0a-7768-4851-be32-b7a44e10bedf",
   "metadata": {},
   "source": [
    "## Evaluating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "320ac630-5e22-4148-9e5c-8e2bc715fc19",
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(123)\n",
    "test_subset = random.sample(ground_truth, 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "508c77cf-a0ec-47da-8df0-287c3b068860",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_rag(examples:list, rag_verstion:str) -> pd.DataFrame:\n",
    "    relevance_list = []\n",
    "    for row in tqdm(examples):\n",
    "        question = row[\"question\"]\n",
    "        answer = rag(question, rag_verstion)\n",
    "        score = evaluate_relevance(question, answer)\n",
    "        score[\"question\"] = question\n",
    "        score[\"answer\"] = answer\n",
    "        relevance_list.append(score)\n",
    "    return pd.DataFrame(relevance_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "874696ac-92ff-43ab-b9fd-35c400113105",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores1 = evaluate_rag(test_subset, \"text+simple\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a5ed1cf-cfef-45d1-8285-4c17aeab0b68",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           46\n",
      "PARTLY_RELEVANT     3\n",
      "NON_RELEVANT        1\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.185495\n",
      "ROUGE_2    0.106079\n",
      "ROUGE_3    0.172999\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8a08756-3cd5-46ca-b7f7-18f78e72b798",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = llm_haiku # to reduce cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73323b11-2bba-4997-8122-68b10dc676e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 50/50 [03:18<00:00,  3.98s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_3</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.161491</td>\n",
       "      <td>0.096491</td>\n",
       "      <td>0.136646</td>\n",
       "      <td>What types of evidence can be provided to prov...</td>\n",
       "      <td>To prove ownership of funds and assets for inv...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.143885</td>\n",
       "      <td>0.082474</td>\n",
       "      <td>0.143885</td>\n",
       "      <td>What are some examples of actions that would b...</td>\n",
       "      <td>Based on the context provided, there are a few...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.116883</td>\n",
       "      <td>0.043860</td>\n",
       "      <td>0.103896</td>\n",
       "      <td>What are the employer's obligations concerning...</td>\n",
       "      <td>Based on the question and the provided context...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.092166</td>\n",
       "      <td>0.185714</td>\n",
       "      <td>What should a visa holder do if their employer...</td>\n",
       "      <td>Here is the answer to your question:\\n\\nIf a v...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer correctly states that the...</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.086957</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>What organization is mentioned in the text?</td>\n",
       "      <td>The text does not mention any specific organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a relevant and t...</td>\n",
       "      <td>0.178571</td>\n",
       "      <td>0.075472</td>\n",
       "      <td>0.178571</td>\n",
       "      <td>What types of information can unlicensed indiv...</td>\n",
       "      <td>Based on the information provided, this questi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.119760</td>\n",
       "      <td>0.070485</td>\n",
       "      <td>0.095808</td>\n",
       "      <td>What types of evidence should be included when...</td>\n",
       "      <td>When applying to extend a student stay in New ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides relevant informa...</td>\n",
       "      <td>0.256881</td>\n",
       "      <td>0.152778</td>\n",
       "      <td>0.256881</td>\n",
       "      <td>Where can employers and employees find informa...</td>\n",
       "      <td>Based on the question, the key points related ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides general informat...</td>\n",
       "      <td>0.252252</td>\n",
       "      <td>0.180556</td>\n",
       "      <td>0.252252</td>\n",
       "      <td>Is there a way to return to the homepage from ...</td>\n",
       "      <td>Based on the context provided, there does not ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.093960</td>\n",
       "      <td>0.019139</td>\n",
       "      <td>0.093960</td>\n",
       "      <td>What is an eVisa and how does it differ from t...</td>\n",
       "      <td>An **eVisa** is an electronic visa that allows...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly provides the ope...</td>\n",
       "      <td>0.280702</td>\n",
       "      <td>0.187500</td>\n",
       "      <td>0.280702</td>\n",
       "      <td>What are the operating hours for Immigration N...</td>\n",
       "      <td>According to the official Immigration New Zeal...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.126126</td>\n",
       "      <td>0.068027</td>\n",
       "      <td>0.108108</td>\n",
       "      <td>Who can sponsor Sponsorship Visa applications ...</td>\n",
       "      <td>According to the information provided, the key...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant ex...</td>\n",
       "      <td>0.190476</td>\n",
       "      <td>0.122137</td>\n",
       "      <td>0.171429</td>\n",
       "      <td>What are some examples of organizations that c...</td>\n",
       "      <td>Based on the question, some examples of organi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that the ori...</td>\n",
       "      <td>0.194175</td>\n",
       "      <td>0.059259</td>\n",
       "      <td>0.174757</td>\n",
       "      <td>How many refugees is New Zealand aiming to res...</td>\n",
       "      <td>According to the information provided, the que...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>0.062222</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>Where can applicants find information about fe...</td>\n",
       "      <td>Based on the question and the context provided...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.152542</td>\n",
       "      <td>0.036585</td>\n",
       "      <td>0.135593</td>\n",
       "      <td>What should pregnant applicants for temporary ...</td>\n",
       "      <td>Based on the question, here is the response:\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.143885</td>\n",
       "      <td>0.067039</td>\n",
       "      <td>0.129496</td>\n",
       "      <td>What must an applicant demonstrate when applyi...</td>\n",
       "      <td>To apply for a second or subsequent permanent ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant co...</td>\n",
       "      <td>0.159091</td>\n",
       "      <td>0.107143</td>\n",
       "      <td>0.159091</td>\n",
       "      <td>What changes are being made to the RSE scheme ...</td>\n",
       "      <td>Based on the provided context, the question ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some general inf...</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.099291</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>How long is the resettlement program at the Mā...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer is partly relevant as it ...</td>\n",
       "      <td>0.145455</td>\n",
       "      <td>0.054054</td>\n",
       "      <td>0.127273</td>\n",
       "      <td>When did applications for the 2021 Resident Vi...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not contain any rele...</td>\n",
       "      <td>0.117647</td>\n",
       "      <td>0.044944</td>\n",
       "      <td>0.117647</td>\n",
       "      <td>Where is the Kyiv VAC located?</td>\n",
       "      <td>The provided context does not contain any info...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.095890</td>\n",
       "      <td>0.051020</td>\n",
       "      <td>0.095890</td>\n",
       "      <td>How can you protect yourself from immigration ...</td>\n",
       "      <td>To protect yourself from immigration and visa ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not provide any info...</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.118644</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>What change was made to temporary work visas i...</td>\n",
       "      <td>Based on the given question and context, I can...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some general inf...</td>\n",
       "      <td>0.146789</td>\n",
       "      <td>0.066667</td>\n",
       "      <td>0.128440</td>\n",
       "      <td>Where should visitors look to find various opt...</td>\n",
       "      <td>Based on the question, it does not appear to b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant co...</td>\n",
       "      <td>0.207407</td>\n",
       "      <td>0.127660</td>\n",
       "      <td>0.207407</td>\n",
       "      <td>What is the minimum amount required to cover m...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer clearly indicates that th...</td>\n",
       "      <td>0.183908</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.160920</td>\n",
       "      <td>As of June 2024, how many 2021 Resident Visa a...</td>\n",
       "      <td>Unfortunately, the given context does not prov...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides relevant informa...</td>\n",
       "      <td>0.124138</td>\n",
       "      <td>0.062827</td>\n",
       "      <td>0.110345</td>\n",
       "      <td>How can cruise ship passengers with an NZeTA e...</td>\n",
       "      <td>To answer your question, here are the key poin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.282609</td>\n",
       "      <td>0.193548</td>\n",
       "      <td>0.282609</td>\n",
       "      <td>Is there a way to go back to the Immigration N...</td>\n",
       "      <td>Based on the information provided, here is the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.130081</td>\n",
       "      <td>0.055866</td>\n",
       "      <td>0.130081</td>\n",
       "      <td>In what circumstances might certified English ...</td>\n",
       "      <td>In the context of New Zealand visa application...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer accurately identifies tha...</td>\n",
       "      <td>0.139535</td>\n",
       "      <td>0.055046</td>\n",
       "      <td>0.139535</td>\n",
       "      <td>What type of organization does this content li...</td>\n",
       "      <td>Based on the context provided, this content li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>0.065728</td>\n",
       "      <td>0.120000</td>\n",
       "      <td>What should you do if you want to change emplo...</td>\n",
       "      <td>To change employment while on a 2021 Interim V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.140625</td>\n",
       "      <td>0.059524</td>\n",
       "      <td>0.140625</td>\n",
       "      <td>What should you do if there is glare or reflec...</td>\n",
       "      <td>The question is not related to New Zealand vis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.121622</td>\n",
       "      <td>0.064516</td>\n",
       "      <td>0.108108</td>\n",
       "      <td>What types of evidence can demonstrate that a ...</td>\n",
       "      <td>To demonstrate that a couple is living togethe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that it does...</td>\n",
       "      <td>0.235294</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.235294</td>\n",
       "      <td>Is the Bank of China loyalty programme partner...</td>\n",
       "      <td>Based on the question and the context provided...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.189474</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.189474</td>\n",
       "      <td>Is there a website available for visa applicat...</td>\n",
       "      <td>Based on the question, it appears you are look...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some general tip...</td>\n",
       "      <td>0.100629</td>\n",
       "      <td>0.043668</td>\n",
       "      <td>0.100629</td>\n",
       "      <td>How can you prevent glare on glasses in a pass...</td>\n",
       "      <td>The question is about preventing glare on glas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.068376</td>\n",
       "      <td>0.011561</td>\n",
       "      <td>0.068376</td>\n",
       "      <td>Where can immigration instructions be found?</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that the giv...</td>\n",
       "      <td>0.317460</td>\n",
       "      <td>0.184211</td>\n",
       "      <td>0.253968</td>\n",
       "      <td>Is there a specific homepage for Immigration N...</td>\n",
       "      <td>Based on the information provided in the given...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not provide any info...</td>\n",
       "      <td>0.101010</td>\n",
       "      <td>0.045113</td>\n",
       "      <td>0.101010</td>\n",
       "      <td>Is the VAC in Kyiv currently open for visa app...</td>\n",
       "      <td>Based on the information provided, this questi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that the que...</td>\n",
       "      <td>0.192982</td>\n",
       "      <td>0.081871</td>\n",
       "      <td>0.175439</td>\n",
       "      <td>What legal compliance is necessary for employe...</td>\n",
       "      <td>The question is asking about the legal complia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that the key...</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>0.201550</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>What is the minimum hourly rate required durin...</td>\n",
       "      <td>Based on the information provided in the quest...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.165138</td>\n",
       "      <td>0.070922</td>\n",
       "      <td>0.146789</td>\n",
       "      <td>Who is allowed to provide immigration advice i...</td>\n",
       "      <td>According to the Immigration Advisers Authorit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.148760</td>\n",
       "      <td>0.060606</td>\n",
       "      <td>0.148760</td>\n",
       "      <td>What are the three skills-based pathways for o...</td>\n",
       "      <td>The three main skills-based pathways for obtai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.109375</td>\n",
       "      <td>0.076923</td>\n",
       "      <td>0.109375</td>\n",
       "      <td>How do employers initiate the recruitment proc...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.122449</td>\n",
       "      <td>0.048077</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>How can employers ensure they are complying wi...</td>\n",
       "      <td>To ensure compliance with employment laws for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.114754</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>0.114754</td>\n",
       "      <td>What information do you need to provide for a ...</td>\n",
       "      <td>To provide a third-party with the necessary in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively covers th...</td>\n",
       "      <td>0.114286</td>\n",
       "      <td>0.029268</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>What steps must an employer take before hiring...</td>\n",
       "      <td>To hire a migrant worker in New Zealand, an em...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.115942</td>\n",
       "      <td>0.060302</td>\n",
       "      <td>0.115942</td>\n",
       "      <td>How can you transfer a valid visa to a new pas...</td>\n",
       "      <td>To transfer a valid visa to a new passport, yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.120690</td>\n",
       "      <td>0.072727</td>\n",
       "      <td>0.120690</td>\n",
       "      <td>What are some situations where fee refunds may...</td>\n",
       "      <td>Based on the context provided, here are some s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not contain any info...</td>\n",
       "      <td>0.273973</td>\n",
       "      <td>0.195122</td>\n",
       "      <td>0.273973</td>\n",
       "      <td>What is the annual quota for Refugee Family Su...</td>\n",
       "      <td>**Key Points:**\\n- The question is specificall...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Relevance                                        Explanation  \\\n",
       "0          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "1          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "2   PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "3          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "4      NON_RELEVANT  The generated answer correctly states that the...   \n",
       "5          RELEVANT  The generated answer provides a relevant and t...   \n",
       "6          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "7          RELEVANT  The generated answer provides relevant informa...   \n",
       "8   PARTLY_RELEVANT  The generated answer provides general informat...   \n",
       "9          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "10         RELEVANT  The generated answer directly provides the ope...   \n",
       "11         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "12  PARTLY_RELEVANT  The generated answer provides some relevant ex...   \n",
       "13  PARTLY_RELEVANT  The generated answer acknowledges that the ori...   \n",
       "14         RELEVANT  The generated answer provides comprehensive an...   \n",
       "15         RELEVANT  The generated answer directly addresses the qu...   \n",
       "16  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "17  PARTLY_RELEVANT  The generated answer provides some relevant co...   \n",
       "18  PARTLY_RELEVANT  The generated answer provides some general inf...   \n",
       "19  PARTLY_RELEVANT  The generated answer is partly relevant as it ...   \n",
       "20     NON_RELEVANT  The generated answer does not contain any rele...   \n",
       "21         RELEVANT  The generated answer provides comprehensive an...   \n",
       "22     NON_RELEVANT  The generated answer does not provide any info...   \n",
       "23  PARTLY_RELEVANT  The generated answer provides some general inf...   \n",
       "24  PARTLY_RELEVANT  The generated answer provides some relevant co...   \n",
       "25     NON_RELEVANT  The generated answer clearly indicates that th...   \n",
       "26  PARTLY_RELEVANT  The generated answer provides relevant informa...   \n",
       "27         RELEVANT  The generated answer directly addresses the qu...   \n",
       "28         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "29         RELEVANT  The generated answer accurately identifies tha...   \n",
       "30         RELEVANT  The generated answer provides detailed and rel...   \n",
       "31         RELEVANT  The generated answer provides detailed and rel...   \n",
       "32         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "33  PARTLY_RELEVANT  The generated answer acknowledges that it does...   \n",
       "34  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "35  PARTLY_RELEVANT  The generated answer provides some general tip...   \n",
       "36         RELEVANT  The generated answer provides comprehensive an...   \n",
       "37  PARTLY_RELEVANT  The generated answer acknowledges that the giv...   \n",
       "38     NON_RELEVANT  The generated answer does not provide any info...   \n",
       "39  PARTLY_RELEVANT  The generated answer acknowledges that the que...   \n",
       "40  PARTLY_RELEVANT  The generated answer acknowledges that the key...   \n",
       "41         RELEVANT  The generated answer directly addresses the qu...   \n",
       "42         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "43         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "44         RELEVANT  The generated answer directly addresses the qu...   \n",
       "45         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "46         RELEVANT  The generated answer comprehensively covers th...   \n",
       "47  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "48         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "49     NON_RELEVANT  The generated answer does not contain any info...   \n",
       "\n",
       "     ROUGE_1   ROUGE_2   ROUGE_3  \\\n",
       "0   0.161491  0.096491  0.136646   \n",
       "1   0.143885  0.082474  0.143885   \n",
       "2   0.116883  0.043860  0.103896   \n",
       "3   0.200000  0.092166  0.185714   \n",
       "4   0.142857  0.086957  0.142857   \n",
       "5   0.178571  0.075472  0.178571   \n",
       "6   0.119760  0.070485  0.095808   \n",
       "7   0.256881  0.152778  0.256881   \n",
       "8   0.252252  0.180556  0.252252   \n",
       "9   0.093960  0.019139  0.093960   \n",
       "10  0.280702  0.187500  0.280702   \n",
       "11  0.126126  0.068027  0.108108   \n",
       "12  0.190476  0.122137  0.171429   \n",
       "13  0.194175  0.059259  0.174757   \n",
       "14  0.133333  0.062222  0.133333   \n",
       "15  0.152542  0.036585  0.135593   \n",
       "16  0.143885  0.067039  0.129496   \n",
       "17  0.159091  0.107143  0.159091   \n",
       "18  0.160000  0.099291  0.140000   \n",
       "19  0.145455  0.054054  0.127273   \n",
       "20  0.117647  0.044944  0.117647   \n",
       "21  0.095890  0.051020  0.095890   \n",
       "22  0.181818  0.118644  0.181818   \n",
       "23  0.146789  0.066667  0.128440   \n",
       "24  0.207407  0.127660  0.207407   \n",
       "25  0.183908  0.083333  0.160920   \n",
       "26  0.124138  0.062827  0.110345   \n",
       "27  0.282609  0.193548  0.282609   \n",
       "28  0.130081  0.055866  0.130081   \n",
       "29  0.139535  0.055046  0.139535   \n",
       "30  0.133333  0.065728  0.120000   \n",
       "31  0.140625  0.059524  0.140625   \n",
       "32  0.121622  0.064516  0.108108   \n",
       "33  0.235294  0.142857  0.235294   \n",
       "34  0.189474  0.125000  0.189474   \n",
       "35  0.100629  0.043668  0.100629   \n",
       "36  0.068376  0.011561  0.068376   \n",
       "37  0.317460  0.184211  0.253968   \n",
       "38  0.101010  0.045113  0.101010   \n",
       "39  0.192982  0.081871  0.175439   \n",
       "40  0.315789  0.201550  0.315789   \n",
       "41  0.165138  0.070922  0.146789   \n",
       "42  0.148760  0.060606  0.148760   \n",
       "43  0.109375  0.076923  0.109375   \n",
       "44  0.122449  0.048077  0.095238   \n",
       "45  0.114754  0.023256  0.114754   \n",
       "46  0.114286  0.029268  0.100000   \n",
       "47  0.115942  0.060302  0.115942   \n",
       "48  0.120690  0.072727  0.120690   \n",
       "49  0.273973  0.195122  0.273973   \n",
       "\n",
       "                                             question  \\\n",
       "0   What types of evidence can be provided to prov...   \n",
       "1   What are some examples of actions that would b...   \n",
       "2   What are the employer's obligations concerning...   \n",
       "3   What should a visa holder do if their employer...   \n",
       "4         What organization is mentioned in the text?   \n",
       "5   What types of information can unlicensed indiv...   \n",
       "6   What types of evidence should be included when...   \n",
       "7   Where can employers and employees find informa...   \n",
       "8   Is there a way to return to the homepage from ...   \n",
       "9   What is an eVisa and how does it differ from t...   \n",
       "10  What are the operating hours for Immigration N...   \n",
       "11  Who can sponsor Sponsorship Visa applications ...   \n",
       "12  What are some examples of organizations that c...   \n",
       "13  How many refugees is New Zealand aiming to res...   \n",
       "14  Where can applicants find information about fe...   \n",
       "15  What should pregnant applicants for temporary ...   \n",
       "16  What must an applicant demonstrate when applyi...   \n",
       "17  What changes are being made to the RSE scheme ...   \n",
       "18  How long is the resettlement program at the Mā...   \n",
       "19  When did applications for the 2021 Resident Vi...   \n",
       "20                     Where is the Kyiv VAC located?   \n",
       "21  How can you protect yourself from immigration ...   \n",
       "22  What change was made to temporary work visas i...   \n",
       "23  Where should visitors look to find various opt...   \n",
       "24  What is the minimum amount required to cover m...   \n",
       "25  As of June 2024, how many 2021 Resident Visa a...   \n",
       "26  How can cruise ship passengers with an NZeTA e...   \n",
       "27  Is there a way to go back to the Immigration N...   \n",
       "28  In what circumstances might certified English ...   \n",
       "29  What type of organization does this content li...   \n",
       "30  What should you do if you want to change emplo...   \n",
       "31  What should you do if there is glare or reflec...   \n",
       "32  What types of evidence can demonstrate that a ...   \n",
       "33  Is the Bank of China loyalty programme partner...   \n",
       "34  Is there a website available for visa applicat...   \n",
       "35  How can you prevent glare on glasses in a pass...   \n",
       "36       Where can immigration instructions be found?   \n",
       "37  Is there a specific homepage for Immigration N...   \n",
       "38  Is the VAC in Kyiv currently open for visa app...   \n",
       "39  What legal compliance is necessary for employe...   \n",
       "40  What is the minimum hourly rate required durin...   \n",
       "41  Who is allowed to provide immigration advice i...   \n",
       "42  What are the three skills-based pathways for o...   \n",
       "43  How do employers initiate the recruitment proc...   \n",
       "44  How can employers ensure they are complying wi...   \n",
       "45  What information do you need to provide for a ...   \n",
       "46  What steps must an employer take before hiring...   \n",
       "47  How can you transfer a valid visa to a new pas...   \n",
       "48  What are some situations where fee refunds may...   \n",
       "49  What is the annual quota for Refugee Family Su...   \n",
       "\n",
       "                                               answer  \n",
       "0   To prove ownership of funds and assets for inv...  \n",
       "1   Based on the context provided, there are a few...  \n",
       "2   Based on the question and the provided context...  \n",
       "3   Here is the answer to your question:\\n\\nIf a v...  \n",
       "4   The text does not mention any specific organiz...  \n",
       "5   Based on the information provided, this questi...  \n",
       "6   When applying to extend a student stay in New ...  \n",
       "7   Based on the question, the key points related ...  \n",
       "8   Based on the context provided, there does not ...  \n",
       "9   An **eVisa** is an electronic visa that allows...  \n",
       "10  According to the official Immigration New Zeal...  \n",
       "11  According to the information provided, the key...  \n",
       "12  Based on the question, some examples of organi...  \n",
       "13  According to the information provided, the que...  \n",
       "14  Based on the question and the context provided...  \n",
       "15  Based on the question, here is the response:\\n...  \n",
       "16  To apply for a second or subsequent permanent ...  \n",
       "17  Based on the provided context, the question ca...  \n",
       "18  Based on the information provided, the questio...  \n",
       "19  Based on the information provided, the questio...  \n",
       "20  The provided context does not contain any info...  \n",
       "21  To protect yourself from immigration and visa ...  \n",
       "22  Based on the given question and context, I can...  \n",
       "23  Based on the question, it does not appear to b...  \n",
       "24  Based on the information provided, the questio...  \n",
       "25  Unfortunately, the given context does not prov...  \n",
       "26  To answer your question, here are the key poin...  \n",
       "27  Based on the information provided, here is the...  \n",
       "28  In the context of New Zealand visa application...  \n",
       "29  Based on the context provided, this content li...  \n",
       "30  To change employment while on a 2021 Interim V...  \n",
       "31  The question is not related to New Zealand vis...  \n",
       "32  To demonstrate that a couple is living togethe...  \n",
       "33  Based on the question and the context provided...  \n",
       "34  Based on the question, it appears you are look...  \n",
       "35  The question is about preventing glare on glas...  \n",
       "36  Based on the information provided, the questio...  \n",
       "37  Based on the information provided in the given...  \n",
       "38  Based on the information provided, this questi...  \n",
       "39  The question is asking about the legal complia...  \n",
       "40  Based on the information provided in the quest...  \n",
       "41  According to the Immigration Advisers Authorit...  \n",
       "42  The three main skills-based pathways for obtai...  \n",
       "43  Based on the information provided, the questio...  \n",
       "44  To ensure compliance with employment laws for ...  \n",
       "45  To provide a third-party with the necessary in...  \n",
       "46  To hire a migrant worker in New Zealand, an em...  \n",
       "47  To transfer a valid visa to a new passport, yo...  \n",
       "48  Based on the context provided, here are some s...  \n",
       "49  **Key Points:**\\n- The question is specificall...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores2 = evaluate_rag(test_subset, \"text+structured\")\n",
    "scores2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "023a3fad-1312-4b7d-ba19-5ea7a826e3e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 50/50 [04:42<00:00,  5.66s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_3</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.168224</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.130841</td>\n",
       "      <td>What types of evidence can be provided to prov...</td>\n",
       "      <td>According to the information provided, when ap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer clearly and directly addr...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.169811</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>What are some examples of actions that would b...</td>\n",
       "      <td>According to the official information, some ex...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.208333</td>\n",
       "      <td>0.091603</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>What are the employer's obligations concerning...</td>\n",
       "      <td>According to the information provided, when an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.241758</td>\n",
       "      <td>0.082645</td>\n",
       "      <td>0.197802</td>\n",
       "      <td>What should a visa holder do if their employer...</td>\n",
       "      <td>According to the information provided, if a vi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer clearly and comprehensive...</td>\n",
       "      <td>0.074766</td>\n",
       "      <td>0.027397</td>\n",
       "      <td>0.074766</td>\n",
       "      <td>What organization is mentioned in the text?</td>\n",
       "      <td>The main organizations mentioned in the provid...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides relevant informa...</td>\n",
       "      <td>0.123711</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.082474</td>\n",
       "      <td>What types of information can unlicensed indiv...</td>\n",
       "      <td>According to the information provided, **unlic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.141593</td>\n",
       "      <td>0.052288</td>\n",
       "      <td>0.106195</td>\n",
       "      <td>What types of evidence should be included when...</td>\n",
       "      <td>To extend a student stay in New Zealand, you s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive in...</td>\n",
       "      <td>0.282609</td>\n",
       "      <td>0.178862</td>\n",
       "      <td>0.260870</td>\n",
       "      <td>Where can employers and employees find informa...</td>\n",
       "      <td>According to the provided context, employers a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.509091</td>\n",
       "      <td>0.444444</td>\n",
       "      <td>0.509091</td>\n",
       "      <td>Is there a way to return to the homepage from ...</td>\n",
       "      <td>Yes, there is a way to return to the homepage ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.145455</td>\n",
       "      <td>0.040541</td>\n",
       "      <td>0.145455</td>\n",
       "      <td>What is an eVisa and how does it differ from t...</td>\n",
       "      <td>An **eVisa** is a visa that is issued and reco...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the operating ho...</td>\n",
       "      <td>0.097561</td>\n",
       "      <td>0.040404</td>\n",
       "      <td>0.097561</td>\n",
       "      <td>What are the operating hours for Immigration N...</td>\n",
       "      <td>According to the context, Immigration New Zeal...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.108696</td>\n",
       "      <td>0.054054</td>\n",
       "      <td>0.108696</td>\n",
       "      <td>Who can sponsor Sponsorship Visa applications ...</td>\n",
       "      <td>According to the New Zealand immigration infor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides specific example...</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.266667</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>What are some examples of organizations that c...</td>\n",
       "      <td>According to the information provided, some ex...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.352941</td>\n",
       "      <td>0.137931</td>\n",
       "      <td>0.313725</td>\n",
       "      <td>How many refugees is New Zealand aiming to res...</td>\n",
       "      <td>According to the information provided, **New Z...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.298507</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>0.268657</td>\n",
       "      <td>Where can applicants find information about fe...</td>\n",
       "      <td>According to the information provided, applica...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.108696</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>What should pregnant applicants for temporary ...</td>\n",
       "      <td>According to the provided context, pregnant ap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the re...</td>\n",
       "      <td>0.277778</td>\n",
       "      <td>0.116279</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>What must an applicant demonstrate when applyi...</td>\n",
       "      <td>According to the New Zealand immigration infor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed informa...</td>\n",
       "      <td>0.129870</td>\n",
       "      <td>0.073394</td>\n",
       "      <td>0.129870</td>\n",
       "      <td>What changes are being made to the RSE scheme ...</td>\n",
       "      <td>According to the information provided, the fol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.271186</td>\n",
       "      <td>0.212121</td>\n",
       "      <td>0.237288</td>\n",
       "      <td>How long is the resettlement program at the Mā...</td>\n",
       "      <td>According to the information provided, the res...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.235294</td>\n",
       "      <td>0.078431</td>\n",
       "      <td>0.156863</td>\n",
       "      <td>When did applications for the 2021 Resident Vi...</td>\n",
       "      <td>According to the information provided:\\n\\n- Th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the exact locati...</td>\n",
       "      <td>0.121212</td>\n",
       "      <td>0.054054</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>Where is the Kyiv VAC located?</td>\n",
       "      <td>According to the context, the **Kyiv VAC (Visa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.104575</td>\n",
       "      <td>0.046512</td>\n",
       "      <td>0.091503</td>\n",
       "      <td>How can you protect yourself from immigration ...</td>\n",
       "      <td>Here are some key ways you can protect yoursel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed informa...</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.046512</td>\n",
       "      <td>What change was made to temporary work visas i...</td>\n",
       "      <td>In 2022, the Accredited Employer Work Visa (AE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.313725</td>\n",
       "      <td>0.135593</td>\n",
       "      <td>0.274510</td>\n",
       "      <td>Where should visitors look to find various opt...</td>\n",
       "      <td>According to the context, visitors to New Zeal...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.373333</td>\n",
       "      <td>0.272727</td>\n",
       "      <td>0.346667</td>\n",
       "      <td>What is the minimum amount required to cover m...</td>\n",
       "      <td>According to the official New Zealand immigrat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.339623</td>\n",
       "      <td>0.172414</td>\n",
       "      <td>0.339623</td>\n",
       "      <td>As of June 2024, how many 2021 Resident Visa a...</td>\n",
       "      <td>According to the information provided, as of *...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.172043</td>\n",
       "      <td>0.102564</td>\n",
       "      <td>0.150538</td>\n",
       "      <td>How can cruise ship passengers with an NZeTA e...</td>\n",
       "      <td>According to the information provided, **cruis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.325000</td>\n",
       "      <td>0.217822</td>\n",
       "      <td>0.325000</td>\n",
       "      <td>Is there a way to go back to the Immigration N...</td>\n",
       "      <td>Yes, according to the context provided, there ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a detailed and c...</td>\n",
       "      <td>0.138614</td>\n",
       "      <td>0.070922</td>\n",
       "      <td>0.138614</td>\n",
       "      <td>In what circumstances might certified English ...</td>\n",
       "      <td>According to the official New Zealand immigrat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer is highly relevant to the...</td>\n",
       "      <td>0.096386</td>\n",
       "      <td>0.037383</td>\n",
       "      <td>0.096386</td>\n",
       "      <td>What type of organization does this content li...</td>\n",
       "      <td>Based on the context provided, this content li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed, releva...</td>\n",
       "      <td>0.235294</td>\n",
       "      <td>0.097087</td>\n",
       "      <td>0.188235</td>\n",
       "      <td>What should you do if you want to change emplo...</td>\n",
       "      <td>If you hold a **2021 Interim Visa** and want t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides two specific rec...</td>\n",
       "      <td>0.392157</td>\n",
       "      <td>0.254545</td>\n",
       "      <td>0.352941</td>\n",
       "      <td>What should you do if there is glare or reflec...</td>\n",
       "      <td>If there is glare or reflection on your glasse...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a detailed and c...</td>\n",
       "      <td>0.238095</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.238095</td>\n",
       "      <td>What types of evidence can demonstrate that a ...</td>\n",
       "      <td>According to the New Zealand immigration guide...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer is relevant to the questi...</td>\n",
       "      <td>0.208955</td>\n",
       "      <td>0.148148</td>\n",
       "      <td>0.208955</td>\n",
       "      <td>Is the Bank of China loyalty programme partner...</td>\n",
       "      <td>No, the Bank of China loyalty programme partne...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.179487</td>\n",
       "      <td>0.101010</td>\n",
       "      <td>0.179487</td>\n",
       "      <td>Is there a website available for visa applicat...</td>\n",
       "      <td>Yes, there is a website available for visa app...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.153846</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>How can you prevent glare on glasses in a pass...</td>\n",
       "      <td>To prevent glare on glasses in a passport phot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.093023</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.093023</td>\n",
       "      <td>Where can immigration instructions be found?</td>\n",
       "      <td>According to the context, **immigration instru...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.392857</td>\n",
       "      <td>0.258065</td>\n",
       "      <td>0.392857</td>\n",
       "      <td>Is there a specific homepage for Immigration N...</td>\n",
       "      <td>Yes, there is a specific homepage for Immigrat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.185185</td>\n",
       "      <td>0.093750</td>\n",
       "      <td>0.185185</td>\n",
       "      <td>Is the VAC in Kyiv currently open for visa app...</td>\n",
       "      <td>No, the VAC (Visa Application Centre) in Kyiv,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively covers th...</td>\n",
       "      <td>0.141732</td>\n",
       "      <td>0.056497</td>\n",
       "      <td>0.141732</td>\n",
       "      <td>What legal compliance is necessary for employe...</td>\n",
       "      <td>According to the provided context, employers m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.483871</td>\n",
       "      <td>0.347826</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>What is the minimum hourly rate required durin...</td>\n",
       "      <td>The minimum hourly rate required during work e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>Who is allowed to provide immigration advice i...</td>\n",
       "      <td>According to the information provided, the fol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a thorough and a...</td>\n",
       "      <td>0.140625</td>\n",
       "      <td>0.054645</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>What are the three skills-based pathways for o...</td>\n",
       "      <td>According to the information provided, there a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.175000</td>\n",
       "      <td>0.100840</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>How do employers initiate the recruitment proc...</td>\n",
       "      <td>To initiate the recruitment process in the RSE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>0.064516</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>How can employers ensure they are complying wi...</td>\n",
       "      <td>To ensure compliance with employment laws for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the specific inf...</td>\n",
       "      <td>0.246575</td>\n",
       "      <td>0.089888</td>\n",
       "      <td>0.164384</td>\n",
       "      <td>What information do you need to provide for a ...</td>\n",
       "      <td>According to the information provided, to have...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively covers th...</td>\n",
       "      <td>0.113475</td>\n",
       "      <td>0.030303</td>\n",
       "      <td>0.099291</td>\n",
       "      <td>What steps must an employer take before hiring...</td>\n",
       "      <td>Before hiring a migrant worker, an employer in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>0.086957</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>How can you transfer a valid visa to a new pas...</td>\n",
       "      <td>To transfer a valid visa to a new passport, yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.148936</td>\n",
       "      <td>0.049180</td>\n",
       "      <td>0.106383</td>\n",
       "      <td>What are some situations where fee refunds may...</td>\n",
       "      <td>According to the official New Zealand immigrat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>0.177215</td>\n",
       "      <td>0.257143</td>\n",
       "      <td>What is the annual quota for Refugee Family Su...</td>\n",
       "      <td>According to the information provided, the ann...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Relevance                                        Explanation   ROUGE_1  \\\n",
       "0   RELEVANT  The generated answer provides a comprehensive ...  0.168224   \n",
       "1   RELEVANT  The generated answer clearly and directly addr...  0.250000   \n",
       "2   RELEVANT  The generated answer directly addresses the qu...  0.208333   \n",
       "3   RELEVANT  The generated answer directly addresses the qu...  0.241758   \n",
       "4   RELEVANT  The generated answer clearly and comprehensive...  0.074766   \n",
       "5   RELEVANT  The generated answer provides relevant informa...  0.123711   \n",
       "6   RELEVANT  The generated answer provides a comprehensive ...  0.141593   \n",
       "7   RELEVANT  The generated answer provides comprehensive in...  0.282609   \n",
       "8   RELEVANT  The generated answer directly addresses the qu...  0.509091   \n",
       "9   RELEVANT  The generated answer provides a comprehensive ...  0.145455   \n",
       "10  RELEVANT  The generated answer provides the operating ho...  0.097561   \n",
       "11  RELEVANT  The generated answer provides a comprehensive ...  0.108696   \n",
       "12  RELEVANT  The generated answer provides specific example...  0.333333   \n",
       "13  RELEVANT  The generated answer directly addresses the qu...  0.352941   \n",
       "14  RELEVANT  The generated answer provides detailed and rel...  0.298507   \n",
       "15  RELEVANT  The generated answer directly addresses the qu...  0.250000   \n",
       "16  RELEVANT  The generated answer directly addresses the re...  0.277778   \n",
       "17  RELEVANT  The generated answer provides detailed informa...  0.129870   \n",
       "18  RELEVANT  The generated answer directly addresses the qu...  0.271186   \n",
       "19  RELEVANT  The generated answer directly addresses the qu...  0.235294   \n",
       "20  RELEVANT  The generated answer provides the exact locati...  0.121212   \n",
       "21  RELEVANT  The generated answer directly addresses the qu...  0.104575   \n",
       "22  RELEVANT  The generated answer provides detailed informa...  0.069767   \n",
       "23  RELEVANT  The generated answer directly addresses the qu...  0.313725   \n",
       "24  RELEVANT  The generated answer directly addresses the qu...  0.373333   \n",
       "25  RELEVANT  The generated answer directly addresses the qu...  0.339623   \n",
       "26  RELEVANT  The generated answer directly addresses the qu...  0.172043   \n",
       "27  RELEVANT  The generated answer directly addresses the qu...  0.325000   \n",
       "28  RELEVANT  The generated answer provides a detailed and c...  0.138614   \n",
       "29  RELEVANT  The generated answer is highly relevant to the...  0.096386   \n",
       "30  RELEVANT  The generated answer provides detailed, releva...  0.235294   \n",
       "31  RELEVANT  The generated answer provides two specific rec...  0.392157   \n",
       "32  RELEVANT  The generated answer provides a detailed and c...  0.238095   \n",
       "33  RELEVANT  The generated answer is relevant to the questi...  0.208955   \n",
       "34  RELEVANT  The generated answer provides comprehensive an...  0.179487   \n",
       "35  RELEVANT  The generated answer directly addresses the qu...  0.250000   \n",
       "36  RELEVANT  The generated answer provides detailed and rel...  0.093023   \n",
       "37  RELEVANT  The generated answer directly addresses the qu...  0.392857   \n",
       "38  RELEVANT  The generated answer directly addresses the qu...  0.185185   \n",
       "39  RELEVANT  The generated answer comprehensively covers th...  0.141732   \n",
       "40  RELEVANT  The generated answer directly addresses the qu...  0.483871   \n",
       "41  RELEVANT  The generated answer directly addresses the qu...  0.145833   \n",
       "42  RELEVANT  The generated answer provides a thorough and a...  0.140625   \n",
       "43  RELEVANT  The generated answer provides a comprehensive ...  0.175000   \n",
       "44  RELEVANT  The generated answer directly addresses the qu...  0.133333   \n",
       "45  RELEVANT  The generated answer provides the specific inf...  0.246575   \n",
       "46  RELEVANT  The generated answer comprehensively covers th...  0.113475   \n",
       "47  RELEVANT  The generated answer provides detailed and rel...  0.145833   \n",
       "48  RELEVANT  The generated answer provides detailed and rel...  0.148936   \n",
       "49  RELEVANT  The generated answer directly addresses the qu...  0.285714   \n",
       "\n",
       "     ROUGE_2   ROUGE_3                                           question  \\\n",
       "0   0.000000  0.130841  What types of evidence can be provided to prov...   \n",
       "1   0.169811  0.250000  What are some examples of actions that would b...   \n",
       "2   0.091603  0.166667  What are the employer's obligations concerning...   \n",
       "3   0.082645  0.197802  What should a visa holder do if their employer...   \n",
       "4   0.027397  0.074766        What organization is mentioned in the text?   \n",
       "5   0.000000  0.082474  What types of information can unlicensed indiv...   \n",
       "6   0.052288  0.106195  What types of evidence should be included when...   \n",
       "7   0.178862  0.260870  Where can employers and employees find informa...   \n",
       "8   0.444444  0.509091  Is there a way to return to the homepage from ...   \n",
       "9   0.040541  0.145455  What is an eVisa and how does it differ from t...   \n",
       "10  0.040404  0.097561  What are the operating hours for Immigration N...   \n",
       "11  0.054054  0.108696  Who can sponsor Sponsorship Visa applications ...   \n",
       "12  0.266667  0.333333  What are some examples of organizations that c...   \n",
       "13  0.137931  0.313725  How many refugees is New Zealand aiming to res...   \n",
       "14  0.069767  0.268657  Where can applicants find information about fe...   \n",
       "15  0.108696  0.250000  What should pregnant applicants for temporary ...   \n",
       "16  0.116279  0.222222  What must an applicant demonstrate when applyi...   \n",
       "17  0.073394  0.129870  What changes are being made to the RSE scheme ...   \n",
       "18  0.212121  0.237288  How long is the resettlement program at the Mā...   \n",
       "19  0.078431  0.156863  When did applications for the 2021 Resident Vi...   \n",
       "20  0.054054  0.090909                     Where is the Kyiv VAC located?   \n",
       "21  0.046512  0.091503  How can you protect yourself from immigration ...   \n",
       "22  0.000000  0.046512  What change was made to temporary work visas i...   \n",
       "23  0.135593  0.274510  Where should visitors look to find various opt...   \n",
       "24  0.272727  0.346667  What is the minimum amount required to cover m...   \n",
       "25  0.172414  0.339623  As of June 2024, how many 2021 Resident Visa a...   \n",
       "26  0.102564  0.150538  How can cruise ship passengers with an NZeTA e...   \n",
       "27  0.217822  0.325000  Is there a way to go back to the Immigration N...   \n",
       "28  0.070922  0.138614  In what circumstances might certified English ...   \n",
       "29  0.037383  0.096386  What type of organization does this content li...   \n",
       "30  0.097087  0.188235  What should you do if you want to change emplo...   \n",
       "31  0.254545  0.352941  What should you do if there is glare or reflec...   \n",
       "32  0.181818  0.238095  What types of evidence can demonstrate that a ...   \n",
       "33  0.148148  0.208955  Is the Bank of China loyalty programme partner...   \n",
       "34  0.101010  0.179487  Is there a website available for visa applicat...   \n",
       "35  0.153846  0.250000  How can you prevent glare on glasses in a pass...   \n",
       "36  0.000000  0.093023       Where can immigration instructions be found?   \n",
       "37  0.258065  0.392857  Is there a specific homepage for Immigration N...   \n",
       "38  0.093750  0.185185  Is the VAC in Kyiv currently open for visa app...   \n",
       "39  0.056497  0.141732  What legal compliance is necessary for employe...   \n",
       "40  0.347826  0.451613  What is the minimum hourly rate required durin...   \n",
       "41  0.100000  0.145833  Who is allowed to provide immigration advice i...   \n",
       "42  0.054645  0.125000  What are the three skills-based pathways for o...   \n",
       "43  0.100840  0.150000  How do employers initiate the recruitment proc...   \n",
       "44  0.064516  0.133333  How can employers ensure they are complying wi...   \n",
       "45  0.089888  0.164384  What information do you need to provide for a ...   \n",
       "46  0.030303  0.099291  What steps must an employer take before hiring...   \n",
       "47  0.086957  0.145833  How can you transfer a valid visa to a new pas...   \n",
       "48  0.049180  0.106383  What are some situations where fee refunds may...   \n",
       "49  0.177215  0.257143  What is the annual quota for Refugee Family Su...   \n",
       "\n",
       "                                               answer  \n",
       "0   According to the information provided, when ap...  \n",
       "1   According to the official information, some ex...  \n",
       "2   According to the information provided, when an...  \n",
       "3   According to the information provided, if a vi...  \n",
       "4   The main organizations mentioned in the provid...  \n",
       "5   According to the information provided, **unlic...  \n",
       "6   To extend a student stay in New Zealand, you s...  \n",
       "7   According to the provided context, employers a...  \n",
       "8   Yes, there is a way to return to the homepage ...  \n",
       "9   An **eVisa** is a visa that is issued and reco...  \n",
       "10  According to the context, Immigration New Zeal...  \n",
       "11  According to the New Zealand immigration infor...  \n",
       "12  According to the information provided, some ex...  \n",
       "13  According to the information provided, **New Z...  \n",
       "14  According to the information provided, applica...  \n",
       "15  According to the provided context, pregnant ap...  \n",
       "16  According to the New Zealand immigration infor...  \n",
       "17  According to the information provided, the fol...  \n",
       "18  According to the information provided, the res...  \n",
       "19  According to the information provided:\\n\\n- Th...  \n",
       "20  According to the context, the **Kyiv VAC (Visa...  \n",
       "21  Here are some key ways you can protect yoursel...  \n",
       "22  In 2022, the Accredited Employer Work Visa (AE...  \n",
       "23  According to the context, visitors to New Zeal...  \n",
       "24  According to the official New Zealand immigrat...  \n",
       "25  According to the information provided, as of *...  \n",
       "26  According to the information provided, **cruis...  \n",
       "27  Yes, according to the context provided, there ...  \n",
       "28  According to the official New Zealand immigrat...  \n",
       "29  Based on the context provided, this content li...  \n",
       "30  If you hold a **2021 Interim Visa** and want t...  \n",
       "31  If there is glare or reflection on your glasse...  \n",
       "32  According to the New Zealand immigration guide...  \n",
       "33  No, the Bank of China loyalty programme partne...  \n",
       "34  Yes, there is a website available for visa app...  \n",
       "35  To prevent glare on glasses in a passport phot...  \n",
       "36  According to the context, **immigration instru...  \n",
       "37  Yes, there is a specific homepage for Immigrat...  \n",
       "38  No, the VAC (Visa Application Centre) in Kyiv,...  \n",
       "39  According to the provided context, employers m...  \n",
       "40  The minimum hourly rate required during work e...  \n",
       "41  According to the information provided, the fol...  \n",
       "42  According to the information provided, there a...  \n",
       "43  To initiate the recruitment process in the RSE...  \n",
       "44  To ensure compliance with employment laws for ...  \n",
       "45  According to the information provided, to have...  \n",
       "46  Before hiring a migrant worker, an employer in...  \n",
       "47  To transfer a valid visa to a new passport, yo...  \n",
       "48  According to the official New Zealand immigrat...  \n",
       "49  According to the information provided, the ann...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores3 = evaluate_rag(test_subset, \"text+big\")\n",
    "scores3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce206419-e18f-4183-9888-fc55ff22b576",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 50/50 [04:05<00:00,  4.91s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_3</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively covers th...</td>\n",
       "      <td>0.183099</td>\n",
       "      <td>0.112676</td>\n",
       "      <td>0.183099</td>\n",
       "      <td>What types of evidence can be provided to prov...</td>\n",
       "      <td>Based on the information provided, here are so...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides several specific...</td>\n",
       "      <td>0.192982</td>\n",
       "      <td>0.113924</td>\n",
       "      <td>0.192982</td>\n",
       "      <td>What are some examples of actions that would b...</td>\n",
       "      <td>Based on the information provided, some exampl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively covers th...</td>\n",
       "      <td>0.165289</td>\n",
       "      <td>0.079096</td>\n",
       "      <td>0.148760</td>\n",
       "      <td>What are the employer's obligations concerning...</td>\n",
       "      <td>Based on the information provided, the key emp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.197183</td>\n",
       "      <td>0.093023</td>\n",
       "      <td>0.183099</td>\n",
       "      <td>What should a visa holder do if their employer...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively lists the...</td>\n",
       "      <td>0.080645</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.064516</td>\n",
       "      <td>What organization is mentioned in the text?</td>\n",
       "      <td>Based on the context provided, the main organi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.158730</td>\n",
       "      <td>0.045714</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>What types of information can unlicensed indiv...</td>\n",
       "      <td>Based on the information provided, unlicensed ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.176471</td>\n",
       "      <td>0.096154</td>\n",
       "      <td>0.176471</td>\n",
       "      <td>What types of evidence should be included when...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and com...</td>\n",
       "      <td>0.247619</td>\n",
       "      <td>0.136646</td>\n",
       "      <td>0.247619</td>\n",
       "      <td>Where can employers and employees find informa...</td>\n",
       "      <td>Based on the information provided, employers a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a detailed and c...</td>\n",
       "      <td>0.169935</td>\n",
       "      <td>0.103004</td>\n",
       "      <td>0.143791</td>\n",
       "      <td>Is there a way to return to the homepage from ...</td>\n",
       "      <td>Based on the information provided, it does not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively addresses...</td>\n",
       "      <td>0.126984</td>\n",
       "      <td>0.041885</td>\n",
       "      <td>0.126984</td>\n",
       "      <td>What is an eVisa and how does it differ from t...</td>\n",
       "      <td>Based on the information provided, an eVisa di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.206897</td>\n",
       "      <td>0.066667</td>\n",
       "      <td>0.183908</td>\n",
       "      <td>What are the operating hours for Immigration N...</td>\n",
       "      <td>Based on the information provided, Immigration...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.127273</td>\n",
       "      <td>0.075000</td>\n",
       "      <td>0.127273</td>\n",
       "      <td>Who can sponsor Sponsorship Visa applications ...</td>\n",
       "      <td>Based on the information provided, the followi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides several relevant...</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.094118</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>What are some examples of organizations that c...</td>\n",
       "      <td>Based on the information provided, some exampl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer accurately states that Ne...</td>\n",
       "      <td>0.318841</td>\n",
       "      <td>0.134831</td>\n",
       "      <td>0.260870</td>\n",
       "      <td>How many refugees is New Zealand aiming to res...</td>\n",
       "      <td>Based on the information provided, New Zealand...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive in...</td>\n",
       "      <td>0.196078</td>\n",
       "      <td>0.066225</td>\n",
       "      <td>0.196078</td>\n",
       "      <td>Where can applicants find information about fe...</td>\n",
       "      <td>Based on the information provided, applicants ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.193548</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.177419</td>\n",
       "      <td>What should pregnant applicants for temporary ...</td>\n",
       "      <td>Based on the information provided, pregnant ap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.192593</td>\n",
       "      <td>0.089552</td>\n",
       "      <td>0.177778</td>\n",
       "      <td>What must an applicant demonstrate when applyi...</td>\n",
       "      <td>Based on the information provided, an applican...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.116959</td>\n",
       "      <td>0.066946</td>\n",
       "      <td>0.105263</td>\n",
       "      <td>What changes are being made to the RSE scheme ...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly answers the ques...</td>\n",
       "      <td>0.268657</td>\n",
       "      <td>0.175000</td>\n",
       "      <td>0.208955</td>\n",
       "      <td>How long is the resettlement program at the Mā...</td>\n",
       "      <td>According to the information provided, the res...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.172840</td>\n",
       "      <td>0.090090</td>\n",
       "      <td>0.172840</td>\n",
       "      <td>When did applications for the 2021 Resident Vi...</td>\n",
       "      <td>Based on the information provided:\\n\\n- Applic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the specific loc...</td>\n",
       "      <td>0.126984</td>\n",
       "      <td>0.056338</td>\n",
       "      <td>0.126984</td>\n",
       "      <td>Where is the Kyiv VAC located?</td>\n",
       "      <td>Based on the information provided, the Kyiv VA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>0.041667</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>How can you protect yourself from immigration ...</td>\n",
       "      <td>Here are some tips to protect yourself from im...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer accurately and comprehens...</td>\n",
       "      <td>0.164948</td>\n",
       "      <td>0.090909</td>\n",
       "      <td>0.164948</td>\n",
       "      <td>What change was made to temporary work visas i...</td>\n",
       "      <td>Based on the information provided, the key cha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.101449</td>\n",
       "      <td>0.020305</td>\n",
       "      <td>0.086957</td>\n",
       "      <td>Where should visitors look to find various opt...</td>\n",
       "      <td>Based on the information provided in the conte...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.277228</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.277228</td>\n",
       "      <td>What is the minimum amount required to cover m...</td>\n",
       "      <td>Based on the information provided, the minimum...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the specific inf...</td>\n",
       "      <td>0.253521</td>\n",
       "      <td>0.113636</td>\n",
       "      <td>0.225352</td>\n",
       "      <td>As of June 2024, how many 2021 Resident Visa a...</td>\n",
       "      <td>According to the information provided, as of a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.127660</td>\n",
       "      <td>0.057416</td>\n",
       "      <td>0.127660</td>\n",
       "      <td>How can cruise ship passengers with an NZeTA e...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.216667</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>Is there a way to go back to the Immigration N...</td>\n",
       "      <td>Based on the information provided, it seems th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a thorough and r...</td>\n",
       "      <td>0.138614</td>\n",
       "      <td>0.062893</td>\n",
       "      <td>0.138614</td>\n",
       "      <td>In what circumstances might certified English ...</td>\n",
       "      <td>Based on the information provided, certified E...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer is highly relevant to the...</td>\n",
       "      <td>0.075472</td>\n",
       "      <td>0.025478</td>\n",
       "      <td>0.075472</td>\n",
       "      <td>What type of organization does this content li...</td>\n",
       "      <td>Based on the context provided, this content li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a detailed and c...</td>\n",
       "      <td>0.144578</td>\n",
       "      <td>0.076336</td>\n",
       "      <td>0.132530</td>\n",
       "      <td>What should you do if you want to change emplo...</td>\n",
       "      <td>Here are the key steps if you want to change e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.101911</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>What should you do if there is glare or reflec...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.147059</td>\n",
       "      <td>0.064865</td>\n",
       "      <td>0.147059</td>\n",
       "      <td>What types of evidence can demonstrate that a ...</td>\n",
       "      <td>Based on the information provided, the key typ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer is relevant to the questi...</td>\n",
       "      <td>0.197183</td>\n",
       "      <td>0.131868</td>\n",
       "      <td>0.197183</td>\n",
       "      <td>Is the Bank of China loyalty programme partner...</td>\n",
       "      <td>Based on the information provided in the conte...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively addresses...</td>\n",
       "      <td>0.104348</td>\n",
       "      <td>0.023529</td>\n",
       "      <td>0.104348</td>\n",
       "      <td>Is there a website available for visa applicat...</td>\n",
       "      <td>Based on the information provided, there does ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.131387</td>\n",
       "      <td>0.061856</td>\n",
       "      <td>0.131387</td>\n",
       "      <td>How can you prevent glare on glasses in a pass...</td>\n",
       "      <td>Here are some tips to prevent glare on glasses...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.074074</td>\n",
       "      <td>0.012121</td>\n",
       "      <td>0.074074</td>\n",
       "      <td>Where can immigration instructions be found?</td>\n",
       "      <td>Based on the information provided, immigration...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.409091</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.409091</td>\n",
       "      <td>Is there a specific homepage for Immigration N...</td>\n",
       "      <td>The text does not mention a specific homepage ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.231884</td>\n",
       "      <td>0.129032</td>\n",
       "      <td>0.231884</td>\n",
       "      <td>Is the VAC in Kyiv currently open for visa app...</td>\n",
       "      <td>Based on the information provided, it seems th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.132450</td>\n",
       "      <td>0.061947</td>\n",
       "      <td>0.119205</td>\n",
       "      <td>What legal compliance is necessary for employe...</td>\n",
       "      <td>Based on the information provided, the key leg...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the specific min...</td>\n",
       "      <td>0.326087</td>\n",
       "      <td>0.229508</td>\n",
       "      <td>0.304348</td>\n",
       "      <td>What is the minimum hourly rate required durin...</td>\n",
       "      <td>Based on the information provided, the minimum...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively addresses...</td>\n",
       "      <td>0.114286</td>\n",
       "      <td>0.067308</td>\n",
       "      <td>0.114286</td>\n",
       "      <td>Who is allowed to provide immigration advice i...</td>\n",
       "      <td>Based on the information provided, the key poi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively covers th...</td>\n",
       "      <td>0.150943</td>\n",
       "      <td>0.082353</td>\n",
       "      <td>0.150943</td>\n",
       "      <td>What are the three skills-based pathways for o...</td>\n",
       "      <td>Based on the information provided, the three s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.052632</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>How do employers initiate the recruitment proc...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer comprehensively addresses...</td>\n",
       "      <td>0.140845</td>\n",
       "      <td>0.066038</td>\n",
       "      <td>0.140845</td>\n",
       "      <td>How can employers ensure they are complying wi...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.162162</td>\n",
       "      <td>0.057143</td>\n",
       "      <td>0.144144</td>\n",
       "      <td>What information do you need to provide for a ...</td>\n",
       "      <td>Based on the information provided, the key det...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.120805</td>\n",
       "      <td>0.043103</td>\n",
       "      <td>0.107383</td>\n",
       "      <td>What steps must an employer take before hiring...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a detailed and c...</td>\n",
       "      <td>0.129032</td>\n",
       "      <td>0.059701</td>\n",
       "      <td>0.129032</td>\n",
       "      <td>How can you transfer a valid visa to a new pas...</td>\n",
       "      <td>Based on the information provided, here are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.128440</td>\n",
       "      <td>0.062893</td>\n",
       "      <td>0.128440</td>\n",
       "      <td>What are some situations where fee refunds may...</td>\n",
       "      <td>Based on the information provided, here are so...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides the specific ann...</td>\n",
       "      <td>0.256410</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.230769</td>\n",
       "      <td>What is the annual quota for Refugee Family Su...</td>\n",
       "      <td>According to the information provided, the ann...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Relevance                                        Explanation  \\\n",
       "0          RELEVANT  The generated answer comprehensively covers th...   \n",
       "1          RELEVANT  The generated answer provides several specific...   \n",
       "2          RELEVANT  The generated answer comprehensively covers th...   \n",
       "3          RELEVANT  The generated answer provides comprehensive an...   \n",
       "4          RELEVANT  The generated answer comprehensively lists the...   \n",
       "5          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "6          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "7          RELEVANT  The generated answer provides detailed and com...   \n",
       "8          RELEVANT  The generated answer provides a detailed and c...   \n",
       "9          RELEVANT  The generated answer comprehensively addresses...   \n",
       "10  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "11         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "12         RELEVANT  The generated answer provides several relevant...   \n",
       "13         RELEVANT  The generated answer accurately states that Ne...   \n",
       "14         RELEVANT  The generated answer provides comprehensive in...   \n",
       "15         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "16         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "17         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "18         RELEVANT  The generated answer directly answers the ques...   \n",
       "19         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "20         RELEVANT  The generated answer provides the specific loc...   \n",
       "21         RELEVANT  The generated answer provides comprehensive an...   \n",
       "22         RELEVANT  The generated answer accurately and comprehens...   \n",
       "23  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "24         RELEVANT  The generated answer directly addresses the qu...   \n",
       "25         RELEVANT  The generated answer provides the specific inf...   \n",
       "26         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "27         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "28         RELEVANT  The generated answer provides a thorough and r...   \n",
       "29         RELEVANT  The generated answer is highly relevant to the...   \n",
       "30         RELEVANT  The generated answer provides a detailed and c...   \n",
       "31         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "32         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "33         RELEVANT  The generated answer is relevant to the questi...   \n",
       "34         RELEVANT  The generated answer comprehensively addresses...   \n",
       "35         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "36         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "37         RELEVANT  The generated answer directly addresses the qu...   \n",
       "38         RELEVANT  The generated answer directly addresses the qu...   \n",
       "39         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "40         RELEVANT  The generated answer provides the specific min...   \n",
       "41         RELEVANT  The generated answer comprehensively addresses...   \n",
       "42         RELEVANT  The generated answer comprehensively covers th...   \n",
       "43         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "44         RELEVANT  The generated answer comprehensively addresses...   \n",
       "45         RELEVANT  The generated answer provides comprehensive an...   \n",
       "46         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "47         RELEVANT  The generated answer provides a detailed and c...   \n",
       "48         RELEVANT  The generated answer directly addresses the qu...   \n",
       "49         RELEVANT  The generated answer provides the specific ann...   \n",
       "\n",
       "     ROUGE_1   ROUGE_2   ROUGE_3  \\\n",
       "0   0.183099  0.112676  0.183099   \n",
       "1   0.192982  0.113924  0.192982   \n",
       "2   0.165289  0.079096  0.148760   \n",
       "3   0.197183  0.093023  0.183099   \n",
       "4   0.080645  0.000000  0.064516   \n",
       "5   0.158730  0.045714  0.142857   \n",
       "6   0.176471  0.096154  0.176471   \n",
       "7   0.247619  0.136646  0.247619   \n",
       "8   0.169935  0.103004  0.143791   \n",
       "9   0.126984  0.041885  0.126984   \n",
       "10  0.206897  0.066667  0.183908   \n",
       "11  0.127273  0.075000  0.127273   \n",
       "12  0.142857  0.094118  0.142857   \n",
       "13  0.318841  0.134831  0.260870   \n",
       "14  0.196078  0.066225  0.196078   \n",
       "15  0.193548  0.083333  0.177419   \n",
       "16  0.192593  0.089552  0.177778   \n",
       "17  0.116959  0.066946  0.105263   \n",
       "18  0.268657  0.175000  0.208955   \n",
       "19  0.172840  0.090090  0.172840   \n",
       "20  0.126984  0.056338  0.126984   \n",
       "21  0.095238  0.041667  0.095238   \n",
       "22  0.164948  0.090909  0.164948   \n",
       "23  0.101449  0.020305  0.086957   \n",
       "24  0.277228  0.181818  0.277228   \n",
       "25  0.253521  0.113636  0.225352   \n",
       "26  0.127660  0.057416  0.127660   \n",
       "27  0.216667  0.111111  0.200000   \n",
       "28  0.138614  0.062893  0.138614   \n",
       "29  0.075472  0.025478  0.075472   \n",
       "30  0.144578  0.076336  0.132530   \n",
       "31  0.222222  0.101911  0.222222   \n",
       "32  0.147059  0.064865  0.147059   \n",
       "33  0.197183  0.131868  0.197183   \n",
       "34  0.104348  0.023529  0.104348   \n",
       "35  0.131387  0.061856  0.131387   \n",
       "36  0.074074  0.012121  0.074074   \n",
       "37  0.409091  0.250000  0.409091   \n",
       "38  0.231884  0.129032  0.231884   \n",
       "39  0.132450  0.061947  0.119205   \n",
       "40  0.326087  0.229508  0.304348   \n",
       "41  0.114286  0.067308  0.114286   \n",
       "42  0.150943  0.082353  0.150943   \n",
       "43  0.100000  0.052632  0.100000   \n",
       "44  0.140845  0.066038  0.140845   \n",
       "45  0.162162  0.057143  0.144144   \n",
       "46  0.120805  0.043103  0.107383   \n",
       "47  0.129032  0.059701  0.129032   \n",
       "48  0.128440  0.062893  0.128440   \n",
       "49  0.256410  0.142857  0.230769   \n",
       "\n",
       "                                             question  \\\n",
       "0   What types of evidence can be provided to prov...   \n",
       "1   What are some examples of actions that would b...   \n",
       "2   What are the employer's obligations concerning...   \n",
       "3   What should a visa holder do if their employer...   \n",
       "4         What organization is mentioned in the text?   \n",
       "5   What types of information can unlicensed indiv...   \n",
       "6   What types of evidence should be included when...   \n",
       "7   Where can employers and employees find informa...   \n",
       "8   Is there a way to return to the homepage from ...   \n",
       "9   What is an eVisa and how does it differ from t...   \n",
       "10  What are the operating hours for Immigration N...   \n",
       "11  Who can sponsor Sponsorship Visa applications ...   \n",
       "12  What are some examples of organizations that c...   \n",
       "13  How many refugees is New Zealand aiming to res...   \n",
       "14  Where can applicants find information about fe...   \n",
       "15  What should pregnant applicants for temporary ...   \n",
       "16  What must an applicant demonstrate when applyi...   \n",
       "17  What changes are being made to the RSE scheme ...   \n",
       "18  How long is the resettlement program at the Mā...   \n",
       "19  When did applications for the 2021 Resident Vi...   \n",
       "20                     Where is the Kyiv VAC located?   \n",
       "21  How can you protect yourself from immigration ...   \n",
       "22  What change was made to temporary work visas i...   \n",
       "23  Where should visitors look to find various opt...   \n",
       "24  What is the minimum amount required to cover m...   \n",
       "25  As of June 2024, how many 2021 Resident Visa a...   \n",
       "26  How can cruise ship passengers with an NZeTA e...   \n",
       "27  Is there a way to go back to the Immigration N...   \n",
       "28  In what circumstances might certified English ...   \n",
       "29  What type of organization does this content li...   \n",
       "30  What should you do if you want to change emplo...   \n",
       "31  What should you do if there is glare or reflec...   \n",
       "32  What types of evidence can demonstrate that a ...   \n",
       "33  Is the Bank of China loyalty programme partner...   \n",
       "34  Is there a website available for visa applicat...   \n",
       "35  How can you prevent glare on glasses in a pass...   \n",
       "36       Where can immigration instructions be found?   \n",
       "37  Is there a specific homepage for Immigration N...   \n",
       "38  Is the VAC in Kyiv currently open for visa app...   \n",
       "39  What legal compliance is necessary for employe...   \n",
       "40  What is the minimum hourly rate required durin...   \n",
       "41  Who is allowed to provide immigration advice i...   \n",
       "42  What are the three skills-based pathways for o...   \n",
       "43  How do employers initiate the recruitment proc...   \n",
       "44  How can employers ensure they are complying wi...   \n",
       "45  What information do you need to provide for a ...   \n",
       "46  What steps must an employer take before hiring...   \n",
       "47  How can you transfer a valid visa to a new pas...   \n",
       "48  What are some situations where fee refunds may...   \n",
       "49  What is the annual quota for Refugee Family Su...   \n",
       "\n",
       "                                               answer  \n",
       "0   Based on the information provided, here are so...  \n",
       "1   Based on the information provided, some exampl...  \n",
       "2   Based on the information provided, the key emp...  \n",
       "3   Based on the information provided, here are th...  \n",
       "4   Based on the context provided, the main organi...  \n",
       "5   Based on the information provided, unlicensed ...  \n",
       "6   Based on the information provided, here are th...  \n",
       "7   Based on the information provided, employers a...  \n",
       "8   Based on the information provided, it does not...  \n",
       "9   Based on the information provided, an eVisa di...  \n",
       "10  Based on the information provided, Immigration...  \n",
       "11  Based on the information provided, the followi...  \n",
       "12  Based on the information provided, some exampl...  \n",
       "13  Based on the information provided, New Zealand...  \n",
       "14  Based on the information provided, applicants ...  \n",
       "15  Based on the information provided, pregnant ap...  \n",
       "16  Based on the information provided, an applican...  \n",
       "17  Based on the information provided, here are th...  \n",
       "18  According to the information provided, the res...  \n",
       "19  Based on the information provided:\\n\\n- Applic...  \n",
       "20  Based on the information provided, the Kyiv VA...  \n",
       "21  Here are some tips to protect yourself from im...  \n",
       "22  Based on the information provided, the key cha...  \n",
       "23  Based on the information provided in the conte...  \n",
       "24  Based on the information provided, the minimum...  \n",
       "25  According to the information provided, as of a...  \n",
       "26  Based on the information provided, here are th...  \n",
       "27  Based on the information provided, it seems th...  \n",
       "28  Based on the information provided, certified E...  \n",
       "29  Based on the context provided, this content li...  \n",
       "30  Here are the key steps if you want to change e...  \n",
       "31  Based on the information provided, here are th...  \n",
       "32  Based on the information provided, the key typ...  \n",
       "33  Based on the information provided in the conte...  \n",
       "34  Based on the information provided, there does ...  \n",
       "35  Here are some tips to prevent glare on glasses...  \n",
       "36  Based on the information provided, immigration...  \n",
       "37  The text does not mention a specific homepage ...  \n",
       "38  Based on the information provided, it seems th...  \n",
       "39  Based on the information provided, the key leg...  \n",
       "40  Based on the information provided, the minimum...  \n",
       "41  Based on the information provided, the key poi...  \n",
       "42  Based on the information provided, the three s...  \n",
       "43  Based on the information provided, here are th...  \n",
       "44  Based on the information provided, here are th...  \n",
       "45  Based on the information provided, the key det...  \n",
       "46  Based on the information provided, here are th...  \n",
       "47  Based on the information provided, here are th...  \n",
       "48  Based on the information provided, here are so...  \n",
       "49  According to the information provided, the ann...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores4 = evaluate_rag(test_subset, \"vector+simple\")\n",
    "scores4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eecb9e14-b923-4d46-9aff-91b9ff9a054a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 50/50 [03:34<00:00,  4.29s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_3</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.135593</td>\n",
       "      <td>0.078125</td>\n",
       "      <td>0.112994</td>\n",
       "      <td>What types of evidence can be provided to prov...</td>\n",
       "      <td>To prove ownership of funds and assets for inv...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides several specific...</td>\n",
       "      <td>0.146667</td>\n",
       "      <td>0.081818</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>What are some examples of actions that would b...</td>\n",
       "      <td>Based on the information provided, some exampl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer covers the key employer o...</td>\n",
       "      <td>0.160584</td>\n",
       "      <td>0.093750</td>\n",
       "      <td>0.160584</td>\n",
       "      <td>What are the employer's obligations concerning...</td>\n",
       "      <td>The key points related to the employer's oblig...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.205479</td>\n",
       "      <td>0.101382</td>\n",
       "      <td>0.191781</td>\n",
       "      <td>What should a visa holder do if their employer...</td>\n",
       "      <td>Here is my answer to the question:\\n\\nIf a vis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not mention any spec...</td>\n",
       "      <td>0.131579</td>\n",
       "      <td>0.043011</td>\n",
       "      <td>0.131579</td>\n",
       "      <td>What organization is mentioned in the text?</td>\n",
       "      <td>The text does not mention any specific organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.117647</td>\n",
       "      <td>0.037879</td>\n",
       "      <td>0.117647</td>\n",
       "      <td>What types of information can unlicensed indiv...</td>\n",
       "      <td>Based on the question and context provided, th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.135802</td>\n",
       "      <td>0.067511</td>\n",
       "      <td>0.135802</td>\n",
       "      <td>What types of evidence should be included when...</td>\n",
       "      <td>When applying to extend a student stay in New ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides relevant informa...</td>\n",
       "      <td>0.260000</td>\n",
       "      <td>0.134328</td>\n",
       "      <td>0.260000</td>\n",
       "      <td>Where can employers and employees find informa...</td>\n",
       "      <td>Based on the question, the key points are:\\n\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.218487</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.218487</td>\n",
       "      <td>Is there a way to return to the homepage from ...</td>\n",
       "      <td>Based on the question, the key points are:\\n\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a clear and deta...</td>\n",
       "      <td>0.160584</td>\n",
       "      <td>0.071066</td>\n",
       "      <td>0.160584</td>\n",
       "      <td>What is an eVisa and how does it differ from t...</td>\n",
       "      <td>An **eVisa** is an electronic visa that is sto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.155172</td>\n",
       "      <td>0.082840</td>\n",
       "      <td>0.137931</td>\n",
       "      <td>What are the operating hours for Immigration N...</td>\n",
       "      <td>Based on the information provided, I do not ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides a general overvi...</td>\n",
       "      <td>0.103704</td>\n",
       "      <td>0.042553</td>\n",
       "      <td>0.103704</td>\n",
       "      <td>Who can sponsor Sponsorship Visa applications ...</td>\n",
       "      <td>Based on the information provided, the key poi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant ex...</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.091954</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>What are some examples of organizations that c...</td>\n",
       "      <td>Based on the question, some examples of organi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides relevant context...</td>\n",
       "      <td>0.229167</td>\n",
       "      <td>0.066116</td>\n",
       "      <td>0.229167</td>\n",
       "      <td>How many refugees is New Zealand aiming to res...</td>\n",
       "      <td>According to the provided context, New Zealand...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.212766</td>\n",
       "      <td>0.114754</td>\n",
       "      <td>0.212766</td>\n",
       "      <td>Where can applicants find information about fe...</td>\n",
       "      <td>To answer your question, applicants can find i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.176000</td>\n",
       "      <td>0.079096</td>\n",
       "      <td>0.176000</td>\n",
       "      <td>What should pregnant applicants for temporary ...</td>\n",
       "      <td>Based on the information provided, pregnant ap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.132530</td>\n",
       "      <td>0.062780</td>\n",
       "      <td>0.108434</td>\n",
       "      <td>What must an applicant demonstrate when applyi...</td>\n",
       "      <td>When applying for a second or subsequent perma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer demonstrates an understan...</td>\n",
       "      <td>0.183486</td>\n",
       "      <td>0.116883</td>\n",
       "      <td>0.165138</td>\n",
       "      <td>What changes are being made to the RSE scheme ...</td>\n",
       "      <td>Based on the information provided in the quest...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant co...</td>\n",
       "      <td>0.140351</td>\n",
       "      <td>0.087500</td>\n",
       "      <td>0.122807</td>\n",
       "      <td>How long is the resettlement program at the Mā...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not provide any info...</td>\n",
       "      <td>0.160920</td>\n",
       "      <td>0.086957</td>\n",
       "      <td>0.160920</td>\n",
       "      <td>When did applications for the 2021 Resident Vi...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.089888</td>\n",
       "      <td>0.033898</td>\n",
       "      <td>0.089888</td>\n",
       "      <td>Where is the Kyiv VAC located?</td>\n",
       "      <td>The given question is about the location of th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.090395</td>\n",
       "      <td>0.047619</td>\n",
       "      <td>0.090395</td>\n",
       "      <td>How can you protect yourself from immigration ...</td>\n",
       "      <td>Here are some tips to protect yourself from im...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer does not contain any info...</td>\n",
       "      <td>0.157895</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>0.157895</td>\n",
       "      <td>What change was made to temporary work visas i...</td>\n",
       "      <td>The question asks about changes made to tempor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer does not directly address...</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.079208</td>\n",
       "      <td>0.175000</td>\n",
       "      <td>Where should visitors look to find various opt...</td>\n",
       "      <td>Based on the question provided, there does not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.241379</td>\n",
       "      <td>0.153846</td>\n",
       "      <td>0.241379</td>\n",
       "      <td>What is the minimum amount required to cover m...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>NON_RELEVANT</td>\n",
       "      <td>The generated answer clearly indicates that th...</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>0.068493</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>As of June 2024, how many 2021 Resident Visa a...</td>\n",
       "      <td>Unfortunately, the given context does not prov...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses how cr...</td>\n",
       "      <td>0.134328</td>\n",
       "      <td>0.062500</td>\n",
       "      <td>0.119403</td>\n",
       "      <td>How can cruise ship passengers with an NZeTA e...</td>\n",
       "      <td>According to the provided context, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.209677</td>\n",
       "      <td>0.144578</td>\n",
       "      <td>0.209677</td>\n",
       "      <td>Is there a way to go back to the Immigration N...</td>\n",
       "      <td>Based on the question, there seems to be no sp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.132353</td>\n",
       "      <td>0.047170</td>\n",
       "      <td>0.117647</td>\n",
       "      <td>In what circumstances might certified English ...</td>\n",
       "      <td>In the context of New Zealand visa application...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer accurately identifies tha...</td>\n",
       "      <td>0.113636</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>0.113636</td>\n",
       "      <td>What type of organization does this content li...</td>\n",
       "      <td>Based on the context provided, this content li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides detailed and rel...</td>\n",
       "      <td>0.170213</td>\n",
       "      <td>0.058824</td>\n",
       "      <td>0.127660</td>\n",
       "      <td>What should you do if you want to change emplo...</td>\n",
       "      <td>To change employment while on a 2021 Interim V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides relevant and use...</td>\n",
       "      <td>0.148148</td>\n",
       "      <td>0.084211</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>What should you do if there is glare or reflec...</td>\n",
       "      <td>The question is not related to New Zealand vis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.132450</td>\n",
       "      <td>0.057692</td>\n",
       "      <td>0.119205</td>\n",
       "      <td>What types of evidence can demonstrate that a ...</td>\n",
       "      <td>The types of evidence that can demonstrate a c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer is partially relevant as ...</td>\n",
       "      <td>0.190476</td>\n",
       "      <td>0.109091</td>\n",
       "      <td>0.190476</td>\n",
       "      <td>Is the Bank of China loyalty programme partner...</td>\n",
       "      <td>Based on the question and the lack of provided...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.113821</td>\n",
       "      <td>0.054054</td>\n",
       "      <td>0.113821</td>\n",
       "      <td>Is there a website available for visa applicat...</td>\n",
       "      <td>Based on the question asked, here is the respo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.131387</td>\n",
       "      <td>0.061856</td>\n",
       "      <td>0.116788</td>\n",
       "      <td>How can you prevent glare on glasses in a pass...</td>\n",
       "      <td>The question is not related to New Zealand vis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.068966</td>\n",
       "      <td>0.011834</td>\n",
       "      <td>0.068966</td>\n",
       "      <td>Where can immigration instructions be found?</td>\n",
       "      <td>Based on the context provided, the question is...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.282051</td>\n",
       "      <td>0.161616</td>\n",
       "      <td>0.230769</td>\n",
       "      <td>Is there a specific homepage for Immigration N...</td>\n",
       "      <td>Based on the provided context, there is no spe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that the giv...</td>\n",
       "      <td>0.149533</td>\n",
       "      <td>0.082192</td>\n",
       "      <td>0.149533</td>\n",
       "      <td>Is the VAC in Kyiv currently open for visa app...</td>\n",
       "      <td>The given question and context do not contain ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges the lack of ...</td>\n",
       "      <td>0.220000</td>\n",
       "      <td>0.112676</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>What legal compliance is necessary for employe...</td>\n",
       "      <td>The question is asking about the legal complia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.186441</td>\n",
       "      <td>0.097561</td>\n",
       "      <td>0.186441</td>\n",
       "      <td>What is the minimum hourly rate required durin...</td>\n",
       "      <td>Based on the information provided, the questio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a comprehensive ...</td>\n",
       "      <td>0.132075</td>\n",
       "      <td>0.090226</td>\n",
       "      <td>0.132075</td>\n",
       "      <td>Who is allowed to provide immigration advice i...</td>\n",
       "      <td>According to the New Zealand government, the f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer directly addresses the qu...</td>\n",
       "      <td>0.131148</td>\n",
       "      <td>0.059880</td>\n",
       "      <td>0.131148</td>\n",
       "      <td>What are the three skills-based pathways for o...</td>\n",
       "      <td>The three main skills-based pathways for obtai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides a thorough and r...</td>\n",
       "      <td>0.096552</td>\n",
       "      <td>0.068293</td>\n",
       "      <td>0.096552</td>\n",
       "      <td>How do employers initiate the recruitment proc...</td>\n",
       "      <td>To answer your question about how employers in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The generated answer provides comprehensive an...</td>\n",
       "      <td>0.100629</td>\n",
       "      <td>0.044053</td>\n",
       "      <td>0.100629</td>\n",
       "      <td>How can employers ensure they are complying wi...</td>\n",
       "      <td>To ensure compliance with employment laws for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.024242</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>What information do you need to provide for a ...</td>\n",
       "      <td>To verify a New Zealand visa with a third part...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.097902</td>\n",
       "      <td>0.029412</td>\n",
       "      <td>0.083916</td>\n",
       "      <td>What steps must an employer take before hiring...</td>\n",
       "      <td>To hire a migrant worker in New Zealand, an em...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides a general overvi...</td>\n",
       "      <td>0.121739</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>0.104348</td>\n",
       "      <td>How can you transfer a valid visa to a new pas...</td>\n",
       "      <td>To transfer a valid visa to a new passport, th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer provides some relevant in...</td>\n",
       "      <td>0.118812</td>\n",
       "      <td>0.073529</td>\n",
       "      <td>0.118812</td>\n",
       "      <td>What are some situations where fee refunds may...</td>\n",
       "      <td>Based on the question, some situations where f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The generated answer acknowledges that it does...</td>\n",
       "      <td>0.178218</td>\n",
       "      <td>0.118519</td>\n",
       "      <td>0.178218</td>\n",
       "      <td>What is the annual quota for Refugee Family Su...</td>\n",
       "      <td>Based on the question and the provided context...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Relevance                                        Explanation  \\\n",
       "0          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "1          RELEVANT  The generated answer provides several specific...   \n",
       "2          RELEVANT  The generated answer covers the key employer o...   \n",
       "3          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "4      NON_RELEVANT  The generated answer does not mention any spec...   \n",
       "5   PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "6          RELEVANT  The generated answer provides a comprehensive ...   \n",
       "7          RELEVANT  The generated answer provides relevant informa...   \n",
       "8          RELEVANT  The generated answer directly addresses the qu...   \n",
       "9          RELEVANT  The generated answer provides a clear and deta...   \n",
       "10  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "11  PARTLY_RELEVANT  The generated answer provides a general overvi...   \n",
       "12  PARTLY_RELEVANT  The generated answer provides some relevant ex...   \n",
       "13  PARTLY_RELEVANT  The generated answer provides relevant context...   \n",
       "14         RELEVANT  The generated answer provides detailed and rel...   \n",
       "15         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "16         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "17  PARTLY_RELEVANT  The generated answer demonstrates an understan...   \n",
       "18  PARTLY_RELEVANT  The generated answer provides some relevant co...   \n",
       "19     NON_RELEVANT  The generated answer does not provide any info...   \n",
       "20  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "21         RELEVANT  The generated answer provides comprehensive an...   \n",
       "22     NON_RELEVANT  The generated answer does not contain any info...   \n",
       "23  PARTLY_RELEVANT  The generated answer does not directly address...   \n",
       "24  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "25     NON_RELEVANT  The generated answer clearly indicates that th...   \n",
       "26         RELEVANT  The generated answer directly addresses how cr...   \n",
       "27         RELEVANT  The generated answer directly addresses the qu...   \n",
       "28         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "29         RELEVANT  The generated answer accurately identifies tha...   \n",
       "30         RELEVANT  The generated answer provides detailed and rel...   \n",
       "31         RELEVANT  The generated answer provides relevant and use...   \n",
       "32         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "33  PARTLY_RELEVANT  The generated answer is partially relevant as ...   \n",
       "34         RELEVANT  The generated answer directly addresses the qu...   \n",
       "35         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "36         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "37         RELEVANT  The generated answer directly addresses the qu...   \n",
       "38  PARTLY_RELEVANT  The generated answer acknowledges that the giv...   \n",
       "39  PARTLY_RELEVANT  The generated answer acknowledges the lack of ...   \n",
       "40  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "41         RELEVANT  The generated answer provides a comprehensive ...   \n",
       "42         RELEVANT  The generated answer directly addresses the qu...   \n",
       "43         RELEVANT  The generated answer provides a thorough and r...   \n",
       "44         RELEVANT  The generated answer provides comprehensive an...   \n",
       "45  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "46  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "47  PARTLY_RELEVANT  The generated answer provides a general overvi...   \n",
       "48  PARTLY_RELEVANT  The generated answer provides some relevant in...   \n",
       "49  PARTLY_RELEVANT  The generated answer acknowledges that it does...   \n",
       "\n",
       "     ROUGE_1   ROUGE_2   ROUGE_3  \\\n",
       "0   0.135593  0.078125  0.112994   \n",
       "1   0.146667  0.081818  0.133333   \n",
       "2   0.160584  0.093750  0.160584   \n",
       "3   0.205479  0.101382  0.191781   \n",
       "4   0.131579  0.043011  0.131579   \n",
       "5   0.117647  0.037879  0.117647   \n",
       "6   0.135802  0.067511  0.135802   \n",
       "7   0.260000  0.134328  0.260000   \n",
       "8   0.218487  0.150000  0.218487   \n",
       "9   0.160584  0.071066  0.160584   \n",
       "10  0.155172  0.082840  0.137931   \n",
       "11  0.103704  0.042553  0.103704   \n",
       "12  0.142857  0.091954  0.142857   \n",
       "13  0.229167  0.066116  0.229167   \n",
       "14  0.212766  0.114754  0.212766   \n",
       "15  0.176000  0.079096  0.176000   \n",
       "16  0.132530  0.062780  0.108434   \n",
       "17  0.183486  0.116883  0.165138   \n",
       "18  0.140351  0.087500  0.122807   \n",
       "19  0.160920  0.086957  0.160920   \n",
       "20  0.089888  0.033898  0.089888   \n",
       "21  0.090395  0.047619  0.090395   \n",
       "22  0.157895  0.095238  0.157895   \n",
       "23  0.200000  0.079208  0.175000   \n",
       "24  0.241379  0.153846  0.241379   \n",
       "25  0.145833  0.068493  0.145833   \n",
       "26  0.134328  0.062500  0.119403   \n",
       "27  0.209677  0.144578  0.209677   \n",
       "28  0.132353  0.047170  0.117647   \n",
       "29  0.113636  0.033333  0.113636   \n",
       "30  0.170213  0.058824  0.127660   \n",
       "31  0.148148  0.084211  0.133333   \n",
       "32  0.132450  0.057692  0.119205   \n",
       "33  0.190476  0.109091  0.190476   \n",
       "34  0.113821  0.054054  0.113821   \n",
       "35  0.131387  0.061856  0.116788   \n",
       "36  0.068966  0.011834  0.068966   \n",
       "37  0.282051  0.161616  0.230769   \n",
       "38  0.149533  0.082192  0.149533   \n",
       "39  0.220000  0.112676  0.200000   \n",
       "40  0.186441  0.097561  0.186441   \n",
       "41  0.132075  0.090226  0.132075   \n",
       "42  0.131148  0.059880  0.131148   \n",
       "43  0.096552  0.068293  0.096552   \n",
       "44  0.100629  0.044053  0.100629   \n",
       "45  0.166667  0.024242  0.150000   \n",
       "46  0.097902  0.029412  0.083916   \n",
       "47  0.121739  0.069767  0.104348   \n",
       "48  0.118812  0.073529  0.118812   \n",
       "49  0.178218  0.118519  0.178218   \n",
       "\n",
       "                                             question  \\\n",
       "0   What types of evidence can be provided to prov...   \n",
       "1   What are some examples of actions that would b...   \n",
       "2   What are the employer's obligations concerning...   \n",
       "3   What should a visa holder do if their employer...   \n",
       "4         What organization is mentioned in the text?   \n",
       "5   What types of information can unlicensed indiv...   \n",
       "6   What types of evidence should be included when...   \n",
       "7   Where can employers and employees find informa...   \n",
       "8   Is there a way to return to the homepage from ...   \n",
       "9   What is an eVisa and how does it differ from t...   \n",
       "10  What are the operating hours for Immigration N...   \n",
       "11  Who can sponsor Sponsorship Visa applications ...   \n",
       "12  What are some examples of organizations that c...   \n",
       "13  How many refugees is New Zealand aiming to res...   \n",
       "14  Where can applicants find information about fe...   \n",
       "15  What should pregnant applicants for temporary ...   \n",
       "16  What must an applicant demonstrate when applyi...   \n",
       "17  What changes are being made to the RSE scheme ...   \n",
       "18  How long is the resettlement program at the Mā...   \n",
       "19  When did applications for the 2021 Resident Vi...   \n",
       "20                     Where is the Kyiv VAC located?   \n",
       "21  How can you protect yourself from immigration ...   \n",
       "22  What change was made to temporary work visas i...   \n",
       "23  Where should visitors look to find various opt...   \n",
       "24  What is the minimum amount required to cover m...   \n",
       "25  As of June 2024, how many 2021 Resident Visa a...   \n",
       "26  How can cruise ship passengers with an NZeTA e...   \n",
       "27  Is there a way to go back to the Immigration N...   \n",
       "28  In what circumstances might certified English ...   \n",
       "29  What type of organization does this content li...   \n",
       "30  What should you do if you want to change emplo...   \n",
       "31  What should you do if there is glare or reflec...   \n",
       "32  What types of evidence can demonstrate that a ...   \n",
       "33  Is the Bank of China loyalty programme partner...   \n",
       "34  Is there a website available for visa applicat...   \n",
       "35  How can you prevent glare on glasses in a pass...   \n",
       "36       Where can immigration instructions be found?   \n",
       "37  Is there a specific homepage for Immigration N...   \n",
       "38  Is the VAC in Kyiv currently open for visa app...   \n",
       "39  What legal compliance is necessary for employe...   \n",
       "40  What is the minimum hourly rate required durin...   \n",
       "41  Who is allowed to provide immigration advice i...   \n",
       "42  What are the three skills-based pathways for o...   \n",
       "43  How do employers initiate the recruitment proc...   \n",
       "44  How can employers ensure they are complying wi...   \n",
       "45  What information do you need to provide for a ...   \n",
       "46  What steps must an employer take before hiring...   \n",
       "47  How can you transfer a valid visa to a new pas...   \n",
       "48  What are some situations where fee refunds may...   \n",
       "49  What is the annual quota for Refugee Family Su...   \n",
       "\n",
       "                                               answer  \n",
       "0   To prove ownership of funds and assets for inv...  \n",
       "1   Based on the information provided, some exampl...  \n",
       "2   The key points related to the employer's oblig...  \n",
       "3   Here is my answer to the question:\\n\\nIf a vis...  \n",
       "4   The text does not mention any specific organiz...  \n",
       "5   Based on the question and context provided, th...  \n",
       "6   When applying to extend a student stay in New ...  \n",
       "7   Based on the question, the key points are:\\n\\n...  \n",
       "8   Based on the question, the key points are:\\n\\n...  \n",
       "9   An **eVisa** is an electronic visa that is sto...  \n",
       "10  Based on the information provided, I do not ha...  \n",
       "11  Based on the information provided, the key poi...  \n",
       "12  Based on the question, some examples of organi...  \n",
       "13  According to the provided context, New Zealand...  \n",
       "14  To answer your question, applicants can find i...  \n",
       "15  Based on the information provided, pregnant ap...  \n",
       "16  When applying for a second or subsequent perma...  \n",
       "17  Based on the information provided in the quest...  \n",
       "18  Based on the information provided, the questio...  \n",
       "19  Based on the information provided, the questio...  \n",
       "20  The given question is about the location of th...  \n",
       "21  Here are some tips to protect yourself from im...  \n",
       "22  The question asks about changes made to tempor...  \n",
       "23  Based on the question provided, there does not...  \n",
       "24  Based on the information provided, the questio...  \n",
       "25  Unfortunately, the given context does not prov...  \n",
       "26  According to the provided context, the questio...  \n",
       "27  Based on the question, there seems to be no sp...  \n",
       "28  In the context of New Zealand visa application...  \n",
       "29  Based on the context provided, this content li...  \n",
       "30  To change employment while on a 2021 Interim V...  \n",
       "31  The question is not related to New Zealand vis...  \n",
       "32  The types of evidence that can demonstrate a c...  \n",
       "33  Based on the question and the lack of provided...  \n",
       "34  Based on the question asked, here is the respo...  \n",
       "35  The question is not related to New Zealand vis...  \n",
       "36  Based on the context provided, the question is...  \n",
       "37  Based on the provided context, there is no spe...  \n",
       "38  The given question and context do not contain ...  \n",
       "39  The question is asking about the legal complia...  \n",
       "40  Based on the information provided, the questio...  \n",
       "41  According to the New Zealand government, the f...  \n",
       "42  The three main skills-based pathways for obtai...  \n",
       "43  To answer your question about how employers in...  \n",
       "44  To ensure compliance with employment laws for ...  \n",
       "45  To verify a New Zealand visa with a third part...  \n",
       "46  To hire a migrant worker in New Zealand, an em...  \n",
       "47  To transfer a valid visa to a new passport, th...  \n",
       "48  Based on the question, some situations where f...  \n",
       "49  Based on the question and the provided context...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores5 = evaluate_rag(test_subset, \"vector+structured\")\n",
    "scores5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2c0e699-9867-4ca7-943e-9cb30d2168d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores6 = evaluate_rag(test_subset, \"vector+big\")\n",
    "scores6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edea22bc-ab69-4bab-a30a-c7ebbbd2870b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           46\n",
      "PARTLY_RELEVANT     3\n",
      "NON_RELEVANT        1\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.185495\n",
      "ROUGE_2    0.106079\n",
      "ROUGE_3    0.172999\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f93452a-9628-40f0-b9a5-67da05636d49",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           26\n",
      "PARTLY_RELEVANT    18\n",
      "NON_RELEVANT        6\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.163282\n",
      "ROUGE_2    0.085720\n",
      "ROUGE_3    0.154784\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb116df-6c43-47ee-a525-f24febf6a80e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT    50\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.217819\n",
      "ROUGE_2    0.116029\n",
      "ROUGE_3    0.199039\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cf6e3e4-0976-4099-982f-74d072845abf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           48\n",
      "PARTLY_RELEVANT     2\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.172731\n",
      "ROUGE_2    0.086049\n",
      "ROUGE_3    0.164021\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c905935-c84e-48bd-9065-10e498b41b33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           27\n",
      "PARTLY_RELEVANT    19\n",
      "NON_RELEVANT        4\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.155240\n",
      "ROUGE_2    0.078514\n",
      "ROUGE_3    0.147519\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7239a6b4-4873-432f-b504-a53131371733",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RELEVANCE STATS\n",
      "Relevance\n",
      "RELEVANT           48\n",
      "PARTLY_RELEVANT     2\n",
      "Name: count, dtype: int64\n",
      "\n",
      "ROUGE\n",
      "ROUGE_1    0.195728\n",
      "ROUGE_2    0.091203\n",
      "ROUGE_3    0.175914\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "display_stats(scores6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28b387cd-9b64-489a-91a0-b57f3066ceae",
   "metadata": {},
   "source": [
    "| Version | RELEVANT | PARTLY_RELEVANT | NON_RELEVANT | ROUGE_1 | ROUGE_2 | ROUGE_3 |\n",
    "|---------|----------|-----------------|--------------|---------|---------|---------|\n",
    "| scores1 (\"text+simple\") | 46 | 3 | 1 | 0.185495 | 0.106079 | 0.172999 |\n",
    "| scores2 (\"text+structured\") | 26 | 18 | 6 | 0.163282 | 0.085720 | 0.154784 |\n",
    "| scores3 (\"text+big\") | 50 | 0 | 0 | 0.217819 | 0.116029 | 0.199039 |\n",
    "| scores4 (\"vector+simple\") | 48 | 2 | 0 | 0.172731 | 0.086049 | 0.164021 |\n",
    "| scores5 (\"vector+structured\") | 27 | 19 | 4 | 0.155240 | 0.078514 | 0.147519 |\n",
    "| scores6 (\"vector+big\") | 48 | 2 | 0 | 0.195728 | 0.091203 | 0.175914 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f4b58f3-0e50-4ca9-8445-76578cc3001c",
   "metadata": {},
   "source": [
    "## Evaluate token usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c042c39-ef2a-4244-8113-7608b974bac6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def llm_haiku_tokens(prompt):\n",
    "    response = clientA.messages.create(\n",
    "        model=\"claude-3-haiku-20240307\",\n",
    "        max_tokens=500,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\n",
    "                        \"type\": \"text\",\n",
    "                        \"text\": prompt\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        ]\n",
    "    )\n",
    "    return response.content[0].text, response.usage.input_tokens + response.usage.output_tokens\n",
    "\n",
    "llm = llm_haiku_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b78cd5a-40da-49f9-a69c-1430f9ebc075",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_relevance(question: str, answer_llm: str) -> dict:\n",
    "    prompt_template = \"\"\"\n",
    "Evaluate answer relevance to question. Classify as NON_RELEVANT, PARTLY_RELEVANT, or RELEVANT.\n",
    "Q: {q}\n",
    "A: {a}\n",
    "Respond with JSON:\n",
    "{{\"Relevance\": \"NON_RELEVANT\"|\"PARTLY_RELEVANT\"|\"RELEVANT\", \"Explanation\": \"Brief reason\"}}\n",
    "\"\"\".strip()\n",
    "\n",
    "    evaluation_prompt = prompt_template.format(q=question, a=answer_llm)\n",
    "    evaluation_response, token_count = llm_haiku(evaluation_prompt)\n",
    "\n",
    "    try:\n",
    "        evaluation_result = json.loads(evaluation_response)\n",
    "        if not all(key in evaluation_result for key in [\"Relevance\", \"Explanation\"]):\n",
    "            raise ValueError(\"Missing expected fields\")\n",
    "        if evaluation_result[\"Relevance\"] not in [\"NON_RELEVANT\", \"PARTLY_RELEVANT\", \"RELEVANT\"]:\n",
    "            raise ValueError(\"Invalid Relevance value\")\n",
    "    except (json.JSONDecodeError, ValueError) as e:\n",
    "        return {\n",
    "            \"Relevance\": \"ERROR\",\n",
    "            \"Explanation\": f\"Error processing LLM response: {str(e)}\",\n",
    "            \"TokensUsed\": token_count\n",
    "        }\n",
    "\n",
    "    rouge = Rouge()\n",
    "    scores = rouge.get_scores(answer_llm, question)[0]\n",
    "    \n",
    "    evaluation_result.update({\n",
    "        \"ROUGE_1\": scores['rouge-1']['f'],\n",
    "        \"ROUGE_2\": scores['rouge-2']['f'],\n",
    "        \"ROUGE_L\": scores['rouge-l']['f'],\n",
    "        \"TokensUsed\": token_count\n",
    "    })\n",
    "\n",
    "    return evaluation_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cf0635d-b59e-428b-ba05-cf89c12c18a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def rag(query, version):\n",
    "    if version == \"text+simple\":\n",
    "        search = elastic_search_boost\n",
    "        build_prompt = build_basic_prompt\n",
    "        extractor = extract_basic_answer\n",
    "    elif version == \"text+structured\":\n",
    "        search = elastic_search_boost\n",
    "        build_prompt = build_structured_prompt\n",
    "        extractor = extract_structured_answer\n",
    "    elif version == \"text+big\":\n",
    "        search = elastic_search_boost\n",
    "        build_prompt = build_big_prompt\n",
    "        extractor = extract_big_answer\n",
    "    elif version == \"vector+simple\":\n",
    "        search = elastic_search_combined_10\n",
    "        build_prompt = build_basic_prompt\n",
    "        extractor = extract_basic_answer\n",
    "    elif version == \"vector+structured\":\n",
    "        search = elastic_search_combined_10\n",
    "        build_prompt = build_structured_prompt\n",
    "        extractor = extract_structured_answer\n",
    "    elif version == \"vector+big\":\n",
    "        search = elastic_search_combined_10\n",
    "        build_prompt = build_big_prompt\n",
    "        extractor = extract_big_answer\n",
    "    \n",
    "    search_results = search(query)\n",
    "    prompt = build_prompt(query, search_results)\n",
    "    answer, tokens = llm(prompt)\n",
    "    return extractor(answer), tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "875ff2e3-325c-46cf-a6a4-0c4cb33274a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_rag(examples:list, rag_verstion:str) -> pd.DataFrame:\n",
    "    relevance_list = []\n",
    "    for row in tqdm(examples):\n",
    "        question = row[\"question\"]\n",
    "        answer, tokens = rag(question, rag_verstion)\n",
    "        score = evaluate_relevance(question, answer)\n",
    "        score[\"question\"] = question\n",
    "        score[\"answer\"] = answer\n",
    "        score[\"tokens\"] = tokens\n",
    "        relevance_list.append(score)\n",
    "    return pd.DataFrame(relevance_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "092c563a-110c-48d7-8b61-bf1167755419",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 10/10 [00:37<00:00,  3.76s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_L</th>\n",
       "      <th>TokensUsed</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides detailed information on wh...</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.100629</td>\n",
       "      <td>0.205128</td>\n",
       "      <td>443</td>\n",
       "      <td>Where can applicants find information about th...</td>\n",
       "      <td>According to the information provided, applica...</td>\n",
       "      <td>4742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides two specific steps a user ...</td>\n",
       "      <td>0.322581</td>\n",
       "      <td>0.219178</td>\n",
       "      <td>0.322581</td>\n",
       "      <td>300</td>\n",
       "      <td>How can someone get back to the main page of I...</td>\n",
       "      <td>Based on the information provided in the conte...</td>\n",
       "      <td>4577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses the question by ...</td>\n",
       "      <td>0.325581</td>\n",
       "      <td>0.235294</td>\n",
       "      <td>0.325581</td>\n",
       "      <td>223</td>\n",
       "      <td>Who certifies all immigration instructions and...</td>\n",
       "      <td>According to the information provided, the **M...</td>\n",
       "      <td>3508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly and accurately identifies ...</td>\n",
       "      <td>0.177215</td>\n",
       "      <td>0.122449</td>\n",
       "      <td>0.177215</td>\n",
       "      <td>366</td>\n",
       "      <td>What are the two main components of the RSE ap...</td>\n",
       "      <td>The two main components of the RSE application...</td>\n",
       "      <td>4120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides detailed and relevant step...</td>\n",
       "      <td>0.150943</td>\n",
       "      <td>0.062112</td>\n",
       "      <td>0.132075</td>\n",
       "      <td>499</td>\n",
       "      <td>How can you transfer your visa to a new passport?</td>\n",
       "      <td>To transfer your valid New Zealand visa to a n...</td>\n",
       "      <td>5431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses the question by ...</td>\n",
       "      <td>0.216216</td>\n",
       "      <td>0.175824</td>\n",
       "      <td>0.216216</td>\n",
       "      <td>315</td>\n",
       "      <td>When does the selection of expressions of inte...</td>\n",
       "      <td>According to the information provided, the sel...</td>\n",
       "      <td>3929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The answer provides information about what hap...</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>257</td>\n",
       "      <td>What will happen after you state your language?</td>\n",
       "      <td>If you call Immigration New Zealand and do not...</td>\n",
       "      <td>7803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides the relevant information t...</td>\n",
       "      <td>0.363636</td>\n",
       "      <td>0.187500</td>\n",
       "      <td>0.327273</td>\n",
       "      <td>314</td>\n",
       "      <td>How can visitors return to the main page of Im...</td>\n",
       "      <td>Based on the information provided in the conte...</td>\n",
       "      <td>3860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides detailed information about...</td>\n",
       "      <td>0.165414</td>\n",
       "      <td>0.078212</td>\n",
       "      <td>0.150376</td>\n",
       "      <td>437</td>\n",
       "      <td>How have the employment agreements for RSE wor...</td>\n",
       "      <td>As of October 1, 2023, the employment agreemen...</td>\n",
       "      <td>7875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses the steps to req...</td>\n",
       "      <td>0.225000</td>\n",
       "      <td>0.056604</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>337</td>\n",
       "      <td>What should you do to request a refund if you ...</td>\n",
       "      <td>According to the information provided, if you ...</td>\n",
       "      <td>5251</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Relevance                                        Explanation  \\\n",
       "0         RELEVANT  The answer provides detailed information on wh...   \n",
       "1         RELEVANT  The answer provides two specific steps a user ...   \n",
       "2         RELEVANT  The answer directly addresses the question by ...   \n",
       "3         RELEVANT  The answer directly and accurately identifies ...   \n",
       "4         RELEVANT  The answer provides detailed and relevant step...   \n",
       "5         RELEVANT  The answer directly addresses the question by ...   \n",
       "6  PARTLY_RELEVANT  The answer provides information about what hap...   \n",
       "7         RELEVANT  The answer provides the relevant information t...   \n",
       "8         RELEVANT  The answer provides detailed information about...   \n",
       "9         RELEVANT  The answer directly addresses the steps to req...   \n",
       "\n",
       "    ROUGE_1   ROUGE_2   ROUGE_L  TokensUsed  \\\n",
       "0  0.222222  0.100629  0.205128         443   \n",
       "1  0.322581  0.219178  0.322581         300   \n",
       "2  0.325581  0.235294  0.325581         223   \n",
       "3  0.177215  0.122449  0.177215         366   \n",
       "4  0.150943  0.062112  0.132075         499   \n",
       "5  0.216216  0.175824  0.216216         315   \n",
       "6  0.160000  0.000000  0.160000         257   \n",
       "7  0.363636  0.187500  0.327273         314   \n",
       "8  0.165414  0.078212  0.150376         437   \n",
       "9  0.225000  0.056604  0.200000         337   \n",
       "\n",
       "                                            question  \\\n",
       "0  Where can applicants find information about th...   \n",
       "1  How can someone get back to the main page of I...   \n",
       "2  Who certifies all immigration instructions and...   \n",
       "3  What are the two main components of the RSE ap...   \n",
       "4  How can you transfer your visa to a new passport?   \n",
       "5  When does the selection of expressions of inte...   \n",
       "6    What will happen after you state your language?   \n",
       "7  How can visitors return to the main page of Im...   \n",
       "8  How have the employment agreements for RSE wor...   \n",
       "9  What should you do to request a refund if you ...   \n",
       "\n",
       "                                              answer  tokens  \n",
       "0  According to the information provided, applica...    4742  \n",
       "1  Based on the information provided in the conte...    4577  \n",
       "2  According to the information provided, the **M...    3508  \n",
       "3  The two main components of the RSE application...    4120  \n",
       "4  To transfer your valid New Zealand visa to a n...    5431  \n",
       "5  According to the information provided, the sel...    3929  \n",
       "6  If you call Immigration New Zealand and do not...    7803  \n",
       "7  Based on the information provided in the conte...    3860  \n",
       "8  As of October 1, 2023, the employment agreemen...    7875  \n",
       "9  According to the information provided, if you ...    5251  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_subset = random.sample(ground_truth, 10)\n",
    "scores3_token = evaluate_rag(test_subset, \"text+big\")\n",
    "scores3_token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2519392e-312e-4c7f-9746-e3f8efd391f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 10/10 [00:47<00:00,  4.74s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Relevance</th>\n",
       "      <th>Explanation</th>\n",
       "      <th>ROUGE_1</th>\n",
       "      <th>ROUGE_2</th>\n",
       "      <th>ROUGE_L</th>\n",
       "      <th>TokensUsed</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses the question by ...</td>\n",
       "      <td>0.303797</td>\n",
       "      <td>0.206186</td>\n",
       "      <td>0.303797</td>\n",
       "      <td>381</td>\n",
       "      <td>Where can applicants find information about th...</td>\n",
       "      <td>Applicants can find information about the **ac...</td>\n",
       "      <td>3504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides two clear and direct metho...</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>0.197183</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>335</td>\n",
       "      <td>How can someone get back to the main page of I...</td>\n",
       "      <td>To get back to the main page of the Immigratio...</td>\n",
       "      <td>3161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses the question by ...</td>\n",
       "      <td>0.186667</td>\n",
       "      <td>0.116505</td>\n",
       "      <td>0.186667</td>\n",
       "      <td>340</td>\n",
       "      <td>Who certifies all immigration instructions and...</td>\n",
       "      <td>According to the information provided, **the M...</td>\n",
       "      <td>2847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer clearly identifies the two main com...</td>\n",
       "      <td>0.186667</td>\n",
       "      <td>0.118812</td>\n",
       "      <td>0.186667</td>\n",
       "      <td>380</td>\n",
       "      <td>What are the two main components of the RSE ap...</td>\n",
       "      <td>The two main components of the RSE application...</td>\n",
       "      <td>3887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides detailed steps on how to t...</td>\n",
       "      <td>0.164948</td>\n",
       "      <td>0.065789</td>\n",
       "      <td>0.144330</td>\n",
       "      <td>481</td>\n",
       "      <td>How can you transfer your visa to a new passport?</td>\n",
       "      <td>To transfer your New Zealand visa to a new pas...</td>\n",
       "      <td>3970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses the question by ...</td>\n",
       "      <td>0.301887</td>\n",
       "      <td>0.266667</td>\n",
       "      <td>0.301887</td>\n",
       "      <td>294</td>\n",
       "      <td>When does the selection of expressions of inte...</td>\n",
       "      <td>According to the information provided, the sel...</td>\n",
       "      <td>3624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>PARTLY_RELEVANT</td>\n",
       "      <td>The answer provides information about the Engl...</td>\n",
       "      <td>0.068182</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.068182</td>\n",
       "      <td>342</td>\n",
       "      <td>What will happen after you state your language?</td>\n",
       "      <td>After you provide information about your langu...</td>\n",
       "      <td>3125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer directly addresses how visitors can...</td>\n",
       "      <td>0.344828</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.275862</td>\n",
       "      <td>253</td>\n",
       "      <td>How can visitors return to the main page of Im...</td>\n",
       "      <td>To return to the main page of the Immigration ...</td>\n",
       "      <td>3515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides detailed information on th...</td>\n",
       "      <td>0.148649</td>\n",
       "      <td>0.069652</td>\n",
       "      <td>0.135135</td>\n",
       "      <td>544</td>\n",
       "      <td>How have the employment agreements for RSE wor...</td>\n",
       "      <td>As of October 1, 2023, the employment agreemen...</td>\n",
       "      <td>4834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RELEVANT</td>\n",
       "      <td>The answer provides clear and detailed instruc...</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.073394</td>\n",
       "      <td>0.164706</td>\n",
       "      <td>466</td>\n",
       "      <td>What should you do to request a refund if you ...</td>\n",
       "      <td>According to the official immigration informat...</td>\n",
       "      <td>3603</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Relevance                                        Explanation  \\\n",
       "0         RELEVANT  The answer directly addresses the question by ...   \n",
       "1         RELEVANT  The answer provides two clear and direct metho...   \n",
       "2         RELEVANT  The answer directly addresses the question by ...   \n",
       "3         RELEVANT  The answer clearly identifies the two main com...   \n",
       "4         RELEVANT  The answer provides detailed steps on how to t...   \n",
       "5         RELEVANT  The answer directly addresses the question by ...   \n",
       "6  PARTLY_RELEVANT  The answer provides information about the Engl...   \n",
       "7         RELEVANT  The answer directly addresses how visitors can...   \n",
       "8         RELEVANT  The answer provides detailed information on th...   \n",
       "9         RELEVANT  The answer provides clear and detailed instruc...   \n",
       "\n",
       "    ROUGE_1   ROUGE_2   ROUGE_L  TokensUsed  \\\n",
       "0  0.303797  0.206186  0.303797         381   \n",
       "1  0.315789  0.197183  0.315789         335   \n",
       "2  0.186667  0.116505  0.186667         340   \n",
       "3  0.186667  0.118812  0.186667         380   \n",
       "4  0.164948  0.065789  0.144330         481   \n",
       "5  0.301887  0.266667  0.301887         294   \n",
       "6  0.068182  0.000000  0.068182         342   \n",
       "7  0.344828  0.181818  0.275862         253   \n",
       "8  0.148649  0.069652  0.135135         544   \n",
       "9  0.211765  0.073394  0.164706         466   \n",
       "\n",
       "                                            question  \\\n",
       "0  Where can applicants find information about th...   \n",
       "1  How can someone get back to the main page of I...   \n",
       "2  Who certifies all immigration instructions and...   \n",
       "3  What are the two main components of the RSE ap...   \n",
       "4  How can you transfer your visa to a new passport?   \n",
       "5  When does the selection of expressions of inte...   \n",
       "6    What will happen after you state your language?   \n",
       "7  How can visitors return to the main page of Im...   \n",
       "8  How have the employment agreements for RSE wor...   \n",
       "9  What should you do to request a refund if you ...   \n",
       "\n",
       "                                              answer  tokens  \n",
       "0  Applicants can find information about the **ac...    3504  \n",
       "1  To get back to the main page of the Immigratio...    3161  \n",
       "2  According to the information provided, **the M...    2847  \n",
       "3  The two main components of the RSE application...    3887  \n",
       "4  To transfer your New Zealand visa to a new pas...    3970  \n",
       "5  According to the information provided, the sel...    3624  \n",
       "6  After you provide information about your langu...    3125  \n",
       "7  To return to the main page of the Immigration ...    3515  \n",
       "8  As of October 1, 2023, the employment agreemen...    4834  \n",
       "9  According to the official immigration informat...    3603  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores6_token = evaluate_rag(test_subset, \"vector+big\")\n",
    "scores6_token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c46a6d4a-d863-4fba-9115-bac48133062e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5109.6"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores3_token.tokens.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfeaa6c1-62d9-43bb-8ac6-7f898a1f986e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3607.0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores6_token.tokens.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c8073a-00fa-49e2-8156-01ed4490d7de",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "1. \"Big\" prompt versions (text+big and vector+big) performed best overall.\n",
    "2. Text+big achieved highest relevance (100%) and ROUGE scores.\n",
    "3. Vector-based approaches showed strong performance.\n",
    "4. Structured prompts unexpectedly underperformed.\n",
    "5. Comprehensive prompts led to better relevance and content quality.\n",
    "6. Token without full articles use way more tokens for answer\n",
    "   \n",
    "The evaluation suggests that using more detailed prompts, especially with text-based retrieval, is most effective for this RAG system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4188d5d9-a786-4cd9-a665-0d69adc905ea",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}