notebooks/2.retrieval_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fb8290ed-5623-44bf-8f6f-d64a0ef9cf1e",
   "metadata": {},
   "source": [
    "# Retrieaval evaluation\n",
    "In this notebook we evaluate different retrieval methods.\n",
    "For perform it we need:\n",
    "* create ground truth dataset, genereate question for our answers\n",
    "* get search_results for our questions\n",
    "* measure performanse of search methods\n",
    "\n",
    "We will try:\n",
    "* minisearch.py\n",
    "* elasticsearch (text search)\n",
    "* elasticsearch (vector search)\n",
    "\n",
    "And separately evaluate search with chunking (split by 1000 symbols)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80d7c691-ae64-4640-8cc3-8e217c1e96ed",
   "metadata": {},
   "source": [
    "To generate ground-truth dataset we use [this script](scripts/generate_ground_truth.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5431446-25ba-402b-a0c1-744b50fdbd23",
   "metadata": {},
   "source": [
    "## Praparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "41580d89-432c-4fea-aca4-47176247e06e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-09-08 17:50:44--  https://raw.githubusercontent.com/alexeygrigorev/minsearch/main/minsearch.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3832 (3.7K) [text/plain]\n",
      "Saving to: ‘minsearch.py’\n",
      "\n",
      "minsearch.py        100%[===================>]   3.74K  --.-KB/s    in 0.002s  \n",
      "\n",
      "2024-09-08 17:50:44 (1.47 MB/s) - ‘minsearch.py’ saved [3832/3832]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f minsearch.py\n",
    "!wget https://raw.githubusercontent.com/alexeygrigorev/minsearch/main/minsearch.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0298b7a2-70fb-4291-88fe-85ed9db35261",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/roman/.local/share/virtualenvs/llm-zoomcamp-project2-jok7aU-Z/lib/python3.12/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:11: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm, trange\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import math\n",
    "from tqdm import tqdm\n",
    "\n",
    "from elasticsearch import Elasticsearch\n",
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "import minsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1c69b55e-56a2-4806-956c-3fce7ab0faa2",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/ground-truth.json', 'r') as f_in:\n",
    "    ground_truth = json.load(f_in)\n",
    "\n",
    "with open('../data/site_content.json', 'r') as f_in:\n",
    "    raw_doc = json.load(f_in)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b9506f-dae7-4be8-a6d3-247cd76c6aee",
   "metadata": {},
   "source": [
    "## Metric functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "44b84d30-44d3-414f-aa5b-fafb5a11e649",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hit_rate(relevance_total):\n",
    "    cnt = 0\n",
    "\n",
    "    for line in relevance_total:\n",
    "        if True in line:\n",
    "            cnt = cnt + 1\n",
    "\n",
    "    return cnt / len(relevance_total)\n",
    "\n",
    "def mrr(relevance_total):\n",
    "    total_score = 0.0\n",
    "\n",
    "    for line in relevance_total:\n",
    "        for rank in range(len(line)):\n",
    "            if line[rank] == True:\n",
    "                total_score = total_score + 1 / (rank + 1)\n",
    "\n",
    "    return total_score / len(relevance_total)\n",
    "def ndcg(relevance_total):\n",
    "    def dcg(relevance):\n",
    "        return sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(relevance))\n",
    "    \n",
    "    def idcg(relevance):\n",
    "        return dcg(sorted(relevance, reverse=True))\n",
    "    \n",
    "    scores = []\n",
    "    for relevance in relevance_total:\n",
    "        if sum(relevance) == 0:\n",
    "            scores.append(0.0)\n",
    "        else:\n",
    "            scores.append(dcg(relevance) / idcg(relevance))\n",
    "    \n",
    "    return sum(scores) / len(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b670161a-6926-4dc1-bf68-140064f4eaa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(ground_truth, search_function):\n",
    "    relevance_total = []\n",
    "\n",
    "    for q in tqdm(ground_truth):\n",
    "        results = search_function(q['question'])\n",
    "        relevance = [d['url'] == q['url'] for d in results]\n",
    "        relevance_total.append(relevance)\n",
    "\n",
    "    return {\n",
    "        'hit_rate': hit_rate(relevance_total),\n",
    "        'mrr': mrr(relevance_total),\n",
    "        'ndsg': ndcg(relevance_total)\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37c57c3-3c3b-466c-8b6d-2c590fc084f8",
   "metadata": {},
   "source": [
    "## Minsearch function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5beefef6-11e7-4870-ae7b-ca0bd2e6ef3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [{'url': k, 'header':v['header'], 'main_content':v['main_content']} for k,v in raw_doc.items()]\n",
    "\n",
    "msearch_index = minsearch.Index(\n",
    "    text_fields=[\"url\", \"header\", \"main_content\"],\n",
    "    keyword_fields=[]\n",
    ")\n",
    "\n",
    "msearch_index.fit(data)\n",
    "\n",
    "def search_msearch(query: str) -> str:\n",
    "    results = msearch_index.search(\n",
    "        query=query,\n",
    "        num_results=5\n",
    "    )\n",
    "\n",
    "    return results\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "52ac1088-136b-4367-a62a-d0500f926885",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 415/415 [00:01<00:00, 238.19it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.4578313253012048,\n",
       " 'mrr': 0.324859437751004,\n",
       " 'ndsg': 0.3580757390552208}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, search_msearch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aa45c7d-5ff6-4acb-9836-c1aea9d37bbd",
   "metadata": {},
   "source": [
    "#### With boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9653d32e-c4a5-4d81-b56b-32f29a589e7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [{'url': k, 'header':v['header'], 'main_content':v['main_content']} for k,v in raw_doc.items()]\n",
    "\n",
    "msearch_index = minsearch.Index(\n",
    "    text_fields=[\"url\", \"header\", \"main_content\"],\n",
    "    keyword_fields=[]\n",
    ")\n",
    "\n",
    "msearch_index.fit(data)\n",
    "\n",
    "def search_msearch_boost(query: str) -> str:\n",
    "    results = msearch_index.search(\n",
    "        query=query,\n",
    "        num_results=5,\n",
    "        boost_dict = {'header':0.5}\n",
    "    )\n",
    "\n",
    "    return results\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "712f0b73-5445-4fbf-b886-4d0ec25738a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 415/415 [00:01<00:00, 223.00it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.4746987951807229,\n",
       " 'mrr': 0.330722891566265,\n",
       " 'ndsg': 0.366675213616622}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, search_msearch_boost)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fc11a4b-f71e-4182-9095-ee1120db225c",
   "metadata": {},
   "source": [
    "## Elastic search (text)\n",
    "\n",
    "first we need to start Elasticsearch locally, if it's not started yet\n",
    "\n",
    "```bash\n",
    "docker run -it \\\n",
    "    --rm \\\n",
    "    --name elasticsearch \\\n",
    "    -p 9200:9200 \\\n",
    "    -p 9300:9300 \\\n",
    "    -e \"discovery.type=single-node\" \\\n",
    "    -e \"xpack.security.enabled=false\" \\\n",
    "    docker.elastic.co/elasticsearch/elasticsearch:8.4.3\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0e8d6286-b841-420e-a410-cf9470d9673d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchtext'})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es_client = Elasticsearch('http://localhost:9200') \n",
    "\n",
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"}\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name = \"esearchtext\"\n",
    "\n",
    "es_client.indices.delete(index=index_name, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "47d48ccc-b1bf-490c-82b9-fb2997de35f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 1030/1030 [00:04<00:00, 209.21it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data):\n",
    "    es_client.index(index=index_name, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "73faaf10-35c9-4094-bb31-33baa2228c31",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search(query):\n",
    "    search_query = {\n",
    "        \"size\": 5,\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"must\": {\n",
    "                    \"multi_match\": {\n",
    "                        \"query\": query,\n",
    "                        \"fields\": [\"url\", \"header\", \"main_content\"],\n",
    "                        \"type\": \"best_fields\"\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "\n",
    "    response = es_client.search(index=index_name, body=search_query)\n",
    "    \n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in response['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "    \n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "85a7881c-50e1-45ed-873f-485735a97151",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 415/415 [00:02<00:00, 204.65it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.6120481927710844,\n",
       " 'mrr': 0.43746987951807226,\n",
       " 'ndsg': 0.48136427158055817}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c943221f-4846-4dea-8e85-d7d3a856cc2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_boost(query):\n",
    "    search_query = {\n",
    "        \"size\": 5,\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"must\": {\n",
    "                    \"multi_match\": {\n",
    "                        \"query\": query,\n",
    "                        \"fields\": [\"url\", \"header\", \"main_content^3\"],\n",
    "                        \"type\": \"best_fields\"\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "\n",
    "    response = es_client.search(index=index_name, body=search_query)\n",
    "    \n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in response['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "    \n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7308f59a-5ff2-44ae-a2b3-2d79ba68a54b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 415/415 [00:02<00:00, 203.21it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.619277108433735,\n",
       " 'mrr': 0.45497991967871476,\n",
       " 'ndsg': 0.49613324092305466}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_boost)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b6aea1c-a09d-491b-87cf-4fc6f450334c",
   "metadata": {},
   "source": [
    "## Elastic search (vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c2c1cb5f-85ba-44c7-a616-b45759fb2983",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/roman/.local/share/virtualenvs/llm-zoomcamp-project2-jok7aU-Z/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchvector'})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"paraphrase-MiniLM-L6-v2\"\n",
    "model = SentenceTransformer(model_name)\n",
    "\n",
    "\n",
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"},\n",
    "            \"header_vector\": {\n",
    "                \"type\": \"dense_vector\",\n",
    "                \"dims\": model.get_sentence_embedding_dimension(),\n",
    "                \"index\": True,\n",
    "                \"similarity\": \"cosine\"\n",
    "            },\n",
    "            \"main_content_vector\": {\n",
    "                \"type\": \"dense_vector\",\n",
    "                \"dims\": model.get_sentence_embedding_dimension(),\n",
    "                \"index\": True,\n",
    "                \"similarity\": \"cosine\"\n",
    "            },\n",
    "            \n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name_vector = \"esearchvector\"\n",
    "\n",
    "es_client.indices.delete(index=index_name_vector, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name_vector, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dd1a55f9-303b-44b0-bbb5-cad13a8d5359",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 1030/1030 [00:36<00:00, 28.16it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data):\n",
    "    doc['header_vector'] = model.encode(doc['header'])\n",
    "    doc['main_content_vector'] = model.encode(doc['main_content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "23d3e960-5f4f-48fa-8ecc-c6920af7973e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 1030/1030 [00:05<00:00, 199.27it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data):\n",
    "    es_client.index(index=index_name_vector, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3ed4ed5d-e374-4377-97af-3b7cb1681971",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_vector(field, vector):\n",
    "    knn = {\n",
    "        \"field\": field,\n",
    "        \"query_vector\": vector,\n",
    "        \"k\": 5,\n",
    "        \"num_candidates\": 10000,\n",
    "    }\n",
    "\n",
    "    search_query = {\n",
    "        \"knn\": knn,\n",
    "        \"_source\": [\"url\", \"header\", \"main_content\", \"header_vector\", \"main_content_vector\"]\n",
    "    }\n",
    "\n",
    "    es_results = es_client.search(\n",
    "        index=index_name_vector,\n",
    "        body=search_query\n",
    "    )\n",
    "    \n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in es_results['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "\n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9ceeaadc-96b5-4e47-9fb9-d1943f37639d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_vector_header(q):\n",
    "    return elastic_search_vector('header_vector', model.encode(q))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0da77e8a-d7ef-40e3-a7b5-e738cf58f783",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:08<00:00, 46.88it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.27228915662650605,\n",
       " 'mrr': 0.16602409638554227,\n",
       " 'ndsg': 0.1922402885934874}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_vector_header)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6e7c8891-bc8e-4c3d-b318-09a2c48e5f88",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_vector_main_content(q):\n",
    "    return elastic_search_vector('main_content_vector', model.encode(q))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "75d097f7-fb8b-4582-8f71-27b36781a355",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:08<00:00, 48.03it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.4506024096385542,\n",
       " 'mrr': 0.33329317269076303,\n",
       " 'ndsg': 0.3625595595195731}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_vector_main_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6466913f-19e6-47bb-949f-4c45d3908085",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_vector_3(q):\n",
    "    query_vector = model.encode(q).tolist()\n",
    "\n",
    "    search_query = {\n",
    "        \"size\": 5,\n",
    "        \"query\": {\n",
    "            \"script_score\": {\n",
    "                \"query\": {\"match_all\": {}},\n",
    "                \"script\": {\n",
    "                    \"source\": \"cosineSimilarity(params.query_vector, 'main_content_vector') + 1.0\",\n",
    "                    \"params\": {\"query_vector\": query_vector}\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "\n",
    "    results = es_client.search(index=index_name_vector, body=search_query)\n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in results['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "\n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "779872f7-f650-44e6-9b7e-a940e99e1767",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:06<00:00, 60.50it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.4578313253012048,\n",
       " 'mrr': 0.34052208835341363,\n",
       " 'ndsg': 0.3697884751822237}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_vector_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0eea9de-22a8-477f-9012-ae4b309857ea",
   "metadata": {},
   "source": [
    "## Minsearch with chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "3d16ae07-4502-4e96-9f93-b0470658db69",
   "metadata": {},
   "outputs": [],
   "source": [
    "def chunk_data(raw_doc, chunk_size=1000, overlap=100):\n",
    "    def chunk_content(content, chunk_size=1000, overlap=100):\n",
    "        chunks = []\n",
    "        start = 0\n",
    "        while start < len(content):\n",
    "            end = start + chunk_size\n",
    "            chunk = content[start:end]\n",
    "            chunks.append(chunk)\n",
    "            start = end - overlap\n",
    "        return chunks\n",
    "\n",
    "    chunked_data = []\n",
    "    for k, v in raw_doc.items():\n",
    "        content_chunks = chunk_content(v['main_content'], chunk_size, overlap)\n",
    "        for i, chunk in enumerate(content_chunks):\n",
    "            chunked_data.append({\n",
    "                'url': k,\n",
    "                'header': v['header'],\n",
    "                'main_content': chunk,\n",
    "                'chunk_index': i\n",
    "            })\n",
    "    \n",
    "    return chunked_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1d32b8f4-1da0-4cfb-8b94-58e050185630",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_chunk = chunk_data(raw_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e1d63ea3-b9fb-4073-9a30-8ca86aed9a96",
   "metadata": {},
   "outputs": [],
   "source": [
    "msearch_index_chunk = minsearch.Index(\n",
    "    text_fields=[\"url\", \"header\", \"main_content\"],\n",
    "    keyword_fields=[]\n",
    ")\n",
    "\n",
    "msearch_index_chunk.fit(data_chunk)\n",
    "\n",
    "def search_msearch_boost_chunk(query: str) -> str:\n",
    "    results = msearch_index_chunk.search(\n",
    "        query=query,\n",
    "        num_results=5,\n",
    "        boost_dict = {'header':0.5}\n",
    "    )\n",
    "\n",
    "    return results\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b51de73b-b460-4452-88b5-4e22ff6b3bcd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 415/415 [00:01<00:00, 207.61it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.3927710843373494,\n",
       " 'mrr': 0.4765461847389559,\n",
       " 'ndsg': 0.3221260677391088}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, search_msearch_boost_chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d21bd04a-5529-42c6-b6bb-c5a64da42c83",
   "metadata": {},
   "source": [
    "## Elastic text with chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "06a14eb0-94dc-4ce5-b3cc-fa44478e8f86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchtext_chunks'})"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"}\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name = \"esearchtext_chunks\"\n",
    "\n",
    "es_client.indices.delete(index=index_name, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4ff411f4-1e6d-4fb8-8912-481e1a7a2b28",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 2834/2834 [00:10<00:00, 257.66it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    es_client.index(index=index_name, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "fc4b8ebe-7a02-416f-8340-5ec1a413f480",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_boost_chunks(query):\n",
    "    search_query = {\n",
    "        \"size\": 5,\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"must\": {\n",
    "                    \"multi_match\": {\n",
    "                        \"query\": query,\n",
    "                        \"fields\": [\"url\", \"header\", \"main_content^3\"],\n",
    "                        \"type\": \"best_fields\"\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "\n",
    "    response = es_client.search(index=index_name, body=search_query)\n",
    "    \n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in response['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "    \n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "78f45acd-08b1-4f30-9e01-2f603fbb8bc9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 415/415 [00:02<00:00, 170.92it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.6240963855421687,\n",
       " 'mrr': 0.6240562248995986,\n",
       " 'ndsg': 0.5197822047592691}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_boost_chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87520b26-0172-4d7a-a749-71b6565b66fc",
   "metadata": {},
   "source": [
    "## Elastic vector chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6362c474-7e59-4578-b365-f8009a85b732",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchvector_chunks'})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"},\n",
    "            \"main_content_vector\": {\n",
    "                \"type\": \"dense_vector\",\n",
    "                \"dims\": model.get_sentence_embedding_dimension(),\n",
    "                \"index\": True,\n",
    "                \"similarity\": \"cosine\"\n",
    "            },\n",
    "            \n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name_vector = \"esearchvector_chunks\"\n",
    "\n",
    "es_client.indices.delete(index=index_name_vector, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name_vector, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c1aa2c89-42c9-449a-b35b-fa74ae916689",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 2834/2834 [02:04<00:00, 22.74it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    doc['header_vector'] = model.encode(doc['header'])\n",
    "    doc['main_content_vector'] = model.encode(doc['main_content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "7509a6cc-9350-45fc-8ebf-9e168d629ac0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 2834/2834 [00:15<00:00, 185.60it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    es_client.index(index=index_name_vector, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "48499789-1638-4727-a5c6-7e0e5c2f157d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_vector(field, vector):\n",
    "    knn = {\n",
    "        \"field\": field,\n",
    "        \"query_vector\": vector,\n",
    "        \"k\": 5,\n",
    "        \"num_candidates\": 10000,\n",
    "    }\n",
    "\n",
    "    search_query = {\n",
    "        \"knn\": knn,\n",
    "        \"_source\": [\"url\", \"header\", \"main_content\", \"header_vector\", \"main_content_vector\"]\n",
    "    }\n",
    "\n",
    "    es_results = es_client.search(\n",
    "        index=index_name_vector,\n",
    "        body=search_query\n",
    "    )\n",
    "    \n",
    "    result_docs = []\n",
    "    \n",
    "    for hit in es_results['hits']['hits']:\n",
    "        result_docs.append(hit['_source'])\n",
    "\n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8d746772-28ed-4b89-81e0-b0499a443acf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_vector_main_content_chunks(q):\n",
    "    return elastic_search_vector('main_content_vector', model.encode(q))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e92fb5b4-65f1-4321-9ff0-1b50664cd3a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:11<00:00, 35.23it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.5060240963855421,\n",
       " 'mrr': 0.46598393574297164,\n",
       " 'ndsg': 0.3986455909302925}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_vector_main_content_chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f671b0ae-83c5-46cd-8e23-b9c519cf4ca6",
   "metadata": {},
   "source": [
    "### Another models Sentense transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ccd26c33-5d0d-4ea8-b917-2b2d694834df",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"all-MiniLM-L12-v2\"\n",
    "model = SentenceTransformer(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b1fe81c9-d0b0-439e-bb5b-70dd0d5b0291",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'esearchvector_chunks_2'})"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_settings = {\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": 1,\n",
    "        \"number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"url\": {\"type\": \"text\"},\n",
    "            \"header\": {\"type\": \"text\"},\n",
    "            \"main_content\": {\"type\": \"text\"},\n",
    "            \"main_content_vector\": {\n",
    "                \"type\": \"dense_vector\",\n",
    "                \"dims\": model.get_sentence_embedding_dimension(),\n",
    "                \"index\": True,\n",
    "                \"similarity\": \"cosine\"\n",
    "            },\n",
    "            \n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "index_name_vector = \"esearchvector_chunks_2\"\n",
    "\n",
    "es_client.indices.delete(index=index_name_vector, ignore_unavailable=True)\n",
    "es_client.indices.create(index=index_name_vector, body=index_settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "0c09f03e-ebb9-429a-94a9-5587b820640a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 2834/2834 [03:58<00:00, 11.89it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    doc['header_vector'] = model.encode(doc['header'])\n",
    "    doc['main_content_vector'] = model.encode(doc['main_content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "80d007fd-4893-42f4-97ed-5104bd02c54f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 2834/2834 [00:17<00:00, 164.99it/s]\n"
     ]
    }
   ],
   "source": [
    "for doc in tqdm(data_chunk):\n",
    "    es_client.index(index=index_name_vector, document=doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "7f569665-e372-4259-87f9-05b13d4e50e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:16<00:00, 24.75it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.5493975903614458,\n",
       " 'mrr': 0.49333333333333323,\n",
       " 'ndsg': 0.4261590511854974}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_vector_main_content_chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d4c7b277-7f79-4ea7-bd40-54ec405259fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_combined(query):\n",
    "    vector = model.encode(query)\n",
    "    search_query = {\n",
    "        \"_source\": [\"url\", \"header\", \"main_content\", \"header_vector\", \"main_content_vector\"],\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"should\": [\n",
    "                    {\n",
    "                        \"multi_match\": {\n",
    "                            \"query\": query,\n",
    "                            \"fields\": [\"header\", \"main_content\"],\n",
    "                            \"type\": \"best_fields\",\n",
    "                            \"tie_breaker\": 0.3\n",
    "                        }\n",
    "                    },\n",
    "                    {\n",
    "                        \"script_score\": {\n",
    "                            \"query\": {\"match_all\": {}},\n",
    "                            \"script\": {\n",
    "                                \"source\": \"cosineSimilarity(params.query_vector, 'main_content_vector') + 1.0\",\n",
    "                                \"params\": {\"query_vector\": vector}\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        },\n",
    "        \"size\": 5\n",
    "    }\n",
    "    \n",
    "    es_results = es_client.search(\n",
    "        index=index_name_vector,\n",
    "        body=search_query\n",
    "    )\n",
    "    \n",
    "    result_docs = [hit['_source'] for hit in es_results['hits']['hits']]\n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a65bda41-3d7a-46b5-8844-ac2f328f7cbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:12<00:00, 34.06it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.5783132530120482,\n",
       " 'mrr': 0.6417670682730925,\n",
       " 'ndsg': 0.47983191351995125}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_combined)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "7c0a6994-932e-44ee-9cc1-ed2a892a7b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_search_combined_10(query):\n",
    "    vector = model.encode(query)\n",
    "    search_query = {\n",
    "        \"_source\": [\"url\", \"header\", \"main_content\", \"header_vector\", \"main_content_vector\"],\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"should\": [\n",
    "                    {\n",
    "                        \"multi_match\": {\n",
    "                            \"query\": query,\n",
    "                            \"fields\": [\"header\", \"main_content\"],\n",
    "                            \"type\": \"best_fields\",\n",
    "                            \"tie_breaker\": 0.3\n",
    "                        }\n",
    "                    },\n",
    "                    {\n",
    "                        \"script_score\": {\n",
    "                            \"query\": {\"match_all\": {}},\n",
    "                            \"script\": {\n",
    "                                \"source\": \"cosineSimilarity(params.query_vector, 'main_content_vector') + 1.0\",\n",
    "                                \"params\": {\"query_vector\": vector}\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        },\n",
    "        \"size\": 10\n",
    "    }\n",
    "    \n",
    "    es_results = es_client.search(\n",
    "        index=index_name_vector,\n",
    "        body=search_query\n",
    "    )\n",
    "    \n",
    "    result_docs = [hit['_source'] for hit in es_results['hits']['hits']]\n",
    "    return result_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "de15b063-4788-4be7-a8f9-161e1219012f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████| 415/415 [00:14<00:00, 28.58it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'hit_rate': 0.6578313253012048,\n",
       " 'mrr': 0.7077184930196975,\n",
       " 'ndsg': 0.49965424064494396}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(ground_truth, elastic_search_combined_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0a77032-8d16-44e5-a670-486e0d69788d",
   "metadata": {},
   "source": [
    "### Final results\n",
    "\n",
    "Results of various search methods, including Minsearch and Elasticsearch with different configurations. The metrics used for evaluation include Hit Rate, MRR (Mean Reciprocal Rank), and NDCG (Normalized Discounted Cumulative Gain).\n",
    "\n",
    "| Method | Hit Rate | MRR | NDCG |\n",
    "|--------|----------|-----|------|\n",
    "| Minsearch | 0.4578 | 0.3249 | 0.3581 |\n",
    "| Minsearch (with boosting) | 0.4747 | 0.3307 | 0.3667 |\n",
    "| Elasticsearch (text search) | 0.6120 | 0.4375 | 0.4814 |\n",
    "| Elasticsearch (text search with boosting) | 0.6193 | 0.4550 | 0.4961 |\n",
    "| Elasticsearch (vector search on header) | 0.2723 | 0.1660 | 0.1922 |\n",
    "| Elasticsearch (vector search on main content) | 0.4506 | 0.3333 | 0.3626 |\n",
    "| Elasticsearch (vector search with cosine similarity) | 0.4578 | 0.3405 | 0.3698 |\n",
    "| Minsearch (with chunks) | 0.3928 | 0.4765 | 0.3221 |\n",
    "| Elasticsearch (text search with chunks) | 0.6241 | 0.6241 | 0.5198 |\n",
    "| Elasticsearch (vector search with chunks, paraphrase-MiniLM-L6-v2) | 0.5060 | 0.4660 | 0.3986 |\n",
    "| Elasticsearch (vector search with chunks, all-MiniLM-L12-v2) | 0.5494 | 0.4933 | 0.4262 |\n",
    "| Elasticsearch (combined search) | 0.5783 | 0.6418 | 0.4798 |\n",
    "| Elasticsearch (combined search, size 10) | 0.6578 | 0.7077 | 0.4997 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef91884-5496-40aa-9b81-e6a367fee81e",
   "metadata": {},
   "source": [
    "## Conclusion:\n",
    "The evaluation of various search methods reveals several important insights:\n",
    "\n",
    "- Elasticsearch consistently outperforms Minsearch across all metrics, with text-based search methods generally showing better results than vector-based methods alone.\n",
    "- Content chunking significantly improves performance, especially for Elasticsearch, as evidenced by the strong results of Elasticsearch text search with chunks.\n",
    "- While vector search methods show mixed results, they improve with chunking and offer complementary strengths to text-based search.\n",
    "- The combined search approach in Elasticsearch, which leverages both text and vector search capabilities, shows promising results with a strong balance across all metrics (Hit Rate: 0.5783, MRR: 0.6418, NDCG: 0.4798).\n",
    "\n",
    "For this project, we have decided to implement the combined search method using Elasticsearch. This approach offers several advantages:\n",
    "\n",
    "- It provides robust performance across different types of queries and content structures.\n",
    "- It balances the strengths of both text-based and vector-based search methods.\n",
    "\n",
    "By choosing this combined method, we aim to create a versatile and effective search solution that can handle a wide range of search scenarios while maintaining high relevance and accuracy in results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c094e8eb-0906-4de6-809a-b9413d0288a0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}