find_similar_texts_with_sentence_transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyO5kfNJFYh9i1OLReKXW5kB",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/howard-haowen/NLP-demos/blob/main/find_similar_texts_with_sentence_transformers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Download the dataset"
      ],
      "metadata": {
        "id": "wpZYqIqWhVDQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget http://od.cdc.gov.tw/pr/CDC_chatbox.csv -O faq.csv"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fu6Wik_dgpZb",
        "outputId": "299bd6e1-b30f-4339-bdeb-83496d819fcb"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-02-09 14:21:12--  http://od.cdc.gov.tw/pr/CDC_chatbox.csv\n",
            "Resolving od.cdc.gov.tw (od.cdc.gov.tw)... 35.229.205.172\n",
            "Connecting to od.cdc.gov.tw (od.cdc.gov.tw)|35.229.205.172|:80... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://od.cdc.gov.tw/pr/CDC_chatbox.csv [following]\n",
            "--2023-02-09 14:21:14--  https://od.cdc.gov.tw/pr/CDC_chatbox.csv\n",
            "Connecting to od.cdc.gov.tw (od.cdc.gov.tw)|35.229.205.172|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 600208 (586K) [text/csv]\n",
            "Saving to: ‘faq.csv’\n",
            "\n",
            "faq.csv             100%[===================>] 586.14K   729KB/s    in 0.8s    \n",
            "\n",
            "2023-02-09 14:21:15 (729 KB/s) - ‘faq.csv’ saved [600208/600208]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load the dataset as dataframe"
      ],
      "metadata": {
        "id": "sfmqleobhYX8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "SuYFpAyARb_T"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('faq.csv')\n",
        "df.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y6aProAnRiIN",
        "outputId": "d6ab8920-807e-449d-c9d1-820052185a5b"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2332, 7)"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "TEXT_COL = 'Question'\n",
        "COLS_TO_SHOW = [TEXT_COL, 'Disease', 'Answer1', 'Answer2']\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "EaEtTY0eRnbE",
        "outputId": "dc8ab3c5-f44b-4a7e-c2f2-6f81234d003c"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        No Disease        Question  \\\n",
              "0        1    A型肝炎        什麼是A型肝炎？   \n",
              "1        2    A型肝炎     A型肝炎會有什麼症狀？   \n",
              "2        3    A型肝炎     罹患A型肝炎的嚴重性？   \n",
              "3        4    A型肝炎      A型肝炎有什麼併發症   \n",
              "4        5    A型肝炎    A型肝炎有哪些高風險族群   \n",
              "...    ...     ...             ...   \n",
              "2327  2330     愛滋病  孕婦好像感染了愛滋病怎麼辦？   \n",
              "2328  2331      淋病      孕婦要如何預防淋病？   \n",
              "2329  2332      淋病   孕婦好像感染了淋病怎麼辦？   \n",
              "2330  2333      梅毒      孕婦要如何預防梅毒？   \n",
              "2331  2334      梅毒   孕婦好像感染了梅毒怎麼辦？   \n",
              "\n",
              "                                                Answer1  \\\n",
              "0                              A型肝炎是由A型肝炎病毒感染所造成的急性肝臟發炎   \n",
              "1     這問題非常好！感染A型肝炎後，可能會出現發燒、全身倦怠不適、食慾不振、嘔吐及腹部不舒服症狀，...   \n",
              "2     A型肝炎的致死率低（約千分之三），造成死亡的情形多半為猛爆性肝炎，通常發生於老年患者或有慢性...   \n",
              "3                 A型肝炎感染很嚴重時，可能造成急性肝衰竭，僅有少數病例會因猛爆性肝炎而死亡   \n",
              "4        像比如說前往A型肝炎流行地區（例如非洲、南美洲、中國大陸、東南亞及南亞地區等）旅遊或工作的人   \n",
              "...                                                 ...   \n",
              "2327            應儘速至愛滋病指定醫院接受篩檢，若確診感染應接受藥物治療，降低傳染給寶寶的風險   \n",
              "2328                      避免不安全性行為及其他感染風險行為等，才能有效預防感染淋病   \n",
              "2329            勿自行至藥局買藥或誤信偏方，應儘速就醫並告知醫師妊娠狀態，以利醫師評估治療方式   \n",
              "2330                  避免不安全性行為，並配合孕婦產檢時程進行梅毒篩檢，才能有效避免感染   \n",
              "2331            勿自行至藥局買藥或誤信偏方，應儘速就醫並告知醫師妊娠狀態，以利醫師評估治療方式   \n",
              "\n",
              "                                                Answer2  \\\n",
              "0     A型肝炎主要為經口（糞口）感染，簡單來說就是食用遭A型肝炎病毒汙染的食物或水而感染，所以前往...   \n",
              "1     症狀的嚴重程度通常隨著年齡增加而增加。大部分6歲以下的小朋友感染後沒有出現症狀或症狀輕微，而...   \n",
              "2                                                   NaN   \n",
              "3                                                   NaN   \n",
              "4                     還有特殊職業如廚師及餐飲食品從業人員、醫療照護者、嬰幼兒保育工作者   \n",
              "...                                                 ...   \n",
              "2327                                                NaN   \n",
              "2328                                                NaN   \n",
              "2329                                                NaN   \n",
              "2330                                                NaN   \n",
              "2331                                                NaN   \n",
              "\n",
              "                                         Answer3 Answer4  \n",
              "0                                            NaN     NaN  \n",
              "1                                            NaN     NaN  \n",
              "2                                            NaN     NaN  \n",
              "3                                            NaN     NaN  \n",
              "4     那患有慢性肝病、血友病、曾經移植肝臟的病人、靜脈藥癮者、男男間性行為者也要特別注意！     NaN  \n",
              "...                                          ...     ...  \n",
              "2327                                         NaN     NaN  \n",
              "2328                                         NaN     NaN  \n",
              "2329                                         NaN     NaN  \n",
              "2330                                         NaN     NaN  \n",
              "2331                                         NaN     NaN  \n",
              "\n",
              "[2332 rows x 7 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7a9e471c-80d8-49f7-881a-e2c382ef7daa\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>No</th>\n",
              "      <th>Disease</th>\n",
              "      <th>Question</th>\n",
              "      <th>Answer1</th>\n",
              "      <th>Answer2</th>\n",
              "      <th>Answer3</th>\n",
              "      <th>Answer4</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>A型肝炎</td>\n",
              "      <td>什麼是A型肝炎？</td>\n",
              "      <td>A型肝炎是由A型肝炎病毒感染所造成的急性肝臟發炎</td>\n",
              "      <td>A型肝炎主要為經口（糞口）感染，簡單來說就是食用遭A型肝炎病毒汙染的食物或水而感染，所以前往...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>A型肝炎</td>\n",
              "      <td>A型肝炎會有什麼症狀？</td>\n",
              "      <td>這問題非常好！感染A型肝炎後，可能會出現發燒、全身倦怠不適、食慾不振、嘔吐及腹部不舒服症狀，...</td>\n",
              "      <td>症狀的嚴重程度通常隨著年齡增加而增加。大部分6歲以下的小朋友感染後沒有出現症狀或症狀輕微，而...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>A型肝炎</td>\n",
              "      <td>罹患A型肝炎的嚴重性？</td>\n",
              "      <td>A型肝炎的致死率低（約千分之三），造成死亡的情形多半為猛爆性肝炎，通常發生於老年患者或有慢性...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>A型肝炎</td>\n",
              "      <td>A型肝炎有什麼併發症</td>\n",
              "      <td>A型肝炎感染很嚴重時，可能造成急性肝衰竭，僅有少數病例會因猛爆性肝炎而死亡</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>A型肝炎</td>\n",
              "      <td>A型肝炎有哪些高風險族群</td>\n",
              "      <td>像比如說前往A型肝炎流行地區（例如非洲、南美洲、中國大陸、東南亞及南亞地區等）旅遊或工作的人</td>\n",
              "      <td>還有特殊職業如廚師及餐飲食品從業人員、醫療照護者、嬰幼兒保育工作者</td>\n",
              "      <td>那患有慢性肝病、血友病、曾經移植肝臟的病人、靜脈藥癮者、男男間性行為者也要特別注意！</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2327</th>\n",
              "      <td>2330</td>\n",
              "      <td>愛滋病</td>\n",
              "      <td>孕婦好像感染了愛滋病怎麼辦？</td>\n",
              "      <td>應儘速至愛滋病指定醫院接受篩檢，若確診感染應接受藥物治療，降低傳染給寶寶的風險</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2328</th>\n",
              "      <td>2331</td>\n",
              "      <td>淋病</td>\n",
              "      <td>孕婦要如何預防淋病？</td>\n",
              "      <td>避免不安全性行為及其他感染風險行為等，才能有效預防感染淋病</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2329</th>\n",
              "      <td>2332</td>\n",
              "      <td>淋病</td>\n",
              "      <td>孕婦好像感染了淋病怎麼辦？</td>\n",
              "      <td>勿自行至藥局買藥或誤信偏方，應儘速就醫並告知醫師妊娠狀態，以利醫師評估治療方式</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2330</th>\n",
              "      <td>2333</td>\n",
              "      <td>梅毒</td>\n",
              "      <td>孕婦要如何預防梅毒？</td>\n",
              "      <td>避免不安全性行為，並配合孕婦產檢時程進行梅毒篩檢，才能有效避免感染</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2331</th>\n",
              "      <td>2334</td>\n",
              "      <td>梅毒</td>\n",
              "      <td>孕婦好像感染了梅毒怎麼辦？</td>\n",
              "      <td>勿自行至藥局買藥或誤信偏方，應儘速就醫並告知醫師妊娠狀態，以利醫師評估治療方式</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2332 rows × 7 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7a9e471c-80d8-49f7-881a-e2c382ef7daa')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7a9e471c-80d8-49f7-881a-e2c382ef7daa button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7a9e471c-80d8-49f7-881a-e2c382ef7daa');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load a pretrained model"
      ],
      "metadata": {
        "id": "mUshErlfhfw5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q -U sentence-transformers"
      ],
      "metadata": {
        "id": "phkH8YshIlKQ"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sentence_transformers import SentenceTransformer"
      ],
      "metadata": {
        "id": "ihFWMuALh6EB"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "embed_model = SentenceTransformer(\"shibing624/text2vec-base-chinese\")\n",
        "sentences = ['如何加入會員', '入會辦法是什麼']\n",
        "sentence_embeddings = embed_model.encode(sentences)\n",
        "\n",
        "print(\"Sentence embeddings:\")\n",
        "print(sentence_embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iLitrXq1IpAv",
        "outputId": "aab113a0-15bf-407c-9a5a-b81082e18b34"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/shibing624_text2vec-base-chinese. Creating a new one with MEAN pooling.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence embeddings:\n",
            "[[ 0.59205085  0.27223465  0.08369207 ... -0.31279552  0.13715449\n",
            "   0.35445943]\n",
            " [ 0.00216622 -0.1309313   0.16524468 ... -0.24950682 -0.56418824\n",
            "   0.5346472 ]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from numpy import dot\n",
        "from numpy.linalg import norm\n",
        "\n",
        "a = sentence_embeddings[0]\n",
        "b = sentence_embeddings[1]\n",
        "cos_sim = dot(a, b)/(norm(a)*norm(b))\n",
        "print(cos_sim)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9cyOBPqDiaAH",
        "outputId": "addee80e-06eb-47a4-e578-f4e3f7763d23"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.60111433\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "texts = df[TEXT_COL]\n",
        "texts[3]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "Lpd48JtII_3v",
        "outputId": "33f3d9e4-932e-48cc-8eb3-82a92619fa0f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'A型肝炎有什麼併發症'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "embeddings = embed_model.encode(texts)"
      ],
      "metadata": {
        "id": "uuIiCJM1Iqpb"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ssJ1k6b4R6O7",
        "outputId": "17f0b785-16a6-4469-df33-3120f5d14648"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2332"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(embeddings[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OQkympmuI-Ts",
        "outputId": "dfbddc39-e9a8-49db-c657-c19e381a7d1e"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "768"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Embed the corpus and build an embedding index"
      ],
      "metadata": {
        "id": "w_puis0dizep"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q faiss-cpu"
      ],
      "metadata": {
        "id": "bxR8bc44Jgz-"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import faiss\n",
        "import numpy as np\n",
        "\n",
        "def create_index_embeddings(embed_arrays: np.array,index_arrays: np.array):\n",
        "\n",
        "\t# Step 1: Change data type\n",
        "\tembeddings = embed_arrays.astype(\"float32\")\n",
        "\t\n",
        "\t# Step 2: Instantiate the index using a type of distance, which is L2 here\n",
        "\tindex = faiss.IndexFlatIP(embeddings.shape[1])\n",
        "\t\n",
        "\t# Step 3: Pass the index to IndexIDMap\n",
        "\tindex = faiss.IndexIDMap(index)\n",
        "\t\n",
        "\t# Step 4: Add vectors and their IDs\n",
        "\tindex.add_with_ids(embeddings, index_arrays)\n",
        "\t\n",
        "\treturn index, embeddings"
      ],
      "metadata": {
        "id": "otV1jd-oJoW7"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "type(embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7Lz_mtz6J4oM",
        "outputId": "395fdec9-c092-4694-eed7-4057677b6e9f"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "numpy.ndarray"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc_ids = df.index.to_numpy()\n",
        "type(doc_ids)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nSK8Cnb8Kpjg",
        "outputId": "76574e25-057d-47d4-89b8-dadead093f51"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "numpy.ndarray"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fs_index, fs_embeddings = create_index_embeddings(embeddings, doc_ids)"
      ],
      "metadata": {
        "id": "ECnVEdH2Js4g"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Search similar texts by a user query"
      ],
      "metadata": {
        "id": "ITUfNB4wjEN-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def search_by_user_query(query: str, #User query text\n",
        "                         embed_model=embed_model, #USE embed model\n",
        "                         index=fs_index, #Faiss index\n",
        "                         df=df, #Corpus in dataframe\n",
        "                         topK=10): #TopK results\n",
        "                    \n",
        "\t\t# embed the query with USE\n",
        "\t\t# Note: Make the query a list to keep it consistent with the format from above\n",
        "\t\tembeddings = embed_model.encode([query])\n",
        "\t\t\n",
        "\t\t# covert the embeddings to conform to the Faiss format\n",
        "\t\tembeddings = np.array(embeddings).astype(\"float32\")\n",
        "\t\t\n",
        "\t\t# get the distances and indexes\n",
        "\t\t# Note: The index is 0 because there's only one document in the embeddings\n",
        "\t\tD, I = index.search(np.array([embeddings[0]]), k=topK)\n",
        "\t\t\n",
        "\t\t# results\n",
        "\t\tresults_df = df.loc[I.flatten(), COLS_TO_SHOW]\n",
        "\n",
        "\t\treturn results_df\n"
      ],
      "metadata": {
        "id": "Q7ADvsR5K8R-"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"破傷風有哪些症狀\"\n",
        "search_by_user_query(query)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 858
        },
        "id": "WKNa1O91L8Oe",
        "outputId": "fa2546a1-d654-4193-93a0-59fb8909e496"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "           Question Disease  \\\n",
              "643      破傷風會有什麼症狀？     破傷風   \n",
              "645      破傷風有什麼危險徵兆     破傷風   \n",
              "642         什麼是破傷風？     破傷風   \n",
              "671       破傷風要怎麼治療？     破傷風   \n",
              "646       破傷風有什麼併發症     破傷風   \n",
              "647     破傷風有哪些高風險族群     破傷風   \n",
              "2042    破傷風用藥有哪些種類？     破傷風   \n",
              "651      破傷風流行地區有哪些     破傷風   \n",
              "2035    罹患破傷風要注意什麼？     破傷風   \n",
              "690   新生兒破傷風會有什麼症狀？  新生兒破傷風   \n",
              "\n",
              "                                                Answer1  \\\n",
              "643   最常見之初症狀為腹部僵硬及肌肉痙攣，典型的破傷風痙攣現象為「角弓反張」(opisthoton...   \n",
              "645                                       無法張開嘴、吞嚥及呼吸困難   \n",
              "642   破傷風由破傷風桿菌之外毒素（exotoxin）所引起，其特徵為痛性之肌肉收縮（最初在咬肌及頸...   \n",
              "671   1.肌肉注射破傷風免疫球蛋白(TIG)，並取少量局部注射於傷口周圍。\\n 2.抗生素治療；抗...   \n",
              "646                          喉痙攣、骨折、肺栓塞、吸入性肺炎、呼吸困難而導致死亡   \n",
              "647   沒有接種過疫苗或距離最後一次破傷風疫苗接種超過 10 ?者。建議在工作中接觸土壤、污物、動物...   \n",
              "2042  1.肌肉注射破傷風免疫球蛋白（TIG），並取少量局部注射於傷口周圍。\\n2.口服（或靜注）m...   \n",
              "651   破傷風病例通常會發生在農業區或低度開發地區，因為該等地區較易與動物之排泄物接觸，或預防接種情...   \n",
              "2035  請儘速至附近醫療院所就醫，接受肌肉注射破傷風免疫球蛋白(TIG)，抗生素需持續治療10～14...   \n",
              "690   典型特徵是嬰兒出生幾天(3至28天，通常6天)後，吸吮動作和哭泣情形由正常漸漸轉變為吸奶困難...   \n",
              "\n",
              "                                                Answer2  \n",
              "643   1.疼痛性肌肉收縮開始為下顎肌與頸部肌，其次為軀幹肌\\n 2.開口障礙，吞嚥困難，四肢僵硬強...  \n",
              "645                                                 NaN  \n",
              "642                                                 NaN  \n",
              "671   4.支持性療法最重要，包括維持患者呼吸道之暢通，暗室中照顧，必要時可以肌肉鬆弛劑保持患者之鎮...  \n",
              "646                                                 NaN  \n",
              "647                                                 NaN  \n",
              "2042                                                NaN  \n",
              "651                                                 NaN  \n",
              "2035                                                NaN  \n",
              "690                                                 NaN  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-524882b5-b85d-4f26-b969-65b38fd9c63c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Question</th>\n",
              "      <th>Disease</th>\n",
              "      <th>Answer1</th>\n",
              "      <th>Answer2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>643</th>\n",
              "      <td>破傷風會有什麼症狀？</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>最常見之初症狀為腹部僵硬及肌肉痙攣，典型的破傷風痙攣現象為「角弓反張」(opisthoton...</td>\n",
              "      <td>1.疼痛性肌肉收縮開始為下顎肌與頸部肌，其次為軀幹肌\\n 2.開口障礙，吞嚥困難，四肢僵硬強...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>645</th>\n",
              "      <td>破傷風有什麼危險徵兆</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>無法張開嘴、吞嚥及呼吸困難</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>642</th>\n",
              "      <td>什麼是破傷風？</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>破傷風由破傷風桿菌之外毒素（exotoxin）所引起，其特徵為痛性之肌肉收縮（最初在咬肌及頸...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>671</th>\n",
              "      <td>破傷風要怎麼治療？</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>1.肌肉注射破傷風免疫球蛋白(TIG)，並取少量局部注射於傷口周圍。\\n 2.抗生素治療；抗...</td>\n",
              "      <td>4.支持性療法最重要，包括維持患者呼吸道之暢通，暗室中照顧，必要時可以肌肉鬆弛劑保持患者之鎮...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>646</th>\n",
              "      <td>破傷風有什麼併發症</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>喉痙攣、骨折、肺栓塞、吸入性肺炎、呼吸困難而導致死亡</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>647</th>\n",
              "      <td>破傷風有哪些高風險族群</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>沒有接種過疫苗或距離最後一次破傷風疫苗接種超過 10 ?者。建議在工作中接觸土壤、污物、動物...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2042</th>\n",
              "      <td>破傷風用藥有哪些種類？</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>1.肌肉注射破傷風免疫球蛋白（TIG），並取少量局部注射於傷口周圍。\\n2.口服（或靜注）m...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>651</th>\n",
              "      <td>破傷風流行地區有哪些</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>破傷風病例通常會發生在農業區或低度開發地區，因為該等地區較易與動物之排泄物接觸，或預防接種情...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2035</th>\n",
              "      <td>罹患破傷風要注意什麼？</td>\n",
              "      <td>破傷風</td>\n",
              "      <td>請儘速至附近醫療院所就醫，接受肌肉注射破傷風免疫球蛋白(TIG)，抗生素需持續治療10～14...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>690</th>\n",
              "      <td>新生兒破傷風會有什麼症狀？</td>\n",
              "      <td>新生兒破傷風</td>\n",
              "      <td>典型特徵是嬰兒出生幾天(3至28天，通常6天)後，吸吮動作和哭泣情形由正常漸漸轉變為吸奶困難...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-524882b5-b85d-4f26-b969-65b38fd9c63c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-524882b5-b85d-4f26-b969-65b38fd9c63c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-524882b5-b85d-4f26-b969-65b38fd9c63c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    }
  ]
}