Store Embeddings in ChromaDB using Langchain.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4553cee2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "import os\n",
    "from langchain_community.embeddings.sentence_transformer import SentenceTransformerEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain.vectorstores.chroma import Chroma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "65504887",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "········\n"
     ]
    }
   ],
   "source": [
    "HUGGINGFACEHUB_API_TOKEN = getpass()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2e7b0f9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['HUGGINGFACEHUB_API_TOKEN'] = HUGGINGFACEHUB_API_TOKEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "04dda609",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.document_loaders.text.TextLoader at 0x1bf2fb4a0d0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = TextLoader(file_path='AI_revolution.txt')\n",
    "loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4e19c838",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='The recent revolution of AI has sparked a paradigm shift across industries, reshaping the way we perceive and interact with technology. Advancements in machine learning algorithms, coupled with vast amounts of data and powerful computing resources, have propelled AI to new heights of capability and applicability. From autonomous vehicles and virtual assistants to personalized healthcare and predictive analytics, AI has permeated nearly every aspect of modern life, revolutionizing how we work, live, and communicate.\\n\\nOne of the most notable aspects of the AI revolution is its impact on automation and efficiency. AI-powered systems are streamlining processes, optimizing workflows, and augmenting human capabilities in ways previously thought impossible. In manufacturing, robots equipped with AI algorithms are revolutionizing production lines, increasing productivity, and reducing errors. In customer service, chatbots and virtual agents are providing instant assistance, improving response times, and enhancing customer satisfaction. The result is a more agile, responsive, and competitive business landscape driven by intelligent automation.\\n\\nMoreover, the AI revolution is fueling innovation and creativity across diverse domains. In healthcare, AI algorithms are revolutionizing diagnostics, drug discovery, and personalized treatment plans. AI-powered imaging techniques can detect diseases at earlier stages with higher accuracy, while predictive analytics models can forecast patient outcomes and recommend tailored interventions. Similarly, in finance, AI algorithms are revolutionizing risk management, fraud detection, and investment strategies. By analyzing vast amounts of financial data in real-time, AI systems can identify trends, predict market fluctuations, and optimize investment portfolios with unparalleled precision.\\n\\nAnother significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.\\n\\nDespite these challenges, the AI revolution holds immense promise for the future, offering unprecedented opportunities for innovation, growth, and societal advancement. By harnessing the power of AI to address complex challenges, improve decision-making, and enhance human capabilities, we can unlock new frontiers of possibility and create a more sustainable, inclusive, and prosperous world for generations to come. As we navigate this transformative era, it is essential to approach the development and deployment of AI technologies with a commitment to ethical principles, transparency, and accountability, ensuring that the benefits of the AI revolution are shared equitably and responsibly.\\n\\n\\n\\n\\n\\n', metadata={'source': 'AI_revolution.txt'})]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "080d8393",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_text_splitters.character.CharacterTextSplitter at 0x1bf3eb4c910>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_splitter = CharacterTextSplitter(chunk_size=100,chunk_overlap=0)\n",
    "text_splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "36fe11c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 520, which is longer than the specified 100\n",
      "Created a chunk of size 630, which is longer than the specified 100\n",
      "Created a chunk of size 693, which is longer than the specified 100\n",
      "Created a chunk of size 831, which is longer than the specified 100\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Document(page_content='The recent revolution of AI has sparked a paradigm shift across industries, reshaping the way we perceive and interact with technology. Advancements in machine learning algorithms, coupled with vast amounts of data and powerful computing resources, have propelled AI to new heights of capability and applicability. From autonomous vehicles and virtual assistants to personalized healthcare and predictive analytics, AI has permeated nearly every aspect of modern life, revolutionizing how we work, live, and communicate.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='One of the most notable aspects of the AI revolution is its impact on automation and efficiency. AI-powered systems are streamlining processes, optimizing workflows, and augmenting human capabilities in ways previously thought impossible. In manufacturing, robots equipped with AI algorithms are revolutionizing production lines, increasing productivity, and reducing errors. In customer service, chatbots and virtual agents are providing instant assistance, improving response times, and enhancing customer satisfaction. The result is a more agile, responsive, and competitive business landscape driven by intelligent automation.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='Moreover, the AI revolution is fueling innovation and creativity across diverse domains. In healthcare, AI algorithms are revolutionizing diagnostics, drug discovery, and personalized treatment plans. AI-powered imaging techniques can detect diseases at earlier stages with higher accuracy, while predictive analytics models can forecast patient outcomes and recommend tailored interventions. Similarly, in finance, AI algorithms are revolutionizing risk management, fraud detection, and investment strategies. By analyzing vast amounts of financial data in real-time, AI systems can identify trends, predict market fluctuations, and optimize investment portfolios with unparalleled precision.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='Another significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='Despite these challenges, the AI revolution holds immense promise for the future, offering unprecedented opportunities for innovation, growth, and societal advancement. By harnessing the power of AI to address complex challenges, improve decision-making, and enhance human capabilities, we can unlock new frontiers of possibility and create a more sustainable, inclusive, and prosperous world for generations to come. As we navigate this transformative era, it is essential to approach the development and deployment of AI technologies with a commitment to ethical principles, transparency, and accountability, ensuring that the benefits of the AI revolution are shared equitably and responsibly.', metadata={'source': 'AI_revolution.txt'})]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = text_splitter.split_documents(data)\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cbfa63a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "HuggingFaceEmbeddings(client=SentenceTransformer(\n",
       "  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel \n",
       "  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})\n",
       "  (2): Normalize()\n",
       "), model_name='all-MiniLM-L6-v2', cache_folder=None, model_kwargs={}, encode_kwargs={}, multi_process=False, show_progress=False)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings = SentenceTransformerEmbeddings(model_name='all-MiniLM-L6-v2')\n",
    "embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a067f915",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.vectorstores.chroma.Chroma at 0x1bf5e4f62d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = Chroma.from_documents(docs,embeddings)\n",
    "db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2426698d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'What are the applications of AI in the society?'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"What are the applications of AI in the society?\"\n",
    "query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1bb50ec4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Another significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='The recent revolution of AI has sparked a paradigm shift across industries, reshaping the way we perceive and interact with technology. Advancements in machine learning algorithms, coupled with vast amounts of data and powerful computing resources, have propelled AI to new heights of capability and applicability. From autonomous vehicles and virtual assistants to personalized healthcare and predictive analytics, AI has permeated nearly every aspect of modern life, revolutionizing how we work, live, and communicate.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='Despite these challenges, the AI revolution holds immense promise for the future, offering unprecedented opportunities for innovation, growth, and societal advancement. By harnessing the power of AI to address complex challenges, improve decision-making, and enhance human capabilities, we can unlock new frontiers of possibility and create a more sustainable, inclusive, and prosperous world for generations to come. As we navigate this transformative era, it is essential to approach the development and deployment of AI technologies with a commitment to ethical principles, transparency, and accountability, ensuring that the benefits of the AI revolution are shared equitably and responsibly.', metadata={'source': 'AI_revolution.txt'}),\n",
       " Document(page_content='One of the most notable aspects of the AI revolution is its impact on automation and efficiency. AI-powered systems are streamlining processes, optimizing workflows, and augmenting human capabilities in ways previously thought impossible. In manufacturing, robots equipped with AI algorithms are revolutionizing production lines, increasing productivity, and reducing errors. In customer service, chatbots and virtual agents are providing instant assistance, improving response times, and enhancing customer satisfaction. The result is a more agile, responsive, and competitive business landscape driven by intelligent automation.', metadata={'source': 'AI_revolution.txt'})]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = db.similarity_search(query)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d581ae83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(Document(page_content='Another significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.', metadata={'source': 'AI_revolution.txt'}),\n",
       "  0.9241192936897278)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_with_similarity_score = db.similarity_search_with_score(query,k=1)\n",
    "res_with_similarity_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "52810113",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(Document(page_content='Another significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.', metadata={'source': 'AI_revolution.txt'}),\n",
       "  0.34654898080667085),\n",
       " (Document(page_content='The recent revolution of AI has sparked a paradigm shift across industries, reshaping the way we perceive and interact with technology. Advancements in machine learning algorithms, coupled with vast amounts of data and powerful computing resources, have propelled AI to new heights of capability and applicability. From autonomous vehicles and virtual assistants to personalized healthcare and predictive analytics, AI has permeated nearly every aspect of modern life, revolutionizing how we work, live, and communicate.', metadata={'source': 'AI_revolution.txt'}),\n",
       "  0.29565952721546507)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_with_similarity_score = db.similarity_search_with_relevance_scores(query,k=2)\n",
    "res_with_similarity_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c647ba2b",
   "metadata": {},
   "source": [
    "## Storing the embeddings of text file in ChromaDB into local disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c620576f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.vectorstores.chroma.Chroma at 0x1bf5eaea190>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db2 = Chroma.from_documents(docs,embedding=embeddings,persist_directory=\"./chromadb\")\n",
    "db2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e29f9677",
   "metadata": {},
   "source": [
    "## Extracting the embeddings of the text from ChromaDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2f39ea31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.vectorstores.chroma.Chroma at 0x1bf5c863ed0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db3 = Chroma(persist_directory=\"./chromadb\",embedding_function=embeddings)\n",
    "db3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c76ed1b",
   "metadata": {},
   "source": [
    "## Using the ChromaDB embeddings to perform similarity search to determine the similarity score of docs with the user query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f41cafa3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(Document(page_content='Another significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.', metadata={'source': 'AI_revolution.txt'}),\n",
       "  0.9241192359160637)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = db3.similarity_search_with_score(query,k=1)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "215b64c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Another significant aspect of the AI revolution is its societal impact, raising important ethical, legal, and socio-economic questions. As AI becomes increasingly autonomous and pervasive, concerns about privacy, security, bias, and accountability are coming to the forefront. Questions about the ethical use of AI, algorithmic transparency, and data privacy are prompting discussions and debates among policymakers, industry leaders, and ethicists. Moreover, the rise of AI-powered automation is reshaping labor markets, leading to concerns about job displacement, income inequality, and workforce re-skilling. Addressing these challenges requires a collaborative effort involving government regulation, industry standards, and public engagement to ensure that AI technologies are developed and deployed responsibly and ethically.\n",
      "0.9241192359160637\n"
     ]
    }
   ],
   "source": [
    "similar_text_corpus, similarity_score = result[0]\n",
    "print(similar_text_corpus.page_content)\n",
    "print(similarity_score)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}