feat(Plugin):ChatPlugin and MetaData Module

.env.template

@@ -55,6 +55,7 @@ EMBEDDING_MODEL=text2vec
 #EMBEDDING_MODEL=bge-large-zh
 KNOWLEDGE_CHUNK_SIZE=500
 KNOWLEDGE_SEARCH_TOP_SIZE=5
+#KNOWLEDGE_CHUNK_OVERLAP=50
 # Control whether to display the source document of knowledge on the front end.
 KNOWLEDGE_CHAT_SHOW_RELATIONS=False
 ## EMBEDDING_TOKENIZER   - Tokenizer to use for chunking large inputs
@@ -83,7 +84,8 @@ LOCAL_DB_TYPE=sqlite
 # LOCAL_DB_PASSWORD=aa12345678
 # LOCAL_DB_HOST=127.0.0.1
 # LOCAL_DB_PORT=3306
-
+### This option determines the storage location of conversation records. The default is not configured to the old version of duckdb. It can be optionally db or file (if the value is db, the database configured by LOCAL_DB will be used)
+#CHAT_HISTORY_STORE_TYPE=db
 
 #*******************************************************************#
 #**                         COMMANDS                              **#
@@ -146,6 +148,28 @@ PROXY_SERVER_URL=https://api.openai.com/v1/chat/completions
 # from https://bard.google.com/     f12-> application-> __Secure-1PSID
 BARD_PROXY_API_KEY={your-bard-token}
 
+#*******************************************************************#
+# **  PROXY_SERVER +                                              **#
+#*******************************************************************#
+
+# Aliyun tongyi
+TONGYI_PROXY_API_KEY={your-tongyi-sk}
+
+## Baidu wenxin
+#WEN_XIN_MODEL_VERSION={version}
+#WEN_XIN_API_KEY={your-wenxin-sk}
+#WEN_XIN_SECRET_KEY={your-wenxin-sct}
+
+## Zhipu
+#ZHIPU_MODEL_VERSION={version}
+#ZHIPU_PROXY_API_KEY={your-zhipu-sk}
+
+## Baichuan
+#BAICHUN_MODEL_NAME={version}
+#BAICHUAN_PROXY_API_KEY={your-baichuan-sk}
+#BAICHUAN_PROXY_API_SECRET={your-baichuan-sct}
+
+
 #*******************************************************************#
 #**    SUMMARY_CONFIG                                             **#
 #*******************************************************************#

.gitignore

@@ -152,5 +152,7 @@ pilot/mock_datas/db-gpt-test.db.wal
 logswebserver.log.*
 .history/*
 .plugin_env
+/pilot/meta_data/alembic/versions/*
+/pilot/meta_data/*.db
 # Ignore for now
 thirdparty

README.md

@@ -58,6 +58,11 @@ Run on an RTX 4090 GPU.
 
 ![demo_en](https://github.com/eosphoros-ai/DB-GPT/assets/17919400/d40118e4-8e76-45b6-b4a6-30e5ff170f42)
 
+
+![7f1bd042-7165-4b9f-a88e-ccce35f9d9aa](https://github.com/eosphoros-ai/DB-GPT/assets/13723926/0f6e0944-24e5-4481-87f5-7c168a63c5ea)
+
+
+
 #### Chat with data, and figure charts.
 
 ![db plugins demonstration](https://github.com/eosphoros-ai/DB-GPT/assets/13723926/4113ac15-83c2-4350-86c0-5fc795677abd)
@@ -84,40 +89,98 @@ Currently, we have released multiple key features, which are listed below to dem
 - ChatDashboard
 - Multi-Agents&Plugins
 - Unified vector storage/indexing of knowledge base
-  - Support for unstructured data such as PDF, TXT, Markdown, CSV, DOC, PPT, and WebURL
+  - Support for unstructured data 
+    - PDF
+    - TXT
+    - Markdown
+    - CSV
+    - DOC
+    - PPT
+    - WebURL
 - Multi LLMs Support, Supports multiple large language models, currently supporting
-  - 🔥 InternLM(7b,20b)
-  - 🔥 Baichuan2(7b,13b)
-  - 🔥 Vicuna-v1.5(7b,13b)
-  - 🔥 llama-2(7b,13b,70b)
-  - WizardLM-v1.2(13b)
-  - Vicuna (7b,13b)
-  - ChatGLM-6b (int4,int8)
-  - ChatGLM2-6b (int4,int8)
-  - guanaco(7b,13b,33b)
-  - Gorilla(7b,13b)
-  - baichuan(7b,13b)
+  - [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
+  - [baichuan2-7b/baichuan2-13b](https://huggingface.co/baichuan-inc)
+  - [internlm/internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b)
+  - [Qwen/Qwen-7B-Chat/Qwen-14B-Chat](https://huggingface.co/Qwen/)
+  - [Vicuna](https://huggingface.co/Tribbiani/vicuna-13b)
+  - [BlinkDL/RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven)
+  - [camel-ai/CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data)
+  - [databricks/dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b)
+  - [FreedomIntelligence/phoenix-inst-chat-7b](https://huggingface.co/FreedomIntelligence/phoenix-inst-chat-7b)
+  - [h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b)
+  - [lcw99/polyglot-ko-12.8b-chang-instruct-chat](https://huggingface.co/lcw99/polyglot-ko-12.8b-chang-instruct-chat)
+  - [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5)
+  - [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat)
+  - [Neutralzz/BiLLa-7B-SFT](https://huggingface.co/Neutralzz/BiLLa-7B-SFT)
+  - [nomic-ai/gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy)
+  - [NousResearch/Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b)
+  - [openaccess-ai-collective/manticore-13b-chat-pyg](https://huggingface.co/openaccess-ai-collective/manticore-13b-chat-pyg)
+  - [OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5)
+  - [project-baize/baize-v2-7b](https://huggingface.co/project-baize/baize-v2-7b)
+  - [Salesforce/codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b)
+  - [StabilityAI/stablelm-tuned-alpha-7b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b)
+  - [THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)
+  - [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)
+  - [tiiuae/falcon-40b](https://huggingface.co/tiiuae/falcon-40b)
+  - [timdettmers/guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged)
+  - [togethercomputer/RedPajama-INCITE-7B-Chat](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat)
+  - [WizardLM/WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0)
+  - [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0)
+  - [baichuan-inc/baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B)
+  - [HuggingFaceH4/starchat-beta](https://huggingface.co/HuggingFaceH4/starchat-beta)
+  - [FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat)
+  - [BAAI/AquilaChat-7B](https://huggingface.co/BAAI/AquilaChat-7B)
+  - [all models of OpenOrca](https://huggingface.co/Open-Orca)
+  - [Spicyboros](https://huggingface.co/jondurbin/spicyboros-7b-2.2?not-for-all-audiences=true) + [airoboros 2.2](https://huggingface.co/jondurbin/airoboros-l2-13b-2.2)
+  - [VMware's OpenLLaMa OpenInstruct](https://huggingface.co/VMware/open-llama-7b-open-instruct)
+
+- Support API Proxy LLMs
+  - [x] [ChatGPT](https://api.openai.com/)
+  - [x] [Tongyi](https://www.aliyun.com/product/dashscope)
+  - [x] [Wenxin](https://cloud.baidu.com/product/wenxinworkshop?track=dingbutonglan)
+  - [x] [ChatGLM](http://open.bigmodel.cn/)
+
+- Support Datasources
+
+| DataSource                                                                      | support     | Notes                                       |
+| ------------------------------------------------------------------------------  | ----------- | ------------------------------------------- |
+| [MySQL](https://www.mysql.com/)                                                 | Yes         |                                             |
+| [PostgresSQL](https://www.postgresql.org/)                                      | Yes         |                                             |
+| [Spark](https://github.com/apache/spark)                                        | Yes         |                                             |
+| [DuckDB](https://github.com/duckdb/duckdb)                                      | Yes         |                                             |
+| [Sqlite](https://github.com/sqlite/sqlite)                                      | Yes         |                                             |
+| [MSSQL](https://github.com/microsoft/mssql-jdbc)                                | Yes         |                                             |
+| [ClickHouse](https://github.com/ClickHouse/ClickHouse)                          | Yes         |                                             |
+| [Oracle](https://github.com/oracle)                                             | No          |           TODO                              |
+| [Redis](https://github.com/redis/redis)                                         | No          |           TODO                              |
+| [MongoDB](https://github.com/mongodb/mongo)                                     | No          |           TODO                              |
+| [HBase](https://github.com/apache/hbase)                                        | No          |           TODO                              |
+| [Doris](https://github.com/apache/doris)                                        | No          |           TODO                              |
+| [DB2](https://github.com/IBM/Db2)                                               | No          |           TODO                              |
+| [Couchbase](https://github.com/couchbase)                                       | No          |           TODO                              |
+| [Elasticsearch](https://github.com/elastic/elasticsearch)                       | No          |           TODO                              |
+| [OceanBase](https://github.com/OceanBase)                                       | No          |           TODO                              |
+| [TiDB](https://github.com/pingcap/tidb)                                         | No          |           TODO                              |
+| [StarRocks](https://github.com/StarRocks/starrocks)                             | No          |           TODO                              |
 
 ## Introduction 
-DB-GPT creates a vast model operating system using [FastChat](https://github.com/lm-sys/FastChat) and offers a large language model powered by vicuna. In addition, we provide private domain knowledge base question-answering capability. Furthermore, we also provide support for additional plugins, and our design natively supports the Auto-GPT plugin.Our vision is to make it easier and more convenient to build  applications around databases and llm.
-
 Is the architecture of the entire DB-GPT shown in the following figure:
 
 <p align="center">
   <img src="./assets/DB-GPT.png" width="800" />
 </p>
 
 The core capabilities mainly consist of the following parts:
-1. Knowledge base capability: Supports private domain knowledge base question-answering capability.
-2. Large-scale model management capability: Provides a large model operating environment based on FastChat.
-3. Unified data vector storage and indexing: Provides a uniform way to store and index various data types.
-4. Connection module: Used to connect different modules and data sources to achieve data flow and interaction.
-5. Agent and plugins: Provides Agent and plugin mechanisms, allowing users to customize and enhance the system's behavior.
-6. Prompt generation and optimization: Automatically generates high-quality prompts and optimizes them to improve system response efficiency.
-7. Multi-platform product interface: Supports various client products, such as web, mobile applications, and desktop applications.
+1. Multi-Models: Support multi-LLMs, such as LLaMA/LLaMA2、CodeLLaMA、ChatGLM, QWen、Vicuna and proxy model ChatGPT、Baichuan、tongyi、wenxin etc
+2. Knowledge Based QA: You can perform high-quality intelligent Q&A based on local documents such as pdf, word, excel and other data.
+3. Embedding: Unified data vector storage and indexing, Embed data as vectors and store them in vector databases, providing content similarity search.
+4. Multi-Datasources: Used to connect different modules and data sources to achieve data flow and interaction. 
+5. Multi-Agents: Provides Agent and plugin mechanisms, allowing users to customize and enhance the system's behavior.
+6. Privacy & Secure: You can be assured that there is no risk of data leakage, and your data is 100% private and secure.
+7. Text2SQL: We enhance the Text-to-SQL performance by applying Supervised Fine-Tuning (SFT) on large language models
 
 ### SubModule
-- [DB-GPT-Hub](https://github.com/eosphoros-ai/DB-GPT-Hub) Text-to-SQL parsing with LLMs
+- [DB-GPT-Hub](https://github.com/eosphoros-ai/DB-GPT-Hub) Text-to-SQL performance by applying Supervised Fine-Tuning (SFT) on large language models.
 - [DB-GPT-Plugins](https://github.com/eosphoros-ai/DB-GPT-Plugins) DB-GPT Plugins, Can run autogpt plugin directly
 - [DB-GPT-Web](https://github.com/eosphoros-ai/DB-GPT-Web)  ChatUI for DB-GPT  
 
@@ -130,7 +193,7 @@ The core capabilities mainly consist of the following parts:
 ![macOS](https://img.shields.io/badge/mac%20os-000000?style=for-the-badge&logo=macos&logoColor=F0F0F0)
 ![Windows](https://img.shields.io/badge/Windows-0078D6?style=for-the-badge&logo=windows&logoColor=white)
 
-[**Quickstart**](https://db-gpt.readthedocs.io/en/latest/getting_started/install/deploy/deploy.html)
+[**Installation && Usage Tutorial**](https://db-gpt.readthedocs.io/en/latest/getting_started/install/deploy/deploy.html)
 
 ### Language Switching
     In the .env configuration file, modify the LANGUAGE parameter to switch to different languages. The default is English (Chinese: zh, English: en, other languages to be added later).
@@ -145,6 +208,110 @@ The core capabilities mainly consist of the following parts:
   <img src="./assets/roadmap.jpg" width="800px" />
 </p>
 
+### KBQA RAG optimization
+- [x] Multi Documents
+  - [x] PDF
+  - [x] Excel, csv
+  - [x] Word
+  - [x] Text
+  - [x] MarkDown
+  - [ ] Code
+  - [ ] Images 
+
+- [x] RAG
+- [ ] Graph Database
+  - [ ] Neo4j Graph
+  - [ ] Nebula Graph
+- [x] Multi Vector Database
+  - [x] Chroma
+  - [x] Milvus
+  - [x] Weaviate
+  - [x] PGVector
+  - [ ] Elasticsearch
+  - [ ] ClickHouse
+  - [ ] Faiss 
+
+- [ ] Testing and Evaluation Capability Building
+  - [ ] Knowledge QA datasets
+  - [ ] Question collection [easy, medium, hard]:
+  - [ ] Scoring mechanism
+  - [ ] Testing and evaluation using Excel + DB datasets
+
+### Multi Datasource Support
+
+- Multi Datasource Support 
+  - [x] MySQL
+  - [x] PostgresSQL
+  - [x] Spark
+  - [x] DuckDB
+  - [x] Sqlite
+  - [x] MSSQL
+  - [x] ClickHouse
+  - [ ] Oracle
+  - [ ] Redis
+  - [ ] MongoDB
+  - [ ] HBase
+  - [ ] Doris
+  - [ ] DB2
+  - [ ] Couchbase
+  - [ ] Elasticsearch
+  - [ ] OceanBase
+  - [ ] TiDB
+  - [ ] StarRocks
+
+### Multi-Models And vLLM
+- [x] [Cluster Deployment](https://db-gpt.readthedocs.io/en/latest/getting_started/install/cluster/vms/index.html)
+- [x] [Fastchat Support](https://github.com/lm-sys/FastChat)
+- [x] [vLLM Support](https://db-gpt.readthedocs.io/en/latest/getting_started/install/llm/vllm/vllm.html)
+- [ ] Cloud-native environment and support for Ray environment
+- [ ] Service Registry(eg:nacos)
+- [ ] Compatibility with OpenAI's interfaces
+- [ ] Expansion and optimization of embedding models
+
+### Agents market and Plugins
+- [x] multi-agents framework
+- [x] custom plugin development 
+- [ ] plugin market
+- [ ] Integration with CoT
+- [ ] Enrich plugin sample library
+- [ ] Support for AutoGPT protocol
+- [ ] Integration of multi-agents and visualization capabilities, defining LLM+Vis new standards
+
+### Cost and Observability
+- [x] [debugging](https://db-gpt.readthedocs.io/en/latest/getting_started/observability.html)
+- [ ] Observability
+- [ ] cost & budgets
+
+### Text2SQL Finetune
+- support llms
+  - [x] LLaMA
+  - [x] LLaMA-2
+  - [x] BLOOM
+  - [x] BLOOMZ
+  - [x] Falcon
+  - [x] Baichuan
+  - [x] Baichuan2
+  - [x] InternLM
+  - [x] Qwen
+  - [x] XVERSE
+  - [x] ChatGLM2
+
+-  SFT Accuracy
+
+As of October 10, 2023, by fine-tuning an open-source model of 13 billion parameters using this project, the execution accuracy on the Spider evaluation dataset has surpassed that of GPT-4!
+
+| name                              | Execution Accuracy | reference                                                                                                                      |
+| ----------------------------------| ------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
+| **GPT-4**                         | **0.762**          | [numbersstation-eval-res](https://www.numbersstation.ai/post/nsql-llama-2-7b)                                                  |
+| ChatGPT                           | 0.728              | [numbersstation-eval-res](https://www.numbersstation.ai/post/nsql-llama-2-7b)                                                  | 
+| **CodeLlama-13b-Instruct-hf_lora**| **0.789**          | sft train by our this project,only used spider train dataset ,the same eval  way in this project  with lora SFT                |
+| CodeLlama-13b-Instruct-hf_qlora   | 0.774              | sft train by our this project,only used spider train dataset ,the same eval  way in this project  with qlora and nf4,bit4 SFT  |
+| wizardcoder                       | 0.610              | [text-to-sql-wizardcoder](https://github.com/cuplv/text-to-sql-wizardcoder/tree/main)                                          |  
+| CodeLlama-13b-Instruct-hf         | 0.556              | eval in this project default param                                                                                             |
+| llama2_13b_hf_lora_best           | 0.744              | sft train by our this project,only used spider train dataset ,the same eval  way in this project                               |
+
+[More Information about Text2SQL finetune](https://github.com/eosphoros-ai/DB-GPT-Hub)
+
 ## Licence
 
 The MIT License (MIT)