chatBGC_benchmark

This is the repository for Benchmarking ChatBGC with different foundational models using th G1034 dataset from DTU Biosustain.

Running the Benchmark

Get access to OpenAI API

To get the OpenAI API key, you can ask @matinnuhamunada for a key or use your own API.

Then, run this on your terminal

OPENAI_API_KEY="<change this to your API Key>"

Installing ChatBGC

Assuming you are using one of our shared machines and have python installed, follow this steps:

# Clone the benchmark repository
git clone git@github.com:NBChub/chatBGC_benchmark.git
cd chatBGC_benchmark

# Copy BGCFlow run result for G1032 dataset (using one the shared machine at the NPGM DTU Biosustain)
mkdir data
cp ~/net/o-drive/CFB-S-NewBioactiveCompounds/11_G1000/bgcflow/G1034_20240208/data/processed/G1032_20240208/antismash_database/antiSMASH_database_7.1.0/antismash_db.duckdb data/antismash_db.duckdb -n

# Create python environment and install ChatBGC
python3 -m venv chatbgc_env
source chatbgc_env/bin/activate
python3 -m pip install --upgrade pip
pip install "git+https://github.com/NBChub/chatBGC.git@dev-0.2.1" #change the version accordingly

# Setup variable environment / secrets
touch .env
echo "export OPENAI_API_KEY=$OPENAI_API_KEY" > .env
source .env

Running ChatBGC Benchmark

1. GPT-4o

bash gpt-4o.sh

2. GPT-4o-mini

bash gpt-4o-mini.sh

3. llama3:70b

bash llama3_70b.sh

Running Ollama models

See: https://github.com/ollama/ollama/blob/main/docs/docker.md

tmux new-session -s ollama
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama run llama3.1:8b

Summarizing Benchmark results

To summarize the result into a single table:

python scripts/summarize.py --models gpt-4o gpt-4o-mini --llm_type openai_chat --chatbgc_version 0.2.0 --benchmark_version bde51dc

The result will be saved in the results folder.

[IMPORTANT UPDATE] - New Database Structure

We have finished the BGCFlow run for the G1034 dataset. Unfortunately, two genomes were dropped because of an issue with the annotation (inconsistent exon ordering for features):

• NBC_01310 (NBC_0131000000000_213589.current.gb)
• NBC_01080 (NBC_0108000000000_76298.current.gb).

Therefore, currently we only have result ready for the 1032 genomes (~/net/o-drive/CFB-S-NewBioactiveCompounds/11_G1000/bgcflow/G1034_20240208/data/processed/G1032_20240208).

For this benchmark, we only use the database containing antiSMASH results without BGCFlow, located in ~/net/o-drive/CFB-S-NewBioactiveCompounds/11_G1000/bgcflow/G1034_20240208/data/processed/G1032_20240208/antismash_database/antiSMASH_database_7.1.0/antismash_db.duckdb.

The new database can be found in ~/net/o-drive/CFB-S-NewBioactiveCompounds/11_G1000/bgcflow/G1034_20240208/data/processed/G1032_20240208/dbt/antiSMASH_7.1.0/dbt_bgcflow.duckdb. The new database structure now host two data schema, one for BGCFlow and another for antiSMASH. In the SQL query, this ca be fetched by explicitly mentioning the schema, for example bgcflow.cdss and antismash.cdss are two different tables. See at the bottom on how it looks like in DBeaver.

Manually exploring the database

If you want to explore the database generated manually, the easiest way is to install DBeaver