Leaderboard #3
Leaderboard #3
Conversation
There was a problem hiding this comment.
Scripts folder are used for standalone scripts, e.g. run parsing or build some plots. For backend and frontend, it's better to move to separate packages:
- Create sub-packages in
goat:parser,backend,frontend,database - Scripts only contain files to start all of them with proper arguments.
For example, in backend sub-package you define your logic with queue to evaluation and use database to store result. And in scripts you will define run_backend.py that recieve credentials of database and starts listening for events.
VyrodovMikhail
force-pushed
the
leaderboard
branch
13 times, most recently
from
c18774e
to
05ef4fc
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
05ef4fc
to
08e7182
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
5de7231
to
8a20476
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
3 times, most recently
from
4549ab1
to
7d1ac7a
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
7d1ac7a
to
61c85c7
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
c224b1b
to
477ff3b
Compare
.github/workflows/main.yaml
Outdated
| pip install -r requirements.txt -r requirements.dev.txt | ||
| - name: "black" | ||
| run: black . --check --diff --color | ||
| - name: "isort" | ||
| run: isort . --check --diff | ||
| - name: "mypy" | ||
| run: mypy | ||
| run: mypy --ignore-missing-imports |
There was a problem hiding this comment.
This already set in pyproject.toml
goat/backend/eval.py
Outdated
| results = evaluator.simple_evaluate(model=lm, tasks=[taskname]) | ||
|
|
||
| model_id = model_name.replace("/", "__") | ||
| Path(f"goat/backend/results/{model_id}").mkdir(exist_ok=True) |
There was a problem hiding this comment.
Store this path into variable, you have multiple reusage later
goat/backend/eval.py
Outdated
|
|
||
| model_id = model_name.replace("/", "__") | ||
| Path(f"goat/backend/results/{model_id}").mkdir(exist_ok=True) | ||
| lm_eval_output_file = f"goat/backend/results/{model_id + '_lm_eval'}.json" |
There was a problem hiding this comment.
Why did you create folder with model_id, but store results in model_id_lm_eval.json. Probably, this json should be inside model directory
| POSTGRES_USER = os.environ.get("POSTGRES_USER") | ||
| POSTGRES_PASSWORD = os.environ.get("POSTGRES_PASSWORD") | ||
| POSTGRES_DB = os.environ.get("POSTGRES_DB") | ||
| POSTGRES_IP = os.environ.get("POSTGRES_IP") | ||
| POSTGRES_PORT = os.environ.get("POSTGRES_PORT") |
There was a problem hiding this comment.
Probably, this should be part of DatabaseHelper class with additional check that these variables set
goat/utils/database_helper.py
Outdated
| if val == "True": | ||
| return True | ||
| else: | ||
| return False |
There was a problem hiding this comment.
| if val == "True": | |
| return True | |
| else: | |
| return False | |
| return val == "True" |
| readme = "README.md" | ||
| license = { "text" = "Apache-2.0" } | ||
| requires-python = ">=3.8" | ||
| dependencies = [ |
There was a problem hiding this comment.
Should we split dependencies by subprojects?
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
e32b355
to
dee8d13
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
dee8d13
to
f7874c2
Compare
VyrodovMikhail
force-pushed
the
leaderboard
branch
from
d022ac6
to
b5cf322
Compare
No description provided.