-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added base scoring program #14
Conversation
The location for data is described in the competition.yaml file. I think we're meant to have the ingestion program pull in the I'll read through what you have here on Monday. Glad you included a test case! |
I still have to update the second scoring program |
* add bioclip code_submission * add ingestion program and bioclip model submission * add model environment and change prediction.txt file * remove defaults in ingestion.py * Update baselines/BioCLIP_code_submission/metadata Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update baselines/BioCLIP_code_submission/model.py Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update baselines/BioCLIP_code_submission/model.py Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update baselines/BioCLIP_code_submission/model.py Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update baselines/BioCLIP_code_submission/model.py Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update baselines/BioCLIP_code_submission/requirements.txt Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update ingestion_program/ingestion.py Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Update baselines/BioCLIP_code_submission/model.py Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com> * Apply suggestions from code review deal with device variable --------- Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>
I updated the sample code submission for the DINOv2 baseline. Some notes:
|
* divide and rename scoring programs by task make helper functions accessible to scoring programs * Update metadata files to point at proper scoring function, input, and output * Update mimic scoring program get proper input files to read, not just directories print all scores and output challenge scores to scores.json for CodaBench to read Add requirements file for temporary fix * Update species A scoring program get proper input files to read, not just directories print all scores and output challenge scores to scores.json for CodaBench to read scores still need proper labels and requirements file is temporary fix pending base container * Add 'ref' to solution filenames to match competition.yaml in formatting branch
When we are evaluating in scoring_program_A, will all the entries be either major and minor subspecies and no others? |
@work4cs, this is working as expected now (just without the container). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We will want to remove the requirements installation from both scoring programs once we have the container sorted, but this is functioning as-is on CodaBench with codalab/competitions-v2-compute-worker.
formatting
@work4cs and @DavidCarlyn I think we're good at this point? We'll change the scoring programs to not require the requirements file once we get the container functioning. |
Addresses #5
I've migrated the major-minor functionality over, but haven't adapted it for this particular format yet, but should be able to add it soon.
The .txt files I added to the reference_data folder were used to quickly test this setup. This will still need testing for an entire approach (once we get the baseline up and running on this format, we can use that).
I made several assumptions on were the data (prediction, solutions, etc.) will be held. @egrace479 let me know if you see a problem with my assumptions.
Also worth noting is we don't have error checking for the input files (predictions for example). Should we?