-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Confident Predictions Selection #269
Comments
Do you consider having a better solution for this issue and do i have some time to work on it? |
I'm afraid you can ruin my work by closing the issue before i get a good score, would you please keep it open? |
@evgenydmitriev I need to ask you about this issue. Did anyone exceed the winner's score significantly or is it impossible? I liked this task very much and I'd like to work on other tasks like this. |
@Annyvichi, I closed the issue because I saw no activity from the challenge organizers for almost a year, but if you email us at challenge@dn.institute, I can forward your resume/profile to a few NLP teams. |
@evgenydmitriev, I will take some time to improve my profile and then will apply at the email and ask about open positions in your team. |
💸 Confident Predictions Selection is a bounty challenge 💸
Your task
To design a model capable of selecting the top 10% of predictions that will exhibit the smallest mean distance.
You have an option to work with either or both of the text data and raw predictions. Additionally, you may perform any form of aggregation or transformation on the raw predictions as you see fit.
Validation Metrics
The primary metric for this challenge is the mean distance of the selected top 10% predictions. Your objective is to minimize this value.
As a secondary metric, we designed the Class Representation Index (CRI). In essence, CRI compares the class distribution before and after filtering, giving a higher weight to classes that were initially larger. The primary purpose of this metric is to detect cases where a class is significantly less represented after filtering compared to its original size.
Please, see the dedicated repository for instructions
The text was updated successfully, but these errors were encountered: