Admin Tools Enhancement and Cost Optimization

[gchhablani]

Project Title: Admin Tools Enhancement and Cost Optimization

Description: The goal of this project is to focus on improving admin experience on EvalAI, as well as target efficient cost-reduction for maintaining EvalAI.

One of the primary focuses will be on enhancing the existing automation of the cancellation of submissions which have expired messages on the SQS queues. The second focus will be identifying underutilized/overutilized ECS instances using AWS health metrics to automatically determine the required compute. Other improvements will include admin actions on Django administration for starting/stopping/restarting the EC2 instance workers, and providing automated deletion of code-upload infrastructure on challenge un-approval.

These features, along with others mentioned in the deliverables, will make EvalAI administrative experience seamless and will also save costs in the longer run.

Deliverable:

• Admin Enhancements:
  • Create admin actions for EC2 worker start/stop/create/restart.
  • Implement an approval button directly on Slack request notifications for challenge approval.
  • Add a feature to automatically delete code-upload infrastructure on unapproving challenges via Django administration.
• Cost Optimization Measures:
  • Use the custom SQS queue retention time to automatically cancel submissions and save costs.
  • Enhance the auto-cancel script to reflect changes in Prometheus metrics for accuracy of metrics.
  • Add improvements for retention of Prometheus metrics on container restart.
  • Identify and stop excessive instances and EC2 clones running on AWS for cost-saving.
  • Identify and remove old ECR repositories (and other avenues) for cost reduction on AWS.
• Infrastructure Monitoring and Automation:
  • Automate ECS monitoring to detect and adjust CPU and memory consumption based on challenge requirements.
  • Address challenges requiring frequent restarts on both EC2 and ECS instances. Migrate problematic workers from ECS to EC2 and improve efficiency of the instances.

Mentor: @gchhablani, Rahul Singh, @gautamjajoo, @RishabhJain2018

Skills: Python, Django, AngularJS, AWS

Skill Level: Medium

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSoC-2024).

Important Links:

• EvalAI Website: eval.ai
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2020-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[mridul45]

Hi @gchhablani , I am new to GSOC and I was wondering how to get started. So I have a question , do I have to present a soluttion to the above project title or I have to download the code , study it and then prepare a detailed proposal about my own solution?

[saif-hacker]

Hi @gchhablani, @gautamjajoo, @RishabhJain2018 I am new to GSOC and i am an absolute beginner in open source contribution i need some assistance to get started so that i could effectively contribute to this issue

[Antoniocolapso]

Interested for the project.

A little bit about me :

I'm Omm Prakash Sahoo from IIT Bhilai, 3rd year b.tech. i recently become ML guy after doing CP and dev in my first 2 years.Was ICPC regionalist expert at CodeForces done decent amount of DEVoPs, Developed both Web and android apps, topped in SystemDesign and ML course at our college, built LIP-Reader (predicting sentence only by lip movement), was lead of INTER-IIT Tech meet team for Adobe behaviour simulation challenge in which we developed 2 LLMs to ease the process of posting new content for marketing for companies in every segment and a lots of interesting works.

After achieving my personal goals in CP in first 2 years now i want to finally contribute to real-world problems.Coz have been member of GDSC, OpenLake(Club for Open Source enthusiasts at IIT Bhilai) and Co-ordinator of Ingenuity (CP club).

Here is my resume link : https://drive.google.com/file/d/1LbGBW9veH75x7IqABSk9ui03JIzZv7cV/view?usp=drive_link

I will be there to solve it soon.

[abdulhameed04]

Hello. I'm Abdul Hameed, pursuing my Statistics Honours from Delhi University. I'm new to this GSoC platform, and also I'm a vivid learner adopting to different work environment. Loved to help you in this project.

[Aisikhue]

Good day,

My name is Ayemhenre Isikhuemhen, I am a student at the University of North Carolina at Charlotte, studying Computer Science with a concentration in Artificial Intelligence. I am looking for hands-on experience with Machine Learning and would love to have the opportunity to contribute to this project.

You can access my resume using the following link:
https://drive.google.com/file/d/1UcbCRY0mC9AqjcN49DwYwSBeKdZoMavT/view?usp=sharing

Kindly,
Ayemehenre Isikhuemhen
aisikhue@uncc.edu

[RedEye124]

Hi @gchhablani, @gautamjajoo, @RishabhJain2018,

I hope you’re all doing well. My name is Ranjith, and I’m excited to be starting my journey with GSoC. As a beginner in both GSoC and open source contributions, I’m eager to learn and make meaningful contributions to the EvalAI project.
would greatly appreciate any guidance or advice you can provide to help me get started.

[abdulhameed04]

It was nice meeting you. We can proceed further through email.

…

On Wed, 14 Aug, 2024, 12:44 pm Narapinni Ranjith, ***@***.***> wrote: Hi @gchhablani <https://github.com/gchhablani>, @gautamjajoo <https://github.com/gautamjajoo>, @RishabhJain2018 <https://github.com/RishabhJain2018>, I hope you’re all doing well. My name is Ranjith, and I’m excited to be starting my journey with GSoC. As a beginner in both GSoC and open source contributions, I’m eager to learn and make meaningful contributions to the EvalAI project. would greatly appreciate any guidance or advice you can provide to help me get started. — Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX2HQIZDCB7OUPFXZBMM5R3ZRL7XFAVCNFSM6AAAAABDB7JOOSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBYGAYTQOJUHE> . You are receiving this because you commented.Message ID: ***@***.***>

[ABHI080201]

Hi @gchhablani, @gautamjajoo, @RishabhJain2018,
Hello everyone, I hope you're doing well! My name is Abhishek, and I'm thrilled to begin my GSoC journey. As a beginner in both GSoC and open source contributions, I'm excited to learn and contribute to the EvalAI project. My primary interests lie in Full Stack AI Development and Data Science. I would greatly appreciate any guidance or advice to help me get started on the right path.