Migrate training into celery and upload results to s3

[andrewpeng02]

Migrate training into celery and upload results to s3

Github Issue Number Here: #1136
What user problem are we solving?

Currently, we do the training during the HTTP request. I plan on changing the train HTTP endpoints by scheduling a train job via Celery and returning the job id in the request. This offers numerous advantages

Long training tasks (>2 min) shouldn't be done in an HTTP request. Scheduling the training job will allow the endpoint to return quickly
Eventually, we can decouple the backend with the training, so that we can use cheaper EC2 instances for the Django backend, and GPU instances for the actual training
Notifying the user will be done in websockets in this issue (#920 (comment)), for now I'll create an HTTP endpoint to retrieve the training results that the user can ping.

What solution does this PR provide?
Training jobs are executed by a celery worker, which polls from the queue set up in aws sqs. Currently you have to run the celery worker locally, but eventually the workers will run in the g4dn.xlarge ec2 instances. In this pr, I added celery endpoints which the backend will call. Then, the frontend will request the training results data from GET /api/training/results/{trainspaceId}, and it'll display the data.

I also moved some files into /celery to make it more clear that the celery worker will operate in there (but it'll also access files in the django app)

Testing Methodology

1. Install the new backend dependencies
2. Start the frontend using dlp-cli frontend start
3. From the training/ folder, start the backend using AWS_PROFILE=dlp docker compose up --build
4. Start the sst locally, using AWS_PROFILE=sst
5. Tested training tabular regression and classification data

newmovie.mov

Any other considerations

[sonarcloud]

Quality Gate passed

Issues
3 New issues
54 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[karkir0003]

@andrewpeng02 Question RE testing video:

1. After you submitted the training requests, were the results shown just dummy data for now or was it actual?

[karkir0003]

Thank you for the detailed PR here. Main question I had was:

1. How does the data flow work from frontend sending a train request to backend to passing the data to celery? What will be the data flow in production?
2. Any high level diagram you could provide/document/link me to in order to understand how celery works and how it's scalable for our use case?

[andrewpeng02]

• The results shown were actual data, not dummy
• Frontend sends training params to backend -> backend submits training job with training params, user id, trainspace id to the queue -> celery worker picks up a job from the queue and trains -> celery worker saves results to s3, which the frontend can request from sst
• I believe the architecture diagram here should answer your question

[karkir0003]

much better. followed up on the comments

[karkir0003]

responded to some of the followups

[codingwithsurya]

just took a look -- this looks awesome!

[karkir0003]

AWESOME PR @andrewpeng02 . I left a few nits in the PR, but looks really good.

Quick question: Will there be any documentation to understand the development process for say adding a new training type now that we have a celery based structure?

[andrewpeng02]

AWESOME PR @andrewpeng02 . I left a few nits in the PR, but looks really good.

Quick question: Will there be any documentation to understand the development process for say adding a new training type now that we have a celery based structure?

It shouldn't be difficult to support new training types, you should be able to figure it out by referencing the existing tabular and image training jobs

[sonarcloud]

Quality Gate passed

Issues
2 New issues
42 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.8% Duplication on New Code

See analysis details on SonarCloud