Download worker refactor #288
Conversation
MattUnderscoreZhang
changed the title
|
Can you rebase please ? |
|
Ok, done |
|
To describe high-level what happened here, I took overlapping logic from subset_worker and download_worker, and moved them to a common file worker.py. I did try to fully combine the two workers into a single class, but was unable to do so right now due to differences in how they load input data. The main reason I did this was because both workers use the video processing subsamplers, which I'm trying to rewrite to not use temp file read/writes. By putting the common video processing logic in a single file, I can reduce the number of changes I have to track later. I have tested both subset_worker and download_worker on a subset of webvid, using my usual workflow with a framerate resampling, resizing, cropping, padding, clip finding, and cutting. I did this both as a single-step download+process, and with a download + subsequent process. Both methods replicated previous results, showing that this refactor should have no functional effect. |
|
looks pretty good to me, let's merge |
* ClippingSubsampler rewrite and bug fixes * More refactoring of ClippingSubsampler, plus a fix to _get_clip_intervals * Finished refactoring ClippingSubsampler * Final code changes * Added docstrings * Passed tests and linting * Made type annotations consistent with Python 3.8 * More annotation fixes * The Python 3.8 annotation needs a lot of hand-holding, it seems * Pylint has to cut it out, I swear to God * No real change, just relauching unit tests which failed due to connection timeouts * Linting issue * Another linting issue * Separated per-shard code from code that should only be executed once * Pulled ShardStatus parameters into their own data type * Cleaned up shard processing error handling * Cleaned up code * Bug fixes * Formatting * Fixed linting issues * Fixing more damn linting * Added a missing docstring * Unified SubsetWorker and DownloadWorker code * Bug fixes * Linting * Linting again * Forgot a docstring * Removed unnecessary manual thread handling * Removed unused import --------- Co-authored-by: iejMac <kilianmaciej6@gmail.com> Co-authored-by: Romain Beaumont <romain.rom1@gmail.com>
Unifies SubsetWorker and DownloadWorker logic.
The two classes can probably be combined at this point.
Merge after #287.