Add cache existence check to wrap_with_run_lock #331
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changed feature
I've added cache existence check to wrap_with_run_lock
What is wrap_with_run_lock?
wrap_with_run_lock is a wrapper function for TaskOnKart.run().
This is used to wrap TaskOnKart.run(), when you want to use efficient task cache collision lock https://gokart.readthedocs.io/en/latest/using_task_cache_collision_lock.html#advanced-using-efficient-task-cache-collision-lock
What is the problem of original code?
Suppose you are to run same task on different nodes at the same time.
In this case, node1 and node2 will do the same thing with the same result, which is inefficient.
By using efficient task cache collision lock, TaskA.run() will not run at the same time.
However, in the original code, after node1 finished running TaskA, node2 will run TaskA again.
Cache collision will not happen, but this is still inefficient, because node2 didn't need to run TaskA again.
How to overcome this problem?
I've added cache existence check just before TaskOnKart.run() to prevent running run() again.
When the cache file is found at the runtime, run() will be skipped.