Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cache existence check to wrap_with_run_lock #331

Conversation

mski-iksm
Copy link
Contributor

Changed feature

I've added cache existence check to wrap_with_run_lock

What is wrap_with_run_lock?

wrap_with_run_lock is a wrapper function for TaskOnKart.run().
This is used to wrap TaskOnKart.run(), when you want to use efficient task cache collision lock https://gokart.readthedocs.io/en/latest/using_task_cache_collision_lock.html#advanced-using-efficient-task-cache-collision-lock

What is the problem of original code?

Suppose you are to run same task on different nodes at the same time.

  • node1: TaskA
  • node2: TaskA

In this case, node1 and node2 will do the same thing with the same result, which is inefficient.

By using efficient task cache collision lock, TaskA.run() will not run at the same time.

However, in the original code, after node1 finished running TaskA, node2 will run TaskA again.
Cache collision will not happen, but this is still inefficient, because node2 didn't need to run TaskA again.

How to overcome this problem?

I've added cache existence check just before TaskOnKart.run() to prevent running run() again.
When the cache file is found at the runtime, run() will be skipped.

@mski-iksm
Copy link
Contributor Author

I will close this PR, because cache existence check must be separated from cache collision lock wrapper.

@mski-iksm mski-iksm closed this Nov 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant