Replay Proto-X #32
Merged
Replay Proto-X #32
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ad or the query timed out
17zhangw
reviewed
Apr 27, 2024
17zhangw
requested changes
May 29, 2024
| if self.logger: | ||
| # We only stash the results if we're not doing HPO, or else the results from concurrent HPO would get | ||
| # stashed in the same directory and potentially cause a race condition. | ||
| if self.logger and not tuning_mode == TuningMode.HPO: |
There was a problem hiding this comment.
Can we also use the ray_trial_id to stash the results during HPO?
There was a problem hiding this comment.
Thanks for catching that!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Can now replay a full tuning run from Proto-X. This is used to see how each step of tuning would have done without query timeouts and without Boot enabled.
Demo:
The image shows the data of a replayed run of TPC-H SF0.01 without Boot enabled during tuning. For each step of tuning, the replay shows the # of queries executed during the original run (which may be < 22 if the workload timed out), the # of queries that timed out during the original run, and whether the workload timed out. It also shows this same information about the replay. You can see that the replayed times are always >= the original times, which makes sense. Whenever the original run is "22,0,False" (i.e. 22 executed, 0 timed out, workload didn't time out), the replayed time matches closely.
Details:
execute_variations()as well as all per-query knob variations tried. We can either replay the best variation or all variations. This is especially useful if the workload timed out in the original run, in which case the "best" variation is a misnomer as it is simply an arbitrary variation.reset()was overwriting the logged replay information for a step, leading to a mismatch between the dumpedaction.pklfile (which contains the DBMS configuration state) and therun.raw.csvfile (which contains the runtime information of the workload during that state)..linkextension to fix a subtle bug where a replay would overwrite theoutput.logfile of the original run.