-
-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add toggle to disable results/backup/
files in AWS mode
#586
Comments
The backup should be made in amlb/results.py#L112, called from the Benchmark, if I am not mistaken. Having an option is call In the meantime, you could disable |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am running large-scale benchmarks in AWS mode and finding that there are files being saved in
results/backup/
that take up significant space (leading to >1 TB of files that cause the host machine to run out of disk during the benchmark run).Where in the code are these files being specified and how can I disable them? Are they necessary for anything? I would assume not.
The problem is that each file in
backup
is concatenating all the results of the benchmark together into a CSV file, causing it to take N^2 space where N is the number of instances being spun up (and in my case, N > 20,000).As an example:
There are around 10 of these files being written a minute, each one larger than the last (currently 108MB per file), meaning 1 GB of disk space is being taken up a minute.
The text was updated successfully, but these errors were encountered: