feat: Improve metadata speed for big directories #272
+29
−19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dmga44
requested review from
uristdwarf,
lgsilva3087,
ralcolea,
antuan96314,
ictus4u and
rolysr
dmga44
force-pushed
the
big-directories-metadata-speedup
branch
from
abb1e6c to
0ee4fab
Compare
rolysr
previously approved these changes
Jan 6, 2025
uristdwarf
reviewed
Jan 6, 2025
dmga44
force-pushed
the
big-directories-metadata-speedup
branch
from
0ee4fab to
568ae81
Compare
dmga44
force-pushed
the
big-directories-metadata-speedup
branch
from
568ae81 to
b104bff
Compare
dmga44
force-pushed
the
big-directories-metadata-speedup
branch
from
b104bff to
f6b6cf0
Compare
uristdwarf
approved these changes
Jan 8, 2025
dmga44
force-pushed
the
big-directories-metadata-speedup
branch
from
f6b6cf0 to
fdacc58
Compare
This commit changes the old flat_map that was advised to use for an ordered_statistics structure that ensures good performance in every case. The custom flat_map structure previously used had O(n) insertion and elimination costs, the new one has O(logn) cost for both of those operations. The goal of this change is to speedup the metadata operations for directories with lots of files. Signed-off-by: Dave <dave@leil.io>
dmga44
force-pushed
the
big-directories-metadata-speedup
branch
from
fdacc58 to
5280a71
Compare
ralcolea
approved these changes
Jan 21, 2025
| typedef flat_map<hstorage::Handle *, FSNode *, | ||
| std::vector<std::pair<hstorage::Handle *, FSNode *>>, | ||
| HandleCompare> | ||
| typedef __gnu_pbds::tree<std::pair<hstorage::Handle *, FSNode *>, |
There was a problem hiding this comment.
Consider to use modern C++ using instruction instead of typedef in the definition of EntriesContainer.
Consider also to apply the same change in the current definition of iterator and const_iterator that also use typedef.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Must be merged after #268 ]
This commit changes the old flat_map that was advised to use for an
ordered_statistics structure that ensures good performance in every
case. The custom flat_map structure previously used had O(n) insertion
and elimination costs, the new one has O(logn) cost for both of those
operations. The goal of this change is to speedup the metadata
operations for directories with lots of files.