Skip to content
This repository has been archived by the owner on May 3, 2024. It is now read-only.

Reduce burst trie memory usage #541

Merged
merged 12 commits into from
Jul 31, 2023
Merged

Reduce burst trie memory usage #541

merged 12 commits into from
Jul 31, 2023

Conversation

MBkkt
Copy link
Contributor

@MBkkt MBkkt commented Jul 31, 2023 •
edited
Loading

Main improvements are

  • clear monotonic buffer after field end
  • release monotonic buffer after writes end

but always keep next block size, also

  • sizeof prefixed_output 96 instead of 104 (node in list)
    because single linked list instead of double linked, so it single pointer in the Node instead of two
  • sizeof entry(block_index_t) 88(8) instead of 120(40)
    because list just a single tail pointer instead of head, tail, size and alloc

@MBkkt MBkkt requested review from gnusi and Dronplane July 31, 2023 04:30
@MBkkt MBkkt marked this pull request as ready for review July 31, 2023 10:39
@MBkkt MBkkt changed the title burst trie wip Reduce burst trie memory usage Jul 31, 2023
Dronplane
Dronplane reviewed Jul 31, 2023
View reviewed changes
core/formats/formats_burst_trie.cpp
IRS_ASSERT(p == reinterpret_cast<byte_type*>(head_));
current_ = p + sizeof(Block);
available_ = next_size_;
next_size_ = (next_size_ * 3) / 2;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could track first call of AllocateMemory and do increase only if not first. So we will not need tricks with re-counting the size.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like if approach because it more cases to think

Dronplane
Dronplane approved these changes Jul 31, 2023
View reviewed changes
Copy link
Contributor

@Dronplane Dronplane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MBkkt MBkkt merged commit ee5015c into master Jul 31, 2023
1 check passed
@MBkkt MBkkt deleted the chore/make-burst-trie-great-again branch July 31, 2023 17:41
@MBkkt MBkkt mentioned this pull request Jul 31, 2023
Merged
19 tasks
This was referenced Aug 14, 2023
Merged
Merged
MBkkt added a commit that referenced this pull request Aug 14, 2023
@MBkkt
Backports 1.2.10 (#546)
4f92c21 
* #516
* #536
* #537
* #543 
* #541
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@Dronplane Dronplane Dronplane approved these changes

@gnusi gnusi Awaiting requested review from gnusi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MBkkt @Dronplane