add new feature flag to intern value string #761
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feature request
Existing scenario
I think the
INTERN_FIELD_NAMES
flag is great thing to have, which is turned on by default. This would save greatly on the memory footprint especially when the message size is huge, imagine ~3.2 million position messages, all with same key like 'portfolio', 'book'.Instead of having ~3.2 million of
portfolio
in parsing each batch of message in heap, withINTERN_FIELD_NAMES
, it would result in only oneportfolio
on the string pool regardless of the message sizes, being ~3.2 million or even more.Changes proposed
A similar feature flag could be provided, even turned on by default as well, when parsing the values.
So that, back to the ~3.2 million records example, instead of having ~3.2 million portfolio names in the heap, the similar feature flag would result in only around ~200
portfolio name
(like Jason, Jackson) in the string pool.#726