Save heap when dynamically mapping arrays #127439
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We can do the merging more eagerly to save on heap at the cost of some cycles (though the overhead should be reasonably limited because we save a little down the line if all the mappers are the same instance). Obviously a bit of a dirty trick but this logic is quite brittle now with things like the vector mapper special case and this seems to me like the shortest path to avoiding runaway heap use. This yet again shows the need to implement actual mapper equality checks and stronger deduplication for them to avoid having mapping merging be our only mechanism for deduplication. closes elastic#117593
original-brownbear
added
>bug
:Search Foundations/Mapping
|
Hi @original-brownbear, I've created a changelog YAML for you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We can do the merging more eagerly to save on heap at the cost of some cycles (though the overhead should be reasonably limited because we save a little down the line if all the mappers are the same instance). Obviously a bit of a dirty trick but this logic is quite brittle now with things like the vector mapper special case and this seems to me like the shortest path to avoiding runaway heap use.
This yet again shows the need to implement actual mapper equality checks and stronger deduplication for them to avoid having mapping merging be our only mechanism for deduplication.
This comes with a slight behavior change too by making unparseable objects with conflicting field types fail to parse a little more eagerly at the parsing stage (as they should IMO). This helps with some of the extreme slowdowns that a steady stream of these eventually unparseable documents has been introducing here and there in heavily loaded deployments.
closes #117593