Add bytes-based flushing to source collectors#1102
Open
46bit wants to merge 2 commits into
Open
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
46bit
force-pushed
the
miki/bytes-based-collector-flushing
branch
from
fd29177 to
9ebfc42
Compare
Sources read data in and collect it up into batches. Batches are flushed based on configurable criteria: until now this has been the number of records collected and how long they have been waiting. This commit adds another criteria for flushing, based on the memory usage of the records. `source-batch-max-bytes`, disabled by default, allows triggering flushing when a certain amount of data has been collected. This new feature enables two useful behaviours: 1) Large records: reduce the risk of OOMs when memory-constrained 2) Small records: create larger, more-efficient batches Memory usage is estimated based on input bytes, rather than by instrumenting Arrow buffer memory. Ideally we would implement this based on Arrow buffer usage. However, Arrow's `ArrayBuilder` trait doesn't expose memory size and we would need to maintain a lot of code ourselves to make it work. I think it's reasonable to use input bytes as an approximation.
Apache policy removes old releases from downloads.apache.org (and its CDN, dlcdn.apache.org) once a version branch is no longer current. Switch to archive.apache.org, which retains all historical releases.
46bit
force-pushed
the
miki/bytes-based-collector-flushing
branch
from
9ebfc42 to
2ccfaf7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Sources read data in and collect it up into batches. Batches are flushed based on configurable criteria: until now this has been the number of records collected and how long they have been waiting.
This commit adds another criteria for flushing, based on the memory usage of the records.
source-batch-max-bytes, disabled by default, allows triggering flushing when a certain amount of data has been collected.Why
This new feature enables two useful behaviours:
How
Memory usage is estimated based on input bytes, rather than by instrumenting Arrow buffer memory.
Ideally we would implement this based on Arrow buffer usage. However, Arrow's
ArrayBuildertrait doesn't expose memory size and we would need to maintain a lot of code ourselves to make it work. I think it's reasonable to use input bytes as an approximation.