Struct LimitedBatchCoalescer 

pub struct LimitedBatchCoalescer {
    inner: BatchCoalescer,
    total_rows: usize,
    fetch: Option<usize>,
    finished: bool,
}

Concatenate multiple [RecordBatch]es and apply a limit

See [BatchCoalescer] for more details on how this works.

Fields

inner: BatchCoalescer

The arrow structure that builds the output batches

total_rows: usize

Total number of rows returned so far

fetch: Option<usize>

Limit: maximum number of rows to fetch, None means fetch all rows

finished: bool

Indicates if the coalescer is finished

Implementations

impl LimitedBatchCoalescer

pub fn new( schema: SchemaRef, target_batch_size: usize, fetch: Option<usize>, ) -> Self

Create a new BatchCoalescer

Arguments

• schema - the schema of the output batches
• target_batch_size - the minimum number of rows for each output batch (until limit reached)
• fetch - the maximum number of rows to fetch, None means fetch all rows

pub fn schema(&self) -> SchemaRef

Return the schema of the output batches

pub fn push_batch(&mut self, batch: RecordBatch) -> Result<PushBatchStatus>

Pushes the next [RecordBatch] into the coalescer and returns its status.

Arguments

• batch - The [RecordBatch] to append.

Returns

• PushBatchStatus::Continue - More batches can still be pushed.
• PushBatchStatus::LimitReached - The row limit was reached after processing this batch. The caller should call Self::finish before retrieving the remaining buffered batches.

Errors

Returns an error if called after Self::finish or if the internal push operation fails.

pub fn is_empty(&self) -> bool

Return true if there is no data buffered

pub fn finish(&mut self) -> Result<()>

Complete the current buffered batch and finish the coalescer

Any subsequent calls to push_batch() will return an Err

pub fn next_completed_batch(&mut self) -> Option<RecordBatch>

Return the next completed batch, if any

Trait Implementations

impl Debug for LimitedBatchCoalescer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.