Struct LimitedBatchCoalescer 

pub struct LimitedBatchCoalescer {
    inner: BatchCoalescer,
    total_rows: usize,
    fetch: Option<usize>,
    finished: bool,
}

Concatenate multiple RecordBatches and apply a limit

See BatchCoalescer for more details on how this works.

Fields

inner: BatchCoalescer

total_rows: usize

fetch: Option<usize>

finished: bool

Implementations

impl LimitedBatchCoalescer

pub fn new( schema: Arc<Schema>, target_batch_size: usize, fetch: Option<usize>, ) -> LimitedBatchCoalescer

Create a new BatchCoalescer

Arguments

• schema - the schema of the output batches
• target_batch_size - the minimum number of rows for each output batch (until limit reached)
• fetch - the maximum number of rows to fetch, None means fetch all rows

pub fn schema(&self) -> Arc<Schema>

Return the schema of the output batches

pub fn push_batch( &mut self, batch: RecordBatch, ) -> Result<PushBatchStatus, DataFusionError>

Pushes the next RecordBatch into the coalescer and returns its status.

Arguments

• batch - The RecordBatch to append.

Returns

• PushBatchStatus::Continue - More batches can still be pushed.
• PushBatchStatus::LimitReached - The row limit was reached after processing this batch. The caller should call Self::finish before retrieving the remaining buffered batches.

Errors

Returns an error if called after Self::finish or if the internal push operation fails.

pub fn is_empty(&self) -> bool

Return true if there is no data buffered

pub fn finish(&mut self) -> Result<(), DataFusionError>

Complete the current buffered batch and finish the coalescer

Any subsequent calls to push_batch() will return an Err

pub fn next_completed_batch(&mut self) -> Option<RecordBatch>

Return the next completed batch, if any

Trait Implementations

impl Debug for LimitedBatchCoalescer

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.