Struct BatchBuilder 

pub struct BatchBuilder {
    schema: SchemaRef,
    batches: Vec<(usize, RecordBatch)>,
    reservation: MemoryReservation,
    cursors: Vec<BatchCursor>,
    indices: Vec<(usize, usize)>,
}

Provides an API to incrementally build a [RecordBatch] from partitioned [RecordBatch]

Fields

schema: SchemaRef

The schema of the RecordBatches yielded by this stream

batches: Vec<(usize, RecordBatch)>

Maintain a list of [RecordBatch] and their corresponding stream

reservation: MemoryReservation

Accounts for memory used by buffered batches

cursors: Vec<BatchCursor>

The current BatchCursor for each stream

indices: Vec<(usize, usize)>

The accumulated stream indexes from which to pull rows Consists of a tuple of (batch_idx, row_idx)

Implementations

impl BatchBuilder

pub fn new( schema: SchemaRef, stream_count: usize, batch_size: usize, reservation: MemoryReservation, ) -> Self

Create a new BatchBuilder with the provided stream_count and batch_size

pub fn push_batch( &mut self, stream_idx: usize, batch: RecordBatch, ) -> Result<()>

Append a new batch in stream_idx

pub fn push_row(&mut self, stream_idx: usize)

Append the next row from stream_idx

pub fn len(&self) -> usize

Returns the number of in-progress rows in this BatchBuilder

pub fn is_empty(&self) -> bool

Returns true if this BatchBuilder contains no in-progress rows

pub fn schema(&self) -> &SchemaRef

Returns the schema of this BatchBuilder

pub fn build_record_batch(&mut self) -> Result<Option<RecordBatch>>

Drains the in_progress row indexes, and builds a new RecordBatch from them

Will then drop any batches for which all rows have been yielded to the output

Returns None if no pending rows

Trait Implementations

impl Debug for BatchBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.