Struct BatchPartitioner 

pub struct BatchPartitioner {
    state: BatchPartitionerState,
    timer: Time,
}

A utility that can be used to partition batches based on Partitioning

Fields

state: BatchPartitionerState

timer: Time

Implementations

impl BatchPartitioner

pub fn try_new(partitioning: Partitioning, timer: Time) -> Result<Self>

Create a new BatchPartitioner with the provided Partitioning

The time spent repartitioning will be recorded to timer

pub fn partition<F>(&mut self, batch: RecordBatch, f: F) -> Result<()>

where F: FnMut(usize, RecordBatch) -> Result<()>,

Partition the provided [RecordBatch] into one or more partitioned [RecordBatch] based on the Partitioning specified on construction

f will be called for each partitioned [RecordBatch] with the corresponding partition index. Any error returned by f will be immediately returned by this function without attempting to publish further [RecordBatch]

The time spent repartitioning, not including time spent in f will be recorded to the metrics::Time provided on construction

fn partition_iter( &mut self, batch: RecordBatch, ) -> Result<impl Iterator<Item = Result<(usize, RecordBatch)>> + Send + '_>

Actual implementation of partition.

The reason this was pulled out is that we need to have a variant of partition that works w/ sync functions, and one that works w/ async. Using an iterator as an intermediate representation was the best way to achieve this (so we don’t need to clone the entire implementation).

fn num_partitions(&self) -> usize