Struct PartitionColumnProjector 

pub struct PartitionColumnProjector {
    key_buffer_cache: ZeroBufferGenerators,
    projected_partition_indexes: Vec<(usize, usize)>,
    projected_schema: SchemaRef,
}

A helper that projects partition columns into the file record batches.

One interesting trick is the usage of a cache for the key buffers of the partition column dictionaries. Indeed, the partition columns are constant, so the dictionaries that represent them have all their keys equal to 0. This enables us to re-use the same “all-zero” buffer across batches, which makes the space consumption of the partition columns O(batch_size) instead of O(record_count).

Fields

key_buffer_cache: ZeroBufferGenerators

An Arrow buffer initialized to zeros that represents the key array of all partition columns (partition columns are materialized by dictionary arrays with only one value in the dictionary, thus all the keys are equal to zero).

projected_partition_indexes: Vec<(usize, usize)>

Mapping between the indexes in the list of partition columns and the target schema. Sorted by index in the target schema so that we can iterate on it to insert the partition columns in the target record batch.

projected_schema: SchemaRef

The schema of the table once the projection was applied.

Implementations

impl PartitionColumnProjector

pub fn new(projected_schema: SchemaRef, table_partition_cols: &[String]) -> Self

pub fn project( &mut self, file_batch: RecordBatch, partition_values: &[ScalarValue], ) -> Result<RecordBatch>