Struct MemorySourceConfig 

pub struct MemorySourceConfig {
    partitions: Vec<Vec<RecordBatch>>,
    schema: SchemaRef,
    projected_schema: SchemaRef,
    projection: Option<Vec<usize>>,
    sort_information: Vec<LexOrdering>,
    show_sizes: bool,
    fetch: Option<usize>,
}

Data source configuration for reading in-memory batches of data

Fields

partitions: Vec<Vec<RecordBatch>>

The partitions to query.

Each partition is a Vec<RecordBatch>.

schema: SchemaRef

Schema representing the data before projection

projected_schema: SchemaRef

Schema representing the data after the optional projection is applied

projection: Option<Vec<usize>>

Optional projection

sort_information: Vec<LexOrdering>

Sort information: one or more equivalent orderings

show_sizes: bool

if partition sizes should be displayed

fetch: Option<usize>

The maximum number of records to read from this plan. If None, all records after filtering are returned.

Implementations

impl MemorySourceConfig

pub fn try_new( partitions: &[Vec<RecordBatch>], schema: SchemaRef, projection: Option<Vec<usize>>, ) -> Result<Self>

Create a new MemorySourceConfig for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_exec( partitions: &[Vec<RecordBatch>], schema: SchemaRef, projection: Option<Vec<usize>>, ) -> Result<Arc<DataSourceExec>>

Create a new DataSourceExec plan for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_as_values( schema: SchemaRef, data: Vec<Vec<Arc<dyn PhysicalExpr>>>, ) -> Result<Arc<DataSourceExec>>

Create a new execution plan from a list of constant values (ValuesExec)

pub fn try_new_from_batches( schema: SchemaRef, batches: Vec<RecordBatch>, ) -> Result<Arc<DataSourceExec>>

Create a new plan using the provided schema and batches.

Errors if any of the batches don’t match the provided schema, or if no batches are provided.

pub fn with_limit(self, limit: Option<usize>) -> Self

Set the limit of the files

pub fn with_show_sizes(self, show_sizes: bool) -> Self

Set show_sizes to determine whether to display partition sizes

pub fn partitions(&self) -> &[Vec<RecordBatch>]

Ref to partitions

pub fn projection(&self) -> &Option<Vec<usize>>

Ref to projection

pub fn show_sizes(&self) -> bool

Show sizes

pub fn sort_information(&self) -> &[LexOrdering]

Ref to sort information

pub fn try_with_sort_information( self, sort_information: Vec<LexOrdering>, ) -> Result<Self>

A memory table can be ordered by multiple expressions simultaneously. [EquivalenceProperties] keeps track of expressions that describe the global ordering of the schema. These columns are not necessarily same; e.g.

┌-------┐
| a | b |
|---|---|
| 1 | 9 |
| 2 | 8 |
| 3 | 7 |
| 5 | 5 |
└---┴---┘

where both a ASC and b DESC can describe the table ordering. With [EquivalenceProperties], we can keep track of these equivalences and treat a ASC and b DESC as the same ordering requirement.

Note that if there is an internal projection, that projection will be also applied to the given sort_information.

pub fn original_schema(&self) -> SchemaRef

Arc clone of ref to original schema

fn repartition_preserving_order( &self, target_partitions: usize, output_ordering: LexOrdering, ) -> Result<Option<Vec<Vec<RecordBatch>>>>

Repartition while preserving order.

Returns Ok(None) if cannot fulfill the requested repartitioning, such as having too few batches to fulfill the target_partitions or if unable to preserve output ordering.

fn repartition_evenly_by_size( &self, target_partitions: usize, ) -> Result<Option<Vec<Vec<RecordBatch>>>>

Repartition into evenly sized chunks (as much as possible without batch splitting), disregarding any ordering.

Current implementation uses a first-fit-decreasing bin packing, modified to enable us to still return the desired count of target_partitions.

Returns Ok(None) if cannot fulfill the requested repartitioning, such as having too few batches to fulfill the target_partitions.

Trait Implementations

impl Clone for MemorySourceConfig

fn clone(&self) -> MemorySourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl DataSource for MemorySourceConfig

fn repartitioned( &self, target_partitions: usize, _repartition_file_min_size: usize, output_ordering: Option<LexOrdering>, ) -> Result<Option<Arc<dyn DataSource>>>

If possible, redistribute batches across partitions according to their size.

Returns Ok(None) if unable to repartition. Preserve output ordering if exists. Refer to DataSource::repartitioned for further details.

fn open( &self, partition: usize, _context: Arc<TaskContext>, ) -> Result<SendableRecordBatchStream>

fn as_any(&self) -> &dyn Any

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format this source for display in explain plans

fn output_partitioning(&self) -> Partitioning

fn eq_properties(&self) -> EquivalenceProperties

fn scheduling_type(&self) -> SchedulingType

fn partition_statistics(&self, partition: Option<usize>) -> Result<Statistics>

Returns statistics for a specific partition, or aggregate statistics across all partitions if partition is None.

fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn DataSource>>

Return a copy of this DataSource with a new fetch limit

fn fetch(&self) -> Option<usize>

fn try_swapping_with_projection( &self, projection: &[ProjectionExpr], ) -> Result<Option<Arc<dyn DataSource>>>

fn statistics(&self) -> Result<Statistics>

👎Deprecated since 51.0.0: Use partition_statistics instead

Returns aggregate statistics across all partitions.

fn metrics(&self) -> ExecutionPlanMetricsSet

fn try_pushdown_filters( &self, filters: Vec<Arc<dyn PhysicalExpr>>, _config: &ConfigOptions, ) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>>

Try to push down filters into this DataSource. See ExecutionPlan::handle_child_pushdown_result for more details.

impl Debug for MemorySourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.