Struct SpillManager 

pub struct SpillManager {
    env: Arc<RuntimeEnv>,
    pub(crate) metrics: SpillMetrics,
    schema: Arc<Schema>,
    batch_read_buffer_capacity: usize,
    pub(crate) compression: SpillCompression,
}

The SpillManager is responsible for the following tasks:

• Reading and writing RecordBatches to raw files based on the provided configurations.
• Updating the associated metrics.

Note: The caller (external operators such as SortExec) is responsible for interpreting the spilled files. For example, all records within the same spill file are ordered according to a specific order.

Fields

env: Arc<RuntimeEnv>

metrics: SpillMetrics

schema: Arc<Schema>

batch_read_buffer_capacity: usize

compression: SpillCompression

Implementations

impl SpillManager

pub fn new( env: Arc<RuntimeEnv>, metrics: SpillMetrics, schema: Arc<Schema>, ) -> SpillManager

pub fn with_batch_read_buffer_capacity( self, batch_read_buffer_capacity: usize, ) -> SpillManager

pub fn with_compression_type( self, spill_compression: SpillCompression, ) -> SpillManager

pub fn create_in_progress_file( &self, request_msg: &str, ) -> Result<InProgressSpillFile, DataFusionError>

Creates a temporary file for in-progress operations, returning an error message if file creation fails. The file can be used to append batches incrementally and then finish the file when done.

pub fn spill_record_batch_and_finish( &self, batches: &[RecordBatch], request_msg: &str, ) -> Result<Option<RefCountedTempFile>, DataFusionError>

Spill input batches into a single file in a atomic operation. If it is intended to incrementally write in-memory batches into the same spill file, use Self::create_in_progress_file instead. None is returned if no batches are spilled.

Errors

• Returns an error if spilling would exceed the disk usage limit configured by max_temp_directory_size in DiskManager

pub fn read_spill_as_stream( &self, spill_file_path: RefCountedTempFile, max_record_batch_memory: Option<usize>, ) -> Result<Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>, DataFusionError>

Reads a spill file as a stream. The file must be created by the current SpillManager. This method will generate output in FIFO order: the batch appended first will be read first.

Trait Implementations

impl Clone for SpillManager

fn clone(&self) -> SpillManager

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SpillManager

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.