Struct SpillManager 

pub struct SpillManager {
    env: Arc<RuntimeEnv>,
    pub(crate) metrics: SpillMetrics,
    schema: SchemaRef,
    batch_read_buffer_capacity: usize,
    pub(crate) compression: SpillCompression,
}

The SpillManager is responsible for the following tasks:

• Reading and writing RecordBatches to raw files based on the provided configurations.
• Updating the associated metrics.

Note: The caller (external operators such as SortExec) is responsible for interpreting the spilled files. For example, all records within the same spill file are ordered according to a specific order.

Fields

env: Arc<RuntimeEnv>

metrics: SpillMetrics

schema: SchemaRef

batch_read_buffer_capacity: usize

Number of batches to buffer in memory during disk reads

compression: SpillCompression

general-purpose compression options

Implementations

impl SpillManager

pub fn new( env: Arc<RuntimeEnv>, metrics: SpillMetrics, schema: SchemaRef, ) -> Self

pub fn with_batch_read_buffer_capacity( self, batch_read_buffer_capacity: usize, ) -> Self

pub fn with_compression_type(self, spill_compression: SpillCompression) -> Self

pub fn create_in_progress_file( &self, request_msg: &str, ) -> Result<InProgressSpillFile>

Creates a temporary file for in-progress operations, returning an error message if file creation fails. The file can be used to append batches incrementally and then finish the file when done.

pub fn spill_record_batch_and_finish( &self, batches: &[RecordBatch], request_msg: &str, ) -> Result<Option<RefCountedTempFile>>

Spill input batches into a single file in a atomic operation. If it is intended to incrementally write in-memory batches into the same spill file, use Self::create_in_progress_file instead. None is returned if no batches are spilled.

Errors

• Returns an error if spilling would exceed the disk usage limit configured by max_temp_directory_size in DiskManager

pub(crate) fn spill_record_batch_by_size_and_return_max_batch_memory( &self, batch: &RecordBatch, request_description: &str, row_limit: usize, ) -> Result<Option<(RefCountedTempFile, usize)>>

Refer to the documentation for Self::spill_record_batch_and_finish. This method additionally spills the RecordBatch into smaller batches, divided by row_limit.

Errors

• Returns an error if spilling would exceed the disk usage limit configured by max_temp_directory_size in DiskManager

pub(crate) async fn spill_record_batch_stream_and_return_max_batch_memory( &self, stream: &mut SendableRecordBatchStream, request_description: &str, ) -> Result<Option<(RefCountedTempFile, usize)>>

Spill a stream of RecordBatches to disk and return the spill file and the size of the largest batch in memory

pub fn read_spill_as_stream( &self, spill_file_path: RefCountedTempFile, max_record_batch_memory: Option<usize>, ) -> Result<SendableRecordBatchStream>

Reads a spill file as a stream. The file must be created by the current SpillManager. This method will generate output in FIFO order: the batch appended first will be read first.

Trait Implementations

impl Clone for SpillManager

fn clone(&self) -> SpillManager

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SpillManager

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.