Struct InProgressSpillFile 

pub struct InProgressSpillFile {
    pub(crate) spill_writer: Arc<SpillManager>,
    writer: Option<IPCStreamWriter>,
    in_progress_file: Option<RefCountedTempFile>,
}

Represents an in-progress spill file used for writing RecordBatches to disk, created by SpillManager. Caller is able to use this struct to incrementally append in-memory batches to the file, and then finalize the file by calling the finish method.

Fields

spill_writer: Arc<SpillManager>

writer: Option<IPCStreamWriter>

Lazily initialized writer

in_progress_file: Option<RefCountedTempFile>

Lazily initialized in-progress file, it will be moved out when the finish method is invoked

Implementations

impl InProgressSpillFile

pub fn new( spill_writer: Arc<SpillManager>, in_progress_file: RefCountedTempFile, ) -> Self

pub fn append_batch(&mut self, batch: &RecordBatch) -> Result<()>

Appends a RecordBatch to the spill file, initializing the writer if necessary.

Errors

• Returns an error if the file is not active (has been finalized)
• Returns an error if appending would exceed the disk usage limit configured by max_temp_directory_size in DiskManager

pub fn finish(&mut self) -> Result<Option<RefCountedTempFile>>

Finalizes the file, returning the completed file reference. If there are no batches spilled before, it returns None.