Module spill 

Defines the spilling functions

Modules

in_progress_spill_file 🔒

Define the InProgressSpillFile struct, which represents an in-progress spill file used for writing RecordBatches to disk, created by SpillManager.

spill_manager 🔒

Define the SpillManager struct, which is responsible for reading and writing RecordBatches to raw files based on the provided configurations.

Structs

IPCStreamWriter 🔒

Write in Arrow IPC Stream format to a file.

SpillReaderStream 🔒

Stream that reads spill files from disk where each batch is read in a spawned blocking task It will read one batch at a time and will not do any buffering, to buffer data use crate::common::spawn_buffered

Enums

SpillReaderStreamState 🔒

Constants

SPILL_BATCH_MEMORY_MARGIN 🔒

Functions

count_array_data_memory_size 🔒

Count the memory usage of array_data and its children recursively.

get_max_alignment_for_schema 🔒

get_record_batch_memory_size

Calculate total used memory of this batch.

spill_record_batch_by_sizeDeprecated

Spill the RecordBatch to disk as smaller batches split by batch_size_rows

Type Aliases

NextRecordBatchResult 🔒

When we poll for the next batch, we will get back both the batch and the reader, so we can call next again.