Module orchestration 

Module containing helper methods/traits related to orchestrating file serialization, streaming to object store, parallelization, and abort handling

Enums

SerializedRecordBatchResult 🔒

Result of calling serialize_rb_stream_to_object_store

Functions

serialize_rb_stream_to_object_store 🔒

Serializes a single data stream in parallel and writes to an ObjectStore concurrently. Data order is preserved.

spawn_writer_tasks_and_join

Orchestrates multipart put of a dynamic number of output files from a single input stream for any statelessly serialized file type. That is, any file type for which each [RecordBatch] can be serialized independently of all other [RecordBatch]s.

stateless_serialize_and_write_files 🔒

Contains the common logic for serializing RecordBatches and writing the resulting bytes to an ObjectStore. Serialization is assumed to be stateless, i.e. each RecordBatch can be serialized without any dependency on the RecordBatches before or after.

Type Aliases

FileWriteBundle 🔒

SerializerType 🔒

WriterType 🔒