serialize_rb_stream_to_object_store

datafusion_datasource::write::orchestration

Function serialize_rb_stream_to_object_store 

Source 
pub(crate) async fn serialize_rb_stream_to_object_store(
    data_rx: Receiver<RecordBatch>,
    serializer: Arc<dyn BatchSerializer>,
    writer: Box<dyn AsyncWrite + Send + Unpin>,
) -> SerializedRecordBatchResult
Expand description

Serializes a single data stream in parallel and writes to an ObjectStore concurrently. Data order is preserved.

In the event of a non-IO error which does not involve the ObjectStore writer, the writer returned to the caller in addition to the error, so that failed writes may be aborted.

In the event of an IO error involving the ObjectStore writer, the writer is dropped to avoid calling further methods on it which might panic.