Struct ArrowFileSink 

struct ArrowFileSink {
    config: FileSinkConfig,
}

Implements FileSink for writing to arrow_ipc files

Fields

config: FileSinkConfig

Implementations

impl ArrowFileSink

fn new(config: FileSinkConfig) -> Self

Trait Implementations

impl DataSink for ArrowFileSink

fn as_any(&self) -> &dyn Any

Returns the data sink as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> &SchemaRef

Returns the sink schema

fn write_all<'life0, 'life1, 'async_trait>( &'life0 self, data: SendableRecordBatchStream, context: &'life1 Arc<TaskContext>, ) -> Pin<Box<dyn Future<Output = Result<u64>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Writes the data to the sink, returns the number of values written

fn metrics(&self) -> Option<MetricsSet>

Return a snapshot of the MetricsSet for this DataSink.

impl Debug for ArrowFileSink

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DisplayAs for ArrowFileSink

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format according to DisplayFormatType, used when verbose representation looks different from the default one

impl FileSink for ArrowFileSink

fn config(&self) -> &FileSinkConfig

Retrieves the file sink configuration.

fn spawn_writer_tasks_and_join<'life0, 'life1, 'async_trait>( &'life0 self, context: &'life1 Arc<TaskContext>, demux_task: SpawnedTask<Result<()>>, file_stream_rx: DemuxedStreamReceiver, object_store: Arc<dyn ObjectStore>, ) -> Pin<Box<dyn Future<Output = Result<u64>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Spawns writer tasks and joins them to perform file writing operations. Is a critical part of FileSink trait, since it’s the very last step for write_all.

fn write_all<'life0, 'life1, 'async_trait>( &'life0 self, data: Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>, context: &'life1 Arc<TaskContext>, ) -> Pin<Box<dyn Future<Output = Result<u64, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: Sync + 'async_trait,

File sink implementation of the DataSink::write_all method.