Struct FileStream 

pub struct FileStream {
    file_iter: VecDeque<PartitionedFile>,
    projected_schema: SchemaRef,
    remain: Option<usize>,
    file_opener: Arc<dyn FileOpener>,
    pc_projector: PartitionColumnProjector,
    state: FileStreamState,
    file_stream_metrics: FileStreamMetrics,
    baseline_metrics: BaselineMetrics,
    on_error: OnError,
}

A stream that iterates record batch by record batch, file over file.

Fields

file_iter: VecDeque<PartitionedFile>

An iterator over input files.

projected_schema: SchemaRef

The stream schema (file schema including partition columns and after projection).

remain: Option<usize>

The remaining number of records to parse, None if no limit

file_opener: Arc<dyn FileOpener>

A dynamic FileOpener. Calling open() returns a FileOpenFuture, which can be resolved to a stream of RecordBatch.

pc_projector: PartitionColumnProjector

The partition column projector

state: FileStreamState

The stream state

file_stream_metrics: FileStreamMetrics

File stream specific metrics

baseline_metrics: BaselineMetrics

runtime baseline metrics

on_error: OnError

Describes the behavior of the FileStream if file opening or scanning fails

Implementations

impl FileStream

pub fn new( config: &FileScanConfig, partition: usize, file_opener: Arc<dyn FileOpener>, metrics: &ExecutionPlanMetricsSet, ) -> Result<Self>

Create a new FileStream using the give FileOpener to scan underlying files

pub fn with_on_error(self, on_error: OnError) -> Self

Specify the behavior when an error occurs opening or scanning a file

If OnError::Skip the stream will skip files which encounter an error and continue If OnError:Fail (default) the stream will fail and stop processing when an error occurs

fn start_next_file( &mut self, ) -> Option<Result<(FileOpenFuture, Vec<ScalarValue>)>>

Begin opening the next file in parallel while decoding the current file in FileStream.

Since file opening is mostly IO (and may involve a bunch of sequential IO), it can be parallelized with decoding.

fn poll_inner( &mut self, cx: &mut Context<'_>, ) -> Poll<Option<Result<RecordBatch>>>

Trait Implementations

impl RecordBatchStream for FileStream

fn schema(&self) -> SchemaRef

Returns the schema of this RecordBatchStream.

impl Stream for FileStream

type Item = Result<RecordBatch, DataFusionError>

Values yielded by the stream.

fn poll_next( self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll<Option<Self::Item>>

Attempt to pull out the next value of this stream, registering the current task for wakeup if the value is not yet available, and returning None if the stream is exhausted.

fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the stream.