Struct CsvOpener 

pub struct CsvOpener {
    config: Arc<CsvSource>,
    file_compression_type: FileCompressionType,
    object_store: Arc<dyn ObjectStore>,
}

A FileOpener that opens a CSV file and yields a FileOpenFuture

Fields

config: Arc<CsvSource>

file_compression_type: FileCompressionType

object_store: Arc<dyn ObjectStore>

Implementations

impl CsvOpener

pub fn new( config: Arc<CsvSource>, file_compression_type: FileCompressionType, object_store: Arc<dyn ObjectStore>, ) -> CsvOpener

Returns a CsvOpener

Trait Implementations

impl FileOpener for CsvOpener

fn open( &self, partitioned_file: PartitionedFile, ) -> Result<Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<RecordBatch, DataFusionError>> + Send>>, DataFusionError>> + Send>>, DataFusionError>

Open a partitioned CSV file.

If file_meta.range is None, the entire file is opened. If file_meta.range is Some(FileRange {start, end}), this signifies that the partition corresponds to the byte range [start, end) within the file.

Note: start or end might be in the middle of some lines. In such cases, the following rules are applied to determine which lines to read:

1. The first line of the partition is the line in which the index of the first character >= start.
2. The last line of the partition is the line in which the byte at position end - 1 resides.

Examples: Consider the following partitions enclosed by braces {}:

{A,1,2,3,4,5,6,7,8,9\n A,1,2,3,4,5,6,7,8,9\n} A,1,2,3,4,5,6,7,8,9\n The lines read would be: [0, 1]

A,{1,2,3,4,5,6,7,8,9\n A,1,2,3,4,5,6,7,8,9\n A},1,2,3,4,5,6,7,8,9\n The lines read would be: [1, 2]