Struct CsvFormat 

pub struct CsvFormat {
    options: CsvOptions,
}

Character Separated Value FileFormat implementation.

Fields

options: CsvOptions

Implementations

impl CsvFormat

pub async fn read_to_delimited_chunks_from_stream<'a>( &self, stream: Pin<Box<dyn Stream<Item = Result<Bytes, DataFusionError>> + Send + 'a>>, ) -> Pin<Box<dyn Stream<Item = Result<Bytes, DataFusionError>> + Send + 'a>>

Convert a stream of bytes into a stream of of [Bytes] containing newline delimited CSV records, while accounting for \ and ".

pub fn with_options(self, options: CsvOptions) -> CsvFormat

Set the csv options

pub fn options(&self) -> &CsvOptions

Retrieve the csv options

pub fn with_schema_infer_max_rec(self, max_rec: usize) -> CsvFormat

Set a limit in terms of records to scan to infer the schema

• default to DEFAULT_SCHEMA_INFER_MAX_RECORD

pub fn with_has_header(self, has_header: bool) -> CsvFormat

Set true to indicate that the first line is a header.

• default to true

pub fn with_truncated_rows(self, truncated_rows: bool) -> CsvFormat

pub fn with_null_regex(self, null_regex: Option<String>) -> CsvFormat

Set the regex to use for null values in the CSV reader.

• default to treat empty values as null.

pub fn has_header(&self) -> Option<bool>

Returns Some(true) if the first line is a header, Some(false) if it is not, and None if it is not specified.

pub fn with_comment(self, comment: Option<u8>) -> CsvFormat

Lines beginning with this byte are ignored.

pub fn with_delimiter(self, delimiter: u8) -> CsvFormat

The character separating values within a row.

• default to ‘,’

pub fn with_quote(self, quote: u8) -> CsvFormat

The quote character in a row.

• default to ‘“’

pub fn with_escape(self, escape: Option<u8>) -> CsvFormat

The escape character in a row.

• default is None

pub fn with_terminator(self, terminator: Option<u8>) -> CsvFormat

The character used to indicate the end of a row.

• default to None (CRLF)

pub fn with_newlines_in_values(self, newlines_in_values: bool) -> CsvFormat

Specifies whether newlines in (quoted) values are supported.

Parsing newlines in quoted values may be affected by execution behaviour such as parallel file scanning. Setting this to true ensures that newlines in values are parsed successfully, which may reduce performance.

The default behaviour depends on the datafusion.catalog.newlines_in_values setting.

pub fn with_file_compression_type( self, file_compression_type: FileCompressionType, ) -> CsvFormat

Set a FileCompressionType of CSV

• defaults to FileCompressionType::UNCOMPRESSED

pub fn with_truncate_rows(self, truncate_rows: bool) -> CsvFormat

Set whether rows should be truncated to the column width

• defaults to false

pub fn delimiter(&self) -> u8

The delimiter character.

pub fn quote(&self) -> u8

The quote character.

pub fn escape(&self) -> Option<u8>

The escape character.

impl CsvFormat

pub async fn infer_schema_from_stream( &self, state: &dyn Session, records_to_read: usize, stream: impl Stream<Item = Result<Bytes, DataFusionError>>, ) -> Result<(Schema, usize), DataFusionError>

Return the inferred schema reading up to records_to_read from a stream of delimited chunks returning the inferred schema and the number of lines that were read.

This method can handle CSV files with different numbers of columns. The inferred schema will be the union of all columns found across all files. Files with fewer columns will have missing columns filled with null values.

Example

If you have two CSV files:

• file1.csv: col1,col2,col3
• file2.csv: col1,col2,col3,col4,col5

The inferred schema will contain all 5 columns, with files that don’t have columns 4 and 5 having null values for those columns.

Trait Implementations

impl Debug for CsvFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for CsvFormat

fn default() -> CsvFormat

Returns the “default value” for a type.

impl FileFormat for CsvFormat

fn as_any(&self) -> &(dyn Any + 'static)

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn get_ext(&self) -> String

Returns the extension for this FileFormat, e.g. “file.csv” -> csv

fn get_ext_with_compression( &self, file_compression_type: &FileCompressionType, ) -> Result<String, DataFusionError>

Returns the extension for this FileFormat when compressed, e.g. “file.csv.gz” -> csv

fn compression_type(&self) -> Option<FileCompressionType>

Returns whether this instance uses compression if applicable

fn infer_schema<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 dyn Session, store: &'life2 Arc<dyn ObjectStore>, objects: &'life3 [ObjectMeta], ) -> Pin<Box<dyn Future<Output = Result<Arc<Schema>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, CsvFormat: 'async_trait,

Infer the common schema of the provided objects. The objects will usually be analysed up to a given number of records or files (as specified in the format config) then give the estimated common schema. This might fail if the files have schemas that cannot be merged.

fn infer_stats<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _store: &'life2 Arc<dyn ObjectStore>, table_schema: Arc<Schema>, _object: &'life3 ObjectMeta, ) -> Pin<Box<dyn Future<Output = Result<Statistics, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, CsvFormat: 'async_trait,

Infer the statistics for the provided object. The cost and accuracy of the estimated statistics might vary greatly between file formats.

fn create_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, state: &'life1 dyn Session, conf: FileScanConfig, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, CsvFormat: 'async_trait,

Take a list of files and convert it to the appropriate executor according to this file format.

fn create_writer_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, input: Arc<dyn ExecutionPlan>, state: &'life1 dyn Session, conf: FileSinkConfig, order_requirements: Option<LexRequirement>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, CsvFormat: 'async_trait,

Take a list of files and the configuration to convert it to the appropriate writer executor according to this file format.

fn file_source(&self) -> Arc<dyn FileSource>

Return the related FileSource such as CsvSource, JsonSource, etc.