Struct ParquetReadOptions 

pub struct ParquetReadOptions<'a> {
    pub file_extension: &'a str,
    pub table_partition_cols: Vec<(String, DataType)>,
    pub parquet_pruning: Option<bool>,
    pub skip_metadata: Option<bool>,
    pub schema: Option<&'a Schema>,
    pub file_sort_order: Vec<Vec<SortExpr>>,
    pub file_decryption_properties: Option<ConfigFileDecryptionProperties>,
    pub metadata_size_hint: Option<usize>,
}

Options that control the reading of Parquet files.

Note this structure is supplied when a datasource is created and can not not vary from statement to statement. For settings that can vary statement to statement see ConfigOptions.

Fields

file_extension: &'a str

File extension; only files with this extension are selected for data input. Defaults to “.parquet”.

table_partition_cols: Vec<(String, DataType)>

Partition Columns

parquet_pruning: Option<bool>

Should the parquet reader use the predicate to prune row groups? If None, uses value in SessionConfig

skip_metadata: Option<bool>

Should the parquet reader to skip any metadata that may be in the file Schema? This can help avoid schema conflicts due to metadata.

If None specified, uses value in SessionConfig

schema: Option<&'a Schema>

An optional schema representing the parquet files. If None, parquet reader will try to infer it based on data in file.

file_sort_order: Vec<Vec<SortExpr>>

Indicates how the file is sorted

file_decryption_properties: Option<ConfigFileDecryptionProperties>

Properties for decryption of Parquet files that use modular encryption

metadata_size_hint: Option<usize>

Metadata size hint for Parquet files reading (in bytes)

Implementations

impl<'a> ParquetReadOptions<'a>

pub fn new() -> Self

Create a new ParquetReadOptions with default values

pub fn file_extension(self, file_extension: &'a str) -> Self

Specify file_extension

pub fn parquet_pruning(self, parquet_pruning: bool) -> Self

Specify parquet_pruning

pub fn skip_metadata(self, skip_metadata: bool) -> Self

Tell the parquet reader to skip any metadata that may be in the file Schema. This can help avoid schema conflicts due to metadata. Defaults to true.

pub fn schema(self, schema: &'a Schema) -> Self

Specify schema to use for parquet read

pub fn table_partition_cols( self, table_partition_cols: Vec<(String, DataType)>, ) -> Self

Specify table_partition_cols for partition pruning

pub fn file_sort_order(self, file_sort_order: Vec<Vec<SortExpr>>) -> Self

Configure if file has known sort order

pub fn file_decryption_properties( self, file_decryption_properties: ConfigFileDecryptionProperties, ) -> Self

Configure file decryption properties for reading encrypted Parquet files

pub fn metadata_size_hint(self, size_hint: Option<usize>) -> Self

Configure metadata size hint for Parquet files reading (in bytes)

Trait Implementations

impl<'a> Clone for ParquetReadOptions<'a>

fn clone(&self) -> ParquetReadOptions<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for ParquetReadOptions<'_>

fn default() -> Self

Returns the “default value” for a type.

impl ReadOptions<'_> for ParquetReadOptions<'_>

fn to_listing_options( &self, config: &SessionConfig, table_options: TableOptions, ) -> ListingOptions

Helper to convert these user facing options to ListingTable options

fn get_resolved_schema<'life0, 'life1, 'async_trait>( &'life0 self, config: &'life1 SessionConfig, state: SessionState, table_path: ListingTableUrl, ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Infer and resolve the schema from the files/sources provided.

fn _get_resolved_schema<'life0, 'async_trait>( &'a self, config: &'life0 SessionConfig, state: SessionState, table_path: ListingTableUrl, schema: Option<&'a Schema>, ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>

where Self: Sync + 'async_trait, 'a: 'async_trait, 'life0: 'async_trait,

helper function to reduce repetitive code. Infers the schema from sources if not provided. Infinite data sources not supported through this function.