Module parquet 

Reexports the datafusion_datasource_parquet crate, containing Parquet based FileSource.

Modules

access_plan

file_format

ParquetFormat: Parquet FileFormat abstractions

metadata

DFParquetMetadata for fetching Parquet file metadata, statistics and schema information.

source

ParquetSource implementation for reading parquet files

Structs

CachedParquetFileReader

Implements [AsyncFileReader] for a Parquet file in object storage. Reads the file metadata from the FileMetadataCache, if available, otherwise reads it directly from the file and then updates the cache.

CachedParquetFileReaderFactory

Implementation of ParquetFileReaderFactory supporting the caching of footer and page metadata. Reads and updates the FileMetadataCache with the [ParquetMetaData] data. This reader always loads the entire metadata (including page index, unless the file is encrypted), even if not required by the current query, to ensure it is always available for those that need it.

CachedParquetMetaData

Wrapper to implement FileMetadata for [ParquetMetaData].

DefaultParquetFileReaderFactory

Default implementation of ParquetFileReaderFactory

ObjectStoreFetch

[MetadataFetch] adapter for reading bytes from an [ObjectStore]

PagePruningAccessPlanFilter

Filters a ParquetAccessPlan based on the Parquet PageIndex, if present

ParquetAccessPlan

A selection of rows and row groups within a ParquetFile to decode.

ParquetFileMetrics

Stores metrics about the parquet execution for a particular parquet file.

ParquetFileReader

Implements [AsyncFileReader] for a parquet file in object storage.

ParquetFormat

The Apache Parquet FileFormat implementation

ParquetFormatFactory

Factory struct used to create ParquetFormat

ParquetSink

Implements DataSink for writing to a parquet file.

RowGroupAccessPlanFilter

Reduces the ParquetAccessPlan based on row group level metadata.

Enums

RowGroupAccess

Describes how the parquet reader will access a row group

Traits

ParquetFileReaderFactory

Interface for reading parquet files.

Functions

apply_file_schema_type_coercions

Apply necessary schema type coercions to make file schema match table schema.

build_row_filter

Build a [RowFilter] from the given predicate Expr if possible

can_expr_be_pushed_down_with_schemas

Recurses through expr as a tree, finds all columns, and checks if any of them would prevent this expression from being predicate pushed down. If any of them would, this returns false. Otherwise, true. Note that the schema passed in here is not the physical file schema (as it is not available at that point in time); it is the schema of the table that this expression is being evaluated against minus any projected columns and partition columns.

coerce_file_schema_to_string_typeDeprecated

If the table schema uses a string type, coerce the file schema to use a string type.

coerce_file_schema_to_view_typeDeprecated

Coerces the file schema if the table schema uses a view type.

coerce_int96_to_resolution

Coerces the file schema’s Timestamps to the provided TimeUnit if Parquet schema contains INT96.

fetch_parquet_metadataDeprecated

Fetches parquet metadata from ObjectStore for given object

fetch_statisticsDeprecated

Read and parse the statistics of the Parquet file at location path

plan_to_parquet

Executes a query and writes the results to a partitioned Parquet file.

statistics_from_parquet_meta_calcDeprecated

transform_binary_to_string

Transform a schema so that any binary types are strings

transform_schema_to_view

Transform a schema to use view types for Utf8 and Binary