Struct DFParquetMetadata 

pub struct DFParquetMetadata<'a> {
    store: &'a (dyn ObjectStore + 'static),
    object_meta: &'a ObjectMeta,
    metadata_size_hint: Option<usize>,
    decryption_properties: Option<Arc<FileDecryptionProperties>>,
    file_metadata_cache: Option<Arc<dyn FileMetadataCache<Extra = ObjectMeta>>>,
    pub coerce_int96: Option<TimeUnit>,
}

Handles fetching Parquet file schema, metadata and statistics from object store.

This component is exposed for low level integrations through ParquetFileReaderFactory.

Fields

store: &'a (dyn ObjectStore + 'static)

object_meta: &'a ObjectMeta

metadata_size_hint: Option<usize>

decryption_properties: Option<Arc<FileDecryptionProperties>>

file_metadata_cache: Option<Arc<dyn FileMetadataCache<Extra = ObjectMeta>>>

coerce_int96: Option<TimeUnit>

timeunit to coerce INT96 timestamps to

Implementations

impl<'a> DFParquetMetadata<'a>

pub fn new( store: &'a (dyn ObjectStore + 'static), object_meta: &'a ObjectMeta, ) -> DFParquetMetadata<'a>

pub fn with_metadata_size_hint( self, metadata_size_hint: Option<usize>, ) -> DFParquetMetadata<'a>

set metadata size hint

pub fn with_decryption_properties( self, decryption_properties: Option<Arc<FileDecryptionProperties>>, ) -> DFParquetMetadata<'a>

set decryption properties

pub fn with_file_metadata_cache( self, file_metadata_cache: Option<Arc<dyn FileMetadataCache<Extra = ObjectMeta>>>, ) -> DFParquetMetadata<'a>

set file metadata cache

pub fn with_coerce_int96( self, time_unit: Option<TimeUnit>, ) -> DFParquetMetadata<'a>

Set timeunit to coerce INT96 timestamps to

pub async fn fetch_metadata( &self, ) -> Result<Arc<ParquetMetaData>, DataFusionError>

Fetch parquet metadata from the remote object store

pub async fn fetch_schema(&self) -> Result<Schema, DataFusionError>

Read and parse the schema of the Parquet file

pub async fn fetch_statistics( &self, table_schema: &Arc<Schema>, ) -> Result<Statistics, DataFusionError>

Fetch the metadata from the Parquet file via Self::fetch_metadata and convert the statistics in the metadata using Self::statistics_from_parquet_metadata

pub fn statistics_from_parquet_metadata( metadata: &ParquetMetaData, table_schema: &Arc<Schema>, ) -> Result<Statistics, DataFusionError>

Convert statistics in [ParquetMetaData] into Statistics using [StatisticsConverter]

The statistics are calculated for each column in the table schema using the row group statistics in the parquet metadata.

Key behaviors:

1. Extracts row counts and byte sizes from all row groups
2. Applies schema type coercions to align file schema with table schema
3. Collects and aggregates statistics across row groups when available

When there are no statistics:

If the Parquet file doesn’t contain any statistics (has_statistics is false), the function returns a Statistics object with:

• Exact row count
• Exact byte size
• All column statistics marked as unknown via Statistics::unknown_column(&table_schema)

When only some columns have statistics:

For columns with statistics:

• Min/max values are properly extracted and represented as Precision::Exact
• Null counts are calculated by summing across row groups

For columns without statistics,

• For min/max, there are two situations:
  1. The column isn’t in arrow schema, then min/max values are set to Precision::Absent
  2. The column is in arrow schema, but not in parquet schema due to schema revolution, min/max values are set to Precision::Exact(null)
• Null counts are set to Precision::Exact(num_rows) (conservatively assuming all values could be null)

Trait Implementations

impl<'a> Debug for DFParquetMetadata<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.