Struct ListingTableConfig 

pub struct ListingTableConfig {
    pub table_paths: Vec<ListingTableUrl>,
    pub file_schema: Option<Arc<Schema>>,
    pub options: Option<ListingOptions>,
    pub(crate) schema_source: SchemaSource,
    pub(crate) schema_adapter_factory: Option<Arc<dyn SchemaAdapterFactory>>,
    pub(crate) expr_adapter_factory: Option<Arc<dyn PhysicalExprAdapterFactory>>,
}

Configuration for creating a crate::ListingTable

Schema Evolution Support

This configuration supports schema evolution through the optional SchemaAdapterFactory. You might want to override the default factory when you need:

• Type coercion requirements: When you need custom logic for converting between different Arrow data types (e.g., Int32 ↔ Int64, Utf8 ↔ LargeUtf8)
• Column mapping: You need to map columns with a legacy name to a new name
• Custom handling of missing columns: By default they are filled in with nulls, but you may e.g. want to fill them in with 0 or "".

If not specified, a datafusion_datasource::schema_adapter::DefaultSchemaAdapterFactory will be used, which handles basic schema compatibility cases.

Fields

table_paths: Vec<ListingTableUrl>

Paths on the ObjectStore for creating crate::ListingTable. They should share the same schema and object store.

file_schema: Option<Arc<Schema>>

Optional SchemaRef for the to be created crate::ListingTable.

See details on ListingTableConfig::with_schema

options: Option<ListingOptions>

Optional ListingOptions for the to be created crate::ListingTable.

See details on ListingTableConfig::with_listing_options

schema_source: SchemaSource

schema_adapter_factory: Option<Arc<dyn SchemaAdapterFactory>>

expr_adapter_factory: Option<Arc<dyn PhysicalExprAdapterFactory>>

Implementations

impl ListingTableConfig

pub fn new(table_path: ListingTableUrl) -> ListingTableConfig

Creates new ListingTableConfig for reading the specified URL

pub fn new_with_multi_paths( table_paths: Vec<ListingTableUrl>, ) -> ListingTableConfig

Creates new ListingTableConfig with multiple table paths.

See ListingTableConfigExt::infer_options for details on what happens with multiple paths

pub fn schema_source(&self) -> SchemaSource

Returns the source of the schema for this configuration

pub fn with_schema(self, schema: Arc<Schema>) -> ListingTableConfig

Set the schema for the overall crate::ListingTable

crate::ListingTable will automatically coerce, when possible, the schema for individual files to match this schema.

If a schema is not provided, it is inferred using Self::infer_schema.

If the schema is provided, it must contain only the fields in the file without the table partitioning columns.

Example: Specifying Table Schema

let schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("name", DataType::Utf8, true),
]));

let config = ListingTableConfig::new(table_paths)
    .with_listing_options(listing_options)  // Set options first
    .with_schema(schema);                    // Then set schema

pub fn with_listing_options( self, listing_options: ListingOptions, ) -> ListingTableConfig

Add listing_options to ListingTableConfig

If not provided, format and other options are inferred via ListingTableConfigExt::infer_options.

Example: Configuring Parquet Files with Custom Options

let options = ListingOptions::new(Arc::new(ParquetFormat::default()))
    .with_file_extension(".parquet")
    .with_collect_stat(true);

let config = ListingTableConfig::new(table_paths).with_listing_options(options);
// Configure file format and options

pub fn infer_file_extension_and_compression_type( path: &str, ) -> Result<(String, Option<String>), DataFusionError>

Returns a tuple of (file_extension, optional compression_extension)

For example a path ending with blah.test.csv.gz returns ("csv", Some("gz")) For example a path ending with blah.test.csv returns ("csv", None)

pub async fn infer_schema( self, state: &dyn Session, ) -> Result<ListingTableConfig, DataFusionError>

Infer the SchemaRef based on table_paths.

This method infers the table schema using the first table_path. See ListingOptions::infer_schema for more details

Errors

• if self.options is not set. See Self::with_listing_options

pub async fn infer_partitions_from_path( self, state: &dyn Session, ) -> Result<ListingTableConfig, DataFusionError>

Infer the partition columns from table_paths.

Errors

• if self.options is not set. See Self::with_listing_options

pub fn with_schema_adapter_factory( self, schema_adapter_factory: Arc<dyn SchemaAdapterFactory>, ) -> ListingTableConfig

Set the SchemaAdapterFactory for the crate::ListingTable

The schema adapter factory is used to create schema adapters that can handle schema evolution and type conversions when reading files with different schemas than the table schema.

If not provided, a default schema adapter factory will be used.

Example: Custom Schema Adapter for Type Coercion

let config = ListingTableConfig::new(table_paths)
    .with_listing_options(listing_options)
    .with_schema(table_schema)
    .with_schema_adapter_factory(Arc::new(MySchemaAdapterFactory));

pub fn schema_adapter_factory(&self) -> Option<&Arc<dyn SchemaAdapterFactory>>

Get the SchemaAdapterFactory for this configuration

pub fn with_expr_adapter_factory( self, expr_adapter_factory: Arc<dyn PhysicalExprAdapterFactory>, ) -> ListingTableConfig

Set the PhysicalExprAdapterFactory for the crate::ListingTable

The expression adapter factory is used to create physical expression adapters that can handle schema evolution and type conversions when evaluating expressions with different schemas than the table schema.

If not provided, a default physical expression adapter factory will be used unless a custom SchemaAdapterFactory is set, in which case only the SchemaAdapterFactory will be used.

See https://github.com/apache/datafusion/issues/16800 for details on this transition.

Trait Implementations

impl Clone for ListingTableConfig

fn clone(&self) -> ListingTableConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ListingTableConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for ListingTableConfig

fn default() -> ListingTableConfig

Returns the “default value” for a type.

impl ListingTableConfigExt for ListingTableConfig

fn infer_options<'life0, 'async_trait>( self, state: &'life0 dyn Session, ) -> Pin<Box<dyn Future<Output = Result<ListingTableConfig>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Infer ListingOptions based on table_path and file suffix.

fn infer<'life0, 'async_trait>( self, state: &'life0 dyn Session, ) -> Pin<Box<dyn Future<Output = Result<Self>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Convenience method to call both Self::infer_options and ListingTableConfig::infer_schema