Expand description
FileScanConfig to configure scanning of possibly partitioned
file sources.
Structs§
- File
Scan Config - The base configurations for a
DataSourceExec, the a physical plan for any given file format. - File
Scan Config Builder - A builder for
FileScanConfig’s. - Partition
Column Projector - A helper that projects partition columns into the file record batches.
- Zero
Buffer 🔒Generator - Generate a arrow [
Buffer] that contains zero values. - Zero
Buffer 🔒Generators
Functions§
- create_
dict_ 🔒array - create_
output_ 🔒array - get_
projected_ 🔒output_ ordering - The various listing tables does not attempt to read all files concurrently, instead they will read files in sequence within a partition. This is an important property as it allows plans to run against 1000s of files and not try to open them all concurrently.
- wrap_
partition_ type_ in_ dict - Convert type to a type suitable for use as a
ListingTablepartition column. ReturnsDictionary(UInt16, val_type), which is a reasonable trade off between a reasonable number of partition values and space efficiency. - wrap_
partition_ value_ in_ dict - Convert a
ScalarValueof partition columns to a type, as described in the documentation ofwrap_partition_type_in_dict, which can wrap the types.