Crate datafusion_pruning 

Modules

file_pruner 🔒

File-level pruning based on partition values and file-level statistics

pruning_predicate 🔒

PruningPredicate to apply filter Expr to prune “containers” based on statistics (e.g. Parquet Row Groups)

Structs

FilePruner

Prune based on partition values and file-level statistics.

PredicateRewriter

Rewrite a predicate expression in terms of statistics (min/max/null_counts) for use as a PruningPredicate.

PruningPredicate

Used to prove that arbitrary predicates (boolean expression) can not possibly evaluate to true given information about a column provided by PruningStatistics.

RequiredColumns

Describes which columns statistics are necessary to evaluate a PruningPredicate.

Traits

PruningStatistics

A source of runtime statistical information to PruningPredicates.

UnhandledPredicateHook

Rewrites predicates that PredicateRewriter can not handle, e.g. certain complex expressions or predicates that reference columns that are not in the schema.

Functions

build_pruning_predicate

Build a pruning predicate from an optional predicate expression. If the predicate is None or the predicate cannot be converted to a pruning predicate, return None. If there is an error creating the pruning predicate it is recorded by incrementing the predicate_creation_errors counter.