Expand description
Module containing helper methods/traits related to enabling dividing input stream into multiple output files at execution time
Functions§
- compute_
hive_ 🔒style_ file_ path - compute_
partition_ 🔒keys_ by_ row - compute_
take_ 🔒arrays - create_
new_ 🔒file_ stream - Helper for row count demuxer
- generate_
file_ 🔒path - Helper for row count demuxer
- hive_
style_ 🔒partitions_ demuxer - Splits an input stream based on the distinct values of a set of columns Assumes standard hive style partition paths such as /col1=val1/col2=val2/outputfile.parquet
- remove_
partition_ 🔒by_ columns - row_
count_ 🔒demuxer - Dynamically partitions input stream to achieve desired maximum rows per file
- start_
demuxer_ 🔒task - Splits a single [SendableRecordBatchStream] into a dynamically determined number of partitions at execution time.