Module file_format 

ArrowFormat: Apache Arrow FileFormat abstractions

Works with files following the Arrow IPC format

Structs

ArrowFileSink 🔒

Implements FileSink for writing to arrow_ipc files

ArrowFormat

Arrow FileFormat implementation.

ArrowFormatFactory

Factory struct used to create ArrowFormat

Constants

ARROW_MAGIC 🔒

BUFFER_FLUSH_BYTES 🔒

If the buffered Arrow data exceeds this size, it is flushed to object store

CONTINUATION_MARKER 🔒

INITIAL_BUFFER_BYTES 🔒

Initial writing buffer size. Note this is just a size hint for efficiency. It will grow beyond the set value if needed.

Functions

collect_at_least_n_bytes 🔒

infer_schema_from_file_stream 🔒

Custom implementation of inferring schema. Should eventually be moved upstream to arrow-rs. See https://github.com/apache/arrow-rs/issues/5021