Struct StreamDecoder 

pub struct StreamDecoder {
    schema: Option<Arc<Schema>>,
    dictionaries: HashMap<i64, Arc<dyn Array>>,
    state: DecoderState,
    buf: MutableBuffer,
    require_alignment: bool,
    skip_validation: UnsafeFlag,
}

A low-level interface for reading RecordBatch data from a stream of bytes

See StreamReader for a higher-level interface

Fields

schema: Option<Arc<Schema>>

dictionaries: HashMap<i64, Arc<dyn Array>>

state: DecoderState

buf: MutableBuffer

require_alignment: bool

skip_validation: UnsafeFlag

Implementations

impl StreamDecoder

pub fn new() -> StreamDecoder

Create a new StreamDecoder

pub fn with_require_alignment(self, require_alignment: bool) -> StreamDecoder

Specifies whether or not array data in input buffers is required to be properly aligned.

If require_alignment is true, this decoder will return an error if any array data in the input buf is not properly aligned. Under the hood it will use arrow_data::ArrayDataBuilder::build to construct arrow_data::ArrayData.

If require_alignment is false (the default), this decoder will automatically allocate a new aligned buffer and copy over the data if any array data in the input buf is not properly aligned. (Properly aligned array data will remain zero-copy.) Under the hood it will use arrow_data::ArrayDataBuilder::build_aligned to construct arrow_data::ArrayData.

pub fn schema(&self) -> Option<Arc<Schema>>

Return the schema if decoded, else None.

pub fn decode( &mut self, buffer: &mut Buffer, ) -> Result<Option<RecordBatch>, ArrowError>

Try to read the next RecordBatch from the provided Buffer

Buffer::advance will be called on buffer for any consumed bytes.

The push-based interface facilitates integration with sources that yield arbitrarily delimited bytes ranges, such as a chunked byte stream received from object storage

fn print_stream<I>(src: impl Iterator<Item = Buffer>) -> Result<(), ArrowError> {
    let mut decoder = StreamDecoder::new();
    for mut x in src {
        while !x.is_empty() {
            if let Some(x) = decoder.decode(&mut x)? {
                println!("{x:?}");
            }
            if let Some(schema) = decoder.schema() {
                println!("Schema: {schema:?}");
            }
        }
    }
    decoder.finish().unwrap();
    Ok(())
}

pub fn finish(&mut self) -> Result<(), ArrowError>

Signal the end of stream

Returns an error if any partial data remains in the stream

Trait Implementations

impl Debug for StreamDecoder

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for StreamDecoder

fn default() -> StreamDecoder

Returns the “default value” for a type.