Enum ByteSourceReader 

pub enum ByteSourceReader<R>
where
    R: BufRead,
{
    UncompressedMemory {
        slice: Buffer<u8>,
        offset: usize,
    },
    UncompressedStream(R),
    Gzip(MultiGzDecoder<R>),
    Zlib(ZlibDecoder<R>),
    Zstd(Decoder<'static, R>),
}

Available on crate feature polars-io only.

A byte source that abstracts over in-memory buffers and streaming readers, with optional transparent decompression and buffering.

Implements BufRead, allowing uniform access regardless of whether the underlying data is an in-memory slice, a raw stream, or a compressed stream (gzip/zlib/zstd).

This is the generic successor to CompressedReader, which only supports in-memory (Buffer<u8>) sources.

Variants

UncompressedMemory

Fields

slice: Buffer<u8>

offset: usize

UncompressedStream(R)

Gzip(MultiGzDecoder<R>)

Available on crate feature decompress only.

Zlib(ZlibDecoder<R>)

Available on crate feature decompress only.

Zstd(Decoder<'static, R>)

Available on crate feature decompress only.

Implementations

impl<R> ByteSourceReader<R>

where R: BufRead,

pub fn try_new( reader: R, compression: Option<SupportedCompression>, ) -> Result<ByteSourceReader<R>, PolarsError>

pub fn is_compressed(&self) -> bool

pub fn compression(&self) -> Option<SupportedCompression>

pub const fn initial_read_size() -> usize

pub const fn ideal_read_size() -> usize

pub fn read_next_slice( &mut self, prev_leftover: &Buffer<u8>, read_size: usize, uncompressed_size_hint: Option<usize>, ) -> Result<(Buffer<u8>, usize), Error>

Reads exactly read_size bytes if possible from the internal readers and creates a new [Buffer] with the content concat(prev_leftover, new_bytes).

Returns the new slice and the number of bytes read, which will be 0 when eof is reached and this function is called again.

If the underlying reader is uncompressed the operation is a cheap zero-copy [Buffer::sliced] operation.

By handling slice concatenation at this level we can implement zero-copy reading and make the interface easier to use.

It’s a logic bug if prev_leftover is neither empty nor the last slice returned by this function.

impl ByteSourceReader<ReaderSource>

pub fn from_memory( slice: Buffer<u8>, ) -> Result<ByteSourceReader<ReaderSource>, PolarsError>