Struct FetchConfig 

pub struct FetchConfig {
    pub chunk_size: usize,
    pub strategy: ConcurrencyStrategy,
}

Fields

chunk_size: usize

strategy: ConcurrencyStrategy

Implementations

impl FetchConfig

pub fn random_access() -> Self

Use for cloud-based file formats that are randomly accessible, i.e. individual row groups (or record batches) and/or individual columns can be fetched directly using the metadata as an input. Example: Parquet, IPC with internal extensions.

The chunk_size should be small enough to enable smooth operation of the bytes-based in-flight concurrency controller.

pub fn streaming() -> Self

Use for cloud-based file formats that have a sequential layout, i.e. the file bytes must be fetched and parsed sequentially. The pipeline is responsible for managing back-pressure and rate-limiting. Example: CSV.

pub fn legacy() -> Self

Used for legacy fetch. @TODO: Deprecate over time.

Trait Implementations

impl Clone for FetchConfig

fn clone(&self) -> FetchConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FetchConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Copy for FetchConfig