polars.scan_arrow_c_stream#

polars.scan_arrow_c_stream(source: Any) → LazyFrame[source]#

Scan a source that implements the Arrow PyCapsule Interface.

This creates a lazy scan over an object that exposes a __arrow_c_stream__ method (e.g. a pyarrow.RecordBatchReader, or a nanoarrow stream). The source is consumed, batch by batch, during query execution.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

Parameters:

source: Any object implementing the Arrow PyCapsule Interface (i.e. it has a __arrow_c_stream__ method).

Returns:

LazyFrame

Notes

The source must produce Arrow RecordBatches with a struct-typed schema. See: https://arrow.apache.org/docs/format/CStreamInterface.html.
Once the stream is consumed, the same source cannot be scanned again.

Examples

Scan from a PyArrow RecordBatchReader:

>>> import pyarrow as pa
>>> schema = pa.schema([("a", pa.int64()), ("b", pa.string())])
>>> batches = [
...     pa.record_batch([[1, 2, 3], ["x", "y", "z"]], schema=schema),
...     pa.record_batch([[4, 5], ["a", "b"]], schema=schema),
... ]
>>> reader = pa.RecordBatchReader.from_batches(schema, batches)
>>> pl.scan_arrow_c_stream(reader).collect()
shape: (5, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1   ┆ x   │
│ 2   ┆ y   │
│ 3   ┆ z   │
│ 4   ┆ a   │
│ 5   ┆ b   │
└─────┴─────┘