polars.StreamingEngine#

class polars.StreamingEngine(*, monitoring: bool | None = None)[source]#

Execute queries in a streaming fashion, supporting larger-than-memory data.

Parameters:
monitoringbool, default None

Whether to monitor queries run by this engine. None uses the setting from Config.enable_monitoring(); True or False overrides it. Setting this to True requires the polars-cloud package.

Examples

Monitor a single query, regardless of the configured default:

>>> lf.collect(engine=pl.StreamingEngine(monitoring=True))  
__init__(*, monitoring: bool | None = None) → None[source]#

Methods

__init__(*[, monitoring])

collect(lf, *, optimizations[, background, ...])

Execute lf, returning a DataFrame or a background query handle.

collect_all(lfs, *, optimizations)

Execute several queries, potentially in parallel.

collect_all_async(lfs, *, optimizations[, ...])

Execute several queries asynchronously.

collect_async(lf, *, optimizations[, gevent])

Execute lf asynchronously.

collect_batches(lf, *, optimizations[, ...])

Execute lf, yielding its result in batches.

execute(lf, *, optimizations)

Execute the query into a QueryResult.

sink_batches(lf, function, *, chunk_size, ...)

See polars.LazyFrame.sink_batches().

sink_csv(lf, path, *, include_bom, ...)

See polars.LazyFrame.sink_csv().

sink_ipc(lf, path, *, compression, ...)

See polars.LazyFrame.sink_ipc().

sink_ndjson(lf, path, *, compression, ...)

See polars.LazyFrame.sink_ndjson().

sink_parquet(lf, path, *, compression, ...)

See polars.LazyFrame.sink_parquet().

Attributes

monitoring

Whether queries are monitored (None uses the configured default).

name

Name of the engine.

plan_engine

Engine name used to render query plans.