polars.collect_all#

polars.collect_all(
lazy_frames: Iterable[LazyFrame],
*,
optimizations: QueryOptFlags = (),
engine: EngineType = 'auto',
lazy: bool = False,
) → list[DataFrame] | LazyFrame[source]#

Collect multiple LazyFrames at the same time.

This can run all the computation graphs in parallel or combined.

Common Subplan Elimination is applied on the combined plan, meaning that diverging queries will run only once.

engine:In MemoryStreamingDistributed
Parameters:
lazy_frames

A list of LazyFrames to collect.

optimizations

The optimization passes done during query optimization.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

engine

Select the engine used to process the query (default "auto"). A Engine instance may also be passed. Supported engine names are:

  • "auto": use the engine set by Config.set_engine_affinity or the POLARS_ENGINE_AFFINITY environment variable, falling back to "streaming" if unset.

  • "in-memory": use the in-memory engine.

  • "streaming": use the streaming engine, which processes queries in batches, reducing memory pressure and often outperforming the in-memory engine.

  • "gpu": use the CUDA GPU engine (requires an Nvidia GPU and cudf-polars). Pass a GPUEngine object for fine-grained control (e.g. device selection on multi-GPU systems).

If the selected engine cannot run the query, Polars falls back to the in-memory engine.

Note

The GPU engine does not support async, or running in the background. If either are enabled, then GPU execution is switched off.

lazy:

Return as LazyFrame that can be collected later. This is only correct if all inputs sink to disk.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

Returns:
list of DataFrames

The collected DataFrames, returned in the same order as the input LazyFrames.

On this page