Version 2.0-rc

Breaking changes

The Lazy API defaults to the streaming engine

LazyFrame.collect and collect_async still default to engine="auto", but "auto" now resolves to the streaming engine for lazy queries (it used to resolve to the in-memory engine). Eager DataFrame operations are unaffected: they continue to resolve "auto" to the in-memory engine internally.

Note that sink_* is not affected by this change: writing to a file was already dispatched to the streaming engine regardless of engine.

explain() and show_graph() are not affected either: they only render a streaming plan when engine="streaming" is passed explicitly (see below).

Danger

This change may silently impact the results of your pipelines. The streaming engine does not guarantee row order for operations that don't require it (unpivot, group_by, joins, ...). If your code relies on incidental ordering, sort explicitly, or pass maintain_order where the operation supports it (e.g. join(..., maintain_order="left")).

Example

Before:

>>> lf = pl.LazyFrame({"a": ["x", "y", "z"], "b": [1, 2, 3], "c": [4, 5, 6]})
>>> lf.unpivot(pl.selectors.numeric(), index="a").collect()
shape: (6, 3)
┌─────┬──────────┬───────┐
│ a   ┆ variable ┆ value │
│ --- ┆ ---      ┆ ---   │
│ str ┆ str      ┆ i64   │
╞═════╪══════════╪═══════╡
│ x   ┆ b        ┆ 1     │
│ y   ┆ b        ┆ 2     │
│ z   ┆ b        ┆ 3     │
│ x   ┆ c        ┆ 4     │
│ y   ┆ c        ┆ 5     │
│ z   ┆ c        ┆ 6     │
└─────┴──────────┴───────┘

After:

>>> lf.unpivot(pl.selectors.numeric(), index="a").collect()
shape: (6, 3)
┌─────┬──────────┬───────┐
│ a   ┆ variable ┆ value │
│ --- ┆ ---      ┆ ---   │
│ str ┆ str      ┆ i64   │
╞═════╪══════════╪═══════╡
│ x   ┆ b        ┆ 1     │
│ x   ┆ c        ┆ 4     │
│ y   ┆ b        ┆ 2     │
│ y   ┆ c        ┆ 5     │
│ z   ┆ b        ┆ 3     │
│ z   ┆ c        ┆ 6     │
└─────┴──────────┴───────┘

The exact row order shown above is not guaranteed — only that it is no longer the pre-2.0 order. Use instead, if you rely on the order:

>>> lf.unpivot(pl.selectors.numeric(), index="a").collect().sort(pl.all())

Joins are affected the same way, and are easy to miss since nothing about the query looks order-sensitive:

>>> left = pl.LazyFrame({"k": [0, 1, 2], "l": ["a", "b", "c"]})
>>> right = pl.LazyFrame({"k": [2, 1, 0], "r": ["x", "y", "z"]})
>>> left.join(right, on="k", how="left").collect()  # row order no longer matches `left`

Use instead, if you rely on the left-hand row order:

>>> left.join(right, on="k", how="left", maintain_order="left").collect()

To restore the previous default engine altogether:

>>> pl.Config.set_engine_affinity("in-memory")  # process-wide
>>> lf.collect(engine="in-memory")  # per query

Or set the POLARS_ENGINE_AFFINITY=in-memory environment variable.

`pl.read_csv` is now dispatched to `pl.scan_csv(...).collect()`.

It gains scan_csv's with_column_names, infer_schema_files, credential_provider, include_file_paths, extra_columns, and missing_columns parameters, and accepts a list of sources.

It loses n_threads, batch_size, sample_size, and rechunk, which have no equivalent in the lazy reader (call .rechunk() on the result if you need it). Its behavior now also matches scan_csv exactly: a schema_overrides list must cover every column in the file, and columns=[...] returns columns in the requested order rather than sorted.

Example

Before:

>>> data = b"a,b,c,d\n1,2,3,4\n1,2,3,4\n"
>>> pl.read_csv(data, schema_overrides=[pl.String]).dtypes
[String, Int64, Int64, Int64]
>>> pl.read_csv(data, columns=[2, 1, 3]).columns
['b', 'c', 'd']

After:

>>> pl.read_csv(data, schema_overrides=[pl.String])
Traceback (most recent call last):
...
polars.exceptions.SchemaError: The number of dtypes in schema override must be equal to the number of fields in the file (1 != 4).
>>> pl.read_csv(data, columns=[2, 1, 3]).columns
['c', 'b', 'd']

Use instead:

>>> pl.read_csv(data, schema_overrides=[pl.String, pl.Int64, pl.Int64, pl.Int64]).dtypes
[String, Int64, Int64, Int64]

`pl.read_ipc` is now dispatched to `pl.scan_ipc(...).collect()` for all non-`use_pyarrow` inputs

The memory_map and rechunk parameters are removed. Unlike scan_ipc's memory_map, which had already become a no-op in 1.40, read_ipc's memory_map had real effect before this release.

Example

Before:

>>> pl.read_ipc("data.arrow", memory_map=True)

After:

>>> pl.read_ipc("data.arrow", memory_map=True)
Traceback (most recent call last):
...
TypeError: read_ipc() got an unexpected keyword argument 'memory_map'

scan_ipc loses the same memory_map parameter in this release (it had been a no-op there since 1.40; see the removal table below), and both readers also lose rechunk. Use .rechunk() on the returned frame instead.

Use instead:

>>> pl.read_ipc("data.arrow")

Change default `show_graph()` `plan_stage` to `"physical"`

LazyFrame.show_graph()'s plan_stage parameter now defaults to "physical" instead of "ir". This does not, by itself, change what the default call renders: plan_stage="physical" only draws the streaming physical graph when engine="streaming" is also selected (explicitly, or via engine affinity); with the default engine="auto" it still falls back to the optimized IR graph, same as before.

Example

Before:

>>> lf.group_by("a").agg(pl.all().sum()).show_graph()  # shows the optimized IR plan

After:

>>> lf.group_by("a").agg(pl.all().sum()).show_graph()  # plan_stage="physical" by default, but still
>>> #                                                     shows the IR plan, since engine="auto" != "streaming"
>>> lf.group_by("a").agg(pl.all().sum()).show_graph(engine="streaming")  # shows the streaming physical plan

Use instead, to restore the previous default:

>>> lf.group_by("a").agg(pl.all().sum()).show_graph(plan_stage="ir")

Remove `LazyFrame.profile()`

LazyFrame.profile() has been removed. It was designed for the in-memory engine; the concurrent nature of the streaming engine, now the default, would make its per-node timings misleading.

Example

Before:

>>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort("a").profile()
(shape: (3, 3)
┌─────┬─────┬─────┐
│ a   ┆ b   ┆ c   │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ x   ┆ 1   ┆ 4   │
│ y   ┆ 2   ┆ 5   │
│ z   ┆ 3   ┆ 6   │
└─────┴─────┴─────┘, shape: (2, 3)
┌──────────────┬───────┬──────┐
│ node         ┆ start ┆ end  │
│ ---          ┆ ---   ┆ ---  │
│ str          ┆ u64   ┆ u64  │
╞══════════════╪═══════╪══════╡
│ optimization ┆ 0     ┆ 371  │
│ sort(a)      ┆ 1162  ┆ 1234 │
└──────────────┴───────┴──────┘)

After:

>>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort("a").profile()
Traceback (most recent call last):
...
polars.exceptions.AttributeRemovedError: `profile` was removed in version 2.0; It was designed for the in-memory engine and would give misleading per-node timings under the streaming engine (now the default).

Users of Polars Cloud / On-Prem can use the Query Profiler functionality for detailed metrics. We're working on a way to provide query profiling for open source Polars.

Update the strict behavior of `pl.concat()`/`pl.union()`

how="horizontal" now always requires equal heights, matching what strict=True used to mean. Previously, the default silently padded shorter frames with null.

Danger

This change may silently impact the results of your pipelines. pl.concat([df1, df2], how="horizontal") used to pad; now it raises if the heights differ.

Example

Before:

>>> df1 = pl.DataFrame({"a": [1, 2, 3]})
>>> df2 = pl.DataFrame({"b": [4, 5]})
>>> pl.concat([df1, df2], how="horizontal")
shape: (3, 2)
┌─────┬──────┐
│ a   ┆ b    │
│ --- ┆ ---  │
│ i64 ┆ i64  │
╞═════╪══════╡
│ 1   ┆ 4    │
│ 2   ┆ 5    │
│ 3   ┆ null │
└─────┴──────┘

After:

>>> pl.concat([df1, df2], how="horizontal")
Traceback (most recent call last):
...
polars.exceptions.ShapeError: cannot concat dataframes with different heights in 'strict' mode

Use instead, padding the shorter frame yourself so that the intent is explicit at the call site:

>>> df2_padded = df2.select(pl.col("b").extend_constant(None, df1.height - df2.height))
>>> pl.concat([df1, df2_padded], how="horizontal")
shape: (3, 2)
┌─────┬──────┐
│ a   ┆ b    │
│ --- ┆ ---  │
│ i64 ┆ i64  │
╞═════╪══════╡
│ 1   ┆ 4    │
│ 2   ┆ 5    │
│ 3   ┆ null │
└─────┴──────┘

If you do want Polars to pad for you, how="horizontal_extend" is the direct equivalent of the old behavior:

>>> pl.concat([df1, df2], how="horizontal_extend")  # same result as above

strict=False is no longer accepted as a way to opt into padding either. It now raises, since how="horizontal" itself no longer pads.

>>> pl.concat([df1, df2], how="horizontal", strict=False)
Traceback (most recent call last):
...
ValueError: `strict=False` is no longer supported for `how='horizontal'`. Use `how='horizontal_extend'` to pad shorter frames with `null`.

strict cannot be combined with how="horizontal_extend" at all:

>>> pl.concat([df1, df2], how="horizontal_extend", strict=True)
Traceback (most recent call last):
...
ValueError: `strict` cannot be used with `how='horizontal_extend'`

Set `explode()` `empty_as_null=False` by default

An empty list now explodes into zero rows, rather than one null row. keep_nulls is unaffected: a null list still explodes into one null row.

Danger

This change may silently impact the results of your pipelines. Row counts change wherever a list column contains empty lists.

Example

Before:

>>> df = pl.DataFrame({"a": [[1, 2, 3], [], [4, 5, 6]]})
>>> df.explode("a")
shape: (7, 1)
┌──────┐
│ a    │
│ ---  │
│ i64  │
╞══════╡
│ 1    │
│ 2    │
│ 3    │
│ null │
│ 4    │
│ 5    │
│ 6    │
└──────┘

After:

>>> df.explode("a")
shape: (6, 1)
┌─────┐
│ a   │
│ --- │
│ i64 │
╞═════╡
│ 1   │
│ 2   │
│ 3   │
│ 4   │
│ 5   │
│ 6   │
└─────┘

Use instead:

>>> df.explode("a", empty_as_null=True)

Change supertype of signed integer types and `UInt64` from `Float64` to `Int128`

Adding (or otherwise combining) a signed integer column with a UInt64 column used to produce a lossy Float64 result. It now produces an exact Int128 result.

Danger

This change may silently impact the results of your pipelines. No error is raised: both the output dtype and the computed values change.

Example

Before:

>>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": [1, 2, 3]}, schema={"a": pl.Int64, "b": pl.UInt64})
>>> lf.select((pl.col("a") + pl.col("b")).alias("result")).collect_schema()
Schema({'result': Float64})

After:

>>> lf.select((pl.col("a") + pl.col("b")).alias("result")).collect_schema()
Schema({'result': Int128})

Make coercion casts for `is_in()` strict instead of lossy

is_in() no longer lossily coerces operands to a shared supertype; it now only coerces when doing so is lossless, and raises otherwise.

Example

Before:

>>> pl.Series([1]).is_in(pl.Series([1.99]))
shape: (1,)
Series: '' [bool]
[
        false
]

After:

>>> pl.Series([1]).is_in(pl.Series([1.99]))
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: 'is_in' cannot check for Int64 values in List(Float64) data.
Hint: Before version 2.0, Polars would perform this check by lossily coercing the operands to Float64. However, since Polars 2.0, for is_in() it is required to explicitly cast (one of) the operands to a compatible type.

Use instead:

>>> pl.Series([1]).is_in(pl.Series([1.99]).cast(pl.Int64))

Stop coercing `pl.col(...)` to a selector in selector `&`, `|`, `^`

Combining a selector with pl.col(...) via &, |, or ^ used to be special-cased to mean a set operation on the selected columns. It now falls through to the ordinary element-wise operator.

Danger

This change may silently impact the results of your pipelines. It only raises when the involved dtypes are incompatible; for compatible dtypes (e.g. two integer columns) it silently switches from a column selection to an element-wise bitwise operation.

Example

Before:

>>> df = pl.DataFrame({"a": [1, 2], "b": [3, 4], "mask": [1, 0]})
>>> df.select(pl.selectors.integer() & pl.col("mask"))
shape: (2, 1)
┌──────┐
│ mask │
│ ---  │
│ i64  │
╞══════╡
│ 1    │
│ 0    │
└──────┘

After:

>>> df.select(pl.selectors.integer() & pl.col("mask"))
shape: (2, 3)
┌─────┬─────┬──────┐
│ a   ┆ b   ┆ mask │
│ --- ┆ --- ┆ ---  │
│ i64 ┆ i64 ┆ i64  │
╞═════╪═════╪══════╡
│ 1   ┆ 1   ┆ 1    │
│ 0   ┆ 0   ┆ 0    │
└─────┴─────┴──────┘

Note that no error is raised: a, b, and mask (every column selected by integer()) are each combined with mask via a bitwise &, element-wise, instead of the query selecting just the mask column.

Use instead:

>>> df.select(pl.selectors.integer() & pl.selectors.by_name("mask"))

Set the output name of `pl.datetime` and `pl.repeat` to their leftmost argument name

pl.datetime(...) and pl.repeat(...) used to always name their output column "datetime" / "repeat". For consistency with other multi-argument functions, they now take the name of their leftmost argument (or "literal", if that argument is itself a literal).

Danger

This change may silently impact the results of your pipelines. No error is raised. Only the output column name changes, which can silently overwrite an existing column in a with_columns call, or break a df["datetime"] lookup.

Example

Before:

>>> df = pl.DataFrame({"year": [2001], "month": [1], "day": [1], "hour": [23]})
>>> df.select(pl.datetime("year", "month", "day", "hour")).columns
['datetime']

After:

>>> df.select(pl.datetime("year", "month", "day", "hour")).columns
['year']

Use instead, if you rely on the name:

>>> df.select(pl.datetime("year", "month", "day", "hour").alias("datetime")).columns
['datetime']

Similarly for pl.repeat:

>>> pl.repeat(3, n=3, dtype=pl.Int8, eager=True)
# before: Series: 'repeat' [i8]
# after:  Series: 'literal' [i8]

Preserve outer `null`s in list/array `to_struct()`

A null list or array row now converts to a null struct, rather than a struct whose fields are all null.

Danger

This change may silently impact the results of your pipelines. No error is raised: row validity changes, which affects is_null(), joins, and to_list().

Example

Before:

>>> s = pl.Series([None], dtype=pl.List(pl.Int64))
>>> s.list.to_struct(fields=["a", "b", "c"]).to_list()
[{'a': None, 'b': None, 'c': None}]

After:

>>> s.list.to_struct(fields=["a", "b", "c"]).to_list()
[None]

Do not reset seek position when scanning from file-like objects

Reading from a file-like object (e.g. io.BytesIO) no longer implicitly rewinds it to the start. Scanning now starts from the object's current position.

Danger

The common buf = io.BytesIO(); df.write_parquet(buf); pl.read_parquet(buf) round trip now needs an explicit buf.seek(0) between the write and the read.

Example

Before:

>>> import io
>>> buf = io.BytesIO()
>>> pl.DataFrame({"a": [1, 2, 3]}).write_parquet(buf)
>>> pl.read_parquet(buf)
shape: (3, 1)
┌─────┐
│ a   │
│ --- │
│ i64 │
╞═════╡
│ 1   │
│ 2   │
│ 3   │
└─────┘

After:

>>> pl.read_parquet(buf)
Traceback (most recent call last):
...
polars.exceptions.ComputeError: parquet: File out of specification: A Parquet file must contain a header and footer with at least 12 bytes

Use instead:

>>> buf.seek(0)
>>> pl.read_parquet(buf)

Preserve height in zero-width `DataFrame`/`LazyFrame` operations

DataFrame/LazyFrame with zero columns now carry a height rather than collapsing to (0, 0). This affects drop() of every column, gather_every() on zero-width frames, and manually constructed empty frames: pl.DataFrame() and pl.LazyFrame() now have a fixed height of 0, so adding a longer column via with_columns() raises instead of adopting the new column's length.

Example

Before:

>>> df = pl.DataFrame({"a": [2, 1, 3], "b": ["a", "b", "c"], "c": [1, 2, 3]})
>>> df.drop("*").shape
(0, 0)
>>> pl.DataFrame().with_columns(pl.Series([None, None])).shape
(2, 1)

After:

>>> df.drop("*").shape
(3, 0)
>>> pl.DataFrame().with_columns(pl.Series([None, None]))
Traceback (most recent call last):
...
polars.exceptions.ShapeError: can't broadcast Series '' of length 2 to length 0

Use instead, to build a frame from scratch:

>>> pl.DataFrame(height=2).with_columns(pl.Series([None, None])).shape
(2, 1)

Always return `Series` from `from_arrow(<ArrowStreamExportable>)`

pl.from_arrow() on an object that supports only the Arrow PyCapsule stream interface (i.e. __arrow_c_stream__, but not a pyarrow.Table/RecordBatch itself) now always returns a Series, rather than a DataFrame. A multi-column source becomes a Series of Struct, one struct per row. The schema/schema_overrides parameters, which had no effect on this path, now raise instead of being silently ignored.

Danger

This change may silently impact the results of your pipelines: the return type changes from DataFrame to Series, and no error is raised.

Example

Before:

>>> class ArrowStreamOnly:
...     """Exposes only `__arrow_c_stream__`, like a third-party Arrow producer."""
...     def __init__(self, obj):
...         self._obj = obj
...     def __arrow_c_stream__(self, requested_schema=None):
...         return self._obj.__arrow_c_stream__(requested_schema)
>>> pl.from_arrow(ArrowStreamOnly(pl.DataFrame({"x": [1, 2, 3]})))
shape: (3, 1)
┌─────┐
│ x   │
│ --- │
│ i64 │
╞═════╡
│ 1   │
│ 2   │
│ 3   │
└─────┘

After:

>>> pl.from_arrow(ArrowStreamOnly(pl.DataFrame({"x": [1, 2, 3]})))
shape: (3,)
Series: '' [struct[1]]
[
        {1}
        {2}
        {3}
]

Use instead, if you need a DataFrame:

>>> pl.from_arrow(ArrowStreamOnly(pl.DataFrame({"x": [1, 2, 3]}))).struct.unnest()

Remove the DataFrame Interchange Protocol

Polars no longer implements the DataFrame Interchange Protocol. DataFrame.__dataframe__() has been removed, along with the polars.interchange.buffer, column, dataframe, from_dataframe, and utils submodules and the polars.interchange.protocol symbols (DtypeKind, Buffer, Column, SupportsInterchange, CopyNotAllowedError, ...). The polars.interchange package itself remains, but now holds only CompatLevel.

pl.from_dataframe() is kept, but only accepts objects supporting the Arrow PyCapsule interface; it no longer falls back to the interchange protocol. Its allow_copy parameter has been removed.

Example

Before:

>>> df = pl.DataFrame({"a": [1, 2, 3]})
>>> df.__dataframe__()
<polars.interchange.dataframe.PolarsDataFrame object at 0x10c0f5d90>

After:

>>> df.__dataframe__()
Traceback (most recent call last):
...
polars.exceptions.AttributeRemovedError: `__dataframe__` was removed in version 2.0; the dataframe interchange protocol is not supported anymore. Consider using `to_arrow` or `to_pandas` instead.

Use instead, for a consumer that accepted an interchange object (e.g. Seaborn):

>>> import seaborn as sns
>>> sns.scatterplot(df.to_pandas(), x="a", y="a")

For Arrow-native interop, use DataFrame.to_arrow() and pl.from_arrow().

Passing an object that supports neither interface to pl.from_dataframe() now raises, rather than silently going through the interchange protocol:

>>> pl.from_dataframe(object())
Traceback (most recent call last):
...
TypeError: expected object supporting the PyCapsule Interface, got 'object'

And allow_copy is gone:

>>> pl.from_dataframe(df, allow_copy=True)
Traceback (most recent call last):
...
polars.exceptions.ArgumentRemovedError: the argument 'allow_copy' for 'from_dataframe' was deprecated in version 1.23.0 and has been removed in version 2.0.0.

Allow transposing an empty `DataFrame`

DataFrame.transpose() on a zero-height or zero-width DataFrame no longer raises; it transposes the frame's shape like any other, consistent with transpose() on a non-empty frame.

Example

Before:

>>> pl.DataFrame().transpose()
Traceback (most recent call last):
...
polars.exceptions.NoDataError: unable to transpose an empty DataFrame

After:

>>> pl.DataFrame().transpose()
shape: (0, 0)
┌┐
╞╡
└┘

Note that a frame's height becomes the transposed frame's width, and vice versa, so a zero-width frame with a nonzero number of rows transposes into a frame with that many columns and no rows. The output columns get the usual generated names, and Null dtype, since there is no input data to derive a supertype from:

>>> pl.DataFrame(height=3).transpose()
shape: (0, 3)
┌──────────┬──────────┬──────────┐
│ column_0 ┆ column_1 ┆ column_2 │
│ ---      ┆ ---      ┆ ---      │
│ null     ┆ null     ┆ null     │
╞══════════╪══════════╪══════════╡
└──────────┴──────────┴──────────┘

The reverse case, a zero-height frame with columns, drops those columns and keeps their count as the new height:

>>> pl.DataFrame(schema={"a": pl.Int32, "b": pl.Int32}).transpose().shape
(2, 0)
>>> pl.DataFrame(schema={"a": pl.Int32, "b": pl.Int32}).transpose(include_header=True)
shape: (2, 1)
┌────────┐
│ column │
│ ---    │
│ str    │
╞════════╡
│ a      │
│ b      │
└────────┘

Casts and operations no longer supported

The following casts and operations, previously deprecated, now raise an error.

Disable casting from integers to categoricals, and from categoricals to integers

Use .cat.to() / .cat.physical() instead.

Example

Before:

>>> dtype = pl.Enum(["a", "b", "c"])
>>> pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(dtype)
shape: (4,)
Series: '' [enum]
[
        null
        "b"
        "a"
        "c"
]

After:

>>> pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(dtype)
Traceback (most recent call last):
...
polars.exceptions.ComputeError: casting from u32 to enum is not supported.
Instead of `.cast(Enum([...])`, use `.cat.to(Enum([...]))`.

Use instead:

>>> pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cat.to(dtype)

And the reverse:

>>> s = pl.Series("a", ["cat2", "cat0", "cat1"], dtype=pl.Enum(["cat0", "cat1", "cat2"]))
>>> s.cast(pl.UInt32)
Traceback (most recent call last):
...
polars.exceptions.ComputeError: cannot cast categorical types to UInt32.
Instead of `.cast(UInt32)`, use `.cat.physical()`.
>>> s.cat.physical()  # instead

Remove casts from string to temporal types

Casting a String column directly to Date/Datetime is no longer supported.

Example

Before:

>>> pl.Series(["2022-08-30"]).cast(pl.Date)
shape: (1,)
Series: '' [date]
[
        2022-08-30
]

After:

>>> pl.Series(["2022-08-30"]).cast(pl.Date)
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: casting from string to date is not supported.
It was removed in Polars 2.0. Use `str.to_date()` instead.

Use instead:

>>> pl.Series(["2022-08-30"]).str.to_date()

The same applies to casting String to Datetime — use str.to_datetime() instead, and to casting a SQL string literal to DATE/TIMESTAMP via ::date/::timestamp — use DATE '...' / DATE(...) instead. String-to-Time casts are unaffected.

Disallow casting from non-nested types to `pl.List(..)`

Example

Before:

>>> pl.Series("a", [1, 2, 3], dtype=pl.Int32).cast(pl.List(pl.Int32))
shape: (3,)
Series: 'a' [list[i32]]
[
        [1]
        [2]
        [3]
]

After:

>>> pl.Series("a", [1, 2, 3], dtype=pl.Int32).cast(pl.List(pl.Int32))
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: casting from Int32 to list type is not supported
Hint: Use pl.list(expr) to turn the Int32 column into a column of single-element lists.

Use instead:

>>> pl.select(pl.list(pl.Series("a", [1, 2, 3], dtype=pl.Int32)))

Do not allow boolean operators between booleans and integer types

Example

Before:

>>> lf = pl.LazyFrame({"bool": [True, False], "int": [1, 2]}, schema={"bool": pl.Boolean, "int": pl.Int32})
>>> lf.select(pl.col("bool") & pl.col("int")).collect_schema()
Schema({'bool': Int32})

After:

>>> lf.select(pl.col("bool") & pl.col("int")).collect_schema()
Traceback (most recent call last):
...
polars.exceptions.ComputeError: & on Boolean and Int32 is not supported
Hint: cast the Boolean to Int32 using pl.Expr.cast().

Use instead:

>>> lf.select(pl.col("bool").cast(pl.Int32) & pl.col("int")).collect_schema()

Remove support for `std()`/`var()` on the `Duration` dtype

Example

Before:

>>> from datetime import timedelta
>>> pl.Series([timedelta(days=1), timedelta(days=2), timedelta(days=4)]).std()
datetime.timedelta(days=1, seconds=45578, microseconds=180014)

After:

>>> pl.Series([timedelta(days=1), timedelta(days=2), timedelta(days=4)]).std()
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: `std` operation not supported for dtype `duration[μs]`

This also applies to var(), ewm_std(), and ewm_var() on Duration columns. Cast to an integer representation first if you need the statistic, e.g. .dt.total_microseconds().std().

Disable invalid struct casts when `strict=True`

Casting between two Struct dtypes with a different number of fields, or with mismatched field names, used to silently succeed by taking the first n matching fields. The same kind of silent truncation struct.rename_fields() used to do (see below). Under the default strict=True it now raises; pass strict=False to keep the old truncating behavior.

Danger

This change may silently impact the results of your pipelines: casts that used to silently drop struct fields now raise, unless you pass strict=False.

Example

Before:

>>> s = pl.Series("x", [{"a": 1, "b": 2}])
>>> s.cast(pl.Struct({"a": pl.Int64}))
shape: (1,)
Series: 'x' [struct[1]]
[
        {1}
]

After:

>>> s.cast(pl.Struct({"a": pl.Int64}))
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: cast from `struct[2]` to `struct[1]` failed in column 'x': structs do not have the same number of fields: 2 vs 1

Ensure that any output struct has the same number of fields as the input, and that all struct field names in the output are present in the input.
Use `strict=False` to force the cast, and Polars will select the first n fields from the struct.

Use instead, to keep the previous behavior:

>>> s.cast(pl.Struct({"a": pl.Int64}), strict=False)

API reshapes

Simplify `list.to_struct` API

fields is now a required, positional argument. The n_field_strategy, upper_bound parameters and the callable form of fields have been removed; the number of names given determines the output width.

Example

Before:

>>> df = pl.DataFrame({"n": [[0, 1, 2], [0, 1]]})
>>> df.select(pl.col("n").list.to_struct(upper_bound=3))
shape: (2, 1)
┌────────────┐
│ n          │
│ ---        │
│ struct[3]  │
╞════════════╡
│ {0,1,2}    │
│ {0,1,null} │
└────────────┘

After:

>>> df.select(pl.col("n").list.to_struct())
Traceback (most recent call last):
...
TypeError: ExprListNameSpace.to_struct() missing 1 required positional argument: 'fields'

Use instead:

>>> df.select(pl.col("n").list.to_struct(["field_0", "field_1", "field_2"]))

Note that arr.to_struct() (for the fixed-size Array dtype) still accepts an omitted fields, since the output width is already known from the dtype; only the callable form of fields was removed there.

Make `add_business_days()` and `str.to_decimal()` keyword-only arguments strict

Expr/Series.dt.add_business_days() (all arguments after n) and Expr/Series.str.to_decimal() (inference_length) have accepted these as keyword-only for several releases, with a DeprecationWarning if passed positionally. That warning is now a hard error.

Example

Before:

>>> import datetime as dt
>>> week_mask = (True, True, True, True, True, False, False)
>>> pl.Series([dt.date(2020, 1, 1)]).dt.add_business_days(5, week_mask)
shape: (1,)
Series: '' [date]
[
        2020-01-08
]
>>> pl.Series(["1.5"]).str.to_decimal(100)
shape: (1,)
Series: '' [decimal[2,1]]
[
        1.5
]

After:

>>> pl.Series([dt.date(2020, 1, 1)]).dt.add_business_days(5, week_mask)
Traceback (most recent call last):
...
TypeError: ExprDateTimeNameSpace.add_business_days() takes 2 positional arguments but 3 were given
>>> pl.Series(["1.5"]).str.to_decimal(100)
Traceback (most recent call last):
...
TypeError: StringNameSpace.to_decimal() takes 1 positional argument but 2 were given

Use instead:

>>> pl.Series([dt.date(2020, 1, 1)]).dt.add_business_days(5, week_mask=week_mask)
>>> pl.Series(["1.5"]).str.to_decimal(inference_length=100)

Replace multi-seed hash API with a single seed

Expr.hash, Series.hash, and DataFrame.hash_rows no longer accept seed_1/seed_2/seed_3.

Hash values computed for the default seed also changed. Note that Polars doesn't guarantee hash stability across versions, only within a single version.

Example

Before:

>>> df = pl.DataFrame({"a": [1, 2, 3]})
>>> df.with_columns(pl.all().hash(10, 20, 30, 40))

After:

>>> df.with_columns(pl.all().hash(10, 20, 30, 40))
Traceback (most recent call last):
...
TypeError: Expr.hash() takes from 1 to 2 positional arguments but 5 were given

Use instead:

>>> df.with_columns(pl.all().hash(10))

Do not support `list` type anymore for `Series.search_sorted()`

Passing a Python list was ambiguous. It could mean either a single list-typed search value, or multiple scalar search targets. It now raises.

Example

Before:

>>> pl.Series([1, 2, 3]).search_sorted([1, 2])
shape: (2,)
Series: '' [u32]
[
        0
        1
]

After:

>>> pl.Series([1, 2, 3]).search_sorted([1, 2])
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: passing a list to `search_sorted` is ambiguous; use `pl.Series([...])` or `pl.lit(...)`

Use instead:

>>> pl.Series([1, 2, 3]).search_sorted(pl.Series([1, 2]))  # many scalar targets
>>> pl.Series("lists", [[0, 1], [0, 2], [1, 4]]).search_sorted(pl.lit([0, 2]))  # one list-typed value

Raise an error on `struct.rename_fields()` with an incorrect number of fields

Passing a number of names that doesn't match the number of struct fields now raises a SchemaError, instead of silently truncating.

Example

Before:

>>> s = pl.Series("s", [{"a": 1, "b": 2}])
>>> s.struct.rename_fields(["x"])
shape: (1,)
Series: 's' [struct[1]]
[
        {1}
]

After:

>>> s.struct.rename_fields(["x"])
Traceback (most recent call last):
...
polars.exceptions.SchemaError: struct.rename_fields() argument has a different number of fields than the struct it operates on (1 vs 2).

Hint: use struct.drop() to drop fields from the struct first.

Use instead, to drop fields first:

>>> s.struct.drop(["b"]).struct.rename_fields(["x"])

Remove the `ordering` parameter and the `'physical'`/`'lexical'` special case on `pl.Categorical()`

Categoricals are now always ordered lexically, so the ordering parameter has been removed. Additionally, the special case where pl.Categorical("physical") / pl.Categorical("lexical") meant an ordering (rather than a category-pool name) has been removed.

Danger

pl.Categorical("physical") no longer means anything about ordering. It silently becomes a named Categories pool called "physical" instead of raising or warning.

Example

Before:

>>> pl.Categorical(ordering="lexical")
Categorical

After:

>>> pl.Categorical(ordering="lexical")
Traceback (most recent call last):
...
TypeError: Categorical.__init__() got an unexpected keyword argument 'ordering'

And the silent case:

>>> pl.Categorical("physical")
Categorical(Categories("physical"))

CSV reading changes

Set csv `infer_schema_files` default value to `10`

Multi-file CSV scans now only inspect the first 10 files (instead of all of them) to infer the schema by default.

Example

Before:

>>> pl.scan_csv([b"a\n1"] * 10 + [b"a\nA"]).collect().tail(1)
shape: (1, 1)
┌─────┐
│ a   │
│ --- │
│ str │
╞═════╡
│ A   │
└─────┘

After:

>>> pl.scan_csv([b"a\n1"] * 10 + [b"a\nA"]).collect()
Traceback (most recent call last):
...
polars.exceptions.ComputeError: could not parse `A` as dtype `i64` at column 'a' (column number 1)

The current offset in the file is 0 bytes.

You might want to try:
- increasing `infer_schema_length` (e.g. `infer_schema_length=10000`),
- specifying correct dtype with the `schema_overrides` argument
- setting `ignore_errors` to `True`,
- adding `A` to the `null_values` list.

Use instead:

>>> pl.scan_csv([b"a\n1"] * 10 + [b"a\nA"], infer_schema_files=11).collect()

Start csv column name counting at 0

Auto-generated column names for headerless files now start at column_0 instead of column_1. This also affects read_excel/read_ods with has_header=False.

Example

Before:

>>> pl.read_csv(b"a,1,10;b,2,20", eol_char=";", has_header=False).columns
['column_1', 'column_2', 'column_3']

After:

>>> pl.read_csv(b"a,1,10;b,2,20", eol_char=";", has_header=False).columns
['column_0', 'column_1', 'column_2']

Respect file column order when schema is given for `scan_csv()`

A user-provided schema for read_csv/scan_csv is now matched by column name against the file's header, and the file's column order is respected, rather than matching positionally. New extra_columns/missing_columns parameters control name mismatches; their literal default is None, which behaves as "raise".

Example

Before:

>>> buf = b"a,b\nA,B\n"
>>> pl.scan_csv(buf, schema={"b": pl.String, "a": pl.String}).collect()
shape: (1, 2)
┌─────┬─────┐
│ b   ┆ a   │
│ --- ┆ --- │
│ str ┆ str │
╞═════╪═════╡
│ A   ┆ B   │
└─────┴─────┘

After:

>>> pl.scan_csv(buf, schema={"b": pl.String, "a": pl.String}).collect()
shape: (1, 2)
┌─────┬─────┐
│ b   ┆ a   │
│ --- ┆ --- │
│ str ┆ str │
╞═════╪═════╡
│ B   ┆ A   │
└─────┴─────┘

Note the values: previously schema was matched positionally against the file (so the first schema entry, b, silently received the file's first column, a's data). Now b correctly receives the file's actual b column.

Use instead, if the file has extra or missing columns relative to the schema:

>>> pl.scan_csv(buf, schema={"b": pl.String}, extra_columns="ignore").collect()
>>> pl.scan_csv(buf, schema={"a": pl.String, "b": pl.String, "c": pl.Int64}, missing_columns="insert").collect()

Automatically disable `raise_if_empty` when `has_header=False` and `schema` is given

raise_if_empty (on read_csv/scan_csv) now has a literal default of None, resolved internally as schema is None or has_header. In the narrow case of an empty, headerless source read with an explicit schema, this returns an empty frame with that schema instead of raising NoDataError. Pass raise_if_empty=True explicitly to keep the old behavior.

A related, undocumented default change: truncate_ragged_lines moves from a literal False to None, resolved as extra_columns == "ignore". The effective default is unchanged unless you also pass extra_columns="ignore", which now implicitly enables ragged-line truncation too. Passing them in direct conflict (extra_columns="ignore" together with an explicit truncate_ragged_lines=False) is a new hard error rather than the extra columns silently winning or losing:

>>> pl.read_csv(b"a,b\n1,2,3\n", extra_columns="ignore", truncate_ragged_lines=False)
Traceback (most recent call last):
...
ValueError: cannot set truncate_ragged_lines=False with extra_columns='ignore'

>>> pl.read_csv(b"", has_header=False, schema={"a": pl.Int64})
shape: (0, 1)
┌─────┐
│ a   │
│ --- │
│ i64 │
╞═════╡
└─────┘

Removal of deprecated functionality

Attempting to use most of the removed functionality below now raises one of two new exception types, each with a hint towards the replacement: polars.exceptions.AttributeRemovedError (an AttributeError) for removed attributes and methods, and polars.exceptions.ArgumentRemovedError (a TypeError) for removed/renamed parameters. Coverage is not complete, though: the entries marked below with [^1] still raise a plain built-in AttributeError or TypeError from ordinary attribute or argument resolution rather than the new typed exceptions.

Removed	Deprecated in	Use instead
`DataFrame`/`LazyFrame.melt()`	1.0	`unpivot()`, with `index` instead of `id_vars` and `on` instead of `value_vars`
`DataFrame`/`LazyFrame.with_row_count()`	0.20	`with_row_index()` (default column name is now `"index"`, not `"row_nr"`)
`DataFrame`/`LazyFrame.approx_n_unique()`	1.0	`select(pl.all().approx_n_unique())`
`LazyFrame.fetch()`	0.19	`collect()` in conjunction with `head()`
`LazyFrame.profile()`	1.43	none (misleading under the streaming engine)
`DataFrame`/`LazyFrame.group_by(...).count()`²	0.20	`len()`
`DataFrame.__dataframe__()`	1.40	`to_arrow()`/`to_pandas()` (see above)
`polars.interchange` submodules and `protocol` symbols¹	1.40	none (only `CompatLevel` remains; see above)
`pl.Array.width`	0.20	`size`
`polars.testing.parametric.columns()`¹	0.20	`column`, in conjunction with a list comprehension
`polars.testing.parametric.create_list_strategy()`¹	0.20	`lists`
`Expr`/`Series.dt.datetime()`	1.0	`dt.replace_time_zone(None)`
`Expr`/`Series.dt.with_time_unit()`	1.0	cast to `Int64`, then cast to the desired dtype
`Expr.agg_groups()`, `pl.groups()`¹	1.0	`df.with_row_index().group_by(...).agg(pl.col("index"))`
`Expr.flatten()`¹	1.0	`list.explode(keep_nulls=False, empty_as_null=False)` (note: different null semantics)
`Expr.rechunk()`¹	1.0	`df.rechunk()` after collecting
`Series.cat.get_categories()`¹	1.0	`Series.unique()`, or `dtype.categories` for an `Enum`
`Series.cat.is_local()`, `to_local()`, `uses_lexical_ordering()`¹	1.0	none (categoricals no longer have a local scope; ordering is always lexical)
`Enum.union()` / `Enum.__or__`¹	1.38	construct explicitly: `pl.Enum(pl.concat([e1.categories, e2.categories]).unique(maintain_order=True))`
`Config.set_auto_structify()`¹	1.32	explicit `pl.struct(...)`
`read_csv_batched()`¹	1.37	`scan_csv().collect_batches()`
`Expr`/`Series.str.explode()`¹	1.0	`str.split("").explode()` (note: empty strings now become `null` instead of being preserved)
`read_csv`/`scan_csv(missing_utf8_is_empty_string=...)`¹	1.43	`empty_string_is_null=...` (note: the boolean is inverted)
`pl.corr(ddof=...)`¹	1.17	none (parameter had no effect)
`scan_ipc(memory_map=...)`¹	1.40	none (parameter had no effect)
`read_parquet`/`scan_parquet(allow_missing_columns=...)`¹	1.30	`missing_columns="insert"` / `"raise"`
`read/scan_*(rechunk=...)`¹	1.x	`.rechunk()` on the result after reading
`LazyFrame.map_batches(no_optimizations=...)`¹	1.30	none (pushdown parameters already default to `False`)
`DataFrame.to_arrow`/`write_ipc`/`write_ipc_stream(future=...)`	1.1	`compat_level=...`
`DataFrame.glimpse(return_as_string=...)`	1.35	`return_type="string"`
`DataFrame`/`LazyFrame.top_k`/`bottom_k(descending=...)`	1.0	`reverse=...`
`DataFrame`/`LazyFrame.rolling`/`group_by_dynamic(by=...)`, `DataFrame.upsample(by=...)`	0.20	`group_by=...`
`DataFrame`/`LazyFrame.join(join_nulls=...)`	1.24	`nulls_equal=...`
`DataFrame.pivot(columns=...)`	1.0	`on=...`
`pl.Array(width=...)`	0.20	`shape=...`
`pl.from_dataframe(allow_copy=...)`	1.23	none (removed along with the interchange protocol)
`LazyFrame.collect`/`collect_async`/`explain`/`show_graph(streaming=...)`	1.25	`engine="streaming"`
`pl.collect_all`/`collect_all_async(streaming=...)`	1.25	`engine="streaming"`
`LazyFrame.explain(tree_format=...)`	0.20	`format="tree"`
`LazyFrame.unpivot(streamable=...)`	1.5	none (parameter had no effect)
`LazyFrame.sink_(retries=...)`, `DataFrame.write_(retries=...)`	1.37	`max_retries` in `storage_options`
The individual optimization-flag parameters (`no_optimization`, `predicate_pushdown`, ...)³	1.30	the `optimizations` parameter

Example

Before:

>>> lf.melt(id_vars="a", value_vars="b")
>>> df.with_row_count()
>>> df.join(df, on="a", join_nulls=True)

After:

>>> lf.melt(id_vars="a", value_vars="b")
Traceback (most recent call last):
...
polars.exceptions.AttributeRemovedError: `melt` was removed in version 2.0; use `LazyFrame.unpivot` instead, with `index` instead of `id_vars` and `on` instead of `value_vars`
>>> df.with_row_count()
Traceback (most recent call last):
...
polars.exceptions.AttributeRemovedError: `with_row_count` was removed in version 2.0; use `with_row_index` instead. Note that the default column name has changed from 'row_nr' to 'index'.
>>> df.join(df, on="a", join_nulls=True)
Traceback (most recent call last):
...
polars.exceptions.ArgumentRemovedError: the argument 'join_nulls' for 'DataFrame.join' was deprecated in version 1.24 and has been removed in 2.0. It was renamed to 'nulls_equal' in version 2.0.

Deprecations

Mark `read_avro()`/`write_avro()` as unstable

Avro support may change at any point without it being considered a breaking change. Enable pl.Config.warn_unstable(True) to be warned when using it.

Deprecate redundant `strict=True` on horizontal `pl.concat()`/`pl.union()`

Since how="horizontal" now always requires equal heights (see above), passing strict=True explicitly no longer does anything and is deprecated. Simply omit it.

These raise a plain built-in AttributeError/TypeError rather than the new typed exceptions. They're mostly pure deletions with no direct rename target (Expr.rechunk(), pl.corr(ddof=...), scan_ipc(memory_map=...), read/scan_*(rechunk=...), map_batches(no_optimizations=...), ...), plus two exceptions that do have a direct replacement but were not wired into the new mechanism: read_parquet/scan_parquet(allow_missing_columns=...) and read_csv/scan_csv(missing_utf8_is_empty_string=...). The removed polars.interchange submodules and polars.testing.parametric strategy helpers are plain deletions too: importing them raises ImportError, and attribute access raises AttributeError. In practice the two rename cases still get a hint, just not the typed one: Python's own TypeError for an unexpected keyword argument appends a Did you mean '...'? suggestion when a similarly-named parameter exists (missing_columns, empty_string_is_null). Pure deletions with no near-name match get no such suggestion. ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
Only LazyFrame.group_by(...).count() and DataFrame.group_by_dynamic(...).count() raise the typed AttributeRemovedError. DataFrame.group_by(...).count() and DataFrame.rolling(...).count() raise a plain AttributeError with no hint, so the eager group_by/rolling paths give you no pointer to len(). ↩
The full set: no_optimization, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, plus the private _eager, _type_check, and _check_order, on LazyFrame.explain/show_graph/collect and pl.collect_all. Pass a pl.QueryOptFlags object via optimizations= instead, e.g. lf.collect(optimizations=pl.QueryOptFlags(predicate_pushdown=False)). ↩

Version 2.0-rc

Breaking changes

The Lazy API defaults to the streaming engine

pl.read_csv is now dispatched to pl.scan_csv(...).collect().

pl.read_ipc is now dispatched to pl.scan_ipc(...).collect() for all non-use_pyarrow inputs

Change default show_graph() plan_stage to "physical"

Remove LazyFrame.profile()

Update the strict behavior of pl.concat()/pl.union()

Set explode() empty_as_null=False by default

Change supertype of signed integer types and UInt64 from Float64 to Int128

Make coercion casts for is_in() strict instead of lossy

Stop coercing pl.col(...) to a selector in selector &, |, ^

Set the output name of pl.datetime and pl.repeat to their leftmost argument name

Preserve outer nulls in list/array to_struct()

Do not reset seek position when scanning from file-like objects

Preserve height in zero-width DataFrame/LazyFrame operations

Always return Series from from_arrow(<ArrowStreamExportable>)

Remove the DataFrame Interchange Protocol

Allow transposing an empty DataFrame

Casts and operations no longer supported

Disable casting from integers to categoricals, and from categoricals to integers

Remove casts from string to temporal types

Disallow casting from non-nested types to pl.List(..)

Do not allow boolean operators between booleans and integer types

Remove support for std()/var() on the Duration dtype

Disable invalid struct casts when strict=True

API reshapes

Simplify list.to_struct API

Make add_business_days() and str.to_decimal() keyword-only arguments strict

Replace multi-seed hash API with a single seed

Do not support list type anymore for Series.search_sorted()

Raise an error on struct.rename_fields() with an incorrect number of fields

Remove the ordering parameter and the 'physical'/'lexical' special case on pl.Categorical()

CSV reading changes

Set csv infer_schema_files default value to 10

Start csv column name counting at 0

Respect file column order when schema is given for scan_csv()

Automatically disable raise_if_empty when has_header=False and schema is given

Removal of deprecated functionality

Deprecations

Mark read_avro()/write_avro() as unstable

Deprecate redundant strict=True on horizontal pl.concat()/pl.union()

`pl.read_csv` is now dispatched to `pl.scan_csv(...).collect()`.

`pl.read_ipc` is now dispatched to `pl.scan_ipc(...).collect()` for all non-`use_pyarrow` inputs

Change default `show_graph()` `plan_stage` to `"physical"`

Remove `LazyFrame.profile()`

Update the strict behavior of `pl.concat()`/`pl.union()`

Set `explode()` `empty_as_null=False` by default

Change supertype of signed integer types and `UInt64` from `Float64` to `Int128`

Make coercion casts for `is_in()` strict instead of lossy

Stop coercing `pl.col(...)` to a selector in selector `&`, `|`, `^`

Set the output name of `pl.datetime` and `pl.repeat` to their leftmost argument name

Preserve outer `null`s in list/array `to_struct()`

Preserve height in zero-width `DataFrame`/`LazyFrame` operations

Always return `Series` from `from_arrow(<ArrowStreamExportable>)`

Allow transposing an empty `DataFrame`

Disallow casting from non-nested types to `pl.List(..)`

Remove support for `std()`/`var()` on the `Duration` dtype

Disable invalid struct casts when `strict=True`

Simplify `list.to_struct` API

Make `add_business_days()` and `str.to_decimal()` keyword-only arguments strict

Do not support `list` type anymore for `Series.search_sorted()`

Raise an error on `struct.rename_fields()` with an incorrect number of fields

Remove the `ordering` parameter and the `'physical'`/`'lexical'` special case on `pl.Categorical()`

Set csv `infer_schema_files` default value to `10`

Respect file column order when schema is given for `scan_csv()`

Automatically disable `raise_if_empty` when `has_header=False` and `schema` is given

Mark `read_avro()`/`write_avro()` as unstable

Deprecate redundant `strict=True` on horizontal `pl.concat()`/`pl.union()`