polars.DataFrame.unique#

DataFrame.unique(
subset: ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None = None,
*,
keep: UniqueKeepStrategy = 'any',
maintain_order: bool = False,
) → DataFrame[source]#

Drop duplicate rows from this dataframe.

Parameters:
subset

Column name(s) or selector(s), to consider when identifying duplicate rows. If set to None (default), use all columns.

keep{‘first’, ‘last’, ‘any’, ‘none’}

Which of the duplicate rows to keep.

  • ‘any’: Does not give any guarantee of which row is kept.

    This allows more optimizations.

  • ‘none’: Don’t keep duplicate rows.

  • ‘first’: Keep first unique row.

  • ‘last’: Keep last unique row.

maintain_order

Keep the same order as the original DataFrame. This is more expensive to compute. Settings this to True blocks the possibility to run on the streaming engine.

Returns:
DataFrame

DataFrame with unique rows.

Warning

This method will fail if there is a column of type List in the DataFrame or subset.

Notes

If you’re coming from pandas, this is similar to pandas.DataFrame.drop_duplicates.

Examples

>>> df = pl.DataFrame(
...     {
...         "foo": [1, 2, 3, 1],
...         "bar": ["a", "a", "a", "a"],
...         "ham": ["b", "b", "b", "b"],
...     }
... )
>>> df.unique(maintain_order=True)
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 1   ┆ a   ┆ b   │
│ 2   ┆ a   ┆ b   │
│ 3   ┆ a   ┆ b   │
└─────┴─────┴─────┘
>>> df.unique(subset=["bar", "ham"], maintain_order=True)
shape: (1, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 1   ┆ a   ┆ b   │
└─────┴─────┴─────┘
>>> df.unique(keep="last", maintain_order=True)
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ ham │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞═════╪═════╪═════╡
│ 2   ┆ a   ┆ b   │
│ 3   ┆ a   ┆ b   │
│ 1   ┆ a   ┆ b   │
└─────┴─────┴─────┘
On this page