Coming from Pandas
Here we set out the key points that anyone who has experience with pandas and wants to try Polars should know. We include both differences in the concepts the libraries are built on and differences in how you should write Polars code compared to pandas code.
Differences in concepts between Polars and pandas
Polars does not have a multi-index/index
pandas gives a label to each row with an index. Polars does not use an index and each row is indexed by its integer position in the table.
Polars aims to have predictable results and readable queries, as such we think an index does not
help us reach that objective. We believe the semantics of a query should not change by the state of
an index or a reset_index call.
In Polars a DataFrame will always be a 2D table with heterogeneous data-types. The data-types may have nesting, but the table itself will not. Operations like resampling will be done by specialized functions or methods that act like 'verbs' on a table explicitly stating the columns that that 'verb' operates on. As such, it is our conviction that not having indices make things simpler, more explicit, more readable and less error-prone.
Note that an 'index' data structure as known in databases will be used by Polars as an optimization technique.
Polars adheres to the Apache Arrow memory format to represent data in memory while pandas uses NumPy arrays
Polars represents data in memory according to the Arrow memory spec while pandas represents data in memory with NumPy arrays. Apache Arrow is an emerging standard for in-memory columnar analytics that can accelerate data load times, reduce memory usage and accelerate calculations.
Polars can convert data to NumPy format with the to_numpy method.
Polars has more support for parallel operations than pandas
Polars exploits the strong support for concurrency in Rust to run many operations in parallel. While
some operations in pandas are multi-threaded the core of the library is single-threaded and an
additional library such as Dask must be used to parallelize operations.
Polars can lazily evaluate queries and apply query optimization
Eager evaluation is when code is evaluated as soon as you run the code. Lazy evaluation is when running a line of code means that the underlying logic is added to a query plan rather than being evaluated.
Polars supports eager evaluation and lazy evaluation whereas pandas only supports eager evaluation. The lazy evaluation mode is powerful because Polars carries out automatic query optimization when it examines the query plan and looks for ways to accelerate the query or reduce memory usage.
Dask also supports lazy evaluation when it generates a query plan.
Key syntax differences
Users coming from pandas generally need to know one thing...
polars != pandas
If your Polars code looks like it could be pandas code, it might run, but it likely runs slower than it should.
Let's go through some typical pandas code and see how we might rewrite it in Polars.
Selecting data
As there is no index in Polars there is no .loc or iloc method in Polars - and there is also no
SettingWithCopyWarning in Polars.
However, the best way to select data in Polars is to use the expression API. For example, if you want to select a column in pandas, you can do one of the following:
df['a']
df.loc[:,'a']
but in Polars you would use the .select method:
df.select('a')
If you want to select rows based on the values then in Polars you use the .filter method:
df.filter(pl.col('a') < 10)
As noted in the section on expressions below, Polars can run operations in .select and filter in
parallel and Polars can carry out query optimization on the full set of data selection criteria.
Be lazy
Working in lazy evaluation mode is straightforward and should be your default in Polars as the lazy mode allows Polars to do query optimization.
We can run in lazy mode by either using an implicitly lazy function (such as scan_csv) or
explicitly using the lazy method.
Take the following simple example where we read a CSV file from disk and do a group by. The CSV file
has numerous columns but we just want to do a group by on one of the id columns (id1) and then sum
by a value column (v1). In pandas this would be:
df = pd.read_csv(csv_file, usecols=['id1','v1'])
grouped_df = df.loc[:,['id1','v1']].groupby('id1').sum('v1')
In Polars you can build this query in lazy mode with query optimization and evaluate it by replacing
the eager pandas function read_csv with the implicitly lazy Polars function scan_csv:
df = pl.scan_csv(csv_file)
grouped_df = df.group_by('id1').agg(pl.col('v1').sum()).collect()
Polars optimizes this query by identifying that only the id1 and v1 columns are relevant and so
will only read these columns from the CSV. By calling the .collect method at the end of the second
line we instruct Polars to eagerly evaluate the query.
If you do want to run this query in eager mode you can just replace scan_csv with read_csv in
the Polars code.
Read more about working with lazy evaluation in the lazy API section.
Express yourself
A typical pandas script consists of multiple data transformations that are executed sequentially. However, in Polars these transformations can be executed in parallel using expressions.
Column assignment
We have a dataframe df with a column called value. We want to add two new columns, a column
called tenXValue where the value column is multiplied by 10 and a column called hundredXValue
where the value column is multiplied by 100.
In pandas this would be:
df.assign(
tenXValue=lambda df_: df_.value * 10,
hundredXValue=lambda df_: df_.value * 100
)
These column assignments are executed sequentially.
In Polars we add columns to df using the .with_columns method:
df.with_columns(
tenXValue=pl.col("value") * 10,
hundredXValue=pl.col("value") * 100,
)
These column assignments are executed in parallel.
Column assignment based on predicate
In this case we have a dataframe df with columns a,b and c. We want to re-assign the values
in column a based on a condition. When the value in column c is equal to 2 then we replace the
value in a with the value in b.
In pandas this would be:
df.assign(a=lambda df_: df_.a.where(df_.c != 2, df_.b))
while in Polars this would be:
df.with_columns(
pl.when(pl.col("c") == 2)
.then(pl.col("b"))
.otherwise(pl.col("a")).alias("a")
)
Polars can compute every branch of an if -> then -> otherwise in parallel. This is valuable, when
the branches get more expensive to compute.
Filtering
We want to filter the dataframe df with housing data based on some criteria.
In pandas you filter the dataframe by passing Boolean expressions to the query method:
df.query("m2_living > 2500 and price < 300000")
or by directly evaluating a mask:
df[(df["m2_living"] > 2500) & (df["price"] < 300000)]
while in Polars you call the filter method:
df.filter(
(pl.col("m2_living") > 2500) & (pl.col("price") < 300000)
)
The query optimizer in Polars can also detect if you write multiple filters separately and combine them into a single filter in the optimized plan.
pandas transform
The pandas documentation demonstrates an operation on a group by called transform. In this case we
have a dataframe df and we want a new column showing the number of rows in each group.
In pandas we have:
df = pd.DataFrame({
"c": [1, 1, 1, 2, 2, 2, 2],
"type": ["m", "n", "o", "m", "m", "n", "n"],
})
df["size"] = df.groupby("c")["type"].transform(len)
Here pandas does a group by on "c", takes column "type", computes the group length and then
joins the result back to the original DataFrame producing:
c type size
0 1 m 3
1 1 n 3
2 1 o 3
3 2 m 4
4 2 m 4
5 2 n 4
6 2 n 4
In Polars the same can be achieved with window functions:
df.with_columns(
pl.col("type").count().over("c").alias("size")
)
shape: (7, 3)
┌─────┬──────┬──────┐
│ c ┆ type ┆ size │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ u32 │
╞═════╪══════╪══════╡
│ 1 ┆ m ┆ 3 │
│ 1 ┆ n ┆ 3 │
│ 1 ┆ o ┆ 3 │
│ 2 ┆ m ┆ 4 │
│ 2 ┆ m ┆ 4 │
│ 2 ┆ n ┆ 4 │
│ 2 ┆ n ┆ 4 │
└─────┴──────┴──────┘
Because we can store the whole operation in a single expression, we can combine several window
functions and even combine different groups!
Polars will cache window expressions that are applied over the same group, so storing them in a
single with_columns is both convenient and optimal. In the following example we look at a case
where we are calculating group statistics over "c" twice:
df.with_columns(
pl.col("c").count().over("c").alias("size"),
pl.col("c").sum().over("type").alias("sum"),
pl.col("type").reverse().over("c").alias("reverse_type")
)
shape: (7, 5)
┌─────┬──────┬──────┬─────┬──────────────┐
│ c ┆ type ┆ size ┆ sum ┆ reverse_type │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ u32 ┆ i64 ┆ str │
╞═════╪══════╪══════╪═════╪══════════════╡
│ 1 ┆ m ┆ 3 ┆ 5 ┆ o │
│ 1 ┆ n ┆ 3 ┆ 5 ┆ n │
│ 1 ┆ o ┆ 3 ┆ 1 ┆ m │
│ 2 ┆ m ┆ 4 ┆ 5 ┆ n │
│ 2 ┆ m ┆ 4 ┆ 5 ┆ n │
│ 2 ┆ n ┆ 4 ┆ 5 ┆ m │
│ 2 ┆ n ┆ 4 ┆ 5 ┆ m │
└─────┴──────┴──────┴─────┴──────────────┘
Missing data
pandas uses NaN and/or None values to indicate missing values depending on the dtype of the
column. In addition the behaviour in pandas varies depending on whether the default dtypes or
optional nullable arrays are used. In Polars missing data corresponds to a null value for all data
types.
For float columns Polars permits the use of NaN values. These NaN values are not considered to
be missing data but instead a special floating point value.
In pandas an integer column with missing values is cast to be a float column with NaN values for
the missing values (unless using optional nullable integer dtypes). In Polars any missing values in
an integer column are simply null values and the column remains an integer column.
See the missing data section for more details.
Pipe littering
A common usage in pandas is utilizing pipe to apply some function to a DataFrame. Copying this
coding style to Polars is unidiomatic and leads to suboptimal query plans.
The snippet below shows a common pattern in pandas.
def add_foo(df: pd.DataFrame) -> pd.DataFrame:
df["foo"] = ...
return df
def add_bar(df: pd.DataFrame) -> pd.DataFrame:
df["bar"] = ...
return df
def add_ham(df: pd.DataFrame) -> pd.DataFrame:
df["ham"] = ...
return df
(df
.pipe(add_foo)
.pipe(add_bar)
.pipe(add_ham)
)
If we do this in polars, we would create 3 with_columns contexts, that forces Polars to run the 3
pipes sequentially, utilizing zero parallelism.
The way to get similar abstractions in polars is creating functions that create expressions. The snippet below creates 3 expressions that run on a single context and thus are allowed to run in parallel.
def get_foo(input_column: str) -> pl.Expr:
return pl.col(input_column).some_computation().alias("foo")
def get_bar(input_column: str) -> pl.Expr:
return pl.col(input_column).some_computation().alias("bar")
def get_ham(input_column: str) -> pl.Expr:
return pl.col(input_column).some_computation().alias("ham")
# This single context will run all 3 expressions in parallel
df.with_columns(
get_ham("col_a"),
get_bar("col_b"),
get_foo("col_c"),
)
If you need the schema in the functions that generate the expressions, you can utilize a single
pipe:
from collections import OrderedDict
def get_foo(input_column: str, schema: OrderedDict) -> pl.Expr:
if "some_col" in schema:
# branch_a
...
else:
# branch b
...
def get_bar(input_column: str, schema: OrderedDict) -> pl.Expr:
if "some_col" in schema:
# branch_a
...
else:
# branch b
...
def get_ham(input_column: str) -> pl.Expr:
return pl.col(input_column).some_computation().alias("ham")
# Use pipe (just once) to get hold of the schema of the LazyFrame.
lf.pipe(lambda lf: lf.with_columns(
get_ham("col_a"),
get_bar("col_b", lf.schema),
get_foo("col_c", lf.schema),
)
Another benefit of writing functions that return expressions, is that these functions are composable as expressions can be chained and partially applied, leading to much more flexibility in the design.