Strings

The following section discusses operations performed on string data, which is a frequently used data type when working with dataframes. String processing functions are available in the namespace str.

Working with strings in other dataframe libraries can be highly inefficient due to the fact that strings have unpredictable lengths. Polars mitigates these inefficiencies by following the Arrow Columnar Format specification, so you can write performant data queries on string data too.

The string namespace

When working with string data you will likely need to access the namespace str, which aggregates 40+ functions that let you work with strings. As an example of how to access functions from within that namespace, the snippet below shows how to compute the length of the strings in a column in terms of the number of bytes and the number of characters:

Python Rust

str.len_bytes · str.len_chars

import polars as pl

df = pl.DataFrame(
    {
        "language": ["English", "Dutch", "Portuguese", "Finish"],
        "fruit": ["pear", "peer", "pêra", "päärynä"],
    }
)

result = df.with_columns(
    pl.col("fruit").str.len_bytes().alias("byte_count"),
    pl.col("fruit").str.len_chars().alias("letter_count"),
)
print(result)

str.len_bytes · str.len_chars

use polars::prelude::*;

let df = df! (
    "language" => ["English", "Dutch", "Portuguese", "Finish"],
    "fruit" => ["pear", "peer", "pêra", "päärynä"],
)?;

let result = df
    .clone()
    .lazy()
    .with_columns([
        col("fruit").str().len_bytes().alias("byte_count"),
        col("fruit").str().len_chars().alias("letter_count"),
    ])
    .collect()?;

println!("{}", result);

shape: (4, 4)
┌────────────┬─────────┬────────────┬──────────────┐
│ language   ┆ fruit   ┆ byte_count ┆ letter_count │
│ ---        ┆ ---     ┆ ---        ┆ ---          │
│ str        ┆ str     ┆ u32        ┆ u32          │
╞════════════╪═════════╪════════════╪══════════════╡
│ English    ┆ pear    ┆ 4          ┆ 4            │
│ Dutch      ┆ peer    ┆ 4          ┆ 4            │
│ Portuguese ┆ pêra    ┆ 5          ┆ 4            │
│ Finish     ┆ päärynä ┆ 10         ┆ 7            │
└────────────┴─────────┴────────────┴──────────────┘

Note

If you are working exclusively with ASCII text, then the results of the two computations will be the same and using len_bytes is recommended since it is faster.

Parsing strings

Polars offers multiple methods for checking and parsing elements of a string column, namely checking for the existence of given substrings or patterns, and counting, extracting, or replacing, them. We will demonstrate some of these operations in the upcoming examples.

Check for the existence of a pattern

We can use the function contains to check for the presence of a pattern within a string. By default, the argument to the function contains is interpreted as a regular expression. If you want to specify a literal substring, set the parameter literal to True.

For the special cases where you want to check if the strings start or end with a fixed substring, you can use the functions starts_with or ends_with, respectively.

Python Rust

str.contains · str.starts_with · str.ends_with

result = df.select(
    pl.col("fruit"),
    pl.col("fruit").str.starts_with("p").alias("starts_with_p"),
    pl.col("fruit").str.contains("p..r").alias("p..r"),
    pl.col("fruit").str.contains("e+").alias("e+"),
    pl.col("fruit").str.ends_with("r").alias("ends_with_r"),
)
print(result)

str.contains · str.starts_with · str.ends_with · Available on feature regex

let result = df
    .clone()
    .lazy()
    .select([
        col("fruit"),
        col("fruit")
            .str()
            .starts_with(lit("p"))
            .alias("starts_with_p"),
        col("fruit").str().contains(lit("p..r"), true).alias("p..r"),
        col("fruit").str().contains(lit("e+"), true).alias("e+"),
        col("fruit").str().ends_with(lit("r")).alias("ends_with_r"),
    ])
    .collect()?;

println!("{}", result);

shape: (4, 5)
┌─────────┬───────────────┬───────┬───────┬─────────────┐
│ fruit   ┆ starts_with_p ┆ p..r  ┆ e+    ┆ ends_with_r │
│ ---     ┆ ---           ┆ ---   ┆ ---   ┆ ---         │
│ str     ┆ bool          ┆ bool  ┆ bool  ┆ bool        │
╞═════════╪═══════════════╪═══════╪═══════╪═════════════╡
│ pear    ┆ true          ┆ true  ┆ true  ┆ true        │
│ peer    ┆ true          ┆ true  ┆ true  ┆ true        │
│ pêra    ┆ true          ┆ false ┆ false ┆ false       │
│ päärynä ┆ true          ┆ true  ┆ false ┆ false       │
└─────────┴───────────────┴───────┴───────┴─────────────┘

Regex specification

Polars relies on the Rust crate regex to work with regular expressions, so you may need to refer to the syntax documentation to see what features and flags are supported. In particular, note that the flavor of regex supported by Polars is different from Python's module re.

Extract a pattern

The function extract allows us to extract patterns from the string values in a column. The function extract accepts a regex pattern with one or more capture groups and extracts the capture group specified as the second argument.

Python Rust

str.extract

df = pl.DataFrame(
    {
        "urls": [
            "http://vote.com/ballon_dor?candidate=messi&ref=polars",
            "http://vote.com/ballon_dor?candidat=jorginho&ref=polars",
            "http://vote.com/ballon_dor?candidate=ronaldo&ref=polars",
        ]
    }
)
result = df.select(
    pl.col("urls").str.extract(r"candidate=(\w+)", group_index=1),
)
print(result)

str.extract

let df = df! (
    "urls" => [
        "http://vote.com/ballon_dor?candidate=messi&ref=polars",
        "http://vote.com/ballon_dor?candidat=jorginho&ref=polars",
        "http://vote.com/ballon_dor?candidate=ronaldo&ref=polars",
    ]
)?;

let result = df
    .clone()
    .lazy()
    .select([col("urls").str().extract(lit(r"candidate=(\w+)"), 1)])
    .collect()?;

println!("{}", result);

shape: (3, 1)
┌─────────┐
│ urls    │
│ ---     │
│ str     │
╞═════════╡
│ messi   │
│ null    │
│ ronaldo │
└─────────┘

To extract all occurrences of a pattern within a string, we can use the function extract_all. In the example below, we extract all numbers from a string using the regex pattern (\d+), which matches one or more digits. The resulting output of the function extract_all is a list containing all instances of the matched pattern within the string.

Python Rust

str.extract_all

df = pl.DataFrame({"text": ["123 bla 45 asd", "xyz 678 910t"]})
result = df.select(
    pl.col("text").str.extract_all(r"(\d+)").alias("extracted_nrs"),
)
print(result)

str.extract_all

let df = df! (
    "text" => ["123 bla 45 asd", "xyz 678 910t"]
)?;

let result = df
    .clone()
    .lazy()
    .select([col("text")
        .str()
        .extract_all(lit(r"(\d+)"))
        .alias("extracted_nrs")])
    .collect()?;

println!("{}", result);

shape: (2, 1)
┌────────────────┐
│ extracted_nrs  │
│ ---            │
│ list[str]      │
╞════════════════╡
│ ["123", "45"]  │
│ ["678", "910"] │
└────────────────┘

Replace a pattern

Akin to the functions extract and extract_all, Polars provides the functions replace and replace_all. These accept a regex pattern or a literal substring (if the parameter literal is set to True) and perform the replacements specified. The function replace will make at most one replacement whereas the function replace_all will make all the non-overlapping replacements it finds.

Python Rust

str.replace · str.replace_all

df = pl.DataFrame({"text": ["123abc", "abc456"]})
result = df.with_columns(
    pl.col("text").str.replace(r"\d", "-"),
    pl.col("text").str.replace_all(r"\d", "-").alias("text_replace_all"),
)
print(result)

str.replace · str.replace_all · Available on feature regex

let df = df! (
    "text" => ["123abc", "abc456"]
)?;

let result = df
    .clone()
    .lazy()
    .with_columns([
        col("text").str().replace(lit(r"\d"), lit("-"), false),
        col("text")
            .str()
            .replace_all(lit(r"\d"), lit("-"), false)
            .alias("text_replace_all"),
    ])
    .collect()?;

println!("{}", result);

shape: (2, 2)
┌────────┬──────────────────┐
│ text   ┆ text_replace_all │
│ ---    ┆ ---              │
│ str    ┆ str              │
╞════════╪══════════════════╡
│ -23abc ┆ ---abc           │
│ abc-56 ┆ abc---           │
└────────┴──────────────────┘

Modifying strings

Case conversion

Converting the casing of a string is a common operation and Polars supports it out of the box with the functions to_lowercase, to_titlecase, and to_uppercase:

Python Rust

str.to_lowercase · str.to_titlecase · str.to_uppercase

addresses = pl.DataFrame(
    {
        "addresses": [
            "128 PERF st",
            "Rust blVD, 158",
            "PoLaRs Av, 12",
            "1042 Query sq",
        ]
    }
)

addresses = addresses.select(
    pl.col("addresses").alias("originals"),
    pl.col("addresses").str.to_titlecase(),
    pl.col("addresses").str.to_lowercase().alias("lower"),
    pl.col("addresses").str.to_uppercase().alias("upper"),
)
print(addresses)

str.to_lowercase · str.to_titlecase · str.to_uppercase · Available on feature nightly

let addresses = df! (
    "addresses" => [
        "128 PERF st",
        "Rust blVD, 158",
        "PoLaRs Av, 12",
        "1042 Query sq",
    ]
)?;

let addresses = addresses
    .clone()
    .lazy()
    .select([
        col("addresses").alias("originals"),
        col("addresses").str().to_titlecase(),
        col("addresses").str().to_lowercase().alias("lower"),
        col("addresses").str().to_uppercase().alias("upper"),
    ])
    .collect()?;

println!("{}", addresses);

shape: (4, 4)
┌────────────────┬────────────────┬────────────────┬────────────────┐
│ originals      ┆ addresses      ┆ lower          ┆ upper          │
│ ---            ┆ ---            ┆ ---            ┆ ---            │
│ str            ┆ str            ┆ str            ┆ str            │
╞════════════════╪════════════════╪════════════════╪════════════════╡
│ 128 PERF st    ┆ 128 Perf St    ┆ 128 perf st    ┆ 128 PERF ST    │
│ Rust blVD, 158 ┆ Rust Blvd, 158 ┆ rust blvd, 158 ┆ RUST BLVD, 158 │
│ PoLaRs Av, 12  ┆ Polars Av, 12  ┆ polars av, 12  ┆ POLARS AV, 12  │
│ 1042 Query sq  ┆ 1042 Query Sq  ┆ 1042 query sq  ┆ 1042 QUERY SQ  │
└────────────────┴────────────────┴────────────────┴────────────────┘

Stripping characters from the ends

Polars provides five functions in the namespace str that let you strip characters from the ends of the string:

Function	Behaviour
strip_chars	Removes leading and trailing occurrences of the characters specified.
strip_chars_end	Removes trailing occurrences of the characters specified.
strip_chars_start	Removes leading occurrences of the characters specified.
strip_prefix	Removes an exact substring prefix if present.
strip_suffix	Removes an exact substring suffix if present.

Similarity to Python string methods

strip_chars is similar to Python's string method strip and strip_prefix/strip_suffix are similar to Python's string methods removeprefix and removesuffix, respectively.

It is important to understand that the first three functions interpret their string argument as a set of characters whereas the functions strip_prefix and strip_suffix do interpret their string argument as a literal string.

Python Rust

str.strip_chars · str.strip_chars_end · str.strip_chars_start · str.strip_prefix · str.strip_suffix

addr = pl.col("addresses")
chars = ", 0123456789"
result = addresses.select(
    addr.str.strip_chars(chars).alias("strip"),
    addr.str.strip_chars_end(chars).alias("end"),
    addr.str.strip_chars_start(chars).alias("start"),
    addr.str.strip_prefix("128 ").alias("prefix"),
    addr.str.strip_suffix(", 158").alias("suffix"),
)
print(result)

str.strip_chars · str.strip_chars_end · str.strip_chars_start · str.strip_prefix · str.strip_suffix

let addr = col("addresses");
let chars = lit(", 0123456789");
let result = addresses
    .clone()
    .lazy()
    .select([
        addr.clone().str().strip_chars(chars.clone()).alias("strip"),
        addr.clone()
            .str()
            .strip_chars_end(chars.clone())
            .alias("end"),
        addr.clone()
            .str()
            .strip_chars_start(chars.clone())
            .alias("start"),
        addr.clone().str().strip_prefix(lit("128 ")).alias("prefix"),
        addr.clone()
            .str()
            .strip_suffix(lit(", 158"))
            .alias("suffix"),
    ])
    .collect()?;

println!("{}", result);

shape: (4, 5)
┌───────────┬───────────────┬────────────────┬────────────────┬───────────────┐
│ strip     ┆ end           ┆ start          ┆ prefix         ┆ suffix        │
│ ---       ┆ ---           ┆ ---            ┆ ---            ┆ ---           │
│ str       ┆ str           ┆ str            ┆ str            ┆ str           │
╞═══════════╪═══════════════╪════════════════╪════════════════╪═══════════════╡
│ Perf St   ┆ 128 Perf St   ┆ Perf St        ┆ Perf St        ┆ 128 Perf St   │
│ Rust Blvd ┆ Rust Blvd     ┆ Rust Blvd, 158 ┆ Rust Blvd, 158 ┆ Rust Blvd     │
│ Polars Av ┆ Polars Av     ┆ Polars Av, 12  ┆ Polars Av, 12  ┆ Polars Av, 12 │
│ Query Sq  ┆ 1042 Query Sq ┆ Query Sq       ┆ 1042 Query Sq  ┆ 1042 Query Sq │
└───────────┴───────────────┴────────────────┴────────────────┴───────────────┘

If no argument is provided, the three functions strip_chars, strip_chars_end, and strip_chars_start, remove whitespace by default.

Slicing

Besides extracting substrings as specified by patterns, you can also slice strings at specified offsets to produce substrings. The general-purpose function for slicing is slice and it takes the starting offset and the optional length of the slice. If the length of the slice is not specified or if it's past the end of the string, Polars slices the string all the way to the end.

The functions head and tail are specialised versions used for slicing the beginning and end of a string, respectively.

Python Rust

str.slice · str.head · str.tail

df = pl.DataFrame(
    {
        "fruits": ["pear", "mango", "dragonfruit", "passionfruit"],
        "n": [1, -1, 4, -4],
    }
)

result = df.with_columns(
    pl.col("fruits").str.slice(pl.col("n")).alias("slice"),
    pl.col("fruits").str.head(pl.col("n")).alias("head"),
    pl.col("fruits").str.tail(pl.col("n")).alias("tail"),
)
print(result)

str.str_slice · str.str_head · str.str_tail

let df = df! (
    "fruits" => ["pear", "mango", "dragonfruit", "passionfruit"],
    "n" => [1, -1, 4, -4],
)?;

let result = df
    .clone()
    .lazy()
    .with_columns([
        col("fruits")
            .str()
            .slice(col("n"), lit(NULL))
            .alias("slice"),
        col("fruits").str().head(col("n")).alias("head"),
        col("fruits").str().tail(col("n")).alias("tail"),
    ])
    .collect()?;

println!("{}", result);

shape: (4, 5)
┌──────────────┬─────┬─────────┬──────────┬──────────┐
│ fruits       ┆ n   ┆ slice   ┆ head     ┆ tail     │
│ ---          ┆ --- ┆ ---     ┆ ---      ┆ ---      │
│ str          ┆ i64 ┆ str     ┆ str      ┆ str      │
╞══════════════╪═════╪═════════╪══════════╪══════════╡
│ pear         ┆ 1   ┆ ear     ┆ p        ┆ r        │
│ mango        ┆ -1  ┆ o       ┆ mang     ┆ ango     │
│ dragonfruit  ┆ 4   ┆ onfruit ┆ drag     ┆ ruit     │
│ passionfruit ┆ -4  ┆ ruit    ┆ passionf ┆ ionfruit │
└──────────────┴─────┴─────────┴──────────┴──────────┘

API documentation

In addition to the examples covered above, Polars offers various other string manipulation functions. To explore these additional methods, you can go to the API documentation of your chosen programming language for Polars.