Strings
The following section discusses operations performed on string data, which is a frequently used data
type when working with dataframes. String processing functions are available in the namespace str.
Working with strings in other dataframe libraries can be highly inefficient due to the fact that strings have unpredictable lengths. Polars mitigates these inefficiencies by following the Arrow Columnar Format specification, so you can write performant data queries on string data too.
The string namespace
When working with string data you will likely need to access the namespace str, which aggregates
40+ functions that let you work with strings. As an example of how to access functions from within
that namespace, the snippet below shows how to compute the length of the strings in a column in
terms of the number of bytes and the number of characters:
import polars as pl
df = pl.DataFrame(
{
"language": ["English", "Dutch", "Portuguese", "Finish"],
"fruit": ["pear", "peer", "pêra", "päärynä"],
}
)
result = df.with_columns(
pl.col("fruit").str.len_bytes().alias("byte_count"),
pl.col("fruit").str.len_chars().alias("letter_count"),
)
print(result)
// Contribute the Rust translation of the Python example by opening a PR.
shape: (4, 4)
┌────────────┬─────────┬────────────┬──────────────┐
│ language ┆ fruit ┆ byte_count ┆ letter_count │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ u32 ┆ u32 │
╞════════════╪═════════╪════════════╪══════════════╡
│ English ┆ pear ┆ 4 ┆ 4 │
│ Dutch ┆ peer ┆ 4 ┆ 4 │
│ Portuguese ┆ pêra ┆ 5 ┆ 4 │
│ Finish ┆ päärynä ┆ 10 ┆ 7 │
└────────────┴─────────┴────────────┴──────────────┘
Note
If you are working exclusively with ASCII text, then the results of the two computations will be the same and using len_bytes is recommended since it is faster.
Parsing strings
Polars offers multiple methods for checking and parsing elements of a string column, namely checking for the existence of given substrings or patterns, and counting, extracting, or replacing, them. We will demonstrate some of these operations in the upcoming examples.
Check for the existence of a pattern
We can use the function contains to check for the presence of a pattern within a string. By
default, the argument to the function contains is interpreted as a regular expression. If you want
to specify a literal substring, set the parameter literal to True.
For the special cases where you want to check if the strings start or end with a fixed substring,
you can use the functions starts_with or ends_with, respectively.
str.contains · str.starts_with · str.ends_with
result = df.select(
pl.col("fruit"),
pl.col("fruit").str.starts_with("p").alias("starts_with_p"),
pl.col("fruit").str.contains("p..r").alias("p..r"),
pl.col("fruit").str.contains("e+").alias("e+"),
pl.col("fruit").str.ends_with("r").alias("ends_with_r"),
)
print(result)
str.contains · str.starts_with · str.ends_with · Available on feature regex
// Contribute the Rust translation of the Python example by opening a PR.
shape: (4, 5)
┌─────────┬───────────────┬───────┬───────┬─────────────┐
│ fruit ┆ starts_with_p ┆ p..r ┆ e+ ┆ ends_with_r │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ bool ┆ bool ┆ bool ┆ bool │
╞═════════╪═══════════════╪═══════╪═══════╪═════════════╡
│ pear ┆ true ┆ true ┆ true ┆ true │
│ peer ┆ true ┆ true ┆ true ┆ true │
│ pêra ┆ true ┆ false ┆ false ┆ false │
│ päärynä ┆ true ┆ true ┆ false ┆ false │
└─────────┴───────────────┴───────┴───────┴─────────────┘
Regex specification
Polars relies on the Rust crate regex to work with regular expressions, so you may need to
refer to the syntax documentation to see what features
and flags are supported. In particular, note that the flavor of regex supported by Polars is
different from Python's module re.
Extract a pattern
The function extract allows us to extract patterns from the string values in a column. The
function extract accepts a regex pattern with one or more capture groups and extracts the capture
group specified as the second argument.
df = pl.DataFrame(
{
"urls": [
"http://vote.com/ballon_dor?candidate=messi&ref=polars",
"http://vote.com/ballon_dor?candidat=jorginho&ref=polars",
"http://vote.com/ballon_dor?candidate=ronaldo&ref=polars",
]
}
)
result = df.select(
pl.col("urls").str.extract(r"candidate=(\w+)", group_index=1),
)
print(result)
// Contribute the Rust translation of the Python example by opening a PR.
shape: (3, 1)
┌─────────┐
│ urls │
│ --- │
│ str │
╞═════════╡
│ messi │
│ null │
│ ronaldo │
└─────────┘
To extract all occurrences of a pattern within a string, we can use the function extract_all. In
the example below, we extract all numbers from a string using the regex pattern (\d+), which
matches one or more digits. The resulting output of the function extract_all is a list containing
all instances of the matched pattern within the string.
df = pl.DataFrame({"text": ["123 bla 45 asd", "xyz 678 910t"]})
result = df.select(
pl.col("text").str.extract_all(r"(\d+)").alias("extracted_nrs"),
)
print(result)
// Contribute the Rust translation of the Python example by opening a PR.
shape: (2, 1)
┌────────────────┐
│ extracted_nrs │
│ --- │
│ list[str] │
╞════════════════╡
│ ["123", "45"] │
│ ["678", "910"] │
└────────────────┘
Replace a pattern
Akin to the functions extract and extract_all, Polars provides the functions replace and
replace_all. These accept a regex pattern or a literal substring (if the parameter literal is
set to True) and perform the replacements specified. The function replace will make at most one
replacement whereas the function replace_all will make all the non-overlapping replacements it
finds.
df = pl.DataFrame({"text": ["123abc", "abc456"]})
result = df.with_columns(
pl.col("text").str.replace(r"\d", "-"),
pl.col("text").str.replace_all(r"\d", "-").alias("text_replace_all"),
)
print(result)
str.replace · str.replace_all · Available on feature regex
// Contribute the Rust translation of the Python example by opening a PR.
shape: (2, 2)
┌────────┬──────────────────┐
│ text ┆ text_replace_all │
│ --- ┆ --- │
│ str ┆ str │
╞════════╪══════════════════╡
│ -23abc ┆ ---abc │
│ abc-56 ┆ abc--- │
└────────┴──────────────────┘
Modifying strings
Case conversion
Converting the casing of a string is a common operation and Polars supports it out of the box with
the functions to_lowercase, to_titlecase, and to_uppercase:
str.to_lowercase · str.to_titlecase · str.to_uppercase
addresses = pl.DataFrame(
{
"addresses": [
"128 PERF st",
"Rust blVD, 158",
"PoLaRs Av, 12",
"1042 Query sq",
]
}
)
addresses = addresses.select(
pl.col("addresses").alias("originals"),
pl.col("addresses").str.to_titlecase(),
pl.col("addresses").str.to_lowercase().alias("lower"),
pl.col("addresses").str.to_uppercase().alias("upper"),
)
print(addresses)
str.to_lowercase · str.to_titlecase · str.to_uppercase · Available on feature nightly
// Contribute the Rust translation of the Python example by opening a PR.
shape: (4, 4)
┌────────────────┬────────────────┬────────────────┬────────────────┐
│ originals ┆ addresses ┆ lower ┆ upper │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str │
╞════════════════╪════════════════╪════════════════╪════════════════╡
│ 128 PERF st ┆ 128 Perf St ┆ 128 perf st ┆ 128 PERF ST │
│ Rust blVD, 158 ┆ Rust Blvd, 158 ┆ rust blvd, 158 ┆ RUST BLVD, 158 │
│ PoLaRs Av, 12 ┆ Polars Av, 12 ┆ polars av, 12 ┆ POLARS AV, 12 │
│ 1042 Query sq ┆ 1042 Query Sq ┆ 1042 query sq ┆ 1042 QUERY SQ │
└────────────────┴────────────────┴────────────────┴────────────────┘
Stripping characters from the ends
Polars provides five functions in the namespace str that let you strip characters from the ends of
the string:
| Function | Behaviour |
|---|---|
strip_chars |
Removes leading and trailing occurrences of the characters specified. |
strip_chars_end |
Removes trailing occurrences of the characters specified. |
strip_chars_start |
Removes leading occurrences of the characters specified. |
strip_prefix |
Removes an exact substring prefix if present. |
strip_suffix |
Removes an exact substring suffix if present. |
??? info "Similarity to Python string methods" strip_chars is similar to Python's string method
strip and strip_prefix/strip_suffix are similar to Python's string methods removeprefix and
strip_suffix, respectively.
It is important to understand that the first three functions interpret their string argument as a
set of characters whereas the functions strip_prefix and strip_suffix do interpret their string
argument as a literal string.
str.strip_chars · str.strip_chars_end · str.strip_chars_start · str.strip_prefix · str.strip_suffix
addr = pl.col("addresses")
chars = ", 0123456789"
result = addresses.select(
addr.str.strip_chars(chars).alias("strip"),
addr.str.strip_chars_end(chars).alias("end"),
addr.str.strip_chars_start(chars).alias("start"),
addr.str.strip_prefix("128 ").alias("prefix"),
addr.str.strip_suffix(", 158").alias("suffix"),
)
print(result)
str.strip_chars · str.strip_chars_end · str.strip_chars_start · str.strip_prefix · str.strip_suffix
// Contribute the Rust translation of the Python example by opening a PR.
shape: (4, 5)
┌───────────┬───────────────┬────────────────┬────────────────┬───────────────┐
│ strip ┆ end ┆ start ┆ prefix ┆ suffix │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ str │
╞═══════════╪═══════════════╪════════════════╪════════════════╪═══════════════╡
│ Perf St ┆ 128 Perf St ┆ Perf St ┆ Perf St ┆ 128 Perf St │
│ Rust Blvd ┆ Rust Blvd ┆ Rust Blvd, 158 ┆ Rust Blvd, 158 ┆ Rust Blvd │
│ Polars Av ┆ Polars Av ┆ Polars Av, 12 ┆ Polars Av, 12 ┆ Polars Av, 12 │
│ Query Sq ┆ 1042 Query Sq ┆ Query Sq ┆ 1042 Query Sq ┆ 1042 Query Sq │
└───────────┴───────────────┴────────────────┴────────────────┴───────────────┘
If no argument is provided, the three functions strip_chars, strip_chars_end, and
strip_chars_start, remove whitespace by default.
Slicing
Besides extracting substrings as specified by patterns, you can also slice
strings at specified offsets to produce substrings. The general-purpose function for slicing is
slice and it takes the starting offset and the optional length of the slice. If the length of
the slice is not specified or if it's past the end of the string, Polars slices the string all the
way to the end.
The functions head and tail are specialised versions used for slicing the beginning and end of a
string, respectively.
str.slice · str.head · str.tail
df = pl.DataFrame(
{
"fruits": ["pear", "mango", "dragonfruit", "passionfruit"],
"n": [1, -1, 4, -4],
}
)
result = df.with_columns(
pl.col("fruits").str.slice(pl.col("n")).alias("slice"),
pl.col("fruits").str.head(pl.col("n")).alias("head"),
pl.col("fruits").str.tail(pl.col("n")).alias("tail"),
)
print(result)
str.str_slice · str.str_head · str.str_tail
// Contribute the Rust translation of the Python example by opening a PR.
shape: (4, 5)
┌──────────────┬─────┬─────────┬──────────┬──────────┐
│ fruits ┆ n ┆ slice ┆ head ┆ tail │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ str ┆ str ┆ str │
╞══════════════╪═════╪═════════╪══════════╪══════════╡
│ pear ┆ 1 ┆ ear ┆ p ┆ r │
│ mango ┆ -1 ┆ o ┆ mang ┆ ango │
│ dragonfruit ┆ 4 ┆ onfruit ┆ drag ┆ ruit │
│ passionfruit ┆ -4 ┆ ruit ┆ passionf ┆ ionfruit │
└──────────────┴─────┴─────────┴──────────┴──────────┘
API documentation
In addition to the examples covered above, Polars offers various other string manipulation functions. To explore these additional methods, you can go to the API documentation of your chosen programming language for Polars.