1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436
//! # Polars: *<small>DataFrames in Rust</small>*
//!
//! Polars is a DataFrame library for Rust. It is based on [Apache Arrow](https://arrow.apache.org/)'s memory model.
//! Apache arrow provides very cache efficient columnar data structures and is becoming the defacto
//! standard for columnar data.
//!
//! ## Quickstart
//! We recommend to build your queries directly with [polars-lazy]. This allows you to combine
//! expression into powerful aggregations and column selections. All expressions are evaluated
//! in parallel and your queries are optimized just in time.
//!
//! [polars-lazy]: polars_lazy
//!
//! ```no_run
//! use polars::prelude::*;
//! # fn example() -> PolarsResult<()> {
//!
//! let lf1 = LazyFrame::scan_parquet("myfile_1.parquet", Default::default())?
//! .group_by([col("ham")])
//! .agg([
//! // expressions can be combined into powerful aggregations
//! col("foo")
//! .sort_by([col("ham").rank(Default::default(), None)], [false])
//! .last()
//! .alias("last_foo_ranked_by_ham"),
//! // every expression runs in parallel
//! col("foo").cum_min(false).alias("cumulative_min_per_group"),
//! // every expression runs in parallel
//! col("foo").reverse().implode().alias("reverse_group"),
//! ]);
//!
//! let lf2 = LazyFrame::scan_parquet("myfile_2.parquet", Default::default())?
//! .select([col("ham"), col("spam")]);
//!
//! let df = lf1
//! .join(lf2, [col("reverse")], [col("foo")], JoinArgs::new(JoinType::Left))
//! // now we finally materialize the result.
//! .collect()?;
//! # Ok(())
//! # }
//! ```
//!
//! This means that Polars data structures can be shared zero copy with processes in many different
//! languages.
//!
//! ## Tree Of Contents
//!
//! * [Cookbooks](#cookbooks)
//! * [Data structures](#data-structures)
//! - [DataFrame](#dataframe)
//! - [Series](#series)
//! - [ChunkedArray](#chunkedarray)
//! * [SIMD](#simd)
//! * [API](#api)
//! * [Expressions](#expressions)
//! * [Compile times](#compile-times)
//! * [Performance](#performance-and-string-data)
//! - [Custom allocator](#custom-allocator)
//! * [Config](#config-with-env-vars)
//! * [User guide](#user-guide)
//!
//! ## Cookbooks
//! See examples in the cookbooks:
//!
//! * [Eager](crate::docs::eager)
//! * [Lazy](crate::docs::lazy)
//!
//! ## Data Structures
//! The base data structures provided by polars are [`DataFrame`], [`Series`], and [`ChunkedArray<T>`].
//! We will provide a short, top-down view of these data structures.
//!
//! [`DataFrame`]: crate::frame::DataFrame
//! [`Series`]: crate::series::Series
//! [`ChunkedArray<T>`]: crate::chunked_array::ChunkedArray
//!
//! ### DataFrame
//! A [`DataFrame`] is a 2 dimensional data structure that is backed by a [`Series`], and it could be
//! seen as an abstraction on [`Vec<Series>`]. Operations that can be executed on [`DataFrame`] are very
//! similar to what is done in a `SQL` like query. You can `GROUP`, `JOIN`, `PIVOT` etc.
//!
//! [`Vec<Series>`]: std::vec::Vec
//!
//! ### Series
//! [`Series`] are the type agnostic columnar data representation of Polars. They provide many
//! operations out of the box, many via the [`Series`] series and
//! [`SeriesTrait`] trait. Whether or not an operation is provided
//! by a [`Series`] is determined by the operation. If the operation can be done without knowing the
//! underlying columnar type, this operation probably is provided by the [`Series`]. If not, you must
//! downcast to the typed data structure that is wrapped by the [`Series`]. That is the [`ChunkedArray<T>`].
//!
//! [`SeriesTrait`]: crate::series::SeriesTrait
//!
//! ### ChunkedArray
//! [`ChunkedArray<T>`] are wrappers around an arrow array, that can contain multiples chunks, e.g.
//! [`Vec<dyn ArrowArray>`]. These are the root data structures of Polars, and implement many operations.
//! Most operations are implemented by traits defined in [chunked_array::ops],
//! or on the [`ChunkedArray`] struct.
//!
//! [`ChunkedArray`]: crate::chunked_array::ChunkedArray
//!
//! ## SIMD
//! Polars / Arrow uses packed_simd to speed up kernels with SIMD operations. SIMD is an optional
//! `feature = "nightly"`, and requires a nightly compiler. If you don't need SIMD, **Polars runs on stable!**
//!
//! ## API
//! Polars supports an eager and a lazy API. The eager API directly yields results, but is overall
//! more verbose and less capable of building elegant composite queries. We recommend to use the Lazy API
//! whenever you can.
//!
//! As neither API is async they should be wrapped in _spawn_blocking_ when used in an async context
//! to avoid blocking the async thread pool of the runtime.
//!
//! ## Expressions
//! Polars has a powerful concept called expressions.
//! Polars expressions can be used in various contexts and are a functional mapping of
//! `Fn(Series) -> Series`, meaning that they have [`Series`] as input and [`Series`] as output.
//! By looking at this functional definition, we can see that the output of an [`Expr`] also can serve
//! as the input of an [`Expr`].
//!
//! [`Expr`]: polars_lazy::dsl::Expr
//!
//! That may sound a bit strange, so lets give an example. The following is an expression:
//!
//! `col("foo").sort().head(2)`
//!
//! The snippet above says select column `"foo"` then sort this column and then take first 2 values
//! of the sorted output.
//! The power of expressions is that every expression produces a new expression and that they can
//! be piped together.
//! You can run an expression by passing them on one of polars execution contexts.
//! Here we run two expressions in the **select** context:
//!
//! ```no_run
//! # use polars::prelude::*;
//! # fn example() -> PolarsResult<()> {
//! # let df = DataFrame::default();
//! df.lazy()
//! .select([
//! col("foo").sort(Default::default()).head(None),
//! col("bar").filter(col("foo").eq(lit(1))).sum(),
//! ])
//! .collect()?;
//! # Ok(())
//! # }
//! ```
//! All expressions are ran in parallel, meaning that separate polars expressions are embarrassingly parallel.
//! (Note that within an expression there may be more parallelization going on).
//!
//! Understanding polars expressions is most important when starting with the polars library. Read more
//! about them in the [user guide](https://docs.pola.rs/user-guide/concepts/expressions).
//!
//! ### Eager
//! Read more in the pages of the following data structures /traits.
//!
//! * [DataFrame struct](crate::frame::DataFrame)
//! * [Series struct](crate::series::Series)
//! * [Series trait](crate::series::SeriesTrait)
//! * [ChunkedArray struct](crate::chunked_array::ChunkedArray)
//! * [ChunkedArray operations traits](crate::chunked_array::ops)
//!
//! ### Lazy
//! Unlock full potential with lazy computation. This allows query optimizations and provides Polars
//! the full query context so that the fastest algorithm can be chosen.
//!
//! **[Read more in the lazy module.](polars_lazy)**
//!
//! ## Compile times
//! A DataFrame library typically consists of
//!
//! * Tons of features
//! * A lot of datatypes
//!
//! Both of these really put strain on compile times. To keep Polars lean, we make both **opt-in**,
//! meaning that you only pay the compilation cost, if you need it.
//!
//! ## Compile times and opt-in features
//! The opt-in features are (not including dtype features):
//!
//! * `performant` - Longer compile times more fast paths.
//! * `lazy` - Lazy API
//! - `regex` - Use regexes in [column selection]
//! - `dot_diagram` - Create dot diagrams from lazy logical plans.
//! * `sql` - Pass SQL queries to polars.
//! * `streaming` - Be able to process datasets that are larger than RAM.
//! * `random` - Generate arrays with randomly sampled values
//! * `ndarray`- Convert from [`DataFrame`] to [ndarray](https://docs.rs/ndarray/)
//! * `temporal` - Conversions between [Chrono](https://docs.rs/chrono/) and Polars for temporal data types
//! * `timezones` - Activate timezone support.
//! * `strings` - Extra string utilities for [`StringChunked`] //! - `string_pad` - `zfill`, `ljust`, `rjust`
//! - `string_to_integer` - `parse_int`
//! * `object` - Support for generic ChunkedArrays called [`ObjectChunked<T>`] (generic over `T`).
//! These are downcastable from Series through the [Any](https://doc.rust-lang.org/std/any/index.html) trait.
//! * Performance related:
//! - `nightly` - Several nightly only features such as SIMD and specialization.
//! - `performant` - more fast paths, slower compile times.
//! - `bigidx` - Activate this feature if you expect >> 2^32 rows. This has not been needed by anyone.
//! This allows polars to scale up way beyond that by using `u64` as an index.
//! Polars will be a bit slower with this feature activated as many data structures
//! are less cache efficient.
//! - `cse` - Activate common subplan elimination optimization
//! * IO related:
//! - `serde` - Support for [serde](https://crates.io/crates/serde) serialization and deserialization.
//! Can be used for JSON and more serde supported serialization formats.
//! - `serde-lazy` - Support for [serde](https://crates.io/crates/serde) serialization and deserialization.
//! Can be used for JSON and more serde supported serialization formats.
//! - `parquet` - Read Apache Parquet format
//! - `json` - JSON serialization
//! - `ipc` - Arrow's IPC format serialization
//! - `decompress` - Automatically infer compression of csvs and decompress them.
//! Supported compressions:
//! * zip
//! * gzip
//!
//! [`StringChunked`]: crate::datatypes::StringChunked
//! [column selection]: polars_lazy::dsl::col
//! [`ObjectChunked<T>`]: polars_core::datatypes::ObjectChunked
//!
//!
//! * [`DataFrame`] operations:
//! - `dynamic_group_by` - Groupby based on a time window instead of predefined keys.
//! Also activates rolling window group by operations.
//! - `sort_multiple` - Allow sorting a [`DataFrame`] on multiple columns
//! - `rows` - Create [`DataFrame`] from rows and extract rows from [`DataFrame`]s.
//! And activates `pivot` and `transpose` operations
//! - `asof_join` - Join ASOF, to join on nearest keys instead of exact equality match.
//! - `cross_join` - Create the Cartesian product of two [`DataFrame`]s.
//! - `semi_anti_join` - SEMI and ANTI joins.
//! - `row_hash` - Utility to hash [`DataFrame`] rows to [`UInt64Chunked`]
//! - `diagonal_concat` - Concat diagonally thereby combining different schemas.
//! - `dataframe_arithmetic` - Arithmetic on ([`Dataframe`] and [`DataFrame`]s) and ([`DataFrame`] on [`Series`])
//! - `partition_by` - Split into multiple [`DataFrame`]s partitioned by groups.
//! * [`Series`]/[`Expr`] operations:
//! - `is_in` - Check for membership in [`Series`].
//! - `zip_with` - [Zip two Series/ ChunkedArrays](crate::chunked_array::ops::ChunkZip).
//! - `round_series` - round underlying float types of [`Series`].
//! - `repeat_by` - [Repeat element in an Array N times, where N is given by another array.
//! - `is_first_distinct` - Check if element is first unique value.
//! - `is_last_distinct` - Check if element is last unique value.
//! - `is_between` - Check if this expression is between the given lower and upper bounds.
//! - `checked_arithmetic` - checked arithmetic/ returning [`None`] on invalid operations.
//! - `dot_product` - Dot/inner product on [`Series`] and [`Expr`].
//! - `concat_str` - Concat string data in linear time.
//! - `reinterpret` - Utility to reinterpret bits to signed/unsigned
//! - `take_opt_iter` - Take from a [`Series`] with [`Iterator<Item=Option<usize>>`](std::iter::Iterator).
//! - `mode` - [Return the most occurring value(s)](polars_ops::chunked_array::mode)
//! - `cum_agg` - [`cum_sum`], [`cum_min`], [`cum_max`] aggregation.
//! - `rolling_window` - rolling window functions, like [`rolling_mean`]
//! - `interpolate` [interpolate None values](polars_ops::series::interpolate())
//! - `extract_jsonpath` - [Run jsonpath queries on StringChunked](https://goessner.net/articles/JsonPath/)
//! - `list` - List utils.
//! - `list_gather` take sublist by multiple indices
//! - `rank` - Ranking algorithms.
//! - `moment` - kurtosis and skew statistics
//! - `ewma` - Exponential moving average windows
//! - `abs` - Get absolute values of [`Series`].
//! - `arange` - Range operation on [`Series`].
//! - `product` - Compute the product of a [`Series`].
//! - `diff` - [`diff`] operation.
//! - `pct_change` - Compute change percentages.
//! - `unique_counts` - Count unique values in expressions.
//! - `log` - Logarithms for [`Series`].
//! - `list_to_struct` - Convert [`List`] to [`Struct`] dtypes.
//! - `list_count` - Count elements in lists.
//! - `list_eval` - Apply expressions over list elements.
//! - `list_sets` - Compute UNION, INTERSECTION, and DIFFERENCE on list types.
//! - `cumulative_eval` - Apply expressions over cumulatively increasing windows.
//! - `arg_where` - Get indices where condition holds.
//! - `search_sorted` - Find indices where elements should be inserted to maintain order.
//! - `offset_by` - Add an offset to dates that take months and leap years into account.
//! - `trigonometry` - Trigonometric functions.
//! - `sign` - Compute the element-wise sign of a [`Series`].
//! - `propagate_nans` - NaN propagating min/max aggregations.
//! - `extract_groups` - Extract multiple regex groups from strings.
//! - `cov` - Covariance and correlation functions.
//! - `find_many` - Find/replace multiple string patterns at once.
//! * [`DataFrame`] pretty printing
//! - `fmt` - Activate [`DataFrame`] formatting
//!
//! [`UInt64Chunked`]: crate::datatypes::UInt64Chunked
//! [`cum_sum`]: polars_ops::prelude::cum_sum
//! [`cum_min`]: polars_ops::prelude::cum_min
//! [`cum_max`]: polars_ops::prelude::cum_max
//! [`rolling_mean`]: crate::series::Series#method.rolling_mean
//! [`diff`]: polars_ops::prelude::diff
//! [`List`]: crate::datatypes::DataType::List
//! [`Struct`]: crate::datatypes::DataType::Struct
//!
//! ## Compile times and opt-in data types
//! As mentioned above, Polars [`Series`] are wrappers around
//! [`ChunkedArray<T>`] without the generic parameter `T`.
//! To get rid of the generic parameter, all the possible value of `T` are compiled
//! for [`Series`]. This gets more expensive the more types you want for a [`Series`]. In order to reduce
//! the compile times, we have decided to default to a minimal set of types and make more [`Series`] types
//! opt-in.
//!
//! Note that if you get strange compile time errors, you probably need to opt-in for that [`Series`] dtype.
//! The opt-in dtypes are:
//!
//! | data type | feature flag |
//! |-------------------------|-------------------|
//! | Date | dtype-date |
//! | Datetime | dtype-datetime |
//! | Time | dtype-time |
//! | Duration | dtype-duration |
//! | Int8 | dtype-i8 |
//! | Int16 | dtype-i16 |
//! | UInt8 | dtype-u8 |
//! | UInt16 | dtype-u16 |
//! | Categorical | dtype-categorical |
//! | Struct | dtype-struct |
//!
//!
//! Or you can choose on of the preconfigured pre-sets.
//!
//! * `dtype-full` - all opt-in dtypes.
//! * `dtype-slim` - slim preset of opt-in dtypes.
//!
//! ## Performance
//! To gains most performance out of Polars we recommend compiling on a nightly compiler
//! with the features `simd` and `performant` activated. The activated cpu features also influence
//! the amount of simd acceleration we can use.
//!
//! See this the features we activate for our python builds, or if you just run locally and want to
//! use all available features on your cpu, set `RUSTFLAGS='-C target-cpu=native'`.
//!
//! ### Custom allocator
//! An OLAP query engine does a lot of heap allocations. It is recommended to use a custom
//! allocator, (we have found this to have up to ~25% runtime influence).
//! [JeMalloc](https://crates.io/crates/jemallocator) and
//! [Mimalloc](https://crates.io/crates/mimalloc) for instance, show a significant
//! performance gain in runtime as well as memory usage.
//!
//! #### Jemalloc Usage
//! ```ignore
//! use jemallocator::Jemalloc;
//!
//! #[global_allocator]
//! static GLOBAL: Jemalloc = Jemalloc;
//! ```
//!
//! #### Cargo.toml
//! ```toml
//! [dependencies]
//! jemallocator = { version = "*" }
//! ```
//!
//! #### Mimalloc Usage
//!
//! ```ignore
//! use mimalloc::MiMalloc;
//!
//! #[global_allocator]
//! static GLOBAL: MiMalloc = MiMalloc;
//! ```
//!
//! #### Cargo.toml
//! ```toml
//! [dependencies]
//! mimalloc = { version = "*", default-features = false }
//! ```
//!
//! #### Notes
//! [Benchmarks](https://github.com/pola-rs/polars/pull/3108) have shown that on Linux and macOS JeMalloc
//! outperforms Mimalloc on all tasks and is therefore the default allocator used for the Python bindings on Unix platforms.
//!
//! ## Config with ENV vars
//!
//! * `POLARS_FMT_TABLE_FORMATTING` -> define styling of tables using any of the following options (default = UTF8_FULL_CONDENSED). These options are defined by comfy-table which provides examples for each at <https://github.com/Nukesor/comfy-table/blob/main/src/style/presets.rs>
//! * `ASCII_FULL`
//! * `ASCII_FULL_CONDENSED`
//! * `ASCII_NO_BORDERS`
//! * `ASCII_BORDERS_ONLY`
//! * `ASCII_BORDERS_ONLY_CONDENSED`
//! * `ASCII_HORIZONTAL_ONLY`
//! * `ASCII_MARKDOWN`
//! * `UTF8_FULL`
//! * `UTF8_FULL_CONDENSED`
//! * `UTF8_NO_BORDERS`
//! * `UTF8_BORDERS_ONLY`
//! * `UTF8_HORIZONTAL_ONLY`
//! * `NOTHING`
//! * `POLARS_FMT_TABLE_CELL_ALIGNMENT` -> define cell alignment using any of the following options (default = LEFT):
//! * `LEFT`
//! * `CENTER`
//! * `RIGHT`
//! * `POLARS_FMT_TABLE_DATAFRAME_SHAPE_BELOW` -> print shape information below the table.
//! * `POLARS_FMT_TABLE_HIDE_COLUMN_NAMES` -> hide table column names.
//! * `POLARS_FMT_TABLE_HIDE_COLUMN_DATA_TYPES` -> hide data types for columns.
//! * `POLARS_FMT_TABLE_HIDE_COLUMN_SEPARATOR` -> hide separator that separates column names from rows.
//! * `POLARS_FMT_TABLE_HIDE_DATAFRAME_SHAPE_INFORMATION"` -> omit table shape information.
//! * `POLARS_FMT_TABLE_INLINE_COLUMN_DATA_TYPE` -> put column data type on the same line as the column name.
//! * `POLARS_FMT_TABLE_ROUNDED_CORNERS` -> apply rounded corners to UTF8-styled tables.
//! * `POLARS_FMT_MAX_COLS` -> maximum number of columns shown when formatting DataFrames.
//! * `POLARS_FMT_MAX_ROWS` -> maximum number of rows shown when formatting DataFrames, `-1` to show all.
//! * `POLARS_FMT_STR_LEN` -> maximum number of characters printed per string value.
//! * `POLARS_TABLE_WIDTH` -> width of the tables used during DataFrame formatting.
//! * `POLARS_MAX_THREADS` -> maximum number of threads used to initialize thread pool (on startup).
//! * `POLARS_VERBOSE` -> print logging info to stderr.
//! * `POLARS_NO_PARTITION` -> polars may choose to partition the group_by operation, based on data
//! cardinality. Setting this env var will turn partitioned group_by's off.
//! * `POLARS_PARTITION_UNIQUE_COUNT` -> at which (estimated) key count a partitioned group_by should run.
//! defaults to `1000`, any higher cardinality will run default group_by.
//! * `POLARS_FORCE_PARTITION` -> force partitioned group_by if the keys and aggregations allow it.
//! * `POLARS_ALLOW_EXTENSION` -> allows for [`ObjectChunked<T>`] to be used in arrow, opening up possibilities like using
//! `T` in complex lazy expressions. However this does require `unsafe` code allow this.
//! * `POLARS_NO_PARQUET_STATISTICS` -> if set, statistics in parquet files are ignored.
//! * `POLARS_PANIC_ON_ERR` -> panic instead of returning an Error.
//! * `POLARS_NO_CHUNKED_JOIN` -> force rechunk before joins.
//!
//! ## User guide
//!
//! If you want to read more, check the [user guide](https://docs.pola.rs/).
#![cfg_attr(docsrs, feature(doc_auto_cfg))]
#![allow(ambiguous_glob_reexports)]
pub mod docs;
#[doc(hidden)]
pub mod export;
pub mod prelude;
#[cfg(feature = "sql")]
pub mod sql;
pub use polars_core::{
apply_method_all_arrow_series, chunked_array, datatypes, df, error, frame, functions, series,
testing,
};
#[cfg(feature = "dtype-categorical")]
pub use polars_core::{enable_string_cache, using_string_cache};
#[cfg(feature = "polars-io")]
pub use polars_io as io;
#[cfg(feature = "lazy")]
pub use polars_lazy as lazy;
#[cfg(feature = "temporal")]
pub use polars_time as time;
/// Polars crate version
pub const VERSION: &str = env!("CARGO_PKG_VERSION");