polars.read_csv#

polars.read_csv( source: str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes], *, columns: Sequence[int] | Sequence[str] | None = None, has_header: bool = True, separator: str = ',', comment_prefix: str | None = None, quote_char: str | None = '"', skip_rows: int = 0, skip_lines: int = 0, schema: SchemaDict | None = None, schema_overrides: SchemaDict | Sequence[PolarsDataType] | None = None, null_values: str | Sequence[str] | dict[str, str] | None = None, empty_string_is_null: bool = True, ignore_errors: bool = False, with_column_names: Callable[[list[str]], list[str]] | None = None, infer_schema: bool = True, infer_schema_length: int | None = 100, infer_schema_files: int = 10, n_rows: int | None = None, encoding: CsvEncoding | str = 'utf8', low_memory: bool = False, skip_rows_after_header: int = 0, row_index_name: str | None = None, row_index_offset: int = 0, try_parse_dates: bool = False, eol_char: str = '\n', new_columns: Sequence[str] | None = None, truncate_ragged_lines: bool | None = None, raise_if_empty: bool | None = None, decimal_comma: bool = False, glob: bool = True, storage_options: StorageOptionsDict | None = None, credential_provider: CredentialProviderFunction | Literal['auto'] | None = 'auto', include_file_paths: str | None = None, extra_columns: Literal['ignore', 'raise'] | None = None, missing_columns: Literal['insert', 'raise'] | None = None, use_pyarrow: bool = False, ) → DataFrame[source]#

Read a CSV file into a DataFrame.

Polars expects CSV data to strictly conform to RFC 4180, unless documented otherwise. Malformed data, though common, may lead to undefined behavior.

Changed in version 0.20.31: The dtypes parameter was renamed schema_overrides.

Changed in version 0.20.4: * The row_count_name parameter was renamed row_index_name. * The row_count_offset parameter was renamed row_index_offset.

Parameters:

source

Path(s) to a file or a file-like object (by “file-like object” we refer to objects that have a read() method, such as a file handler like the builtin open function, or a BytesIO instance). If fsspec is installed, it might be used to open remote files. Compressed files (gzip and zstd) are supported when reading from a path or a file-like object. For file-like objects, the stream position may not be updated accordingly after reading.

columns

Columns to select. Accepts a list of column indices (starting at zero) or a list of column names.

has_header

Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: column_x, with x being an enumeration over every column in the dataset, starting at 0.

separator

Single byte character to use as separator in the file.

comment_prefix

A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are # and //.

quote_char

Single byte character used for csv quoting, default = ". Set to None to turn off special handling and escaping of quotes.

skip_rows

Start reading after skip_rows rows. The header will be parsed at this offset. Note that we respect CSV escaping/comments when skipping rows. If you want to skip by newline char only, use skip_lines.

skip_lines

Start reading after skip_lines lines. The header will be parsed at this offset. Note that CSV escaping will not be respected when skipping lines. If you want to skip valid CSV rows, use skip_rows.

schema

Provide the schema. This means that polars doesn’t do schema inference. This argument expects the complete schema, whereas schema_overrides can be used to partially overwrite a schema. Note that the order of the columns in the provided schema must match the order of the columns in the CSV being read.

schema_overrides

Overwrite dtypes during inference; should be a {colname:dtype,} dict or, if providing a list of strings to new_columns, a list of dtypes of the same length.

null_values

Values to interpret as null values. You can provide a:

str: All values equal to this string will be null.
List[str]: All values equal to any string in this list will be null.
Dict[str, str]: A dictionary that maps column name to a null value string.

empty_string_is_null

By default a missing string value is considered to be null. If empty_string_is_null is set to False, missing string values are considered to decoded as empty strings.

ignore_errors

Try to keep reading lines if some lines yield errors. First try infer_schema=False to read all columns as pl.String to check which values might cause an issue.

with_column_names

Apply a function over the column names just in time (when they are determined); this function will receive (and should return) a list of column names.

infer_schema

When True, the schema is inferred from the data using the first infer_schema_length rows. When False, the schema is not inferred and will be pl.String if not specified in schema or schema_overrides.

infer_schema_length

The maximum number of rows to scan for schema inference. If set to None, the full data will be scanned into memory (this is slow). Alternatively set infer_schema=False to read all columns as pl.String.

infer_schema_files

How many files to use when inferring schema.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

n_rows

Stop reading from CSV file after reading n_rows.

encoding{‘utf8’, ‘utf8-lossy’, ‘windows-1252’, ‘windows-1252-lossy’, …}

Lossy means that invalid utf8 values are replaced with � characters. Defaults to “utf8”.

low_memory

Reduce memory pressure at the expense of performance.

skip_rows_after_header

Skip this number of rows when the header is parsed.

row_index_name

If not None, this will insert a row index column with the given name into the DataFrame.

row_index_offset

Offset to start the row index column (only used if the name is set).

try_parse_dates

Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type pl.String.

eol_char

Single byte end of line character (default: \n). When encountering a file with windows line endings (\r\n), one can go with the default \n. The extra \r will be removed when processed.

new_columns

Provide an explicit list of string column names to use (for example, when scanning a headerless CSV file). If the given list is shorter than the width of the DataFrame the remaining columns will have their original name.

raise_if_empty

Control behavior when scanning empty files:

True: Raise a NoDataError.
False: Return 0 rows.

This defaults to False if has_header=False and a schema is specified, and True otherwise.

truncate_ragged_lines

Truncate lines that are longer than the schema.

decimal_comma

Parse floats using a comma as the decimal separator instead of a period.

glob

Expand path given via globbing rules.

storage_options

Options that indicate how to connect to a cloud provider.

The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here:

aws
gcp
azure
Hugging Face (hf://): Accepts an API key under the token parameter: {'token': '...'}, or by setting the HF_TOKEN environment variable.

If storage_options is not provided, Polars will try to infer the information from environment variables.

credential_provider

Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

include_file_paths

Include the path of the source file(s) as a column with this name.

extra_columns

Configuration for behavior when extra columns outside of the defined schema are encountered in the data:

ignore: Silently ignores.
raise: Raises an error.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

missing_columns

Configuration for behavior when columns defined in the schema are missing from the data:

"insert": Insert the missing columns with NULL values.
"raise": Raise an error.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

use_pyarrow

Try to use pyarrow’s native CSV parser. This will always parse dates, even if try_parse_dates=False. This is not always possible. The set of arguments given to this function determines if it is possible to use pyarrow’s native parser. Note that pyarrow and polars may have a different strategy regarding type inference.

Returns:

DataFrame

Warning

Calling read_csv().lazy() is an antipattern as this forces Polars to materialize a full csv file and therefore cannot push any optimizations into the reader. Therefore always prefer scan_csv if you want to work with LazyFrame s.