polars.scan_ipc#

polars.scan_ipc(
source: str | Path | IO[bytes] | bytes | list[str] | list[Path] | list[IO[bytes]] | list[bytes],
*,
n_rows: int | None = None,
row_index_name: str | None = None,
row_index_offset: int = 0,
glob: bool = True,
storage_options: StorageOptionsDict | None = None,
credential_provider: CredentialProviderFunction | Literal['auto'] | None = 'auto',
hive_partitioning: bool | None = None,
hive_schema: SchemaDict | None = None,
try_parse_hive_dates: bool = True,
include_file_paths: str | None = None,
_record_batch_statistics: bool = False,
) → LazyFrame[source]#

Lazily read from an Arrow IPC (Feather v2) file or multiple files via glob patterns.

This allows the query optimizer to push down predicates and projections to the scan level, thereby potentially reducing memory overhead.

engine:In MemoryStreamingDistributed
Parameters:
source

Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the storage_options parameter.

n_rows

Stop reading from IPC file after reading n_rows.

row_index_name

If not None, this will insert a row index column with give name into the DataFrame

row_index_offset

Offset to start the row index column (only use if the name is set)

glob

Expand path given via globbing rules.

storage_options

Options that indicate how to connect to a cloud provider.

The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here:

  • aws

  • gcp

  • azure

  • Hugging Face (hf://): Accepts an API key under the token parameter: {'token': '...'}, or by setting the HF_TOKEN environment variable.

If storage_options is not provided, Polars will try to infer the information from environment variables.

credential_provider

Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

hive_partitioning

Infer statistics and schema from Hive partitioned URL and use them to prune reads. This is unset by default (i.e. None), meaning it is automatically enabled when a single directory is passed, and otherwise disabled.

hive_schema

The column names and data types of the columns by which the data is partitioned. If set to None (default), the schema of the Hive partitions is inferred.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

try_parse_hive_dates

Whether to try parsing hive values as date/datetime types.

include_file_paths

Include the path of the source file(s) as a column with this name.

On this page