polars.LazyFrame.sink_iceberg#

LazyFrame.sink_iceberg(
target: str | pyiceberg.table.Table,
*,
mode: Literal['append', 'overwrite'],
schema_mode: Literal['merge', 'overwrite'] | None = None,
snapshot_properties: dict[str, str] | None = None,
catalog: pyiceberg.catalog.Catalog | polars.io.iceberg.IcebergCatalogConfig | None = None,
storage_options: StorageOptionsDict | None = None,
compression: ParquetCompression = 'zstd',
compression_level: int | None = None,
row_group_size: int | None = None,
maintain_order: bool = True,
engine: EngineType = 'auto',
) → DataFrame[source]#

Sink a LazyFrame to an Iceberg table.

Warning

This functionality is currently considered unstable. It may be changed at any point without it being considered a breaking change.

Parameters:
target

A PyIceberg Table object, or a ‘namespace.table_name’ identifier string.

mode{‘append’, ‘overwrite’}

How to handle existing data.

  • If ‘append’, will add new data.

  • If ‘overwrite’, will replace table with new data.

schema_mode{‘merge’, ‘overwrite’}

How to handle differences between the incoming and table schemas.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

  • If ‘merge’, evolve the table schema with incoming fields and compatible type promotions.

  • If ‘overwrite’, replace the table schema with the incoming schema. This requires mode='overwrite'.

  • If None, require the schemas to match.

snapshot_properties

Custom properties to add to the Iceberg snapshot summary.

catalog

PyIceberg catalog to load the table from if the provided target was a table identifier.

storage_options

Extra options for the storage backends supported by pyiceberg. For cloud storages, this may include configurations for authentication etc.

More info is available here.

compression

Parquet compression codec.

compression_level

Compression level to use. Higher compression levels usually reduce file size at the expense of write throughput.

row_group_size

Row group size in number of rows.

maintain_order

Maintain the input row order in the written files. Setting this to False can improve throughput. For tables with a sort order, this controls the order of rows with equal sort keys; files always follow the table’s sort order.

engine

Engine used to produce rows for the local pyiceberg writer.

Returns:
DataFrame

Contains the new metadata path.

Notes

Partitioned writes support identity, year, month, day, hour, and truncate transforms. Truncation is supported for integer, long, string, and binary source columns, including fields nested within structs. Top-level partition source columns must use an Iceberg metrics mode that records lower and upper bounds (full or truncate). Bucket, void, and decimal truncation are not supported.

mode="overwrite" replaces all table data; dynamic partition overwrite is not supported. schema_mode="overwrite" is not supported for partitioned tables or tables with a sort order. Custom location providers are not supported.

The table’s default sort order is applied to incoming rows and recorded on each new data file. Existing files are not rewritten. Sorting materializes the incoming data in memory. Bucket sort transforms require pyiceberg-core; their sort keys are computed in batches while Parquet I/O remains native. This does not guarantee a globally sorted table scan.