polars.SQLContext.register_catalog#

SQLContext.register_catalog(
name: str,
catalog: pyiceberg.catalog.Catalog | IcebergCatalogConfig | str | None = None,
*,
default_namespace: str | Sequence[str] | None = None,
default: bool = False,
storage_options: StorageOptionsDict | None = None,
) → Self[source]#

Attach an Iceberg catalog, so that queries can read its tables by name.

Warning

This functionality is considered unstable. It may be changed at any point without it being considered a breaking change.

A table name that starts with name refers to a table of this catalog (name.namespace.table). With default=True, so does a table name that starts with no attached catalog (namespace.table). A table name without a namespace gets default_namespace. Registered tables and CTEs take precedence over catalog tables, and an attached catalog takes precedence over a namespace of the same name in the default catalog.

Catalog tables are scanned lazily, as with scan_iceberg(): a table is loaded when the query’s schema is first needed, and a table that does not exist raises then. The rest of the table name is passed to the catalog as written; catalog identifiers are case-sensitive.

Parameters:
name

The catalog’s name in SQL. Registering the same name again replaces the catalog: catalog, default_namespace and storage_options take the new values, including their defaults when omitted, while the catalog stays the default if it was (see default). A name that differs from an attached catalog’s only in case is an error.

catalog

A PyIceberg catalog, an IcebergCatalogConfig, or a catalog name. None means use the name parameter as a catalog name.

default_namespace

The namespace of table names without one: a string (split on “.”) or a sequence of namespace parts.

default

Make this catalog the default: table names that start with no attached catalog refer to it. At most one catalog is the default; a later default replaces it. False never removes the default: re-registering the default catalog keeps it the default, and re-registering another catalog leaves it. Only unregister_catalog leaves no default.

storage_options

Options for the storage of the catalog’s tables, as for scan_iceberg(); CREATE TABLE ... AS also writes the rows with them. Pass the storage credentials here when a plan is serialized after its tables loaded: the serialized plan then reads the tables’ metadata directly, with these options rather than the catalog’s properties. DROP TABLE ... PURGE does not use them: the catalog deletes the files with its own properties.

Notes

Catalog tables can be created and dropped with SQL, at execute(). The table name resolves as in a query: with a default catalog, CREATE TABLE tpch.t ... creates tpch.t in it. CREATE TEMP TABLE creates a registered table instead.

  • CREATE TABLE <name> (<column>, ...) creates an empty table, and CREATE TABLE <name> AS <query> one holding the query’s rows. The rows are committed together with the create, so a failed query creates no table; a catalog that cannot stage a create gets an empty table first, dropped again if filling it fails, unless another writer changed it. A registered table of that name is an error, as it would hide the new table. See CREATE TABLE for the options.

  • DROP TABLE [IF EXISTS] <name> [PURGE] drops a registered table of that name, else the catalog table, permanently. PURGE also deletes the table’s data files.

  • CREATE SCHEMA [IF NOT EXISTS] <name> [WITH (...)] creates a namespace, and DROP SCHEMA [IF EXISTS] <name> drops an empty one. A name starting with an attached catalog is a namespace of it, any other name one of the default catalog, not under default_namespace. See CREATE SCHEMA.

  • ALTER TABLE [IF EXISTS] <name> <operation> changes a catalog table’s top-level columns or properties, one operation per statement: ADD COLUMN, DROP COLUMN, RENAME COLUMN, ALTER COLUMN ... SET DATA TYPE (as the catalog allows: int to long, float to double, a wider decimal), ALTER COLUMN ... DROP NOT NULL and SET TBLPROPERTIES. Column names are case-sensitive. See ALTER TABLE.

  • INSERT INTO <name> [(<column>, ...)] <query> appends the rows of a VALUES list or a query in one commit, and INSERT OVERWRITE replaces all of the table’s rows. Values convert to the column types as described in INSERT.

  • TRUNCATE [TABLE] [IF EXISTS] <name> deletes all of a catalog table’s rows in one commit, without writing data files.

  • DELETE FROM <name> [WHERE <condition>] deletes the matching rows in one commit, keeping rows where the condition is NULL. The condition compares columns with constants. It returns the statement, not the remaining rows. See DELETE FROM.

  • UPDATE <name> SET <column> = <value>, ... [WHERE <condition>] changes the matching rows in one commit, with the condition limited as for DELETE. Values convert to the column types as for INSERT. See UPDATE.

A catalog table can be read as of a snapshot, a branch or tag, or a point in time: FROM <name> FOR VERSION AS OF <id or 'ref'>, FOR TIMESTAMP AS OF <timestamp>, or AT (VERSION => ...). See FROM.

Catalogs attached inside a with SQLContext() block are detached on exit, like the tables registered there. If that detaches the default catalog, the default catalog on entry is restored.

A serialized query plan over catalog tables can contain the catalog’s properties and the storage options, which may include credentials: treat serialized plans as secret.

SQLContext is not thread-safe: do not register catalogs or tables while another thread executes a query on the same context.

Examples

>>> from pyiceberg.catalog import load_catalog  
>>> ctx = pl.SQLContext()  
>>> ctx.register_catalog(
...     "lake", load_catalog("lake"), default_namespace="tpch", default=True
... )  
>>> ctx.execute("SELECT COUNT(*) FROM lineitem")

Create a catalog table holding a query’s rows, then drop it:

>>> ctx.execute(
...     "CREATE TABLE big_orders AS SELECT * FROM orders WHERE amount > 1000"
... )  
>>> ctx.execute("DROP TABLE big_orders")