Google Cloud has added native support for querying external Apache Iceberg REST catalogs from BigQuery, allowing engineers to run analytics directly against tables registered in Databricks Unity Catalog, Snowflake Horizon Catalog, and AWS Glue Data Catalog without moving the underlying data. Practitioners benefit from lower latency, eliminated storage duplication, and reduced pipeline complexity while keeping all compute inside Google Cloud.
How the borderless lakehouse works
The solution relies on the Lakehouse runtime catalog, which periodically synchronizes schema definitions, snapshot pointers, and table metadata from each remote catalog. This metadata synchronization runs on a schedule you define and does not touch the actual data files. When a query is executed, the runtime catalog provides the storage locations, and a compatible engine reads the Parquet files directly from the remote object store (for example, Amazon S3) using temporary credentials issued by the source platform. All query processing – parsing, planning, and execution – occurs in Google Cloud services such as BigQuery or the Managed Service for Apache Spark. No remote Databricks clusters, Snowflake warehouses, or AWS compute resources are invoked during execution.
Deploying a federated catalog
Google Cloud recommends a three‑step deployment pattern:
- Create the federated catalog in Google Cloud and point it at the external Iceberg REST endpoint.
- Inspect the newly created catalog to retrieve the automatically generated service account.
- Grant that service account the necessary permissions in the remote platform (Databricks, Snowflake, or AWS Glue) so it can read the underlying storage.
Once the service account has read access, BigQuery can resolve the external table references and issue direct reads against the remote files.
-- Run inside Databricks SQL workspace CREATE CATALOG IF NOT EXISTS dbx_retail_db; CREATE SCHEMA IF NOT EXISTS dbx_retail_db.db;
Operational considerations
Because metadata and data access are decoupled, operators must monitor two separate aspects:
- Metadata refresh cadence: Too‑infrequent syncs can cause queries to see stale schemas or missing partitions; too‑frequent syncs may increase API traffic to the remote catalog.
- Credential lifecycle: The temporary credentials used for direct storage reads are issued per query. Ensure the remote platform’s IAM policies allow short‑lived token issuance for the federated service account.
All compute costs are incurred on Google Cloud resources, and egress fees are limited to the data read from the remote object store. Since no data is copied into BigQuery storage, storage cost growth is avoided, but network egress from the source cloud still applies.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
Adopting the federated Iceberg catalog pattern lets you replace traditional ETL pipelines with a metadata‑driven federation layer. Evaluate the refresh schedule that balances schema freshness against API rate limits, and verify that the remote platform’s IAM configuration permits the required temporary read tokens. Monitor cross‑cloud egress costs and ensure that any compliance requirements around data residency are satisfied, as the data remains in its original bucket while being processed in Google Cloud.
