The borderless Lakehouse introduces a direct query path from BigQuery to Apache Iceberg tables that Databricks writes. Instead of building ETL pipelines that copy Parquet files between environments, BigQuery now reads the same files in place and sees updates after a scheduled metadata refresh (as frequent as five minutes). For engineers who manage concurrent Spark and analytics workloads, the change eliminates a storage‑costly duplication step and shortens data‑to‑insight latency.
How the borderless Lakehouse works
When Databricks creates a managed Iceberg table, it writes Parquet files to Google Cloud Storage and simultaneously generates Iceberg‑compatible metadata alongside the native Delta logs. The Lakehouse layer reads that metadata directly, allowing BigQuery’s vectorized engine to scan the open‑format files without an intermediate load. The same mechanism extends to external buckets in Amazon S3 or Azure Blob Storage via the Apache Iceberg REST Catalog; each federated catalog resides in a GCP region close to the source data, keeping read paths short. Remote egress charges still apply, but the architecture can mitigate them with Cross‑Cloud Interconnect and Lakehouse‑level caching.
Security and governance implications
Access control is consolidated through a BigQuery Cloud Resource Connection that references a service account with IAM permissions on the underlying GCS bucket. Granting a user access to the Lakehouse table entity automatically enforces BigQuery‑level policies—row‑level security, column‑level security, and dynamic data masking—while Databricks continues to own the ingestion pipeline. Catalog federation relies on Databricks Unity Catalog as the source of truth; when a query runs, Lakehouse contacts Unity Catalog for the current table snapshot and receives short‑lived tokens via credential vending, granting temporary read rights to the storage location.
Performance and cost considerations
BigQuery leverages Iceberg manifests to plan queries, enabling partition pruning based on logical column filters. Because Iceberg stores column‑level statistics in the manifests, BigQuery can skip entire data files that cannot satisfy the WHERE clause, reducing bytes scanned, slot consumption, and associated on‑demand costs. Databricks‑managed Iceberg tables do not support partition transforms; the platform recommends liquid clustering instead, which still benefits from manifest‑driven pruning. Additionally, Lakehouse caching can keep frequently accessed metadata in memory, further accelerating query planning.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
Teams should evaluate existing data pipelines for opportunities to replace copy jobs with a Lakehouse‑based federation. Verify that service accounts used for the Cloud Resource Connection have the minimal IAM scope required for the target buckets, and test row‑level and column‑level policies on external Iceberg tables to confirm they behave as expected. Monitor metadata refresh intervals to balance freshness against query planning overhead, and assess egress patterns when accessing data in other clouds. Finally, incorporate Iceberg manifest statistics into query design to maximize partition pruning and keep cost per query low.