Cross‑cloud caching is now in preview for Google’s Borderless Lakehouse, extending BigQuery’s ability to query remote datasets while storing frequently accessed column blocks locally in Google Cloud. The change reduces the amount of data that must traverse public networks, cuts transfer costs, and introduces built‑in encryption and tenant isolation without requiring manual cache provisioning.
What Changed
Google Cloud announced two preview capabilities:
- Cross‑cloud caching for Lakehouse queries in BigQuery. The cache stores sub‑file blocks—specifically column chunks and dictionary pages—from remote Apache Iceberg tables, allowing subsequent queries to read from the local copy.
- BigQuery cross‑cloud connections that enable queries against non‑Iceberg data sources in other clouds.
Both features are delivered as part of the Borderless Lakehouse architecture and are intended to operate transparently, with no user‑visible knobs or separate storage resources to manage.
Why It Matters to Practitioners
AI engineers gain faster access to training data that lives in S3, ADLS, or other external stores, because only the column blocks required for a model’s feature set need to be transferred. Cloud and platform engineers can design pipelines that keep data in place while still achieving low‑latency analytics, reducing the need for costly ETL duplication. DevOps and SRE teams benefit from predictable network usage: the cache limits cross‑cloud traffic to a fraction of the original dataset size, simplifying capacity planning and cost forecasting. Security engineers see that cached data inherits Google‑managed encryption at rest and is isolated per project and region, preserving existing compliance postures without additional configuration.
Architectural and Operational Implications
The preview introduces several concrete considerations:
- Sub‑file block granularity: Instead of moving whole files, BigQuery fetches only the column blocks required by a query. This reduces bandwidth but also means cache hit rates depend on query patterns that repeatedly access the same columns.
- Encryption at rest: Cached blocks are encrypted by default with Google‑managed encryption keys (GMEK). No extra key‑management steps are required, but teams should verify that their key‑management policies align with GMEK usage.
- Tenant and regional isolation: Cache entries are partitioned by Google Cloud project and catalog, and they are bound to a specific region (e.g.,
us-east4). This design supports data residency requirements but also requires that workloads be region‑aware to avoid cross‑region cache misses. - Zero‑configuration operation: The service automatically creates and evicts cache entries. Operators should monitor cache size metrics to ensure that local storage quotas are not unintentionally exceeded.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
Start by enabling the preview in a non‑production project and run a representative query workload against remote Iceberg tables. Observe cache hit ratios and network egress to confirm the claimed reduction to under 5 % of transferred data. Verify that the region of the BigQuery dataset matches the intended cache region to stay compliant with residency rules. Finally, incorporate cache‑related metrics into existing observability dashboards so that SREs can track performance and storage impact over time.