How Lakehouse and Databricks work together
Managed Iceberg tables automatically generates Iceberg-compatible metadata alongside native Delta logs. Data engineering teams can store Parquet files once in Google Cloud Storage (GCS) using Databricks. Rather than loading this information into BigQuery, Lakehouse reads the Iceberg metadata directly, applying BigQuery’s vectorized engine directly to open-format files in GCS.
Although Google Cloud Storage (GCS) serves as the primary storage layer in the above example, Lakehouse extends this borderless federation directly to external cloud environments like Amazon S3 and Azure Blob Storage. By pairing the open Apache Iceberg REST Catalog with cross-cloud connections, Lakehouse lets BigQuery query data across clouds without moving it. Each federated catalog lives in a Google Cloud location you choose, ideally the region closest to your data. This lets BigQuery read open-format Parquet files where they live, delivering multi-cloud analytics without replication pipelines. Remote-cloud egress charges still apply, but Cross-Cloud Interconnect (AWS) and Lakehouse caching can reduce them.
Beyond simply querying external data, Lakehouse framework establishes true read and write (R/W) interoperability across engines. Because the platform is built on an open Apache Iceberg foundation, data teams can effortlessly execute multi-statement SQL commands directly inside BigQuery.
Centralized security and governance
Handling access controls across disparate systems can introduce vulnerabilities. Lakehouse addresses this through delegated access. Administrators can configure a BigQuery Cloud Resource Connection tied to a service account that manages IAM rights for GCS buckets. Granting users access to the Lakehouse table entity lets you enforce BigQuery controls including row-level, column-level security, and dynamic data masking on external Iceberg tables while Databricks handles ingestion.
Catalog federation and credential vending
Previously, connecting these platforms meant manually configuring BigQuery queries to point at static metadata paths. Modern setups leverage the open Apache Iceberg REST Catalog API. Databricks Unity Catalog functions as the central metadata layer, allowing a BigQuery administrator to establish a Lakehouse runtime catalog with a federated connection. When running a query, Lakehouse contacts Unity Catalog for current table state, and short-lived tokens are issued through credential vending to grant BigQuery temporary read access to GCS.
Partition pruning
BigQuery uses each table’s Iceberg metadata to plan the query, then reads only the relevant data files in place. Iceberg’s hidden partitioning lets you filter on logical columns, such as timestamps, without knowing the directory layout. Note that Databricks managed Iceberg tables don’t support partition transforms; Databricks recommends liquid clustering instead.
Filters in a standard WHERE clause let BigQuery use the Iceberg manifests to skip partitions that can’t match. Compare partition columns with literal values so pruning can apply. These manifests maintain detailed column-level statistical bounds, enabling BigQuery to bypass irrelevant manifest records and unneeded Parquet files immediately. Consequently, queries scan only necessary byte ranges, accelerating query execution while significantly reducing on-demand slot consumption and storage scan costs.
Fast queries with Iceberg metadata and Lakehouse caching
Iceberg manifests already identify which data files to read, so BigQuery doesn’t need a metadata cache for Iceberg tables. For cross-cloud tables, such as those in Amazon S3, Lakehouse caching (Preview) keeps retrieved data blocks in Google Cloud so repeat queries read them locally. BigQuery reads the Iceberg manifest files directly when it plans each query; there’s no metadata caching to enable. This dynamic partition pruning bypasses unneeded files, significantly reducing runtime and bytes scanned.
Maintenance and lifecycle management
Modern lakehouses demand active upkeep; without routine care, performance degrades into an inefficient repository. Fortunately, both BigQuery and Databricks supply native capabilities to preserve table hygiene and accelerate analytical workloads.
Keeping your tables optimized
To maintain optimal query performance, active maintenance of your lakehouse storage footprint remains necessary.
-
Trigger regular compaction pipelines via Databricks APIs to merge tiny Parquet files into larger, optimized layouts and use Predictive Optimization.
-
Execute the Databricks VACUUM command to clear unreferenced files outside active snapshots, avoiding extra GCS storage costs.
Compaction and the small file problem
Frequent write operations, continuous ingestion jobs, and streaming workloads continuously produce thousands of small Parquet files. This fragmentation creates a storage bottleneck that can limit BigQuery’s reading efficiency. Establishing thousands of network connections to GCS and processing metadata footers across numerous tiny files consumes significant compute time, diminishing the performance advantages of vectorized query execution.
Running routine compaction within Iceberg resolves these inefficiencies. Databricks manages this automatically using Liquid Clustering which dynamically adjusts file layouts based on access patterns or through native table management utilities.
To compact small files manually on Databricks, run OPTIMIZE:
