Google Cloud unveiled its Spanner columnar engine in public preview today, delivering analytical query speeds up to 200 times faster while maintaining real-time transactional capabilities. The new dual-storage architecture enables the Cloud Spanner database to process both operational and analytical workloads simultaneously without performance degradation, positioning it as a unified solution for organizations seeking to eliminate complex data pipelines between transactional and analytical systems.
The enhancement works by maintaining data in both row-based and columnar formats simultaneously, with Spanner’s query processor automatically routing requests to the optimal storage layer. Short transactional queries continue using the row store for high-throughput operations, while analytical queries leverage the columnar store for large-scale scans and aggregations, according to Google Cloud documentation.
The system employs vectorized execution, processing data in batches rather than row-by-row to achieve its performance gains. Google Cloud benchmarks using Clickbench showed specific queries running 46.3 times and 58.6 times faster on a single node, contributing to the overall improvement claims of up to 200-fold acceleration for analytical workloads.
Apache Iceberg Integration
A key architectural decision involves Spanner’s integration with Apache Iceberg lakehouses through what Google calls a “reverse ETL” pattern. Rather than directly querying Iceberg files in data lakes, Spanner ingests curated data from cold storage and transforms it into hot operational data for low-latency access, the company explained in its blog post.
The integration relies on several pathways, with BigQuery’s BigLake component serving as the primary connector for reading Iceberg tables in Google Cloud Storage. Organizations can also use Dataflow templates for complex pipelines or connect data from platforms like Databricks UniForm and Snowflake through BigQuery, expanding the system’s interoperability.
Commercial Impact and Availability
Early adopters include Palo Alto Networks, which requires real-time threat detection insights, and Vodafone, seeking to bridge analytical and operational data for enhanced customer experiences, according to Google Cloud’s announcement.
The columnar engine is currently available in public preview, with pricing based on storage consumption. Enabling the feature creates additional columnar representations that are billed at standard Spanner storage rates, Google Cloud documentation states. Notable limitations during preview include instance backups not containing columnar data and potential performance impacts from high-rate updates or random inserts.
The system also features direct integration with Vertex AI, allowing users to invoke machine learning models within SQL queries using the ML.PREDICT function, positioning Spanner as both a database and serving layer for AI-driven applications.
Sources
- Google Cloud Blog
- Google Cloud Documentation
