# BigQuery Published articles for BigQuery. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Kotlin on the server at Khan Academy DevFeed: [Kotlin on the server at Khan Academy]() Original publisher: [Read original article]() Author: Khan Academy Published: 2018-06-28T22:00:00Z Content type: article Language: en Sources: [Khan Academy]() Topics: [Kotlin](), [backend-development](), [Python](), [Google]() Tags: [api](), [bigquery](), [blog-post](), [developer-experience](), [engineering](), [google](), [http](), [infrastructure](), [java](), [kotlin](), [news](), [python](), [server-side-development]() ### AI overview Khan Academy describes testing Kotlin for user-facing server requests during a 2018 hackathon and deciding to adopt Kotlin as a second server-side language after reporting positive efficiency results and a positive developer experience. ### Source excerpt By Colin Fuller At Khan Academy, we run our web application using Python 2.7 on Google's App Engine ... Read more ## New data pipeline management platform at Khan Academy DevFeed: [New data pipeline management platform at Khan Academy]() Original publisher: [Read original article]() Author: Khan Academy Published: 2018-04-30T22:00:00Z Content type: article Language: en Sources: [Khan Academy]() Topics: [data](), [DataOps](), [systems]() Tags: [batch](), [bigquery](), [cloud-dataflow](), [data](), [data-pipelines](), [debugging](), [engineering](), [infrastructure](), [news](), [pipeline](), [scheduling]() ### AI overview Khan Academy developed Khanalytics to manage its growing collection of data pipelines. The platform provides a sandboxed environment for batch jobs, a web interface, automatic parallelization, centralized logs, and pipeline scheduling with dependencies. ### Source excerpt By Ragini Gupta Data is very crucial to Khan Academy and is itself an internal product for the ... Read more ## Measuring real-time performance per dollar under continuous load: CostBench's first end-to-end results DevFeed: [Measuring real-time performance per dollar under continuous load: CostBench's first end-to-end results]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2026-09-08T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [AI, ML & Data Engineering](), [Amazon Redshift]() Tags: [analytics](), [architecture](), [benchmark](), [bigquery](), [clickhouse](), [cloud](), [cost](), [data](), [performance](), [real-time](), [serverless](), [streaming](), [systems](), [tracing](), [work]() ### AI overview CostBench benchmarks the cost and performance of real-time cloud data warehouses under continuous ingestion and query load. It compares ClickHouse Cloud with Snowflake, BigQuery, and Redshift Serverless, reporting better end-to-end performance per dollar for ClickHouse Cloud in the tested workload. ### Source excerpt CostBench puts cloud data warehouses under continuous load. Across the complete path from fresh data to fast answers, ClickHouse Cloud delivers 412-1,996x better performance per dollar. ## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-09-07T06:33:31Z Content type: article Language: en Sources: [Housing.com]() Topics: [BigQuery](), [databricks](), [Optimization](), [AWS Database Migration Service](), [data-platforms](), [Platform Engineering](), [migration](), [Cloud]() Tags: [bigquery](), [cloud-computing](), [cost](), [cost-optimization](), [data-engineering](), [data-platforms](), [databricks](), [delta-lake](), [finops](), [infrastructure](), [migration](), [optimization](), [platform-engineering]() ### AI overview This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation. ### Source excerpt How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea ## How dbt works, and why orchestrators shouldn't split it into tasks DevFeed: [How dbt works, and why orchestrators shouldn't split it into tasks]() Original publisher: [Read original article]() Author: Ruben Fiszel Published: 2026-08-20T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [Compiler](), [Job](), [airflow](), [data](), [Command-line interface]() Tags: [airflow](), [bigquery](), [cli](), [dagster](), [data](), [data-pipelines](), [dbt](), [dbt-data-pipelines-airflow-dagster-orchestration](), [job](), [orchestration](), [scheduler](), [series]() ### AI overview This primer explains dbt's architecture and examines five ways teams orchestrate it in production. It argues that running a project as one dbt command, while reading dbt's per-model execution state, is generally more efficient and less fragile than creating one orchestrator task per model. It discusses Dagster and astronomer-cosmos's convergence on this approach, including a reported cost comparison. ### Source excerpt How is dbt actually orchestrated, and why does running it as one job beat one task per model? A primer on dbt as a compiler with a scheduler attached, the five ways teams wrap it, and why both Dagster and astronomer-cosmos converged on a single dbt invocation with per-model state projected out of it. ## How Sony LIV uses ClickHouse Cloud to deliver live streaming analytics at billion-row scale DevFeed: [How Sony LIV uses ClickHouse Cloud to deliver live streaming analytics at billion-row scale]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-08-18T16:20:22Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Cloud](), [real-time](), [Streaming](), [data](), [telemetry](), [Platform Engineering](), [BigQuery](), [elasticsearch]() Tags: [bigquery](), [clickhouse](), [cloud](), [data](), [elasticsearch](), [latency](), [platform-engineering](), [real-time](), [streaming](), [telemetry]() ### AI overview Sony LIV rebuilt its streaming analytics infrastructure around ClickHouse Cloud, consolidating fragmented batch pipelines, Elasticsearch workflows, BigQuery workloads, and telemetry stores. The cloud-native platform processes billions of daily events and delivers sub-second analytics for operational visibility during live events. ### Source excerpt Sony LIV consolidated fragmented batch, Elasticsearch, and BigQuery workloads on ClickHouse Cloud, delivering sub-second analytics across billions of daily streaming events. ## Announcing Port's new GCP integration DevFeed: [Announcing Port's new GCP integration]() Original publisher: [Read original article]() Author: Matan Geva Published: 2026-07-30T10:06:55Z Content type: release Language: en Sources: [Developer Experience & Platform Engineering Blog | Port]() Topics: [internal developer portal](), [Google](), [cloud-infrastructure](), [Software]() Tags: [bigquery](), [cloud](), [cloud-run](), [devops](), [gcp](), [github](), [integration](), [software-catalog]() ### AI overview Port announces a new GCP integration that replaces its previous GCP exporter. The integration brings live Google Cloud resource metadata, runtime data, state, and real-time changes into Port's software catalog across projects and services. ### Source excerpt You can now integrate live cloud resource data into the software catalog, displaying both metadata and runtime data in context, making it easy. ## Indexing the Data Lake for Online Point Queries DevFeed: [Indexing the Data Lake for Online Point Queries]() Original publisher: [Read original article]() Author: Spotify Engineering Published: 2026-07-27T20:34:28Z Content type: article Language: en Sources: [Spotify Engineering](), [Spotify Engineering Blog]() Topics: [AI, ML & Data Engineering](), [Amazon S3]() Tags: [ai-agent](), [analytics](), [bigquery](), [data](), [infrastructure](), [low-latency](), [parquet](), [sql](), [storage]() ### AI overview Spotify describes Random Access Parquet (RAP), which uses an external index and ranged reads to support low-latency key-based lookups in data-lake Parquet files without maintaining separate serving copies. ### Source excerpt Companies like Spotify need vast quantities of data accessible at low latency for online services and,... The post Indexing the Data Lake for Online Point Queries appeared first on Spotify Engineering. ## Supabase Pipelines is now in Public Alpha DevFeed: [Supabase Pipelines is now in Public Alpha]() Original publisher: [Read original article]() Author: Riccardo Busetti Published: 2026-07-21T07:00:00Z Content type: release Language: en Sources: [Supabase Blog]() Topics: [Supabase](), [Replication](), [BigQuery](), [clickhouse](), [data](), [Databases](), [dashboards](), [Open Source](), [Rust]() Tags: [bigquery](), [clickhouse](), [dashboards](), [data](), [database](), [google](), [latency](), [memory](), [monitoring](), [open-source](), [performance](), [real-time](), [replication](), [rust](), [sync]() ### AI overview Supabase Pipelines has entered public alpha with automatic handling of supported schema changes, faster initial synchronization, and a request form for additional destinations including ClickHouse, Snowflake, and DuckLake. The service replicates Supabase Postgres data to analytical systems in near real time, with Google BigQuery as the first generally available destination. ### Source excerpt Supabase Pipelines is now in public alpha with schema change support, a faster initial sync, and a new destination request form for ClickHouse, Snowflake, and DuckLake. ## Tableflow: Turn Kafka Topics into Iceberg Tables DevFeed: [Tableflow: Turn Kafka Topics into Iceberg Tables]() Original publisher: [Read original article]() Author: Mohtasham Sayeed Mohiuddin Published: 2026-07-10T15:36:14Z Content type: tutorial Language: en Sources: [Confluent: Data in motion]() Topics: [Apache Iceberg](), [Kafka](), [schema-evolution](), [Confluent Cloud](), [parquet](), [Amazon S3](), [Streaming](), [AWS Glue](), [Amazon Redshift](), [BigQuery]() Tags: [amazon-redshift](), [amazon-s3](), [apache-iceberg](), [aws-glue](), [bigquery](), [confluent-cloud](), [kafka](), [parquet](), [schema-evolution](), [streaming](), [technologies](), [tutorial]() ### AI overview This tutorial explains how Confluent Cloud Tableflow continuously materializes Apache Kafka topics as Apache Iceberg or Delta Lake tables. It covers automatic schema handling, type conversion, schema evolution, Parquet conversion, catalog publishing, and table maintenance for querying streaming data with analytics engines and warehouses. ### Source excerpt Learn how Confluent Tableflow turns Kafka topics into Iceberg tables for zero-ETL analytics with automatic schema evolution and open catalog access. ## Introducing TabFM: A zero-shot foundation model for tabular data DevFeed: [Introducing TabFM: A zero-shot foundation model for tabular data]() Original publisher: [Read original article]() Published: 2026-06-30T10:26:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [foundation-models](), [datasets](), [Machine learning](), [Google](), [Feature Engineering](), [Hyperparameter optimization](), [Language models](), [BigQuery](), [GitHub](), [hugging face](), [AI Inference]() Tags: [bigquery](), [classification](), [data](), [data-management](), [feature-engineering](), [github](), [google](), [hugging-face](), [hyperparameter-optimization](), [machine-intelligence](), [model](), [product](), [zero-shot]() ### AI overview Google Research introduces TabFM, a zero-shot foundation model for tabular-data classification and regression. It frames prediction as in-context learning, reducing the need for dataset-specific training, hyperparameter optimization, and feature engineering, with availability through Hugging Face, GitHub, and BigQuery. ### Source excerpt Data Management ## Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud DevFeed: [Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-06-18T20:31:40Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Cloud](), [data](), [data-engineering](), [observability](), [real-time](), [Hadoop](), [log management](), [BigQuery](), [Artificial Intelligence]() Tags: [analytics](), [bigquery](), [clickhouse](), [cloud](), [data](), [data-engineering](), [hadoop](), [infrastructure](), [logging](), [observability](), [real-time]() ### AI overview METRO Markets replaced its self-hosted Hadoop-based data stack with ClickHouse Cloud, consolidating company data into a single warehouse that supports data warehousing, real-time seller analytics, credit risk modeling, observability, operational logging, and new AI use cases. ### Source excerpt How METRO Markets replaced a failing Hadoop-based stack with ClickHouse Cloud to build a single platform now powering data warehousing, real-time seller analytics, credit risk modeling, observability, and AI across the whole company. ## TPC-H for less than a cent: ClickHouse Cloud vs. Snowflake, Databricks, BigQuery, and Redshift DevFeed: [TPC-H for less than a cent: ClickHouse Cloud vs. Snowflake, Databricks, BigQuery, and Redshift]() Original publisher: [Read original article]() Author: Tom Schreiber; Mark Needham; Alexander Gololobov; Andriy Yakovlev; Robert Schulze Published: 2026-06-02T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [benchmarking](), [Cloud](), [Amazon Redshift](), [BigQuery](), [databricks](), [SQL]() Tags: [aws](), [benchmark](), [bigquery](), [clickhouse](), [cloud](), [cost](), [databricks](), [performance](), [sql]() ### AI overview The article benchmarks ClickHouse Cloud against Snowflake, Databricks, BigQuery, and Redshift using the TPC-H workload. It reports competitive runtime at SF100, first-place cost-performance, and an SF10 workload cost of less than one cent. ### Source excerpt ClickHouse Cloud takes on Snowflake, Databricks, BigQuery, and Redshift on TPC-H, ranking first on SF100 cost-performance and running SF10 for less than one cent. ## BigQuery CI/CD with Harness Database DevOps DevFeed: [BigQuery CI/CD with Harness Database DevOps]() Original publisher: [Read original article]() Author: Animesh Pathak Stephen Atwell Published: 2026-05-29T00:00:00Z Content type: release Language: en Sources: [Harness Blog]() Topics: [BigQuery](), [CI/CD](), [Database](), [Deployment](), [DevOps](), [Authentication](), [OpenID connect (OIDC)](), [Security]() Tags: [authentication](), [automation](), [bigquery](), [ci-cd](), [database](), [deployment](), [devops](), [google-cloud](), [oidc](), [pipelines](), [schema](), [security]() ### AI overview Harness announces BigQuery support for Database DevOps, enabling pipeline-driven schema deployments with versioning, approvals, governance, promotion across environments, auditing, and OIDC-based keyless authentication. ### Source excerpt Automate BigQuery schema deployments with Harness using secure OIDC authentication and CI/CD pipelines. | Blog ## Bringing full YAML anchor support to zizmor DevFeed: [Bringing full YAML anchor support to zizmor]() Original publisher: [Read original article]() Author: "Alexis Challande" Published: 2026-05-22T11:00:00Z Content type: article Language: en Sources: [The Trail of Bits Blog](), [The Trail of Bits Blog]() Topics: [GitHub Actions](), [YAML](), [GitHub](), [ci](), [BigQuery](), [bug](), [Open Source](), [dataset]() Tags: [backdoor](), [bigquery](), [bug](), [bugs](), [ci](), [ecosystem-security](), [engineering-practice](), [github](), [github-actions](), [open-source](), [post-mortem](), [pull-requests](), [secrets](), [supply-chain](), [testing]() ### AI overview Trail of Bits describes how it collaborated with zizmor maintainers to add full YAML anchor support to the GitHub Actions static analyzer. The work fixed parsing and deserialization issues, aligned expression evaluation with GitHub's tests, and was validated against 41,253 workflows from 6,612 open-source repositories. ### Source excerpt In March 2026, attackers exploited a pull_request_target misconfiguration in the aquasecurity/trivy-action GitHub Action to exfiltrate organization and repository secrets, then used those credentials to backdoor LiteLLM on PyPI (see Trivy's post-mortem for the full timeline). zizmor is a static analyzer that GitHub Actions users run to catch exactly these misconfigurations before they ship. When GitHub Actions added support for YAML anchors in September 2025, a small but high-value slice of the ecosystem started writing workflows that zizmor could only analyze on a best-effort basis. Over the past three months, Trail of Bits collaborated with the zizmor maintainers to bring zizmor's anchor support up to full coverage. First, we fixed parsing bugs that caused crashes, produced wrong-location findings, and silently mishandled aliased values. Second, we surfaced deserialization edge cases that broke zizmor on otherwise valid workflows. Finally, we helped align zizmor's expression evaluator with GitHub's own Known Answer Tests. We validated all of this against a new corpus of 41,253 workflows from 6,612 high-value open-source repositories. The result: 20 filed issues, 15 merged pull requests. Building the test corpus To understand how anchors are used in CI today and to stress-test zizmor against the full variety of YAML it encounters in the wild, we built a corpus of real workflows. We used BigQuery's GitHub dataset to identify the 10,000 most-starred repositories created between 2022 and 2025, filtered to the 6,612 that use GitHub Actions, and downloaded every workflow file. That gave us 41,253 YAML files. Figure 1: Building a testing corpus When we ran zizmor against the corpus, it crashed on 45 of the 41,253 workflows. That's a low rate, but each crash means a bug in zizmor. How anchors are used in the wild zizmor's anchor support was deliberately limited, and for good reason. YAML anchors make workflows non-local: an alias defined in one place changes behavior else ## ClickHouse integrates with Lakehouse Runtime Catalog DevFeed: [ClickHouse integrates with Lakehouse Runtime Catalog]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2026-04-22T13:27:18Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [Google](), [BigQuery](), [Apache Spark](), [Cloud](), [API](), [data](), [Authentication]() Tags: [analytics](), [apache-iceberg](), [api](), [authentication](), [bigquery](), [clickhouse](), [cloud](), [data](), [google](), [google-cloud](), [integration](), [spark]() ### AI overview ClickHouse announces a beta integration with Google's Lakehouse Runtime Catalog, enabling direct querying of Google Cloud Lakehouse Apache Iceberg tables through the Iceberg REST Catalog. The integration supports ClickHouse 26.2 and later, ClickHouse Cloud, Google Application Default Credentials, and querying or loading data without data movement or metadata synchronization. ### Source excerpt We're excited to announce a new integration between ClickHouse and Google's Lakehouse Runtime CatalogCatalog ## Terraform & OpenAPI for ClickPipes is now Generally Available DevFeed: [Terraform & OpenAPI for ClickPipes is now Generally Available]() Original publisher: [Read original article]() Author: Marta Paes Published: 2026-04-20T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [Terraform](), [Infrastructure as code](), [OpenAPI Specification](), [clickhouse](), [Provisioning](), [API](), [API keys](), [Amazon S3](), [Azure](), [BigQuery](), [MongoDB](), [MySQL]() Tags: [api](), [api-keys](), [azure](), [bigquery](), [clickhouse](), [infrastructure](), [infrastructure-as-code](), [mysql](), [openapi](), [s3](), [terraform]() ### AI overview ClickHouse announces the general availability of Terraform resources and OpenAPI endpoints for ClickPipes. The update adds full connector coverage, including CDC ClickPipes for PostgreSQL, MySQL, and MongoDB, plus additional connectors, storage modes, and reverse private endpoint management. ### Source excerpt Provision and manage ClickPipes resources "as code" using Terraform and OpenAPI, now with full connector coverage and improved usability ## Apache Airflow 3.2.0: Data-Aware Workflows at Scale DevFeed: [Apache Airflow 3.2.0: Data-Aware Workflows at Scale]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2026-04-07T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [releases](), [data](), [Orchestration](), [Release notes](), [SDK](), [Amazon S3](), [BigQuery]() Tags: [airflow](), [apache](), [apache-airflow](), [bigquery](), [data](), [docker-image](), [orchestration](), [pypi](), [release](), [release-notes](), [s3](), [sdk](), [workflows]() ### AI overview Apache Airflow 3.2.0 introduces asset partitioning for more granular data-aware scheduling, along with multi-team deployments, synchronous deadline alert callbacks, and continued Task SDK separation. Partition-aware workflows trigger downstream DAGs only when the relevant partition is updated. ### Source excerpt We're proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Details: 📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: docker pull apache/airflow:3.2.0 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 🗂 Asset Partitioning (AIP-76): Only the Right Work Gets Triggered Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. Key Capabilities Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464) Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depe ## Exploring Data Systems and Building with Rust After Leaving Google DevFeed: [Exploring Data Systems and Building with Rust After Leaving Google]() Original publisher: [Read original article]() Author: Nevin Zheng Published: 2026-02-27T00:00:00Z Content type: opinion Language: en Sources: [nevzheng]() Topics: [Data Infrastructure](), [BigQuery](), [Google](), [Rust](), [systems]() Tags: [bigquery](), [data-infrastructure](), [google](), [rust](), [systems]() ### AI overview A former BigQuery engineer describes leaving Google after four years, exploring storage engines and Rust, and looking for a smaller team working on challenging data-infrastructure problems. ### Source excerpt Left Google, exploring data systems, building things for fun, and figuring out what's next. ## BigQuery connector for ClickPipes is now in Private Preview DevFeed: [BigQuery connector for ClickPipes is now in Private Preview]() Original publisher: [Read original article]() Author: Marta Paes Published: 2026-01-28T16:38:57Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [BigQuery](), [clickhouse](), [migration](), [real-time](), [Performance Testing](), [data-architecture](), [datasets](), [Cloud](), [Replication]() Tags: [architecture](), [bigquery](), [clickhouse](), [cloud](), [data](), [datasets](), [migration](), [open-source](), [performance](), [performance-testing](), [real-time](), [replication]() ### AI overview The article announces a private-preview BigQuery connector for ClickPipes, enabling users to load BigQuery data into ClickHouse Cloud for rapid exploration, prototyping, and performance testing. It presents ClickHouse as a low-latency speed layer alongside BigQuery's batch-processing and warehousing workloads, while describing automated data movement and future continuous-ingestion capabilities. ### Source excerpt Load data from BigQuery into ClickHouse Cloud in a few clicks for fast exploration and prototyping. The connector simplifies data migration for testing ClickHouse's real-time query performance on your BigQuery datasets. ## ClickPy at 2 Trillion rows: Scaling ingestion and fixing the past DevFeed: [ClickPy at 2 Trillion rows: Scaling ingestion and fixing the past]() Original publisher: [Read original article]() Author: Lionel Palacin Published: 2026-01-21T10:48:32Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data](), [Python](), [BigQuery](), [dataset](), [cloud-infrastructure](), [Code](), [Scripting]() Tags: [bigquery](), [blog](), [blog-post](), [clickhouse](), [cloud-storage](), [code](), [data](), [database](), [operational](), [python](), [scale](), [scripting](), [storage]() ### AI overview ClickPy has grown to more than 2 trillion Python package-download rows dating back to 2011. This article describes replacing a custom, cron-based ingestion script with ClickPipes and correcting historical data while keeping the service available and ingestion uninterrupted. ### Source excerpt A look at how ClickPy handles over 2 trillion Python package downloads, from ingestion redesign to fixing historical data at scale. ## Introducing Supabase Pipelines DevFeed: [Introducing Supabase Pipelines]() Original publisher: [Read original article]() Author: Riccardo Busetti Published: 2025-12-02T07:00:00Z Content type: article Language: en Sources: [Supabase Blog]() Topics: [Supabase](), [Replication](), [BigQuery](), [Databases](), [SQL](), [datasets](), [Open Source](), [GitHub]() Tags: [bigquery](), [data](), [datasets](), [github](), [open-source](), [postgres](), [real-time](), [replication](), [sql]() ### AI overview Supabase Pipelines provides managed change-data-capture pipelines that replicate Postgres tables to analytical destinations such as Google BigQuery in near real time. ### Source excerpt Managed change-data-capture pipelines that replicate your Postgres tables to analytical destinations like BigQuery in near real time. ## How the 5 major cloud data warehouses compare on cost-performance DevFeed: [How the 5 major cloud data warehouses compare on cost-performance]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2025-12-02T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [benchmarking](), [Cloud](), [clickhouse](), [Amazon Redshift](), [BigQuery](), [databricks](), [dataset]() Tags: [benchmark](), [benchmarking](), [bigquery](), [clickhouse](), [cloud](), [cloud-data](), [compare](), [comparison](), [cost](), [data](), [databricks](), [dataset](), [performance](), [redshift](), [storage](), [warehouse]() ### AI overview This article compares the cost-performance of Snowflake, Databricks, ClickHouse Cloud, BigQuery, and Redshift across analytical workloads containing 1 billion, 10 billion, and 100 billion rows. Using each system's real compute billing model, the benchmark concludes that ClickHouse Cloud provides substantially better value than the other systems at scale. ### Source excerpt We benchmarked the five major cloud data warehouses at 1B-100B rows using their real billing models to measure performance per dollar. Results show how cost-performance shifts as data grows. ## How the 5 major cloud data warehouses really bill you: A unified, engineer-friendly guide DevFeed: [How the 5 major cloud data warehouses really bill you: A unified, engineer-friendly guide]() Original publisher: [Read original article]() Author: Tom Schreiber & Lionel Palacin Published: 2025-12-01T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [Amazon Redshift](), [cpu]() Tags: [amazon-redshift](), [bigquery](), [clickhouse](), [cloud](), [compute](), [cost](), [cpu](), [data](), [databricks](), [guide](), [performance](), [scale](), [serverless](), [sql](), [storage]() ### AI overview A guide to comparing how five cloud data warehouses meter and bill analytical compute. It explains why published price lists alone do not reveal real query costs and focuses on the execution and scaling models behind those costs. ### Source excerpt This guide explains how the five major cloud data warehouses--Snowflake, Databricks, ClickHouse Cloud, BigQuery, and Redshift--allocate, meter, and bill compute, giving engineers a clear understanding of what the billing units mean and how to compare them. [Next page]()