# BigQuery BigQuery is a Google Cloud data analytics platform for loading, querying, transforming, and analyzing data. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-09-07T06:33:31Z Content type: article Language: en Sources: [Housing.com]() Topics: [BigQuery](), [databricks](), [Optimization](), [AWS Database Migration Service](), [data-platforms](), [Platform Engineering](), [migration](), [Cloud]() Tags: [bigquery](), [cloud-computing](), [cost](), [cost-optimization](), [data-engineering](), [data-platforms](), [databricks](), [delta-lake](), [finops](), [infrastructure](), [migration](), [optimization](), [platform-engineering]() ### AI overview This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation. ### Source excerpt How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea ## Supabase Pipelines is now in Public Alpha DevFeed: [Supabase Pipelines is now in Public Alpha]() Original publisher: [Read original article]() Author: Riccardo Busetti Published: 2026-07-21T07:00:00Z Content type: release Language: en Sources: [Supabase Blog]() Topics: [Supabase](), [Replication](), [BigQuery](), [clickhouse](), [data](), [Databases](), [dashboards](), [Open Source](), [Rust]() Tags: [bigquery](), [clickhouse](), [dashboards](), [data](), [database](), [google](), [latency](), [memory](), [monitoring](), [open-source](), [performance](), [real-time](), [replication](), [rust](), [sync]() ### AI overview Supabase Pipelines has entered public alpha with automatic handling of supported schema changes, faster initial synchronization, and a request form for additional destinations including ClickHouse, Snowflake, and DuckLake. The service replicates Supabase Postgres data to analytical systems in near real time, with Google BigQuery as the first generally available destination. ### Source excerpt Supabase Pipelines is now in public alpha with schema change support, a faster initial sync, and a new destination request form for ClickHouse, Snowflake, and DuckLake. ## Introducing TabFM: A zero-shot foundation model for tabular data DevFeed: [Introducing TabFM: A zero-shot foundation model for tabular data]() Original publisher: [Read original article]() Published: 2026-06-30T10:26:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [foundation-models](), [datasets](), [Machine learning](), [Google](), [Feature Engineering](), [Hyperparameter optimization](), [Language models](), [BigQuery](), [GitHub](), [hugging face](), [AI Inference]() Tags: [bigquery](), [classification](), [data](), [data-management](), [feature-engineering](), [github](), [google](), [hugging-face](), [hyperparameter-optimization](), [machine-intelligence](), [model](), [product](), [zero-shot]() ### AI overview Google Research introduces TabFM, a zero-shot foundation model for tabular-data classification and regression. It frames prediction as in-context learning, reducing the need for dataset-specific training, hyperparameter optimization, and feature engineering, with availability through Hugging Face, GitHub, and BigQuery. ### Source excerpt Data Management ## BigQuery CI/CD with Harness Database DevOps DevFeed: [BigQuery CI/CD with Harness Database DevOps]() Original publisher: [Read original article]() Author: Animesh Pathak Stephen Atwell Published: 2026-05-29T00:00:00Z Content type: release Language: en Sources: [Harness Blog]() Topics: [BigQuery](), [CI/CD](), [Database](), [Deployment](), [DevOps](), [Authentication](), [OpenID connect (OIDC)](), [Security]() Tags: [authentication](), [automation](), [bigquery](), [ci-cd](), [database](), [deployment](), [devops](), [google-cloud](), [oidc](), [pipelines](), [schema](), [security]() ### AI overview Harness announces BigQuery support for Database DevOps, enabling pipeline-driven schema deployments with versioning, approvals, governance, promotion across environments, auditing, and OIDC-based keyless authentication. ### Source excerpt Automate BigQuery schema deployments with Harness using secure OIDC authentication and CI/CD pipelines. | Blog ## Bringing full YAML anchor support to zizmor DevFeed: [Bringing full YAML anchor support to zizmor]() Original publisher: [Read original article]() Author: "Alexis Challande" Published: 2026-05-22T11:00:00Z Content type: article Language: en Sources: [The Trail of Bits Blog](), [The Trail of Bits Blog]() Topics: [GitHub Actions](), [YAML](), [GitHub](), [ci](), [BigQuery](), [bug](), [Open Source](), [dataset]() Tags: [backdoor](), [bigquery](), [bug](), [bugs](), [ci](), [ecosystem-security](), [engineering-practice](), [github](), [github-actions](), [open-source](), [post-mortem](), [pull-requests](), [secrets](), [supply-chain](), [testing]() ### AI overview Trail of Bits describes how it collaborated with zizmor maintainers to add full YAML anchor support to the GitHub Actions static analyzer. The work fixed parsing and deserialization issues, aligned expression evaluation with GitHub's tests, and was validated against 41,253 workflows from 6,612 open-source repositories. ### Source excerpt In March 2026, attackers exploited a pull_request_target misconfiguration in the aquasecurity/trivy-action GitHub Action to exfiltrate organization and repository secrets, then used those credentials to backdoor LiteLLM on PyPI (see Trivy's post-mortem for the full timeline). zizmor is a static analyzer that GitHub Actions users run to catch exactly these misconfigurations before they ship. When GitHub Actions added support for YAML anchors in September 2025, a small but high-value slice of the ecosystem started writing workflows that zizmor could only analyze on a best-effort basis. Over the past three months, Trail of Bits collaborated with the zizmor maintainers to bring zizmor's anchor support up to full coverage. First, we fixed parsing bugs that caused crashes, produced wrong-location findings, and silently mishandled aliased values. Second, we surfaced deserialization edge cases that broke zizmor on otherwise valid workflows. Finally, we helped align zizmor's expression evaluator with GitHub's own Known Answer Tests. We validated all of this against a new corpus of 41,253 workflows from 6,612 high-value open-source repositories. The result: 20 filed issues, 15 merged pull requests. Building the test corpus To understand how anchors are used in CI today and to stress-test zizmor against the full variety of YAML it encounters in the wild, we built a corpus of real workflows. We used BigQuery's GitHub dataset to identify the 10,000 most-starred repositories created between 2022 and 2025, filtered to the 6,612 that use GitHub Actions, and downloaded every workflow file. That gave us 41,253 YAML files. Figure 1: Building a testing corpus When we ran zizmor against the corpus, it crashed on 45 of the 41,253 workflows. That's a low rate, but each crash means a bug in zizmor. How anchors are used in the wild zizmor's anchor support was deliberately limited, and for good reason. YAML anchors make workflows non-local: an alias defined in one place changes behavior else ## ClickHouse integrates with Lakehouse Runtime Catalog DevFeed: [ClickHouse integrates with Lakehouse Runtime Catalog]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2026-04-22T13:27:18Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [Google](), [BigQuery](), [Apache Spark](), [Cloud](), [API](), [data](), [Authentication]() Tags: [analytics](), [apache-iceberg](), [api](), [authentication](), [bigquery](), [clickhouse](), [cloud](), [data](), [google](), [google-cloud](), [integration](), [spark]() ### AI overview ClickHouse announces a beta integration with Google's Lakehouse Runtime Catalog, enabling direct querying of Google Cloud Lakehouse Apache Iceberg tables through the Iceberg REST Catalog. The integration supports ClickHouse 26.2 and later, ClickHouse Cloud, Google Application Default Credentials, and querying or loading data without data movement or metadata synchronization. ### Source excerpt We're excited to announce a new integration between ClickHouse and Google's Lakehouse Runtime CatalogCatalog ## Exploring Data Systems and Building with Rust After Leaving Google DevFeed: [Exploring Data Systems and Building with Rust After Leaving Google]() Original publisher: [Read original article]() Author: Nevin Zheng Published: 2026-02-27T00:00:00Z Content type: opinion Language: en Sources: [nevzheng]() Topics: [Data Infrastructure](), [BigQuery](), [Google](), [Rust](), [systems]() Tags: [bigquery](), [data-infrastructure](), [google](), [rust](), [systems]() ### AI overview A former BigQuery engineer describes leaving Google after four years, exploring storage engines and Rust, and looking for a smaller team working on challenging data-infrastructure problems. ### Source excerpt Left Google, exploring data systems, building things for fun, and figuring out what's next. ## BigQuery connector for ClickPipes is now in Private Preview DevFeed: [BigQuery connector for ClickPipes is now in Private Preview]() Original publisher: [Read original article]() Author: Marta Paes Published: 2026-01-28T16:38:57Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [BigQuery](), [clickhouse](), [migration](), [real-time](), [Performance Testing](), [data-architecture](), [datasets](), [Cloud](), [Replication]() Tags: [architecture](), [bigquery](), [clickhouse](), [cloud](), [data](), [datasets](), [migration](), [open-source](), [performance](), [performance-testing](), [real-time](), [replication]() ### AI overview The article announces a private-preview BigQuery connector for ClickPipes, enabling users to load BigQuery data into ClickHouse Cloud for rapid exploration, prototyping, and performance testing. It presents ClickHouse as a low-latency speed layer alongside BigQuery's batch-processing and warehousing workloads, while describing automated data movement and future continuous-ingestion capabilities. ### Source excerpt Load data from BigQuery into ClickHouse Cloud in a few clicks for fast exploration and prototyping. The connector simplifies data migration for testing ClickHouse's real-time query performance on your BigQuery datasets. ## ClickPy at 2 Trillion rows: Scaling ingestion and fixing the past DevFeed: [ClickPy at 2 Trillion rows: Scaling ingestion and fixing the past]() Original publisher: [Read original article]() Author: Lionel Palacin Published: 2026-01-21T10:48:32Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data](), [Python](), [BigQuery](), [dataset](), [cloud-infrastructure](), [Code](), [Scripting]() Tags: [bigquery](), [blog](), [blog-post](), [clickhouse](), [cloud-storage](), [code](), [data](), [database](), [operational](), [python](), [scale](), [scripting](), [storage]() ### AI overview ClickPy has grown to more than 2 trillion Python package-download rows dating back to 2011. This article describes replacing a custom, cron-based ingestion script with ClickPipes and correcting historical data while keeping the service available and ingestion uninterrupted. ### Source excerpt A look at how ClickPy handles over 2 trillion Python package downloads, from ingestion redesign to fixing historical data at scale. ## Introducing Supabase Pipelines DevFeed: [Introducing Supabase Pipelines]() Original publisher: [Read original article]() Author: Riccardo Busetti Published: 2025-12-02T07:00:00Z Content type: article Language: en Sources: [Supabase Blog]() Topics: [Supabase](), [Replication](), [BigQuery](), [Databases](), [SQL](), [datasets](), [Open Source](), [GitHub]() Tags: [bigquery](), [data](), [datasets](), [github](), [open-source](), [postgres](), [real-time](), [replication](), [sql]() ### AI overview Supabase Pipelines provides managed change-data-capture pipelines that replicate Postgres tables to analytical destinations such as Google BigQuery in near real time. ### Source excerpt Managed change-data-capture pipelines that replicate your Postgres tables to analytical destinations like BigQuery in near real time. ## How the 5 major cloud data warehouses compare on cost-performance DevFeed: [How the 5 major cloud data warehouses compare on cost-performance]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2025-12-02T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [benchmarking](), [Cloud](), [clickhouse](), [Amazon Redshift](), [BigQuery](), [databricks](), [dataset]() Tags: [benchmark](), [benchmarking](), [bigquery](), [clickhouse](), [cloud](), [cloud-data](), [compare](), [comparison](), [cost](), [data](), [databricks](), [dataset](), [performance](), [redshift](), [storage](), [warehouse]() ### AI overview This article compares the cost-performance of Snowflake, Databricks, ClickHouse Cloud, BigQuery, and Redshift across analytical workloads containing 1 billion, 10 billion, and 100 billion rows. Using each system's real compute billing model, the benchmark concludes that ClickHouse Cloud provides substantially better value than the other systems at scale. ### Source excerpt We benchmarked the five major cloud data warehouses at 1B-100B rows using their real billing models to measure performance per dollar. Results show how cost-performance shifts as data grows. ## WeatherNext 2: Our most advanced weather forecasting model DevFeed: [WeatherNext 2: Our most advanced weather forecasting model]() Original publisher: [Read original article]() Author: The WeatherNext team Published: 2025-11-17T15:09:23Z Content type: article Language: en Sources: [Google DeepMind News]() Topics: [Artificial Intelligence](), [Google](), [model architecture](), [Inference](), [BigQuery](), [Google Cloud Platform (GCP)](), [AI Platform]() Tags: [ai](), [bigquery](), [google](), [inference](), [model](), [model-architecture](), [none](), [tpu](), [vertex-ai]() ### AI overview Google DeepMind and Google Research introduce WeatherNext 2, an AI weather forecasting model that generates hundreds of possible scenarios faster, at higher resolution, and with improved accuracy. Its forecast data is available through Earth Engine and BigQuery, with custom model inference offered through Vertex AI, and its technology is being integrated into several Google products. ### Source excerpt The new AI model delivers more efficient, more accurate and higher-resolution global weather predictions. ## Platform Thinking: Three Views from Two Sigma Leaders DevFeed: [Platform Thinking: Three Views from Two Sigma Leaders]() Original publisher: [Read original article]() Author: Emily Majewski Published: 2025-10-23T14:49:45Z Content type: article Language: en Sources: [Two Sigma Engineering]() Topics: [data-engineering](), [data](), [Large Language Model](), [Development](), [Feature Engineering](), [Artificial Intelligence](), [BigQuery](), [CI/CD](), [AI research agents](), [Open Source Models & Datasets]() Tags: [ai](), [ai-agents](), [bigquery](), [ci-cd](), [data](), [data-engineering](), [datasets](), [feature-engineering](), [llms](), [open-source](), [platform](), [post]() ### AI overview Two Sigma leaders describe the company's platform-oriented approach to data, technology, and management. The article highlights parallel delivery of raw and curated datasets, BigQuery and CI/CD practices, reusable data contracts, LLM-assisted feature engineering, controls for temporal leakage, and possible uses of multiple AI agents in quantitative research. ### Source excerpt The post Platform Thinking: Three Views from Two Sigma Leaders appeared first on Two Sigma. ## The 2025 AI + Data Engineering Roadmap DevFeed: [The 2025 AI + Data Engineering Roadmap]() Original publisher: [Read original article]() Author: Zach Wilson Published: 2025-10-17T22:35:45Z Content type: tutorial Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [Python](), [SQL](), [Artificial Intelligence](), [airflow](), [BigQuery](), [Orchestration](), [Apache Spark](), [data-modeling](), [Data Quality](), [Embeddings](), [Retrieval Augmented Generation (RAG)]() Tags: [2025](), [ai](), [airflow](), [bigquery](), [count](), [course](), [data-engineering](), [data-modeling](), [data-quality](), [embeddings](), [engineering](), [framer](), [newsletter](), [orchestration](), [python](), [rag](), [real-time](), [right-join](), [spark](), [sql]() ### AI overview A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding. ### Source excerpt Getting a data engineering job is complicated. ## Empowering Data Through Self-Service: Behind the Scenes of Our Data Platform DevFeed: [Empowering Data Through Self-Service: Behind the Scenes of Our Data Platform]() Original publisher: [Read original article]() Author: Karen Fehmer Published: 2025-06-02T03:53:58Z Content type: article Language: en Sources: [Kogan.com]() Topics: [data](), [Architecture & Design](), [BigQuery](), [CI/CD](), [GitHub Actions](), [Git]() Tags: [architecture](), [bigquery](), [ci-cd](), [data](), [dbt](), [github-actions](), [looker](), [self-service](), [tableau]() ### AI overview Kogan.com describes replacing a request-based BI model with a governed self-service data platform built around BigQuery, dbt, Looker, and Acryl (DataHub). The architecture uses layered models, GitHub Actions CI/CD, testing, documentation, and controlled environments to support trusted analysis and reporting. ### Source excerpt At Kogan.com, our data needs have grown alongside the business. As more teams relied on insights to move quickly, it became clear our request-based BI model couldn't scale. We needed a platform that empowered teams to answer their own questions, trust the numbers, and move independently. That journey led us to build a self-service platform grounded in governance, transparency, and scalability--powered by dbt, Looker, and Acryl (DataHub). Rethinking Our BI Model We originally relied on Tableau. It served us well but had limitations: duplicated logic, inconsistent metrics, and limited collaboration with dbt. Tableau workbooks weren't version-controlled, which made maintaining consistency difficult. To bridge modeling and reporting, we often created extra presentation tables in dbt, adding complexity. We needed a platform that integrated tightly with dbt and supported governed exploration. A New Architecture: Modular, Transparent, Scalable We redesigned the platform around a clean, modular flow: Raw Sources -> BigQuery -> dbt -> Looker -> Acryl (DataHub) Our data transformations are built in dbt, where we follow a layered modeling structure. While we use stg_ (staging) and int_ (intermediate) models primarily for data cleaning and standardization, the marts_ models are the ones that power our analysis and reporting. These models contain our fact and dimension tables, fully aligned with business logic and ready for consumption in Looker. We've integrated CI/CD pipelines using GitHub Actions, and every change is tested before deployment. This includes dbt tests, schema validations, and model documentation to ensure confidence at every layer. Why Looker Was the Right Fit for Self-Service Looker offered a structured, governed approach that aligned with our dbt-first architecture. LookML let us centralize business logic, version it with Git, and deploy changes through CI/CD. With support for multiple environments (UAT and Production), we can test safely before releasing to users ## How we handle sensitive data in BigQuery DevFeed: [How we handle sensitive data in BigQuery]() Original publisher: [Read original article]() Author: Lambert Le Manh Published: 2024-11-14T15:21:00Z Content type: article Language: en Sources: [The incident.io Blog]() Topics: [BigQuery](), [pii](), [AWS IAM](), [data]() Tags: [bigquery](), [data](), [false-positives](), [iam](), [incident](), [incident-channel](), [incident-management](), [incident-response](), [outage](), [pii](), [policy](), [post-mortem](), [slack-incident]() ### AI overview The article explains how incident.io handles sensitive customer data in BigQuery. New columns are marked sensitive by default and masked for users without appropriate IAM permissions, while separate service accounts provide controlled access for customer-facing and internal analytics workflows. YAML files identify non-sensitive columns that can be untagged to reduce false positives. ### Source excerpt We take handling sensitive customer data seriously. This blog explains how we manage PII and confidential data in BigQuery through default masking, automated tagging, and strict access controls. ## If you don't know about HTTP Archive's Web Almanac yet, you should! DevFeed: [If you don't know about HTTP Archive's Web Almanac yet, you should!]() Original publisher: [Read original article]() Author: Brian Clark Published: 2024-11-01T05:00:00Z Content type: article Language: en Sources: [Blog RSS Feed | Snyk]() Topics: [State of the Web](), [BigQuery](), [Security](), [HTTP](), [Open Source](), [Web Development]() Tags: [bigquery](), [blog](), [community](), [developer](), [devrel](), [http](), [interest](), [javascript](), [open-source](), [open-source-security](), [security](), [snyk-open-source](), [state-of-the-web](), [web]() ### AI overview This article introduces the Web Almanac, a community-driven report that analyzes how the web is built and used through data from millions of websites. It describes the report's chapters, contributors, HTTP Archive web-crawl data, public BigQuery database, and the forthcoming 2024 edition. ### Source excerpt Learn about the HTTP Archive's Web Almanac, a valuable, community-driven resource for today's security teams. ## Introducing Log Drains DevFeed: [Introducing Log Drains]() Original publisher: [Read original article]() Author: Lee TzeYiing Published: 2024-08-15T07:00:00Z Content type: release Language: en Sources: [Supabase Blog]() Topics: [Logging](), [observability pipelines](), [Supabase](), [SIEM, Security](), [observability](), [Self-hosted](), [Cross-origin resource sharing (CORS)](), [backends](), [JSON](), [BigQuery](), [PostgreSQL]() Tags: [analytics](), [bigquery](), [datadog](), [http](), [json](), [logging](), [logs](), [monitoring](), [observability](), [observability-pipelines](), [postgresql](), [product](), [security](), [self-hosted](), [self-hosting]() ### AI overview Supabase introduces Log Drains for Team and Enterprise users, enabling logs from Database, Storage, Realtime, and Auth to be exported to Datadog Logs or custom HTTP endpoints. The feature supports alerting, observability pipelines, SIEM integrations, extended retention, self-hosting, and local development. ### Source excerpt Log Drains for exporting product logs is now available under Public Alpha ## My first 3 months at incident.io DevFeed: [My first 3 months at incident.io]() Original publisher: [Read original article]() Author: Lambert Le Manh Published: 2024-08-02T15:14:00Z Content type: article Language: en Sources: [The incident.io Blog]() Topics: [data](), [BigQuery](), [Docker](), [Cloud](), [Docker Image](), [ci](), [PostgreSQL](), [cloud-infrastructure](), [Docker Compose]() Tags: [bigquery](), [ci](), [cloud](), [cost](), [data](), [development](), [docker](), [docker-image](), [incident](), [incident-channel](), [incident-management](), [incident-response](), [json](), [outage](), [platform](), [post-mortem](), [postgresql](), [registry](), [slack-incident](), [storage](), [tools]() ### AI overview A Data Engineer reviews their first three months at incident.io, focusing on scaling the development platform, evaluating alternatives to Fivetran for PostgreSQL data ingestion, and reducing platform costs. The work included storing dbt artifacts in Google Cloud Storage and using custom Docker images in CircleCI and production workflows. ### Source excerpt A review of my first three months as Data Engineer on the Data team working on improving the development platform, experimenting with different data ingestion tools, and cutting down on overall platform costs. ## Build a real-time dashboard over BigQuery DevFeed: [Build a real-time dashboard over BigQuery]() Original publisher: [Read original article]() Author: Cameron Archer Published: 2023-10-20T00:00:00Z Content type: tutorial Language: en Sources: [Tinybird]() Topics: [BigQuery](), [dashboards](), [real-time](), [data]() Tags: [bigquery](), [build](), [data](), [how-to](), [i-built-this](), [real-time](), [warehouse]() ### AI overview A tutorial about building a real-time dashboard over BigQuery, with sub-second updates to warehouse data. ### Source excerpt BigQuery real-time dashboard sounds impossible. It's not. Here's how to add sub-second updates to your warehouse data. ## Serverless Data Pipelines in GCP using Dataform and BigQuery Remote Functions DevFeed: [Serverless Data Pipelines in GCP using Dataform and BigQuery Remote Functions]() Original publisher: [Read original article]() Author: Austen Novis Published: 2023-08-23T13:35:22Z Content type: tutorial Language: en Sources: [Engineering at Premise - Medium]() Topics: [Google Cloud Platform (GCP)](), [Serverless](), [data-engineering](), [BigQuery](), [SQL](), [GitHub](), [GitLab](), [dataset]() Tags: [bigquery](), [cloud-computing](), [dataform](), [deployment](), [gcp](), [github](), [gitlab](), [goblet](), [how-to](), [pipelines](), [serverless](), [sql]() ### AI overview This tutorial explains how to build serverless data pipelines on Google Cloud Platform using Dataform and BigQuery Remote Functions. It describes Dataform's SQL-based pipeline orchestration, version-control and monitoring integrations, and shows how Remote Functions can add custom logic through external Cloud Run or Cloud Functions services. ### Source excerpt Photo by Growtika on Unsplash Dataform is an amazing new offering from Google Cloud Platform (GCP) that allows for serverless orchestration for data pipelines. Previously, data engineering pipelines were either supported by separate data engineering teams, which many smaller organizations don't have the resources to support, or were stitched together using a series of interrelated BigQuery scheduled queries, which could be time-consuming to build and error-prone to run and maintain. Dataform solves these problems, especially for data scientists and data analysts, by including features such as Ability to build data pipelines using SQL Integrates with GitHub and GitLab for code versioning and deployment Monitor pipelines for failures and trigger alerts The main limitation of Dataform is that it only supports SQL, with limited ability to add javascript, but this limitation can be overcome with BigQuery Remote Functions. BigQuery Remote Functions allow you to call external services running in Cloudrun or Cloudfunctions from BigQuery, making it possible to have a serverless data pipeline that can run custom logic in any programming language that you may need. The rest of this blog post will go through a complete example on how to setup both Dataform and BigQuery Remote Functions in just a few steps. Setup Dataform We will begin by setting up a simple Dataform pipeline in the GCP console. Navigate to Dataform and create a repository. Create Repository Make sure to grant the default Dataform service account service-PROJECT_ID@gcp-sa-dataform.iam.gserviceaccount.com BigQuery read and write permissions. This can be accomplished using the roles/bigquery.user role. Next create a new development workspace and allow GCP to initialize it, which will add the default files needed to run the pipeline. Dataform files Next create a new BigQuery dataset called tutorial , and inside a table called test with schema id: int, val: int, type: str . BigQuery Schema We can insert some test da ## How the Guardian's Ophan Analytics Tool Moved from Elasticsearch Rollups to BigQuery DevFeed: [How the Guardian's Ophan Analytics Tool Moved from Elasticsearch Rollups to BigQuery]() Original publisher: [Read original article]() Author: Sam Hession Published: 2023-06-07T12:44:33Z Content type: article Language: en Sources: [Guardian]() Topics: [Monitoring](), [elasticsearch](), [real-time](), [BigQuery](), [data]() Tags: [analytics](), [bigquery](), [data](), [elasticsearch](), [monitoring](), [pipeline](), [real-time](), [warehouse]() ### AI overview The Guardian describes how Ophan, its real-time analytics tool, expanded from short-term pageview monitoring to historical data analysis. Because Elasticsearch Rollups remained in technical preview and lacked service-level guarantees, the team considered moving the long-term data pipeline to BigQuery. ### Source excerpt How the Guardian's real time analytics tool pivoted from ElasticSearch Rollups to BigQuery and what we learnt along the way Ophan is the Guardian's in-house developed real time analytics tool which allows us to see how our content is performing in real-time, providing our Editorial teams with the insights they need to curate and promote our journalism. Its intuitive ways of monitoring reader engagement with content help the teams to evaluate whether it's performing optimally. Ophan's distinguishing feature has always been to provide journalists with visualised real time pageview information, with features based on the previous two weeks. More recently we have expanded these capabilities to provide insights for wider timescales with years of data available at their fingertips. Continue reading... ## Managing Cloud Messaging Tokens DevFeed: [Managing Cloud Messaging Tokens]() Original publisher: [Read original article]() Author: Andrea Wu Published: 2023-04-13T00:00:00Z Content type: tutorial Language: en Sources: [Firebase Blog]() Topics: [Firebase](), [SDKs](), [Cloud Functions](), [Firestore](), [BigQuery](), [API](), [App](), [backends]() Tags: [android](), [apis](), [bigquery](), [cloud-functions](), [cloud-messaging](), [code](), [firebase](), [firestore](), [sdk](), [tutorials]() ### AI overview This Firebase tutorial explains how to generate, retrieve, store, and update Firebase Cloud Messaging registration tokens. It covers token changes, token freshness, and examples using Cloud Firestore and Cloud Functions for Firebase, with notes for applications that use their own servers. ### Source excerpt News, tutorials, and updates from the Firebase team. ## Using Tinybird with BigQuery to Build Real-Time Applications and APIs DevFeed: [Using Tinybird with BigQuery to Build Real-Time Applications and APIs]() Original publisher: [Read original article]() Author: Alejandro Martín Published: 2023-02-28T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [BigQuery](), [real-time](), [data]() Tags: [apis](), [bigquery](), [data](), [product-updates](), [real-time]() ### AI overview The article presents Tinybird's BigQuery connector for querying BigQuery data in milliseconds and publishing APIs without moving data warehouses. ### Source excerpt Your BigQuery data is powerful but slow for apps. Now you can query it in milliseconds and publish APIs without moving warehouses. [Next page]()