# Serverless Data Pipelines in GCP using Dataform and BigQuery Remote Functions

DevFeed: [Serverless Data Pipelines in GCP using Dataform and BigQuery Remote Functions](<https://devfeed.tech/articles/serverless-data-pipelines-in-gcp-using-dataform-and-bigquery-remote-functions-23880.md>)

Original publisher: [Read original article](<https://engineering.premise.com/serverless-data-pipelines-in-gcp-using-dataform-and-bigquery-remote-functions-9ee235d0cb18?source=rss----c5fada0a103d---4>)

Author: Austen Novis

Published: 2023-08-23T13:35:22Z

Content type: tutorial

Language: en

Sources: [Engineering at Premise - Medium](<https://devfeed.tech/sources/engineering-at-premise-medium.md>)

Topics: [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [GitHub](<https://devfeed.tech/topics/github.md>), [GitLab](<https://devfeed.tech/topics/gitlab.md>), [dataset](<https://devfeed.tech/topics/dataset.md>)

Tags: [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [dataform](<https://devfeed.tech/tags/dataform.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [github](<https://devfeed.tech/tags/github.md>), [gitlab](<https://devfeed.tech/tags/gitlab.md>), [goblet](<https://devfeed.tech/tags/goblet.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [sql](<https://devfeed.tech/tags/sql.md>)

## AI overview

This tutorial explains how to build serverless data pipelines on Google Cloud Platform using Dataform and BigQuery Remote Functions. It describes Dataform's SQL-based pipeline orchestration, version-control and monitoring integrations, and shows how Remote Functions can add custom logic through external Cloud Run or Cloud Functions services.

## Source excerpt

Photo by Growtika on Unsplash Dataform is an amazing new offering from Google Cloud Platform (GCP) that allows for serverless orchestration for data pipelines. Previously, data engineering pipelines were either supported by separate data engineering teams, which many smaller organizations don't have the resources to support, or were stitched together using a series of interrelated BigQuery scheduled queries, which could be time-consuming to build and error-prone to run and maintain. Dataform solves these problems, especially for data scientists and data analysts, by including features such as Ability to build data pipelines using SQL Integrates with GitHub and GitLab for code versioning and deployment Monitor pipelines for failures and trigger alerts The main limitation of Dataform is that it only supports SQL, with limited ability to add javascript, but this limitation can be overcome with BigQuery Remote Functions. BigQuery Remote Functions allow you to call external services running in Cloudrun or Cloudfunctions from BigQuery, making it possible to have a serverless data pipeline that can run custom logic in any programming language that you may need. The rest of this blog post will go through a complete example on how to setup both Dataform and BigQuery Remote Functions in just a few steps. Setup Dataform We will begin by setting up a simple Dataform pipeline in the GCP console. Navigate to Dataform and create a repository. Create Repository Make sure to grant the default Dataform service account service-PROJECT_ID@gcp-sa-dataform.iam.gserviceaccount.com BigQuery read and write permissions. This can be accomplished using the roles/bigquery.user role. Next create a new development workspace and allow GCP to initialize it, which will add the default files needed to run the pipeline. Dataform files Next create a new BigQuery dataset called tutorial , and inside a table called test with schema id: int, val: int, type: str . BigQuery Schema We can insert some test da