# warehouse Published articles for warehouse. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## MYWAI ports its VILMA visual imitation learning toolkit to Arduino UNO Q and VENTUNO Q DevFeed: [MYWAI ports its VILMA visual imitation learning toolkit to Arduino UNO Q and VENTUNO Q]() Original publisher: [Read original article]() Author: Arduino Team Published: 2026-09-15T14:26:17Z Content type: article Language: en Sources: [Arduino Blog]() Topics: [Robotics](), [Artificial Intelligence](), [Computer vision](), [Arduino](), [Qualcomm](), [UNO Q](), [VENTUNO Q]() Tags: [ai](), [arduino](), [computer-vision](), [edge-ai](), [industrial](), [machine-learning](), [qualcomm](), [robotics](), [robots](), [uno-q](), [ventuno-q](), [warehouse]() ### AI overview The article describes VILMA, an AI-powered toolkit from MYWAI that enables robots and humanoids to learn manipulation tasks from one-shot human demonstrations. It reports that the toolkit is being ported to Arduino UNO Q and VENTUNO Q boards powered by Qualcomm Dragonwing processors. ### Source excerpt Every day, hundreds of thousands of kits are prepared in warehouses before components ever reach an automotive production line. While robots have become commonplace in modern manufacturing, many upstream logistics activities still rely heavily on human operators performing repetitive pick-and-place and kitting tasks. What if robots could learn these operations the same way humans do: [...] The post MYWAI™ VILMA™ is designed to bring human-like learning to robots via one-shot demonstration appeared first on Arduino Blog. ## The hard part of an MCP gateway is auth DevFeed: [The hard part of an MCP gateway is auth]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-09-11T15:22:28Z Content type: opinion Language: en Sources: [WorkOS Blog]() Topics: [MCP Server](), [Authorization](), [Model Context Protocol](), [Artificial Intelligence]() Tags: [audit](), [auth](), [github](), [integration](), [mcp](), [mcp-gateway](), [mcp-server](), [salesforce](), [scopes](), [slack](), [warehouse]() ### AI overview The article examines Sierra's internal MCP gateway and argues that its most difficult challenges are authentication, identity, per-tool scopes, consent, and audit rather than MCP protocol design. It also discusses integration work and the lack of standardized gateway behavior. ### Source excerpt Sierra's MCP gateway iceberg is a field report on agent auth: identity, per-tool scopes, consent, and audit are the submerged mass, and they ship off the shelf. ## Your Data Warehouse Isn't Integrated Just Because the Tables Are in One Place DevFeed: [Your Data Warehouse Isn't Integrated Just Because the Tables Are in One Place]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-07-18T23:26:52Z Content type: opinion Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [centralization](), [data]() Tags: [centralization](), [data](), [warehouse]() ### AI overview The article argues that placing tables in one data warehouse centralizes data but does not by itself integrate it. ### Source excerpt Centralization is not integration! ## How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes]() Original publisher: [Read original article]() Author: Amit Prabhu Published: 2026-07-14T14:06:16Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [data](), [airflow](), [Apache Spark](), [Amazon S3](), [Apache Iceberg](), [microservices architecture](), [parquet](), [Architecture & Design]() Tags: [airflow](), [analytics](), [apache-iceberg](), [architecture](), [batch](), [data](), [graphs](), [infrastructure](), [parquet](), [pipelines](), [razorpay](), [spark](), [trino](), [warehouse]() ### AI overview Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions. ### Source excerpt Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o ## How the 5 major cloud data warehouses compare on cost-performance DevFeed: [How the 5 major cloud data warehouses compare on cost-performance]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2025-12-02T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [benchmarking](), [Cloud](), [clickhouse](), [Amazon Redshift](), [BigQuery](), [databricks](), [dataset]() Tags: [benchmark](), [benchmarking](), [bigquery](), [clickhouse](), [cloud](), [cloud-data](), [compare](), [comparison](), [cost](), [data](), [databricks](), [dataset](), [performance](), [redshift](), [storage](), [warehouse]() ### AI overview This article compares the cost-performance of Snowflake, Databricks, ClickHouse Cloud, BigQuery, and Redshift across analytical workloads containing 1 billion, 10 billion, and 100 billion rows. Using each system's real compute billing model, the benchmark concludes that ClickHouse Cloud provides substantially better value than the other systems at scale. ### Source excerpt We benchmarked the five major cloud data warehouses at 1B-100B rows using their real billing models to measure performance per dollar. Results show how cost-performance shifts as data grows. ## Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration DevFeed: [Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration]() Original publisher: [Read original article]() Author: Sudarshan Hiray Published: 2025-11-05T20:05:38Z Content type: article Language: en Sources: [Heroku]() Topics: [Heroku](), [data-architecture](), [Databases](), [Monitoring]() Tags: [analytics](), [applications](), [article](), [building](), [data](), [ecosystems](), [engineering](), [enterprise](), [etl](), [heroku](), [heroku-postgres](), [integration](), [platforms](), [salesforce](), [services](), [systems](), [tools](), [warehouse]() ### AI overview Heroku describes an enterprise data warehouse architecture that unifies Salesforce and data from multiple application databases for real-time analytics. The approach uses Heroku Connect, Heroku Postgres follower databases, and a unified analytics platform to address API limits, ETL complexity, production database load, and fragmented monitoring. ### Source excerpt Modern businesses don't just run on Salesforce--they run on entire ecosystems of applications. At Heroku, we operate dozens of services alongside our Salesforce instance such as billing systems, user management platforms, analytics engines, and support tools. Traditional approaches to unifying this data create more problems than they solve. In this article, we'll see how we [...] The post Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration appeared first on Heroku. ## A Date-Stamped Data Warehouse Setup DevFeed: [A Date-Stamped Data Warehouse Setup]() Original publisher: [Read original article]() Author: Sahar Massachi Published: 2025-10-24T21:03:30Z Content type: article Language: en Sources: [DataExpert.io Newsletter]() Topics: [data-engineering](), [data](), [Apache Iceberg]() Tags: [ab-testing](), [data](), [data-engineering](), [storage](), [warehouse]() ### AI overview The article presents date-stamping data as a simple organizing principle for building resilient data warehouses and pipelines. It explains how this approach can manage changes over time, support experimentation and metrics, and work with systems including Hive metastore, Iceberg, Delta, and Hudi. ### Source excerpt Storage is cheap, your time is not! ## Postgres Meets Analytics: CDC From Neon to ClickHouse Via PeerDB DevFeed: [Postgres Meets Analytics: CDC From Neon to ClickHouse Via PeerDB]() Original publisher: [Read original article]() Author: Sai Srirampur Published: 2024-10-02T16:39:48Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [clickhouse](), [Databases](), [real-time](), [Data analysis](), [Replication](), [data](), [autoscaling](), [web applications](), [Serverless](), [Open Source]() Tags: [analysis](), [analytics](), [autoscaling](), [clickhouse](), [code](), [community](), [data](), [databases](), [latency](), [low-latency](), [olap](), [open-source](), [performance](), [postgres](), [real-time](), [replication](), [warehouse](), [web-applications]() ### AI overview This article explains how Neon serverless Postgres and ClickHouse can be combined for transactional workloads and real-time analytics. It presents PeerDB's Change Data Capture and replication capabilities as the bridge for synchronizing Neon data to ClickHouse for customer-facing analytics and data warehousing. ### Source excerpt If you're building a data-driven application that handles large amounts of data, you may need to balance two different types of databases: a purpose-built operational DB for your transactions, and an analytical database for large-scale data analysis. A purpose-built analytical da... ## Building a Data Pipeline to Track Strava's Bad Events DevFeed: [Building a Data Pipeline to Track Strava's Bad Events]() Original publisher: [Read original article]() Author: Bisman Sodhi Published: 2024-01-08T20:19:46Z Content type: opinion Language: en Sources: [Strava Engineering]() Topics: [data](), [airflow](), [Amazon S3](), [Python](), [Amazon Web Services](), [JSON](), [dashboards](), [Computer science]() Tags: [airflow](), [apache-airflow](), [aws](), [data](), [data-engineering](), [data-pipeline](), [data-platforms](), [integrity](), [json](), [python](), [s3](), [software-engineering](), [strava](), [tableau](), [warehouse]() ### AI overview A software engineering intern describes building a daily Apache Airflow pipeline that extracts schema-invalid user behavior events from S3, decompresses them into JSON, and loads them into Snowflake. Staging tables protect production data from partial loads, while materialized SQL views and a Tableau dashboard improve querying and monitoring. ### Source excerpt Hi my name is Bisman and I studied Computer Science at University of California, Santa Barbara. During summer of 2022, I had the most amazing experience working as a Software Engineer Intern on Strava's Data Platform Team. In the first fews weeks, I learned the tools my team uses and then spent the rest of the time working on my project. TRACKING BAD EVENTS For my major summer project, I created a data pipeline that pulls user behavior data out of external storage and persists it in our data warehouse. Strava uses a service called Snowplow to collect this user behavior data, like loading a club page or uploading a profile photo. Sometimes, this data fails to match the schema that we've set, and a piece of data that fails this schema validation is called a bad event. Previously, these bad events were temporarily stored in an Elastic Search. Persisting this data in Snowflake, our data warehouse, makes it accessible to a wider audience. It also makes it easier to incorporate the bad events data with other services used at Strava. To start my project, I created a directed acyclic graph in Apache Airflow, a scheduling framework, using python that extracts bad events data from the S3, AWS's storage service, buckets on a daily cadence. This data was stored as gzip files on S3 which I decompressed and stored the data as JSON blobs. As I was working with billions of rows of data, it was important to maintain data integrity and take measures in case data failed to load from S3. Therefore, I loaded data into a staging table in Snowflake. The staging table ensured that if loading from S3 failed, the production table would remain untouched. This data was then loaded into the production table free of any partial data. After all the data was loaded into the production table, I created six view tables because there were six different types of bad events stored in the production table. I collaborated with our stakeholders -- data analysts -- throughout this process to craft tables bas ## Build a real-time dashboard over BigQuery DevFeed: [Build a real-time dashboard over BigQuery]() Original publisher: [Read original article]() Author: Cameron Archer Published: 2023-10-20T00:00:00Z Content type: tutorial Language: en Sources: [Tinybird]() Topics: [BigQuery](), [dashboards](), [real-time](), [data]() Tags: [bigquery](), [build](), [data](), [how-to](), [i-built-this](), [real-time](), [warehouse]() ### AI overview A tutorial about building a real-time dashboard over BigQuery, with sub-second updates to warehouse data. ### Source excerpt BigQuery real-time dashboard sounds impossible. It's not. Here's how to add sub-second updates to your warehouse data. ## How the Guardian's Ophan Analytics Tool Moved from Elasticsearch Rollups to BigQuery DevFeed: [How the Guardian's Ophan Analytics Tool Moved from Elasticsearch Rollups to BigQuery]() Original publisher: [Read original article]() Author: Sam Hession Published: 2023-06-07T12:44:33Z Content type: article Language: en Sources: [Guardian]() Topics: [Monitoring](), [elasticsearch](), [real-time](), [BigQuery](), [data]() Tags: [analytics](), [bigquery](), [data](), [elasticsearch](), [monitoring](), [pipeline](), [real-time](), [warehouse]() ### AI overview The Guardian describes how Ophan, its real-time analytics tool, expanded from short-term pageview monitoring to historical data analysis. Because Elasticsearch Rollups remained in technical preview and lacked service-level guarantees, the team considered moving the long-term data pipeline to BigQuery. ### Source excerpt How the Guardian's real time analytics tool pivoted from ElasticSearch Rollups to BigQuery and what we learnt along the way Ophan is the Guardian's in-house developed real time analytics tool which allows us to see how our content is performing in real-time, providing our Editorial teams with the insights they need to curate and promote our journalism. Its intuitive ways of monitoring reader engagement with content help the teams to evaluate whether it's performing optimally. Ophan's distinguishing feature has always been to provide journalists with visualised real time pageview information, with features based on the previous two weeks. More recently we have expanded these capabilities to provide insights for wider timescales with years of data available at their fingertips. Continue reading... ## Adding Speed to Snowflake Warehouses for Real-Time Solutions DevFeed: [Adding Speed to Snowflake Warehouses for Real-Time Solutions]() Original publisher: [Read original article]() Author: Alejandro Martín Published: 2023-04-19T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [real-time]() Tags: [product-updates](), [real-time](), [solutions](), [speed](), [warehouse]() ### AI overview The article discusses workarounds for adding speed to Snowflake warehouses for real-time solutions without replacing the warehouse. ### Source excerpt Real-time solutions with Snowflake require workarounds. Here's how to add speed to your warehouse without replacing it. ## Using Trino with Apache Airflow for (almost) all your data problems DevFeed: [Using Trino with Apache Airflow for (almost) all your data problems]() Original publisher: [Read original article]() Author: Philippe Gagnon, Brian Olsen Published: 2022-12-21T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [airflow](), [data](), [Data Infrastructure](), [Scalability](), [Availability](), [Databases](), [Software as a service]() Tags: [airflow](), [analytics](), [apache](), [architecture](), [availability](), [batch](), [data](), [data-infrastructure](), [databases](), [integration](), [saas](), [scalability](), [summit](), [trading](), [warehouse](), [workflow]() ### AI overview The article recaps a Trino Summit 2022 talk about using Apache Airflow to orchestrate Trino batch queries. It explains Trino's fault-tolerant execution mode and discusses moving queries closer to federated data sources to improve data availability, performance, scalability, and collaboration. ### Source excerpt As we close in on the final talks from Trino Summit 2022, this next talk dives into how to set up Trino for batch processing. Trino has historically been well-known for facilitating fast adhoc analytics queries as opposed to long-running, resource intensive batch/ETL queries. This is due to the fact that Trino kills queries that run out of resources in order to prioritize faster query execution. Earlier this year, Trino added features to better support batch queries with a new fault-tolerant execution mode. This mode backs up intermediate data during execution time, allowing Trino to restart individual query tasks on failure rather than a query stage or the query itself. Batch queries don't typically involve human intervention and run asynchronously. These tasks may depend on each other and have a complex workflow. This talk describes how to orchestrate this complexity using Airflow's new Trino integration to run Trino batch queries to solve (almost) all your data problems. ## Understanding the Data Warehouse DevFeed: [Understanding the Data Warehouse]() Original publisher: [Read original article]() Author: Alasdair Brown Published: 2022-11-29T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [data]() Tags: [blog](), [data](), [series](), [the-data-base](), [warehouse]() ### AI overview This article introduces the origins of data warehouses and explains what they enable. It is presented as the first article in a two-part series. ### Source excerpt What gave rise to the Data Warehouse? What do they enable? I answer these questions in this first of a two-blog series. ## Let me automate that for you II, Electric Bugaloo DevFeed: [Let me automate that for you II, Electric Bugaloo]() Original publisher: [Read original article]() Author: GameChanger Published: 2021-05-07T18:29:38Z Content type: article Language: en Sources: [GameChanger]() Topics: [SQL](), [Amazon Redshift](), [Pull Request](), [Optimization](), [Structured-data](), [Script](), [Slack](), [Documentation]() Tags: [automation](), [blog](), [blog-post](), [building](), [data](), [documentation](), [github](), [kafka](), [optimization](), [performance](), [pipeline](), [reactive](), [redshift](), [sql](), [systems](), [warehouse]() ### AI overview The article describes evolving an embedded SQL generator and related scripts into a standalone SQL producer for warehouse schema and table management. The system generates SQL migrations, creates or updates tables, optimizes table performance, documents proposed changes, opens pull requests, and notifies engineers in Slack for review. The article also discusses limitations of the reactive original implementation and outlines the design of the improved system. ### Source excerpt Improving our original, embedded SQL generator and some related scripts by converting them to a better, long term, stand alone SQL producer that's faster, more reliable, and more obvious. About seventeen years ago, in 2019, I published my blog post "Let me automate that for you" about a design for automating creating warehouse tables based on schemas for new event data. The idea was when our ETL system couldn't load waiting data into a warehouse table (as there was no table to be found), it would look up the schema for that data, convert the schema to a SQL statement, then issue a PR to the repo where SQL migrations for such needs are kept. Eventually creating tables made a friend, updating tables when there was a mismatch between the schema of the data we were loading and the schema of the table in the warehouse, and a third buddy joined the part, optimizing a table to improve its performance. The system had some absolutely great qualities: it automated acting on errors it saw, it generated great documentation in the PR and the SQL statement (with comments for discussions and places to review more closely), and it posted to Slack to let engineers know that there was something for them to do a final review on. However... it wasn't perfect. Reading is going toward something that is about to be, and no one yet knows what it will be. [1] Let me take you through the evolution of our embedded SQL generator to stand-alone SQL producer. Limitations of previous implementation Opportunities to build it better Building blocks of a stand-alone SQL producer Joining the human needs with the computer's logic Detailed breakdown of the services available Troubleshooting live Final thoughts Appendix A: Redshift optimization queries Appendix B: select Github logic Limitations of previous implementation While the SQL generator eased so much work for so many different people in the company, it had some... strange caveats, shall we say. Some were more noticable than others but all were, in ## From pipeline to beyond DevFeed: [From pipeline to beyond]() Original publisher: [Read original article]() Author: GameChanger Published: 2021-05-05T09:00:18Z Content type: article Language: en Sources: [GameChanger]() Topics: [Kafka](), [Amazon S3](), [data](), [streaming-data-processing](), [Back end]() Tags: [analysis](), [backend](), [consul](), [data](), [infrastructure](), [kafka](), [pipeline](), [redshift](), [s3](), [terraform](), [warehouse]() ### AI overview The article examines options for moving data out of Kafka into a data warehouse and archive. It discusses off-the-shelf tools such as Kafka Connect, Secor, and Gobblin, the limitations encountered, and the development of a custom solution. The requirements include preserving Avro data and schemas, writing to S3, and partitioning data by processing time or event time. ### Source excerpt An overview of off-the-shelf solutions for moving data out of Kafka, problems we had making those systems work, and how we wrote our own solution and stood it up for those in a similar situation. You cannot know everything a system will be used for when you start: it is only at the end of its life you can have such certainty. [1] Many moons ago I wrote about our design for upgrading our data pipeline, which lightly touched on how we'd move data out of our pipeline (Kafka) to downstream systems, namely our data warehouse and data archive. At that time we hadn't really been able to dive into focusing on getting the data out of Kafka, because getting data in to Kafka is often much more custom and complex, and we thought we'd be able to use an off the shelf solution like Kafka Connect to move data out, don't even worry about it. We were, uh -- we were wrong. Let me take you on our journey, in case you're on this journey too. The problem space Solution 1: Kafka Connect Solution 2: Secor or Gobblin Solution 3: we'll do this ourselves Tangent: naming things How you can do this yourselves, code edition How you can do this yourselves, infrastructure and metrics edition Takeaways The problem space Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis. [4] At a high level, the problem we needed a solution for was as follows: Data enters the data pipeline from numerous backend systems. This crossover point is producers into the pipeline, which we'd already implemented. Data from the data pipeline needs to move into the data warehouse and the data archive. This crossover point would be a consumer on the pipeline. Ideally we'd like the same consumer for both needs that we can simply configure differently. We want to preserve our data's Avro format along side its schemas. This would allow every system that interacts with the data to use the same language. We want to write our data to S3. data warehouse: This will be our ## Crash Course to Redshift DevFeed: [Crash Course to Redshift]() Original publisher: [Read original article]() Author: GameChanger Published: 2020-03-30T14:36:13Z Content type: tutorial Language: en Sources: [GameChanger]() Topics: [Amazon Redshift](), [Databases](), [data](), [IO](), [Optimization](), [Development](), [debugging]() Tags: [data](), [database](), [debugging](), [development](), [distributed](), [documentation](), [how-to](), [io](), [optimization](), [performance](), [redshift](), [scale](), [warehouse]() ### AI overview A crash course to Amazon Redshift that introduces its database architecture, large-scale table design considerations, data distribution, data loading, debugging, and query-performance optimization. ### Source excerpt Redshift. It can store insane amounts of data. It can also store insane amounts of surprises, considerations, new ideas to learn, skewed tables to fix, distributions to get in line, what's a WLM, what am I doing‽ This post is meant to give you a crash course into working with Redshift, to get you off and running until you have the time and resources to come back and internalize what it all means. This is by no means a comprehensive review of Redshift, as then it'd no longer be a crash course, nor does this dive into data warehousing specifics, which I can cover in another post if people want. At a high level what I'll be covering is: Introduction to Redshift Table design Table analysis Data loading Debugging The vast majority of this post actually comes from our internal documentation, so you can trust that we do use this to help educate those less familiar with Redshift, and get them ramped up and feeling comfortable. Introduction to Redshift On Redshift The Redshift database will behave like other databases you've encountered, but under the hood it has some extra considerations to take into account. The main difference between Redshift and most other databases you'll have encountered is due to scale, with the cluster being important to keep in mind in table design along with standard table design considerations. And since the scale is so much larger, the impact of IO can go up considerably, especially if the cluster needs to move or share data to perform a query. The reasons for this and how to best avoid these inefficiencies are detailed below. More on Redshift database development here. On distributing data Within a Redshift cluster, there is a leader node and many compute nodes. The leader node helps orchestrate the work the compute nodes do. For example, if a query is operating only on data from May of 2017, and all of that data is stored on a single compute node, the leader only needs that node to perform the work. If instead a query is operating on data from ## Let me automate that for you DevFeed: [Let me automate that for you]() Original publisher: [Read original article]() Author: GameChanger Published: 2019-09-20T18:29:38Z Content type: article Language: en Sources: [GameChanger]() Topics: [data](), [systems](), [Refactoring](), [Back end](), [Database](), [SQL]() Tags: [architecture](), [automation](), [backend](), [data](), [database](), [pipeline](), [refactor](), [refactoring](), [sql](), [tests](), [warehouse]() ### AI overview The article describes GameChanger's data pipeline and warehouse, where engineers configure producers, pipeline connections, and warehouse tables. A refactoring project simplified producers, removed boilerplate, and added tests, but creating or updating warehouse tables remained a difficult manual task because the warehouse uses different SQL syntax and lacks library-generated code. ### Source excerpt As GameChanger's data engineer, I oversee the data pipeline and data warehouse. Sounds simple, right? And at a high level, it is! Fig 1.1: high level architecture diagram. Some complexity removed due to it being kinda boring for this post. Producers produce into the pipeline, and our main consumer is the ETL job which moves data to our warehouse, enabling anybody to come get answers to their questions and see what's happening across all our systems. Boom: easy. Well, not quite. Who owns what Since data can come from any number of backend systems and teams, engineers are responsible for writing the setup that shepherds their data through the system: a producer that lives near their data, the pipe it travels through in the pipeline, and the warehouse table. This often means new data that I'm unfamiliar with arrives in our warehouse without me even knowing it's been set up, which is actually kind of neat: the system should be so easy to work with that you don't need the data engineer. After a recent refactoring project, producers were made as simple as possible with removed boilerplate and plenty of tests to automatically catch the most common bugs engineers encounter. Typically, engineers have no problems with making their producers. Fig 2.1: engineers before and after producer refactor project. Studies have shown that engineers prefer to be happy. The pipe their data travels through is set up by filling in a form and pressing a button. Again, engineers typically have no problems with this. It's the warehouse table that becomes a pain point. Follow the readme There are two times non data engineers need to interact with warehouse tables: they've created a new producer which needs a table for their data to land in. they've updated an existing producer which needs its table updated as well. The second point is trickier and easier to get wrong, but the first point proved just as difficult for many engineers and far more common, especially if the engineers in question had ## Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada DevFeed: [Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada]() Original publisher: [Read original article]() Author: Jason Hiltz-Laforge Published: 2019-02-07T16:30:00Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [Google Cloud Platform (GCP)](), [Terraform](), [Cloud](), [Kubernetes](), [migration](), [data](), [Networks](), [DDoS]() Tags: [attacks](), [cloud](), [cloud-platform](), [clusters](), [compute](), [core](), [engineering](), [google-cloud](), [google-cloud-platform](), [industry](), [infrastructure](), [kubernetes](), [migration](), [networks](), [platform](), [resiliency](), [retail](), [shopify](), [terraform](), [warehouse]() ### AI overview Shopify describes the engineering work required to support cannabis retailers in Canada after legalization in 2018. The team built a Montreal-based regional deployment on Google Cloud Platform to satisfy Canadian data-residency requirements, using Google Kubernetes Engine, Google Compute Engine, Terraform, regional clusters, workload segregation, and a regional data warehouse. It also modeled traffic scenarios and provisioned capacity for launches, media-driven surges, sales peaks, and possible denial-of-service attacks. ### Source excerpt On October 17th, 2018, Canada ended a 95-year history of cannabis prohibition. For Shopify, the legalization of cannabis marked a new industry entering the Canadian retail market and we worked with governments and licensed sellers across the country to provide a safe, reliable and scalable platform for their business. For our engineering team, it meant significant changes to our platform to meet the strict requirements for this newly regulated industry. ## Things to consider before building a serverless data warehouse DevFeed: [Things to consider before building a serverless data warehouse]() Original publisher: [Read original article]() Author: Ashan Fernando Published: 2018-08-29T00:00:00Z Content type: article Language: en Sources: [Serverless Blog]() Topics: [Serverless](), [data]() Tags: [aws](), [aws-lambda](), [cloud-computing](), [data](), [engineering](), [faas](), [function-as-a-service](), [infrastructure-as-code](), [post](), [serverless](), [serverless-architecture](), [serverless-framework](), [tips](), [warehouse]() ### AI overview A post about considerations and practical tips for building a serverless data warehouse. ### Source excerpt Is it time for the rise of the serverless data warehouse? Read this post to find out, and for some serverless data warehousing pro-tips and considerations. ## A Journey Towards a Custom Data Warehouse Solution Part 2: We Need Storage DevFeed: [A Journey Towards a Custom Data Warehouse Solution Part 2: We Need Storage]() Original publisher: [Read original article]() Author: Robert Bordo (robert@upday.com) Published: 2017-08-22T04:39:55Z Content type: article Language: en Sources: [Upday]() Topics: [data](), [data lake](), [Amazon S3](), [Amazon Redshift](), [Amazon Web Services](), [Scalability]() Tags: [aws](), [business-intelligence](), [data](), [data-lake](), [s3](), [scalability](), [storage](), [warehouse]() ### AI overview This article examines storage choices for a custom data warehouse. It describes application log data, the limitations of time-series databases for additional master and historical data, business intelligence access needs, and the use of Amazon S3 as a data lake while considering other storage options including AWS Redshift. ### Source excerpt In the beginning we created a cluster. And the cluster was without form, and void; and nulls were upon the face of the storage. As we learned in part 1 of our series, a data warehouse consists of several components. The key component is the storage. All the others group around it. But how can one draw a decision on which storage solution to adopt? What is out there anyway? Preface In a perfect world there would be only one kind of storage that fits all the needs of current DWH development and analysis. But since we are not living in that kind of place, we have several options. And the number of options increase the deeper one dives into the topic. There seem to be solutions for every use case you can think of. That might be a good starting point. What is our most common use case? What are we going to store? And how would we like to access our data in the end? Our major source is a massive amount of log data coming from our app. Everything the user does (e.g swiping through articles, selecting categories, leaving the app) is tracked, enriched with metadata (e.g. the user's location, app version, article identifier) and stored by a third-party service in big, semi-structured log files. Having only this source, a time series database like Graphite or InfluxDB could do the job. But also having slow changing master data, like user profiles, article metadata and maybe even to keep a history of data, this solution would not satisfy our current and future needs. Another thing that comes to my mind is how the data will be accessed by our final consumer (namely: Business Intelligence). Usually they use tools like Jasper Reports or Tableau for generating reports. For analyses we have to pre-aggregate the data to make queries more performant and translate raw information into a digestible format. What else is on the market? Storage good at bad at Example S3/Flat Files scalability, easy to use, data lake querying S3 Time Series DB handling time series data non time series data G