# redshift Published articles for redshift. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Five Years of Kafka at Razorpay's UPI Switch DevFeed: [Five Years of Kafka at Razorpay's UPI Switch]() Original publisher: [Read original article]() Author: Kshitij Nawandar Published: 2026-09-07T09:09:58Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [Kafka](), [Architecture & Design](), [event driven](), [Amazon Simple Queue Service (SQS)](), [distributed-systems](), [Microservices](), [Amazon Web Services]() Tags: [aws](), [aws-sns](), [distributed-systems](), [event-driven](), [kafka](), [microservices](), [razorpay](), [redshift](), [sns](), [sqs]() ### AI overview Razorpay describes five years of evolution in its UPI Switch, from a monolith using AWS SQS to Kafka-based infrastructure. The article covers architectural decisions, operational challenges, and optimization work affecting payment-processing performance, reliability, and scale. ### Source excerpt Preface The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay's total UPI volume. Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure. This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on. The First Version: Monolith and SQS When we began building the UPI Switch, we weren't thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately. For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn't need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL. The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows: Update NPCI with an API call Send callbacks to merchants about payment status Push structured data into our warehouse (AWS Redshift) To handle this, we started bolting on AWS SNS plus SQS for fan-ou ## Optimizing Redshift Write Patterns: Tackling Tombstones and Ghost Rows DevFeed: [Optimizing Redshift Write Patterns: Tackling Tombstones and Ghost Rows]() Original publisher: [Read original article]() Author: Harichandan Pulagam Published: 2026-08-24T20:18:12Z Content type: article Language: en Sources: [Wealthfront]() Topics: [Amazon Redshift](), [dashboards](), [Data Quality]() Tags: [amazon-redshift](), [batch](), [dashboards](), [data-quality](), [latency](), [load](), [performance](), [redshift](), [space](), [wealthfront-engineering]() ### AI overview This Wealthfront engineering post examines how Redshift tables grew to nearly 10 times the size of their useful data because deleted rows remained on disk as ghost rows. It describes the resulting read and write latency and the write strategies adopted to control table size. ### Source excerpt Amazon Redshift is a core part of our analytics platform, powering dashboards, data quality checks, ad-hoc analytical workloads, and downstream reporting on a shared cluster. Because everything runs on the same cluster, the size and health of our tables directly affects every workload. At Wealthfront, data drives every decision we make, which means any performance... Read more ## How the 5 major cloud data warehouses compare on cost-performance DevFeed: [How the 5 major cloud data warehouses compare on cost-performance]() Original publisher: [Read original article]() Author: Tom Schreiber; Lionel Palacin Published: 2025-12-02T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [benchmarking](), [Cloud](), [clickhouse](), [Amazon Redshift](), [BigQuery](), [databricks](), [dataset]() Tags: [benchmark](), [benchmarking](), [bigquery](), [clickhouse](), [cloud](), [cloud-data](), [compare](), [comparison](), [cost](), [data](), [databricks](), [dataset](), [performance](), [redshift](), [storage](), [warehouse]() ### AI overview This article compares the cost-performance of Snowflake, Databricks, ClickHouse Cloud, BigQuery, and Redshift across analytical workloads containing 1 billion, 10 billion, and 100 billion rows. Using each system's real compute billing model, the benchmark concludes that ClickHouse Cloud provides substantially better value than the other systems at scale. ### Source excerpt We benchmarked the five major cloud data warehouses at 1B-100B rows using their real billing models to measure performance per dollar. Results show how cost-performance shifts as data grows. ## Why HTAP Databases Are Giving Way to Disaggregated Architectures DevFeed: [Why HTAP Databases Are Giving Way to Disaggregated Architectures]() Original publisher: [Read original article]() Author: Zhou Sun Published: 2025-05-04T10:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Databases](), [olap](), [Cloud](), [Latency](), [consistency](), [NoSQL](), [MongoDB](), [Hadoop](), [hdfs](), [CockroachDB](), [Amazon Redshift](), [vitess]() Tags: [cloud](), [cloud-data](), [cockroachdb](), [consistency](), [data-lake](), [databases](), [engineering](), [hadoop](), [hdfs](), [latency](), [mongodb](), [olap](), [redshift](), [sql](), [vitess]() ### AI overview The article traces the separation of transactional and analytical database workloads, explaining how differing storage and scaling requirements led to specialized OLTP and OLAP systems. It argues that HTAP as a single database architecture is declining while its underlying ideas persist in today's disaggregated data stack. ### Source excerpt This blog is inspired by Jordan Tigani's "Big Data is Dead." Jordan and I actually spent some time building an HTAP database at SingleStore. From the one database that did everything in the '80s, to the great divide, to HTAP, to today's disaggregated stack--here's why HTAP as a database is dead, but its spirit lives on. ## Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block DevFeed: [Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block]() Original publisher: [Read original article]() Author: Unni Krishnan Published: 2025-03-20T00:00:00Z Content type: article Language: en Sources: [Cash App Code Blog]() Topics: [data-processing](), [Kafka](), [migration](), [databricks](), [data lake](), [Apache Spark](), [Amazon S3](), [parquet](), [Amazon Redshift]() Tags: [acquisition](), [amazon](), [cloud](), [data](), [data-lake](), [data-processing](), [databricks](), [delta-lake](), [emr](), [kafka](), [parquet](), [pipelines](), [redshift](), [s3](), [scale](), [storage]() ### AI overview Project Teleport is Block's cross-region Kafka data-processing system for integrating Afterpay's Sydney-hosted data lake into Block's US-based ecosystem. Built with Delta Lake, Spark on Databricks, and object storage, it supports migration of legacy pipelines and reduced cloud egress costs by USD 540,000 per year. ### Source excerpt Teleport achieves efficient and reliable cross-region Kafka data processing at scale. Using this approach, Afterpay data team reduced cloud egress costs by USD 540,000 per year. ## Logging millions of requests reliably with our new data ingestion pipeline DevFeed: [Logging millions of requests reliably with our new data ingestion pipeline]() Original publisher: [Read original article]() Published: 2022-07-01T00:00:00Z Content type: article Language: en Sources: [HackerEarth]() Topics: [Logging](), [Amazon Redshift](), [log management](), [real-time](), [Amazon Web Services](), [cloud-infrastructure](), [web applications](), [Security](), [Architecture & Design]() Tags: [analytics](), [aws](), [infrastructure](), [logging](), [pipeline](), [real-time](), [redshift](), [security](), [server]() ### AI overview HackerEarth describes redesigning its request-logging architecture to reliably ingest millions of HTTP/HTTPS requests in near real time. The new pipeline uses Kinesis Firehose data streams to deliver compressed and encrypted data from web servers to Amazon Redshift, reducing operational overhead and infrastructure cost while providing retries and delivery within 10-15 minutes. ### Source excerpt Introduction HackerEarth handles millions of requests every day. To understand the user access patterns, get the usage of any particular feature or a page, figure out the daily active users or users who have been active for the past 6 months, etc in near real time, it is important to stream that data from across different services and ingest it to the analytics pipeline reliably. Problem Our old request logging architecture was complex and has many moving components. There were a lot of operational overheads involved in maintaining and scaling each of those components independently to ensure that all the self-hosted components were up and running all the time. Architecture Solution Last year, we revamped the way we log our web requests. It was done mainly to increase the reliability in logging the HTTP/HTTPS request data from across web services and also to reduce the operational overheads and the infrastructure cost associated with it. The new flow is making use of Kinesis Firehose data streams to deliver the request data from our web servers to Redshift (the database that we use to log and query request data) reliably with much lower cost. Amazon Kinesis Firehose is a fully managed service that automatically scales to match the throughput of our incoming request log data and requires no ongoing administration. It also allows us to compress and encrypt the data before loading it, minimizing the amount of storage used at the destination with increased security and we only need to pay for the amount of data we transmit through the service. The new flow is a fully managed solution with almost no operational overhead. We tried to keep the flow simple and straightforward with less number of moving components. The request data is now guaranteed to appear in the Redshift table within 10-15 minutes from the point we received a request at our web server's end. There are retries configured between consecutive infra components in the new flow to make sure there are no message ## Teleport 6.2 - Redshift, Listing Databases, and K8S in the UI DevFeed: [Teleport 6.2 - Redshift, Listing Databases, and K8S in the UI]() Original publisher: [Read original article]() Author: ben@goteleport.com (Ben Arent) Published: 2021-05-27T00:00:00Z Content type: release Language: en Sources: [Teleport]() Topics: [Amazon Redshift](), [Databases](), [Kubernetes clusters](), [Grafana](), [JSON Web Tokens](), [Jenkins]() Tags: [amazon-redshift](), [databases](), [grafana](), [jenkins](), [jwt](), [kubernetes-clusters](), [redshift](), [release]() ### AI overview Teleport 6.2 is a release with improvements to Application Access, Kubernetes Access, Database Access, Server Access, and Trusted Clusters. It adds Amazon Redshift support, database and Kubernetes cluster listing in the UI, pass-through headers, native JWT authentication for Grafana, and other fixes. ### Source excerpt Teleport 6.2 brings enhancements across the board, including the ability to list Databases and Kubernetes Clusters in Teleport and Amazon Redshift support. ## Let me automate that for you II, Electric Bugaloo DevFeed: [Let me automate that for you II, Electric Bugaloo]() Original publisher: [Read original article]() Author: GameChanger Published: 2021-05-07T18:29:38Z Content type: article Language: en Sources: [GameChanger]() Topics: [SQL](), [Amazon Redshift](), [Pull Request](), [Optimization](), [Structured-data](), [Script](), [Slack](), [Documentation]() Tags: [automation](), [blog](), [blog-post](), [building](), [data](), [documentation](), [github](), [kafka](), [optimization](), [performance](), [pipeline](), [reactive](), [redshift](), [sql](), [systems](), [warehouse]() ### AI overview The article describes evolving an embedded SQL generator and related scripts into a standalone SQL producer for warehouse schema and table management. The system generates SQL migrations, creates or updates tables, optimizes table performance, documents proposed changes, opens pull requests, and notifies engineers in Slack for review. The article also discusses limitations of the reactive original implementation and outlines the design of the improved system. ### Source excerpt Improving our original, embedded SQL generator and some related scripts by converting them to a better, long term, stand alone SQL producer that's faster, more reliable, and more obvious. About seventeen years ago, in 2019, I published my blog post "Let me automate that for you" about a design for automating creating warehouse tables based on schemas for new event data. The idea was when our ETL system couldn't load waiting data into a warehouse table (as there was no table to be found), it would look up the schema for that data, convert the schema to a SQL statement, then issue a PR to the repo where SQL migrations for such needs are kept. Eventually creating tables made a friend, updating tables when there was a mismatch between the schema of the data we were loading and the schema of the table in the warehouse, and a third buddy joined the part, optimizing a table to improve its performance. The system had some absolutely great qualities: it automated acting on errors it saw, it generated great documentation in the PR and the SQL statement (with comments for discussions and places to review more closely), and it posted to Slack to let engineers know that there was something for them to do a final review on. However... it wasn't perfect. Reading is going toward something that is about to be, and no one yet knows what it will be. [1] Let me take you through the evolution of our embedded SQL generator to stand-alone SQL producer. Limitations of previous implementation Opportunities to build it better Building blocks of a stand-alone SQL producer Joining the human needs with the computer's logic Detailed breakdown of the services available Troubleshooting live Final thoughts Appendix A: Redshift optimization queries Appendix B: select Github logic Limitations of previous implementation While the SQL generator eased so much work for so many different people in the company, it had some... strange caveats, shall we say. Some were more noticable than others but all were, in ## From pipeline to beyond DevFeed: [From pipeline to beyond]() Original publisher: [Read original article]() Author: GameChanger Published: 2021-05-05T09:00:18Z Content type: article Language: en Sources: [GameChanger]() Topics: [Kafka](), [Amazon S3](), [data](), [streaming-data-processing](), [Back end]() Tags: [analysis](), [backend](), [consul](), [data](), [infrastructure](), [kafka](), [pipeline](), [redshift](), [s3](), [terraform](), [warehouse]() ### AI overview The article examines options for moving data out of Kafka into a data warehouse and archive. It discusses off-the-shelf tools such as Kafka Connect, Secor, and Gobblin, the limitations encountered, and the development of a custom solution. The requirements include preserving Avro data and schemas, writing to S3, and partitioning data by processing time or event time. ### Source excerpt An overview of off-the-shelf solutions for moving data out of Kafka, problems we had making those systems work, and how we wrote our own solution and stood it up for those in a similar situation. You cannot know everything a system will be used for when you start: it is only at the end of its life you can have such certainty. [1] Many moons ago I wrote about our design for upgrading our data pipeline, which lightly touched on how we'd move data out of our pipeline (Kafka) to downstream systems, namely our data warehouse and data archive. At that time we hadn't really been able to dive into focusing on getting the data out of Kafka, because getting data in to Kafka is often much more custom and complex, and we thought we'd be able to use an off the shelf solution like Kafka Connect to move data out, don't even worry about it. We were, uh -- we were wrong. Let me take you on our journey, in case you're on this journey too. The problem space Solution 1: Kafka Connect Solution 2: Secor or Gobblin Solution 3: we'll do this ourselves Tangent: naming things How you can do this yourselves, code edition How you can do this yourselves, infrastructure and metrics edition Takeaways The problem space Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis. [4] At a high level, the problem we needed a solution for was as follows: Data enters the data pipeline from numerous backend systems. This crossover point is producers into the pipeline, which we'd already implemented. Data from the data pipeline needs to move into the data warehouse and the data archive. This crossover point would be a consumer on the pipeline. Ideally we'd like the same consumer for both needs that we can simply configure differently. We want to preserve our data's Avro format along side its schemas. This would allow every system that interacts with the data to use the same language. We want to write our data to S3. data warehouse: This will be our ## Crash Course to Redshift DevFeed: [Crash Course to Redshift]() Original publisher: [Read original article]() Author: GameChanger Published: 2020-03-30T14:36:13Z Content type: tutorial Language: en Sources: [GameChanger]() Topics: [Amazon Redshift](), [Databases](), [data](), [IO](), [Optimization](), [Development](), [debugging]() Tags: [data](), [database](), [debugging](), [development](), [distributed](), [documentation](), [how-to](), [io](), [optimization](), [performance](), [redshift](), [scale](), [warehouse]() ### AI overview A crash course to Amazon Redshift that introduces its database architecture, large-scale table design considerations, data distribution, data loading, debugging, and query-performance optimization. ### Source excerpt Redshift. It can store insane amounts of data. It can also store insane amounts of surprises, considerations, new ideas to learn, skewed tables to fix, distributions to get in line, what's a WLM, what am I doing‽ This post is meant to give you a crash course into working with Redshift, to get you off and running until you have the time and resources to come back and internalize what it all means. This is by no means a comprehensive review of Redshift, as then it'd no longer be a crash course, nor does this dive into data warehousing specifics, which I can cover in another post if people want. At a high level what I'll be covering is: Introduction to Redshift Table design Table analysis Data loading Debugging The vast majority of this post actually comes from our internal documentation, so you can trust that we do use this to help educate those less familiar with Redshift, and get them ramped up and feeling comfortable. Introduction to Redshift On Redshift The Redshift database will behave like other databases you've encountered, but under the hood it has some extra considerations to take into account. The main difference between Redshift and most other databases you'll have encountered is due to scale, with the cluster being important to keep in mind in table design along with standard table design considerations. And since the scale is so much larger, the impact of IO can go up considerably, especially if the cluster needs to move or share data to perform a query. The reasons for this and how to best avoid these inefficiencies are detailed below. More on Redshift database development here. On distributing data Within a Redshift cluster, there is a leader node and many compute nodes. The leader node helps orchestrate the work the compute nodes do. For example, if a query is operating only on data from May of 2017, and all of that data is stored on a single compute node, the leader only needs that node to perform the work. If instead a query is operating on data from ## ETL job processing with Serverless, Lambda, and AWS Redshift DevFeed: [ETL job processing with Serverless, Lambda, and AWS Redshift]() Original publisher: [Read original article]() Author: Rupak Ganguly Published: 2018-02-20T00:00:00Z Content type: tutorial Language: en Sources: [Serverless Blog]() Topics: [Serverless](), [Amazon Redshift](), [Amazon Web Services](), [API](), [data]() Tags: [api](), [aws](), [aws-lambda](), [cloud-computing](), [data](), [engineering](), [etl](), [faas](), [function-as-a-service](), [infrastructure-as-code](), [redshift](), [serverless](), [serverless-architecture](), [serverless-framework]() ### AI overview The article describes building an ETL job service that fetches data from a public API endpoint and stores it in an AWS Redshift database. ### Source excerpt Build an ETL job service by fetching data from a public API endpoint and dumping it into an AWS Redshift database. ## Faster Snowflake Queries through Clustering DevFeed: [Faster Snowflake Queries through Clustering]() Original publisher: [Read original article]() Author: Michal Klos Published: 2017-03-13T14:17:01Z Content type: article Language: en Sources: [Localytics]() Topics: [data-processing](), [Databases](), [Query (disambiguation)](), [Amazon Redshift]() Tags: [clustering](), [data-processing](), [database](), [databases](), [join](), [mpp](), [mysql](), [partitioning](), [performance](), [redshift](), [s3](), [scale](), [snowflake](), [time-series]() ### AI overview This article explains how Snowflake clustering and re-clustering can improve query performance by helping the query engine prune unneeded data. It discusses choosing cluster keys, comparisons with MPP database techniques, and the caveat that naturally ordered time-series data may not need clustering. ### Source excerpt At Localytics we have petabytes of data that needs to be served at low latencies and we use Snowflake in our mix of data processing technologies. Snowflake, like many other MPP databases, has a way of partitioning data to optimize read-time performance by allowing the query engine to prune ## Speeding things up with Redshift DevFeed: [Speeding things up with Redshift]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-04-25T14:30:19Z Content type: opinion Language: en Sources: [Bitly]() Topics: [Amazon Redshift](), [Data analysis](), [SQL](), [Hadoop](), [Python](), [amazon]() Tags: [amazon](), [analysis](), [data-analysis](), [gotchas](), [hadoop](), [python](), [redshift](), [speed](), [sql](), [workflow]() ### AI overview The article describes bitly's experience using Amazon Redshift to speed up ad hoc analysis of large volumes of click data. It contrasts Redshift SQL queries with a slower Hadoop and Python MapReduce workflow, reporting answers within minutes or seconds and an overall positive experience, while noting some gotchas. ### Source excerpt Recently we've started to experiment with using Redshift, Amazon's new data warehousing service. More specifically, we're using it to speed up and expand our ad hoc data analysis. The Challenge bitly sees billions of clicks and shortens each month. Often we have various questions about the data generated from this activity. Sometimes these questions are driven by business needs (how much traffic do we see from a potential enterprise customer), sometimes they are more technically driven (how much traffic will a new sub-system need to deal with), and sometimes we like to just have fun (what are the top trashy celeb stories this week). Unfortunately, when working with that volume of data it can be pretty difficult to do much of anything quickly. Pre-Redshift, all of these questions were answered by writing map-reduce jobs to be run on our Hadoop cluster or on Amazon's EMR. Whenever we wanted to answer a question with our data, the process would look something like this: Write map-reduce job in Python Run it on some local test data Fix bugs. Run it on the Hadoop cluster Wait 20-30 minutes for results Get an error back from Hadoop Dig through the logs to find the error. GOTO 3 This is clearly not ideal when all you want to do is get a simple count. For a lot of the work we do Hadoop + Python make for an awesome combination, but for these ad hoc aggregation queries they're very blunt instruments. In both cases, they are general purpose tools that are super flexible, but slow and difficult to use for this specific use case. Redshift, on the other hand, is specifically built and optimized for doing aggregation queries over large sets of data. When we want to answer a question with Redshift, we just write a SQL query and get an answer within a few minutes--if not seconds. Overall, our experience with Redshift has been a positive one but we have run into some gotchas that we'll get into below. The Good News User Experience From a user perspective, we're really happy with Redsh