# sqs Published articles for sqs. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Five Years of Kafka at Razorpay's UPI Switch DevFeed: [Five Years of Kafka at Razorpay's UPI Switch]() Original publisher: [Read original article]() Author: Kshitij Nawandar Published: 2026-09-07T09:09:58Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [Kafka](), [Architecture & Design](), [event driven](), [Amazon Simple Queue Service (SQS)](), [distributed-systems](), [Microservices](), [Amazon Web Services]() Tags: [aws](), [aws-sns](), [distributed-systems](), [event-driven](), [kafka](), [microservices](), [razorpay](), [redshift](), [sns](), [sqs]() ### AI overview Razorpay describes five years of evolution in its UPI Switch, from a monolith using AWS SQS to Kafka-based infrastructure. The article covers architectural decisions, operational challenges, and optimization work affecting payment-processing performance, reliability, and scale. ### Source excerpt Preface The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay's total UPI volume. Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure. This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on. The First Version: Monolith and SQS When we began building the UPI Switch, we weren't thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately. For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn't need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL. The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows: Update NPCI with an API call Send callbacks to merchants about payment status Push structured data into our warehouse (AWS Redshift) To handle this, we started bolting on AWS SNS plus SQS for fan-ou ## Your SQS consumer can hang forever by default DevFeed: [Your SQS consumer can hang forever by default]() Original publisher: [Read original article]() Author: Ivan Cernja Published: 2026-07-28T00:00:00Z Content type: article Language: en Sources: [Encore Updates]() Topics: [Amazon Simple Queue Service (SQS)](), [SDK](), [Rust](), [Amazon Web Services]() Tags: [aws](), [health-check](), [queue](), [retry](), [rust](), [sdk](), [sqs](), [tcp](), [timeout]() ### AI overview The article explains that the AWS Rust SDK has no request timeout by default. A failed or silently dropped connection can leave an SQS receive waiting indefinitely, stopping a sequential consumer loop while the process and health check still appear healthy. It describes fixing the issue with a per-attempt timeout. ### Source excerpt The AWS Rust SDK ships no request timeout by default, so one SQS receive on a dead connection can hang a whole consumer with nothing in the logs. ## Incident Response as Code: Managing PagerDuty with Pulumi DevFeed: [Incident Response as Code: Managing PagerDuty with Pulumi]() Original publisher: [Read original article]() Author: Engin Diri Published: 2026-07-20T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [pulumi](), [Incident response](), [Amazon Web Services](), [TypeScript](), [Amazon Simple Queue Service (SQS)]() Tags: [aws](), [devops](), [incident](), [incident-response](), [infrastructure-as-code](), [pagerduty](), [pulumi](), [sqs](), [tutorials](), [typescript]() ### AI overview A tutorial for managing PagerDuty incident-response resources and AWS alarm wiring together in a Pulumi program. The demo uses TypeScript to define an on-call team, escalation policy, PagerDuty service, SQS dead-letter queue, CloudWatch alarm, and SNS forwarding. ### Source excerpt You usually find out in the postmortem: the alarm fired, but it paged a schedule nobody was on anymore. Or the service had been running in production for three months before anyone created the matching PagerDuty service, so the first person to notice the outage was a customer. The infrastructure was code, reviewed and versioned. The incident response setup was forty clicks in a web UI, done once, by someone who has since changed teams. PagerDuty's own engineering team has been making the case for managing PagerDuty as code for years, and the OneUptime folks recently published a hands-on guide to the PagerDuty Terraform provider that walks the whole resource catalog. I liked both posts, and you probably know where this is going: I want the same thing in TypeScript, with the on-call setup and the infrastructure it protects living in the same Pulumi program. Teams, schedules, escalation policies, services, and the CloudWatch alarm that pages you: one pulumi up. What you are building The demo models a small platform team that owns a checkout service on AWS. On the PagerDuty side: a team with three members, a weekly on-call rotation, an escalation policy with two levels, and a service with a CloudWatch integration. On the AWS side: a dead-letter queue, an alarm that fires when a message lands in it, and an SNS topic that forwards the alarm to PagerDuty. The interesting part is where the two meet: the integration key PagerDuty mints flows straight into the SNS subscription without ever touching a clipboard. flowchart LR DLQ["SQS dead-letter queue"] --> Alarm["CloudWatch alarm"] Alarm --> SNS["SNS topic"] SNS -->|"HTTPS subscription with integration key"| Int["CloudWatch integration"] Int --> Svc["PagerDuty service Checkout API"] Svc --> EP["Escalation policy"] EP -->|"level 1"| Sched["Weekly rotation Lena, Jonas, Mia"] EP -->|"level 2, after 15 min"| Lead["Team lead"] Everything on both sides of that diagram is one TypeScript program. If you would rather write ## Kafka vs RabbitMQ vs SQS DevFeed: [Kafka vs RabbitMQ vs SQS]() Original publisher: [Read original article]() Author: Ashish Pratap Singh Published: 2026-07-02T03:31:11Z Content type: comparison Language: en Sources: [AlgoMaster Newsletter]() Topics: [Amazon Simple Queue Service (SQS)](), [Kafka](), [RabbitMQ](), [Messaging](), [Back end]() Tags: [backend](), [kafka](), [messaging](), [queues](), [rabbitmq](), [sqs]() ### AI overview This comparison examines Kafka, RabbitMQ, and Amazon SQS for asynchronous systems. It explains that Kafka uses an append-only log with consumer offsets and replay, RabbitMQ routes messages through exchanges and queues with acknowledgments, and SQS provides managed queue-based messaging that consumers poll. ### Source excerpt Most backend systems eventually need one service to hand off work to another asynchronously. ## Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools DevFeed: [Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools]() Original publisher: [Read original article]() Author: R Tyler Croy Published: 2025-08-01T00:00:00Z Content type: tutorial Language: en Sources: [Scribd Tech]() Topics: [data](), [Amazon Web Services](), [databricks](), [event driven](), [Amazon Simple Queue Service (SQS)](), [Kafka](), [Open Source](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [architecture](), [aws](), [azure](), [catalog](), [cloud](), [cloud-native](), [data](), [databricks](), [deltalake](), [event-driven](), [featured](), [google-cloud](), [kafka](), [open-source](), [rust](), [sqs]() ### AI overview This article presents Scribd's cloud-native data-ingestion architecture for building large datasets for Delta Lake. It describes using AWS services and open-source tools such as kafka-delta-ingest, oxbow, and Airbyte in a more event-driven and reliable platform, with Databricks and Unity Catalog. The approach can also be adapted to Azure, Google Cloud Platform, or on-premises environments. ### Source excerpt Delta Lake is a fantastic technology for quickly querying massive data sets, but first you need those massive data sets! In this talk from Data and AI Summit 2025 I dive into the cloud-native architecture Scribd has adopted to ingest data from AWS Aurora, SQS, Kinesis Data Firehose and more! ## Order Dispatch Systems at Scale DevFeed: [Order Dispatch Systems at Scale]() Original publisher: [Read original article]() Author: Guest User Published: 2025-07-28T01:53:56Z Content type: tutorial Language: en Sources: [Kogan.com]() Topics: [Amazon Simple Queue Service (SQS)](), [event driven](), [Scalability](), [Amazon Web Services](), [Microservices](), [microservices architecture](), [Latency]() Tags: [aws](), [concurrency](), [event](), [event-driven](), [latency](), [microservices-architecture](), [scale](), [sqs]() ### AI overview This tutorial compares vertical and horizontal scaling, pull-based queues using AWS Simple Queue Service, and event-driven architecture for order dispatch systems. It describes queue trade-offs and explains how event subscribers can process orders independently and scale separately. ### Source excerpt How to load-balance like a seasoned waiter Software systems often parallel the real world. Imagine running a busy restaurant, where customers line up to make orders whilst the kitchen prepares the meals. In the software world, your users are the customers, and your backend services are the kitchen. With more people online than ever before, that line might start to grow out the front door. The ability to scale is no longer optional, it is essential. Know Your Options Vertical Scaling - Scale Up Horizontal Scaling - Scale Out Expanding your restaurant by adding more tables or a larger kitchen. In software terms, this means scaling up your infrastructure. More powerful CPUs, larger memory, increased throughput etc. This is a relative quick fix, but comes with diminishing returns and limits on how big everything can get. Opening new restaurant locations to serve more customers simultaneously and distribute existing flows. In software terms, this means adding more API servers, more worker nodes or creating many database replicas. This approach is more flexible and scalable than vertical scaling in the long term. A Steppingstone - Queues Just like how customers queue for their order, we create a pull-based task queue for our order management system using AWS Simple Queue Service (SQS). Tasks get queued into the SQS, and a consumer service will continuously poll this queue to process the tasks. This gives a lot of control for the queue consumer to dictate the frequency of polling, which works well in systems that cannot handle high throughput or requires non-concurrency like the SAP ERP (more on that later). SQS also provides built-in dead-letter-queues, retry policies, at least once delivery guarantee and scales automatically. Vertical scaling involves sizing up the compute power of the consumer (CPU, RAM etc). Horizontal scaling involves spinning up more consumers of the SQS. However, queues have limitations: Latency between order arrival and processing. Inefficient poll ## Replacing AWS Step Functions with SQS FIFO queues and cutting the cost in half DevFeed: [Replacing AWS Step Functions with SQS FIFO queues and cutting the cost in half]() Original publisher: [Read original article]() Author: Alexei Liulin Published: 2025-03-14T14:47:27Z Content type: article Language: en Sources: [Homegate Engineering Blog - Medium]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Step Functions](), [Amazon Web Services]() Tags: [aws](), [aws-step-functions](), [batching](), [cloud-computing](), [cost](), [cost-optimization](), [email](), [fifo](), [lambda](), [notifications](), [push-notifications](), [queue](), [sqs](), [sqs-queue](), [workflow]() ### AI overview The article describes SMG Real Estate's Search Alerts system, which batches property-match notifications for delivery by email or mobile push notification. It explains the original AWS Step Functions implementation and its transition-cost problem, motivating a more cost-efficient approach using SQS FIFO queues. ### Source excerpt One of the core services for SMG Real Estate is Search Alerts -- a service that allows users to be notified about new properties published on ImmoScout24 and Homegate. For example, if a user is looking for a new apartment having N rooms and costing less than X, and not finding any matches now, they can create a search alert with those search criteria. When a new property matching those criteria is published, the user will be notified either via email or a mobile push notification. About the Search Alerts system Users can choose the frequency of notifications -- either every 5 minutes or every 4 hours (assuming there are any to be delivered). The vast majority of search alerts are configured with the 5-minute frequency. Just to get an idea of the scale, we have: Millions of published properties match saved search alerts every day More than a million emails and push notifications sent daily Check out our "Homegate's fast and modern search experience helps users find their dream home" blog post for an overview of how Search Alerts work. The Original Implementation In the original implementation we used an AWS Step Function to achieve the 5- minute batching of notifications: When a new property matching a search alert arrived, the StartSendNotificationProcess lambda would start a Step Function execution specific to that search alert. All it did was waiting for 5 minutes while the matches accumulated in the matches-{searchAlertId} SQS queue, which was programmatically created for that specific search alert If more matching listings arrived during the 5-minute waiting period, the StartSendNotificationProcess lambda attempted to start the step function execution with the same name. When it failed with the ExecutionAlreadyExists error, we knew there was already a SF for waiting. That way the deduplication of notification processes was guaranteed. After 5 minutes of waiting time, the Step Function execution proceeded with triggering the LoadMatches lambda that received the acc ## Terraform module to manage Oxbow Lambda and its components DevFeed: [Terraform module to manage Oxbow Lambda and its components]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-03-14T00:00:00Z Content type: release Language: en Sources: [Scribd Tech]() Topics: [Terraform](), [AWS Lambda](), [infrastructure as code (IAC)](), [Amazon Web Services](), [AWS Glue](), [Amazon Simple Queue Service (SQS)](), [data-processing](), [event driven](), [DynamoDB](), [Amazon S3]() Tags: [aws](), [aws-glue](), [aws-lambda](), [aws-s3](), [deltalake](), [dynamodb](), [event-driven](), [iac](), [oxbow](), [rust](), [security](), [sqs](), [terraform]() ### AI overview The article introduces terraform-oxbow, an open-source Terraform module for deploying and managing an Oxbow AWS Lambda workflow and its supporting components. It describes configurable integrations including AWS Glue, Kinesis Data Firehose, SQS, DynamoDB, IAM policies, and S3 notifications, while noting AWS notification limits and least-privilege considerations. ### Source excerpt Oxbow is a project to take an existing storage location which contains Apache Parquet files into a Delta Lake table. It is intended to run both as an AWS Lambda or as a command line application. We are excited to introduce terraform-oxbow, an open-source Terraform module that simplifies the deployment and management of AWS Lambda and its supporting components. Whether you're working with AWS Glue, Kinesis Data Firehose, SQS, or DynamoDB, this module provides a streamlined approach to infrastructure as code (IaC) in AWS. ## Building Reliable Messaging Patterns in AWS with SQS and SNS DevFeed: [Building Reliable Messaging Patterns in AWS with SQS and SNS]() Original publisher: [Read original article]() Author: Guillermo Ojeda Published: 2024-12-20T15:20:38Z Content type: tutorial Language: en Sources: [Guille Ojeda]() Topics: [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Messaging](), [distributed-systems]() Tags: [architecture](), [aws](), [distributed-system](), [distributed-systems](), [error-handling](), [fifo](), [messaging](), [scale](), [sns](), [sqs](), [technical]() ### AI overview This tutorial explains how to build reliable distributed messaging systems with AWS SQS and SNS. It covers queue configuration, error handling, scaling patterns, message retention, long polling, visibility timeouts, and the differences between Standard and FIFO queues, using Node.js examples. ### Source excerpt Building distributed systems requires putting a lot of attention on communication between components. These components often need to exchange information asynchronously, and that's where message queues and pub/sub systems are the go-to solution. AWS ... ## Scaling Spin Apps With KEDA DevFeed: [Scaling Spin Apps With KEDA]() Original publisher: [Read original article]() Author: Thorsten Hans Published: 2024-12-19T12:00:00Z Content type: tutorial Language: en Sources: [Fermyon - Experience the next wave of cloud computing.]() Topics: [Kubernetes](), [autoscaling](), [event driven](), [Amazon Simple Queue Service (SQS)](), [Amazon Web Services]() Tags: [autoscaling](), [aws](), [event-driven](), [how-to](), [kubernetes](), [scale](), [sqs]() ### AI overview A tutorial on scaling Spin apps on Kubernetes with SpinKube and KEDA, including event-driven autoscaling with AWS SQS queues to optimize resource usage and application efficiency. ### Source excerpt Learn how to scale Spin apps on Kubernetes with SpinKube and KEDA. Explore event-driven autoscaling using AWS SQS queues to optimize resource usage and enhance application efficiency. ## Building a Scalable Job Consumer System in Go DevFeed: [Building a Scalable Job Consumer System in Go]() Original publisher: [Read original article]() Author: ansu jain Published: 2024-10-16T06:48:50Z Content type: tutorial Language: en Sources: [Mindorks - Medium]() Topics: [Go Language](), [Amazon Simple Queue Service (SQS)](), [Concurrency](), [Orchestration]() Tags: [consumer](), [go](), [goroutines](), [message-queue](), [orchestration](), [parallel](), [sqs](), [workers]() ### AI overview This tutorial presents a scalable job consumer system in Go. It describes interfaces for queue consumption, job management, job handling, and configuration, with parallel workers processing messages from queues such as AWS SQS. ### Source excerpt In this article, I will walk through the design of a scalable and extensible job consumer system using Go. This system allows the flexible registration of different jobs and the parallel consumption of messages from a message queue. I'll focus on how this system can be used to handle real-world scenarios like payment processing, order proccesing etc where different jobs consume messages from a queue (e.g., AWS SQS) and execute business logic based on the type of message received. Key Components of the Design IMessageConsumer - A generic interface for interacting with message queues (e.g., SQS). IJobManager - An interface that coordinates jobs and workers, responsible for consuming messages and distributing them to registered jobs. IJobHandler - An interface representing individual jobs. Each job implements specific business logic. IQueueConfig - Provides configurations for the queue (e.g., queue name, region, broker type). IJobConfig - Provides job-level configurations (e.g., number of consumers, timeouts). Core Concepts1. IMessageConsumer This interface abstracts queue operations, exposing methods for dequeuing and deleting messages. It's implemented by specific brokers (such as SQS in this case). Here's a basic structure: type IMessageConsumer interface { Dequeue(noOfMessages int64) ([]Message, error) Delete(messageIdentifier string) error } Dequeue: Fetches messages from the queue. Delete: Deletes a message after it has been successfully processed. 2. IJobManager This is where the system's core orchestration happens. The JobManager manages a pool of registered jobs and starts multiple workers to process messages concurrently. It uses a Consume method to begin processing and a RegisterJob method to register jobs into the system. type IJobManager interface { Consume(ctx context.Context) RegisterJob(job IJobHandler) } RegisterJob: Adds a job to the manager's job list. Consume: Starts the workers that process messages from the queue in parallel for each registered jo ## Comparing the Dependency Size of AWS Java SDK v1 and v2 SQS Clients DevFeed: [Comparing the Dependency Size of AWS Java SDK v1 and v2 SQS Clients]() Original publisher: [Read original article]() Author: Doug Tangren Published: 2024-01-19T16:40:54Z Content type: article Language: en Sources: [Making Meetup - Medium]() Topics: [Java](), [Amazon Web Services](), [SDKs](), [Amazon Simple Queue Service (SQS)](), [Netty](), [Gradle](), [Architecture & Design]() Tags: [aws](), [dependencies](), [dependency-management](), [gradle](), [java](), [maintenance](), [sdk](), [sdks](), [security-vulnerabilities](), [sqs]() ### AI overview This article from Meetup compares the dependency size of AWS SDK for Java v1 and v2 using SQS client versions 1.12.637 and 2.23.3. It attributes the larger v2 footprint largely to bundled defaults and transitive dependencies, including Netty, and discusses the maintenance and security costs of dependencies. ### Source excerpt Photo by Nathan Dumlao on Unsplash At Meetup, two of our core engineering principles are to be cost conscious and use technologies that are proven to scale. On Meetup every time you click an attend button, schedule an event, create a group, start a conversation, decide join the local puppy group in your neighborhood, or any other activity your request is guaranteed to pass through multiple JVMs and within those, likely half a dozen AWS services along the way which themselves are often sitting in front of multiple JVMs. Both the JVM and AWS APIs are considered proven, rock-solid, and scalable technologies at Meetup. This is why we're heavily invested in Java and AWS for our core platform services as well as keeping both up to date. Like many companies, AWS included, we've completed the spiritual journey of migrating our largest primary platform codebase from Java 8 to 11, then to 17, and most recently to 21. The renaissance happening with the Java community has been wonderful and has unlocked a number of options for us, one being the subject of this post. Being a 20 year strong engineering focused company, we've accumulated a lot learnings in the area of understanding the cost of code dependencies. We've learned it's much easier to add than to remove dependencies and that the simplest solution to avoid the future burden of maintenance tax and security vulnerabilities attached to dependencies is simply not to invite them to the party in the first place so we're relatively conservative when evaluating new dependencies. We understand the long term tax and cost involved in doing so. When evaluating the new v2 AWS SDKs for Java, the first surprise we encountered was that it was nearing twice the size of that of the equivalent v1 SDK clients. While we started our V2 migration journey relatively long ago we recently revisited and this is still more or less the same case. We'll compare the AWS SDK SQS client versionsv1.12.637vs v2.23.3 respectively in this post. Here is an e ## Amazon SNS: Cost Reduction and Reliable Delivery for Startups DevFeed: [Amazon SNS: Cost Reduction and Reliable Delivery for Startups]() Original publisher: [Read original article]() Author: ratneshmaurya2311@gmail.com (Ratnesh Maurya) Published: 2023-12-10T00:00:00Z Content type: article Language: en Sources: [Ratn Labs]() Topics: [Messaging](), [Amazon Simple Queue Service (SQS)](), [amazon]() Tags: [amazon](), [automatic](), [aws](), [backend](), [cloud-devops](), [cost](), [delivery](), [messaging](), [queues](), [sqs](), [system-design](), [system-design-backend-cloud-devops]() ### AI overview This article discusses how Amazon SNS can help startups reduce messaging costs through pay-per-message pricing and support message delivery with automatic retries, dead letter queues, and multi-region delivery. It also considers when Amazon SQS may be a better fit. ### Source excerpt How Amazon SNS helps startups cut messaging costs with pay-per-message pricing, automatic retries, dead letter queues, and multi-region delivery -- and when SQS might be a better fit. ## Using SQS to Throttle Writes to DynamoDB DevFeed: [Using SQS to Throttle Writes to DynamoDB]() Original publisher: [Read original article]() Author: Guillermo Ojeda Published: 2023-09-07T00:12:11Z Content type: tutorial Language: en Sources: [Guille Ojeda]() Topics: [Amazon SQS](), [Messaging](), [Database]() Tags: [amazon-sqs](), [aws](), [backend](), [cache](), [caching](), [cloud](), [databases](), [dynamodb](), [e-commerce](), [microservices](), [queue](), [queuing](), [sqs]() ### AI overview A technical article describes an e-commerce order-processing system whose DynamoDB provisioned capacity cannot keep up with sudden write traffic. It examines using Amazon SQS to queue writes and throttle database load, while noting that DynamoDB on-demand capacity is an alternative with higher stated cost. ### Source excerpt We're running an e-commerce platform, where people publish products and other people purchase those products. Our backend has some highly scalable microservices running on well-designed Lambdas, and there's a lot of caching involved. Our order proces... ## Vapor: Individual Queue Concurrency DevFeed: [Vapor: Individual Queue Concurrency]() Original publisher: [Read original article]() Author: Joe Dixon Published: 2023-01-30T14:36:00Z Content type: news Language: en Sources: [Laravel Blog]() Topics: [Laravel](), [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Concurrent Programming]() Tags: [announce](), [aws](), [concurrency](), [job](), [laravel](), [new-feature](), [queue](), [sqs]() ### AI overview Laravel Vapor now supports setting maximum concurrency for individual queues. The feature distributes a function's available concurrency across SQS queues and prevents a busy queue from consuming all resources or throttling other work. ### Source excerpt Today, we're excited to announce it's now possible to set the maximum concurrency of individual Vapor queues. This new feature will allow you to ensure that not one of your queues can consume all of the available resources. ## Micro Service Events (AWS EventBridge vs. SNS) DevFeed: [Micro Service Events (AWS EventBridge vs. SNS)]() Original publisher: [Read original article]() Author: Kristian Müller Published: 2022-09-16T12:06:51Z Content type: comparison Language: en Sources: [Homegate Engineering Blog - Medium]() Topics: [event driven](), [Messaging](), [Amazon Web Services](), [Amazon Simple Queue Service (SQS)](), [Publish-subscribe pattern](), [Architecture & Design](), [AWS Lambda](), [Amazon API Gateway](), [DynamoDB](), [Amazon S3](), [TypeScript]() Tags: [api](), [api-gateway](), [architecture](), [aws](), [aws-eventbridge](), [aws-sns](), [data](), [data-processing](), [database](), [dynamodb](), [event](), [event-driven](), [eventbridge](), [eventbus](), [events](), [gateway](), [lambda](), [messaging](), [microservice-architecture](), [s3](), [sns](), [sqs](), [typescript]() ### AI overview This article examines Homegate's event-driven microservice architecture on AWS and compares Amazon SNS with EventBridge as messaging infrastructure. It describes how services use SNS, SQS, Lambda, API Gateway, DynamoDB, and S3, and explains that EventBridge has a different focus, so replacing SNS is not necessarily the right question. ### Source excerpt During the last few years, we've been busy rebuilding the Homegate platform in an event driven, micro service architecture based on AWS. One of the most important buildings blocks, or shall we rather say the mortar that connects the building blocks, is the way our services send events to each other. While Amazon offers its notification service SNS since more than 12 years; in 2019 it started a newcomer in the messaging bus field by introducing EventBridge. This was very interesting to us as we're using SNS a lot. Bus on a Bridge (Photo by "Blue Elf" on Wikimedia Commons) As EventBridge was introduced and gained traction we asked ourselves what advantaged we would gain by switching our SNS based service glue to EventBridge. Maybe this turned out to be the wrong question for us. Let's look at our system to see why. Micro Services Connected by SNS Homegate is connecting dozens of services via SNS right now. Those services range from single purpose Lambda functions written in TypeScript to database heavy services using Aurora or DynamoDB and Step Functions sending events to other services. Some of the services are also interfacing external APIs such as Salesforce or SendGrid. Here our Lambda functions can call APIs when needed. On the other hand, we have web services to be called as APIs by other systems and end users. This is mostly achieved via API Gateway. A typical service connects to the system by subscribing to events needed to fulfill its purpose. Those inbound SNS events are then ingested via SQS allowing for orderly data processing and DLQs to handle errors. Outgoing results are pushed back to the system in form of events to a SNS topics. As Marko Savic has pointed out, we're also utilizing S3 to store and replay those events if needed. Basic AWS Lambda function connected via SNS If we would replace the SNS topic by streams on the EventBridge we would initially gain the same functionality by utilizing SQS as well. However, EventBridge has a different focus. Let ## AWS FIFO Queues with Message Groups for Atomic Processing at Scale DevFeed: [AWS FIFO Queues with Message Groups for Atomic Processing at Scale]() Original publisher: [Read original article]() Author: Nick De Cooman Published: 2022-03-03T09:55:58Z Content type: tutorial Language: en Sources: [Homegate Engineering Blog - Medium]() Topics: [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Serverless](), [event driven](), [Scalability](), [Publish-subscribe pattern]() Tags: [architecture](), [aws](), [decoupling](), [distributed](), [event](), [event-driven](), [events](), [fifo](), [serverless](), [sns](), [sqs]() ### AI overview A tutorial on using AWS FIFO queues and message groups to process events for multiple objects in parallel while ensuring that only one instance processes events for the same object at a time. It presents a serverless architecture using SNS, SQS, and Lambda for atomic listing ingestion. ### Source excerpt How to process multiple objects in parallel with the guarantee that for each unique object, only a single instance is processed simultaneously In a distributed, event-driven architecture, many challenges arise when processing large sets of data. One of them is the need for scalability when handling event streams with a substantial throughput. The challenge becomes even more complicated when processing these events involves atomic operations. In this case, we cannot simply process multiple events in parallel. As an example, let's say that we are building a service that ingests listings into our platform. The ingestion process involves several steps and requires multiple API operations. As a result, the whole ingestion can take up to several seconds per listing to complete. Now, here is where it gets tricky: a constraint of our service is that, while we are processing a listing, we cannot process other events for the same listing. The ingestion should be considered an atomic operation, and hence, only a single execution per listing can be performed simultaneously. At Homegate, we run most of our infrastructure on AWS in a serverless fashion. At the core of this, Lambda functions execute application logic. A common pattern is to use SNS topics to fan-out messages, and SQS for acting as a decoupling buffer between an SNS topic and some Lambda function. For our ingester service, a logical architecture could look like this: Listing events are published to a dedicated SNS topic. An SQS queue subscribes to this topic and consumes the events. Here, the events are buffered until a Lambda instance removes them from the queue and processes the corresponding listing. Under normal circumstances -- in which the processing would not involve an atomic operation -- this architecture would be a good fit. It would scale automatically relative to the number of Lambda instances that consume events from the queue, without much additional configuration. However, in the case where we cannot p ## Should we consider migrate to Amazon EventBridge from Amazon SNS + SQS ? DevFeed: [Should we consider migrate to Amazon EventBridge from Amazon SNS + SQS ?]() Original publisher: [Read original article]() Author: Guillaume Lannebere Published: 2022-02-01T14:45:47Z Content type: article Language: en Sources: [betclic-tech - Medium]() Topics: [Amazon EventBridge](), [Amazon Simple Queue Service (SQS)](), [event driven](), [Architecture & Design](), [Microservice](), [Amazon Web Services](), [Monitoring](), [Latency]() Tags: [amazon-eventbridge](), [architecture](), [aws](), [event-driven](), [eventbridge](), [latency](), [microservices](), [monitoring](), [net](), [sns](), [sqs]() ### AI overview The article examines Betclic's event-driven architecture requirements and its early-2019 decision to combine Amazon SNS with Amazon SQS. It presents the decision in the context of unpredictable traffic, multiple consumers, event replay, monitoring, schema management, language interoperability, analytics, autoscaling, low latency, high availability, AWS integration, and cost. ### Source excerpt Should we consider migrating to Amazon EventBridge from Amazon SNS + SQS? In early 2019, Betclic was in a corner, and we decided to migrate to microservices approach and migrate to the cloud. The first step of this change was the implementation of an Event Driven Architecture (EDA). Betclic presentation Before speaking of architecture and EDA, it's essential to know what Betclic is doing to understand our choice quickly. Betclic is a French online gambling company. Betclic operates in Sports betting, horse race betting, and online poker. The specificity of gambling is that we don't have predictive traffic. We have high traffic peaks during significant events and weekends. We can see an example of traffic on a soccer event: Example of trafficEDA wishlist When beginning 2019, we did a workshop to identify the key features of our ideal EDA; we would like something: allowing multiple reads: the same message can be read by various services with error recovery: in case of error, we would like to replay events easy to monitor: particularly with Datadog, our centralized monitoring solution with schema registry language-agnostic: we use 3 main languages .net, Kotlin, and Typescript (for Lambda), and the chosen solution must work with all languages with analytics capabilities: we would like to integrate all our events of EDA in our data lake fully managed with auto scaling: essential feature for us; we have unpredictable traffic with big spikes at the end of matches, so we need to have an EDA to be able to absorb our spikes with low latency and high availability: also an essential feature, we would like our EDA to serve critical use cases for our final users, so we want something the most real-time and the most available possible well-integrated with AWS services: EDA is one of our first bricks in the cloud, but we want something to help us to migrate to AWS easy to use: we want all the time something easy to use :) but here it's particularly true because EDA is a centralized ## Guest Post: Developing a Serverless Scheduler Using DynamoDB TTL and Filtered Streams DevFeed: [Guest Post: Developing a Serverless Scheduler Using DynamoDB TTL and Filtered Streams]() Original publisher: [Read original article]() Author: Frédéric Barthelet Published: 2021-12-21T00:00:00Z Content type: tutorial Language: en Sources: [Serverless Blog]() Topics: [Serverless](), [Amazon Web Services](), [serverless framework](), [Streams](), [Amazon Simple Queue Service (SQS)]() Tags: [aws](), [aws-lambda](), [cloud-computing](), [deep-dive](), [engineering](), [faas](), [function-as-a-service](), [guest-post](), [infrastructure-as-code](), [new-feature](), [serverless](), [serverless-architecture](), [serverless-framework](), [sqs](), [streams]() ### AI overview This guest post explains AWS Lambda event filtering and describes how Serverless Framework supports filtering for stream and SQS event types through the filterPatterns option. It presents a serverless scheduler example involving DynamoDB TTL and filtered streams. ### Source excerpt AWS announced Lambda event filtering during re:Invent 2021. This new feature helps reduce the quantity of Lambda's invocation to exactly match your requirements and possibly save on costs. We'll deep dive into a specific example in this article. We're happy to announce that Serverless Framework now supports event filtering for both `stream` and `sqs` event types via the new `filterPatterns` option ## AWS re:Invent 2021 Updates for Serverless Framework, Dashboard, and Cloud DevFeed: [AWS re:Invent 2021 Updates for Serverless Framework, Dashboard, and Cloud]() Original publisher: [Read original article]() Author: Steven Yi Published: 2021-12-02T00:00:00Z Content type: news Language: en Sources: [Serverless Blog]() Topics: [Amazon Web Services](), [serverless framework](), [dashboards](), [Amazon Simple Queue Service (SQS)](), [AWS Lambda](), [Cloud]() Tags: [aws](), [aws-lambda](), [cloud](), [cloud-computing](), [error-handling](), [faas](), [function-as-a-service](), [infrastructure-as-code](), [news](), [serverless](), [serverless-architecture](), [serverless-framework](), [sqs](), [updates]() ### AI overview The article covers AWS re:Invent 2021 updates from the Serverless team, including how Serverless Framework, Dashboard, and Cloud use AWS enhancements such as improved SQS batch error handling with AWS Lambda. ### Source excerpt A lot of news came out of AWS re:Invent 2021 - get updates from our team on how Serverless Framework, Dashboard, and Cloud are taking advantage of the latest AWS enhancements (like improved SQS batch error handling with AWS Lambda). ## Improved SQS batch error handling with AWS Lambda DevFeed: [Improved SQS batch error handling with AWS Lambda]() Original publisher: [Read original article]() Author: Matthieu Napoli Published: 2021-11-30T00:00:00Z Content type: tutorial Language: en Sources: [Serverless Blog]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Lambda](), [serverless framework]() Tags: [aws](), [aws-lambda](), [cloud-computing](), [engineering](), [error-handling](), [faas](), [function-as-a-service](), [how-to](), [infrastructure-as-code](), [serverless](), [serverless-architecture](), [serverless-framework](), [sqs]() ### AI overview This tutorial explains how AWS Lambda's partial batch response feature improves error handling for Amazon SQS batch processing and shows how Serverless Framework supports it with a complete example. ### Source excerpt With the new "partial batch response", handling errors in SQS batch processing with Lambda can be much more robust. Serverless Framework supports that new feature today. Learn how to use that new feature with a complete example ## AWS OIDC Authentication with SPIFFE DevFeed: [AWS OIDC Authentication with SPIFFE]() Original publisher: [Read original article]() Author: Roy Xu Published: 2021-08-26T19:00:00Z Content type: tutorial Language: en Sources: [Square Corner Blog](), [Square Corner Blog RSS Feed]() Topics: [Amazon Web Services](), [OpenID connect (OIDC)](), [SPIFFE](), [SPIRE](), [Security](), [Secrets Management](), [Authentication](), [datacenter](), [Amazon S3](), [Amazon Simple Queue Service (SQS)](), [migration]() Tags: [app](), [authorization](), [automated](), [aws](), [cloud](), [datacenter](), [engineering](), [kubernetes](), [oidc](), [s3](), [secrets-management](), [security](), [spiffe](), [spire](), [sqs]() ### AI overview Square describes using SPIFFE and SPIRE's OIDC Discovery Provider to exchange SPIFFE JWTs for AWS credentials, allowing datacenter applications to assume AWS roles through an automated configuration-based process. ### Source excerpt Easy authentication with automated AWS credentials ## Real-Time Machine Learning at Headspace DevFeed: [Real-Time Machine Learning at Headspace]() Original publisher: [Read original article]() Author: Headspace Published: 2021-08-16T16:53:19Z Content type: article Language: en Sources: [Stories by Headspace on Medium]() Topics: [Machine Learning & Artificial Intelligence](), [real-time](), [Inference](), [recommendations](), [Apache Spark](), [databricks](), [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Amazon SageMaker AI](), [Streaming](), [Orchestration](), [Model Development]() Tags: [android](), [apache-spark](), [aws](), [data](), [databricks](), [headspace-engineering](), [inference](), [machine-learning](), [orchestration](), [real-time](), [recommendations](), [sqs](), [streaming](), [web]() ### AI overview This article describes Headspace's infrastructure for real-time machine learning inference and personalized content recommendations. It explains how the team uses modular Publishing, Receiver, Orchestration, and Serving layers with Apache Spark Structured Streaming on Databricks, AWS SQS, Lambda, and SageMaker. ### Source excerpt Author: Yu Chen, Senior Software Engineer / Co-author: Koyuki Nakamori, Senior Engineering Manager Introduction Data is often most valuable when it can be immediately leveraged to make decisions in the moment, but traditionally consumer data is ingested, transformed, persisted, and sits dormant for lengthy periods of time before being used by machine learning and analytics teams. Headspace's core products are iOS, Android, and web-based apps that focus on improving the health and happiness of its users through mindfulness, meditation, sleep, exercise, and focus content. Machine learning models are core to our user experiences by offering recommendations that engage our users with new relevant, personalized content that builds consistent habits in their lifelong journey. Finding a way to leverage user data to generate real-time insights and decisions means that consumer-facing products like the Headspace app can dramatically shorten the end-to-end user feedback loop: actions that users perform a few seconds, or minutes ago can be incorporated in generating a more relevant, personalized, and context-specific content recommendation for the user. This means our ML models could incorporate dynamic features that update throughout the course of a user's day, or even an individual session: Current session bounce rates for sleep content Semantic embeddings for recent user search terms (if a user recently searched for "preparing for big exam", the ML model can assign more weight to Focus-themed meditations) Users' biometric data (i.e., if step counts and heart rate are increasing over the last 10 minutes, we can recommend Move or Exercise content) The Headspace Machine Learning team architected a solution by decomposing the infrastructure systems into modular Publishing, Receiver, Orchestration, and Serving layers, leveraging Apache Spark Structured Streaming on Databricks, AWS SQS, Lambda, and Sagemaker to deliver real-time inference capabilities for our machine learning mod ## Remediating AWS IMDSv1 DevFeed: [Remediating AWS IMDSv1]() Original publisher: [Read original article]() Published: 2021-08-11T16:16:23Z Content type: tutorial Language: en Sources: [Latacora]() Topics: [Amazon Web Services](), [Amazon EC2](), [AWS IAM](), [Exploit](), [vulnerability](), [Amazon Simple Queue Service (SQS)](), [DynamoDB]() Tags: [aws](), [bugs](), [credentials](), [defense-in-depth](), [dynamodb](), [ec2](), [exploit](), [iam](), [infrastructure](), [sqs](), [technical](), [vulnerability]() ### AI overview This technical article explains why AWS Instance Metadata Service version 1 is risky. Applications vulnerable to SSRF, XXE, or RCE may allow attackers to obtain temporary IAM role credentials from EC2-hosted applications and use them to access AWS services. The article presents defense in depth as a way to limit the impact of application vulnerabilities, although the supplied text ends before the remediation details. ### Source excerpt 2024-12-17 Updated to include Declarative Policies Compute resources in AWS (for example, EC2 instances, ECS tasks/services, etc.) get access to AWS credentials, such as temporary instance role credentials, via the Instance Metadata Service (IMDS). The compute resources use these credentials to access other AWS services such as SQS, DynamoDB and Secrets Manager. Introduction: Problems with IMDSv1 # There was originally only one version of IMDS, now called "v1," which unfortunately many people still use. The technical risks and high profile incidents (the Capital One breach comes to mind) associated with v1, as well as the existence of v2 are well-documented. When an application hosted on an EC2 instance is vulnerable to SSRF, XXE or RCE, attackers can likely steal the temporary AWS credentials of the IAM role configured for the instance. This service is a particularly interesting target for attackers: [Next page]()