# Amazon Simple Queue Service (SQS) Fully managed message queuing service for microservices, distributed systems, and serverless applications. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Symfony 8.1.7 released DevFeed: [Symfony 8.1.7 released]() Original publisher: [Read original article]() Author: Fabien Potencier Published: 2026-09-15T07:15:39Z Content type: release Language: en Sources: [Symfony Blog]() Topics: [Symfony](), [bug](), [Amazon Simple Queue Service (SQS)]() Tags: [amazon-sqs](), [bug](), [release](), [symfony]() ### AI overview Symfony 8.1.7 has been released. The release includes security, translation, Messenger, HTTP, form, console, and other fixes and minor changes, including a fix for keepalive handling on the Amazon SQS transport. ### Source excerpt Symfony 8.1.7 has just been released. Read the Symfony upgrade guide to learn more about upgrading Symfony and use the SymfonyInsight upgrade reports to detect the code you will need to change in your project. Tip... ## Testing application resilience with Amazon SQS and AWS Fault Injection Service DevFeed: [Testing application resilience with Amazon SQS and AWS Fault Injection Service]() Original publisher: [Read original article]() Author: Richard Whitworth Published: 2026-09-09T21:33:59Z Content type: tutorial Language: en Sources: [AWS Architecture Blog]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Fault Injection Service (FIS)](), [Chaos Engineering](), [AWS IAM](), [AWS Identity and Access Management (IAM)]() Tags: [advanced-300](), [amazon-cloudwatch](), [amazon-simple-queue-service-sqs](), [amazon-sqs](), [automation](), [aws](), [aws-fault-injection-service-fis](), [experiments](), [how-to](), [iam](), [observability](), [resilience](), [technical-how-to](), [testing]() ### AI overview A tutorial on testing application resilience when Amazon SQS data-plane operations fail. It uses AWS Fault Injection Service and Systems Manager Automation to progressively deny queue access, evaluate recovery and observability with CloudWatch metrics, and avoid IAM deny-policy lockouts. ### Source excerpt Learn how to use AWS Fault Injection Service and AWS Systems Manager Automation to run progressive chaos experiments against Amazon SQS queues. Validate that your retry logic, circuit breakers, and dead-letter queues actually work under failure before a real outage hits production. ## Five Years of Kafka at Razorpay's UPI Switch DevFeed: [Five Years of Kafka at Razorpay's UPI Switch]() Original publisher: [Read original article]() Author: Kshitij Nawandar Published: 2026-09-07T09:09:58Z Content type: article Language: en Sources: [Razorpay Engineering - Medium]() Topics: [Kafka](), [Architecture & Design](), [event driven](), [Amazon Simple Queue Service (SQS)](), [distributed-systems](), [Microservices](), [Amazon Web Services]() Tags: [aws](), [aws-sns](), [distributed-systems](), [event-driven](), [kafka](), [microservices](), [razorpay](), [redshift](), [sns](), [sqs]() ### AI overview Razorpay describes five years of evolution in its UPI Switch, from a monolith using AWS SQS to Kafka-based infrastructure. The article covers architectural decisions, operational challenges, and optimization work affecting payment-processing performance, reliability, and scale. ### Source excerpt Preface The UPI Switch at Razorpay has evolved significantly in the five years since we started building it. The Switch is the platform that enables real-time payment processing with NPCI. When the team began, it was little more than an idea. Today it powers more than 70% of Razorpay's total UPI volume. Because UPI is inherently asynchronous, a messaging system sits at the heart of the Switch and has a direct impact on performance, reliability, and scale. What began as a straightforward queue became the core of the system, shaping how every new feature was designed and delivered. This post covers that evolution: the decisions that enabled growth, the ones that slowed us down, the operational issues that forced us to rethink our assumptions, and the optimizations that ultimately stabilized our Kafka-based infrastructure. This is the story of what we got right, what we got wrong, and how we eventually built something stable enough to grow on. The First Version: Monolith and SQS When we began building the UPI Switch, we weren't thinking about massive scale, distributed systems, or elegant event routing. So we built Switch v1 as a monolith. No microservices, no distributed orchestration: just one solid block of code doing everything. That was the right call. We needed to move fast, experiment, and learn, and we followed the Keep It Simple, Stupid (KISS) principle deliberately. For messaging, we picked AWS SQS: reliable, managed, and low on cognitive load. We didn't need ordering guarantees at the time, so a standard queue worked fine. We started with just two queues, and this setup held its ground. It handled a peak of 400 TPS during the IPL. The limitations showed up as the ecosystem grew. A single event, like a successful payment, needed to fan out into multiple workflows: Update NPCI with an API call Send callbacks to merchants about payment status Push structured data into our warehouse (AWS Redshift) To handle this, we started bolting on AWS SNS plus SQS for fan-ou ## Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode DevFeed: [Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode]() Original publisher: [Read original article]() Author: Ben Freiberg Published: 2026-08-10T13:42:03Z Content type: article Language: en Sources: [AWS Compute Blog]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Lambda](), [Latency](), [Low Latency](), [Amazon Web Services](), [Concurrency](), [Kafka](), [Internet of things](), [telemetry]() Tags: [amazon-simple-queue-service-sqs](), [amazon-sqs](), [apache-kafka](), [aws](), [aws-lambda](), [concurrency](), [event-driven](), [flash](), [fraud](), [kafka](), [latency](), [low-latency](), [payment-processing](), [real-time](), [telemetry]() ### AI overview AWS explains how provisioned mode for Amazon SQS event source mappings with AWS Lambda provides explicit control over event pollers to support low-latency, high-throughput event processing. The article describes default and provisioned scaling, concurrency limits, throughput, and use cases including real-time payments, fraud detection, IoT telemetry, and flash-sale order fulfillment. ### Source excerpt Customers building event-driven applications on AWS rely on Amazon Simple Queue Service (Amazon SQS) and AWS Lambda event source mappings (ESMs) to process millions of events every day. The fully managed polling infrastructure of ESMs eliminates the need to write and maintain custom code. You can focus on business logic while Lambda handles scaling, batching, [...] ## Your SQS consumer can hang forever by default DevFeed: [Your SQS consumer can hang forever by default]() Original publisher: [Read original article]() Author: Ivan Cernja Published: 2026-07-28T00:00:00Z Content type: article Language: en Sources: [Encore Updates]() Topics: [Amazon Simple Queue Service (SQS)](), [SDK](), [Rust](), [Amazon Web Services]() Tags: [aws](), [health-check](), [queue](), [retry](), [rust](), [sdk](), [sqs](), [tcp](), [timeout]() ### AI overview The article explains that the AWS Rust SDK has no request timeout by default. A failed or silently dropped connection can leave an SQS receive waiting indefinitely, stopping a sequential consumer loop while the process and health check still appear healthy. It describes fixing the issue with a per-attempt timeout. ### Source excerpt The AWS Rust SDK ships no request timeout by default, so one SQS receive on a dead connection can hang a whole consumer with nothing in the logs. ## Announcing the AWS Exporter | Port DevFeed: [Announcing the AWS Exporter | Port]() Original publisher: [Read original article]() Author: Tal Sabag Published: 2026-07-22T12:04:28Z Content type: article Language: en Sources: [Developer Experience & Platform Engineering Blog | Port]() Topics: [internal developer portal](), [Amazon Web Services](), [Amazon S3](), [Amazon Simple Queue Service (SQS)](), [AWS IAM]() Tags: [aws](), [catalog](), [cloud](), [configuration](), [developer-portal](), [encryption](), [iam](), [internal-developer-portal](), [queue](), [resource](), [s3](), [serverless](), [software-catalog]() ### AI overview Port's AWS exporter imports live AWS resource data into a software catalog and internal developer portal. It supports more than 500 resource types, maps them to blueprints and relations, and exposes metadata, tags, console links, and resource-specific details such as encryption and access configuration. ### Source excerpt The AWS exporter brings us even closer to realizing our vision of getting everything into the software catalog and the internal developer portal. ## Incident Response as Code: Managing PagerDuty with Pulumi DevFeed: [Incident Response as Code: Managing PagerDuty with Pulumi]() Original publisher: [Read original article]() Author: Engin Diri Published: 2026-07-20T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [pulumi](), [Incident response](), [Amazon Web Services](), [TypeScript](), [Amazon Simple Queue Service (SQS)]() Tags: [aws](), [devops](), [incident](), [incident-response](), [infrastructure-as-code](), [pagerduty](), [pulumi](), [sqs](), [tutorials](), [typescript]() ### AI overview A tutorial for managing PagerDuty incident-response resources and AWS alarm wiring together in a Pulumi program. The demo uses TypeScript to define an on-call team, escalation policy, PagerDuty service, SQS dead-letter queue, CloudWatch alarm, and SNS forwarding. ### Source excerpt You usually find out in the postmortem: the alarm fired, but it paged a schedule nobody was on anymore. Or the service had been running in production for three months before anyone created the matching PagerDuty service, so the first person to notice the outage was a customer. The infrastructure was code, reviewed and versioned. The incident response setup was forty clicks in a web UI, done once, by someone who has since changed teams. PagerDuty's own engineering team has been making the case for managing PagerDuty as code for years, and the OneUptime folks recently published a hands-on guide to the PagerDuty Terraform provider that walks the whole resource catalog. I liked both posts, and you probably know where this is going: I want the same thing in TypeScript, with the on-call setup and the infrastructure it protects living in the same Pulumi program. Teams, schedules, escalation policies, services, and the CloudWatch alarm that pages you: one pulumi up. What you are building The demo models a small platform team that owns a checkout service on AWS. On the PagerDuty side: a team with three members, a weekly on-call rotation, an escalation policy with two levels, and a service with a CloudWatch integration. On the AWS side: a dead-letter queue, an alarm that fires when a message lands in it, and an SNS topic that forwards the alarm to PagerDuty. The interesting part is where the two meet: the integration key PagerDuty mints flows straight into the SNS subscription without ever touching a clipboard. flowchart LR DLQ["SQS dead-letter queue"] --> Alarm["CloudWatch alarm"] Alarm --> SNS["SNS topic"] SNS -->|"HTTPS subscription with integration key"| Int["CloudWatch integration"] Int --> Svc["PagerDuty service Checkout API"] Svc --> EP["Escalation policy"] EP -->|"level 1"| Sched["Weekly rotation Lena, Jonas, Mia"] EP -->|"level 2, after 15 min"| Lead["Team lead"] Everything on both sides of that diagram is one TypeScript program. If you would rather write ## Amazon SQS turns 20: Two decades of reliable messaging at scale DevFeed: [Amazon SQS turns 20: Two decades of reliable messaging at scale]() Original publisher: [Read original article]() Author: Esra Kayabali Published: 2026-07-13T18:13:57Z Content type: article Language: en Sources: [AWS News Blog]() Topics: [Amazon Simple Queue Service (SQS)](), [distributed-systems](), [Amazon Web Services](), [Encryption](), [Security](), [Availability]() Tags: [amazon](), [amazon-simple-queue-service-sqs](), [amazon-sqs](), [announcements](), [availability](), [distributed-systems](), [encryption](), [launch](), [messaging](), [news](), [performance](), [scale](), [security](), [systems](), [transactions]() ### AI overview Amazon SQS marks its 20th anniversary with a retrospective on reliable, asynchronous messaging and the decoupling of producers from consumers. The article highlights recent developments including higher-throughput FIFO queues, default server-side encryption, and dead-letter queue redrive enhancements. ### Source excerpt On July 13, 2006, we launched Amazon Simple Queue Service (Amazon SQS) as one of the first three services available to customers, alongside Amazon EC2 and Amazon S3. We had learned firsthand that distributed systems need a reliable way to pass messages between components without creating tight dependencies. If one service called another directly and [...] ## Kafka vs RabbitMQ vs SQS DevFeed: [Kafka vs RabbitMQ vs SQS]() Original publisher: [Read original article]() Author: Ashish Pratap Singh Published: 2026-07-02T03:31:11Z Content type: comparison Language: en Sources: [AlgoMaster Newsletter]() Topics: [Amazon Simple Queue Service (SQS)](), [Kafka](), [RabbitMQ](), [Messaging](), [Back end]() Tags: [backend](), [kafka](), [messaging](), [queues](), [rabbitmq](), [sqs]() ### AI overview This comparison examines Kafka, RabbitMQ, and Amazon SQS for asynchronous systems. It explains that Kafka uses an append-only log with consumer offsets and replay, RabbitMQ routes messages through exchanges and queues with acknowledgments, and SQS provides managed queue-based messaging that consumers poll. ### Source excerpt Most backend systems eventually need one service to hand off work to another asynchronously. ## Migrating a Hybrid Architecture to Laravel Cloud: Top Q&A Questions Answered DevFeed: [Migrating a Hybrid Architecture to Laravel Cloud: Top Q&A Questions Answered]() Original publisher: [Read original article]() Author: Ana Tavares Published: 2026-05-21T14:13:10Z Content type: article Language: en Sources: [Laravel Blog]() Topics: [Laravel](), [migration](), [Cloud](), [Vapor](), [Cloudflare Workers](), [API](), [Routing (disambiguation)](), [Amazon Simple Queue Service (SQS)](), [data](), [Database](), [MongoDB](), [networking]() Tags: [api](), [cloud](), [cloudflare-workers](), [database](), [developer](), [laravel](), [migration](), [networking](), [routing](), [s3]() ### AI overview Trybe migrated a high-volume hybrid booking platform from Laravel Vapor to Laravel Cloud without dropping requests. The article covers its isolated multi-application architecture, routing through Cloudflare Workers, operational scale, and migration-related technical questions. ### Source excerpt Trybe shared how they migrated from Laravel Vapor to Laravel Cloud in a live webinar. Here are the top developer questions from the Q&A, answered. ## Unordered mode for S3 ClickPipes is now available DevFeed: [Unordered mode for S3 ClickPipes is now available]() Original publisher: [Read original article]() Author: Marta Paes Published: 2025-12-09T15:06:25Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [Amazon S3](), [clickhouse](), [Amazon Simple Queue Service (SQS)](), [event driven](), [data]() Tags: [amazon-s3](), [amazon-sqs](), [clickhouse](), [data](), [event-driven](), [object-storage](), [queue](), [s3]() ### AI overview ClickPipes now supports unordered ingestion from Amazon S3 into ClickHouse Cloud using S3 Event Notifications through Amazon SQS. This handles backfills, retries, late-arriving data, and other out-of-order files without relying on lexicographical file order. ### Source excerpt Ingest data from Amazon S3 into ClickHouse Cloud in any order for event-driven, blazing-fast analytics. ## HubSpot Incident Report for October 20, 2025 DevFeed: [HubSpot Incident Report for October 20, 2025]() Original publisher: [Read original article]() Author: Kartik Vishwanath Published: 2025-11-11T17:09:36Z Content type: news Language: en Sources: [HubSpot]() Topics: [incident](), [Amazon Web Services](), [Cloud](), [Resilience](), [Amazon Simple Queue Service (SQS)](), [Amazon EC2](), [AWS IAM](), [DynamoDB]() Tags: [amazon](), [amazon-sqs](), [analysis](), [aws](), [cloud](), [data](), [database](), [distributed](), [dynamodb](), [ec2](), [engineering](), [failover](), [hubspot](), [iam](), [improvements](), [incident](), [message-queue](), [outage](), [recovery](), [report](), [resilience]() ### AI overview HubSpot reports that a severe AWS outage in the us-east-1 region on October 20, 2025 disrupted multiple product features and affected third-party vendors. The incident caused failures involving DynamoDB, IAM, SQS, and EC2, and degraded HubSpot's TQ2 background task processing. HubSpot says it completed an analysis and is implementing resilience improvements. ### Source excerpt On October 20, 2025, HubSpot experienced a significant service disruption affecting multiple product features due to a severe AWS outage in the us-east-1 region. While our infrastructure remained intact, the widespread nature of the cloud provider failure impacted both our services and critical third-party vendors we rely on. We've completed a thorough analysis of this incident and are implementing comprehensive improvements to strengthen our resilience against future cloud provider disruptions. ## Building event-driven pipelines with SQS and S3 DevFeed: [Building event-driven pipelines with SQS and S3]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2025-08-21T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Amazon S3](), [Amazon Simple Queue Service (SQS)](), [event driven](), [real-time](), [Serverless](), [data]() Tags: [amazon-s3](), [amazon-sqs](), [amazon-sqs-s3-integration](), [automate-data-ingestion-s3](), [building-data-pipelines-with-sqs-and-s3](), [building-event-driven-pipelines-with-sqs-and-s3](), [data-pipeline-automation-with-sqs](), [event-driven](), [event-driven-data-processing](), [event-driven-pipelines-sqs-s3](), [how-to](), [object-storage](), [real-time](), [real-time-data-pipelines-s3](), [s3-to-redpanda-streaming](), [serverless](), [set-up-event-driven-architecture-aws](), [sqs-s3-data-pipeline-tutorial](), [sqs-s3-notifications-for-data-pipelines](), [streaming-data-to-redpanda-from-s3](), [tutorial]() ### AI overview This tutorial explains how to use S3 Event Notifications and Amazon SQS to trigger Redpanda Connect processing whenever new objects arrive in an S3 bucket. It demonstrates an automated, continuously running pipeline that ingests object-storage data into real-time streams. ### Source excerpt Learn how to build real-time, event-driven data pipelines directly from your object storage with Amazon SQS and S3. ## Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools DevFeed: [Cloud-native data ingestion architecture using AWS, Databricks, and open-source tools]() Original publisher: [Read original article]() Author: R Tyler Croy Published: 2025-08-01T00:00:00Z Content type: tutorial Language: en Sources: [Scribd Tech]() Topics: [data](), [Amazon Web Services](), [databricks](), [event driven](), [Amazon Simple Queue Service (SQS)](), [Kafka](), [Open Source](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [architecture](), [aws](), [azure](), [catalog](), [cloud](), [cloud-native](), [data](), [databricks](), [deltalake](), [event-driven](), [featured](), [google-cloud](), [kafka](), [open-source](), [rust](), [sqs]() ### AI overview This article presents Scribd's cloud-native data-ingestion architecture for building large datasets for Delta Lake. It describes using AWS services and open-source tools such as kafka-delta-ingest, oxbow, and Airbyte in a more event-driven and reliable platform, with Databricks and Unity Catalog. The approach can also be adapted to Azure, Google Cloud Platform, or on-premises environments. ### Source excerpt Delta Lake is a fantastic technology for quickly querying massive data sets, but first you need those massive data sets! In this talk from Data and AI Summit 2025 I dive into the cloud-native architecture Scribd has adopted to ingest data from AWS Aurora, SQS, Kinesis Data Firehose and more! ## Order Dispatch Systems at Scale DevFeed: [Order Dispatch Systems at Scale]() Original publisher: [Read original article]() Author: Guest User Published: 2025-07-28T01:53:56Z Content type: tutorial Language: en Sources: [Kogan.com]() Topics: [Amazon Simple Queue Service (SQS)](), [event driven](), [Scalability](), [Amazon Web Services](), [Microservices](), [microservices architecture](), [Latency]() Tags: [aws](), [concurrency](), [event](), [event-driven](), [latency](), [microservices-architecture](), [scale](), [sqs]() ### AI overview This tutorial compares vertical and horizontal scaling, pull-based queues using AWS Simple Queue Service, and event-driven architecture for order dispatch systems. It describes queue trade-offs and explains how event subscribers can process orders independently and scale separately. ### Source excerpt How to load-balance like a seasoned waiter Software systems often parallel the real world. Imagine running a busy restaurant, where customers line up to make orders whilst the kitchen prepares the meals. In the software world, your users are the customers, and your backend services are the kitchen. With more people online than ever before, that line might start to grow out the front door. The ability to scale is no longer optional, it is essential. Know Your Options Vertical Scaling - Scale Up Horizontal Scaling - Scale Out Expanding your restaurant by adding more tables or a larger kitchen. In software terms, this means scaling up your infrastructure. More powerful CPUs, larger memory, increased throughput etc. This is a relative quick fix, but comes with diminishing returns and limits on how big everything can get. Opening new restaurant locations to serve more customers simultaneously and distribute existing flows. In software terms, this means adding more API servers, more worker nodes or creating many database replicas. This approach is more flexible and scalable than vertical scaling in the long term. A Steppingstone - Queues Just like how customers queue for their order, we create a pull-based task queue for our order management system using AWS Simple Queue Service (SQS). Tasks get queued into the SQS, and a consumer service will continuously poll this queue to process the tasks. This gives a lot of control for the queue consumer to dictate the frequency of polling, which works well in systems that cannot handle high throughput or requires non-concurrency like the SAP ERP (more on that later). SQS also provides built-in dead-letter-queues, retry policies, at least once delivery guarantee and scales automatically. Vertical scaling involves sizing up the compute power of the consumer (CPU, RAM etc). Horizontal scaling involves spinning up more consumers of the SQS. However, queues have limitations: Latency between order arrival and processing. Inefficient poll ## Replacing AWS Step Functions with SQS FIFO queues and cutting the cost in half DevFeed: [Replacing AWS Step Functions with SQS FIFO queues and cutting the cost in half]() Original publisher: [Read original article]() Author: Alexei Liulin Published: 2025-03-14T14:47:27Z Content type: article Language: en Sources: [Homegate Engineering Blog - Medium]() Topics: [Amazon Simple Queue Service (SQS)](), [AWS Step Functions](), [Amazon Web Services]() Tags: [aws](), [aws-step-functions](), [batching](), [cloud-computing](), [cost](), [cost-optimization](), [email](), [fifo](), [lambda](), [notifications](), [push-notifications](), [queue](), [sqs](), [sqs-queue](), [workflow]() ### AI overview The article describes SMG Real Estate's Search Alerts system, which batches property-match notifications for delivery by email or mobile push notification. It explains the original AWS Step Functions implementation and its transition-cost problem, motivating a more cost-efficient approach using SQS FIFO queues. ### Source excerpt One of the core services for SMG Real Estate is Search Alerts -- a service that allows users to be notified about new properties published on ImmoScout24 and Homegate. For example, if a user is looking for a new apartment having N rooms and costing less than X, and not finding any matches now, they can create a search alert with those search criteria. When a new property matching those criteria is published, the user will be notified either via email or a mobile push notification. About the Search Alerts system Users can choose the frequency of notifications -- either every 5 minutes or every 4 hours (assuming there are any to be delivered). The vast majority of search alerts are configured with the 5-minute frequency. Just to get an idea of the scale, we have: Millions of published properties match saved search alerts every day More than a million emails and push notifications sent daily Check out our "Homegate's fast and modern search experience helps users find their dream home" blog post for an overview of how Search Alerts work. The Original Implementation In the original implementation we used an AWS Step Function to achieve the 5- minute batching of notifications: When a new property matching a search alert arrived, the StartSendNotificationProcess lambda would start a Step Function execution specific to that search alert. All it did was waiting for 5 minutes while the matches accumulated in the matches-{searchAlertId} SQS queue, which was programmatically created for that specific search alert If more matching listings arrived during the 5-minute waiting period, the StartSendNotificationProcess lambda attempted to start the step function execution with the same name. When it failed with the ExecutionAlreadyExists error, we knew there was already a SF for waiting. That way the deduplication of notification processes was guaranteed. After 5 minutes of waiting time, the Step Function execution proceeded with triggering the LoadMatches lambda that received the acc ## Terraform module to manage Oxbow Lambda and its components DevFeed: [Terraform module to manage Oxbow Lambda and its components]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-03-14T00:00:00Z Content type: release Language: en Sources: [Scribd Tech]() Topics: [Terraform](), [AWS Lambda](), [infrastructure as code (IAC)](), [Amazon Web Services](), [AWS Glue](), [Amazon Simple Queue Service (SQS)](), [data-processing](), [event driven](), [DynamoDB](), [Amazon S3]() Tags: [aws](), [aws-glue](), [aws-lambda](), [aws-s3](), [deltalake](), [dynamodb](), [event-driven](), [iac](), [oxbow](), [rust](), [security](), [sqs](), [terraform]() ### AI overview The article introduces terraform-oxbow, an open-source Terraform module for deploying and managing an Oxbow AWS Lambda workflow and its supporting components. It describes configurable integrations including AWS Glue, Kinesis Data Firehose, SQS, DynamoDB, IAM policies, and S3 notifications, while noting AWS notification limits and least-privilege considerations. ### Source excerpt Oxbow is a project to take an existing storage location which contains Apache Parquet files into a Delta Lake table. It is intended to run both as an AWS Lambda or as a command line application. We are excited to introduce terraform-oxbow, an open-source Terraform module that simplifies the deployment and management of AWS Lambda and its supporting components. Whether you're working with AWS Glue, Kinesis Data Firehose, SQS, or DynamoDB, this module provides a streamlined approach to infrastructure as code (IaC) in AWS. ## Building Reliable Messaging Patterns in AWS with SQS and SNS DevFeed: [Building Reliable Messaging Patterns in AWS with SQS and SNS]() Original publisher: [Read original article]() Author: Guillermo Ojeda Published: 2024-12-20T15:20:38Z Content type: tutorial Language: en Sources: [Guille Ojeda]() Topics: [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Messaging](), [distributed-systems]() Tags: [architecture](), [aws](), [distributed-system](), [distributed-systems](), [error-handling](), [fifo](), [messaging](), [scale](), [sns](), [sqs](), [technical]() ### AI overview This tutorial explains how to build reliable distributed messaging systems with AWS SQS and SNS. It covers queue configuration, error handling, scaling patterns, message retention, long polling, visibility timeouts, and the differences between Standard and FIFO queues, using Node.js examples. ### Source excerpt Building distributed systems requires putting a lot of attention on communication between components. These components often need to exchange information asynchronously, and that's where message queues and pub/sub systems are the go-to solution. AWS ... ## Scaling Spin Apps With KEDA DevFeed: [Scaling Spin Apps With KEDA]() Original publisher: [Read original article]() Author: Thorsten Hans Published: 2024-12-19T12:00:00Z Content type: tutorial Language: en Sources: [Fermyon - Experience the next wave of cloud computing.]() Topics: [Kubernetes](), [autoscaling](), [event driven](), [Amazon Simple Queue Service (SQS)](), [Amazon Web Services]() Tags: [autoscaling](), [aws](), [event-driven](), [how-to](), [kubernetes](), [scale](), [sqs]() ### AI overview A tutorial on scaling Spin apps on Kubernetes with SpinKube and KEDA, including event-driven autoscaling with AWS SQS queues to optimize resource usage and application efficiency. ### Source excerpt Learn how to scale Spin apps on Kubernetes with SpinKube and KEDA. Explore event-driven autoscaling using AWS SQS queues to optimize resource usage and enhance application efficiency. ## Building a Scalable Job Consumer System in Go DevFeed: [Building a Scalable Job Consumer System in Go]() Original publisher: [Read original article]() Author: ansu jain Published: 2024-10-16T06:48:50Z Content type: tutorial Language: en Sources: [Mindorks - Medium]() Topics: [Go Language](), [Amazon Simple Queue Service (SQS)](), [Concurrency](), [Orchestration]() Tags: [consumer](), [go](), [goroutines](), [message-queue](), [orchestration](), [parallel](), [sqs](), [workers]() ### AI overview This tutorial presents a scalable job consumer system in Go. It describes interfaces for queue consumption, job management, job handling, and configuration, with parallel workers processing messages from queues such as AWS SQS. ### Source excerpt In this article, I will walk through the design of a scalable and extensible job consumer system using Go. This system allows the flexible registration of different jobs and the parallel consumption of messages from a message queue. I'll focus on how this system can be used to handle real-world scenarios like payment processing, order proccesing etc where different jobs consume messages from a queue (e.g., AWS SQS) and execute business logic based on the type of message received. Key Components of the Design IMessageConsumer - A generic interface for interacting with message queues (e.g., SQS). IJobManager - An interface that coordinates jobs and workers, responsible for consuming messages and distributing them to registered jobs. IJobHandler - An interface representing individual jobs. Each job implements specific business logic. IQueueConfig - Provides configurations for the queue (e.g., queue name, region, broker type). IJobConfig - Provides job-level configurations (e.g., number of consumers, timeouts). Core Concepts1. IMessageConsumer This interface abstracts queue operations, exposing methods for dequeuing and deleting messages. It's implemented by specific brokers (such as SQS in this case). Here's a basic structure: type IMessageConsumer interface { Dequeue(noOfMessages int64) ([]Message, error) Delete(messageIdentifier string) error } Dequeue: Fetches messages from the queue. Delete: Deletes a message after it has been successfully processed. 2. IJobManager This is where the system's core orchestration happens. The JobManager manages a pool of registered jobs and starts multiple workers to process messages concurrently. It uses a Consume method to begin processing and a RegisterJob method to register jobs into the system. type IJobManager interface { Consume(ctx context.Context) RegisterJob(job IJobHandler) } RegisterJob: Adds a job to the manager's job list. Consume: Starts the workers that process messages from the queue in parallel for each registered jo ## Comparing the Dependency Size of AWS Java SDK v1 and v2 SQS Clients DevFeed: [Comparing the Dependency Size of AWS Java SDK v1 and v2 SQS Clients]() Original publisher: [Read original article]() Author: Doug Tangren Published: 2024-01-19T16:40:54Z Content type: article Language: en Sources: [Making Meetup - Medium]() Topics: [Java](), [Amazon Web Services](), [SDKs](), [Amazon Simple Queue Service (SQS)](), [Netty](), [Gradle](), [Architecture & Design]() Tags: [aws](), [dependencies](), [dependency-management](), [gradle](), [java](), [maintenance](), [sdk](), [sdks](), [security-vulnerabilities](), [sqs]() ### AI overview This article from Meetup compares the dependency size of AWS SDK for Java v1 and v2 using SQS client versions 1.12.637 and 2.23.3. It attributes the larger v2 footprint largely to bundled defaults and transitive dependencies, including Netty, and discusses the maintenance and security costs of dependencies. ### Source excerpt Photo by Nathan Dumlao on Unsplash At Meetup, two of our core engineering principles are to be cost conscious and use technologies that are proven to scale. On Meetup every time you click an attend button, schedule an event, create a group, start a conversation, decide join the local puppy group in your neighborhood, or any other activity your request is guaranteed to pass through multiple JVMs and within those, likely half a dozen AWS services along the way which themselves are often sitting in front of multiple JVMs. Both the JVM and AWS APIs are considered proven, rock-solid, and scalable technologies at Meetup. This is why we're heavily invested in Java and AWS for our core platform services as well as keeping both up to date. Like many companies, AWS included, we've completed the spiritual journey of migrating our largest primary platform codebase from Java 8 to 11, then to 17, and most recently to 21. The renaissance happening with the Java community has been wonderful and has unlocked a number of options for us, one being the subject of this post. Being a 20 year strong engineering focused company, we've accumulated a lot learnings in the area of understanding the cost of code dependencies. We've learned it's much easier to add than to remove dependencies and that the simplest solution to avoid the future burden of maintenance tax and security vulnerabilities attached to dependencies is simply not to invite them to the party in the first place so we're relatively conservative when evaluating new dependencies. We understand the long term tax and cost involved in doing so. When evaluating the new v2 AWS SDKs for Java, the first surprise we encountered was that it was nearing twice the size of that of the equivalent v1 SDK clients. While we started our V2 migration journey relatively long ago we recently revisited and this is still more or less the same case. We'll compare the AWS SDK SQS client versionsv1.12.637vs v2.23.3 respectively in this post. Here is an e ## Vapor: Individual Queue Concurrency DevFeed: [Vapor: Individual Queue Concurrency]() Original publisher: [Read original article]() Author: Joe Dixon Published: 2023-01-30T14:36:00Z Content type: news Language: en Sources: [Laravel Blog]() Topics: [Laravel](), [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Concurrent Programming]() Tags: [announce](), [aws](), [concurrency](), [job](), [laravel](), [new-feature](), [queue](), [sqs]() ### AI overview Laravel Vapor now supports setting maximum concurrency for individual queues. The feature distributes a function's available concurrency across SQS queues and prevents a busy queue from consuming all resources or throttling other work. ### Source excerpt Today, we're excited to announce it's now possible to set the maximum concurrency of individual Vapor queues. This new feature will allow you to ensure that not one of your queues can consume all of the available resources. ## Micro Service Events (AWS EventBridge vs. SNS) DevFeed: [Micro Service Events (AWS EventBridge vs. SNS)]() Original publisher: [Read original article]() Author: Kristian Müller Published: 2022-09-16T12:06:51Z Content type: comparison Language: en Sources: [Homegate Engineering Blog - Medium]() Topics: [event driven](), [Messaging](), [Amazon Web Services](), [Amazon Simple Queue Service (SQS)](), [Publish-subscribe pattern](), [Architecture & Design](), [AWS Lambda](), [Amazon API Gateway](), [DynamoDB](), [Amazon S3](), [TypeScript]() Tags: [api](), [api-gateway](), [architecture](), [aws](), [aws-eventbridge](), [aws-sns](), [data](), [data-processing](), [database](), [dynamodb](), [event](), [event-driven](), [eventbridge](), [eventbus](), [events](), [gateway](), [lambda](), [messaging](), [microservice-architecture](), [s3](), [sns](), [sqs](), [typescript]() ### AI overview This article examines Homegate's event-driven microservice architecture on AWS and compares Amazon SNS with EventBridge as messaging infrastructure. It describes how services use SNS, SQS, Lambda, API Gateway, DynamoDB, and S3, and explains that EventBridge has a different focus, so replacing SNS is not necessarily the right question. ### Source excerpt During the last few years, we've been busy rebuilding the Homegate platform in an event driven, micro service architecture based on AWS. One of the most important buildings blocks, or shall we rather say the mortar that connects the building blocks, is the way our services send events to each other. While Amazon offers its notification service SNS since more than 12 years; in 2019 it started a newcomer in the messaging bus field by introducing EventBridge. This was very interesting to us as we're using SNS a lot. Bus on a Bridge (Photo by "Blue Elf" on Wikimedia Commons) As EventBridge was introduced and gained traction we asked ourselves what advantaged we would gain by switching our SNS based service glue to EventBridge. Maybe this turned out to be the wrong question for us. Let's look at our system to see why. Micro Services Connected by SNS Homegate is connecting dozens of services via SNS right now. Those services range from single purpose Lambda functions written in TypeScript to database heavy services using Aurora or DynamoDB and Step Functions sending events to other services. Some of the services are also interfacing external APIs such as Salesforce or SendGrid. Here our Lambda functions can call APIs when needed. On the other hand, we have web services to be called as APIs by other systems and end users. This is mostly achieved via API Gateway. A typical service connects to the system by subscribing to events needed to fulfill its purpose. Those inbound SNS events are then ingested via SQS allowing for orderly data processing and DLQs to handle errors. Outgoing results are pushed back to the system in form of events to a SNS topics. As Marko Savic has pointed out, we're also utilizing S3 to store and replay those events if needed. Basic AWS Lambda function connected via SNS If we would replace the SNS topic by streams on the EventBridge we would initially gain the same functionality by utilizing SQS as well. However, EventBridge has a different focus. Let ## Building a reliable asynchronous job pipeline DevFeed: [Building a reliable asynchronous job pipeline]() Original publisher: [Read original article]() Published: 2022-06-17T00:00:00Z Content type: article Language: en Sources: [HackerEarth]() Topics: [Amazon Simple Queue Service (SQS)](), [Library](), [Scalability](), [web applications](), [Architecture & Design](), [Amazon Web Services](), [Amazon S3](), [Serverless](), [Provisioning](), [Security](), [Kafka](), [Self-hosted]() Tags: [applications](), [architecture](), [aws](), [infrastructure](), [jobs](), [kafka](), [libraries](), [open-source](), [pipeline](), [provisioning](), [s3](), [scalability](), [security](), [serverless]() ### AI overview The article presents he-messenger, a custom serverless library for reliable asynchronous background-job processing. Built on AWS SQS, SNS, and S3, it replaces a self-hosted RabbitMQ and Kafka solution to reduce operational overhead, improve security, and support scalable service-to-service communication. ### Source excerpt Asynchronous background jobs can dramatically improve the performance and scalability of web applications by offloading resource-intensive and time consuming processing from the request-response cycle of an application. Last year, in an effort to make our asynchronous flows more reliable, secure, and scalable, we decided to move away from our self-hosted solution that was based on RabbitMQ and Kafka, to a fully-managed one. This was done mainly to reduce the operational overheads in managing and scaling the underlying infrastructure and also to improve our overall security posture. To support this flow, we created a new library called he-messenger that implements an end-to-end queuing solution for allowing different services or different components of the same service to communicate with each other asynchronously. This library is built on top of the SQS, SNS, and S3 - managed services provided by AWS. It is a fully serverless solution that ingests events from different services, buffers them, and then delivers those events to the subscribed services in a reliable way. It simplifies the otherwise laborious process of provisioning and scaling self-hosted infrastructure. Since its introduction, this library has become one of the critical pieces of our architecture powering a lot of different use-cases with a very high number of transactions There are many open-source libraries available in the market that use AWS managed services to support asynchronous background jobs. However, none of them offered an end-to-end solution and the kind of guarantees we needed. Therefore, we decided to implement our own custom library to support this flow. The purpose of this blog is to give you an overview of the internal and the code level details of the he-messenger library along with a sample reference architecture, supported flows, features,and benefits of this solution. Terminology Channel: An abstract communication layer responsible for passing messages between producers and consu [Next page]()