# Amazon SQS

Published articles for Amazon SQS.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Symfony 8.1.7 released

DevFeed: [Symfony 8.1.7 released](<https://devfeed.tech/articles/symfony-8-1-7-released-26607.md>)

Original publisher: [Read original article](<https://symfony.com/blog/symfony-8-1-7-released>)

Author: Fabien Potencier

Published: 2026-09-15T07:15:39Z

Content type: release

Language: en

Sources: [Symfony Blog](<https://devfeed.tech/sources/symfony-blog.md>)

Topics: [Symfony](<https://devfeed.tech/topics/symfony.md>), [bug](<https://devfeed.tech/topics/bug.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>)

Tags: [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [bug](<https://devfeed.tech/tags/bug.md>), [release](<https://devfeed.tech/tags/release.md>), [symfony](<https://devfeed.tech/tags/symfony.md>)

### AI overview

Symfony 8.1.7 has been released. The release includes security, translation, Messenger, HTTP, form, console, and other fixes and minor changes, including a fix for keepalive handling on the Amazon SQS transport.

### Source excerpt

Symfony 8.1.7 has just been released. Read the Symfony upgrade guide to learn more about upgrading Symfony and use the SymfonyInsight upgrade reports to detect the code you will need to change in your project. Tip...

## Testing application resilience with Amazon SQS and AWS Fault Injection Service

DevFeed: [Testing application resilience with Amazon SQS and AWS Fault Injection Service](<https://devfeed.tech/articles/testing-application-resilience-with-amazon-sqs-and-aws-fault-injection-service-4651.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/testing-application-resilience-with-amazon-sqs-and-aws-fault-injection-service/>)

Author: Richard Whitworth

Published: 2026-09-09T21:33:59Z

Content type: tutorial

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [AWS Fault Injection Service (FIS)](<https://devfeed.tech/topics/aws-fault-injection-service-fis.md>), [Chaos Engineering](<https://devfeed.tech/topics/chaos-engineering.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>), [AWS Identity and Access Management (IAM)](<https://devfeed.tech/topics/aws-identity-and-access-management-iam.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-simple-queue-service-sqs](<https://devfeed.tech/tags/amazon-simple-queue-service-sqs.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [automation](<https://devfeed.tech/tags/automation.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-fault-injection-service-fis](<https://devfeed.tech/tags/aws-fault-injection-service-fis.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iam](<https://devfeed.tech/tags/iam.md>), [observability](<https://devfeed.tech/tags/observability.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

A tutorial on testing application resilience when Amazon SQS data-plane operations fail. It uses AWS Fault Injection Service and Systems Manager Automation to progressively deny queue access, evaluate recovery and observability with CloudWatch metrics, and avoid IAM deny-policy lockouts.

### Source excerpt

Learn how to use AWS Fault Injection Service and AWS Systems Manager Automation to run progressive chaos experiments against Amazon SQS queues. Validate that your retry logic, circuit breakers, and dead-letter queues actually work under failure before a real outage hits production.

## Scheduling email campaigns at scale with Amazon EventBridge Scheduler

DevFeed: [Scheduling email campaigns at scale with Amazon EventBridge Scheduler](<https://devfeed.tech/articles/scheduling-email-campaigns-at-scale-with-amazon-eventbridge-scheduler-4671.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/compute/scheduling-email-campaigns-at-scale-with-amazon-eventbridge-scheduler/>)

Author: Oluwaseun Ademuwagun

Published: 2026-09-01T13:09:17Z

Content type: tutorial

Language: en

Sources: [AWS Compute Blog](<https://devfeed.tech/sources/aws-compute-blog.md>)

Topics: [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Software as a service](<https://devfeed.tech/topics/saas.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-eventbridge](<https://devfeed.tech/tags/amazon-eventbridge.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer](<https://devfeed.tech/tags/customer.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [integration](<https://devfeed.tech/tags/integration.md>), [local](<https://devfeed.tech/tags/local.md>), [post](<https://devfeed.tech/tags/post.md>), [saas](<https://devfeed.tech/tags/saas.md>), [scale](<https://devfeed.tech/tags/scale.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

This tutorial explains how to schedule personalized email campaigns at scale with Amazon EventBridge Scheduler. It uses one schedule per recipient to deliver messages at individually optimal times, discusses batch cron jobs, Amazon SQS delay queues, and third-party campaign tools, and describes scaling schedule creation with AWS Step Functions Distributed Map and delivery through Amazon SES.

### Source excerpt

Learn how to use Amazon EventBridge Scheduler to deliver email campaigns at per-recipient optimal send times. This post shows how to create one schedule per recipient with zero idle compute cost, scale schedule creation with AWS Step Functions Distributed Map, and deliver through Amazon SES.

## Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode

DevFeed: [Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode](<https://devfeed.tech/articles/low-latency-high-throughput-sqs-event-processing-with-aws-lambda-provisioned-mode-4668.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/compute/low-latency-high-throughput-sqs-event-processing-with-aws-lambda-provisioned-mode/>)

Author: Ben Freiberg

Published: 2026-08-10T13:42:03Z

Content type: article

Language: en

Sources: [AWS Compute Blog](<https://devfeed.tech/sources/aws-compute-blog.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [AWS Lambda](<https://devfeed.tech/topics/aws-lambda.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [amazon-simple-queue-service-sqs](<https://devfeed.tech/tags/amazon-simple-queue-service-sqs.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [flash](<https://devfeed.tech/tags/flash.md>), [fraud](<https://devfeed.tech/tags/fraud.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [payment-processing](<https://devfeed.tech/tags/payment-processing.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>)

### AI overview

AWS explains how provisioned mode for Amazon SQS event source mappings with AWS Lambda provides explicit control over event pollers to support low-latency, high-throughput event processing. The article describes default and provisioned scaling, concurrency limits, throughput, and use cases including real-time payments, fraud detection, IoT telemetry, and flash-sale order fulfillment.

### Source excerpt

Customers building event-driven applications on AWS rely on Amazon Simple Queue Service (Amazon SQS) and AWS Lambda event source mappings (ESMs) to process millions of events every day. The fully managed polling infrastructure of ESMs eliminates the need to write and maintain custom code. You can focus on business logic while Lambda handles scaling, batching, [...]

## Amazon SQS turns 20: Two decades of reliable messaging at scale

DevFeed: [Amazon SQS turns 20: Two decades of reliable messaging at scale](<https://devfeed.tech/articles/amazon-sqs-turns-20-two-decades-of-reliable-messaging-at-scale-4607.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/aws/amazon-sqs-turns-20-two-decades-of-reliable-messaging-at-scale/>)

Author: Esra Kayabali

Published: 2026-07-13T18:13:57Z

Content type: article

Language: en

Sources: [AWS News Blog](<https://devfeed.tech/sources/aws-news-blog.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Encryption](<https://devfeed.tech/topics/encryption.md>), [Security](<https://devfeed.tech/topics/security.md>), [Availability](<https://devfeed.tech/topics/availability.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-simple-queue-service-sqs](<https://devfeed.tech/tags/amazon-simple-queue-service-sqs.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [availability](<https://devfeed.tech/tags/availability.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [encryption](<https://devfeed.tech/tags/encryption.md>), [launch](<https://devfeed.tech/tags/launch.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [news](<https://devfeed.tech/tags/news.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scale](<https://devfeed.tech/tags/scale.md>), [security](<https://devfeed.tech/tags/security.md>), [systems](<https://devfeed.tech/tags/systems.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Amazon SQS marks its 20th anniversary with a retrospective on reliable, asynchronous messaging and the decoupling of producers from consumers. The article highlights recent developments including higher-throughput FIFO queues, default server-side encryption, and dead-letter queue redrive enhancements.

### Source excerpt

On July 13, 2006, we launched Amazon Simple Queue Service (Amazon SQS) as one of the first three services available to customers, alongside Amazon EC2 and Amazon S3. We had learned firsthand that distributed systems need a reliable way to pass messages between components without creating tight dependencies. If one service called another directly and [...]

## Unordered mode for S3 ClickPipes is now available

DevFeed: [Unordered mode for S3 ClickPipes is now available](<https://devfeed.tech/articles/unordered-mode-for-s3-clickpipes-is-now-available-5182.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/clickpipes-s3-unordered-mode>)

Author: Marta Paes

Published: 2025-12-09T15:06:25Z

Content type: release

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [data](<https://devfeed.tech/tags/data.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [queue](<https://devfeed.tech/tags/queue.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

ClickPipes now supports unordered ingestion from Amazon S3 into ClickHouse Cloud using S3 Event Notifications through Amazon SQS. This handles backfills, retries, late-arriving data, and other out-of-order files without relying on lexicographical file order.

### Source excerpt

Ingest data from Amazon S3 into ClickHouse Cloud in any order for event-driven, blazing-fast analytics.

## HubSpot Incident Report for October 20, 2025

DevFeed: [HubSpot Incident Report for October 20, 2025](<https://devfeed.tech/articles/hubspot-incident-report-for-october-20-2025-29106.md>)

Original publisher: [Read original article](<https://product.hubspot.com/blog/incident-report-for-october-20-2025>)

Author: Kartik Vishwanath

Published: 2025-11-11T17:09:36Z

Content type: news

Language: en

Sources: [HubSpot](<https://devfeed.tech/sources/hubspot.md>)

Topics: [incident](<https://devfeed.tech/topics/incident.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [AWS IAM](<https://devfeed.tech/topics/aws-iam.md>), [DynamoDB](<https://devfeed.tech/topics/dynamodb.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [dynamodb](<https://devfeed.tech/tags/dynamodb.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [failover](<https://devfeed.tech/tags/failover.md>), [hubspot](<https://devfeed.tech/tags/hubspot.md>), [iam](<https://devfeed.tech/tags/iam.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [incident](<https://devfeed.tech/tags/incident.md>), [message-queue](<https://devfeed.tech/tags/message-queue.md>), [outage](<https://devfeed.tech/tags/outage.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [report](<https://devfeed.tech/tags/report.md>), [resilience](<https://devfeed.tech/tags/resilience.md>)

### AI overview

HubSpot reports that a severe AWS outage in the us-east-1 region on October 20, 2025 disrupted multiple product features and affected third-party vendors. The incident caused failures involving DynamoDB, IAM, SQS, and EC2, and degraded HubSpot's TQ2 background task processing. HubSpot says it completed an analysis and is implementing resilience improvements.

### Source excerpt

On October 20, 2025, HubSpot experienced a significant service disruption affecting multiple product features due to a severe AWS outage in the us-east-1 region. While our infrastructure remained intact, the widespread nature of the cloud provider failure impacted both our services and critical third-party vendors we rely on. We've completed a thorough analysis of this incident and are implementing comprehensive improvements to strengthen our resilience against future cloud provider disruptions.

## Building event-driven pipelines with SQS and S3

DevFeed: [Building event-driven pipelines with SQS and S3](<https://devfeed.tech/articles/building-event-driven-pipelines-with-sqs-and-s3-12679.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/building-event-driven-pipelines-sqs-s3>)

Author: Chandler Mayo

Published: 2025-08-21T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [amazon-sqs-s3-integration](<https://devfeed.tech/tags/amazon-sqs-s3-integration.md>), [automate-data-ingestion-s3](<https://devfeed.tech/tags/automate-data-ingestion-s3.md>), [building-data-pipelines-with-sqs-and-s3](<https://devfeed.tech/tags/building-data-pipelines-with-sqs-and-s3.md>), [building-event-driven-pipelines-with-sqs-and-s3](<https://devfeed.tech/tags/building-event-driven-pipelines-with-sqs-and-s3.md>), [data-pipeline-automation-with-sqs](<https://devfeed.tech/tags/data-pipeline-automation-with-sqs.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [event-driven-data-processing](<https://devfeed.tech/tags/event-driven-data-processing.md>), [event-driven-pipelines-sqs-s3](<https://devfeed.tech/tags/event-driven-pipelines-sqs-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data-pipelines-s3](<https://devfeed.tech/tags/real-time-data-pipelines-s3.md>), [s3-to-redpanda-streaming](<https://devfeed.tech/tags/s3-to-redpanda-streaming.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [set-up-event-driven-architecture-aws](<https://devfeed.tech/tags/set-up-event-driven-architecture-aws.md>), [sqs-s3-data-pipeline-tutorial](<https://devfeed.tech/tags/sqs-s3-data-pipeline-tutorial.md>), [sqs-s3-notifications-for-data-pipelines](<https://devfeed.tech/tags/sqs-s3-notifications-for-data-pipelines.md>), [streaming-data-to-redpanda-from-s3](<https://devfeed.tech/tags/streaming-data-to-redpanda-from-s3.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to use S3 Event Notifications and Amazon SQS to trigger Redpanda Connect processing whenever new objects arrive in an S3 bucket. It demonstrates an automated, continuously running pipeline that ingests object-storage data into real-time streams.

### Source excerpt

Learn how to build real-time, event-driven data pipelines directly from your object storage with Amazon SQS and S3.

## Using SQS to Throttle Writes to DynamoDB

DevFeed: [Using SQS to Throttle Writes to DynamoDB](<https://devfeed.tech/articles/using-sqs-to-throttle-writes-to-dynamodb-18018.md>)

Original publisher: [Read original article](<https://blog.guilleojeda.com/sqs-throttle-database-writes-dynamodb>)

Author: Guillermo Ojeda

Published: 2023-09-07T00:12:11Z

Content type: tutorial

Language: en

Sources: [Guille Ojeda](<https://devfeed.tech/sources/guille-ojeda.md>)

Topics: [Amazon SQS](<https://devfeed.tech/topics/amazon-sqs.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Database](<https://devfeed.tech/topics/database.md>)

Tags: [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [aws](<https://devfeed.tech/tags/aws.md>), [backend](<https://devfeed.tech/tags/backend.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [databases](<https://devfeed.tech/tags/databases.md>), [dynamodb](<https://devfeed.tech/tags/dynamodb.md>), [e-commerce](<https://devfeed.tech/tags/e-commerce.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [queue](<https://devfeed.tech/tags/queue.md>), [queuing](<https://devfeed.tech/tags/queuing.md>), [sqs](<https://devfeed.tech/tags/sqs.md>)

### AI overview

A technical article describes an e-commerce order-processing system whose DynamoDB provisioned capacity cannot keep up with sudden write traffic. It examines using Amazon SQS to queue writes and throttle database load, while noting that DynamoDB on-demand capacity is an alternative with higher stated cost.

### Source excerpt

We're running an e-commerce platform, where people publish products and other people purchase those products. Our backend has some highly scalable microservices running on well-designed Lambdas, and there's a lot of caching involved. Our order proces...

## Message Batching to Increase Throughput and Reduce Costs

DevFeed: [Message Batching to Increase Throughput and Reduce Costs](<https://devfeed.tech/articles/message-batching-to-increase-throughput-and-reduce-costs-24979.md>)

Original publisher: [Read original article](<https://codeahoy.com/2017/08/03/message-batching-to-increase-throughput-and-reduce-costs/>)

Author: umer

Published: 2017-08-03T00:00:00Z

Content type: tutorial

Language: en

Sources: [Code Ahoy - Articles](<https://devfeed.tech/sources/code-ahoy-articles.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>)

Tags: [advice](<https://devfeed.tech/tags/advice.md>), [amazon-sqs](<https://devfeed.tech/tags/amazon-sqs.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [aws](<https://devfeed.tech/tags/aws.md>), [backend](<https://devfeed.tech/tags/backend.md>), [batching](<https://devfeed.tech/tags/batching.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [latency](<https://devfeed.tech/tags/latency.md>), [performance](<https://devfeed.tech/tags/performance.md>), [queue](<https://devfeed.tech/tags/queue.md>), [services](<https://devfeed.tech/tags/services.md>)

### AI overview

The article describes using batching and buffering with Amazon SQS to group messages before sending them. In the author's system, batches of up to 15 messages or 50 milliseconds increased throughput, reduced SQS costs, and had little effect on latency, while retries can mitigate message loss after a crash.

### Source excerpt

A while ago, I was working on a backend system on the AWS cloud. Individual services in the system communicated by exchanging asynchronous messages with each other using Amazon SQS. During early stages of development, we ran small load tests and found that CPU use was high and we would need more servers to handle the load. (Estimated peak load was 50,000 requests per second.) Services were generating lots of small messages every second and the profiler showed that threads responsible for handling and sending messages to SQS, one message at a time, were using CPU more than they should. This was affecting performance and throughput. To overcome this challenge, we took a page from Amazon's best practices advice, and introduced batching and buffering on senders to group multiple messages and send as one batch. This simple fix increased the throughput, cut down SQS costs (SQS pricing is by number of messages), and had very little impact on latency. Win, win, win. The custom batching algorithm was simple: batch up to 15 messages or wait up to a maximum of 50 milliseconds. If either 15 messages arrive quickly to form a batch or 50 milliseconds elapse, the batch is sent out. These figures (batch size and maximum time) were established after trial and error, tuned for the best throughput and latency. The application was multi-threaded with hundreds of active threads, so the system allowed multiple concurrent batches to be active at the same time to reduce thread blocking. The only downside is that if the application crashes after retrieving batch from the queue but before processing messages in it, we can lose some or all of the messages. To deal with complete message loss, the system can implement retries and resend messages upon timeout. Clients will see 2x response times, but at least get their response back. If you are building applications which generate and exchange a lot of messages in short periods, batching can increase throughput, performance and in the case of SQS