# AWS FIFO Queues with Message Groups for Atomic Processing at Scale

DevFeed: [AWS FIFO Queues with Message Groups for Atomic Processing at Scale](<https://devfeed.tech/articles/aws-fifo-queues-with-message-groups-for-atomic-processing-at-scale-23895.md>)

Original publisher: [Read original article](<https://medium.com/smg-real-estate/aws-fifo-queues-with-message-groups-for-atomic-processing-at-scale-dc70f8820b16?source=rss----2186e5b9bd8f---4>)

Author: Nick De Cooman

Published: 2022-03-03T09:55:58Z

Content type: tutorial

Language: en

Sources: [Homegate Engineering Blog - Medium](<https://devfeed.tech/sources/homegate-engineering-blog-medium.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [decoupling](<https://devfeed.tech/tags/decoupling.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [event](<https://devfeed.tech/tags/event.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [events](<https://devfeed.tech/tags/events.md>), [fifo](<https://devfeed.tech/tags/fifo.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [sns](<https://devfeed.tech/tags/sns.md>), [sqs](<https://devfeed.tech/tags/sqs.md>)

## AI overview

A tutorial on using AWS FIFO queues and message groups to process events for multiple objects in parallel while ensuring that only one instance processes events for the same object at a time. It presents a serverless architecture using SNS, SQS, and Lambda for atomic listing ingestion.

## Source excerpt

How to process multiple objects in parallel with the guarantee that for each unique object, only a single instance is processed simultaneously In a distributed, event-driven architecture, many challenges arise when processing large sets of data. One of them is the need for scalability when handling event streams with a substantial throughput. The challenge becomes even more complicated when processing these events involves atomic operations. In this case, we cannot simply process multiple events in parallel. As an example, let's say that we are building a service that ingests listings into our platform. The ingestion process involves several steps and requires multiple API operations. As a result, the whole ingestion can take up to several seconds per listing to complete. Now, here is where it gets tricky: a constraint of our service is that, while we are processing a listing, we cannot process other events for the same listing. The ingestion should be considered an atomic operation, and hence, only a single execution per listing can be performed simultaneously. At Homegate, we run most of our infrastructure on AWS in a serverless fashion. At the core of this, Lambda functions execute application logic. A common pattern is to use SNS topics to fan-out messages, and SQS for acting as a decoupling buffer between an SNS topic and some Lambda function. For our ingester service, a logical architecture could look like this: Listing events are published to a dedicated SNS topic. An SQS queue subscribes to this topic and consumes the events. Here, the events are buffered until a Lambda instance removes them from the queue and processes the corresponding listing. Under normal circumstances -- in which the processing would not involve an atomic operation -- this architecture would be a good fit. It would scale automatically relative to the number of Lambda instances that consume events from the queue, without much additional configuration. However, in the case where we cannot p