# message queue Published articles for message queue. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## What is Infrastructure from Code? DevFeed: [What is Infrastructure from Code?]() Original publisher: [Read original article]() Author: Ivan Cernja Published: 2026-03-17T00:00:00Z Content type: article Language: en Sources: [Encore Updates]() Topics: [Code](), [Cloud](), [Artificial Intelligence](), [Terraform](), [Databases](), [AWS IAM](), [Caching](), [Amazon Simple Queue Service (SQS)]() Tags: [ai](), [ai-agents](), [aws](), [caching](), [cloud](), [code](), [database](), [infrastructure](), [least-privilege](), [message-queue](), [terraform]() ### AI overview The article explains Infrastructure from Code (IFC), an approach that declares infrastructure such as databases, queues, and scheduled jobs in application code. It describes how frameworks derive cloud resources from those declarations and argues that AI-generated application changes have increased the need to keep infrastructure synchronized. ### Source excerpt How it works, why AI made it urgent, and what the trade-offs are. ## HubSpot Incident Report for October 20, 2025 DevFeed: [HubSpot Incident Report for October 20, 2025]() Original publisher: [Read original article]() Author: Kartik Vishwanath Published: 2025-11-11T17:09:36Z Content type: news Language: en Sources: [HubSpot]() Topics: [incident](), [Amazon Web Services](), [Cloud](), [Resilience](), [Amazon Simple Queue Service (SQS)](), [Amazon EC2](), [AWS IAM](), [DynamoDB]() Tags: [amazon](), [amazon-sqs](), [analysis](), [aws](), [cloud](), [data](), [database](), [distributed](), [dynamodb](), [ec2](), [engineering](), [failover](), [hubspot](), [iam](), [improvements](), [incident](), [message-queue](), [outage](), [recovery](), [report](), [resilience]() ### AI overview HubSpot reports that a severe AWS outage in the us-east-1 region on October 20, 2025 disrupted multiple product features and affected third-party vendors. The incident caused failures involving DynamoDB, IAM, SQS, and EC2, and degraded HubSpot's TQ2 background task processing. HubSpot says it completed an analysis and is implementing resilience improvements. ### Source excerpt On October 20, 2025, HubSpot experienced a significant service disruption affecting multiple product features due to a severe AWS outage in the us-east-1 region. While our infrastructure remained intact, the widespread nature of the cloud provider failure impacted both our services and critical third-party vendors we rely on. We've completed a thorough analysis of this incident and are implementing comprehensive improvements to strengthen our resilience against future cloud provider disruptions. ## A fictional story about vanishing messages in Snap's microservices DevFeed: [A fictional story about vanishing messages in Snap's microservices]() Original publisher: [Read original article]() Author: Tim Imkin Published: 2024-10-31T07:00:00Z Content type: opinion Language: en Sources: [Temporal Blog]() Topics: [Messaging](), [Microservice](), [Traces]() Tags: [community](), [debug](), [message-queue](), [messages](), [microservices](), [snap](), [traces]() ### AI overview A fictional spooky story follows an on-call engineer investigating failures in Snap's microservices, including disappearing messages, queue errors, traces pointing to nonexistent workflows, and unexplained anomalies. ### Source excerpt They say something strange happens deep within Snap Tower, hidden behind the pristine walls and endless lines of code. ## Building a Scalable Job Consumer System in Go DevFeed: [Building a Scalable Job Consumer System in Go]() Original publisher: [Read original article]() Author: ansu jain Published: 2024-10-16T06:48:50Z Content type: tutorial Language: en Sources: [Mindorks - Medium]() Topics: [Go Language](), [Amazon Simple Queue Service (SQS)](), [Concurrency](), [Orchestration]() Tags: [consumer](), [go](), [goroutines](), [message-queue](), [orchestration](), [parallel](), [sqs](), [workers]() ### AI overview This tutorial presents a scalable job consumer system in Go. It describes interfaces for queue consumption, job management, job handling, and configuration, with parallel workers processing messages from queues such as AWS SQS. ### Source excerpt In this article, I will walk through the design of a scalable and extensible job consumer system using Go. This system allows the flexible registration of different jobs and the parallel consumption of messages from a message queue. I'll focus on how this system can be used to handle real-world scenarios like payment processing, order proccesing etc where different jobs consume messages from a queue (e.g., AWS SQS) and execute business logic based on the type of message received. Key Components of the Design IMessageConsumer - A generic interface for interacting with message queues (e.g., SQS). IJobManager - An interface that coordinates jobs and workers, responsible for consuming messages and distributing them to registered jobs. IJobHandler - An interface representing individual jobs. Each job implements specific business logic. IQueueConfig - Provides configurations for the queue (e.g., queue name, region, broker type). IJobConfig - Provides job-level configurations (e.g., number of consumers, timeouts). Core Concepts1. IMessageConsumer This interface abstracts queue operations, exposing methods for dequeuing and deleting messages. It's implemented by specific brokers (such as SQS in this case). Here's a basic structure: type IMessageConsumer interface { Dequeue(noOfMessages int64) ([]Message, error) Delete(messageIdentifier string) error } Dequeue: Fetches messages from the queue. Delete: Deletes a message after it has been successfully processed. 2. IJobManager This is where the system's core orchestration happens. The JobManager manages a pool of registered jobs and starts multiple workers to process messages concurrently. It uses a Consume method to begin processing and a RegisterJob method to register jobs into the system. type IJobManager interface { Consume(ctx context.Context) RegisterJob(job IJobHandler) } RegisterJob: Adds a job to the manager's job list. Consume: Starts the workers that process messages from the queue in parallel for each registered jo ## How Hotstar Built a Scalable Real-Time Emoji System for Sports Viewing DevFeed: [How Hotstar Built a Scalable Real-Time Emoji System for Sports Viewing]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2024-03-26T15:32:38Z Content type: article Language: en Sources: [High Scalability]() Topics: [Emoji](), [Architecture & Design](), [Scalability](), [real-time](), [API](), [HTTP](), [client]() Tags: [api](), [architecture](), [article](), [asynchronous](), [blog-post](), [emojis](), [http](), [message-queue](), [real-time](), [scalability]() ### AI overview This article explains how Hotstar built an in-house system to collect user-submitted emojis in real time, condense them into an emoji swarm showing audience mood, and support very high submission volumes. It covers horizontal scalability, component decomposition, asynchronous processing, load balancing, auto-scaling, and HTTP API request handling. ### Source excerpt This blog post was written by Dedeepya Bonthu. This is a repost from her Medium article, approved by the author. In stadiums, sports fans love to express themselves by cheering for their favorite teams, holding up placards and team logos. Emoji's allow fans at home to rapidly express ## Recap: Apache Pulsar Sessions in CommunityOverCode Asia 2023 DevFeed: [Recap: Apache Pulsar Sessions in CommunityOverCode Asia 2023]() Original publisher: [Read original article]() Author: tison Published: 2023-08-30T00:00:00Z Content type: article Language: en Sources: [Apache Pulsar Blog]() Topics: [Apache Pulsar](), [Kafka](), [Cloud](), [Containers](), [Optimization](), [huawei]() Tags: [apache](), [aws](), [cloud-native](), [clusters](), [conference](), [huawei](), [io](), [kafka](), [load-balancer](), [load-balancing](), [lts](), [message-queue](), [new-features](), [optimization](), [production](), [recap]() ### AI overview A recap of Apache Pulsar sessions at CommunityOverCode Asia 2023, covering the Pulsar 3.0 first LTS release, new load-balancing and delayed-message features, Direct IO optimization, container-focused cloud-native improvements, and traffic-limiting practices involving Pulsar and Kafka. ### Source excerpt The CommunityOverCode Asia 2023 conference (previously known as ApacheCon Asia) has been held from August 18th to August 20th. ## Replicating Database Changes to a Message Queue is Tricky DevFeed: [Replicating Database Changes to a Message Queue is Tricky]() Original publisher: [Read original article]() Published: 2022-12-13T02:01:42Z Content type: tutorial Language: en Sources: [Evan Jones]() Topics: [Database](), [Kafka](), [systems]() Tags: [database](), [kafka](), [message-queue](), [systems]() ### AI overview This article explains why replicating database changes to a message queue can produce missing or extra updates when components fail. It compares parallel and sequential approaches and recommends using a database transaction to update application state and record pending messages at a single commit point. ### Source excerpt Let's imagine we have an program that stores its state in a database, and we want other programs to do things when changes occur. For example, we might want to send email notifications if a bank balance drops below a threshold. This is a very common reason applications use message queues like Kafka. Unfortunately, the "trivial" implementation does not work when components fail. I suspect there are many real applications that get this wrong. Most of the time, these applications work correctly, and the changes are replicated across multiple systems. However, when things restart, updates can go missing, or extra updates can appear. In this article, I'm going to try to explain how this can go wrong, and some ways to fix it. Attempt one: Update both in parallel The application performs the following operations: Write the change to the database. At the same time, publish the message to the message queue. The problem in this case is that the database update could fail, but publishing the message succeeds. This means applications consuming the stream receive an "extra" update that does not exist in the database. Attempt two: Update database, then publish message Okay, let's try again, and make sure that updating the database succeeds: Write the change to the database. Wait for the database to confirm the write occurred. Publish the message to the message queue. We fixed the "extra" message update problem! However, we still have a problems: If the application crashes after writing to the database, but before publishing the message, the stream is missing an update. This can be particularly bad if the message queue is unavailable. The application can retry publishing the message for a while. However, if the message queue is down for long enough, it is likely the application will run out of memory, or be restarted. In this case, all the pending updates are lost. So now what? We can't do the operations sequentially, and we can't do them in parallel. The trick is to order the wor ## Never ever use a database as a message queue. Do this instead. DevFeed: [Never ever use a database as a message queue. Do this instead.]() Original publisher: [Read original article]() Author: Glauber Costa Published: 2022-08-23T00:00:00Z Content type: article Language: en Sources: [Turso Blog]() Topics: [Turso](), [SQLite](), [Back end](), [libSQL]() Tags: [backend](), [database](), [javascript](), [kafka](), [libsql](), [message-queue](), [queue](), [sqlite](), [turso]() ### AI overview The article argues against using a database as a message queue and proposes an alternative. The supplied text describes Turso as a backend and data-layer platform built from a rewrite of SQLite, with database access over the wire and synchronization to devices and servers. It also notes that the post references an older version of Turso. ### Source excerpt If you are an application developer and need a full backend, including the data layer, you have two choices: Relying on a backend team may be the solution we all want. But often, especially with... ## Live Video Transmuxing/Transcoding: FFmpeg vs TwitchTranscoder, Part II DevFeed: [Live Video Transmuxing/Transcoding: FFmpeg vs TwitchTranscoder, Part II]() Original publisher: [Read original article]() Author: Yueshi Shen Published: 2017-10-23T21:08:55Z Content type: article Language: en Sources: [Twitch]() Topics: [FFmpeg (Fast Forward Moving Picture Experts Group)](), [Transcodings](), [Code](), [Concurrency](), [C]() Tags: [c](), [code](), [decoding](), [encoding](), [engineering](), [ffmpeg](), [files](), [function](), [loops](), [message-queue](), [programming](), [thread](), [threading](), [threads](), [video]() ### AI overview This second part of a two-part series examines FFmpeg Release 3.3's threading model and transcoding pipeline for a one-input, multiple-output scenario. It traces how FFmpeg orchestrates I/O, filtering, decoding, encoding, and output processing, and reports that the number of input files determines input-thread creation, so a one-input scenario uses a single thread. ### Source excerpt By: Jeff Gong, Software Engineer, jeffgon@twitch.tv Sahil Dhanju, Software Engineer Intern Chih-Chiang Lu, Senior Software Engineer, chihchil@twitch.tv Yueshi Shen, Principal Research Engineer, yshen@twitch.tv Special thanks go to: Christopher Kennedy, Staff Video Engineer at Crunchyroll/Ellation John Nichols, Principal Software Engineer at Xilinx, jnichol@xilinx.com for their information on FFmpeg and reviewing this article. Note: This is the second part of a 2-part series. Make sure you read Part 1 first. FFmpeg's 1-In-N-Out Pipeline. Why doesn't it handle the technical issues discussed earlier? How does FFmpeg programmatically deal with instances where a single input stream is required to generate multiple transcoded and/or transmuxed outputs? We went directly into the latest FFmpeg Release 3.3. source code in order to understand its threading model and transcoding pipeline. In the top-level ffmpeg.c file, the transcode() function (line 4544) loops and repeatedly calls transcode_step() (line 4478) until its inputs are completely processed, or until the user interrupts the execution. Transcode_step() wraps the main pipeline and orchestrates file I/O, filtering, decoding and encoding amongst many other immediate steps. During the initial setup phase, init_input_threads() (line 4020) is called, and based on the number of input files, a number of new threads may be spawned to process the input. if (nb_input_files == 1) { return 0;}for (i = 0; i < nb_input_files; i++) { ... ret = av_thread_message_queue_alloc(&f->in_thread_queue, f->thread_queue_size, sizeof(AVPacket)); // line 4033} In line 4033, we see that the number of threads spawned is solely determined by the number of inputs. This means FFmpeg will process a 1-in-N-out scenario using only a single thread. In get_input_packet() (line 4055), the multithreaded companion function get_input_packet_mt() (line 4047) is only called if the number of input files is greater than one. get_input_packet_mt() can read input ## Disque: A Message Broker Designed for Reliable Message Delivery DevFeed: [Disque: A Message Broker Designed for Reliable Message Delivery]() Original publisher: [Read original article]() Published: 2015-03-15T22:32:15Z Content type: article Language: en Sources: [Antirez]() Topics: [Redis]() Tags: [blog](), [broker](), [message-queue](), [queue](), [redis]() ### AI overview The article introduces Disque, an in-development message broker, and explains why developers often use Redis as a message queue. It discusses the differences between Redis data structures and immutable messages, along with at-most-once and at-least-once delivery semantics. ### Source excerpt EDIT: In case you missed it, Disque source code is now available at http://github.com/antirez/disque It is a few months that I spend ~ 15-20% of my time, mostly hours stolen to nights and weekends, working to a new system. It's a message broker and it's called Disque. I've an implementation of 80% of what was in the original specification, but still I don't feel like it's ready to be released. Since I can't ship, I'll at least blog... so that's the story of how it started and a few details about what it is. ~ First steps ~ Many developers use Redis as a message queue, often wrappered via some library abstracting away Redis low level primitives, other times directly building a simple, ad-hoc queue, using the Redis raw API. This use case is covered mainly using blocking list operations, and list push operations. Redis apparently is at the same time the best and the worst system to use like that. It's good because it is fast, easy to inspect, deploy and use, and in many environments it was already one piece of the infrastructure. However it has disadvantages because Redis mutable data structures are very different than immutable messages. Redis HA / Cluster tradeoffs are totally biased towards large mutable values, but the same tradeoffs are not the best ones to deal with messages. One thing that is important to guarantee for a message broker is that a message is delivered either at least one time, or at most one time. In short given that to guarantee an exact single delivery of a message (where for delivery we intent a message that was received *and* processed by a worker) is practically impossible, the choices are that the message broker is able to guarantee either 0 or 1 deliveries, or 1 to infinite deliveries. This is often referred as at-most-once semantics, and at-least-once semantics. There are use cases for the first, but the most interesting and practical semantics is the latter, that is, to guarantee that a message is delivered at least one time, and deliver mu ## A journey on the Android Main Thread -- Lifecycle bits DevFeed: [A journey on the Android Main Thread -- Lifecycle bits]() Original publisher: [Read original article]() Author: P-Y Ricau Published: 2013-12-26T17:05:00Z Content type: article Language: en Sources: [Square Corner Blog RSS Feed]() Topics: [Android](), [Code]() Tags: [android](), [components](), [configuration](), [crash](), [engineering](), [lifecycle](), [main-thread](), [message-queue](), [queue](), [thread]() ### AI overview This article examines how Android's main thread interacts with activity lifecycle events, especially configuration changes. It explains that an orientation change can recreate an activity by posting messages such as CONFIGURATION_CHANGED and RELAUNCH_ACTIVITY to the main thread looper queue. ### Source excerpt Written by Pierre-Yves Ricau. ## The Robust Realtime Server DevFeed: [The Robust Realtime Server]() Original publisher: [Read original article]() Published: 2013-05-31T00:00:00Z Content type: article Language: en Sources: [HackerEarth]() Topics: [Server](), [Socket.IO](), [Meteor](), [Database](), [Web](), [Cache]() Tags: [blog-post](), [browser](), [cache](), [code](), [communication](), [database](), [message-queue](), [server](), [web]() ### AI overview This developer article describes a realtime server that updates webpages when database or cache data changes. It discusses scaling problems with nowjs and socket.io, including connection limits and file descriptor leaks, then explains choosing Tornado as an alternative. The described code-submission workflow uses a web server, RabbitMQ message queues, a code-checker engine, asynchronous processing, and browser notifications. ### Source excerpt This is going to be a long blog post but I promise you will find some interesting piece of engineering here, so stay till the end. The realtime server manages the live update of webpages when the data changes in the data storage system (database or cache). We had a realtime server in-place but there was a big problem with scaling it. ####Problem with nowjs I was told beforehand that I will be primarily working first on writing a realtime server beside many other things. Vivek Prakash told me he had written a realtime server implementation sometime ago with nowjs. But the problem with it is that it doesn't scale well beyond ~200 simultaneous connections. In a conversation on Google Groups, I came across this: In my experience, the underlying "socket.io" module is not able to scale well (more than 150 connections was a problem for me), so I had to retreat from using "nowjs" or more specifically, "socket.io" in one of my applications. After further inspection, we also saw that there was an issue with file descriptor leak and nowjs server reported ENFILE/EMFILE (Too many open files). Also nowjs project was abandoned in 2012 and last commit in github repo is that of 1 year ago. So there was need of some good alternative which can handle large number of simultaneous connections (or users). I didn't have to do much research as Vivek had already researched about it. He found Tornado and Meteor.js to be good alternative. Going by order of preference and popularity I chose Tornado, and also because it's integration with existing system looked simpler and more efficient. ####The Use Case Vivek pretty much explained me how different components of code submission works. Here is a quick explanation of it. User submits the code and a POST request is sent to webserver which further sends submission details to a message queue in RabbitMQ server (a message broker to connect various application components). Code-checker engine (consumer of RabbitMQ here) gets the submission details, e ## Building NSQ Client Libraries DevFeed: [Building NSQ Client Libraries]() Original publisher: [Read original article]() Author: Wordbitly Published: 2013-05-09T18:58:23Z Content type: tutorial Language: en Sources: [Bitly]() Topics: [Messaging](), [client](), [configuration](), [servers](), [HTTP]() Tags: [asynchronous](), [client-library](), [configuration](), [developers](), [distributed](), [guide](), [infrastructure](), [message-queue](), [messaging](), [nsq](), [tcp](), [thundering-herd]() ### AI overview A guide to the responsibilities and design expectations of NSQ client libraries, focusing on consumers. It covers configuration, discovery, TCP connections, message handling, pipelining, asynchronous processing, and techniques for maintaining cluster robustness and performance. ### Source excerpt Brace yourself, this is a long one. The following guide was originally intended for client library developers to describe in detail all the important features and functionality we expected in an NSQ client library. While writing it we began to realize that it had value beyond client library developers. It incorporates a comprehensive analysis of most of the capabilities of NSQ (both client and server) and is therefore interesting and useful for end-users as well (or anyone using or interested in infrastructure messaging platforms). If you need some background on NSQ please see our original blog post or its follow up, spray some NSQ on it. Intro NSQ's design pushes a lot of responsibility onto client libraries in order to maintain overall cluster robustness and performance. This guide attempts to outline the various responsibilities well-behaved client libraries need to fulfill. Because publishing to nsqd is trivial (just an HTTP POST to the /put endpoint), this document focuses on consumers. By setting these expectations we hope to provide a foundation for achieving consistency across languages for NSQ users. Overview Configuration Discovery (optional) Connection Handling Feature Negotiation Data Flow / Heartbeats Message Handling RDY State Backoff Configuration At a high level, our philosophy with respect to configuration is to design the system to have the flexibility to support different workloads, use sane defaults that run well "out of the box", and minimize the number of dials. A client subscribes to a topic on a channel over a TCP connection to nsqd instance(s). You can only subscribe to one topic per connection so multiple topic consumption needs to be structured accordingly. Using nsqlookupd for discovery is optional so client libraries should support a configuration where a client connects directly to one or more nsqd instances or where it is configured to poll one or more nsqlookupd instances. When a client is configured to poll nsqlookupd the polling int ## Choosing a message queue for Python on Ubuntu on a VPS DevFeed: [Choosing a message queue for Python on Ubuntu on a VPS]() Original publisher: [Read original article]() Author: Graham King Published: 2009-08-10T05:05:13Z Content type: comparison Language: en Sources: [Graham King]() Topics: [Messaging](), [RabbitMQ](), [NumPy](), [Ubuntu](), [Redis]() Tags: [comparison](), [gearman](), [message-queue](), [python](), [queue](), [rabbitmq](), [redis](), [software](), [ubuntu]() ### AI overview A comparison of RabbitMQ, Gearman, Beanstalkd, and Redis as message queues for Python web applications running on Ubuntu VPS environments. It considers Python compatibility, memory usage, reliability, and suitability for background work. ### Source excerpt Queuing up the best message options: A straightforward comparison of popular message queues.