# Publish-subscribe pattern

The publish-subscribe pattern (short: pubsub) is an event handling and messaging pattern where the consumer subscribes to the provider, which in return can then publish a message that can be picked up and processed by each subscribed consumer.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## EP224: MCP vs RAG vs AI Agents

DevFeed: [EP224: MCP vs RAG vs AI Agents](<https://devfeed.tech/articles/ep224-mcp-vs-rag-vs-ai-agents-17986.md>)

Original publisher: [Read original article](<https://blog.bytebytego.com/p/ep224-mcp-vs-rag-vs-ai-agents>)

Author: ByteByteGo

Published: 2026-09-05T15:30:58Z

Content type: article

Language: en

Sources: [ByteByteGo](<https://devfeed.tech/sources/bytebytego.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Model Context Protocol](<https://devfeed.tech/topics/model-context-protocol.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Replication](<https://devfeed.tech/topics/replication.md>)

Tags: [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [rag](<https://devfeed.tech/tags/rag.md>), [replication](<https://devfeed.tech/tags/replication.md>)

### AI overview

This article compares MCP, RAG, and AI agents. It explains MCP as a standard protocol for connecting AI models to external tools and data, RAG as a way to retrieve current information from external sources, and AI agents as systems that perform tasks and make decisions autonomously. It also summarizes several distributed-systems patterns, including replication, sharding, publish-subscribe, circuit breakers, retries, and leader election.

### Source excerpt

An AI agent is kind of an AI system where the agent performs the task autonomously and takes the decisions.

## We turned off Pub/Sub and nobody noticed

DevFeed: [We turned off Pub/Sub and nobody noticed](<https://devfeed.tech/articles/we-turned-off-pub-sub-and-nobody-noticed-12060.md>)

Original publisher: [Read original article](<https://incident.io/blog/we-turned-off-pub-sub-and-nobody-noticed>)

Author: Patrick Hamann; Mike Fisher

Published: 2026-08-11T13:56:40Z

Content type: article

Language: en

Sources: [The incident.io Blog](<https://devfeed.tech/sources/the-incident-io-blog.md>)

Topics: [event driven](<https://devfeed.tech/topics/event-driven.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Apache Pulsar](<https://devfeed.tech/topics/pulsar.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Google](<https://devfeed.tech/topics/google.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [broker](<https://devfeed.tech/tags/broker.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-channel](<https://devfeed.tech/tags/incident-channel.md>), [incident-management](<https://devfeed.tech/tags/incident-management.md>), [incident-response](<https://devfeed.tech/tags/incident-response.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [outage](<https://devfeed.tech/tags/outage.md>), [post-mortem](<https://devfeed.tech/tags/post-mortem.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [slack](<https://devfeed.tech/tags/slack.md>), [slack-incident](<https://devfeed.tech/tags/slack-incident.md>)

### AI overview

This developer article explains how incident.io made its predominantly event-driven platform more resilient by adding a secondary message broker alongside Google Cloud Pub/Sub. It describes the role of message brokers and publish-subscribe processing, the risks of a single point of failure, and the successful production test in which Pub/Sub was turned off without affecting customers.

### Source excerpt

Our entire event-driven platform ran through a single message broker, which made it a single point of failure. So we added a second one. This is the story of building an event load balancer, the queuing theory behind it, and the final chaos test where we turned off Pub/Sub in production and nobody noticed.

## How Discord Fans Out One Message to a Million Users

DevFeed: [How Discord Fans Out One Message to a Million Users](<https://devfeed.tech/articles/how-discord-fans-out-one-message-to-a-million-users-18023.md>)

Original publisher: [Read original article](<https://blog.levelupcoding.com/p/discord-case-study-one-message-million-users>)

Author: Nikki Siapno

Published: 2026-07-07T13:22:26Z

Content type: article

Language: en

Sources: [Level Up Coding System Design Newsletter](<https://devfeed.tech/sources/level-up-coding-system-design-newsletter.md>)

Topics: [Discord](<https://devfeed.tech/topics/discord.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [WebSocket](<https://devfeed.tech/topics/websocket.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [Authorization](<https://devfeed.tech/topics/authorization.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [channel](<https://devfeed.tech/tags/channel.md>), [discord](<https://devfeed.tech/tags/discord.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [permission](<https://devfeed.tech/tags/permission.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scale](<https://devfeed.tech/tags/scale.md>), [server](<https://devfeed.tech/tags/server.md>), [websocket](<https://devfeed.tech/tags/websocket.md>)

### AI overview

This case study explains how Discord fans out a single message to many online users. It describes a real-time backend built around pub/sub, guild routing processes, permission checks, session processes, and WebSocket delivery.

### Source excerpt

A case study in fanout, bottlenecks, and the engineering decisions behind Discord's scale.

## When the sensor starts thinking: SnortML, agentic AI, and the evolving architecture of intrusion detection

DevFeed: [When the sensor starts thinking: SnortML, agentic AI, and the evolving architecture of intrusion detection](<https://devfeed.tech/articles/when-the-sensor-starts-thinking-snortml-agentic-ai-and-the-evolving-architecture-of-intrusion-detection-2187.md>)

Original publisher: [Read original article](<https://stackoverflow.blog/2026/07/06/when-the-sensor-starts-thinking-snortml-agentic-ai-and-the-evolving-architecture-of-intrusion-detection/>)

Author: Samaresh Kumar Singh

Published: 2026-07-06T15:23:34Z

Content type: article

Language: en

Sources: [Stack Overflow Blog](<https://devfeed.tech/sources/stack-overflow-blog.md>)

Topics: [Machine Learning & Artificial Intelligence](<https://devfeed.tech/topics/machine-learning-artificial-intelligence.md>), [Security](<https://devfeed.tech/topics/security.md>), [Cisco Talos](<https://devfeed.tech/topics/cisco-talos.md>), [Network](<https://devfeed.tech/topics/network.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [cc-by-sa](<https://devfeed.tech/tags/cc-by-sa.md>), [cisco-talos](<https://devfeed.tech/tags/cisco-talos.md>), [inference](<https://devfeed.tech/tags/inference.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [network](<https://devfeed.tech/tags/network.md>), [se-stackoverflow](<https://devfeed.tech/tags/se-stackoverflow.md>), [se-tech](<https://devfeed.tech/tags/se-tech.md>), [security](<https://devfeed.tech/tags/security.md>)

### AI overview

The article examines how SnortML adds local machine-learning inference to Snort 3's intrusion-detection pipeline. It contrasts signature-based detection, which provides precise coverage for known exploits, with machine-learning detection aimed at identifying novel or modified attack behavior. It also places SnortML alongside the broader rise of agentic AI in network defense, while treating the two as developments operating at different architectural layers.

### Source excerpt

Signature-based detection has always known what it was looking for. Machine learning and autonomous agents are changing the question entirely, shifting from "does this match a known pattern?" to "does this actually make sense in context?"

## An Interactive Study of Publish/Subscribe Messaging for Event-Driven Backend Systems

DevFeed: [An Interactive Study of Publish/Subscribe Messaging for Event-Driven Backend Systems](<https://devfeed.tech/articles/what-is-pub-sub-17831.md>)

Original publisher: [Read original article](<https://encore.dev/blog/pubsub>)

Author: Ivan Cernja

Published: 2026-04-24T00:00:00Z

Content type: tutorial

Language: en

Sources: [Encore Updates](<https://devfeed.tech/sources/encore-updates.md>)

Topics: [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>)

Tags: [decoupling](<https://devfeed.tech/tags/decoupling.md>), [developers](<https://devfeed.tech/tags/developers.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [events](<https://devfeed.tech/tags/events.md>), [flow](<https://devfeed.tech/tags/flow.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [publisher](<https://devfeed.tech/tags/publisher.md>), [subscriber](<https://devfeed.tech/tags/subscriber.md>)

### AI overview

An interactive tutorial explains publish/subscribe messaging by contrasting it with direct service calls. It shows how topics decouple publishers from subscribers, support fan-out, and let subscribers process messages independently, including when downstream services are unavailable.

### Source excerpt

An interactive study of publish/subscribe messaging

## Heroku Eventing: A Router for All Your Events

DevFeed: [Heroku Eventing: A Router for All Your Events](<https://devfeed.tech/articles/heroku-eventing-a-router-for-all-your-events-26429.md>)

Original publisher: [Read original article](<https://www.heroku.com/blog/heroku-eventing-router-for-your-events/>)

Author: Kim Harrison

Published: 2025-03-04T15:01:00Z

Content type: release

Language: en

Sources: [Heroku](<https://devfeed.tech/sources/heroku.md>)

Topics: [Heroku](<https://devfeed.tech/topics/heroku.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Authentication](<https://devfeed.tech/topics/authentication.md>), [API](<https://devfeed.tech/topics/api.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [app-architecture](<https://devfeed.tech/tags/app-architecture.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [events](<https://devfeed.tech/tags/events.md>), [heroku](<https://devfeed.tech/tags/heroku.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [news](<https://devfeed.tech/tags/news.md>), [observability](<https://devfeed.tech/tags/observability.md>), [performance-optimization](<https://devfeed.tech/tags/performance-optimization.md>), [product-features](<https://devfeed.tech/tags/product-features.md>)

### AI overview

Heroku introduces Heroku Eventing, a pilot tool for managing, subscribing to, and publishing events across services. It provides a unified API, centralized event management, integrations with Heroku Kafka and Postgres, secure credential storage, and monitoring across connected sources.

### Source excerpt

Managing event-driven architecture can be challenging. For many organizations, this includes a diverse set of eventing services and buses, often across multiple organizations. Developers must manage authentication and pub/sub services across teams and applications. We're thrilled to introduce Heroku Eventing, a powerful tool designed to help teams manage events more efficiently and securely. This new [...] The post Heroku Eventing: A Router for All Your Events appeared first on Heroku.

## Repost: The Real LISP Mobility Use Case

DevFeed: [Repost: The Real LISP Mobility Use Case](<https://devfeed.tech/articles/repost-the-real-lisp-mobility-use-case-10989.md>)

Original publisher: [Read original article](<https://blog.ipspace.net/2024/04/repost-lisp-mobility/>)

Published: 2024-04-30T05:59:00Z

Content type: article

Language: en

Sources: [ipSpace.net blog](<https://devfeed.tech/sources/ipspace-net-blog.md>)

Topics: [Lisp](<https://devfeed.tech/topics/lisp.md>), [BGP](<https://devfeed.tech/topics/bgp.md>), [networking](<https://devfeed.tech/topics/networking.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [bgp](<https://devfeed.tech/tags/bgp.md>), [lisp](<https://devfeed.tech/tags/lisp.md>), [mobility](<https://devfeed.tech/tags/mobility.md>), [networks](<https://devfeed.tech/tags/networks.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

The article discusses using LISP with PUBSUB for multilink mobility in ground-to-airplane networks. It argues that LISP may handle frequent mobility events, scale, and safety-critical requirements better than BGP in this use case, while noting that several protocol extensions and standardized reliable transport are still needed.

### Source excerpt

Béla Várkonyi is working on an interesting challenge: building ground-to-airplane(s) networks providing multilink mobility. Due to its relative simplicity, he claims LISP works much better than BGP in that environment. In some newer routers BGP would not be such a big bottleneck, but you need a lot of knob turning in BGP to get it right, while in LISP it is quite simple. If you have many thousands concurrent airplanes with multi-link and max. 16 subnets with different routing policies on each, and the radio links are going up and down, then you have a large number of mobility events. Read more ...

## Build a Real-time Materialized View from Postgres Changes using Confluent's ksqlDB

DevFeed: [Build a Real-time Materialized View from Postgres Changes using Confluent's ksqlDB](<https://devfeed.tech/articles/build-a-real-time-materialized-view-from-postgres-changes-using-confluent-s-ksqldb-5763.md>)

Original publisher: [Read original article](<https://neon.com/blog/real-time-materialized-view-postgres-kafka-ksqldb>)

Author: Evan Shortiss

Published: 2024-02-28T17:33:42Z

Content type: tutorial

Language: en

Sources: [Blog -- Neon Docs](<https://devfeed.tech/sources/blog-neon-docs.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Replication](<https://devfeed.tech/topics/replication.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Microservice](<https://devfeed.tech/topics/microservice.md>), [data](<https://devfeed.tech/topics/data.md>), [Databases](<https://devfeed.tech/topics/databases.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [ide](<https://devfeed.tech/topics/ide.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [community](<https://devfeed.tech/tags/community.md>), [components](<https://devfeed.tech/tags/components.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [events](<https://devfeed.tech/tags/events.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [performance](<https://devfeed.tech/tags/performance.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [replication](<https://devfeed.tech/tags/replication.md>), [sql](<https://devfeed.tech/tags/sql.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

This tutorial explains how to stream changes from a Neon Postgres database to Apache Kafka and use ksqlDB on Confluent Cloud to build a real-time materialized view. It contrasts this approach with refreshing a Postgres materialized view after each write, highlighting performance, history retention, and reliable downstream notifications.

### Source excerpt

Neon's support for Postgres' logical replication features opens up a variety of interesting use cases, including for real-time streaming architectures based on change data capture. We previously demonstrated how to use Debezium to fan-out changes from Postgres by using Redis as a...

## Fan-out from Postgres with Change Data Capture using Debezium and Upstash Redis

DevFeed: [Fan-out from Postgres with Change Data Capture using Debezium and Upstash Redis](<https://devfeed.tech/articles/fan-out-from-postgres-with-change-data-capture-using-debezium-and-upstash-redis-5270.md>)

Original publisher: [Read original article](<https://neon.com/blog/fan-out-postgres-changes-using-debezium-and-upstash-redis>)

Author: Evan Shortiss

Published: 2024-02-14T23:32:41Z

Content type: tutorial

Language: en

Sources: [Blog -- Neon Docs](<https://devfeed.tech/sources/blog-neon-docs.md>)

Topics: [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [community](<https://devfeed.tech/tags/community.md>), [database](<https://devfeed.tech/tags/database.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [redis](<https://devfeed.tech/tags/redis.md>), [replication](<https://devfeed.tech/tags/replication.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

A tutorial on using Postgres logical replication and change data capture with Debezium Server and Upstash Redis Streams to implement fan-out messaging with at-least-once delivery.

### Source excerpt

Neon now has beta support for Postgres Logical Replication. This enables teams to use Change Data Capture (CDC) to observe database changes - such as INSERT and UPDATE operations - and stream these changes to downstream systems. We previously wrote about the benefits of CDC and h...

## Using Stand-by Servers for Postgres Logical Replication

DevFeed: [Using Stand-by Servers for Postgres Logical Replication](<https://devfeed.tech/articles/using-stand-by-servers-for-postgres-logical-replication-18871.md>)

Original publisher: [Read original article](<https://www.morling.dev/blog/stand-by-servers-for-postgres-logical-replication/>)

Published: 2023-12-19T00:00:00Z

Content type: tutorial

Language: en

Sources: [Gunnar Morling](<https://devfeed.tech/sources/gunnar-morling.md>)

Topics: [Replication](<https://devfeed.tech/topics/replication.md>), [Database](<https://devfeed.tech/topics/database.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [postgres](<https://devfeed.tech/tags/postgres.md>), [replication](<https://devfeed.tech/tags/replication.md>), [servers](<https://devfeed.tech/tags/servers.md>)

### AI overview

This tutorial explains Postgres logical replication from stand-by servers, including how replication supports high availability and load distribution. It contrasts logical replication with streaming replication and introduces the publish/subscribe model.

### Source excerpt

Table of Contents What is Postgres Logical Replication? Why Logical Replication On Stand-By Servers? Provisioning a Testing Environment Testing Things Out This post originally appeared on the Decodable blog. All rights reserved. For users of Change Data Capture (CDC), one of the most exciting features in Postgres version 16 (released in September this year) is the support for logical replication from stand-by servers. Instead of connecting to your primary server, you can now point CDC tools such as Debezium to a replica server, which is very interesting for instance from a load distribution perspective. I am going to take a closer look at this new feature in this two-part blog series:

## Replay replay: Durable Execution, the way forward for event-driven architectures

DevFeed: [Replay replay: Durable Execution, the way forward for event-driven architectures](<https://devfeed.tech/articles/replay-replay-durable-execution-the-way-forward-for-event-driven-architectures-35959.md>)

Original publisher: [Read original article](<https://temporal.io/blog/replay-replay-durable-execution-the-way-forward-for-event-driven>)

Author: Jim Walker

Published: 2023-10-24T06:00:00Z

Content type: opinion

Language: en

Sources: [Temporal Blog](<https://devfeed.tech/sources/temporal-blog.md>)

Topics: [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Developer experience](<https://devfeed.tech/topics/developer-experience.md>), [backend-development](<https://devfeed.tech/topics/backend-development.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [backend](<https://devfeed.tech/tags/backend.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [community](<https://devfeed.tech/tags/community.md>), [developer-experience](<https://devfeed.tech/tags/developer-experience.md>), [resilience](<https://devfeed.tech/tags/resilience.md>)

### AI overview

A Replay 2023 keynote discusses using Temporal and Durable Execution to simplify event-driven architectures. The talk explains how an execution-centered design can reduce complexity and improve the developer experience while supporting scalable and resilient applications.

### Source excerpt

At Replay 2023, our CEO and co-founder, Max provided an enlightening keynote talk that outlined how Durable Execution is the future of event-driven architectures.

## Mexico City Rails Meet-up Reboot

DevFeed: [Mexico City Rails Meet-up Reboot](<https://devfeed.tech/articles/mexico-city-rails-meet-up-reboot-20036.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/mexico-city-rails-meet-up-reboot>)

Author: Doximity

Published: 2022-11-01T10:39:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [Ruby on Rails](<https://devfeed.tech/topics/ruby-on-rails.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Microservice](<https://devfeed.tech/topics/microservice.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Bootcamp](<https://devfeed.tech/topics/bootcamp.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [developers](<https://devfeed.tech/tags/developers.md>), [development](<https://devfeed.tech/tags/development.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [rails](<https://devfeed.tech/tags/rails.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [services](<https://devfeed.tech/tags/services.md>)

### AI overview

This article describes the restart of the Mexico on Rails meetup in Mexico City after pandemic-related disruption. It covers the event, its attendees, a presentation on syncing data across Rails microservices, Kafka as publish-subscribe middleware, and discussion about software education and in-person community.

### Source excerpt

In recent years Mexico City has developed a thriving tech scene, with new startups putting roots, vying for talent, and contributing to a slew of events and conferences in the city. Mexico on Rails meetup has been an integral part of the ecosystem, meeting regularly to connect and discuss Ruby on Rails and web development. But the pandemic put most of this activity on hold. While discussing this recently with my Doximity colleague Samus Gray, who also lives in Mexico City, he suggested we try and restart the meetup activity. I nodded along, as something that would be nice to do. A few days later, Samus had already convinced the previous meetup organizer to transfer the group to us, found a venue, and got Doximity to sponsor it. That's when I realized he meant it. Last week we met in Le Wagon, a Rails Bootcamp in the fashionable Roma Norte neighborhood. Despite a rainy day, about 20 people showed up. This was a diverse crowd of seasoned developers working for local startups such as Telos Labs and Jetty, Bootcamp students, and digital nomads working remotely. After breaking the ice over Detroit-style pizza and beers, we moved to a classroom, where I gave a presentation on data syncing between Rails microservices. Presented originally by our colleague Austin Story in RailsConf 2021, it tracked Doximity's thorny path from a monolith app with a limited number of data sources to an architecture in which multiple rails applications and services can consume and sync over billions of data updates from a myriad of sources. During the Q&A session, Samus shared his experience working with Kafka, Doximity's tool of choice for our publish-subscribe middleware. Finally, Le Wagon co-founder Mathiew Le Roux took the floor to share his vision for the future of software education. Eager to connect, most of us stayed long after the formal part was over, chatting and sharing our experiences. The enthusiasm shared by everyone for the meetup reboot was a reminder of the importance of meet

## Micro Service Events (AWS EventBridge vs. SNS)

DevFeed: [Micro Service Events (AWS EventBridge vs. SNS)](<https://devfeed.tech/articles/micro-service-events-aws-eventbridge-vs-sns-23899.md>)

Original publisher: [Read original article](<https://medium.com/smg-real-estate/micro-service-events-aws-eventbridge-vs-sns-2d86b5919c22?source=rss----2186e5b9bd8f---4>)

Author: Kristian Müller

Published: 2022-09-16T12:06:51Z

Content type: comparison

Language: en

Sources: [Homegate Engineering Blog - Medium](<https://devfeed.tech/sources/homegate-engineering-blog-medium.md>)

Topics: [event driven](<https://devfeed.tech/topics/event-driven.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [AWS Lambda](<https://devfeed.tech/topics/aws-lambda.md>), [Amazon API Gateway](<https://devfeed.tech/topics/amazon-api-gateway.md>), [DynamoDB](<https://devfeed.tech/topics/dynamodb.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [TypeScript](<https://devfeed.tech/topics/typescript.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [api-gateway](<https://devfeed.tech/tags/api-gateway.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-eventbridge](<https://devfeed.tech/tags/aws-eventbridge.md>), [aws-sns](<https://devfeed.tech/tags/aws-sns.md>), [data](<https://devfeed.tech/tags/data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [database](<https://devfeed.tech/tags/database.md>), [dynamodb](<https://devfeed.tech/tags/dynamodb.md>), [event](<https://devfeed.tech/tags/event.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [eventbridge](<https://devfeed.tech/tags/eventbridge.md>), [eventbus](<https://devfeed.tech/tags/eventbus.md>), [events](<https://devfeed.tech/tags/events.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [lambda](<https://devfeed.tech/tags/lambda.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [microservice-architecture](<https://devfeed.tech/tags/microservice-architecture.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sns](<https://devfeed.tech/tags/sns.md>), [sqs](<https://devfeed.tech/tags/sqs.md>), [typescript](<https://devfeed.tech/tags/typescript.md>)

### AI overview

This article examines Homegate's event-driven microservice architecture on AWS and compares Amazon SNS with EventBridge as messaging infrastructure. It describes how services use SNS, SQS, Lambda, API Gateway, DynamoDB, and S3, and explains that EventBridge has a different focus, so replacing SNS is not necessarily the right question.

### Source excerpt

During the last few years, we've been busy rebuilding the Homegate platform in an event driven, micro service architecture based on AWS. One of the most important buildings blocks, or shall we rather say the mortar that connects the building blocks, is the way our services send events to each other. While Amazon offers its notification service SNS since more than 12 years; in 2019 it started a newcomer in the messaging bus field by introducing EventBridge. This was very interesting to us as we're using SNS a lot. Bus on a Bridge (Photo by "Blue Elf" on Wikimedia Commons) As EventBridge was introduced and gained traction we asked ourselves what advantaged we would gain by switching our SNS based service glue to EventBridge. Maybe this turned out to be the wrong question for us. Let's look at our system to see why. Micro Services Connected by SNS Homegate is connecting dozens of services via SNS right now. Those services range from single purpose Lambda functions written in TypeScript to database heavy services using Aurora or DynamoDB and Step Functions sending events to other services. Some of the services are also interfacing external APIs such as Salesforce or SendGrid. Here our Lambda functions can call APIs when needed. On the other hand, we have web services to be called as APIs by other systems and end users. This is mostly achieved via API Gateway. A typical service connects to the system by subscribing to events needed to fulfill its purpose. Those inbound SNS events are then ingested via SQS allowing for orderly data processing and DLQs to handle errors. Outgoing results are pushed back to the system in form of events to a SNS topics. As Marko Savic has pointed out, we're also utilizing S3 to store and replay those events if needed. Basic AWS Lambda function connected via SNS If we would replace the SNS topic by streams on the EventBridge we would initially gain the same functionality by utilizing SQS as well. However, EventBridge has a different focus. Let

## AWS FIFO Queues with Message Groups for Atomic Processing at Scale

DevFeed: [AWS FIFO Queues with Message Groups for Atomic Processing at Scale](<https://devfeed.tech/articles/aws-fifo-queues-with-message-groups-for-atomic-processing-at-scale-23895.md>)

Original publisher: [Read original article](<https://medium.com/smg-real-estate/aws-fifo-queues-with-message-groups-for-atomic-processing-at-scale-dc70f8820b16?source=rss----2186e5b9bd8f---4>)

Author: Nick De Cooman

Published: 2022-03-03T09:55:58Z

Content type: tutorial

Language: en

Sources: [Homegate Engineering Blog - Medium](<https://devfeed.tech/sources/homegate-engineering-blog-medium.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [decoupling](<https://devfeed.tech/tags/decoupling.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [event](<https://devfeed.tech/tags/event.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [events](<https://devfeed.tech/tags/events.md>), [fifo](<https://devfeed.tech/tags/fifo.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [sns](<https://devfeed.tech/tags/sns.md>), [sqs](<https://devfeed.tech/tags/sqs.md>)

### AI overview

A tutorial on using AWS FIFO queues and message groups to process events for multiple objects in parallel while ensuring that only one instance processes events for the same object at a time. It presents a serverless architecture using SNS, SQS, and Lambda for atomic listing ingestion.

### Source excerpt

How to process multiple objects in parallel with the guarantee that for each unique object, only a single instance is processed simultaneously In a distributed, event-driven architecture, many challenges arise when processing large sets of data. One of them is the need for scalability when handling event streams with a substantial throughput. The challenge becomes even more complicated when processing these events involves atomic operations. In this case, we cannot simply process multiple events in parallel. As an example, let's say that we are building a service that ingests listings into our platform. The ingestion process involves several steps and requires multiple API operations. As a result, the whole ingestion can take up to several seconds per listing to complete. Now, here is where it gets tricky: a constraint of our service is that, while we are processing a listing, we cannot process other events for the same listing. The ingestion should be considered an atomic operation, and hence, only a single execution per listing can be performed simultaneously. At Homegate, we run most of our infrastructure on AWS in a serverless fashion. At the core of this, Lambda functions execute application logic. A common pattern is to use SNS topics to fan-out messages, and SQS for acting as a decoupling buffer between an SNS topic and some Lambda function. For our ingester service, a logical architecture could look like this: Listing events are published to a dedicated SNS topic. An SQS queue subscribes to this topic and consumes the events. Here, the events are buffered until a Lambda instance removes them from the queue and processes the corresponding listing. Under normal circumstances -- in which the processing would not involve an atomic operation -- this architecture would be a good fit. It would scale automatically relative to the number of Lambda instances that consume events from the queue, without much additional configuration. However, in the case where we cannot p

## Tiny Letter from Kafka

DevFeed: [Tiny Letter from Kafka](<https://devfeed.tech/articles/tiny-letter-from-kafka-2169.md>)

Original publisher: [Read original article](<https://developers.soundcloud.com/blog//tiny-letter-from-kafka>)

Published: 2021-10-13T00:00:00Z

Content type: article

Language: en

Sources: [SoundCloud Backstage Blog](<https://devfeed.tech/sources/soundcloud-backstage-blog.md>)

Topics: [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [article](<https://devfeed.tech/tags/article.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [event](<https://devfeed.tech/tags/event.md>), [event-streaming](<https://devfeed.tech/tags/event-streaming.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [messaging](<https://devfeed.tech/tags/messaging.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

This article introduces Apache Kafka as an open-source distributed event streaming and messaging platform. It explains the publish-subscribe messaging pattern and uses the story behind Kafka's name to illustrate the system's core flow, with a brief reference to how SoundCloud teams consume event data.

### Source excerpt

This article discusses the powerful design choice of Apache Kafka, "an open-source distributed event streaming platform," and gives a sneak...

## Effective batching in streaming dataflow jobs

DevFeed: [Effective batching in streaming dataflow jobs](<https://devfeed.tech/articles/effective-batching-in-streaming-dataflow-jobs-15619.md>)

Original publisher: [Read original article](<https://developer.squareup.com/blog/effective-batching-in-streaming-dataflow-jobs>)

Author: Fasih Awan

Published: 2019-07-15T11:00:00Z

Content type: article

Language: en

Sources: [Square Corner Blog RSS Feed](<https://devfeed.tech/sources/square-corner-blog-rss-feed.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [Google](<https://devfeed.tech/topics/google.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [batching](<https://devfeed.tech/tags/batching.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [payload](<https://devfeed.tech/tags/payload.md>), [payments](<https://devfeed.tech/tags/payments.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

Square describes how it batches payment messages in a Google Dataflow job to produce near-real-time monitoring metrics. The pipeline consumes PubSub messages, applies five-second windows, aggregates payments by dimensions, and sends metrics to SignalFX while accounting for API restrictions.

### Source excerpt

How we make near real time monitoring possible

## Scheduling Cloud Functions for Firebase (cron)

DevFeed: [Scheduling Cloud Functions for Firebase (cron)](<https://devfeed.tech/articles/scheduling-cloud-functions-for-firebase-cron-16316.md>)

Original publisher: [Read original article](<https://firebase.blog/posts/2019/04/schedule-cloud-functions-firebase-cron>)

Author: Doug Stevenson

Published: 2019-04-18T00:00:00Z

Content type: tutorial

Language: en

Sources: [Firebase Blog](<https://devfeed.tech/sources/firebase-blog.md>)

Topics: [Firebase](<https://devfeed.tech/topics/firebase.md>), [Cloud Functions](<https://devfeed.tech/topics/cloud-functions.md>), [Firebase CLI](<https://devfeed.tech/topics/firebase-cli.md>), [SDKs](<https://devfeed.tech/topics/sdks.md>), [Google](<https://devfeed.tech/topics/google.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>)

Tags: [cli](<https://devfeed.tech/tags/cli.md>), [cloud-functions](<https://devfeed.tech/tags/cloud-functions.md>), [cloud-pub-sub](<https://devfeed.tech/tags/cloud-pub-sub.md>), [firebase](<https://devfeed.tech/tags/firebase.md>), [firebase-cli](<https://devfeed.tech/tags/firebase-cli.md>), [github](<https://devfeed.tech/tags/github.md>), [launch](<https://devfeed.tech/tags/launch.md>), [payment](<https://devfeed.tech/tags/payment.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [sdks](<https://devfeed.tech/tags/sdks.md>), [tutorials](<https://devfeed.tech/tags/tutorials.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

Firebase introduces scheduled Pub/Sub functions built on Cloud Scheduler, allowing developers to schedule Cloud Functions for Firebase using Firebase tools and SDKs. The article covers cron syntax, setup requirements, billing, and costs.

### Source excerpt

News, tutorials, and updates from the Firebase team.

## Kafka, a walking skeleton

DevFeed: [Kafka, a walking skeleton](<https://devfeed.tech/articles/kafka-a-walking-skeleton-26067.md>)

Original publisher: [Read original article](<https://blog.arkey.fr/2017/04/29/kafka-a-walking-skeleton/>)

Author: brice.dutheil@gmail.com (Brice Dutheil)

Published: 2017-04-29T00:00:00Z

Content type: article

Language: fr

Sources: [The Coffee Workshop](<https://devfeed.tech/sources/the-coffee-workshop.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>)

Tags: [article](<https://devfeed.tech/tags/article.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [broker](<https://devfeed.tech/tags/broker.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [messages](<https://devfeed.tech/tags/messages.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [production](<https://devfeed.tech/tags/production.md>), [queuing](<https://devfeed.tech/tags/queuing.md>)

### AI overview

This French article introduces Apache Kafka through a minimal end-to-end implementation, based on Kafka 0.9. It explains Kafka's origins at LinkedIn and describes its performance, durable message retention, scalability, fault resilience, and support for queuing and publish-subscribe messaging.

### Source excerpt

Ce blog post est une republication de mon article sur Kafka parut dans le magazine Programmez numéro 196 en mai 2016 (pages 70-73). Il se base sur la version 0.9 de Kafka.

## Can HTTP/2 Replace MQTT?

DevFeed: [Can HTTP/2 Replace MQTT?](<https://devfeed.tech/articles/can-http-2-replace-mqtt-33410.md>)

Original publisher: [Read original article](<https://timkellogg.me/blog/2015/02/20/can-http2-replace-mqtt>)

Published: 2015-02-20T00:00:00Z

Content type: opinion

Language: en

Sources: [Tim Kellogg](<https://devfeed.tech/sources/tim-kellogg.md>)

Topics: [HTTP](<https://devfeed.tech/topics/http.md>), [MQTT](<https://devfeed.tech/topics/mqtt.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [client](<https://devfeed.tech/topics/client.md>)

Tags: [client](<https://devfeed.tech/tags/client.md>), [http](<https://devfeed.tech/tags/http.md>), [messages](<https://devfeed.tech/tags/messages.md>), [mqtt](<https://devfeed.tech/tags/mqtt.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [server](<https://devfeed.tech/tags/server.md>), [stream](<https://devfeed.tech/tags/stream.md>), [subscription](<https://devfeed.tech/tags/subscription.md>)

### AI overview

The article examines whether HTTP/2 with HPACK could replace MQTT. It explains MQTT's compact headers and publish/subscribe model, then describes how HTTP/2 multiplexing, server push, persistent streams, and headers could approximate MQTT-style messaging.

### Source excerpt

Yesterday I got an interesting question:

## Finn.no shares eventHub and Awesome Board repositories on GitHub

DevFeed: [Finn.no shares eventHub and Awesome Board repositories on GitHub](<https://devfeed.tech/articles/spreading-love-and-good-vibrations-on-github-31961.md>)

Original publisher: [Read original article](<https://tech.finn.no2011/11/21/spreading-love-and-good-vibrations-on-github/>)

Author: espen

Published: 2011-11-21T19:02:40Z

Content type: release

Language: en

Sources: [Finn.no](<https://devfeed.tech/sources/finn-no.md>)

Topics: [GitHub](<https://devfeed.tech/topics/github.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [jQuery](<https://devfeed.tech/topics/jquery.md>), [Document Object Model (DOM)](<https://devfeed.tech/topics/dom.md>), [App](<https://devfeed.tech/topics/app.md>)

Tags: [awesome](<https://devfeed.tech/tags/awesome.md>), [code](<https://devfeed.tech/tags/code.md>), [event](<https://devfeed.tech/tags/event.md>), [github](<https://devfeed.tech/tags/github.md>), [libraries](<https://devfeed.tech/tags/libraries.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [repositories](<https://devfeed.tech/tags/repositories.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

Finn.no announces that it has shared the eventHub and Awesome Board repositories on GitHub under the finn-no organization. eventHub uses the publish-subscribe pattern for loosely coupled components and applications, while Awesome Board supports quality visualization. The shared versions are incomplete compared with the internal version.

### Source excerpt

We have finally made an effort to put some of our repos out on GitHub under the finn-no organization. It is not exactly filled with stuff right now, but we aim to put a lot more of our libraries and tools out here. Currently there are a few repositories out there: eventHub Awesome Board[Gone] eventHub This is just a tiny piece of code which enables you to build loosely coupled components or applications by utilizing the publish-subscribe pattern. The eventHub is just an object which publishes event to those to registered event listeners functions. We have written one-page JS applications using only the hub besides jQuery for DOM manipulation. It scales pretty well and it is pretty much what you'd need to create an application. Once your application grows to be quite large you might want to add some other architectural components, but for small to medium sized projects it is brilliant. It is used for the dashboard part for the Awesome Board[Gone]. Awesome Board The Awesome Board[Gone] is the application which has featured in two articles about visualizing quality. We have shown it a few times to different audiences and due to request for making it available we decided it was a good thing to share it on GitHub. There are some pieces missing from the version we run internally, but have patience the rest might still make it to a repo near you. Enjoy! Feel free to drop us a line in the comments section or follow us on Twitter.