# Kafka Producer Pipeline for Ruby on Rails

DevFeed: [Kafka Producer Pipeline for Ruby on Rails](<https://devfeed.tech/articles/kafka-producer-pipeline-for-ruby-on-rails-1461.md>)

Original publisher: [Read original article](<https://shopify.engineering/kafka-producer-pipeline-for-ruby-on-rails>)

Author: Simon Eskildsen

Published: 2014-07-22T15:28:00Z

Content type: article

Language: en

Sources: [Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering.md>), [Shopify Engineering - Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering-shopify-engineering.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [Containers](<https://devfeed.tech/topics/containers.md>), [Go](<https://devfeed.tech/topics/go.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [container](<https://devfeed.tech/tags/container.md>), [containers](<https://devfeed.tech/tags/containers.md>), [data](<https://devfeed.tech/tags/data.md>), [go](<https://devfeed.tech/tags/go.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [ipc](<https://devfeed.tech/tags/ipc.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [library](<https://devfeed.tech/tags/library.md>), [logs](<https://devfeed.tech/tags/logs.md>), [queue](<https://devfeed.tech/tags/queue.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [shopify](<https://devfeed.tech/tags/shopify.md>)

## AI overview

This article describes how Shopify integrated Kafka with its Ruby on Rails and Go infrastructure. It explains the use of a Go-based Kafka producer and kernel-level SysV message queues to buffer events, support restarts, and tolerate Kafka downtime, followed by the operational challenges introduced by Docker containerization.

## Source excerpt

In the early fall our infrastructure team was considering Kafka, a highly available message bus. We were looking to solve several infrastructure problems that had come up around that time. We were looking for a reliable way to collect event data and send it to our data warehouse. We were considering a more service-oriented architecture, and needed a standardized way of message passing between the components. We were starting to evaluate containerization of Shopify, and were searching for a way to get logs out of containers. We were intrigued by Kafka due to its highly available design. However, Kafka runs on the JVM, and its primary user, LinkedIn, runs a full JVM stack. Shopify is mainly Ruby on Rails and Go, so we had to figure out how to integrate Kafka into our infrastructure.