# Getting Started with Ruby and an AWS Managed Kafka Server

DevFeed: [Getting Started with Ruby and an AWS Managed Kafka Server](<https://devfeed.tech/articles/getting-started-with-ruby-and-an-aws-managed-kafka-server-28123.md>)

Original publisher: [Read original article](<http://fuzzyblog.io/blog/aws/2020/02/12/getting-started-with-ruby-and-an-aws-managed-kafka-server.html>)

Author: Fuzzygroup

Published: 2020-02-12T00:00:00Z

Content type: tutorial

Language: en

Sources: [Scott Johnson](<https://devfeed.tech/sources/scott-johnson.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [VPC](<https://devfeed.tech/topics/vpc.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [python](<https://devfeed.tech/tags/python.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [sqs](<https://devfeed.tech/tags/sqs.md>), [vpc](<https://devfeed.tech/tags/vpc.md>)

## AI overview

This tutorial explains how to create and configure an AWS Managed Kafka cluster and connect to it from Ruby. It also outlines a data pipeline using Twitter, Kafka, Python machine-learning routines, and Amazon SQS, and discusses baseline cluster pricing and networking considerations.

## Source excerpt

I am in the position of building a data pipeline that looks like this: Kafka as the core repository using an AWS Managed Kafka Server A data feed from Twitter feeding into Kafka using the ruby-kafka gem from Zendesk Python Machine Learning routines processing the data in Kafka and outputting it to SQS In this blog post I will walk through creating the Kafka server and then show you the initial configuration to get Ruby to talk to it. Core Assumptions Here are the core assumptions: AWS Managed Kafka Instance in the same VPC as an EC2 Instance Ruby 2.7 Minimal ruby script; just enough to connect and put one thing into a topic AWS Kafka Pricing The baseline pricing for an AWS kafka.m5.large cluster is: 0.21 / hour without factoring in storage. 0.21 / hour means nothing to me so here's the calculation: 0.21 * 24 = $5.04 / day 0.21 * 24 * 7 = $35.28 / week 0.21 * 24 * 30 = $151.20 / month 0.21 * 24 * 365 = $1839.60 / year Whether or not this is a good deal depends on your budget and application but what I can tell you is that the AWS Managed Kafka is phenomenally easy to setup and get started with. Note: Read thru the creation process below because I suspect it has pricing implications. Creating the Kafka Instance The new Managed Kafka service from AWS makes creating a Kafka instance absolutely trivial. You do need to know the VPC id of your instance to make things easier. You want to start here to create a Kafka cluster. Here are screenshots of the entire creation flow along with notes on anything I thought relevant. Naming your Kafka cluster is pretty simple and unless you are an expert I'd go with the standard configuration. AWS sometimes lags behind on their versions of open source stuff so keep that in mind. The standard configuration options are generally pretty good. I might argue that auto.create.topics.enable should be enabled by default but that's a debatable point and really one tied to application needs. The networking options are where you likely will get in