# Sidekiq

Published articles for Sidekiq.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Why Gusto hired Evil Martians for Sidekiq infrastructure

DevFeed: [Why Gusto hired Evil Martians for Sidekiq infrastructure](<https://devfeed.tech/articles/why-gusto-hired-evil-martians-for-sidekiq-infrastructure-19795.md>)

Original publisher: [Read original article](<https://evilmartians.com/chronicles/why-gusto-hired-evil-martians-for-sidekiq-infrastructure>)

Author: Irina Nazarova (inazarova@evilmartians.com)

Published: 2026-09-02T00:00:00Z

Content type: article

Language: en

Sources: [Evil Martians](<https://devfeed.tech/sources/evil-martians.md>)

Topics: [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [rails](<https://devfeed.tech/tags/rails.md>), [redis](<https://devfeed.tech/tags/redis.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [scale](<https://devfeed.tech/tags/scale.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

This case study examines why Gusto hired Evil Martians to work on the Sidekiq and Redis infrastructure supporting background jobs in its large Rails monolith. It discusses the operational demands of running payroll-related jobs at scale and the tradeoffs involved when a company uses consultants instead of waiting for a perfect hire.

### Source excerpt

Gusto runs payroll for 500,000+ businesses on one of the largest Rails monoliths anywhere. Why a team this strong hired Evil Martians for Sidekiq at scale, what running background jobs at that size actually takes, and when to stop waiting for the perfect hire.

## Fair by design: orchestrating background jobs in Ruby

DevFeed: [Fair by design: orchestrating background jobs in Ruby](<https://devfeed.tech/articles/fair-by-design-orchestrating-background-jobs-in-ruby-19782.md>)

Original publisher: [Read original article](<https://evilmartians.com/chronicles/fair-by-design-orchestrating-background-jobs-in-ruby>)

Author: Travis Turner (richardturner@evilmartians.com)

Published: 2026-08-11T00:00:00Z

Content type: tutorial

Language: en

Sources: [Evil Martians](<https://devfeed.tech/sources/evil-martians.md>)

Topics: [Ruby](<https://devfeed.tech/topics/ruby.md>), [jobs](<https://devfeed.tech/topics/jobs.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [autoscaling](<https://devfeed.tech/topics/autoscaling.md>)

Tags: [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [latency](<https://devfeed.tech/tags/latency.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rails](<https://devfeed.tech/tags/rails.md>), [redis](<https://devfeed.tech/tags/redis.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

This tutorial examines fairness in Ruby background-job processing. It explains how queue latency affects quality of service, why adding workers or autoscaling may be limited by shared resources and operational cost, and introduces background-job prioritization as a way to address bottlenecks.

### Source excerpt

Are you treating your users fairly? They could be stuck in the queue while a greedy user monopolizes resources. And you might not even know it! In this post, you'll see if it's time for you to take background job prioritization seriously, and how to make it fair for all users.

## A Pre-Deployment Checklist for Risky Data Migrations

DevFeed: [A Pre-Deployment Checklist for Risky Data Migrations](<https://devfeed.tech/articles/the-checklist-for-deploying-a-scary-change-19165.md>)

Original publisher: [Read original article](<https://artsy.github.io/blog/2023/09/13/deploying-a-scary-change/>)

Published: 2023-09-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Artsy](<https://devfeed.tech/sources/artsy.md>)

Topics: [Command-line interface](<https://devfeed.tech/topics/cli.md>), [Docker](<https://devfeed.tech/topics/docker.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [command-line](<https://devfeed.tech/tags/command-line.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [deploy-process](<https://devfeed.tech/tags/deploy-process.md>), [docker](<https://devfeed.tech/tags/docker.md>), [error-reporting](<https://devfeed.tech/tags/error-reporting.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [latency](<https://devfeed.tech/tags/latency.md>), [logs](<https://devfeed.tech/tags/logs.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [retry](<https://devfeed.tech/tags/retry.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

This article presents a pre-deployment checklist for risky data migrations and other changes that could affect production services. It recommends planning rollback steps and monitoring error rates, reported errors, latency, logs, functionality, and failing Sidekiq retries after deployment.

### Source excerpt

Lately, I've been getting involved with some sketchy stuff. You know what I'm talking about-data migrations. I've been rolling out changes that have a significant risk of breaking our production environment for mission-critical services. It's been exciting work (keep your eyes out for more posts on the exact project, coming soon™), but I've definitely caused a couple incidents along the way. After accidentally taking down a key service for a couple hours, I decided I needed to have a better pre-deploy process for these changes. I did some thinking and came up with a short checklist to run through before I press the shiny green button. Here's the checklist I came up with: What is your plan if something goes wrong? Run through ramifications of rolling back. If there's a reason you're worried about rolling back, then you're not ready to deploy the change yet! Figure out exactly what command(s) you will need to run to roll back. At Artsy, this is usually a one-liner using Hokusai, our command-line Docker/Kubernetes CLI How will you tell if something is going wrong after you deploy? Error rate (DataDog) Specific error reporting (Sentry) Latency (DataDog) Logs (Papertrail) Functionality (does it still work? Are people using it successfully? Important for things where errors may not be bubbled up correctly or reported immediately) Sidekiq (are there lots of jobs queued to retry that are failing?) With this checklist in hand, I'm deploying more confidently and causing fewer incidents along the way. Do you have something similar? Are there things you think this checklist should include? Let me know in the comments!

## Introducing Simplekiq: A Lightweight Orchestration Framework for Ruby

DevFeed: [Introducing Simplekiq: A Lightweight Orchestration Framework for Ruby](<https://devfeed.tech/articles/introducing-simplekiq-a-lightweight-orchestration-framework-for-ruby-20033.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/introducing-simplekiq-a-lightweight-orchestration-framework-for-ruby>)

Author: Doximity

Published: 2023-05-03T11:28:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [Framework](<https://devfeed.tech/topics/framework.md>)

Tags: [declarative](<https://devfeed.tech/tags/declarative.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [refactoring](<https://devfeed.tech/tags/refactoring.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

The article introduces Simplekiq, a lightweight Ruby workflow framework built on Sidekiq Pro batching. It presents declarative, single-file orchestrations with serial and parallel execution as a way to make complex background-job workflows easier to understand, diagnose, and refactor.

### Source excerpt

Sidekiq is an amazing background job framework for Ruby with a long history of the best kind of minimalism - one of performance and succinctness. However, a downside of this is that long workflows built from dozens of background jobs queueing subsequent background jobs can be difficult to consistently design and maintain. We especially struggled with refactoring these types of structures since one has to read through all the code for the jobs to understand when and from where each job might get queued. The added complexity of manually defining Sidekiq Pro batches leads to difficult-to-diagnose triage in more creative workflows, especially when batch callbacks are involved. Introducing Simplekiq: A Solution to Our Challenges We solved our issue by building Simplekiq (special thanks to Daniel Pepper for freeing the rubygems simplekiq name for us) --a lightweight background job workflow framework built on top of Sidekiq Pro batching. Simplekiq solves the issue of having to follow long chains of jobs that queue other jobs ad nauseam. Workflows that span many files and job classes are inherently difficult to understand, diagnose and refactor. Simplekiq flattens out these complex, multi-file workflows into declarative, single-file ones which we call orchestrations. Each orchestration can contain a mix of serial and parallel execution so that even the most complex workflow can be expressed in an efficient, readable syntax. Comparing Sidekiq Pro and Simplekiq with an Example It's easier to demonstrate this with an example: making an apple pie. The late, great Carl Sagan taught us that to make an apple pie from scratch we must first invent the universe. That's a bit out of budget for most bakeries, but in the spirit of simplicity let's explore how a workflow for baking an apple pie from basic ingredients looks in standard Sidekiq Pro structures and compare it to the flattened Simplekiq paradigm. Our example will also show how refactoring differs between the two. Implementing

## 5 SRE tips for Rails workloads in Kubernetes

DevFeed: [5 SRE tips for Rails workloads in Kubernetes](<https://devfeed.tech/articles/5-sre-tips-for-rails-workloads-in-kubernetes-20020.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/5-sre-tips-for-rails-workloads-in-kubernetes>)

Author: Doximity

Published: 2022-12-02T13:00:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [site-reliability-engineering](<https://devfeed.tech/topics/site-reliability-engineering.md>)

Tags: [bugs](<https://devfeed.tech/tags/bugs.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [deployment-strategies](<https://devfeed.tech/tags/deployment-strategies.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rails](<https://devfeed.tech/tags/rails.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

The article presents five changes made to Kubernetes clusters to better support Rails workloads and workflows. The supplied text focuses on deployment strategies: RollingUpdate is used for web traffic, while Recreate is used for Kafka and Sidekiq workloads because RollingUpdate caused rebalancing, performance, and compatibility issues during deployments.

### Source excerpt

In a previous post we shared some suggestions for how to adjust a Rails application to make it run better in a containerized setting. In this post I'd like to explore the opposite relationship: five changes we have made to our Kubernetes clusters to better accommodate Rails workloads and workflows. All of these ideas can be relevant to non-Rails apps too! Selecting the right deployment strategies The two deployment strategies Kubernetes offers out of the box are RollingUpdate or Recreate. Initially we selected RollingUpdate for all of our workloads; with this strategy some new pods are created before old ones are terminated. The RollingUpdate strategy is ideal for our web traffic because if Kubernetes shuts down all of the existing pods before it starts up any new pods, users will see 5XX errors until the new web pods are healthy, not good. That said, the RollingUpdate strategy created issues with our Sidekiq and Kafka workloads so we switched those to use the Recreate strategy. Looking at Kafka first, consumers are basically just a group of processes working together to handle messages. Kafka consumers follow a rebalancing protocol when a new member wants to join a consumer group. During deployments we encountered bugs and performance issues while using RollingUpdate approach because new pods kept trickling-in and wanting to join the consumer group therefore triggering multiple rebalances. Switching our Kafka consumers to deploy using the Recreate strategy means all the existing Pods get killed before any new pods get created. This results in much more predictable deployment behavior, and now we see only a single rebalance with all new consumers joining the group at basically the same time. Next, looking at Sidekiq, it does not stop workers from processing jobs when a new worker process joins, so we didn't have quite as many issues with using the RollingUpdate approach. However, it did occasionally cause subtle bugs on some deployments. For example, if a team creat

## Square Payroll's Migration from a Monolith to Microservices

DevFeed: [Square Payroll's Migration from a Monolith to Microservices](<https://devfeed.tech/articles/square-payroll-s-migration-from-a-monolith-to-microservices-15879.md>)

Original publisher: [Read original article](<https://developer.squareup.com/blog/square-payrolls-migration-from-a-monolith-to-microservices>)

Author: Harold Sultan

Published: 2022-10-12T04:00:00Z

Content type: article

Language: en

Sources: [Square Corner Blog](<https://devfeed.tech/sources/square-corner-blog-medium.md>), [Square Corner Blog RSS Feed](<https://devfeed.tech/sources/square-corner-blog-rss-feed.md>)

Topics: [Microservices](<https://devfeed.tech/topics/microservices.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [serverless architecture](<https://devfeed.tech/topics/serverless-architecture.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [migration](<https://devfeed.tech/tags/migration.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-architecture](<https://devfeed.tech/tags/serverless-architecture.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

Square describes an early step in migrating Square Payroll from a Ruby monolith backed by MySQL to a more scalable architecture. The article focuses on redesigning its tax filing system with AWS serverless architecture and using a Step Function to reduce code complexity and improve transparency.

### Source excerpt

Scaling with AWS serverless architecture

## To Thread or Not to Thread: An In-Depth Look at Ruby's Execution Models

DevFeed: [To Thread or Not to Thread: An In-Depth Look at Ruby's Execution Models](<https://devfeed.tech/articles/to-thread-or-not-to-thread-an-in-depth-look-at-ruby-s-execution-models-1555.md>)

Original publisher: [Read original article](<https://shopify.engineering/ruby-execution-models>)

Author: Jean Boussier

Published: 2022-05-31T14:00:00Z

Content type: article

Language: en

Sources: [Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering.md>), [Shopify Engineering - Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering-shopify-engineering.md>)

Topics: [Ruby](<https://devfeed.tech/topics/ruby.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [web applications](<https://devfeed.tech/topics/web-applications.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Heroku](<https://devfeed.tech/topics/heroku.md>)

Tags: [heroku](<https://devfeed.tech/tags/heroku.md>), [processes](<https://devfeed.tech/tags/processes.md>), [rails](<https://devfeed.tech/tags/rails.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>), [thread](<https://devfeed.tech/tags/thread.md>), [threads](<https://devfeed.tech/tags/threads.md>), [web-applications](<https://devfeed.tech/tags/web-applications.md>)

### AI overview

This article examines threaded servers and processes in Ruby web applications. It explains why threads are a popular default, compares their throughput and memory characteristics with processes, and discusses static and request-processing memory, garbage collection, and Copy on Write.

### Source excerpt

An in-depth look at threads vs processes in Ruby web applications, and when you should use each.

## Logster and our error logging strategy at Discourse

DevFeed: [Logster and our error logging strategy at Discourse](<https://devfeed.tech/articles/logster-and-our-error-logging-strategy-at-discourse-22431.md>)

Original publisher: [Read original article](<https://samsaffron.com/archive/2018/12/27/logster-and-our-logging-strategy-at-discourse>)

Author: Sam Saffron

Published: 2019-01-02T07:11:20Z

Content type: article

Language: en

Sources: [Sam Saffron](<https://devfeed.tech/sources/sam-saffron.md>)

Topics: [log management](<https://devfeed.tech/topics/log-management.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [Ruby on Rails](<https://devfeed.tech/topics/ruby-on-rails.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [nginx](<https://devfeed.tech/topics/nginx.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Docker](<https://devfeed.tech/topics/docker.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [blog](<https://devfeed.tech/tags/blog.md>), [c-sharp](<https://devfeed.tech/tags/c-sharp.md>), [docker](<https://devfeed.tech/tags/docker.md>), [errors](<https://devfeed.tech/tags/errors.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [net](<https://devfeed.tech/tags/net.md>), [nginx](<https://devfeed.tech/tags/nginx.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [production](<https://devfeed.tech/tags/production.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [rails](<https://devfeed.tech/tags/rails.md>), [redis](<https://devfeed.tech/tags/redis.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [saffron](<https://devfeed.tech/tags/saffron.md>), [sam](<https://devfeed.tech/tags/sam.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>), [sql](<https://devfeed.tech/tags/sql.md>), [visibility](<https://devfeed.tech/tags/visibility.md>), [weblog](<https://devfeed.tech/tags/weblog.md>)

### AI overview

The article describes Discourse's logging strategy and Logster, an open-source log viewer for Ruby on Rails or Rack applications. It explains how Logster stores and analyzes logs with Redis, aggregates production errors, and provides detailed development logs, alongside broader monitoring and observability tools.

### Source excerpt

I have always been somewhat fascinated with logs. I tend to see the warning and error logs in production as a valuable heartbeat of an application. Proper handling of error logs is a very strong complement to a robust test suite. It shows us what really happens when real world data meets our application. 9 years ago, at Stack Overflow we had a daily ritual where we would open up our fork of ELMAH every morning and fish through our logs for problems. This had a dramatic positive effect on Stack Overflow. Almost 7 years into our journey building Discourse, every single week we find and fix issues in our application thanks to our error logs and Logster. Error logs are the pulse of our application, they let us know immediately if there are any urgent issues and where. Since we host more than 1500 sites running many different code branches, we needed to evolve a sane and robust set of practices and tools. Top level structure of logging and monitoring at Discourse We have lots of logs at Discourse and many systems for dealing with them. We keep raw Docker, Postgres, Redis, NGINX, Rails and HAProxy and so on in Elastic Search and use Kibana for business intelligence. We have a monitoring system built on alertmanager and Prometheus, with business intelligence in Grafana and alert escalation in our internal Discourse instance and opsgenie. We have logster which we use for web application aka. "Rails / Sidekiq" warnings and errors. I would like to focus on logster and our Rails / Sidekiq portion for this blog post, but think it is worth mentioning other mechanisms cause I don't want people to think we are not good data hoarders and only have very limited visibility into our systems. About Logster At Discourse we developed a log viewer called logster. Logster is a free and open source tool you can embed into any Ruby on Rails or Rack application in production and development. It runs as Rack middleware and uses Redis as its backend for log storage and analysis. It operates in

## Finding where STDOUT/STDERR debug messages are coming from

DevFeed: [Finding where STDOUT/STDERR debug messages are coming from](<https://devfeed.tech/articles/finding-where-stdout-stderr-debug-messages-are-coming-from-22430.md>)

Original publisher: [Read original article](<https://samsaffron.com/archive/2018/08/07/finding-where-stdout-stderr-debug-messages-are-coming-from>)

Author: Sam Saffron

Published: 2018-08-07T06:06:55Z

Content type: tutorial

Language: en

Sources: [Sam Saffron](<https://devfeed.tech/sources/sam-saffron.md>)

Topics: [debug](<https://devfeed.tech/topics/debug.md>), [Development](<https://devfeed.tech/topics/development.md>), [servers](<https://devfeed.tech/topics/servers.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [c-sharp](<https://devfeed.tech/tags/c-sharp.md>), [console](<https://devfeed.tech/tags/console.md>), [debug](<https://devfeed.tech/tags/debug.md>), [debugger](<https://devfeed.tech/tags/debugger.md>), [development](<https://devfeed.tech/tags/development.md>), [net](<https://devfeed.tech/tags/net.md>), [rails](<https://devfeed.tech/tags/rails.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [saffron](<https://devfeed.tech/tags/saffron.md>), [sam](<https://devfeed.tech/tags/sam.md>), [server](<https://devfeed.tech/tags/server.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>), [sql](<https://devfeed.tech/tags/sql.md>), [weblog](<https://devfeed.tech/tags/weblog.md>)

### AI overview

A debugging tip for tracing unexpected messages printed to STDOUT or STDERR in development. The article shows how to reopen STDERR and attach a method that prints caller locations when write is invoked, helping identify the source of stray output in web servers, test suites, or upgraded gems.

### Source excerpt

Recently, we have been experiencing "stalls" in the Puma web server in development, this means that quite often during our dev cycle we would hit CTRL-C and be stuck waiting many many seconds for Puma to stop. Sometimes needing to fallback to kill -9 on the Puma process. We definitely want this Puma issue fixed, however our "web application server of choice" is Unicorn not Puma. It makes little sense for us to run Puma in development. Our Unicorn configuration is very mature and handles all sorts of magic including automatic forking of our Sidekiq job scheduler which is awesome in dev. A major problem though is that when we run Puma in dev our console is pristine, run Unicorn in dev and it is noise central. 127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/pretty-text-bundle.js?1533620338.6222095 HTTP/1.1" 200 112048 0.0481 127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/plugin.js?1533620338.6222444 HTTP/1.1" 200 146176 0.0726 127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/plugin-third-party.js?1533620338.6222594 HTTP/1.1" 200 3364 0.0569 127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/application.js?1533620338.6222193 HTTP/1.1" 200 3039095 0.2049 127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/fontawesome-webfont.woff2?http://l.discourse&2&v=4.7.0 HTTP/1.1" 304 - 0.0016 I am a puts debugger and being barred from being a puts debugger in development is a blocking feature for me. So, how do we find where these messages are coming from? Before we start the little tip here first... if you have not yet... take a break and read _why's classic seeing metaclasses clearly. Now that you know about metaclasses, time to have some fun, let's reopen STDERR and glue a little debug method to it that will output caller locations when we invoke write on STDERR (note this will work on STDOUT as well if you want): class << STDERR alias_method :orig_write, :write def write(x) orig_write(caller[0..3].join("\n")) orig_write(x) end end /home/sam/.rbenv/v

## Debugging 100% CPU usage in production Ruby on Rails systems

DevFeed: [Debugging 100% CPU usage in production Ruby on Rails systems](<https://devfeed.tech/articles/debugging-100-cpu-usage-in-production-ruby-on-rails-systems-41343.md>)

Original publisher: [Read original article](<https://samsaffron.com/archive/2018/01/18/my-production-ruby-on-rails-cpu-is-at-100-now-what>)

Author: Sam Saffron

Published: 2018-01-18T02:24:25Z

Content type: tutorial

Language: en

Sources: [Sam Saffron](<https://devfeed.tech/sources/sam-saffron.md>)

Topics: [Ruby on Rails](<https://devfeed.tech/topics/ruby-on-rails.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [debug](<https://devfeed.tech/topics/debug.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [debugging](<https://devfeed.tech/tags/debugging.md>), [performance](<https://devfeed.tech/tags/performance.md>), [production](<https://devfeed.tech/tags/production.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

This article explains how to investigate 100% CPU usage in a production Ruby on Rails system. It demonstrates using rbtrace and stackprof with Sidekiq to capture CPU profiles, identify the code consuming the processor, and analyze the resulting performance data. The reported culprit is a regular expression in Typography.to_html_french.

### Source excerpt

Today I noticed one of our customer containers was running really high on CPU. # top -bn1 190 discour+ 20 0 2266292 205128 15656 S 86.7 0.3 9377:35 ruby # ps aux discour+ 190 19.4 0.3 2266292 207096 ? Sl 2017 9364:38 sidekiq 5.0.5 discourse [1 of 5 busy] Looks like sidekiq is stuck on a job. Where is it stuck? Usually, this is where the story ends and another series of questions start Can we reproduce this on staging or development? What code changed recently? Why is perf trace not giving me anything I can work with? How awesome is my Sidekiq logging? Where is my divining rod? Julia Evans is working on a fantastic profiler that will allow us to answer this kind of question real quick. But in the meantime, is there anything we can do? rbtrace + stackprof At Discourse we include rbtrace and stackprof in our Gemfile. gem 'rbtrace' gem 'stackprof', require: false We always load up rbtrace in production, it allows us a large variety of production level debugging. stackprof is loaded on-demand. In this particular case I simply run: # rbtrace -p 190 -e 'Thread.new{ require "stackprof"; StackProf.start(mode: :cpu); sleep 2; StackProf.stop; StackProf.results("/tmp/perf"); }' This injects a new thread that enables stackprof globally and finally writes the performance data to /tmp/perf This dump can easily be analyzed: # stackprof /tmp/perf ================================== Mode: cpu(1000) Samples: 475 (0.63% miss rate) GC: 0 (0.00%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 473 (99.6%) 473 (99.6%) #<Class:0x00007fac08218020>::Typography.to_html_french 4 (0.8%) 1 (0.2%) #<Module:0x00007fac080c3620>.reap_connections In fact, we can even collect backtraces and generate flamegraphs in stackprof, so I strongly recommend reading through the readme. So there you have it, my culprit is: text.gsub(/(\s|)+([!?;]+(\s|\z))/, '&thinsp;\2\3') A regular expression consuming 100% CPU in production, this has got to be a first

## Writing a one-time script in Rails

DevFeed: [Writing a one-time script in Rails](<https://devfeed.tech/articles/writing-a-one-time-script-in-rails-26280.md>)

Original publisher: [Read original article](<https://www.justinweiss.com/articles/writing-a-one-time-script-in-rails/>)

Author: Justin Weiss

Published: 2017-02-14T06:08:51Z

Content type: tutorial

Language: en

Sources: [Justin Weiss](<https://devfeed.tech/sources/justin-weiss.md>)

Topics: [Rails](<https://devfeed.tech/topics/rails.md>), [Script](<https://devfeed.tech/topics/script.md>), [data](<https://devfeed.tech/topics/data.md>), [migration](<https://devfeed.tech/topics/migration.md>), [import](<https://devfeed.tech/topics/import.md>), [CSV](<https://devfeed.tech/topics/csv.md>), [Database](<https://devfeed.tech/topics/database.md>)

Tags: [capistrano](<https://devfeed.tech/tags/capistrano.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [heroku](<https://devfeed.tech/tags/heroku.md>), [import](<https://devfeed.tech/tags/import.md>), [job](<https://devfeed.tech/tags/job.md>), [migration](<https://devfeed.tech/tags/migration.md>), [other](<https://devfeed.tech/tags/other.md>), [rails](<https://devfeed.tech/tags/rails.md>), [rake](<https://devfeed.tech/tags/rake.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [script](<https://devfeed.tech/tags/script.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>), [structure](<https://devfeed.tech/tags/structure.md>)

### AI overview

This Rails tutorial explains how to handle one-time data tasks, comparing database migrations with rake tasks and scheduled background jobs. It recommends rake tasks for code that must be tested locally and run in production, while noting when migrations are appropriate and what risks model changes can create.

### Source excerpt

Have you ever wanted to import a bunch of data into your app from a CSV file? Or maybe you need to fix badly encoded characters in some of your customer reviews. Or you changed your mind about how you wanted to store data in Redis, and had to move everything from the old format to the new one. At Avvo, we called these "ad-hoc tasks." As in, you probably only need to run them once. So what's the best way to handle an ad-hoc task in Rails? Write a database migration A migration works well if you need to change the structure of the data in your database. It tracks whether the task was run, it carries over changes to other environments - it's what migrations were built for. It's also what you're probably already using them for. If you're changing data at the same time, a migration might work well. But there are some things to watch out for. Calling something like Permissions.create(...) in your migration can cause you trouble. If the model has changed, your migration might break, because your model might not be available when the migration runs. Or your model might have changed between the time you wrote the migration and when it ran. There are ways to get around this, but they're error-prone and can fail in weird ways. Migrations are also less useful if your task doesn't involve ActiveRecord. These aren't deal-breakers. But I tend not to import or change much data in migrations. There are better options. Write a rake task You have a task. You probably only want to run it once. And you want to be able to test it on your machine and run it in production. Rake tasks work really well for this. Rails can even generate rake tasks for you: $ be rails g task locations import create lib/tasks/locations.rake This creates a file for you to stash your code into: lib/tasks/locations.rakenamespace :locations do desc "TODO" task import: :environment do end end Inside that task block, you can use all your models and the rest of the code in your Rails app. It's easy to import and chang

## Optimum Sidekiq Configuration on Heroku with Puma

DevFeed: [Optimum Sidekiq Configuration on Heroku with Puma](<https://devfeed.tech/articles/optimum-sidekiq-configuration-on-heroku-with-puma-20072.md>)

Original publisher: [Read original article](<http://jollygoodcode.com/blog/2015/12/08/optimum-sidekiq-configuration-on-heroku-with-puma.html>)

Published: 2015-12-08T02:49:47Z

Content type: article

Language: en

Sources: [Jolly Good Code](<https://devfeed.tech/sources/jolly-good-code.md>)

Topics: [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Heroku](<https://devfeed.tech/topics/heroku.md>), [configuration](<https://devfeed.tech/topics/configuration.md>)

Tags: [configuration](<https://devfeed.tech/tags/configuration.md>), [heroku](<https://devfeed.tech/tags/heroku.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

The article presents an optimum configuration for running Sidekiq on Heroku with Puma.

### Source excerpt

What's the optimum config for Sidekiq on Heroku with Puma? Check out our config!

## Debugging memory leaks in Ruby

DevFeed: [Debugging memory leaks in Ruby](<https://devfeed.tech/articles/debugging-memory-leaks-in-ruby-41338.md>)

Original publisher: [Read original article](<https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby>)

Author: Sam Saffron

Published: 2015-03-31T06:44:39Z

Content type: tutorial

Language: en

Sources: [Sam Saffron](<https://devfeed.tech/sources/sam-saffron.md>)

Topics: [Memory Leaks](<https://devfeed.tech/topics/memory-leaks.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [debug](<https://devfeed.tech/topics/debug.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Docker](<https://devfeed.tech/topics/docker.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>)

Tags: [debugging](<https://devfeed.tech/tags/debugging.md>), [docker](<https://devfeed.tech/tags/docker.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [memory-leaks](<https://devfeed.tech/tags/memory-leaks.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [process](<https://devfeed.tech/tags/process.md>), [rails](<https://devfeed.tech/tags/rails.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

A practical guide to diagnosing memory leaks in Ruby and Rails applications. It recommends graphing memory over time, tracking RSS for key processes such as Unicorn and Sidekiq, and examining memory-growth trends across Docker containers.

### Source excerpt

At some point in the life of every Rails developer you are bound to hit a memory leak. It may be a tiny amount of constant memory growth, or a spurt of growth that hits you on the job queue when certain jobs run. Sadly, most Ruby devs out there simply employ monit , inspeqtor or unicorn worker killers. This allows you to move along and do more important work, tidily sweeping the problem snugly under the carpet. Unfortunately, this approach leads to quite a few bad side effects. Besides performance pains, instability and larger memory requirements it also leaves the community with a general lack of confidence in Ruby. Monitoring and restarting processes is a very important tool for your arsenal, but at best it is a stopgap and safeguard. It is not a solution. We have some amazing tools out there for attacking and resolving memory leaks, especially the easy ones - managed memory leaks. Are we leaking ? The first and most important step for dealing with memory problems is graphing memory over time. At Discourse we use a combo of Graphite , statsd and Grafana to graph application metrics. A while back I packaged up a Docker image for this work which is pretty close to what we are running today. If rolling your own is not your thing you could look at New Relic, Datadog or any other cloud based metric provider. The key metric you first need to track is RSS for your key Ruby processes. At Discourse we look at max RSS for Unicorn our web server and Sidekiq our job queue. Discourse is deployed on multiple machines in multiple Docker containers. We use a custom built Docker container to watch all the other Docker containers. This container is launched with access to the Docker socket so it can interrogate Docker about Docker. It uses docker exec to get all sorts of information about the processes running inside the container. Note: Discourse uses the unicorn master process to launch multiple workers and job queues, it is impossible to achieve the same setup (which shares memo

## Discourse as my blogging platform

DevFeed: [Discourse as my blogging platform](<https://devfeed.tech/articles/discourse-as-my-blogging-platform-41328.md>)

Original publisher: [Read original article](<https://samsaffron.com/archive/2013/10/10/discourse-as-my-blogging-platform>)

Author: Sam Saffron

Published: 2013-10-10T10:44:37Z

Content type: article

Language: en

Sources: [Sam Saffron](<https://devfeed.tech/sources/sam-saffron.md>)

Topics: [Rails](<https://devfeed.tech/topics/rails.md>), [Code](<https://devfeed.tech/topics/code.md>), [Digital Ocean](<https://devfeed.tech/topics/digital-ocean.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [GitHub](<https://devfeed.tech/topics/github.md>)

Tags: [blogging](<https://devfeed.tech/tags/blogging.md>), [code](<https://devfeed.tech/tags/code.md>), [digital-ocean](<https://devfeed.tech/tags/digital-ocean.md>), [email](<https://devfeed.tech/tags/email.md>), [github](<https://devfeed.tech/tags/github.md>), [integration](<https://devfeed.tech/tags/integration.md>), [plugin](<https://devfeed.tech/tags/plugin.md>), [rails](<https://devfeed.tech/tags/rails.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

The author describes moving a blog into Discourse by building a Discourse plugin with Rails engines. The post also discusses hosting on DigitalOcean and experimenting with Rails, Unicorn, Rack Hijack, Sidekiq, and email integration.

### Source excerpt

This Blog has gone through several incarnations. When I started blogging I used subtext. Then I decided it was time to reinvent wheels and rolled my own, hodge-podge, blogging solution. It worked ok, but I found I had extreme difficulty engaging with the community. People would leave replies and I had no idea if they ever got my response. Further more creating content was becoming a huge pain. I ended up using the Discourse editor for some posts and then pasting the body of the post here. Crazytown. So I had this idea, what if I could have Discourse turtles all the way down. I took some time to extend our plugin story and using the magic of Rails engines managed to host my blog inside Discourse. You can see the full source code of this blog as a Discourse plugin at: GitHub - SamSaffron/blog: samsaffron.com blog plugin for Discourse I am hosted on Digital Ocean (a 2GB instance). I am also experimenting with some less traditional Discourse configs: I am running Rails 4 I am running unicorn, with Rack Hijack to handle long polling I am running no bluepill, I tricked unicorn into daemonizing sidekiq and use an upstart job. Let's see how it goes. All of this and more deserves a much more detailed blog post. Before any of this, let me address the elephant in the room. Commenting on this blog just became way more complicated. I plan to address that, in a future update, this is a temporary state. I want to bring back the traditional commenting box. That said, I have email integration set up, so now when I respond to you, you will get an email you can respond to. You will not even need to visit this site to spam my blog Overall, I am super excited about this. It pushes the limits of what can be done with Discourse and leaves breadcrumbs for others to follow. More to come, now that blogging became so much easier.