# Sidekiq Sidekiq is a background job system for Ruby applications. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why Gusto hired Evil Martians for Sidekiq infrastructure DevFeed: [Why Gusto hired Evil Martians for Sidekiq infrastructure]() Original publisher: [Read original article]() Author: Irina Nazarova (inazarova@evilmartians.com) Published: 2026-09-02T00:00:00Z Content type: article Language: en Sources: [Evil Martians]() Topics: [Sidekiq](), [Rails](), [Redis](), [Ruby]() Tags: [ai](), [case-study](), [engineering](), [infrastructure](), [rails](), [redis](), [ruby](), [scale](), [sidekiq]() ### AI overview This case study examines why Gusto hired Evil Martians to work on the Sidekiq and Redis infrastructure supporting background jobs in its large Rails monolith. It discusses the operational demands of running payroll-related jobs at scale and the tradeoffs involved when a company uses consultants instead of waiting for a perfect hire. ### Source excerpt Gusto runs payroll for 500,000+ businesses on one of the largest Rails monoliths anywhere. Why a team this strong hired Evil Martians for Sidekiq at scale, what running background jobs at that size actually takes, and when to stop waiting for the perfect hire. ## A Pre-Deployment Checklist for Risky Data Migrations DevFeed: [A Pre-Deployment Checklist for Risky Data Migrations]() Original publisher: [Read original article]() Published: 2023-09-13T00:00:00Z Content type: tutorial Language: en Sources: [Artsy]() Topics: [Command-line interface](), [Docker](), [Kubernetes](), [Sidekiq](), [Latency]() Tags: [command-line](), [data-migrations](), [deploy-process](), [docker](), [error-reporting](), [kubernetes](), [latency](), [logs](), [migrations](), [retry](), [ruby-on-rails](), [sidekiq]() ### AI overview This article presents a pre-deployment checklist for risky data migrations and other changes that could affect production services. It recommends planning rollback steps and monitoring error rates, reported errors, latency, logs, functionality, and failing Sidekiq retries after deployment. ### Source excerpt Lately, I've been getting involved with some sketchy stuff. You know what I'm talking about-data migrations. I've been rolling out changes that have a significant risk of breaking our production environment for mission-critical services. It's been exciting work (keep your eyes out for more posts on the exact project, coming soon™), but I've definitely caused a couple incidents along the way. After accidentally taking down a key service for a couple hours, I decided I needed to have a better pre-deploy process for these changes. I did some thinking and came up with a short checklist to run through before I press the shiny green button. Here's the checklist I came up with: What is your plan if something goes wrong? Run through ramifications of rolling back. If there's a reason you're worried about rolling back, then you're not ready to deploy the change yet! Figure out exactly what command(s) you will need to run to roll back. At Artsy, this is usually a one-liner using Hokusai, our command-line Docker/Kubernetes CLI How will you tell if something is going wrong after you deploy? Error rate (DataDog) Specific error reporting (Sentry) Latency (DataDog) Logs (Papertrail) Functionality (does it still work? Are people using it successfully? Important for things where errors may not be bubbled up correctly or reported immediately) Sidekiq (are there lots of jobs queued to retry that are failing?) With this checklist in hand, I'm deploying more confidently and causing fewer incidents along the way. Do you have something similar? Are there things you think this checklist should include? Let me know in the comments! ## Introducing Simplekiq: A Lightweight Orchestration Framework for Ruby DevFeed: [Introducing Simplekiq: A Lightweight Orchestration Framework for Ruby]() Original publisher: [Read original article]() Author: Doximity Published: 2023-05-03T11:28:00Z Content type: article Language: en Sources: [Doximity]() Topics: [Orchestration](), [Sidekiq](), [Ruby](), [Framework]() Tags: [declarative](), [orchestration](), [parallel](), [refactoring](), [ruby](), [sidekiq](), [workflow]() ### AI overview The article introduces Simplekiq, a lightweight Ruby workflow framework built on Sidekiq Pro batching. It presents declarative, single-file orchestrations with serial and parallel execution as a way to make complex background-job workflows easier to understand, diagnose, and refactor. ### Source excerpt Sidekiq is an amazing background job framework for Ruby with a long history of the best kind of minimalism - one of performance and succinctness. However, a downside of this is that long workflows built from dozens of background jobs queueing subsequent background jobs can be difficult to consistently design and maintain. We especially struggled with refactoring these types of structures since one has to read through all the code for the jobs to understand when and from where each job might get queued. The added complexity of manually defining Sidekiq Pro batches leads to difficult-to-diagnose triage in more creative workflows, especially when batch callbacks are involved. Introducing Simplekiq: A Solution to Our Challenges We solved our issue by building Simplekiq (special thanks to Daniel Pepper for freeing the rubygems simplekiq name for us) --a lightweight background job workflow framework built on top of Sidekiq Pro batching. Simplekiq solves the issue of having to follow long chains of jobs that queue other jobs ad nauseam. Workflows that span many files and job classes are inherently difficult to understand, diagnose and refactor. Simplekiq flattens out these complex, multi-file workflows into declarative, single-file ones which we call orchestrations. Each orchestration can contain a mix of serial and parallel execution so that even the most complex workflow can be expressed in an efficient, readable syntax. Comparing Sidekiq Pro and Simplekiq with an Example It's easier to demonstrate this with an example: making an apple pie. The late, great Carl Sagan taught us that to make an apple pie from scratch we must first invent the universe. That's a bit out of budget for most bakeries, but in the spirit of simplicity let's explore how a workflow for baking an apple pie from basic ingredients looks in standard Sidekiq Pro structures and compare it to the flattened Simplekiq paradigm. Our example will also show how refactoring differs between the two. Implementing ## 5 SRE tips for Rails workloads in Kubernetes DevFeed: [5 SRE tips for Rails workloads in Kubernetes]() Original publisher: [Read original article]() Author: Doximity Published: 2022-12-02T13:00:00Z Content type: article Language: en Sources: [Doximity]() Topics: [Kubernetes](), [Rails](), [Deployment Strategies](), [Kafka](), [Sidekiq](), [site-reliability-engineering]() Tags: [bugs](), [deployment](), [deployment-strategies](), [kafka](), [kubernetes](), [performance](), [rails](), [sidekiq]() ### AI overview The article presents five changes made to Kubernetes clusters to better support Rails workloads and workflows. The supplied text focuses on deployment strategies: RollingUpdate is used for web traffic, while Recreate is used for Kafka and Sidekiq workloads because RollingUpdate caused rebalancing, performance, and compatibility issues during deployments. ### Source excerpt In a previous post we shared some suggestions for how to adjust a Rails application to make it run better in a containerized setting. In this post I'd like to explore the opposite relationship: five changes we have made to our Kubernetes clusters to better accommodate Rails workloads and workflows. All of these ideas can be relevant to non-Rails apps too! Selecting the right deployment strategies The two deployment strategies Kubernetes offers out of the box are RollingUpdate or Recreate. Initially we selected RollingUpdate for all of our workloads; with this strategy some new pods are created before old ones are terminated. The RollingUpdate strategy is ideal for our web traffic because if Kubernetes shuts down all of the existing pods before it starts up any new pods, users will see 5XX errors until the new web pods are healthy, not good. That said, the RollingUpdate strategy created issues with our Sidekiq and Kafka workloads so we switched those to use the Recreate strategy. Looking at Kafka first, consumers are basically just a group of processes working together to handle messages. Kafka consumers follow a rebalancing protocol when a new member wants to join a consumer group. During deployments we encountered bugs and performance issues while using RollingUpdate approach because new pods kept trickling-in and wanting to join the consumer group therefore triggering multiple rebalances. Switching our Kafka consumers to deploy using the Recreate strategy means all the existing Pods get killed before any new pods get created. This results in much more predictable deployment behavior, and now we see only a single rebalance with all new consumers joining the group at basically the same time. Next, looking at Sidekiq, it does not stop workers from processing jobs when a new worker process joins, so we didn't have quite as many issues with using the RollingUpdate approach. However, it did occasionally cause subtle bugs on some deployments. For example, if a team creat ## Square Payroll's Migration from a Monolith to Microservices DevFeed: [Square Payroll's Migration from a Monolith to Microservices]() Original publisher: [Read original article]() Author: Harold Sultan Published: 2022-10-12T04:00:00Z Content type: article Language: en Sources: [Square Corner Blog RSS Feed]() Topics: [Microservices](), [migration](), [Amazon Web Services](), [Serverless](), [serverless architecture](), [Architecture & Design](), [Ruby](), [MySQL](), [Sidekiq]() Tags: [architecture](), [aws](), [engineering](), [microservices](), [migration](), [mysql](), [ruby](), [serverless](), [serverless-architecture](), [sidekiq]() ### AI overview Square describes an early step in migrating Square Payroll from a Ruby monolith backed by MySQL to a more scalable architecture. The article focuses on redesigning its tax filing system with AWS serverless architecture and using a Step Function to reduce code complexity and improve transparency. ### Source excerpt Scaling with AWS serverless architecture ## Logster and our error logging strategy at Discourse DevFeed: [Logster and our error logging strategy at Discourse]() Original publisher: [Read original article]() Author: Sam Saffron Published: 2019-01-02T07:11:20Z Content type: article Language: en Sources: [Sam Saffron]() Topics: [log management](), [Logging](), [Ruby](), [Ruby on Rails](), [Open Source](), [Redis](), [Sidekiq](), [Grafana](), [nginx](), [Prometheus](), [Docker]() Tags: [backend](), [blog](), [c-sharp](), [docker](), [errors](), [grafana](), [logging](), [logs](), [monitoring](), [net](), [nginx](), [open-source](), [production](), [prometheus](), [rails](), [redis](), [ruby](), [ruby-on-rails](), [saffron](), [sam](), [sidekiq](), [sql](), [visibility](), [weblog]() ### AI overview The article describes Discourse's logging strategy and Logster, an open-source log viewer for Ruby on Rails or Rack applications. It explains how Logster stores and analyzes logs with Redis, aggregates production errors, and provides detailed development logs, alongside broader monitoring and observability tools. ### Source excerpt I have always been somewhat fascinated with logs. I tend to see the warning and error logs in production as a valuable heartbeat of an application. Proper handling of error logs is a very strong complement to a robust test suite. It shows us what really happens when real world data meets our application. 9 years ago, at Stack Overflow we had a daily ritual where we would open up our fork of ELMAH every morning and fish through our logs for problems. This had a dramatic positive effect on Stack Overflow. Almost 7 years into our journey building Discourse, every single week we find and fix issues in our application thanks to our error logs and Logster. Error logs are the pulse of our application, they let us know immediately if there are any urgent issues and where. Since we host more than 1500 sites running many different code branches, we needed to evolve a sane and robust set of practices and tools. Top level structure of logging and monitoring at Discourse We have lots of logs at Discourse and many systems for dealing with them. We keep raw Docker, Postgres, Redis, NGINX, Rails and HAProxy and so on in Elastic Search and use Kibana for business intelligence. We have a monitoring system built on alertmanager and Prometheus, with business intelligence in Grafana and alert escalation in our internal Discourse instance and opsgenie. We have logster which we use for web application aka. "Rails / Sidekiq" warnings and errors. I would like to focus on logster and our Rails / Sidekiq portion for this blog post, but think it is worth mentioning other mechanisms cause I don't want people to think we are not good data hoarders and only have very limited visibility into our systems. About Logster At Discourse we developed a log viewer called logster. Logster is a free and open source tool you can embed into any Ruby on Rails or Rack application in production and development. It runs as Rack middleware and uses Redis as its backend for log storage and analysis. It operates in ## Optimum Sidekiq Configuration on Heroku with Puma DevFeed: [Optimum Sidekiq Configuration on Heroku with Puma]() Original publisher: [Read original article]() Published: 2015-12-08T02:49:47Z Content type: article Language: en Sources: [Jolly Good Code]() Topics: [Sidekiq](), [Heroku](), [configuration]() Tags: [configuration](), [heroku](), [sidekiq]() ### AI overview The article presents an optimum configuration for running Sidekiq on Heroku with Puma. ### Source excerpt What's the optimum config for Sidekiq on Heroku with Puma? Check out our config!