# production-issue

Published articles for production-issue.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Shared Host Ephemeral Port Exhaustion Caused Overnight Job Failures

DevFeed: [Shared Host Ephemeral Port Exhaustion Caused Overnight Job Failures](<https://devfeed.tech/articles/we-ran-out-of-tcp-ports-at-1-30-am-the-crack-was-tcp-tw-reuse-0-26518.md>)

Original publisher: [Read original article](<https://medium.com/engineering-housing/we-ran-out-of-tcp-ports-at-1-30-am-the-crack-was-tcp-tw-reuse-0-54952bf5f7bd?source=rss----3a69e32e2594---4>)

Author: PULKIT GARG

Published: 2026-08-11T05:58:23Z

Content type: article

Language: en

Sources: [Housing.com](<https://devfeed.tech/sources/housing-com.md>)

Topics: [ephemeral ports](<https://devfeed.tech/topics/ephemeral-ports.md>), [incident](<https://devfeed.tech/topics/incident.md>), [Kernel](<https://devfeed.tech/topics/kernel.md>)

Tags: [incident](<https://devfeed.tech/tags/incident.md>), [port](<https://devfeed.tech/tags/port.md>), [production-issue](<https://devfeed.tech/tags/production-issue.md>), [rate-limiting](<https://devfeed.tech/tags/rate-limiting.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [tcp](<https://devfeed.tech/tags/tcp.md>)

### AI overview

An engineering incident report explains how a neighboring bulk job exhausted the shared operating system's ephemeral TCP port pool, causing scheduled background jobs to fail until the temporary traffic pressure subsided.

### Source excerpt

A neighbor's bulk job drained every ephemeral port on a shared box. Twice. Here's the one kernel setting that would have stopped it -- and the architecture that makes it a non-issue. At 1:38 AM, one of our background-job services started failing. Scheduled jobs -- the nudges and reminders we send users overnight -- began throwing an error most engineers rarely see: Errno::EADDRNOTAVAIL -- Cannot assign requested address Not "connection refused." Not "timeout." Cannot assign requested address. The machine wasn't saying the other end was down. It was saying it had run out of addresses to call from. Here's the strange part: we hadn't deployed anything. Our code was untouched for days. And by ~3 AM it healed itself, with nobody doing a thing. Two nights later, same window, it happened again. The bug wasn't in our service. It was in a service we don't own -- and don't even talk to. Act 1: A crime scene with no suspect ~7,800 job executions failed across the two nights. The first night slipped in under the alerting threshold entirely -- we only found it in the errors inbox after the fact. Everything about it was confusing: Our service hadn't changed. The internal APIs we call were all healthy. It happened in a tight window (~1:30-3:00 AM), then vanished. No human fixed it. It just... stopped. An outage that fixes itself is telling you something important: the thing that broke wasn't yours, and the pressure that caused it was temporary. Act 2: The noisy neighbor That service doesn't run on its own machine. It shares one box with about ten other apps -- a normal, cost-efficient setup. What's easy to forget is what they silently share: not just CPU and memory, but the operating system's ephemeral port pool. Quick refresher. Every time a program opens an outbound TCP connection, the OS assigns it a temporary "ephemeral" source port. That pool is finite -- by default, roughly 28,000 ports. And on a shared box, every app draws from the same pool. Ephemeral ports are an unmonitored, multi

## Using the OSI Model for Effective Production Issue Debugging

DevFeed: [Using the OSI Model for Effective Production Issue Debugging](<https://devfeed.tech/articles/using-the-osi-model-for-effective-production-issue-debugging-26515.md>)

Original publisher: [Read original article](<https://medium.com/engineering-housing/using-the-osi-model-for-effective-production-issue-debugging-c37052e87b48?source=rss----3a69e32e2594---4>)

Author: Kamal Kumar

Published: 2025-08-28T17:37:45Z

Content type: tutorial

Language: en

Sources: [Housing.com](<https://devfeed.tech/sources/housing-com.md>)

Topics: [debugging](<https://devfeed.tech/topics/debugging.md>), [networking](<https://devfeed.tech/topics/networking.md>), [Network](<https://devfeed.tech/topics/network.md>), [Routing (disambiguation)](<https://devfeed.tech/topics/routing.md>), [Server](<https://devfeed.tech/topics/server.md>), [Authentication](<https://devfeed.tech/topics/authentication.md>), [Encryption](<https://devfeed.tech/topics/encryption.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [Framework](<https://devfeed.tech/topics/framework.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [apis](<https://devfeed.tech/tags/apis.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [devops](<https://devfeed.tech/tags/devops.md>), [errors](<https://devfeed.tech/tags/errors.md>), [firewalls](<https://devfeed.tech/tags/firewalls.md>), [load](<https://devfeed.tech/tags/load.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [network](<https://devfeed.tech/tags/network.md>), [osi-model](<https://devfeed.tech/tags/osi-model.md>), [production-debugging](<https://devfeed.tech/tags/production-debugging.md>), [production-issue](<https://devfeed.tech/tags/production-issue.md>), [rca](<https://devfeed.tech/tags/rca.md>), [root-cause-analysis](<https://devfeed.tech/tags/root-cause-analysis.md>), [routing](<https://devfeed.tech/tags/routing.md>), [servers](<https://devfeed.tech/tags/servers.md>), [sre](<https://devfeed.tech/tags/sre.md>), [tcp](<https://devfeed.tech/tags/tcp.md>), [troubleshooting](<https://devfeed.tech/tags/troubleshooting.md>), [udp](<https://devfeed.tech/tags/udp.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

This tutorial explains how to use the seven-layer OSI model to structure root cause analysis for production alerts. It recommends checking lower layers for network and gateway errors, and higher layers for application and client errors, including connectivity, routing, logs, APIs, authentication, and configuration.

### Source excerpt

In production environments, debugging alerts can sometimes feel like finding a needle in a haystack. Over the years, I've found the OSI (Open Systems Interconnection) model to be a reliable guide during Root Cause Analysis (RCA) of production issues. What is the OSI Model? The OSI model is a conceptual framework that standardizes the functions of a telecommunication or computing system into seven layers: Physical Layer -- Hardware, cables, switches Data Link Layer -- MAC addresses, switches, network topology Network Layer -- IP addressing, routing Transport Layer -- TCP/UDP, ports, session reliability Session Layer -- Session management, authentication Presentation Layer -- Data translation, encryption Application Layer -- APIs, web servers, applications How I Use OSI Layers in RCA: When I debug production alerts, I follow different approaches depending on the type of error: Network / Gateway Errors (e.g., 502, 504): These errors usually indicate communication issues between services. I start from the bottom layers (Physical -> Network -> Transport) to check connectivity, firewalls, routing, or load balancers. Application / Client Errors (e.g., 500, 503, 404): These errors generally originate from the application or business logic. I start from the top layers (Application -> Presentation -> Session) to check service logs, APIs, authentication issues, or configuration problems. Why this approach works: Following the OSI model provides a structured, layer-by-layer method for troubleshooting, ensuring that we don't miss low-level network issues or high-level application errors. It helps reduce mean time to resolution (MTTR) and improves the quality of RCA reports. Takeaway: The OSI model is not just a theoretical concept -- it's a practical tool that can guide engineers through complex production debugging. Next time you face a tricky alert, try mapping it to the OSI layers, and you might find the root cause faster than you think. Using the OSI Model for Effective Production Issue