# Housing.com Engineering and technology articles from developers - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-09-07T06:33:31Z Content type: article Language: en Sources: [Housing.com]() Topics: [BigQuery](), [databricks](), [Optimization](), [AWS Database Migration Service](), [data-platforms](), [Platform Engineering](), [migration](), [Cloud]() Tags: [bigquery](), [cloud-computing](), [cost](), [cost-optimization](), [data-engineering](), [data-platforms](), [databricks](), [delta-lake](), [finops](), [infrastructure](), [migration](), [optimization](), [platform-engineering]() ### AI overview This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation. ### Source excerpt How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea ## Shared Host Ephemeral Port Exhaustion Caused Overnight Job Failures DevFeed: [Shared Host Ephemeral Port Exhaustion Caused Overnight Job Failures]() Original publisher: [Read original article]() Author: PULKIT GARG Published: 2026-08-11T05:58:23Z Content type: article Language: en Sources: [Housing.com]() Topics: [ephemeral ports](), [incident](), [Kernel]() Tags: [incident](), [port](), [production-issue](), [rate-limiting](), [ruby-on-rails](), [tcp]() ### AI overview An engineering incident report explains how a neighboring bulk job exhausted the shared operating system's ephemeral TCP port pool, causing scheduled background jobs to fail until the temporary traffic pressure subsided. ### Source excerpt A neighbor's bulk job drained every ephemeral port on a shared box. Twice. Here's the one kernel setting that would have stopped it -- and the architecture that makes it a non-issue. At 1:38 AM, one of our background-job services started failing. Scheduled jobs -- the nudges and reminders we send users overnight -- began throwing an error most engineers rarely see: Errno::EADDRNOTAVAIL -- Cannot assign requested address Not "connection refused." Not "timeout." Cannot assign requested address. The machine wasn't saying the other end was down. It was saying it had run out of addresses to call from. Here's the strange part: we hadn't deployed anything. Our code was untouched for days. And by ~3 AM it healed itself, with nobody doing a thing. Two nights later, same window, it happened again. The bug wasn't in our service. It was in a service we don't own -- and don't even talk to. Act 1: A crime scene with no suspect ~7,800 job executions failed across the two nights. The first night slipped in under the alerting threshold entirely -- we only found it in the errors inbox after the fact. Everything about it was confusing: Our service hadn't changed. The internal APIs we call were all healthy. It happened in a tight window (~1:30-3:00 AM), then vanished. No human fixed it. It just... stopped. An outage that fixes itself is telling you something important: the thing that broke wasn't yours, and the pressure that caused it was temporary. Act 2: The noisy neighbor That service doesn't run on its own machine. It shares one box with about ten other apps -- a normal, cost-efficient setup. What's easy to forget is what they silently share: not just CPU and memory, but the operating system's ephemeral port pool. Quick refresher. Every time a program opens an outbound TCP connection, the OS assigns it a temporary "ephemeral" source port. That pool is finite -- by default, roughly 28,000 ports. And on a shared box, every app draws from the same pool. Ephemeral ports are an unmonitored, multi ## We Gave Our Service More CPU -- and It Took Down Production DevFeed: [We Gave Our Service More CPU -- and It Took Down Production]() Original publisher: [Read original article]() Author: PULKIT GARG Published: 2026-08-10T06:48:00Z Content type: article Language: en Sources: [Housing.com]() Topics: [cpu](), [Kubernetes](), [incident]() Tags: [cpu](), [deadcode](), [incident](), [jvm](), [kubernetes](), [parallelstream](), [production](), [production-outage]() ### AI overview An incident report explains how increasing a service from 1.5 to 2.5 CPU cores caused a lead-creation API to stop creating leads across the web and app. The investigation found that Kubernetes CPU requests and limits had been treated differently: the request was reverted, but the elevated limit remained, leaving the service in its broken state. ### Source excerpt Every engineer has a story about the "harmless" change that wasn't. This is mine. Ours didn't start with a bad deploy. No risky feature flag, no sketchy migration, no Friday-evening hotfix. It started with a change every ops playbook calls safe: We gave a struggling service more CPU. We went from 1.5 cores to 2.5 cores. That's it. And within minutes, our lead-creation API -- the thing that turns a visitor tapping "Contact" into an actual business lead -- stopped creating leads. Entirely across web and app. The twist that cost us hours: the more CPU we threw at it, the worse it got. This is the story of why -- and the three-year-old landmine we stepped on. The incident in five beats -- including the "rollback" that changed nothing.Act 1: The pager goes off It's mid-afternoon. Leads flatline. Not "slow" -- zero. New listings get intermittently sluggish too. Instinct says: something downstream is down. A database. A cache. Elasticsearch. So we do what everyone does -- we start pointing fingers at infrastructure: Is Postgres healthy? Looks fine. Redis? Fine. Elasticsearch? Fine. Is the DB connection pool exhausted? Let's bump it -- 20 connections to 30. Let's add a live monitor to watch it. We even stopped incoming traffic to let the system breathe. Nothing helped. And here's the detail that made everyone's head hurt: CPU utilization wasn't even high. We had just added CPU. The graphs looked calm. The service looked healthy by every metric we trusted. Yet requests went in and never came out. When your dashboards say "fine" and your users say "down," you're not looking at the right layer. Act 2: The rollback that didn't roll back Eventually someone said the obvious thing: "Wasn't the only change today the CPU bump? Roll it back." So we did. Or we thought we did. We reverted the CPU request. The service kept failing. What we missed -- and what burned an embarrassing amount of the incident -- is that Kubernetes has two CPU knobs: request -- what the scheduler reserves for you limit ## Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently) DevFeed: [Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently)]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-07-20T10:07:52Z Content type: article Language: en Sources: [Housing.com]() Topics: [data](), [product analytics]() Tags: [analytics](), [dashboards](), [data](), [data-architecture](), [data-engineering](), [data-strategy](), [databricks](), [governance](), [leadership](), [product-analytics](), [sql](), [technical]() ### AI overview This article argues that Single Source of Truth initiatives often fail because teams use different definitions for shared business metrics. It presents governance, business ownership of KPI definitions, and alignment between teams as more important than centralizing tables, pipelines, or dashboards. ### Source excerpt "We have multiple dashboards showing different numbers. Which one is correct?" If you've worked in data long enough, you've probably heard this question more times than you'd like. Sales reports one revenue figure. Finance reports another. Product Analytics has a third. Executives spend more time debating whose dashboard is correct than discussing what action to take. The natural response is often: "Let's build a Single Source of Truth." Sounds simple. Build a few centralized tables. Move everyone onto the same dashboards. Problem solved. Except...it rarely is. After leading an enterprise-wide Single Source of Truth (SSOT) initiative, I learned an important lesson: The hardest part wasn't building pipelines or writing SQL. It was aligning people. Technology was the easy part. Changing how the organization thought about data was the real challenge. The Biggest Myth About Single Source of Truth Many organizations believe an SSOT is simply a technical project. The thinking usually goes like this: Collect Data ↓ Transform Data ↓ Build Gold Tables ↓ Everyone Uses Them Unfortunately, reality looks more like this: Different Teams ↓ Different Definitions ↓ Different Dashboards ↓ Different Decisions ↓ Lost Trust The problem isn't that data lives in different places. The problem is that different teams define the same business metrics differently. Figure 1: Moving from fragmented metric definitions to a trusted Single Source of Truth is as much about standardization and governance as it is about technology. A table cannot solve that. Only governance can. Technology Doesn't Create Trust Imagine a metric as simple as Revenue. Ask five departments what "Revenue" means, and you might receive five different answers. Finance may recognize revenue after invoicing. Sales may count closed deals. Marketing may include projected pipeline. Product Analytics may track subscription purchases. Customer Success may exclude refunds. None of them are necessarily wrong. They're answering differen ## How We Built an AI Agent to Clean Up Dead Code After A/B Tests DevFeed: [How We Built an AI Agent to Clean Up Dead Code After A/B Tests]() Original publisher: [Read original article]() Author: Aseem Upadhyay Published: 2026-06-23T10:54:43Z Content type: article Language: en Sources: [Housing.com]() Topics: [AI Agent](), [experiments](), [Code](), [Pull Request](), [context](), [Android]() Tags: [agent](), [ai](), [ai-agent](), [ai-agents-in-action](), [automated](), [code](), [concurrent](), [experiment](), [experiments](), [infrastructure](), [jira](), [llm]() ### AI overview Housing.com describes building an AI-agent pipeline to help clean up code after A/B tests. The workflow interprets experiment tickets, reports experiment status, and applies instructions to code, while using scripts for deterministic steps and an LLM where judgment is required. ### Source excerpt Photo by Microsoft Copilot on Unsplash At Housing.com, running product experiments is a continuous cycle. A/B tests go live, collect data, and eventually reach a conclusion. That's the exciting part. Then comes the mundane reality where someone has to clean up the code .i.e. remove a feature flag, promote a winning variant or revert the loser variant and finally raise a change request. Sounds simple? Maybe Is it tedious and quietly expensive? Yes! lifecycle of a taskThe Problem Worth Solving An experiment conclusion ticket typically lands on an engineer's desk looking something like this: Experiment: show_listing_map_widget Platform: Android Result: Negative - revert to control The job of the assigned engineer involves four distinct steps: Find every reference to the flag across the codebase. Delete the losing variant's code path. Trace every side-effect that only existed to support that variant Commit, open a PR, and comment on the Jira ticket. Step 3 is where the trap lies. Be it applying or removing a change, changing all the infrastructure code dependent on it could increase the complexity and risk of creating technical debt. But what if we automated a part of it? The AI Agent Pipelineupdated AI enabled lifecycle The problem statement became simple: Let stakeholders own the trigger. We built two agents to make it happen, 1. to interpret tickets and report experiment status 2. to take the instructions and code. Then came the hard part. Navigating Roadblocks The real complexity lies in building an AI agent that runs autonomously and serves different users across different use cases We found ourselves wrestling with questions we hadn't fully anticipated: How do we optimise on the tokens used per request? How do we handle concurrent requests? How do we ensure that the consistency in the output? So we went looking for answers.. Optimising Tokens per request Not every step needs AI. At each point in the workflow, we asked one question: is this operation deterministic ## Using the OSI Model for Effective Production Issue Debugging DevFeed: [Using the OSI Model for Effective Production Issue Debugging]() Original publisher: [Read original article]() Author: Kamal Kumar Published: 2025-08-28T17:37:45Z Content type: tutorial Language: en Sources: [Housing.com]() Topics: [debugging](), [networking](), [Network](), [Routing (disambiguation)](), [Server](), [Authentication](), [Encryption](), [configuration](), [business logic](), [Framework]() Tags: [analysis](), [apis](), [authentication](), [configuration](), [debugging](), [devops](), [errors](), [firewalls](), [load](), [mapping](), [network](), [osi-model](), [production-debugging](), [production-issue](), [rca](), [root-cause-analysis](), [routing](), [servers](), [sre](), [tcp](), [troubleshooting](), [udp](), [web]() ### AI overview This tutorial explains how to use the seven-layer OSI model to structure root cause analysis for production alerts. It recommends checking lower layers for network and gateway errors, and higher layers for application and client errors, including connectivity, routing, logs, APIs, authentication, and configuration. ### Source excerpt In production environments, debugging alerts can sometimes feel like finding a needle in a haystack. Over the years, I've found the OSI (Open Systems Interconnection) model to be a reliable guide during Root Cause Analysis (RCA) of production issues. What is the OSI Model? The OSI model is a conceptual framework that standardizes the functions of a telecommunication or computing system into seven layers: Physical Layer -- Hardware, cables, switches Data Link Layer -- MAC addresses, switches, network topology Network Layer -- IP addressing, routing Transport Layer -- TCP/UDP, ports, session reliability Session Layer -- Session management, authentication Presentation Layer -- Data translation, encryption Application Layer -- APIs, web servers, applications How I Use OSI Layers in RCA: When I debug production alerts, I follow different approaches depending on the type of error: Network / Gateway Errors (e.g., 502, 504): These errors usually indicate communication issues between services. I start from the bottom layers (Physical -> Network -> Transport) to check connectivity, firewalls, routing, or load balancers. Application / Client Errors (e.g., 500, 503, 404): These errors generally originate from the application or business logic. I start from the top layers (Application -> Presentation -> Session) to check service logs, APIs, authentication issues, or configuration problems. Why this approach works: Following the OSI model provides a structured, layer-by-layer method for troubleshooting, ensuring that we don't miss low-level network issues or high-level application errors. It helps reduce mean time to resolution (MTTR) and improves the quality of RCA reports. Takeaway: The OSI model is not just a theoretical concept -- it's a practical tool that can guide engineers through complex production debugging. Next time you face a tricky alert, try mapping it to the OSI layers, and you might find the root cause faster than you think. Using the OSI Model for Effective Production Issue ## Optimistic Locking Overview DevFeed: [Optimistic Locking Overview]() Original publisher: [Read original article]() Author: somesh sharma Published: 2025-06-19T09:02:40Z Content type: tutorial Language: en Sources: [Housing.com]() Topics: [Concurrency](), [Transactions](), [Database]() Tags: [concurrency](), [concurrency-control](), [database](), [databases](), [isolation-level](), [locks](), [optimistic-locking](), [retry](), [sql](), [stateless](), [transaction-management](), [transactions]() ### AI overview An overview of optimistic locking as a concurrency-control mechanism for database transactions. It explains how version columns detect conflicting updates, when optimistic locking is suitable, and when transaction isolation may make it unnecessary. ### Source excerpt Optimistic locking is a concurrency control mechanism where we assume that multiple transactions can safely access data without conflict, allowing them to proceed without locking the data upfront. Unlike pessimistic locking, where resources are locked to avoid conflicts, optimistic locking allows transactions to proceed without locks and checks for conflicts only when updating the data. If a conflict is detected (e.g., another transaction has already modified the data), the operation fails, and you can retry it. When Should You Use Optimistic Locking? Optimistic locking is ideal for use in scenarios where: Low contention exists: If it's unlikely that multiple users or processes will try to update the same data at the same time, optimistic locking is a good fit. This is especially true in systems where most operations involve reading data rather than writing it. High read-to-write ratio: If your application is mostly about reading data and writing happens less frequently, optimistic locking helps avoid the overhead of locking rows during reads. Non-critical updates: In cases where it's okay to retry a failed update without much impact, optimistic locking is a good choice. When conflicts arise, either the user or the system can simply retry. Stateless operations: Optimistic locking is suitable in stateless environments where holding onto locks across multiple requests or sessions isn't feasible. Long-running transactions: If your transactions take a long time to complete, holding locks during the entire process isn't feasible. Optimistic locking provides flexibility while still maintaining data integrity. How Optimistic Locking Works in Databases In databases, optimistic locking is usually implemented with a version column. Each row in the database has a version field (like a number or timestamp) that gets updated whenever the row is modified. When updating a record, the database checks if the version in the database matches the version the transaction originally read ## Enhancing Customer Engagement: Exploring Gupshup API Integration for Effective Messaging DevFeed: [Enhancing Customer Engagement: Exploring Gupshup API Integration for Effective Messaging]() Original publisher: [Read original article]() Author: Jitendra Kumar Published: 2024-06-12T06:55:06Z Content type: tutorial Language: en Sources: [Housing.com]() Topics: [API](), [Messaging]() Tags: [api](), [api-integration](), [customer-engagement](), [google-play](), [gupshup](), [integration](), [lead-generation](), [whatsapp](), [whatsapp-marketing]() ### AI overview This tutorial introduces Gupshup's API for adding messaging capabilities to applications and explains initial steps for connecting a WhatsApp Business account, including account creation, Gupshup registration, API access requests, and WhatsApp approval. ### Source excerpt Introduction In today's world, effective communication plays an important role in the success of businesses and organizations. With the rise of mobile devices and messaging, businesses continue to find new ways to connect and interact with customers. Gupshup is an interactive platform that provides a powerful API that allows businesses to connect resources in their applications, websites or systems. In this blog, we will explore the advantages and steps of integrating the Gupshup API to better communicate with our customers. What is Gupshup API? The Gupshup API is a set of tools and resources that allow developers to integrate messaging into their own applications. SMS messages, WhatsApp messages and Facebook Messenger etc. other social media apps have great features like sending and receiving. With the Gupshup API, businesses can harness the power of messaging to improve the customer experience, increase operational efficiency and drive business growth. Gupshup's WhatsApp Business API allows businesses to message and communicate with WhatsApp users. Before getting into the detail integration of gupshup api, we need one more step. we need a whatsapp bussiness account that need to be linked the gupshup account. To link a WhatsApp Business account with Gupshup, you can follow the steps outlined below: Create a WhatsApp Business Account: First, you need to create a WhatsApp Business account for your business or organization. You can download the WhatsApp Business app from the Google Play Store or Apple App Store and follow the registration process to create your account. Sign up for Gupshup API: Visit the Gupshup website (www.gupshup.io) and sign up for an account. Choose the appropriate plan and subscription based on your business requirements. Request WhatsApp Business API Access: Once you have signed up for Gupshup, you need to request access to the WhatsApp Business API. This step involves providing necessary information about your business, such as its name, websi ## Getting Started with Socket.IO: Building Real-Time Web Applications DevFeed: [Getting Started with Socket.IO: Building Real-Time Web Applications]() Original publisher: [Read original article]() Author: Jitendra Kumar Published: 2024-06-12T06:49:56Z Content type: tutorial Language: en Sources: [Housing.com]() Topics: [Socket.IO](), [WebSocket](), [Node.js](), [HTTP](), [servers]() Tags: [client-library](), [event](), [http](), [latency](), [node-js](), [real-time](), [server](), [web-applications](), [websocket]() ### AI overview This tutorial introduces Socket.IO as a Node.js server-side library for low-latency, bidirectional, event-based communication. It explains the client-server connection process, including the HTTP handshake, transport selection, optional upgrade to WebSocket, real-time messaging, and heartbeat keep-alive. ### Source excerpt What is Socket.io ? Socket.IO is a server-side library for nodejs that enables low-latency, bidirectional and event-based communication between a client and a server using client-server architecture while it is a wrapper around WebSockets For Node.js, it is super easy and simple to use especially when dealing with chat messages or Real-time data. Socket Client-Server Architecture How persistent bi-directional connection is established between client(s) and server ? The connection process between a Socket.IO client and server involves several steps. Below is a detailed breakdown of the key steps: 1. Handshake Initialization: The client initiates a connection to the Socket.IO server by sending an HTTP request (often a GET request) to the server using constructor provided by socket-io.client library. // Connect to the server (URL is optional if both client and server are at same origin ) const socket = io('http://localhost:3000'); The request includes a specific path, typically /socket.io/, and it may also include query parameters indicating the supported Socket.IO protocol version (EIO=3), transports, and other details. Example : GET /socket.io/?EIO=3&transport=polling&t=timestamp 2. Server Handshake Response: The Socket.IO server receives the initial HTTP request and responds with a handshake. The response includes the selected transport mechanism and a unique session ID (sid) that will be used to identify the connection. 3. Connection Upgrade (Optional): If the client and server both support WebSocket and agree to use it, they may negotiate to upgrade the connection from HTTP to WebSocket. This is an optimization for more efficient and bidirectional communication. 4. Transport Initialization: Based on the handshake response, the client initializes the chosen transport (e.g., WebSocket, polling). If using WebSocket, a WebSocket connection is established directly. If using polling, the client may start polling the server for updates. 5. Real-Time Communication: Once t ## Harnessing the Power of Service Workers: Building Offline-Ready Web Apps DevFeed: [Harnessing the Power of Service Workers: Building Offline-Ready Web Apps]() Original publisher: [Read original article]() Author: Jitendra Kumar Published: 2024-06-12T06:49:30Z Content type: tutorial Language: en Sources: [Housing.com]() Topics: [Web Development](), [web applications](), [Caching](), [browser](), [interception](), [JavaScript](), [Persistence]() Tags: [browser](), [building](), [cache](), [caching](), [interception](), [javascript](), [offline](), [persistence](), [web](), [web-apps]() ### AI overview This tutorial explains how service workers run in the background of web applications, act as proxies between pages and the network, and enable offline functionality by serving cached content. It compares service worker caching with standard browser caching and describes Cache Storage as distinct from Local Storage and Session Storage. ### Source excerpt In the dynamic world of web development, where user experience reigns supreme, having a web application that functions seamlessly even in offline mode is no longer a luxury but a necessity. Enter service workers -- the unsung heroes behind the scenes, making this magic possible. In this blog post, we'll delve into the world of service workers, exploring what they are, how they work, and how you can harness their power to supercharge your web applications. What is a service worker ? At its core, a service worker is a script that runs in the background of a web application, separate from the main JavaScript thread. It acts as a proxy between your web page and the network, enabling a host of powerful features such as offline functionality. This means that even if the user loses internet connection, the service worker can still serve cached content, providing a seamless offline experience. How service worker caching is different than standard browser caching ? Location and Scope: Service Worker: JavaScript files running in the background, separate from the main browser thread. Browser Caching: Resources stored within the browser itself, part of the main browsing environment. 2. Control Over Network Requests: Service Worker: Provides fine-grained control over network requests, allowing interception and customized handling. Browser Caching: More passive, resources are stored and retrieved by the browser without direct developer intervention. 3. Offline Functionality: Service Worker: Enables creation of offline experiences by caching and serving resources even when offline. Browser Caching: Improves performance but doesn't inherently provide offline functionality. 4. Persistence and Management: Service Worker: Persistent and programmatically managed caches, controlled by developers. Browser Caching: Managed by the browser, with limited developer control over cache lifecycle. Where service worker data is stored ? Service worker caching is stored in a separate storage called