# Diagnosing a Deadlock in Trino's Hudi Connector That Stalled Blinkit's Inventory Pipeline

DevFeed: [Diagnosing a Deadlock in Trino's Hudi Connector That Stalled Blinkit's Inventory Pipeline](<https://devfeed.tech/articles/how-a-deadlock-froze-blinkit-s-supply-chain-20085.md>)

Original publisher: [Read original article](<https://lambda.blinkit.com/how-a-deadlock-froze-blinkits-supply-chain-4b7c4d6d4a3f?source=rss----42df4a1e8725---4>)

Author: Ratul Dawar

Published: 2026-05-29T09:26:27Z

Content type: article

Language: en

Sources: [Grofers](<https://devfeed.tech/sources/grofers.md>)

Topics: [Deadlock](<https://devfeed.tech/topics/deadlock.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [blinkit](<https://devfeed.tech/tags/blinkit.md>), [bug](<https://devfeed.tech/tags/bug.md>), [deadlock](<https://devfeed.tech/tags/deadlock.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [s3](<https://devfeed.tech/tags/s3.md>), [thread](<https://devfeed.tech/tags/thread.md>), [trino](<https://devfeed.tech/tags/trino.md>), [trinos](<https://devfeed.tech/tags/trinos.md>), [yield](<https://devfeed.tech/tags/yield.md>)

## AI overview

Blinkit describes how a deadlock in Trino's Hudi connector stalled inventory replenishment queries without errors or resource saturation. The issue involved one thread pool handling both file-split production and signalling; the reported fix used cooperative scheduling and was contributed upstream.

## Source excerpt

A silent deadlock in our query engine was stalling inventory replenishment jobs with no error, no crash -- just infinite waiting. This is the story of how we found it, traced it to an open-source bug, and fixed it upstream. TL;DRTrino's Hudi connector used a single thread pool for both producing file splits and signalling when there was room for more. Under load, every thread ended up waiting for a signal that had no thread left to run it. The fix was to switch the producer side to a cooperative scheduling pattern: yield the thread when the buffer is full, and resume when space opens. Our inventory replenishment pipeline was frozen. CPU was idle. Memory was fine. There were no errors anywhere. Queries just... stopped moving. The first signal was a long queue on one of our analytics clusters. Queries were piling up. Inventory replenishment jobs -- the jobs that decide how much stock every warehouse and store needs to hold -- were delayed. Blinkit's supply chain was being impacted. Dashboards were turning amber, but nothing was crashing. That was the unsettling part. Investigation: Resources Doing Nothing The affected cluster runs analytical workloads on Trino, reading data stored in Apache Hudi tables on S3. The natural first instinct in a queue build-up is to look at resource saturation -- a CPU spike, memory pressure, network bottleneck. There was none of that. The cluster was sitting largely idle, with CPU barely above baseline and heap usage well within limits. Every new query touching a Hudi table joined the queue and stayed there indefinitely. Queries that were already mid-execution completed fine. Only freshly submitted ones were affected. And crucially, there were no errors. No timeouts, no exceptions in the logs -- just silence and a growing backlog. A thread dump -- a snapshot of what every thread in the process is doing right now -- was our next move. It showed dozens of producer threads all stuck in the same parked state, waiting on the exact same internal signal.