# Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts

DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts](<https://devfeed.tech/articles/inside-cilium-cni-solving-mysterious-kubernetes-pod-setup-timeouts-26260.md>)

Original publisher: [Read original article](<https://medium.com/adyen/inside-cilium-cni-solving-mysterious-kubernetes-pod-setup-timeouts-8529dacd5e81?source=rss----64941d9fbc09---4>)

Author: Adyen

Published: 2026-07-13T09:17:40Z

Content type: article

Language: en

Sources: [Adyen Tech](<https://devfeed.tech/sources/adyen-tech.md>)

Topics: [Cilium](<https://devfeed.tech/topics/cilium.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Kubernetes networking](<https://devfeed.tech/topics/kubernetes-networking.md>), [networking](<https://devfeed.tech/topics/networking.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [hdfs](<https://devfeed.tech/topics/hdfs.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [cpu](<https://devfeed.tech/topics/cpu.md>)

Tags: [big-data](<https://devfeed.tech/tags/big-data.md>), [cilium](<https://devfeed.tech/tags/cilium.md>), [cni](<https://devfeed.tech/tags/cni.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data](<https://devfeed.tech/tags/data.md>), [devops](<https://devfeed.tech/tags/devops.md>), [ebpf](<https://devfeed.tech/tags/ebpf.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-networking](<https://devfeed.tech/tags/kubernetes-networking.md>), [latency](<https://devfeed.tech/tags/latency.md>), [spark](<https://devfeed.tech/tags/spark.md>), [speed](<https://devfeed.tech/tags/speed.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

## AI overview

An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs.

## Source excerpt

By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra