# high performance

Published articles for high performance.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Inside the Tech That Powers Dream Cricket's Multiplayer Experience

DevFeed: [Inside the Tech That Powers Dream Cricket's Multiplayer Experience](<https://devfeed.tech/articles/inside-the-tech-that-powers-dream-cricket-s-multiplayer-experience-22623.md>)

Original publisher: [Read original article](<https://medium.com/dreamlockerroom/inside-the-tech-that-powers-dream-crickets-multiplayer-experience-7922a41af559?source=rss----5c7a7f580b01---4>)

Author: Dream Blog

Published: 2025-12-13T07:38:59Z

Content type: article

Language: en

Sources: [Dream11 Engineering](<https://devfeed.tech/sources/dream11-engineering.md>)

Topics: [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [cloud-infrastructure](<https://devfeed.tech/topics/cloud-infrastructure.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Server](<https://devfeed.tech/topics/server.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cricket](<https://devfeed.tech/tags/cricket.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [game-development](<https://devfeed.tech/tags/game-development.md>), [game-servers](<https://devfeed.tech/tags/game-servers.md>), [games](<https://devfeed.tech/tags/games.md>), [gaming](<https://devfeed.tech/tags/gaming.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [latency](<https://devfeed.tech/tags/latency.md>), [multiplayer](<https://devfeed.tech/tags/multiplayer.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [tech](<https://devfeed.tech/tags/tech.md>)

### AI overview

This article describes how Dream Cricket scaled its multiplayer infrastructure after a simple queue and fixed server pool could no longer handle growing adoption and large-scale events. It discusses challenges including peak matchmaking delays, unfair pairings, idle capacity, risky rollouts, and limited operational visibility, then outlines a design using Open Match, Agones, Kubernetes, and a health and latency service.

### Source excerpt

By Vaibhav Naik Every great game starts with great tech -- and great people behind it. If you're passionate about crafting seamless, high-performance gaming experiences that bring players closer to the sport they love, we'd love to have you on our team. Explore our open roles at Dream Cricket. Why Multiplayer Matters on Dream Cricket We've always believed that the best games are the ones that bring people together. And that's exactly what multiplayer on Dream Cricket does -- it turns gaming from a solo challenge into a shared moment. The banter, the thrill of a close finish, the satisfaction of a fair match -- that's what keeps players coming back. Over the years, as our player base grew and expectations rose, we knew our multiplayer system needed to be scaled. It had to be faster. Fairer. The kind of system that doesn't break under pressure, whether it's lakhs of players logging in during a live event or two friends battling it out over spotty mobile data. This blog is our story of getting there. It's about how we rethought everything, from how we match players to how we run thousands of servers all at once. You'll get a look at the roadblocks we hit, the calls we had to make, and how open-source tech like Open Match and Agones helped us build something truly scalable. Building for the Next Era of MultiplayerThe Challenge Our initial multiplayer setup used a simple queue and a fixed pool of servers, which worked well. As adoption grew and large-scale events became frequent, the system was unable to manage the load. We had to move from basic matchmaking to skill-based fairness, surge handling, safe releases, and cloud portability. And as we scaled, familiar pain points surfaced: High P95 time-to-match at peak; cancellations rose Unfair pairings in upper tiers; noticeable re-matches Idle server waste off-peak; slow warm-up during spikes Risky rollouts without clean canaries; limited on-call visibility The challenge was clear: we needed to decouple matchmaking logic from

## SOTA OCR with Core ML and dots.ocr

DevFeed: [SOTA OCR with Core ML and dots.ocr](<https://devfeed.tech/articles/sota-ocr-with-core-ml-and-dots-ocr-7174.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/dots-ocr-ne>)

Author: Christopher Fleetwood; Pedro Cuenca

Published: 2025-10-02T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [MLX](<https://devfeed.tech/topics/mlx.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [apple](<https://devfeed.tech/tags/apple.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [battery](<https://devfeed.tech/tags/battery.md>), [coreml](<https://devfeed.tech/tags/coreml.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [developers](<https://devfeed.tech/tags/developers.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [images](<https://devfeed.tech/tags/images.md>), [mlx](<https://devfeed.tech/tags/mlx.md>), [model](<https://devfeed.tech/tags/model.md>), [ocr](<https://devfeed.tech/tags/ocr.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [precision](<https://devfeed.tech/tags/precision.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [repo](<https://devfeed.tech/tags/repo.md>), [tools](<https://devfeed.tech/tags/tools.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

This tutorial explains how to convert dots.ocr from PyTorch to Core ML for on-device execution on Apple hardware. It discusses the roles of the Neural Engine, GPU, MLX, and Core ML, then outlines a staged conversion process beginning with GPU execution, FLOAT32 precision, and static shapes.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Running Serverless Wasm Functions on the Edge with k3s and SpinKube

DevFeed: [Running Serverless Wasm Functions on the Edge with k3s and SpinKube](<https://devfeed.tech/articles/running-serverless-wasm-functions-on-the-edge-with-k3s-and-spinkube-15381.md>)

Original publisher: [Read original article](<https://www.fermyon.com/blog/spinkube-k3s>)

Author: Matt Butcher

Published: 2025-07-22T12:00:00Z

Content type: article

Language: en

Sources: [Fermyon - Experience the next wave of cloud computing.](<https://devfeed.tech/sources/fermyon-experience-the-next-wave-of-cloud-computing.md>)

Topics: [k3s](<https://devfeed.tech/topics/k3s.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [wasm](<https://devfeed.tech/topics/wasm.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [rancher](<https://devfeed.tech/topics/rancher.md>), [Edge](<https://devfeed.tech/topics/edge.md>)

Tags: [edge](<https://devfeed.tech/tags/edge.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [k3s](<https://devfeed.tech/tags/k3s.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [rancher](<https://devfeed.tech/tags/rancher.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [wasm](<https://devfeed.tech/tags/wasm.md>)

### AI overview

The article examines running SpinKube with Rancher k3s, a lightweight Kubernetes distribution, to run high-performance serverless WebAssembly functions at the edge.

### Source excerpt

We look at running SpinKube with SUSE's lightweight Kubernetes distribution Rancher k3s - to run high performance serverless even the edge.

## Optimizing Apache Iceberg tables for real-time analytics

DevFeed: [Optimizing Apache Iceberg tables for real-time analytics](<https://devfeed.tech/articles/optimizing-apache-iceberg-tables-for-real-time-analytics-18585.md>)

Original publisher: [Read original article](<https://www.tinybird.co/blog/optimizing-apache-iceberg-tables-for-real-time-analytics>)

Author: Alberto Romeu

Published: 2025-06-03T10:00:00Z

Content type: tutorial

Language: en

Sources: [Tinybird](<https://devfeed.tech/sources/tinybird.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Apache Iceberg tables](<https://devfeed.tech/topics/apache-iceberg-tables.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Sorting](<https://devfeed.tech/topics/sorting.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [performance](<https://devfeed.tech/tags/performance.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scalable-analytics-architecture](<https://devfeed.tech/tags/scalable-analytics-architecture.md>), [sorting](<https://devfeed.tech/tags/sorting.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

A tutorial on using Apache Iceberg partitioning, sorting, and compaction features to build high-performance real-time analytics systems.

### Source excerpt

Learn how to use Iceberg's partitioning, sorting, and compaction features to build high-performance real-time analytics systems

## Designing high-performance financial ledgers with Temporal

DevFeed: [Designing high-performance financial ledgers with Temporal](<https://devfeed.tech/articles/designing-high-performance-financial-ledgers-with-temporal-35770.md>)

Original publisher: [Read original article](<https://temporal.io/blog/designing-high-performance-financial-ledgers-with-temporal>)

Author: Tim Imkin

Published: 2025-03-18T00:00:00Z

Content type: article

Language: en

Sources: [Temporal Blog](<https://devfeed.tech/sources/temporal-blog.md>)

Topics: [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>), [integrity](<https://devfeed.tech/topics/integrity.md>)

Tags: [accounting](<https://devfeed.tech/tags/accounting.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [crashes](<https://devfeed.tech/tags/crashes.md>), [design](<https://devfeed.tech/tags/design.md>), [financial](<https://devfeed.tech/tags/financial.md>), [fintech](<https://devfeed.tech/tags/fintech.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [ledger](<https://devfeed.tech/tags/ledger.md>), [payment](<https://devfeed.tech/tags/payment.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [temporal-concepts](<https://devfeed.tech/tags/temporal-concepts.md>)

### AI overview

The article discusses designing high-performance financial ledgers around scalable transaction handling, double-entry bookkeeping, flexible metadata, and resilient workflow coordination. It presents Temporal's durable execution as a way to address failures caused by network issues, service downtime, and process crashes.

### Source excerpt

Learn how Temporal's durable execution solves core challenges -- scalability, reconciliation, and compliance -- to create reliable, future-proof ledger systems.

## Novel Uses of Core Java for Low-Latency and High-Performance Systems

DevFeed: [Novel Uses of Core Java for Low-Latency and High-Performance Systems](<https://devfeed.tech/articles/novel-uses-of-core-java-for-low-latency-and-high-performance-systems-30745.md>)

Original publisher: [Read original article](<http://blog.vanillajava.blog/2024/12/novel-uses-of-core-java-for-low-latency.html>)

Author: Peter Lawrey (noreply@blogger.com)

Published: 2024-12-09T16:24:00Z

Content type: article

Language: en

Sources: [Vanilla Java](<https://devfeed.tech/sources/vanilla-java.md>)

Topics: [Java](<https://devfeed.tech/topics/java.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Code](<https://devfeed.tech/topics/code.md>), [Exception](<https://devfeed.tech/topics/exception.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>)

Tags: [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [exception](<https://devfeed.tech/tags/exception.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [info](<https://devfeed.tech/tags/info.md>), [java](<https://devfeed.tech/tags/java.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [opinion](<https://devfeed.tech/tags/opinion.md>), [performance](<https://devfeed.tech/tags/performance.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

This article examines unconventional Core Java techniques for low-latency and high-performance systems. It discusses capturing stack traces without throwing exceptions, system-wide unique timestamps, trivially copyable data types, zero-garbage strategies, and conditional diagnostic tracing, with attention to performance, determinism, and production use.

### Source excerpt

Standard Java libraries and idioms may only sometimes suffice in high-performance and low-latency Java systems. This article explores unconventional yet practical techniques that push Core Java to its limits, focusing on performance, diagnostics, and determinism. Drawing on experiences from building ultra-low-latency libraries and infrastructure, we will highlight patterns such as capturing stack traces without exceptions, system-wide unique timestamps, "trivially copyable" data types, zero-garbage strategies, and more. We will also discuss lessons from applying these approaches in production environments, where nanosecond-level considerations are the norm. This is taken from the transcript for Novel Uses of Core Java for Low-Latency and High-Performance Systems Moderated by Melissa McKay. Introduction Developers often rely on standard Java idioms--throwable hierarchies, BigDecimal for financial calculations, thread-local resources, or off-the-shelf message queues. While straightforward, these approaches can impose unwanted latency, garbage generation, or diagnostic blind spots. Even microseconds matter in low-latency trading, market data processing, or other time-sensitive domains. We aim to present "novel uses" of Core Java that remove such bottlenecks, allowing developers to produce cleaner, more deterministic, and more insightful code. These techniques are not always common knowledge, yet they can deliver substantial benefits in the right context. Capturing Stack Traces Without Exceptions Most Java developers assume that Throwable subclasses like Exception or Error must represent actual errors. However, you can extend Throwable to capture a stack trace at any point, even without throwing it. public final class StackTrace extends Throwable { // Intentionally extends Throwable but never thrown } Why do this? Treating a stack trace as a standalone data structure lets you record where and when certain critical events occur. For example, if a resource is closed premat

## Trivially Copyable Objects in Java

DevFeed: [Trivially Copyable Objects in Java](<https://devfeed.tech/articles/trivially-copyable-objects-in-java-30748.md>)

Original publisher: [Read original article](<http://blog.vanillajava.blog/2024/12/trivially-copyable-objects-in-java.html>)

Author: Peter Lawrey (noreply@blogger.com)

Published: 2024-12-09T15:38:00Z

Content type: tutorial

Language: en

Sources: [Vanilla Java](<https://devfeed.tech/sources/vanilla-java.md>)

Topics: [Java](<https://devfeed.tech/topics/java.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [java](<https://devfeed.tech/tags/java.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [opinion](<https://devfeed.tech/tags/opinion.md>), [performance](<https://devfeed.tech/tags/performance.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

This article explains how Java objects composed only of fixed-size primitive fields can emulate C++ trivially copyable objects for more efficient serialization. It describes reducing object graph traversal, reflection, and per-field copying through bulk memory operations, and discusses using Chronicle Bytes when strict trivial copyability is unnecessary. The approach trades flexibility for lower latency and higher throughput in latency-sensitive systems.

### Source excerpt

TL;DR Problem: Java's standard serialisation can be slow due to scattered object fields and reflection-based overhead. Approach: Emulate C++-style trivially copyable objects by restricting fields to primitives, enabling bulk memory copies. Result: Near C++-like serialisation performance, dramatically reducing latency and improving throughput. Trade-offs: Requires careful design, limited flexibility, and testing for JVM compatibility. Outcome: Low-latency systems with high performance, suitable for financial data feeds, real-time analytics, and other latency-sensitive domains. Introduction For low-latency systems, every microsecond has tangible business impact. In high-frequency trading, real-time analytics, and similarly time-sensitive workloads, even minor inefficiencies in serialisation and deserialisation can degrade throughput and responsiveness. The seemingly mundane act of converting objects into bytes and back often becomes a performance bottleneck. This article explores how we can emulate a C++-like concept of Trivially Copyable Objects within Java to achieve far more efficient serialisation. By ensuring objects contain only fixed-size primitives, we can sidestep the traditional overheads of object graph traversal, reflection, and per-field copying. Instead, we can treat them as contiguous memory blocks, dramatically reducing the time taken to read and write data. We shall also consider how to get very close to this performance using Chronicle Bytes without strictly requiring trivial copyability. This technique blends the low-level efficiency with the safety and familiarity of Java's ecosystem. The Core Challenge of Java Serialisation Most Java object graphs are composed of references linking scattered heap allocations. Serialising such objects typically involves visiting numerous memory locations, reading each field individually, and writing them out one at a time. This is akin to foraging around a warehouse for individual items whenever you need to pack a

## .NET Support on Heroku

DevFeed: [.NET Support on Heroku](<https://devfeed.tech/articles/net-support-on-heroku-26397.md>)

Original publisher: [Read original article](<https://www.heroku.com/blog/dotnet-support-on-heroku/>)

Author: Rune Soerensen

Published: 2024-12-03T00:00:00Z

Content type: release

Language: en

Sources: [Heroku](<https://devfeed.tech/sources/heroku.md>)

Topics: [Heroku](<https://devfeed.tech/topics/heroku.md>), [.NET](<https://devfeed.tech/topics/net.md>), [ASP.NET Core](<https://devfeed.tech/topics/asp-net-core.md>), [Blazor](<https://devfeed.tech/topics/blazor.md>), [C#](<https://devfeed.tech/topics/csharp.md>), [F#](<https://devfeed.tech/topics/fsharp.md>), [Visual Basic](<https://devfeed.tech/topics/visual-basic.md>), [cross-platform](<https://devfeed.tech/topics/cross-platform.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Databases](<https://devfeed.tech/topics/databases.md>)

Tags: [announce](<https://devfeed.tech/tags/announce.md>), [apis](<https://devfeed.tech/tags/apis.md>), [app](<https://devfeed.tech/tags/app.md>), [asp-net](<https://devfeed.tech/tags/asp-net.md>), [asp-net-core](<https://devfeed.tech/tags/asp-net-core.md>), [blazor](<https://devfeed.tech/tags/blazor.md>), [buildpacks](<https://devfeed.tech/tags/buildpacks.md>), [c-sharp](<https://devfeed.tech/tags/c-sharp.md>), [can](<https://devfeed.tech/tags/can.md>), [community](<https://devfeed.tech/tags/community.md>), [core](<https://devfeed.tech/tags/core.md>), [cross-platform](<https://devfeed.tech/tags/cross-platform.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [developers](<https://devfeed.tech/tags/developers.md>), [f-sharp](<https://devfeed.tech/tags/f-sharp.md>), [heroku](<https://devfeed.tech/tags/heroku.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [languages](<https://devfeed.tech/tags/languages.md>), [maintainability](<https://devfeed.tech/tags/maintainability.md>), [net](<https://devfeed.tech/tags/net.md>), [news](<https://devfeed.tech/tags/news.md>), [nextgen](<https://devfeed.tech/tags/nextgen.md>), [platform](<https://devfeed.tech/tags/platform.md>)

### AI overview

Heroku announces that official .NET support is entering public beta. Developers can build and deploy C#, F#, and Visual Basic applications using frameworks including ASP.NET Core and Blazor. The support includes a buildpack, documentation, testing, and guidance for deploying applications, including Blazor apps with Heroku Postgres.

### Source excerpt

We're excited to announce that official support for .NET on Heroku is entering public beta starting today. Developers can now build and deploy applications in C#, F#, and Visual Basic, using frameworks like ASP.NET Core and Blazor, all with the simplicity and flexibility of the Heroku platform. .NET has long been one of the most [...] The post .NET Support on Heroku appeared first on Heroku.

## Kafka for System Design Interviews

DevFeed: [Kafka for System Design Interviews](<https://devfeed.tech/articles/kafka-for-system-design-interviews-32314.md>)

Original publisher: [Read original article](<https://evanking1.medium.com/kafka-for-system-design-interviews-f031ac05d186?source=rss-9736778727ef------2>)

Author: Evan King

Published: 2024-07-01T03:18:09Z

Content type: tutorial

Language: en

Sources: [Evan King](<https://devfeed.tech/sources/evan-king.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Apache-Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [durability](<https://devfeed.tech/tags/durability.md>), [google](<https://devfeed.tech/tags/google.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [meta](<https://devfeed.tech/tags/meta.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [system-design-interview](<https://devfeed.tech/tags/system-design-interview.md>)

### AI overview

A tutorial on Apache Kafka for system design interviews. It explains Kafka as an open-source distributed event streaming platform that can function as a message queue or stream processing system, then uses a real-time sports statistics example to discuss producers, consumers, scaling, and event ordering.

### Source excerpt

By the Co-founder of www.hellointerview.comIntro There is a good chance you've heard of Kafka. It's popular. In fact, according to their website, it's used by 80% of the Fortune 100. If it's good enough to help scale the largest companies in the world, it's probably good enough for your next system design interview. In fact, it's one of the top 5 technologies we see used in design interviews. Apache Kafka is an open-source distributed event streaming platform that can be used either as a message queue or as a stream processing system. Kafka excels in delivering high performance, scalability, and durability. It's engineered to handle vast volumes of data in real-time, ensuring that no message is ever lost and that each piece of data is processed as swiftly as possible. In this deep dive, we're going to take a top down approach. Starting with a zoomed out view of Kafka and progressing into more and more detail. If you know the basics, feel free to skip ahead to the more advanced sections. A Motivating Example It's the World Cup (my personal favorite competition). And we run a website that provides real-time statistics on the matches. Each time a goal is scored, a player is booked, or a substitution is made, we want to update our website with the latest information. Events are placed on a queue when they occur. We call the server or process responsible for putting these events on the queue the producer. Downstream, we have a server that reads events off the queue and updates the website. We call this the consumer. Now, imagine the World Cup expanded from just the top 48 teams to a hypothetical 1,000-team tournament, and all the games are now played at the same time. The number of events has increased significantly, and our single server hosting the queue is struggling to keep up. Similarly, our consumer feels like it has its mouth under a firehose and is crashing under the load. We need to scale the system by adding more servers to distribute our queue. But how do we e

## Lessons Learned Running Presto at Meta Scale

DevFeed: [Lessons Learned Running Presto at Meta Scale](<https://devfeed.tech/articles/lessons-learned-running-presto-at-meta-scale-27907.md>)

Original publisher: [Read original article](<http://highscalability.com/blog/2023/7/16/lessons-learned-running-presto-at-meta-scale.html>)

Author: Philip Bell

Published: 2023-07-16T16:42:21Z

Content type: tutorial

Language: en

Sources: [High Scalability](<https://devfeed.tech/sources/high-scalability.md>), [High Scalability](<https://devfeed.tech/sources/high-scalability-2.md>)

Topics: [SQL](<https://devfeed.tech/topics/sql.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [gateway](<https://devfeed.tech/topics/gateway.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [releases](<https://devfeed.tech/topics/releases.md>), [data centers](<https://devfeed.tech/topics/data-centers.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [automation](<https://devfeed.tech/tags/automation.md>), [availability](<https://devfeed.tech/tags/availability.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [database-scalability](<https://devfeed.tech/tags/database-scalability.md>), [ddatabase](<https://devfeed.tech/tags/ddatabase.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-availablilty](<https://devfeed.tech/tags/high-availablilty.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [high-scalability](<https://devfeed.tech/tags/high-scalability.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [presto](<https://devfeed.tech/tags/presto.md>), [releases](<https://devfeed.tech/tags/releases.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

This article presents lessons from running Presto, an open source SQL query engine, at Meta scale. It discusses deploying new releases across many clusters, maintaining availability during updates, routing queries through the Gateway, and automating cluster operations.

### Source excerpt

Presto is a free, open source SQL query engine. We've been using it at Meta for the past ten years, and learned a lot while doing so. Running anything at scale - tools, processes, services - takes problem solving to overcome unexpected challenges. Here are four things we learned while scaling up Presto to Meta scale, and some advice if you're interested in running your own queries at scale. Scaling Presto rapidly to meet growing demands: What challenges did we face? Deploying new Presto releases

## Lessons Learned Running Presto at Meta Scale

DevFeed: [Lessons Learned Running Presto at Meta Scale](<https://devfeed.tech/articles/lessons-learned-running-presto-at-meta-scale-33604.md>)

Original publisher: [Read original article](<https://highscalability.com/lessons-learned-running-presto-at-meta-scale/>)

Author: Philip Bell

Published: 2023-07-16T15:42:21Z

Content type: article

Language: en

Sources: [High Scalability](<https://devfeed.tech/sources/high-scalability-3.md>)

Topics: [presto](<https://devfeed.tech/topics/presto.md>), [Meta](<https://devfeed.tech/topics/meta.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [gateway](<https://devfeed.tech/topics/gateway.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [datacenter](<https://devfeed.tech/topics/datacenter.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [automation](<https://devfeed.tech/tags/automation.md>), [availability](<https://devfeed.tech/tags/availability.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [database-scalability](<https://devfeed.tech/tags/database-scalability.md>), [ddatabase](<https://devfeed.tech/tags/ddatabase.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [high-availablilty](<https://devfeed.tech/tags/high-availablilty.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [high-scalability](<https://devfeed.tech/tags/high-scalability.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [presto](<https://devfeed.tech/tags/presto.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

The article presents lessons from running Presto, an open source SQL query engine, at Meta scale. It discusses deploying releases across many clusters, maintaining availability for interactive queries, using the Gateway load balancer to route queries, and automating cluster updates and capacity management.

### Source excerpt

Presto is a free, open source SQL query engine. We've been using it at Meta for the past ten years, and learned a lot while doing so. Running anything at scale - tools, processes, services - takes problem solving to overcome unexpected challenges. Here are four things we learned while

## Great Rust Libraries / Tools

DevFeed: [Great Rust Libraries / Tools](<https://devfeed.tech/articles/great-rust-libraries-tools-28317.md>)

Original publisher: [Read original article](<http://fuzzyblog.io/blog/rust/2020/02/21/great-rust-libraries-tools.html>)

Author: Fuzzygroup

Published: 2020-02-21T00:00:00Z

Content type: article

Language: en

Sources: [Scott Johnson](<https://devfeed.tech/sources/scott-johnson.md>)

Topics: [Rust](<https://devfeed.tech/topics/rust.md>), [Tooling](<https://devfeed.tech/topics/tooling.md>), [client library](<https://devfeed.tech/topics/client-library.md>), [Database](<https://devfeed.tech/topics/database.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Erlang](<https://devfeed.tech/topics/erlang.md>)

Tags: [client-library](<https://devfeed.tech/tags/client-library.md>), [database](<https://devfeed.tech/tags/database.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [libraries](<https://devfeed.tech/tags/libraries.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [rust](<https://devfeed.tech/tags/rust.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The author introduces Rust after years of primarily using Ruby and describes several Rust libraries and tools, including a Reddit client, string matcher, Kafka client, filesystem watchers, an embedded database, and a distributed runtime. The article highlights Rust's performance and cites benchmark figures for Sled.

### Source excerpt

I don't often do the dance with new languages. I suspect you could view my approach to languages as monogamy - I've been in a long term, monogamous relationship with Ruby now since 2007. But 12 years is a long time and deciding to dance with Python this past fall was perhaps my break through event away from "language monogamy". I'm now happily dating other languages and the latest one to turn my head is Rust. I've been very impressed with the quality of the core tooling around Rust and here are some of those tools: https://github.com/IntrepidPig/orca - Reddit client library https://github.com/BurntSushi/aho-corasick - String / Substring matcher https://crates.io/crates/kafka - Kafka client https://github.com/notify-rs/notify - Filesystem watcher; this is what drew me to using Rust; I have a high performance application where I'm virtually certain Ruby is going to fall over and die. Rust gives me an alternative. https://github.com/francesca64/hotwatch - make notify easier to use http://eradman.com/entrproject/ - A filesystem watcher built on top of Rust; possible alternative. https://github.com/watchexec/watchexec - Another filesystem watcher built on top of Rust; possible alternative. https://github.com/spacejam/sled - An embedded database built in Rust https://github.com/bastion-rs/bastion - Seemingly an Erlang like distributed runtime; fascinating AF if it is true One of the reasons you use Rust is performance and the metrics on Sled are pretty impressive: over a billion operations in under a minute at 95% read 5% writes on 16 cores on a small dataset a crash-safe monotonic ID generator capable of generating 75-125 million unique ID's per second Sources / Things to Read Getting Started with Rust Rust and Ruby Rust and Kafka Rust File Watcher Textmate 2 and Rust Rust Package Index

## How trivago Reduced Memcached Memory Usage by 50%

DevFeed: [How trivago Reduced Memcached Memory Usage by 50%](<https://devfeed.tech/articles/how-trivago-reduced-memcached-memory-usage-by-50-27956.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2017-12-19-memcached-optimization/>)

Author: Ivana Petrovic Software backend developer; Optimisation; Clean Architecture

Published: 2017-12-19T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [Caching](<https://devfeed.tech/topics/caching.md>), [Code](<https://devfeed.tech/topics/code.md>), [PHP](<https://devfeed.tech/topics/php.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [Network](<https://devfeed.tech/topics/network.md>), [Database](<https://devfeed.tech/topics/database.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [errors](<https://devfeed.tech/tags/errors.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [logging](<https://devfeed.tech/tags/logging.md>), [memcached](<https://devfeed.tech/tags/memcached.md>), [memory](<https://devfeed.tech/tags/memory.md>), [network](<https://devfeed.tech/tags/network.md>), [performance](<https://devfeed.tech/tags/performance.md>), [php](<https://devfeed.tech/tags/php.md>), [profiling](<https://devfeed.tech/tags/profiling.md>), [uptime](<https://devfeed.tech/tags/uptime.md>)

### AI overview

trivago describes how a Memcached cache failure caused database overload and platform downtime during a botnet-driven spike in spider traffic. The team added cache-value-size logging, used Blackfire profiling, and found that a method was caching about 10 MB of data and being called 30 times per page load.

### Source excerpt

One day, Memcached ran out of free memory. The method `get` failed and all requests went directly to the database. Of course these calls also failed under the huge load, and eventually it caused downtime for the whole trivago website. Yikes!

## Speaking at Tech Mesh

DevFeed: [Speaking at Tech Mesh](<https://devfeed.tech/articles/speaking-at-tech-mesh-30648.md>)

Original publisher: [Read original article](<http://bad-concurrency.blogspot.com/2012/11/speaking-at-tech-mesh.html>)

Author: Michael Barker (noreply@blogger.com)

Published: 2012-11-07T23:15:00Z

Content type: opinion

Language: en

Sources: [Bad Concurrency](<https://devfeed.tech/sources/bad-concurrency.md>)

Topics: [Development](<https://devfeed.tech/topics/development.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [announce](<https://devfeed.tech/tags/announce.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [disruptor](<https://devfeed.tech/tags/disruptor.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [history](<https://devfeed.tech/tags/history.md>), [jvm](<https://devfeed.tech/tags/jvm.md>), [performance](<https://devfeed.tech/tags/performance.md>), [perspectives](<https://devfeed.tech/tags/perspectives.md>), [speaking](<https://devfeed.tech/tags/speaking.md>), [systems](<https://devfeed.tech/tags/systems.md>), [tech](<https://devfeed.tech/tags/tech.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

The author announces an upcoming Tech Mesh talk about the Disruptor, covering its history and motivations, the challenges of building high-performance concurrent systems, and how JVM and hardware changes could support them.

### Source excerpt

I'm happy to announce that I will speaking at Tech Mesh in December. I'll be speaking about the Disruptor from two perspectives, firstly looking briefly back at some of the history and motivations behind the Disruptor. Then spending some time explaining at the challenges of building high performance concurrent systems (like the Disruptor) and delving into how the JVM and hardware could change to support the development of these systems.