# William Kennedy

Recent content in Ardan Labs Blog on

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## RAG in Go: A Vulnerability Research Tool

DevFeed: [RAG in Go: A Vulnerability Research Tool](<https://devfeed.tech/articles/rag-in-go-a-vulnerability-research-tool-22285.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2026/04/rag-in-go-a-vulnerability-research-tool/>)

Published: 2026-04-20T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [vector embeddings in AI](<https://devfeed.tech/topics/vector-embeddings-in-ai.md>), [LanceDB vector database](<https://devfeed.tech/topics/lancedb-vector-database.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Vulnerabilities](<https://devfeed.tech/topics/vulnerabilities.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>), [information retrieval](<https://devfeed.tech/topics/information-retrieval.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [cve](<https://devfeed.tech/tags/cve.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [database-schema](<https://devfeed.tech/tags/database-schema.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embedding-model](<https://devfeed.tech/tags/embedding-model.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [json](<https://devfeed.tech/tags/json.md>), [llm](<https://devfeed.tech/tags/llm.md>), [openai-api](<https://devfeed.tech/tags/openai-api.md>), [programming](<https://devfeed.tech/tags/programming.md>), [rag](<https://devfeed.tech/tags/rag.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [retrieval-augmented-generation](<https://devfeed.tech/tags/retrieval-augmented-generation.md>), [vector](<https://devfeed.tech/tags/vector.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>), [vulnerability-research](<https://devfeed.tech/tags/vulnerability-research.md>)

### AI overview

The post demonstrates a Go RAG workflow using the Go Vulnerability Database: ingestion parses vulnerability JSON, embeds it, and stores the CVE text and ID; search embeds a user query to retrieve relevant records. It uses Kronk Model Server through an OpenAI-compatible connection, with a mock API key, and DuckDB's vector support so the database can be embedded in the Go program without a separate server. The author emphasizes designing the schema around expected queries and access rules, such as filtering records by tenant. The supplied excerpt ends during the vulnerability struct implementation, so it does not show the complete search code or results.

### Source excerpt

Introduction In the previous post, you saw how you can use tools to add information to an LLM query. In this post, we'll see another method of adding information to an LLM called RAG, or Retrieval-Augmented Generation. The idea of RAG is that you want the LLM to have access to information that wasn't available to it when it was initially trained. You do it by storing documents in your own database along with their embedding. I won't go into the technical details of embedding, but think of it as a way to convert a piece of text into a vector. The magic is that if two pieces of text have similar meaning, an embedding model can create vectors that mathematically show they are similar.

## Using Tools: A Meeting Scheduler

DevFeed: [Using Tools: A Meeting Scheduler](<https://devfeed.tech/articles/using-tools-a-meeting-scheduler-22284.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2026/03/using-tools-a-meeting-scheduler/>)

Published: 2026-03-16T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [function calling](<https://devfeed.tech/topics/function-calling.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [llamaindex](<https://devfeed.tech/topics/llamaindex.md>)

Tags: [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [data](<https://devfeed.tech/tags/data.md>), [function-calling](<https://devfeed.tech/tags/function-calling.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [llms](<https://devfeed.tech/tags/llms.md>), [meetings](<https://devfeed.tech/tags/meetings.md>), [programming](<https://devfeed.tech/tags/programming.md>), [rag](<https://devfeed.tech/tags/rag.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The excerpt explains how function calling lets an LLM use changing or private data: provide tool descriptions, let the model return a call with parameters, execute it, and send the result back. A query may require multiple agent-LLM round trips before a final answer. The example builds an agent to suggest meeting times using a tool that returns a user's meetings on a given date. The author uses Kronk locally as a model server and SDK, noting that Ollama, OpenAI, Claude, and other systems can also be used. The excerpt begins to describe the system prompt and tool definition but omits their actual contents and parts of the setup.

### Source excerpt

Introduction LLMs are great, but they are trained on public data sets. In some cases, you need the LLM to use data that's not publicly available or that's frequently changing. There are several ways to make such data available to LLMs: Tool/function calls Retrieval-augmented generation (aka RAG) MCP In coding agents, you can also add skills. In this post we'll focus on function calling. How Does It Work? When interacting with an LLM, you can provide a description of available tools if the model supports tool calling. If the LLM reasons that the best answer is to use one of the tools, it will return a reply that contains a tool call with the parameters to use. Then you make the function call and return the answer back to the LLM.

## Range-Over Functions in Go

DevFeed: [Range-Over Functions in Go](<https://devfeed.tech/articles/range-over-functions-in-go-22234.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/04/range-over-functions-in-go.html>)

Published: 2026-02-16T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [iteration](<https://devfeed.tech/topics/iteration.md>)

Tags: [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [control](<https://devfeed.tech/tags/control.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [features](<https://devfeed.tech/tags/features.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [linked-list](<https://devfeed.tech/tags/linked-list.md>), [model](<https://devfeed.tech/tags/model.md>), [package](<https://devfeed.tech/tags/package.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [programming](<https://devfeed.tech/tags/programming.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [syntax](<https://devfeed.tech/tags/syntax.md>)

### AI overview

The range-over-functions experiment proposes a standard iterator protocol for Go while retaining familiar `for range` syntax. The excerpt motivates it by showing that custom container iteration otherwise requires ad hoc patterns and that storing iteration state on a container complicates concurrent traversals. Its stack example instead gives each traversal its own iterator, which tracks the current node and returns the next value. The excerpt mentions the `iter` package and sequence abstractions, but cuts off before showing their use, so it does not support further implementation details.

### Source excerpt

Iteration has long been one of the more fragmented areas of Go, with developers relying on ad hoc patterns to traverse custom data structures. This article explores the range-over-functions experiment, a proposed evolution of the language that introduces a standardized iterator model while preserving Go's familiar for range syntax. Using the new iter package and sequence abstractions, it shows how iteration logic can be expressed more clearly, flexibly, and idiomatically. Originally published in April 2024, the concepts remain highly relevant as Go continues to evolve toward more expressive yet simple language features.

## Query Database Using Plain English

DevFeed: [Query Database Using Plain English](<https://devfeed.tech/articles/query-database-using-plain-english-22283.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2026/02/query-database-using-plain-english/>)

Published: 2026-02-11T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Code](<https://devfeed.tech/topics/code.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Ollama](<https://devfeed.tech/topics/ollama.md>), [context](<https://devfeed.tech/topics/context.md>), [GitHub](<https://devfeed.tech/topics/github.md>), [API](<https://devfeed.tech/topics/api.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [code](<https://devfeed.tech/tags/code.md>), [context](<https://devfeed.tech/tags/context.md>), [database](<https://devfeed.tech/tags/database.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [github](<https://devfeed.tech/tags/github.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [llms](<https://devfeed.tech/tags/llms.md>), [ollama](<https://devfeed.tech/tags/ollama.md>), [programming](<https://devfeed.tech/tags/programming.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

This tutorial shows how to build a system that lets users query a relational database using plain English. It uses an LLM to generate SQL, executes the query against a database, and then uses the query results to generate an answer. The example uses the Austin Bike Share dataset, Kronk Model Server, and DuckDB, while noting that other model servers and SQL databases can be used.

### Source excerpt

Introduction In this post you'll see how you can create a system that allows users to query a relational database using plain English. This allows users not familiar with SQL or business intelligence systems to get insights from data. Setting Up If you want to follow along, you'll need to clone the code from the GitHub repo. This will download the code, and the database file containing the data (bikes.ddb) Note: The data is from the Austin Bike Share dataset.

## Kubernetes Memory Limits and Go

DevFeed: [Kubernetes Memory Limits and Go](<https://devfeed.tech/articles/kubernetes-memory-limits-and-go-22232.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/02/kubernetes-memory-limits-go.html>)

Published: 2026-01-27T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [rancher](<https://devfeed.tech/topics/rancher.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>)

Tags: [go](<https://devfeed.tech/tags/go.md>), [go-runtime](<https://devfeed.tech/tags/go-runtime.md>), [golang](<https://devfeed.tech/tags/golang.md>), [gomemlimit](<https://devfeed.tech/tags/gomemlimit.md>), [k8s](<https://devfeed.tech/tags/k8s.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [limits](<https://devfeed.tech/tags/limits.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [memory](<https://devfeed.tech/tags/memory.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [oom](<https://devfeed.tech/tags/oom.md>)

### AI overview

The excerpt frames Kubernetes memory limits as a source of Go pod OOMs and describes an experiment to investigate them. The author measures a service's memory under load using Go metrics exposed through expvar, aiming to find a level near the OOM boundary rather than choosing an arbitrarily low limit. The experiment asks whether GOMEMLIMIT set to the Kubernetes memory limit affects performance and how requests, limits, and GOMEMLIMIT might help prevent OOMs. The author cautions that the results may not apply to other services.

### Source excerpt

Kubernetes memory limits introduce a subtle but critical interaction with the Go runtime that can determine whether a service runs efficiently or fails under load. This article explores how Go manages memory under normal conditions and what changes when Kubernetes enforces hard memory constraints. Using controlled load testing, it shows why Go is generally excellent at self regulating memory, how OOM events emerge once limits are imposed, and how tools like GOMEMLIMIT can be used to align the runtime with container boundaries. Originally published in July 2024, the guidance remains highly relevant for teams running Go services in containerized environments where stability and throughput are tightly coupled to memory configuration.

## Everything You Wanted to Ask About Rust - Answered by Herbert Wolverson

DevFeed: [Everything You Wanted to Ask About Rust - Answered by Herbert Wolverson](<https://devfeed.tech/articles/everything-you-wanted-to-ask-about-rust-answered-by-herbert-wolverson-22282.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2026/01/everything-you-wanted-to-ask-about-rust-answered-by-herbert-wolverson/>)

Published: 2026-01-07T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Rust](<https://devfeed.tech/topics/rust.md>), [Memory Safety](<https://devfeed.tech/topics/memory-safety.md>), [Ardan Labs](<https://devfeed.tech/topics/ardan-labs.md>)

Tags: [advanced](<https://devfeed.tech/tags/advanced.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [beginner](<https://devfeed.tech/tags/beginner.md>), [blog](<https://devfeed.tech/tags/blog.md>), [book](<https://devfeed.tech/tags/book.md>), [box](<https://devfeed.tech/tags/box.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [build](<https://devfeed.tech/tags/build.md>), [c](<https://devfeed.tech/tags/c.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [c-sharp](<https://devfeed.tech/tags/c-sharp.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [generics](<https://devfeed.tech/tags/generics.md>), [github](<https://devfeed.tech/tags/github.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [jetbrains](<https://devfeed.tech/tags/jetbrains.md>), [livestream](<https://devfeed.tech/tags/livestream.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-management](<https://devfeed.tech/tags/memory-management.md>), [programming](<https://devfeed.tech/tags/programming.md>), [questions](<https://devfeed.tech/tags/questions.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

The excerpt introduces Herbert Wolverson's Rust livestream and summarizes its first four questions. His book Advanced Hands-On Rust follows Hands-On Rust into generics, macros, and library design. For developers coming from Java or C#, the main shift is Rust's explicit but safe memory model: ownership and smart pointers provide control without garbage collection and help prevent data races and use-after-free. Wolverson says Rust's learning curve has eased with friendlier syntax, libraries, and tooling, and recommends that newcomers can start with Rust directly rather than learning C first. The excerpt does not include the remaining livestream questions.

### Source excerpt

In a recent livestream with JetBrains, Vitaly Bragilevsky sat down with Herbert Wolverson, our Lead Rust Consultant and Instructor here at Ardan Labs, to talk about everything Rust developers - beginners and pros alike - are curious about. Watch the full livestream replay on the Ardan Labs Channel and check out the GitHub repo with Herbert's prepared answers and code samples. Below is a short summary of the 23 questions that were covered during the livestream.

## Getting Friendly With CPU Caches

DevFeed: [Getting Friendly With CPU Caches](<https://devfeed.tech/articles/getting-friendly-with-cpu-caches-22223.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2023/07/getting-friendly-with-cpu-caches.html>)

Published: 2025-10-21T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [CPU Cache](<https://devfeed.tech/topics/cpu-cache.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Jule](<https://devfeed.tech/topics/jule.md>)

Tags: [2023](<https://devfeed.tech/tags/2023.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [array](<https://devfeed.tech/tags/array.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cpu-caches](<https://devfeed.tech/tags/cpu-caches.md>), [cpus](<https://devfeed.tech/tags/cpus.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [go](<https://devfeed.tech/tags/go.md>), [golang](<https://devfeed.tech/tags/golang.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [locality](<https://devfeed.tech/tags/locality.md>), [memory](<https://devfeed.tech/tags/memory.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

The article explains that cache misses can make data access much slower than access from CPU cache, then introduces a Go case study: a user struct embeds a 16 KB icon array, and a benchmark counts active users by country. The supplied excerpt says replacing the embedded array with a slice improved performance by more than 40 times by reducing cache misses and improving data locality, but it cuts off during benchmark setup, so the detailed implementation and measurements are not available here.

### Source excerpt

Understanding how your data structures interact with hardware is one of the most powerful ways to improve application performance. This blogpost explores how CPU caches influence speed and how thoughtful struct design in Go can yield massive gains. Through a real-world case study, it shows how replacing a large embedded array with a slice improved performance by more than 40 times by reducing cache misses and improving data locality. Originally published in July 2023, its lessons remain highly relevant today for developers optimizing for memory efficiency and cache-aware programming.

## Kubernetes CPU Limits and Go

DevFeed: [Kubernetes CPU Limits and Go](<https://devfeed.tech/articles/kubernetes-cpu-limits-and-go-22231.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/02/kubernetes-cpu-limits-go.html>)

Published: 2025-09-22T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>)

Tags: [argocd](<https://devfeed.tech/tags/argocd.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cpu-throttling](<https://devfeed.tech/tags/cpu-throttling.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [go](<https://devfeed.tech/tags/go.md>), [golang](<https://devfeed.tech/tags/golang.md>), [k8s](<https://devfeed.tech/tags/k8s.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [limits](<https://devfeed.tech/tags/limits.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

The excerpt explains that a Kubernetes CPU limit expressed in millicores caps a service's CPU time: 250m is described as 25% of one CPU. The author investigates how such limits affect a Go service after finding that value in a deployment configuration and argues that Go services may run less efficiently when they are not configured for the limit. The excerpt introduces the topic but omits the detailed throttling mechanics and any configuration recommendations.

### Source excerpt

Kubernetes CPU limits can look straightforward on the surface, but their impact on application performance is anything but simple. This article unpacks how Go services interact with Kubernetes CPU throttling and why a seemingly harmless configuration such as setting a limit of 250m can dramatically constrain performance in production. First published in February 2024, its insights remain just as relevant today for anyone running Go applications in containerized environments and for developers who want to avoid costly slowdowns.

## Garbage Collection In Go : Part III - GC Pacing

DevFeed: [Garbage Collection In Go : Part III - GC Pacing](<https://devfeed.tech/articles/garbage-collection-in-go-part-iii-gc-pacing-22148.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2019/07/garbage-collection-in-go-part3-gcpacing.html>)

Published: 2025-08-26T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Go Language](<https://devfeed.tech/topics/go-language.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Code](<https://devfeed.tech/topics/code.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [code](<https://devfeed.tech/tags/code.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [memory](<https://devfeed.tech/tags/memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [programming](<https://devfeed.tech/tags/programming.md>)

### AI overview

This third article in a three-part series explains how Go's garbage collector paces itself according to workload demands. It uses sequential and concurrent examples to examine adaptive behavior and emphasizes reducing allocations per unit of work to reduce garbage-collection latency and improve application performance.

### Source excerpt

Go's garbage collector is designed not only to manage memory safely but also to pace itself intelligently, striking a balance between low latency and high throughput. This blogpost explores how the GC adapts its pace to workload demands, demonstrated through both sequential and concurrent program examples, and why reducing allocations per unit of work is the most effective way to lighten its load. Originally published in 2019, its core principles remain just as relevant today, offering Go developers a deeper understanding of the runtime's adaptive behavior and confidence that the GC can find the right rhythm without extensive manual tuning.

## Garbage Collection In Go : Part II - GC Traces

DevFeed: [Garbage Collection In Go : Part II - GC Traces](<https://devfeed.tech/articles/garbage-collection-in-go-part-ii-gc-traces-22145.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2019/05/garbage-collection-in-go-part2-gctraces.html>)

Published: 2025-07-14T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Go Language](<https://devfeed.tech/topics/go-language.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Web app](<https://devfeed.tech/topics/webapp.md>)

Tags: [2019](<https://devfeed.tech/tags/2019.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [gc](<https://devfeed.tech/tags/gc.md>), [generate](<https://devfeed.tech/tags/generate.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [heap](<https://devfeed.tech/tags/heap.md>), [latency](<https://devfeed.tech/tags/latency.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pprof](<https://devfeed.tech/tags/pprof.md>), [programming](<https://devfeed.tech/tags/programming.md>), [services](<https://devfeed.tech/tags/services.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

This excerpt explains how to investigate Go garbage collection overhead in a web application by generating and interpreting GC traces and application profiles. It carries forward the prior installment's conclusion that reducing heap stress lowers GC-related latency, and cautions that slowing collection cadence by extending the time between collections is not a good strategy; a consistent pace can better support performance. The excerpt introduces an RSS-search application used for the walkthrough, but ends before showing the trace and profile analysis or specific optimization steps.

### Source excerpt

Originally published in 2019, this article is part two of a three-part series exploring Go's garbage collector. Though the Go runtime has continued to evolve, the performance principles covered here remain highly relevant today. This installment focuses on practical techniques for analyzing and reducing garbage collection (GC) overhead in real-world Go applications. It walks through how to interpret GC traces using GODEBUG=gctrace=1 and uncover allocation hotspots with pprof, with a clear message: reducing unnecessary allocations, especially in tight loops, has a direct and measurable impact on latency and throughput. Whether you're optimizing high-performance services or just becoming GC-aware, the insights here are as applicable now as ever.

## Garbage Collection In Go : Part I - Semantics

DevFeed: [Garbage Collection In Go : Part I - Semantics](<https://devfeed.tech/articles/garbage-collection-in-go-part-i-semantics-22140.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html>)

Published: 2025-06-10T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Go Language](<https://devfeed.tech/topics/go-language.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [2018](<https://devfeed.tech/tags/2018.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [article](<https://devfeed.tech/tags/article.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [blog](<https://devfeed.tech/tags/blog.md>), [collector](<https://devfeed.tech/tags/collector.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [gc](<https://devfeed.tech/tags/gc.md>), [go](<https://devfeed.tech/tags/go.md>), [go-internals](<https://devfeed.tech/tags/go-internals.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [performance](<https://devfeed.tech/tags/performance.md>), [practical](<https://devfeed.tech/tags/practical.md>), [programming](<https://devfeed.tech/tags/programming.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [stop](<https://devfeed.tech/tags/stop.md>), [trace](<https://devfeed.tech/tags/trace.md>), [world](<https://devfeed.tech/tags/world.md>)

### AI overview

The article builds a practical mental model of Go's garbage collector around its behavior rather than runtime implementation details, which change across releases. For Go 1.12, it describes a non-generational, concurrent tri-color mark-and-sweep collector and introduces the semantics behind how it tracks heap allocations, reclaims unused memory, and retains live allocations. This is Part I of a three-part series; the excerpt says later parts cover GC traces and pacing.

### Source excerpt

This article was originally published in 2018, yet its core insights into Go's garbage collection model remain highly relevant for developers today. While some implementation details of Go's runtime have evolved, the foundational concepts explored here--such as the semantics of the tri-color mark and sweep algorithm, Stop The World (STW) events, and GC trace interpretation--are still essential to understanding how Go manages memory. Whether you're optimizing performance or deepening your knowledge of Go internals, this post continues to offer a clear and practical guide to working with the garbage collector, not against it.

## JSON - The Fine Print: Part 3 - Zero vs Missing Values

DevFeed: [JSON - The Fine Print: Part 3 - Zero vs Missing Values](<https://devfeed.tech/articles/json-the-fine-print-part-3-zero-vs-missing-values-22281.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2025/05/json-the-fine-print-part-3.html>)

Published: 2025-05-19T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [encoding/json](<https://devfeed.tech/topics/encoding-json.md>), [JSON](<https://devfeed.tech/topics/json.md>), [Data serialization](<https://devfeed.tech/topics/data-serialization.md>)

Tags: [api-layers](<https://devfeed.tech/tags/api-layers.md>), [common-serialization-errors](<https://devfeed.tech/tags/common-serialization-errors.md>), [data-serialization](<https://devfeed.tech/tags/data-serialization.md>), [data-structure-separation](<https://devfeed.tech/tags/data-structure-separation.md>), [data-transmission](<https://devfeed.tech/tags/data-transmission.md>), [data-validation](<https://devfeed.tech/tags/data-validation.md>), [encoding-json](<https://devfeed.tech/tags/encoding-json.md>), [go](<https://devfeed.tech/tags/go.md>), [go-json-types](<https://devfeed.tech/tags/go-json-types.md>), [go-language-serialization](<https://devfeed.tech/tags/go-language-serialization.md>), [go-language-validation](<https://devfeed.tech/tags/go-language-validation.md>), [go-type-mapping](<https://devfeed.tech/tags/go-type-mapping.md>), [json](<https://devfeed.tech/tags/json.md>), [json-best-practices](<https://devfeed.tech/tags/json-best-practices.md>), [json-go-mapping](<https://devfeed.tech/tags/json-go-mapping.md>), [json-marshaling](<https://devfeed.tech/tags/json-marshaling.md>), [json-unmarshaling](<https://devfeed.tech/tags/json-unmarshaling.md>), [json-validation](<https://devfeed.tech/tags/json-validation.md>), [missing-values](<https://devfeed.tech/tags/missing-values.md>), [passing-serialized-data](<https://devfeed.tech/tags/passing-serialized-data.md>), [rest-apis](<https://devfeed.tech/tags/rest-apis.md>), [schema-less-data-format](<https://devfeed.tech/tags/schema-less-data-format.md>), [serialization](<https://devfeed.tech/tags/serialization.md>), [serialization-mistakes](<https://devfeed.tech/tags/serialization-mistakes.md>), [serialization-protocols](<https://devfeed.tech/tags/serialization-protocols.md>), [unmarshalling](<https://devfeed.tech/tags/unmarshalling.md>), [values](<https://devfeed.tech/tags/values.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

Go's basic JSON unmarshalling leaves struct fields at their zero values when the corresponding JSON fields are absent, and it does not fail merely because fields are missing or unknown to the struct. This makes an omitted value indistinguishable from an explicitly supplied zero value. The excerpt also notes that `json.Decoder.DisallowUnknownFields` can reject JSON fields that have no matching struct field when exact input validation is needed. The text cuts off before showing how to distinguish an omitted field from an explicit zero.

### Source excerpt

Introduction In part 1 we took a higher level view on serialization in general and JSON in specific. In part 2 we looked at emitting JSON. In part 3, we'll look at an issue you might encounter when consuming JSON, Zero vs NULL field values. To clarify the definition of NULL, this means the absence of value. So here is the question: Given a field in a Go struct set to its zero value, how do you know that zero value was set by the user or it's zero because it was never provided?

## Scheduling In Go : Part III - Concurrency

DevFeed: [Scheduling In Go : Part III - Concurrency](<https://devfeed.tech/articles/scheduling-in-go-part-iii-concurrency-22142.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2018/12/scheduling-in-go-part3.html>)

Published: 2025-05-12T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Sorting](<https://devfeed.tech/topics/sorting.md>)

Tags: [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [programming](<https://devfeed.tech/tags/programming.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>)

### AI overview

The excerpt defines concurrency as executing instructions out of order while preserving the same result, and recommends first building a working sequential solution before deciding whether concurrency is practical. It distinguishes CPU-bound work, such as summing or sorting, from I/O-bound work, such as file reading: the described examples and benchmarks show parallelism improving CPU-bound performance, while concurrency alone benefits I/O-bound workloads. Identifying the workload type helps guide the engineering choice and avoid unnecessary complexity. This is an excerpt, so it does not establish the article's full treatment or detailed benchmark results.

### Source excerpt

Although originally written in 2018, the following concepts remain essential for developers working with concurrency. This blogpost focuses on concurrency, distinguishing it from parallelism by defining it as "out of order" execution. It emphasizes the importance of understanding workload types--CPU bound (e.g., summing, sorting) and IO bound (e.g., file reading)--to assess when concurrency is appropriate. Through practical examples and benchmarks, it shows that parallelism boosts performance for CPU bound tasks, while concurrency alone benefits IO bound workloads. The post underscores that identifying workload type is key to applying concurrency effectively without added complexity.

## Scheduling In Go : Part II - Go Scheduler

DevFeed: [Scheduling In Go : Part II - Go Scheduler](<https://devfeed.tech/articles/scheduling-in-go-part-ii-go-scheduler-22138.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part2.html>)

Published: 2025-04-14T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Go Language](<https://devfeed.tech/topics/go-language.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2018](<https://devfeed.tech/tags/2018.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [blog](<https://devfeed.tech/tags/blog.md>), [complex](<https://devfeed.tech/tags/complex.md>), [components](<https://devfeed.tech/tags/components.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [context](<https://devfeed.tech/tags/context.md>), [design](<https://devfeed.tech/tags/design.md>), [global](<https://devfeed.tech/tags/global.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [local](<https://devfeed.tech/tags/local.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [programming](<https://devfeed.tech/tags/programming.md>), [queue](<https://devfeed.tech/tags/queue.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [switching](<https://devfeed.tech/tags/switching.md>), [system](<https://devfeed.tech/tags/system.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

This excerpt introduces the Go scheduler's high-level model to help developers reason about concurrency. At startup, Go creates one logical processor (P) for each virtual core the host reports; with Hyper-Threading, hardware threads are counted as virtual cores. The post says it will cover global and local run queues, context switching, work stealing, and synchronous and asynchronous system calls, while emphasizing behavior over low-level mechanics. The supplied text ends during an example, so it does not provide the promised detailed explanations.

### Source excerpt

This blogpost is the second installment in a three-part series exploring the mechanics and semantics of the Go scheduler. Despite being published in 2018, the content remains relevant today, as the Go scheduler's design continues to influence the development of efficient and scalable concurrent systems. In this post, we will go into the inner workings of the Go scheduler, discussing its components, such as the Global Run Queue (GRQ) and Local Run Queue (LRQ), and its behavior, including context switching, work stealing, and the handling of synchronous and asynchronous system calls. By understanding these concepts, developers can make informed decisions about concurrency and optimization in their Go applications.

## Scheduling In Go : Part I - OS Scheduler

DevFeed: [Scheduling In Go : Part I - OS Scheduler](<https://devfeed.tech/articles/scheduling-in-go-part-i-os-scheduler-22137.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part1.html>)

Published: 2025-03-04T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Operating system](<https://devfeed.tech/topics/operating-system.md>), [NUMA](<https://devfeed.tech/topics/numa.md>)

Tags: [2018](<https://devfeed.tech/tags/2018.md>), [2025](<https://devfeed.tech/tags/2025.md>), [applications](<https://devfeed.tech/tags/applications.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [blog](<https://devfeed.tech/tags/blog.md>), [blogpost](<https://devfeed.tech/tags/blogpost.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [fundamentals](<https://devfeed.tech/tags/fundamentals.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [operating-system](<https://devfeed.tech/tags/operating-system.md>), [os](<https://devfeed.tech/tags/os.md>), [overview](<https://devfeed.tech/tags/overview.md>), [performant](<https://devfeed.tech/tags/performant.md>), [principles](<https://devfeed.tech/tags/principles.md>), [programming](<https://devfeed.tech/tags/programming.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [series](<https://devfeed.tech/tags/series.md>)

### AI overview

The excerpt frames efficient multithreaded Go design as depending on understanding how the Go scheduler works with the operating system scheduler. This first post in a three-part series focuses on OS scheduling and its mechanics and semantics, aiming to build a practical mental model for engineering decisions. It notes that OS schedulers account for hardware features such as multiple processors and cores, CPU caches, and NUMA, while saying a useful general model does not require deep study of those topics. The excerpt ends as it begins explaining how the OS executes a program's sequential machine instructions, so it does not provide the detailed scheduling mechanism.

### Source excerpt

Although this blogpost was originally published in 2018, the concepts and principles discussed remain crucial for building efficient and performant multithreaded applications in Go now in 2025 (Go 1.24.0). As the Go ecosystem continues to evolve, understanding how the Go scheduler interacts with the operating system scheduler is more important than ever. This three-part series provides a comprehensive overview of the mechanics and semantics behind Go's scheduling, starting with the fundamentals of the operating system scheduler.

## Context Package Semantics In Go

DevFeed: [Context Package Semantics In Go](<https://devfeed.tech/articles/context-package-semantics-in-go-22150.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2019/09/context-package-semantics-in-go.html>)

Published: 2025-02-10T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [API](<https://devfeed.tech/topics/api.md>)

Tags: [1-7](<https://devfeed.tech/tags/1-7.md>), [2014](<https://devfeed.tech/tags/2014.md>), [ardan-labs](<https://devfeed.tech/tags/ardan-labs.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [blog](<https://devfeed.tech/tags/blog.md>), [built-in](<https://devfeed.tech/tags/built-in.md>), [cancellation](<https://devfeed.tech/tags/cancellation.md>), [context](<https://devfeed.tech/tags/context.md>), [critical](<https://devfeed.tech/tags/critical.md>), [data](<https://devfeed.tech/tags/data.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [go](<https://devfeed.tech/tags/go.md>), [go-programming](<https://devfeed.tech/tags/go-programming.md>), [golang](<https://devfeed.tech/tags/golang.md>), [gotham](<https://devfeed.tech/tags/gotham.md>), [handler](<https://devfeed.tech/tags/handler.md>), [package](<https://devfeed.tech/tags/package.md>), [pitfalls](<https://devfeed.tech/tags/pitfalls.md>), [practices](<https://devfeed.tech/tags/practices.md>), [programming](<https://devfeed.tech/tags/programming.md>), [semantics](<https://devfeed.tech/tags/semantics.md>)

### AI overview

Go has no direct mechanism for terminating goroutines, so services need a way to manage timeouts and cancellation. The excerpt recommends creating a Context as early as possible while processing a request or task, which encourages APIs to accept Context as their first parameter. In the described server example, the incoming request already carries a Context in Go 1.7; with Go 1.8, the author says an empty Context would need to be created before that call. The excerpt ends during this example, so it does not provide the promised broader set of semantics or pitfalls.

### Source excerpt

Although first introduced in 2014, the Context package remains a crucial component of Go programming, enabling efficient management of request-scoped data, deadlines, and cancellation signals. As the Go ecosystem continues to evolve, understanding the Context package's semantics is vital for developing reliable and maintainable software. This blogpost provides an in-depth exploration of the Context package's semantics, highlighting best practices and common pitfalls to help developers effectively leverage this powerful tool.

## JSON - The Fine Print: Part 2 - Emitting JSON

DevFeed: [JSON - The Fine Print: Part 2 - Emitting JSON](<https://devfeed.tech/articles/json-the-fine-print-part-2-emitting-json-22267.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/10/json-the-fine-print-part-2.html>)

Published: 2025-02-04T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [JSON](<https://devfeed.tech/topics/json.md>), [encoding/json](<https://devfeed.tech/topics/encoding-json.md>), [Tutorial](<https://devfeed.tech/topics/tutorial.md>)

Tags: [api-layers](<https://devfeed.tech/tags/api-layers.md>), [apis](<https://devfeed.tech/tags/apis.md>), [common-serialization-errors](<https://devfeed.tech/tags/common-serialization-errors.md>), [data-serialization](<https://devfeed.tech/tags/data-serialization.md>), [data-structure-separation](<https://devfeed.tech/tags/data-structure-separation.md>), [data-transmission](<https://devfeed.tech/tags/data-transmission.md>), [data-validation](<https://devfeed.tech/tags/data-validation.md>), [encoding-json](<https://devfeed.tech/tags/encoding-json.md>), [go-json-types](<https://devfeed.tech/tags/go-json-types.md>), [go-language-serialization](<https://devfeed.tech/tags/go-language-serialization.md>), [go-language-validation](<https://devfeed.tech/tags/go-language-validation.md>), [go-type-mapping](<https://devfeed.tech/tags/go-type-mapping.md>), [http](<https://devfeed.tech/tags/http.md>), [json](<https://devfeed.tech/tags/json.md>), [json-best-practices](<https://devfeed.tech/tags/json-best-practices.md>), [json-go-mapping](<https://devfeed.tech/tags/json-go-mapping.md>), [json-marshaling](<https://devfeed.tech/tags/json-marshaling.md>), [json-unmarshaling](<https://devfeed.tech/tags/json-unmarshaling.md>), [json-validation](<https://devfeed.tech/tags/json-validation.md>), [map](<https://devfeed.tech/tags/map.md>), [passing-serialized-data](<https://devfeed.tech/tags/passing-serialized-data.md>), [rest-apis](<https://devfeed.tech/tags/rest-apis.md>), [schema-less-data-format](<https://devfeed.tech/tags/schema-less-data-format.md>), [serialization](<https://devfeed.tech/tags/serialization.md>), [serialization-mistakes](<https://devfeed.tech/tags/serialization-mistakes.md>), [serialization-protocols](<https://devfeed.tech/tags/serialization-protocols.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [struct](<https://devfeed.tech/tags/struct.md>), [tag](<https://devfeed.tech/tags/tag.md>)

### AI overview

Go's `encoding/json` offers `Marshal`, which returns bytes, and `NewEncoder`, which writes to an `io.Writer`. For HTTP handlers, encoding before sending can expose encoding errors early enough to choose a failure status; streaming through an encoder may fail after the response has started, so such errors should be logged. The excerpt also contrasts quick map-based marshaling with structs, which define an outgoing schema, and notes that struct field tags customize JSON property names.

### Source excerpt

Introduction In part 1 we took a high-level view on serialization and JSON. In this part, we'll roll our sleeves and start working with JSON, focused on emitting JSON. You might think this is a basic topic, but there is much more to it than just calling json.Marshal. json.Marshal vs json.Encoder The encoding/json package has two main APIs: Marshal and NewEncoder. The Marshal function returns a []byte while the NewEncoder function will write to an io.Writer. The question is: When should you use one API over the other?

## Optimizing Databases on Kubernetes: Kubernetes Backup and Recovery with CNPG and ZFS Snapshots

DevFeed: [Optimizing Databases on Kubernetes: Kubernetes Backup and Recovery with CNPG and ZFS Snapshots](<https://devfeed.tech/articles/optimizing-databases-on-kubernetes-kubernetes-backup-and-recovery-with-cnpg-and-zfs-snapshots-22280.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2025/01/optimizing-databases-on-kubernetes-ep5-kubernetes-backup-and-recovery-with-cnpg-and-zfs-snapshots.html>)

Published: 2024-12-30T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Featured-Backing Up](<https://devfeed.tech/topics/featured-backing-up.md>), [CloudNativePG](<https://devfeed.tech/topics/cloudnativepg.md>), [ZFS](<https://devfeed.tech/topics/zfs-file-system-features.md>), [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [backup](<https://devfeed.tech/tags/backup.md>), [backup-and-recovery](<https://devfeed.tech/tags/backup-and-recovery.md>), [backups](<https://devfeed.tech/tags/backups.md>), [cloud-block-storage-benchmarking](<https://devfeed.tech/tags/cloud-block-storage-benchmarking.md>), [data](<https://devfeed.tech/tags/data.md>), [databases](<https://devfeed.tech/tags/databases.md>), [durability](<https://devfeed.tech/tags/durability.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [fast](<https://devfeed.tech/tags/fast.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-backup](<https://devfeed.tech/tags/kubernetes-backup.md>), [kubernetes-database-performance](<https://devfeed.tech/tags/kubernetes-database-performance.md>), [kubernetes-database-throughput](<https://devfeed.tech/tags/kubernetes-database-throughput.md>), [kubernetes-local-storage-pros-and-cons](<https://devfeed.tech/tags/kubernetes-local-storage-pros-and-cons.md>), [kubernetes-production-grade-storage](<https://devfeed.tech/tags/kubernetes-production-grade-storage.md>), [kubernetes-storage-benchmarking](<https://devfeed.tech/tags/kubernetes-storage-benchmarking.md>), [kubernetes-storage-efficiency](<https://devfeed.tech/tags/kubernetes-storage-efficiency.md>), [kubernetes-storage-trade-offs](<https://devfeed.tech/tags/kubernetes-storage-trade-offs.md>), [kubernetes-zfs-advantages](<https://devfeed.tech/tags/kubernetes-zfs-advantages.md>), [logs](<https://devfeed.tech/tags/logs.md>), [optimizing-postgresql-on-kubernetes](<https://devfeed.tech/tags/optimizing-postgresql-on-kubernetes.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pgbench-kubernetes-testing](<https://devfeed.tech/tags/pgbench-kubernetes-testing.md>), [point](<https://devfeed.tech/tags/point.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [production](<https://devfeed.tech/tags/production.md>), [rancher-local-path-performance](<https://devfeed.tech/tags/rancher-local-path-performance.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>), [storage](<https://devfeed.tech/tags/storage.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [zfs](<https://devfeed.tech/tags/zfs.md>), [zfs-compression-benefits](<https://devfeed.tech/tags/zfs-compression-benefits.md>), [zfs-lz4-compression](<https://devfeed.tech/tags/zfs-lz4-compression.md>), [zfs-vs-cloud-storage](<https://devfeed.tech/tags/zfs-vs-cloud-storage.md>)

### AI overview

The excerpt describes a PostgreSQL backup and recovery approach using CNPG and ZFS: automate backups and ship WAL to S3-compatible object storage, with barman cloud cited for managing replication logs, to support point-in-time recovery. ZFS copy-on-write snapshots are presented as a way to clone databases quickly with low storage overhead, aided by compression. The episode also flags balancing performance and storage needs, and notes possible configuration and durability challenges; it does not provide the actual YAML configuration details in the supplied text.

### Source excerpt

Introduction: In the final episode of the Optimizing Databases on Kubernetes series, Jérôme Petazzoni dives into advanced backup and recovery techniques for PostgreSQL, showcasing how CNPG (Cloud Native PostgreSQL) and ZFS snapshots ensure durability and fast recovery in production environments. This episode focuses on safeguarding data with comprehensive backup strategies, point-in-time recovery, and rapid database cloning, while leveraging ZFS's efficiency for optimized storage performance. Comprehensive Backup Strategies: Exploring backups with CNPG, including write-ahead logs (WAL) and volume snapshots.

## Optimizing Databases on Kubernetes Ep.4: Kubernetes Storage: Benchmarking ZFS, Cloud Disks, and Local Paths

DevFeed: [Optimizing Databases on Kubernetes Ep.4: Kubernetes Storage: Benchmarking ZFS, Cloud Disks, and Local Paths](<https://devfeed.tech/articles/optimizing-databases-on-kubernetes-ep-4-kubernetes-storage-benchmarking-zfs-cloud-disks-and-local-paths-22276.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/12/optimizing-databases-on-kubernetes-ep4-kubernetes-storage-benchmarking-zfs-cloud-disks-and-local-paths.html>)

Published: 2024-12-23T00:00:00Z

Content type: comparison

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [ZFS](<https://devfeed.tech/topics/zfs-file-system-features.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [cloud block storage benchmarking](<https://devfeed.tech/topics/cloud-block-storage-benchmarking.md>), [Rancher Local Path Provisioner](<https://devfeed.tech/topics/rancher-local-path-persistent-volumes.md>), [CloudNativePG](<https://devfeed.tech/topics/cloudnativepg.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cloud-block-storage-benchmarking](<https://devfeed.tech/tags/cloud-block-storage-benchmarking.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [databases](<https://devfeed.tech/tags/databases.md>), [deduplication](<https://devfeed.tech/tags/deduplication.md>), [durability](<https://devfeed.tech/tags/durability.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-database-performance](<https://devfeed.tech/tags/kubernetes-database-performance.md>), [kubernetes-database-throughput](<https://devfeed.tech/tags/kubernetes-database-throughput.md>), [kubernetes-local-storage-pros-and-cons](<https://devfeed.tech/tags/kubernetes-local-storage-pros-and-cons.md>), [kubernetes-production-grade-storage](<https://devfeed.tech/tags/kubernetes-production-grade-storage.md>), [kubernetes-storage-benchmarking](<https://devfeed.tech/tags/kubernetes-storage-benchmarking.md>), [kubernetes-storage-efficiency](<https://devfeed.tech/tags/kubernetes-storage-efficiency.md>), [kubernetes-storage-trade-offs](<https://devfeed.tech/tags/kubernetes-storage-trade-offs.md>), [kubernetes-zfs-advantages](<https://devfeed.tech/tags/kubernetes-zfs-advantages.md>), [lz4](<https://devfeed.tech/tags/lz4.md>), [methodology](<https://devfeed.tech/tags/methodology.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimizing-postgresql-on-kubernetes](<https://devfeed.tech/tags/optimizing-postgresql-on-kubernetes.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pgbench-kubernetes-testing](<https://devfeed.tech/tags/pgbench-kubernetes-testing.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [production](<https://devfeed.tech/tags/production.md>), [rancher](<https://devfeed.tech/tags/rancher.md>), [rancher-local-path-performance](<https://devfeed.tech/tags/rancher-local-path-performance.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [storage](<https://devfeed.tech/tags/storage.md>), [zfs](<https://devfeed.tech/tags/zfs.md>), [zfs-compression-benefits](<https://devfeed.tech/tags/zfs-compression-benefits.md>), [zfs-lz4-compression](<https://devfeed.tech/tags/zfs-lz4-compression.md>), [zfs-vs-cloud-storage](<https://devfeed.tech/tags/zfs-vs-cloud-storage.md>)

### AI overview

Jérôme Petazzoni benchmarks Kubernetes storage for CloudNativePG clusters using pgbench. In the excerpt, cloud block storage serves as the baseline at about 1,400 transactions per second; it is described as durable but slower. ZFS with LZ4 compression reaches about 3,400 transactions per second while reducing actual disk use by nearly 80%, presented as a balance of speed and storage efficiency. Local storage is described as fast but lacking production-grade safeguards. The episode recommends benchmarking and tuning configurations for the workload; the excerpt provides no specific results for Local Path or detailed durability and scalability comparisons.

### Source excerpt

Introduction: In Episode 4 of the Optimizing Databases on Kubernetes series, Jérôme Petazzoni benchmarks the performance of various Kubernetes storage classes, including cloud block storage, ZFS, and Rancher's Local Path provisioner. This episode dives into the practical aspects of measuring transactions per second, storage efficiency, and durability, offering insights into selecting the right storage solution for database workloads. Through detailed performance comparisons, Jérôme highlights how features like ZFS compression can optimize resource usage and boost database throughput.

## Optimizing Databases on Kubernetes Ep.3: Exploring Kubernetes Storage with OpenEBS and ZFS

DevFeed: [Optimizing Databases on Kubernetes Ep.3: Exploring Kubernetes Storage with OpenEBS and ZFS](<https://devfeed.tech/articles/optimizing-databases-on-kubernetes-ep-3-exploring-kubernetes-storage-with-openebs-and-zfs-22275.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/12/optimizing-databases-on-kubernetes-ep3-exploring-kubernetes-storage-with-openebs-and-zfs.html>)

Published: 2024-12-18T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [ZFS](<https://devfeed.tech/topics/zfs-file-system-features.md>), [Database](<https://devfeed.tech/topics/database.md>), [openebs localpv zfs](<https://devfeed.tech/topics/openebs-localpv-zfs.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Rancher Local Path Provisioner](<https://devfeed.tech/topics/rancher-local-path-persistent-volumes.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [automated-zfs-setup-kubernetes](<https://devfeed.tech/tags/automated-zfs-setup-kubernetes.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [classes](<https://devfeed.tech/tags/classes.md>), [comparing-zfs-and-block-storage](<https://devfeed.tech/tags/comparing-zfs-and-block-storage.md>), [compression](<https://devfeed.tech/tags/compression.md>), [copy-on-write](<https://devfeed.tech/tags/copy-on-write.md>), [databases](<https://devfeed.tech/tags/databases.md>), [deduplication](<https://devfeed.tech/tags/deduplication.md>), [encryption](<https://devfeed.tech/tags/encryption.md>), [file-system](<https://devfeed.tech/tags/file-system.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-advanced-storage-solutions](<https://devfeed.tech/tags/kubernetes-advanced-storage-solutions.md>), [kubernetes-local-provisioners-efficiency](<https://devfeed.tech/tags/kubernetes-local-provisioners-efficiency.md>), [kubernetes-storage-optimization-zfs](<https://devfeed.tech/tags/kubernetes-storage-optimization-zfs.md>), [kubernetes-storage-provisioners-comparison](<https://devfeed.tech/tags/kubernetes-storage-provisioners-comparison.md>), [kubernetes-volume-management-zfs](<https://devfeed.tech/tags/kubernetes-volume-management-zfs.md>), [kubernetes-zfs-performance-tuning](<https://devfeed.tech/tags/kubernetes-zfs-performance-tuning.md>), [openebs-localpv-zfs](<https://devfeed.tech/tags/openebs-localpv-zfs.md>), [performance](<https://devfeed.tech/tags/performance.md>), [proxmox-zfs-database-management](<https://devfeed.tech/tags/proxmox-zfs-database-management.md>), [raid](<https://devfeed.tech/tags/raid.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [scalable-storage-kubernetes-zfs](<https://devfeed.tech/tags/scalable-storage-kubernetes-zfs.md>), [setup](<https://devfeed.tech/tags/setup.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>), [solutions](<https://devfeed.tech/tags/solutions.md>), [storage](<https://devfeed.tech/tags/storage.md>), [storage-classes-benchmarking-zfs](<https://devfeed.tech/tags/storage-classes-benchmarking-zfs.md>), [storage-solutions](<https://devfeed.tech/tags/storage-solutions.md>), [zfs](<https://devfeed.tech/tags/zfs.md>), [zfs-compression-and-encryption](<https://devfeed.tech/tags/zfs-compression-and-encryption.md>), [zfs-daemonset-deployment](<https://devfeed.tech/tags/zfs-daemonset-deployment.md>), [zfs-deduplication-k8s](<https://devfeed.tech/tags/zfs-deduplication-k8s.md>), [zfs-file-system-features](<https://devfeed.tech/tags/zfs-file-system-features.md>), [zfs-integration-with-kubernetes](<https://devfeed.tech/tags/zfs-integration-with-kubernetes.md>), [zfs-pools-in-k8s](<https://devfeed.tech/tags/zfs-pools-in-k8s.md>), [zfs-snapshots-and-compression](<https://devfeed.tech/tags/zfs-snapshots-and-compression.md>), [zfs-vs-rancher-local-path-provisioner](<https://devfeed.tech/tags/zfs-vs-rancher-local-path-provisioner.md>)

### AI overview

Jérôme Petazzoni demonstrates using ZFS with Kubernetes and OpenEBS LocalPV to provision volumes with configurable features such as compression and deduplication. The excerpt describes installing ZFS on Debian-based systems, creating pools, and automating setup across nodes with a DaemonSet. It also introduces storage-class configuration and comparisons with cloud block storage and Rancher's local path provisioner, with benchmarking mentioned as a way to examine performance. The text does not provide benchmark results.

### Source excerpt

Introduction: In Episode 3 of the Optimizing Databases on Kubernetes series, Jérôme Petazzoni introduces ZFS, a versatile file system renowned for its features like compression, deduplication, and snapshots. Leveraging ZFS with Kubernetes, Jérôme demonstrates how to create efficient and flexible storage solutions, complete with automated setup and configuration using tools like OpenEBS LocalPV. This episode showcases the potential of ZFS to optimize storage performance and reliability in containerized environments. Overview of ZFS: Exploring the features of this advanced file system, from snapshots to RAID configurations.

## Optimizing Databases on Kubernetes Ep.1: Provisioning a Cluster and Configuring PostgreSQL on Kubernetes

DevFeed: [Optimizing Databases on Kubernetes Ep.1: Provisioning a Cluster and Configuring PostgreSQL on Kubernetes](<https://devfeed.tech/articles/optimizing-databases-on-kubernetes-ep-1-provisioning-a-cluster-and-configuring-postgresql-on-kubernetes-22273.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/12/optimizing-databases-on-kubernetes-ep1-provisioning-a-cluster-and-configuring-postgresql-on-kubernetes.html>)

Published: 2024-12-09T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [CloudNativePG](<https://devfeed.tech/topics/cloudnativepg.md>), [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [Provisioning](<https://devfeed.tech/topics/provisioning.md>), [Rancher Local Path Provisioner](<https://devfeed.tech/topics/rancher-local-path-persistent-volumes.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [build vs buy database solutions](<https://devfeed.tech/topics/build-vs-buy-database-solutions.md>), [Linode](<https://devfeed.tech/topics/linode.md>)

Tags: [backups](<https://devfeed.tech/tags/backups.md>), [build-vs-buy-database-solutions](<https://devfeed.tech/tags/build-vs-buy-database-solutions.md>), [cloudnativepg](<https://devfeed.tech/tags/cloudnativepg.md>), [cloudnativepg-setup](<https://devfeed.tech/tags/cloudnativepg-setup.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [cnpg-postgresql-operator](<https://devfeed.tech/tags/cnpg-postgresql-operator.md>), [configure](<https://devfeed.tech/tags/configure.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-efficient-kubernetes-clusters](<https://devfeed.tech/tags/cost-efficient-kubernetes-clusters.md>), [cost-optimization](<https://devfeed.tech/tags/cost-optimization.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [database-infrastructure-on-linode](<https://devfeed.tech/tags/database-infrastructure-on-linode.md>), [database-optimization-on-kubernetes](<https://devfeed.tech/tags/database-optimization-on-kubernetes.md>), [databases](<https://devfeed.tech/tags/databases.md>), [efficient-storage-with-zfspv](<https://devfeed.tech/tags/efficient-storage-with-zfspv.md>), [high-availability-postgresql-kubernetes](<https://devfeed.tech/tags/high-availability-postgresql-kubernetes.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-clusters](<https://devfeed.tech/tags/kubernetes-clusters.md>), [kubernetes-for-databases](<https://devfeed.tech/tags/kubernetes-for-databases.md>), [kubernetes-for-startups](<https://devfeed.tech/tags/kubernetes-for-startups.md>), [linode](<https://devfeed.tech/tags/linode.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [postgresql-clusters](<https://devfeed.tech/tags/postgresql-clusters.md>), [postgresql-clusters-on-kubernetes](<https://devfeed.tech/tags/postgresql-clusters-on-kubernetes.md>), [postgresql-replication-on-k8s](<https://devfeed.tech/tags/postgresql-replication-on-k8s.md>), [production](<https://devfeed.tech/tags/production.md>), [production-ready-postgresql-on-k8s](<https://devfeed.tech/tags/production-ready-postgresql-on-k8s.md>), [provisioning](<https://devfeed.tech/tags/provisioning.md>), [provisioning-kubernetes-clusters](<https://devfeed.tech/tags/provisioning-kubernetes-clusters.md>), [rancher-local-path-persistent-volumes](<https://devfeed.tech/tags/rancher-local-path-persistent-volumes.md>), [reducing-hosting-costs-with-kubernetes](<https://devfeed.tech/tags/reducing-hosting-costs-with-kubernetes.md>), [replication](<https://devfeed.tech/tags/replication.md>), [scalable-databases-on-kubernetes](<https://devfeed.tech/tags/scalable-databases-on-kubernetes.md>), [scalable-postgresql-deployments](<https://devfeed.tech/tags/scalable-postgresql-deployments.md>), [self-hosted-databases-vs-managed-services](<https://devfeed.tech/tags/self-hosted-databases-vs-managed-services.md>), [setting-up-postgresql-on-kubernetes](<https://devfeed.tech/tags/setting-up-postgresql-on-kubernetes.md>)

### AI overview

The episode provisions a Kubernetes cluster on Linode, then configures ZFSPV and Rancher Local Path for storage before introducing CloudNativePG (CNPG) to deploy PostgreSQL primary and replica nodes with replication and backups. It presents Kubernetes and in-house database operations as a way to reduce hosting costs compared with managed services, while framing the choice as a build-versus-buy tradeoff--especially relevant to startups and small teams. The excerpt describes these as introductory setup steps for later optimization episodes; it does not provide measured cost savings or detailed performance comparisons.

### Source excerpt

Introduction: In this introductory episode, Jérôme Petazzoni walks through the critical steps of provisioning a Kubernetes cluster tailored for running databases like PostgreSQL. Leveraging his extensive experience, Jérôme demonstrates practical tools and techniques to establish a scalable, cost-efficient database infrastructure while laying the groundwork for future optimizations covered in this series. Provisioning Kubernetes Clusters: A step-by-step guide to creating a reliable cluster using Linode. Setting Up PostgreSQL with CNPG: Deploying production-ready PostgreSQL clusters with built-in replication and backups.

## Fearless Concurrency Ep.7: Lock-Free Structures and Channels for Scalable Rust Code

DevFeed: [Fearless Concurrency Ep.7: Lock-Free Structures and Channels for Scalable Rust Code](<https://devfeed.tech/articles/fearless-concurrency-ep-7-lock-free-structures-and-channels-for-scalable-rust-code-22272.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/12/fearless-concurrency-ep7-lock-free-structures-and-channels-for-scalable-rust-code.html>)

Published: 2024-12-05T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [Code](<https://devfeed.tech/topics/code.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [advanced-rust-concurrency](<https://devfeed.tech/tags/advanced-rust-concurrency.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [backpressure](<https://devfeed.tech/tags/backpressure.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrency-in-rust](<https://devfeed.tech/tags/concurrency-in-rust.md>), [crossbeam-crate-rust](<https://devfeed.tech/tags/crossbeam-crate-rust.md>), [efficient-multithreading-rust](<https://devfeed.tech/tags/efficient-multithreading-rust.md>), [fearless-concurrency-rust](<https://devfeed.tech/tags/fearless-concurrency-rust.md>), [lock-free](<https://devfeed.tech/tags/lock-free.md>), [lock-free-data-structures-rust](<https://devfeed.tech/tags/lock-free-data-structures-rust.md>), [managing-shared-resources-rust](<https://devfeed.tech/tags/managing-shared-resources-rust.md>), [mpsc-channels-rust](<https://devfeed.tech/tags/mpsc-channels-rust.md>), [one-shot-channels-rust](<https://devfeed.tech/tags/one-shot-channels-rust.md>), [optimizing-concurrency-in-rust](<https://devfeed.tech/tags/optimizing-concurrency-in-rust.md>), [performance](<https://devfeed.tech/tags/performance.md>), [robust-multithreaded-rust-apps](<https://devfeed.tech/tags/robust-multithreaded-rust-apps.md>), [rust](<https://devfeed.tech/tags/rust.md>), [rust-atomiccell](<https://devfeed.tech/tags/rust-atomiccell.md>), [rust-bounded-channels](<https://devfeed.tech/tags/rust-bounded-channels.md>), [rust-concurrency](<https://devfeed.tech/tags/rust-concurrency.md>), [rust-dashmap-dashset](<https://devfeed.tech/tags/rust-dashmap-dashset.md>), [rust-high-concurrency-techniques](<https://devfeed.tech/tags/rust-high-concurrency-techniques.md>), [rust-thread-multiplexing](<https://devfeed.tech/tags/rust-thread-multiplexing.md>), [rust-thread-safe-work-queues](<https://devfeed.tech/tags/rust-thread-safe-work-queues.md>), [rust-timeout-options](<https://devfeed.tech/tags/rust-timeout-options.md>), [scalable-rust-applications](<https://devfeed.tech/tags/scalable-rust-applications.md>), [thread](<https://devfeed.tech/tags/thread.md>), [thread-communication-rust](<https://devfeed.tech/tags/thread-communication-rust.md>)

### AI overview

The seventh and final episode of a Rust concurrency series covers lock-free data structures, channels for communication between threads, and techniques for managing shared resources. It discusses tools including DashMap, DashSet, Crossbeam, MPSC and one-shot channels, multiplexing, bounded channels, backpressure, memory usage, and timeouts.

### Source excerpt

Introduction: Welcome to Episode 7 of the Fearless Concurrency in Rust series! In this final episode, we explore advanced concurrency techniques that enable efficient, scalable, and robust multithreaded applications in Rust. The focus is on leveraging tools like lock-free data structures, channels for thread communication, and strategies for safely managing shared resources in complex systems. These approaches ensure developers can push the limits of Rust's concurrency model while maintaining safety and performance.

## Scalable JSON Streaming with HTTP and Go - Ep.5

DevFeed: [Scalable JSON Streaming with HTTP and Go - Ep.5](<https://devfeed.tech/articles/scalable-json-streaming-with-http-and-go-ep-5-22271.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/11/scalable-json-streaming-with--http-and-go.html>)

Published: 2024-11-25T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [encoding/json](<https://devfeed.tech/topics/encoding-json.md>), [json lines format](<https://devfeed.tech/topics/json-lines-format.md>), [JSON](<https://devfeed.tech/topics/json.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Error Handling](<https://devfeed.tech/topics/error-handling.md>)

Tags: [advanced](<https://devfeed.tech/tags/advanced.md>), [chunked-transfer-json](<https://devfeed.tech/tags/chunked-transfer-json.md>), [decoding-json-streams](<https://devfeed.tech/tags/decoding-json-streams.md>), [efficient-json-handling](<https://devfeed.tech/tags/efficient-json-handling.md>), [encoding-json](<https://devfeed.tech/tags/encoding-json.md>), [encoding-json-in-go](<https://devfeed.tech/tags/encoding-json-in-go.md>), [go](<https://devfeed.tech/tags/go.md>), [go-json-encoding](<https://devfeed.tech/tags/go-json-encoding.md>), [go-json-memory-management](<https://devfeed.tech/tags/go-json-memory-management.md>), [handling-large-json-files](<https://devfeed.tech/tags/handling-large-json-files.md>), [http](<https://devfeed.tech/tags/http.md>), [http-chunked-encoding](<https://devfeed.tech/tags/http-chunked-encoding.md>), [json](<https://devfeed.tech/tags/json.md>), [json-chunked-transfer](<https://devfeed.tech/tags/json-chunked-transfer.md>), [json-error-logging](<https://devfeed.tech/tags/json-error-logging.md>), [json-lines-format](<https://devfeed.tech/tags/json-lines-format.md>), [json-lines-ndjson](<https://devfeed.tech/tags/json-lines-ndjson.md>), [json-object](<https://devfeed.tech/tags/json-object.md>), [json-streaming](<https://devfeed.tech/tags/json-streaming.md>), [json-streaming-in-go](<https://devfeed.tech/tags/json-streaming-in-go.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [logging](<https://devfeed.tech/tags/logging.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-efficient-data-transmission](<https://devfeed.tech/tags/memory-efficient-data-transmission.md>), [memory-efficient-json](<https://devfeed.tech/tags/memory-efficient-json.md>), [practical](<https://devfeed.tech/tags/practical.md>), [query](<https://devfeed.tech/tags/query.md>), [real-time-json-streaming](<https://devfeed.tech/tags/real-time-json-streaming.md>), [scalable-json-processing](<https://devfeed.tech/tags/scalable-json-processing.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data](<https://devfeed.tech/tags/streaming-data.md>), [streaming-json-over-http](<https://devfeed.tech/tags/streaming-json-over-http.md>), [streaming-large-json-datasets](<https://devfeed.tech/tags/streaming-large-json-datasets.md>), [streams](<https://devfeed.tech/tags/streams.md>)

### AI overview

For large datasets, the text recommends streaming JSON Lines (NDJSON) instead of building one large JSON object in memory, sending records incrementally to reduce memory use. In Go, an encoder can write each object with a newline; decoding requires a loop that handles incoming data without memory reuse causing stale or incorrect values. Over HTTP/1.1, the response writer can be used to encode objects and flush after each one for incremental delivery. Once streaming begins, the HTTP status code cannot be changed, so the text emphasizes logging errors. It mentions Netcat demonstrations but provides no further details in the supplied excerpt.

### Source excerpt

Introduction: Welcome to the final episode of the JSON for Engineers series! In this concluding session, we tackle the challenges of working with large JSON datasets, exploring efficient strategies for streaming data while minimizing memory usage. These techniques enable developers to handle massive payloads without overburdening system resources, ensuring scalable and cost-effective applications. JSON Streaming: Using JSON Lines for memory-efficient data transmission. HTTP Chunked Encoding: Leveraging HTTP/1.1 chunked transfer encoding for streaming large datasets. Practical Error Handling: Logging and managing errors in streaming JSON responses. This episode starts by addressing the inefficiencies of constructing large JSON objects in memory when working with massive datasets, such as database query results. Instead of consuming significant memory to create one monolithic JSON object, the recommended approach involves using JSON Lines (NDJSON), a format where each line represents a separate JSON object. This method reduces memory requirements by transmitting data incrementally. Using Go's encoding/json package, developers can easily encode and stream multiple JSON objects, as it automatically appends newlines between objects. On the receiving end, decoding JSON streams requires careful looping to handle incoming data dynamically while avoiding memory reuse issues, which could lead to errors or stale data.

## Fearless Concurrency Ep.6: Understanding Rust ARC for Efficient Multithreading

DevFeed: [Fearless Concurrency Ep.6: Understanding Rust ARC for Efficient Multithreading](<https://devfeed.tech/articles/fearless-concurrency-ep-6-understanding-rust-arc-for-efficient-multithreading-22270.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/11/fearless-concurrency-ep6-understanding-rust-arc-for-efficient-multithreading.html>)

Published: 2024-11-21T00:00:00Z

Content type: article

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [atomic reference counting](<https://devfeed.tech/topics/atomic-reference-counting.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [multithreading](<https://devfeed.tech/topics/multithreading.md>), [thread safe data sharing](<https://devfeed.tech/topics/thread-safe-data-sharing.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>)

Tags: [applications](<https://devfeed.tech/tags/applications.md>), [arc](<https://devfeed.tech/tags/arc.md>), [arc-best-practices](<https://devfeed.tech/tags/arc-best-practices.md>), [arc-for-performance](<https://devfeed.tech/tags/arc-for-performance.md>), [arc-in-rust](<https://devfeed.tech/tags/arc-in-rust.md>), [arc-memory-management](<https://devfeed.tech/tags/arc-memory-management.md>), [arc-ownership-model](<https://devfeed.tech/tags/arc-ownership-model.md>), [arc-shared-resources](<https://devfeed.tech/tags/arc-shared-resources.md>), [arc-vs-garbage-collection](<https://devfeed.tech/tags/arc-vs-garbage-collection.md>), [arc-with-mutex](<https://devfeed.tech/tags/arc-with-mutex.md>), [arc-with-rwlock](<https://devfeed.tech/tags/arc-with-rwlock.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [atomic-reference-counting](<https://devfeed.tech/tags/atomic-reference-counting.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrency-in-rust](<https://devfeed.tech/tags/concurrency-in-rust.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [connection-pool](<https://devfeed.tech/tags/connection-pool.md>), [database](<https://devfeed.tech/tags/database.md>), [efficient-multithreading-rust](<https://devfeed.tech/tags/efficient-multithreading-rust.md>), [garbage-collection](<https://devfeed.tech/tags/garbage-collection.md>), [global](<https://devfeed.tech/tags/global.md>), [heap](<https://devfeed.tech/tags/heap.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [interior-mutability-arc](<https://devfeed.tech/tags/interior-mutability-arc.md>), [large](<https://devfeed.tech/tags/large.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [lightweight-memory-management](<https://devfeed.tech/tags/lightweight-memory-management.md>), [memory](<https://devfeed.tech/tags/memory.md>), [multithreading](<https://devfeed.tech/tags/multithreading.md>), [multithreading-with-arc](<https://devfeed.tech/tags/multithreading-with-arc.md>), [ownership](<https://devfeed.tech/tags/ownership.md>), [rust](<https://devfeed.tech/tags/rust.md>), [rust-arc-tutorial](<https://devfeed.tech/tags/rust-arc-tutorial.md>), [rust-arc-use-cases](<https://devfeed.tech/tags/rust-arc-use-cases.md>), [rust-concurrency-tools](<https://devfeed.tech/tags/rust-concurrency-tools.md>), [rust-shared-ownership](<https://devfeed.tech/tags/rust-shared-ownership.md>), [rust-thread-safe-programming](<https://devfeed.tech/tags/rust-thread-safe-programming.md>), [thread-safe-data-sharing](<https://devfeed.tech/tags/thread-safe-data-sharing.md>)

### AI overview

Rust ARC shares a heap-allocated value across threads by cloning a pointer and incrementing its reference count; dropping each pointer decrements the count, and the value is destroyed when the count reaches zero. This avoids copying the underlying data and can suit high fan-out sharing, such as sharing configuration or a database connection pool. The excerpt also mentions combining ARC with interior mutability for thread-safe updates and flags circular references as a pitfall. It presents ARC as shared ownership without a traditional garbage collector; the excerpt does not identify the interior-mutability types or give their usage details.

### Source excerpt

Introduction: Welcome to Episode 6 of the Fearless Concurrency in Rust series! In this episode, we explore the powerful concept of ARC (Atomic Reference Counting) and its critical role in managing shared resources in multithreaded Rust programs. By leveraging ARC, developers can safely share ownership of data across threads without relying on a traditional garbage collector, enabling efficient, flexible, and scalable concurrency. What is ARC?: Understanding Atomic Reference Counting as a lightweight, manual garbage collection system.

[Next page](<https://devfeed.tech/sources/william-kennedy.md?cursor=WyIyMDI0LTExLTIxVDAwOjAwOjAwKzAwOjAwIiwgIjAwYTc1NWY3LWFkZGYtNGUzNS1hMTkxLWIxMjNlZGZhOGNjOCJd>)