# Scalable JSON Streaming with HTTP and Go - Ep.5

DevFeed: [Scalable JSON Streaming with HTTP and Go - Ep.5](<https://devfeed.tech/articles/scalable-json-streaming-with-http-and-go-ep-5-22271.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/11/scalable-json-streaming-with--http-and-go.html>)

Published: 2024-11-25T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [encoding/json](<https://devfeed.tech/topics/encoding-json.md>), [json lines format](<https://devfeed.tech/topics/json-lines-format.md>), [JSON](<https://devfeed.tech/topics/json.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Error Handling](<https://devfeed.tech/topics/error-handling.md>)

Tags: [advanced](<https://devfeed.tech/tags/advanced.md>), [chunked-transfer-json](<https://devfeed.tech/tags/chunked-transfer-json.md>), [decoding-json-streams](<https://devfeed.tech/tags/decoding-json-streams.md>), [efficient-json-handling](<https://devfeed.tech/tags/efficient-json-handling.md>), [encoding-json](<https://devfeed.tech/tags/encoding-json.md>), [encoding-json-in-go](<https://devfeed.tech/tags/encoding-json-in-go.md>), [go](<https://devfeed.tech/tags/go.md>), [go-json-encoding](<https://devfeed.tech/tags/go-json-encoding.md>), [go-json-memory-management](<https://devfeed.tech/tags/go-json-memory-management.md>), [handling-large-json-files](<https://devfeed.tech/tags/handling-large-json-files.md>), [http](<https://devfeed.tech/tags/http.md>), [http-chunked-encoding](<https://devfeed.tech/tags/http-chunked-encoding.md>), [json](<https://devfeed.tech/tags/json.md>), [json-chunked-transfer](<https://devfeed.tech/tags/json-chunked-transfer.md>), [json-error-logging](<https://devfeed.tech/tags/json-error-logging.md>), [json-lines-format](<https://devfeed.tech/tags/json-lines-format.md>), [json-lines-ndjson](<https://devfeed.tech/tags/json-lines-ndjson.md>), [json-object](<https://devfeed.tech/tags/json-object.md>), [json-streaming](<https://devfeed.tech/tags/json-streaming.md>), [json-streaming-in-go](<https://devfeed.tech/tags/json-streaming-in-go.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [logging](<https://devfeed.tech/tags/logging.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-efficient-data-transmission](<https://devfeed.tech/tags/memory-efficient-data-transmission.md>), [memory-efficient-json](<https://devfeed.tech/tags/memory-efficient-json.md>), [practical](<https://devfeed.tech/tags/practical.md>), [query](<https://devfeed.tech/tags/query.md>), [real-time-json-streaming](<https://devfeed.tech/tags/real-time-json-streaming.md>), [scalable-json-processing](<https://devfeed.tech/tags/scalable-json-processing.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data](<https://devfeed.tech/tags/streaming-data.md>), [streaming-json-over-http](<https://devfeed.tech/tags/streaming-json-over-http.md>), [streaming-large-json-datasets](<https://devfeed.tech/tags/streaming-large-json-datasets.md>), [streams](<https://devfeed.tech/tags/streams.md>)

## AI overview

For large datasets, the text recommends streaming JSON Lines (NDJSON) instead of building one large JSON object in memory, sending records incrementally to reduce memory use. In Go, an encoder can write each object with a newline; decoding requires a loop that handles incoming data without memory reuse causing stale or incorrect values. Over HTTP/1.1, the response writer can be used to encode objects and flush after each one for incremental delivery. Once streaming begins, the HTTP status code cannot be changed, so the text emphasizes logging errors. It mentions Netcat demonstrations but provides no further details in the supplied excerpt.

## Source excerpt

Introduction: Welcome to the final episode of the JSON for Engineers series! In this concluding session, we tackle the challenges of working with large JSON datasets, exploring efficient strategies for streaming data while minimizing memory usage. These techniques enable developers to handle massive payloads without overburdening system resources, ensuring scalable and cost-effective applications. JSON Streaming: Using JSON Lines for memory-efficient data transmission. HTTP Chunked Encoding: Leveraging HTTP/1.1 chunked transfer encoding for streaming large datasets. Practical Error Handling: Logging and managing errors in streaming JSON responses. This episode starts by addressing the inefficiencies of constructing large JSON objects in memory when working with massive datasets, such as database query results. Instead of consuming significant memory to create one monolithic JSON object, the recommended approach involves using JSON Lines (NDJSON), a format where each line represents a separate JSON object. This method reduces memory requirements by transmitting data incrementally. Using Go's encoding/json package, developers can easily encode and stream multiple JSON objects, as it automatically appends newlines between objects. On the receiving end, decoding JSON streams requires careful looping to handle incoming data dynamically while avoiding memory reuse issues, which could lead to errors or stale data.