# real-time data streaming

Published articles for real-time data streaming.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Hemut: Building the Internet of Freight on Real-Time Data

DevFeed: [Hemut: Building the Internet of Freight on Real-Time Data](<https://devfeed.tech/articles/hemut-building-the-internet-of-freight-on-real-time-data-11551.md>)

Original publisher: [Read original article](<https://www.confluent.io/blog/hemut-building-the-internet-of-freight-on-real-time-data/>)

Author: Tim Graczewski

Published: 2026-08-27T20:23:15Z

Content type: article

Language: en

Sources: [Confluent: Data in motion](<https://devfeed.tech/sources/confluent-data-in-motion.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Network](<https://devfeed.tech/topics/network.md>), [Software](<https://devfeed.tech/topics/software.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Operating system](<https://devfeed.tech/topics/operating-system.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [business](<https://devfeed.tech/tags/business.md>), [ceo](<https://devfeed.tech/tags/ceo.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [confluent](<https://devfeed.tech/tags/confluent.md>), [confluent-cloud](<https://devfeed.tech/tags/confluent-cloud.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [erp](<https://devfeed.tech/tags/erp.md>), [network](<https://devfeed.tech/tags/network.md>), [production](<https://devfeed.tech/tags/production.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [software](<https://devfeed.tech/tags/software.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [supply-chain](<https://devfeed.tech/tags/supply-chain.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

Hemut uses Confluent Cloud and real-time data streaming to provide an AI-native operating system for trucking carriers and brokers. Its platform combines ERP and TMS capabilities to automate tasks, improve fleet efficiency, reduce operating costs, and support the company's vision of an Internet of Freight.

### Source excerpt

Hemut uses Confluent, real-time data streaming, and AI to automate trucking operations, improve fleet efficiency, and build the Internet of Freight.

## Streaming optimized data to S3 for analytics with Parquet

DevFeed: [Streaming optimized data to S3 for analytics with Parquet](<https://devfeed.tech/articles/streaming-optimized-data-to-s3-for-analytics-with-parquet-12776.md>)

Original publisher: [Read original article](<https://www.redpanda.com/blog/streaming-data-s3-analytics-parquet>)

Author: Chandler Mayo

Published: 2025-08-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redpanda](<https://devfeed.tech/sources/redpanda.md>)

Topics: [parquet](<https://devfeed.tech/topics/parquet.md>), [Redpanda-Connect](<https://devfeed.tech/topics/redpanda-connect.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data analytics](<https://devfeed.tech/topics/data-analytics.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [pandas](<https://devfeed.tech/topics/pandas.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-systems-data-feed](<https://devfeed.tech/tags/analytics-systems-data-feed.md>), [apache-spark-and-athena-data-queries](<https://devfeed.tech/tags/apache-spark-and-athena-data-queries.md>), [athena](<https://devfeed.tech/tags/athena.md>), [automating-data-pipelines-in-s3](<https://devfeed.tech/tags/automating-data-pipelines-in-s3.md>), [building-dashboards-with-s3-data](<https://devfeed.tech/tags/building-dashboards-with-s3-data.md>), [compressing-data-with-parquet](<https://devfeed.tech/tags/compressing-data-with-parquet.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipeline-for-analytics](<https://devfeed.tech/tags/data-pipeline-for-analytics.md>), [event-driven-pipelines-with-s3](<https://devfeed.tech/tags/event-driven-pipelines-with-s3.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [learn](<https://devfeed.tech/tags/learn.md>), [pandas](<https://devfeed.tech/tags/pandas.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files-in-amazon-s3](<https://devfeed.tech/tags/parquet-files-in-amazon-s3.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [redpanda-connect](<https://devfeed.tech/tags/redpanda-connect.md>), [redpanda-data-streaming](<https://devfeed.tech/tags/redpanda-data-streaming.md>), [s3](<https://devfeed.tech/tags/s3.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [setting-up-redpanda-observability](<https://devfeed.tech/tags/setting-up-redpanda-observability.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-data-to-s3-with-parquet](<https://devfeed.tech/tags/streaming-data-to-s3-with-parquet.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

This tutorial explains how to use Redpanda Connect to continuously batch streaming data and write compressed Apache Parquet files to Amazon S3 for analytical workloads. It covers the benefits of Parquet and querying the resulting files with tools such as Pandas, Apache Spark, and Athena.

### Source excerpt

Learn how to build a powerful data pipeline that feeds analytics systems from Redpanda using clean, compressed Parquet files in Amazon S3.

## FIPS-ing the Un-FIPS-able: Apache Spark

DevFeed: [FIPS-ing the Un-FIPS-able: Apache Spark](<https://devfeed.tech/articles/fips-ing-the-un-fips-able-apache-spark-13045.md>)

Original publisher: [Read original article](<https://www.chainguard.dev/unchained/fips-ing-the-un-fips-able-apache-spark>)

Published: 2025-04-17T00:00:00Z

Content type: article

Language: en

Sources: [Chainguard: Unchained](<https://devfeed.tech/sources/chainguard-unchained.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [chainguard](<https://devfeed.tech/topics/chainguard.md>), [container images](<https://devfeed.tech/topics/container-images.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Cybersecurity](<https://devfeed.tech/topics/cybersecurity.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [chainguard](<https://devfeed.tech/tags/chainguard.md>), [chainguard-containers](<https://devfeed.tech/tags/chainguard-containers.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [container-images](<https://devfeed.tech/tags/container-images.md>), [cryptography](<https://devfeed.tech/tags/cryptography.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [fedramp](<https://devfeed.tech/tags/fedramp.md>), [fips](<https://devfeed.tech/tags/fips.md>), [fraud-detection](<https://devfeed.tech/tags/fraud-detection.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [recommendation-systems](<https://devfeed.tech/tags/recommendation-systems.md>), [spark](<https://devfeed.tech/tags/spark.md>), [spark-operator](<https://devfeed.tech/tags/spark-operator.md>)

### AI overview

Chainguard announces FIPS-validated container images for Apache Spark and Spark Operator, built entirely from source. The article explains the demand for FIPS-compatible Spark in regulated environments and describes the effort to overcome incompatibilities between Spark and FIPS-approved cryptographic libraries.

### Source excerpt

Chainguard now offers FIPS-validated container images for Apache Spark and Spark Operator. See how we did it.

## Scalable JSON Streaming with HTTP and Go - Ep.5

DevFeed: [Scalable JSON Streaming with HTTP and Go - Ep.5](<https://devfeed.tech/articles/scalable-json-streaming-with-http-and-go-ep-5-22271.md>)

Original publisher: [Read original article](<https://www.ardanlabs.com/blog/2024/11/scalable-json-streaming-with--http-and-go.html>)

Published: 2024-11-25T00:00:00Z

Content type: tutorial

Language: en

Sources: [William Kennedy](<https://devfeed.tech/sources/william-kennedy.md>)

Topics: [Go Language](<https://devfeed.tech/topics/go-language.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [JSON](<https://devfeed.tech/topics/json.md>), [HTTP](<https://devfeed.tech/topics/http.md>), [Error Handling](<https://devfeed.tech/topics/error-handling.md>), [datasets](<https://devfeed.tech/topics/datasets.md>)

Tags: [chunked-transfer-json](<https://devfeed.tech/tags/chunked-transfer-json.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [data-transmission](<https://devfeed.tech/tags/data-transmission.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [decoding-json-streams](<https://devfeed.tech/tags/decoding-json-streams.md>), [efficient-json-handling](<https://devfeed.tech/tags/efficient-json-handling.md>), [encoding-json](<https://devfeed.tech/tags/encoding-json.md>), [encoding-json-in-go](<https://devfeed.tech/tags/encoding-json-in-go.md>), [error-handling](<https://devfeed.tech/tags/error-handling.md>), [examples](<https://devfeed.tech/tags/examples.md>), [go](<https://devfeed.tech/tags/go.md>), [go-json-encoding](<https://devfeed.tech/tags/go-json-encoding.md>), [go-json-memory-management](<https://devfeed.tech/tags/go-json-memory-management.md>), [handling-large-json-files](<https://devfeed.tech/tags/handling-large-json-files.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [http](<https://devfeed.tech/tags/http.md>), [http-chunked-encoding](<https://devfeed.tech/tags/http-chunked-encoding.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [json](<https://devfeed.tech/tags/json.md>), [json-chunked-transfer](<https://devfeed.tech/tags/json-chunked-transfer.md>), [json-error-logging](<https://devfeed.tech/tags/json-error-logging.md>), [json-for-engineers](<https://devfeed.tech/tags/json-for-engineers.md>), [json-lines-format](<https://devfeed.tech/tags/json-lines-format.md>), [json-lines-ndjson](<https://devfeed.tech/tags/json-lines-ndjson.md>), [json-streaming](<https://devfeed.tech/tags/json-streaming.md>), [json-streaming-in-go](<https://devfeed.tech/tags/json-streaming-in-go.md>), [memory-efficient-data-transmission](<https://devfeed.tech/tags/memory-efficient-data-transmission.md>), [memory-efficient-json](<https://devfeed.tech/tags/memory-efficient-json.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>), [real-time-json-streaming](<https://devfeed.tech/tags/real-time-json-streaming.md>), [scalable-json-processing](<https://devfeed.tech/tags/scalable-json-processing.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-json-over-http](<https://devfeed.tech/tags/streaming-json-over-http.md>), [streaming-large-json-datasets](<https://devfeed.tech/tags/streaming-large-json-datasets.md>)

### AI overview

The final episode of the JSON for Engineers series explains how to stream large JSON datasets efficiently using JSON Lines, Go's encoding/json package, and HTTP/1.1 chunked transfer encoding. It covers incremental encoding and decoding, memory usage, response flushing, and logging errors after streaming begins.

### Source excerpt

Introduction: Welcome to the final episode of the JSON for Engineers series! In this concluding session, we tackle the challenges of working with large JSON datasets, exploring efficient strategies for streaming data while minimizing memory usage. These techniques enable developers to handle massive payloads without overburdening system resources, ensuring scalable and cost-effective applications. JSON Streaming: Using JSON Lines for memory-efficient data transmission. HTTP Chunked Encoding: Leveraging HTTP/1.1 chunked transfer encoding for streaming large datasets. Practical Error Handling: Logging and managing errors in streaming JSON responses. This episode starts by addressing the inefficiencies of constructing large JSON objects in memory when working with massive datasets, such as database query results. Instead of consuming significant memory to create one monolithic JSON object, the recommended approach involves using JSON Lines (NDJSON), a format where each line represents a separate JSON object. This method reduces memory requirements by transmitting data incrementally. Using Go's encoding/json package, developers can easily encode and stream multiple JSON objects, as it automatically appends newlines between objects. On the receiving end, decoding JSON streams requires careful looping to handle incoming data dynamically while avoiding memory reuse issues, which could lead to errors or stale data.

## Kafka 101

DevFeed: [Kafka 101](<https://devfeed.tech/articles/kafka-101-33612.md>)

Original publisher: [Read original article](<https://highscalability.com/untitled-2/>)

Author: ByteByteGo

Published: 2024-05-09T18:55:21Z

Content type: tutorial

Language: en

Sources: [High Scalability](<https://devfeed.tech/sources/high-scalability-3.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [Microservice](<https://devfeed.tech/topics/microservice.md>), [Persistence](<https://devfeed.tech/topics/persistence.md>)

Tags: [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [data-structure](<https://devfeed.tech/tags/data-structure.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [real-time-data-streaming](<https://devfeed.tech/tags/real-time-data-streaming.md>)

### AI overview

This introductory article explains Apache Kafka's origins, distributed streaming architecture, and use as a central platform for coordinating data among services and downstream systems. It also describes Kafka's log-based storage model, including immutability, ordered records, concurrent reads, and optimization for high-throughput, cost-efficient HDD storage.

### Source excerpt

This is a guest article by Stanislav Kozlovski, an Apache Kafka Committer. If you would like to connect with Stanislav, you can do so on Twitter and LinkedIn. Originally developed in LinkedIn during 2011, Apache Kafka is one of the most popular open-source Apache projects out