# Big data infrastructure

Computing infrastructure for managing, storing, and analyzing large-scale data, including reference architectures and application interfaces.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Interactive Databricks Reference Architectures Across 70 Industries

DevFeed: [Interactive Databricks Reference Architectures Across 70 Industries](<https://devfeed.tech/articles/interactive-databricks-reference-architectures-across-70-industries-146371.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/interactive-databricks-reference-architectures-across-70-industries>)

Author: Ashraf Osman; Amr Ali; Shelley Shaffery

Published: 2026-11-20T14:48:28Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [architectures](<https://devfeed.tech/tags/architectures.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [industries](<https://devfeed.tech/tags/industries.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [platform](<https://devfeed.tech/tags/platform.md>)

### AI overview

The Databricks Reference Architecture is a free, interactive board with versions for 70 industries and a standard industry-neutral view. Each board maps sector-specific source systems, data formats, teams, medallion layers, use cases, Genie Agents, dashboards, and apps around the shared Databricks Platform. Simplified and Detailed views serve different levels of detail. Region and release-stage filters show product availability based on public Databricks documentation, which is refreshed before board updates; automated update pipelines are planned. Boards work on Azure, AWS, and GCP, support 16 languages, and can be copied, edited, or exported in formats including PowerPoint, PDF, YAML, and Visio. The article presents each board as a starting point that should be adapted to a company's systems and constraints.

### Source excerpt

Why we built itMost Databricks architecture talks used to start with the same slide: ingestion on the left...

## Edge Computing for IoT-Enabled Smart Cities: A Review of Middleware, Explainable AI, and User Behavior Analytics

DevFeed: [Edge Computing for IoT-Enabled Smart Cities: A Review of Middleware, Explainable AI, and User Behavior Analytics](<https://devfeed.tech/articles/edge-computing-for-iot-enabled-smart-cities-a-review-of-middleware-explainable-ai-and-user-behavior-analytics-142170.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23310>)

Author: Alexandros Gazis, George Vavouras, Nikos E. Mastorakis, Kleanthi Santamouri, Nikolaos Pournaras, Stylianos Pappas

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Internet of things](<https://devfeed.tech/topics/iot.md>), [middleware](<https://devfeed.tech/topics/middleware.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [accountability](<https://devfeed.tech/tags/accountability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [behavior-analytics](<https://devfeed.tech/tags/behavior-analytics.md>), [behavioral-analytics](<https://devfeed.tech/tags/behavioral-analytics.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [case-studies](<https://devfeed.tech/tags/case-studies.md>), [cities](<https://devfeed.tech/tags/cities.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [computing](<https://devfeed.tech/tags/computing.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [explainable-ai](<https://devfeed.tech/tags/explainable-ai.md>), [iot](<https://devfeed.tech/tags/iot.md>), [middleware](<https://devfeed.tech/tags/middleware.md>), [review](<https://devfeed.tech/tags/review.md>), [smart-cities](<https://devfeed.tech/tags/smart-cities.md>)

### AI overview

The review maps user-behavior analytics in smart cities across five application domains, including EV charging, energy demand, environmental exposure, and crowd safety. It proposes X-EdgeCity Middleware, an XAI-EdgeSOA framework combining IoT sensing, edge preprocessing, fog coordination, event-driven communication, cloud analytics, explainability, privacy-by-design, and benchmark logging. The aim is to support energy-sensitive and mission-critical urban services where latency, reliability, explainability, and accountability matter. The supplied text is an abstract and does not provide detailed comparisons of case studies or framework performance.

### Source excerpt

arXiv:2611.23310v1 Announce Type: new Abstract: This review looks at cloud-IoT edge middleware for analyzing user behavior in smart cities, with a particular focus on linking behavioral analytics to energy-aware and mission-critical urban services. The topic sits at the crossroads of smart-city platforms, IoT sensing, edge/fog/cloud computing, service-oriented architectures, machine learning, explainable AI, privacy-aware governance, and energy-critical applications such as EV charging, energy demand, environmental exposure, crowd safety, and the operational resilience of urban infrastructure. As such, this paper maps out the main categories of user-behavior data, examines five application domains, compares several case studies, and puts forward a new XAI-EdgeSOA Middleware Framework called X-EdgeCity Middleware. This framework brings together IoT sensing, edge preprocessing, fog coordination, event-driven communication, cloud analytics, XAI explanations, privacy-by-design, and benchmark logging. The goal is to support not just general smart-city applications, but also energy-sensitive or mission-critical services, where latency, reliability, explainability, and accountability have a direct impact on urban safety and continuity of operations.

## S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way

DevFeed: [S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way](<https://devfeed.tech/articles/s3-vs-scylladb-for-tiny-telemetry-what-red-canary-learned-the-hard-way-140898.md>)

Original publisher: [Read original article](<https://thenewstack.io/scylladb-s3-tiny-telemetry/>)

Author: Cynthia Dunlop

Published: 2026-11-18T15:00:00Z

Content type: article

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [scylladb](<https://devfeed.tech/topics/scylladb.md>), [Endpoint Security & XDR](<https://devfeed.tech/topics/endpoint-security-xdr.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [experience report](<https://devfeed.tech/topics/experience-report.md>)

Tags: [account](<https://devfeed.tech/tags/account.md>), [aws](<https://devfeed.tech/tags/aws.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batching](<https://devfeed.tech/tags/batching.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [canary](<https://devfeed.tech/tags/canary.md>), [data](<https://devfeed.tech/tags/data.md>), [databases](<https://devfeed.tech/tags/databases.md>), [detection](<https://devfeed.tech/tags/detection.md>), [detection-and-response](<https://devfeed.tech/tags/detection-and-response.md>), [edr](<https://devfeed.tech/tags/edr.md>), [endpoint-detection-and-response](<https://devfeed.tech/tags/endpoint-detection-and-response.md>), [finops](<https://devfeed.tech/tags/finops.md>), [large](<https://devfeed.tech/tags/large.md>), [linux](<https://devfeed.tech/tags/linux.md>), [post-contributed](<https://devfeed.tech/tags/post-contributed.md>), [red](<https://devfeed.tech/tags/red.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scylladb](<https://devfeed.tech/tags/scylladb.md>), [small](<https://devfeed.tech/tags/small.md>), [sponsor-scylladb](<https://devfeed.tech/tags/sponsor-scylladb.md>), [sponsored-post-contributed](<https://devfeed.tech/tags/sponsored-post-contributed.md>), [storage](<https://devfeed.tech/tags/storage.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [tiny](<https://devfeed.tech/tags/tiny.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Red Canary reduced costs and improved endpoint data visibility by moving Linux EDR health and telemetry storage from S3 and Postgres to ScyllaDB Cloud. Buffering uploads through Firehose and Kinesis cut S3 PUT costs, but imposed a one-megabyte-per-shard limit, delayed visibility and made endpoint debugging harder; Postgres also struggled with write spikes. In ScyllaDB, the team used short TTLs and faster tombstone cleanup to improve reads, and split large customers across numbered buckets to avoid hot partitions. For telemetry, it partitioned by sensor ID and moved large process-memory arrays into an auxiliary table. A fork of the Rust driver enabled token lookup and execution without serializing statements twice, while separating CPU work from I/O; file processing fell from about 150 ms to 50 ms in this use case. Red Canary reports about 20x fewer writes than inserted rows and endpoint retrieval in seconds rather than minutes for its largest customers. The team is exploring further uses, including detection data; the excerpt does not establish results for those future plans.

### Source excerpt

At Red Canary, storing large amounts of small data is a big deal. Their Linux Endpoint Detection and Response (EDR) The post S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way appeared first on The New Stack.

## ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining

DevFeed: [ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining](<https://devfeed.tech/articles/zarrfm-a-canonical-tensor-storage-architecture-for-eeg-foundation-model-pretraining-125330.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11149>)

Author: Dheeraj Chintapalli, Sunkalp Chandra

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [sharding](<https://devfeed.tech/topics/sharding.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [canonical](<https://devfeed.tech/tags/canonical.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [eeg](<https://devfeed.tech/tags/eeg.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [storage](<https://devfeed.tech/tags/storage.md>), [storage-architecture](<https://devfeed.tech/tags/storage-architecture.md>)

### AI overview

ZarrFM stores heterogeneous EEG corpora in a single Zarr v3 hierarchy, complementing BIDS with indexed metadata, canonical channel spaces, explicit masks, stable subject splits, verified event tables, and block-aligned sharding. In tests on real PhysioNet recordings and synthetic corpora, its two-level index reduced metadata queries across 200 recordings from 601 to 14. Preserving native signal codes used 22.7 MB/hour versus 42.4 MB/hour for EDF, and block-aligned sampling reached 3,131 windows/s, up to 11 times Parquet's rate. Reads were bit-identical across independent Zarr implementations, and failure-injected concurrent appends recovered without data loss. These reported results suggest a compact, interoperable storage option for EEG pretraining; the excerpt does not provide broader evaluation details.

### Source excerpt

arXiv:2611.11149v1 Announce Type: new Abstract: Statistical models for brain-computer interfaces (BCIs) increasingly require large, heterogeneous EEG corpora, yet existing datasets vary in sampling rates, channel layouts, window lengths, and storage formats, creating substantial preprocessing and I/O overhead. We propose ZarrFM, an execution-oriented representation that complements BIDS by storing a corpus as a single Zarr v3 hierarchy with indexed metadata, canonical channel spaces, explicit masks, hash-stable subject splits, verified event tables, and block-aligned sharding. On real PhysioNet recordings and synthetic corpora, we compare ZarrFM with EDF, FIF, NPY, HDF5, and Parquet. Across 200 recordings, its two-level index reduces metadata queries from 601 requests to 14. Preserving native signal codes reduces storage from 42.4 MB/hour in EDF to 22.7 MB/hour, while block-aligned sampling reaches 3,131 windows/s, up to 11 times faster than Parquet. Reads remain bit-identical across independent Zarr implementations, and concurrent failure-injected appends recover without data loss. These results show that ZarrFM can provide a compact, interoperable, and execution-efficient storage layer for large-scale EEG pretraining.

## Ocient raises $100 million as agentic AI drives growth in large-scale analytics

DevFeed: [Ocient raises $100 million as agentic AI drives growth in large-scale analytics](<https://devfeed.tech/articles/ocient-raises-100-million-as-agentic-ai-drives-growth-in-large-scale-analytics-118837.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/05/ocient-raises-100-million-as-agentic-ai-drives-growth-in-large-scale-analytics/>)

Author: Paul Gillin

Published: 2026-11-05T12:00:23Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-artificial-intelligence](<https://devfeed.tech/tags/agentic-artificial-intelligence.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business](<https://devfeed.tech/tags/business.md>), [c](<https://devfeed.tech/tags/c.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analyltics](<https://devfeed.tech/tags/data-analyltics.md>), [data-analytics](<https://devfeed.tech/tags/data-analytics.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [demand](<https://devfeed.tech/tags/demand.md>), [funding](<https://devfeed.tech/tags/funding.md>), [growth](<https://devfeed.tech/tags/growth.md>), [large](<https://devfeed.tech/tags/large.md>), [national-security](<https://devfeed.tech/tags/national-security.md>), [news](<https://devfeed.tech/tags/news.md>), [ocient-inc](<https://devfeed.tech/tags/ocient-inc.md>), [ocientaiq-solutions](<https://devfeed.tech/tags/ocientaiq-solutions.md>), [ocientaiq-unified-data-platform](<https://devfeed.tech/tags/ocientaiq-unified-data-platform.md>), [platform](<https://devfeed.tech/tags/platform.md>), [raises](<https://devfeed.tech/tags/raises.md>), [scale](<https://devfeed.tech/tags/scale.md>), [the-latest](<https://devfeed.tech/tags/the-latest.md>), [top-story-5](<https://devfeed.tech/tags/top-story-5.md>), [top-story-6](<https://devfeed.tech/tags/top-story-6.md>)

### AI overview

Ocient raised $100 million in Series C funding to expand its OcientAIQ analytics business, bringing total funding to $260 million. Its platform lets AI agents and analysts query petabyte-scale datasets without sampling, combining data ingestion, analytics, governance and agent orchestration while minimizing data movement and using forward error correction instead of conventional replication to reduce storage overhead. The company reports growing demand from national security and telecommunications customers, including for cybersecurity analysis of complete historical network data; it provides infrastructure, not threat detection algorithms. Ocient says it often complements platforms such as Snowflake and Databricks, and that many current opportunities are on premises for cost, sovereignty or data protection reasons. It plans to use the funding for sales and customer delivery hiring, engineering, and geographic expansion; its CEO said the funding should help reach breakeven but gave no timetable.

### Source excerpt

Ocient Inc., the maker of a speedy data analytics platform, today said it has raised $100 million in a Series C funding to expand its agentic artificial intelligence analytics business. That business is fueled by growing demand from national security organizations and telecommunications providers seeking to analyze massive datasets without sampling. The new round brings [...] The post Ocient raises $100 million as agentic AI drives growth in large-scale analytics appeared first on SiliconANGLE.

## Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale

DevFeed: [Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale](<https://devfeed.tech/articles/knowledge-graph-lakehouse-a-cloud-native-architecture-for-managing-enterprise-knowledge-graphs-at-scale-115964.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04899>)

Author: Bashar Ahmad, Christoph G. Schuetz, Loris Bozzato

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Microservices](<https://devfeed.tech/topics/microservices.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [cloud-native-architecture](<https://devfeed.tech/tags/cloud-native-architecture.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

The Knowledge Graph Lakehouse constructs application-specific contextualized knowledge graphs on demand, rather than maintaining one monolithic graph. It indexes incoming documents along configurable context dimensions and stores them in a data lake; independently deployable microservices use configuration for context dimensions and document formats, keeping the architecture domain-independent. The authors report a proof of concept evaluated in experiments on a distributed cluster, but the supplied abstract gives no performance results.

### Source excerpt

arXiv:2611.04899v1 Announce Type: new Abstract: Modern data architectures for integrating heterogeneous data sources to discover actionable knowledge include the data warehouse, the data lake, and the data lakehouse, which combines the flexibility of the data lake with the structured data management capabilities of the data warehouse through metadata, indexes, and caching. Enterprise knowledge graphs (KGs) offer an alternative approach to such integration for the purposes of data analysis, but handling them at scale must address the Three Vs of big data: volume, velocity, and variety. Contextualized KGs (CKGs) meet part of that challenge by enriching facts with contextual information, yet many applications require only a context-specific subset of that knowledge. We propose the Knowledge Graph Lakehouse (KG Lakehouse), a cloud-native microservice architecture that adapts the data lakehouse paradigm for management of CKGs at scale. The KG Lakehouse indexes incoming source documents along configurable context dimensions and stores these documents in a data lake, from which application-specific CKGs are constructed on demand instead of maintaining a single, monolithic KG. We realize the proposed architecture as a proof of concept, with independently deployable services built from established cloud technology; context dimensions and format-specific document handling enter as configuration, leaving the architecture independent of the application domain. We evaluate the implemented system in experiments on a distributed cluster deployment.

## How fixing our data infrastructure before it broke us gave back 6,200 engineering hours

DevFeed: [How fixing our data infrastructure before it broke us gave back 6,200 engineering hours](<https://devfeed.tech/articles/how-fixing-our-data-infrastructure-before-it-broke-us-gave-back-6-200-engineering-hours-113146.md>)

Original publisher: [Read original article](<https://www.infoworld.com/article/4241788/how-fixing-our-data-infrastructure-before-it-broke-us-gave-back-6200-engineering-hours.html>)

Author: John Mitchem

Published: 2026-11-02T09:00:00Z

Content type: opinion

Language: en

Sources: [InfoWorld](<https://devfeed.tech/sources/infoworld.md>)

Topics: [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [digital sovereignty](<https://devfeed.tech/topics/digital-sovereignty.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [vendor lock-in](<https://devfeed.tech/topics/vendor-lock-in.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [data-infrastructure](<https://devfeed.tech/tags/data-infrastructure.md>), [data-management](<https://devfeed.tech/tags/data-management.md>), [data-management-databases](<https://devfeed.tech/tags/data-management-databases.md>), [databases](<https://devfeed.tech/tags/databases.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [observability](<https://devfeed.tech/tags/observability.md>), [platform](<https://devfeed.tech/tags/platform.md>), [storytype-opinion](<https://devfeed.tech/tags/storytype-opinion.md>)

### AI overview

A five-person contact center observability startup faced a prospective bank customer expecting five times its usual traffic, while a flaky hosted Elasticsearch setup and slow vendor support threatened customer commitments. The founders estimated that onboarding the bank without changing course would tie up three engineers in infrastructure work. Before launch, they moved to open source OpenSearch and had NetApp Instaclustr manage it and Postgres, later adding ClickHouse clusters. A key requirement was keeping data within their own environment for regulated customers. They report reclaiming more than 6,200 engineering hours annually, processing over 500 million interactions, and seeing fewer performance scares and routine failovers and upgrades. The author presents early evaluation as a way to make infrastructure choices on a planned schedule, while noting that running infrastructure in-house can be a real advantage for some companies.

### Source excerpt

Just about every CTO at a growth-stage company that I know is carrying an infrastructure problem they are quietly deciding not to look at. The system works, so the problem stays out of sight right up until something drags it into the open. That "something," in our particular case, was a large UK-based bank. The new customer arrived with the expectation of five times our usual traffic. For a nascent startup, the product-market fit really began to feel real. It's also in this moment that what had previously been seen as a technical chore can morph into a business risk. We're a contact center observability platform, so our work runs on processing especially large volumes of real-time interaction data for some of the world's largest contact centers. A couple years ago we were five engineers, and we ran our own data infrastructure. Frankly we were proud of it, enough that it showed up in our pitch. When that bank prospect came along, our reflex was to treat its scale as one more thing we would, and could, handle in-house. The math, for its part, said otherwise, or at least not without breaking something we had promised our customers. The bank did not create that risk, which had been quietly compounding for the better part of a year. We were leaning on a hosted Elasticsearch setup that had gotten flaky, from a vendor who was not fast enough on support for us to honor the promises we were making to our own customers. We knew all of this, but by then had gotten used to working around it like you might ignore the rattle in your car hoping it will just work itself out. What the bank deal did, in an instant, was take away our ability to keep ignoring it. The data infrastructure debt you stop noticing This is the trap that I now watch other teams wander into constantly. The system keeps humming right up until it doesn't, ergo the problem never forces its way onto a roadmap. CTOs and other decision makers end up making the call only once the stakes have climbed high enough that

## Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why

DevFeed: [Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why](<https://devfeed.tech/articles/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why-112261.md>)

Original publisher: [Read original article](<https://hackernoon.com/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why?source=rss>)

Author: Vikrant Bachhav

Published: 2026-11-02T06:00:04Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Messaging](<https://devfeed.tech/topics/messaging.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>)

Tags: [2019](<https://devfeed.tech/tags/2019.md>), [2022](<https://devfeed.tech/tags/2022.md>), [2023](<https://devfeed.tech/tags/2023.md>), [adtech](<https://devfeed.tech/tags/adtech.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [bi](<https://devfeed.tech/tags/bi.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [billing](<https://devfeed.tech/tags/billing.md>), [business](<https://devfeed.tech/tags/business.md>), [click](<https://devfeed.tech/tags/click.md>), [cloud-cost-optimization](<https://devfeed.tech/tags/cloud-cost-optimization.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-sampling-techniques](<https://devfeed.tech/tags/data-sampling-techniques.md>), [events](<https://devfeed.tech/tags/events.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [executor](<https://devfeed.tech/tags/executor.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stream](<https://devfeed.tech/tags/stream.md>), [unity](<https://devfeed.tech/tags/unity.md>), [volume](<https://devfeed.tech/tags/volume.md>), [workers](<https://devfeed.tech/tags/workers.md>)

### AI overview

Unity reduced ad exchange stream volume from about 10 GB/s to 2 GB/s by sampling pre-filter events at 0.5% while retaining post-filter events at full fidelity. Sampling by auction ID kept related events joinable; counts from sampled data were scaled by the inverse sampling probability, while ratios remained comparable when both parts used the same sample. After reviewing reporting needs with report owners, the team moved reports to a separate sampled Kafka topic and tuned Spark executors. Worker count fell from 240 to 30 and CPU utilization rose 41%; DSP Report transform billing was about 41% lower for March 9-April 6, 2023 than the prior period. Two weeks of parallel comparisons preceded report cutover. Validation also uncovered longstanding join duplication that inflated BidWin and WinPrice metrics. The account notes that dimensions considered unsafe to sample were kept at full fidelity.

### Source excerpt

How we reduced Kafka stream volume by 80%, cut Spark workers from 240 to 30, and lowered reporting costs through sampling and executor tuning at Unity.

## GESIS Surf: An Open-Source Infrastructure for Privacy-Preserving Longitudinal Web Browsing Data Collection

DevFeed: [GESIS Surf: An Open-Source Infrastructure for Privacy-Preserving Longitudinal Web Browsing Data Collection](<https://devfeed.tech/articles/gesis-surf-an-open-source-infrastructure-for-privacy-preserving-longitudinal-web-browsing-data-collection-111223.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.41441>)

Author: Mario Alberto Ramirez, Frank Mangold, Sebastian Stier

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Firefox extension](<https://devfeed.tech/topics/firefox-extension.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [Digital Public Good](<https://devfeed.tech/topics/digital-public-goods.md>), [Firefox](<https://devfeed.tech/topics/firefox.md>), [Google Chrome](<https://devfeed.tech/topics/google-chrome.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [backend](<https://devfeed.tech/tags/backend.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [browser](<https://devfeed.tech/tags/browser.md>), [browser-extension](<https://devfeed.tech/tags/browser-extension.md>), [browsing](<https://devfeed.tech/tags/browsing.md>), [capture](<https://devfeed.tech/tags/capture.md>), [chrome](<https://devfeed.tech/tags/chrome.md>), [client](<https://devfeed.tech/tags/client.md>), [collection](<https://devfeed.tech/tags/collection.md>), [content](<https://devfeed.tech/tags/content.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [data](<https://devfeed.tech/tags/data.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [database](<https://devfeed.tech/tags/database.md>), [digital](<https://devfeed.tech/tags/digital.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [web](<https://devfeed.tech/tags/web.md>), [web-browsing](<https://devfeed.tech/tags/web-browsing.md>)

### AI overview

GESIS Surf is an open-source infrastructure for longitudinal web behavior research, combining Chrome and Firefox extensions with a REST API backend and relational database. It passively records interactions, page metadata, and full HTML snapshots, organizing observations in a hierarchical session model. Privacy controls include strict opt-in participation, per-domain collection rules, and client-side data minimization. The abstract reports deployment with more than 1,000 participants in an academic panel that also collects survey data, supporting reproducible analysis of browsing behavior over time.

### Source excerpt

arXiv:2610.41441v1 Announce Type: new Abstract: Understanding how people navigate digital media environments requires behavioral data beyond survey self-reports. However, existing infrastructures for collecting such data reproducibly, sustainably, and with strong privacy guarantees remain limited: commercial solutions lack transparency and data governance, while academic tools are typically designed for narrow, short-term use cases and do not support general-purpose academic panel data collections. We address this gap with GESIS Surf, a research infrastructure for large-scale longitudinal collection of web behavioral data in naturalistic environments. The system consists of a browser extension (Chrome and Firefox), a backend with a REST application programming interface (API), and a relational database. It passively captures user interactions, page metadata, and full HTML snapshots of each rendered page, embedding each observation within a hierarchical session model that preserves the structure of browsing behavior. GESIS Surf implements privacy-by-design through strict opt-in participation, per-domain collection rules, and client-side data minimization, ensuring that sensitive data are controlled at the point of collection. The system has been deployed with more than 1,000 participants in a large-scale academic panel infrastructure that includes surveys of participants, and supports reproducible, content-level, and structurally rich analysis of digital behavior. By combining hierarchical session modeling, content-level capture, and integrated privacy enforcement within a unified system, GESIS Surf provides a general-purpose infrastructure for studying digital behavior in a reproducible manner over time.

## SPECTRUM: A Strategic Framework and Technical Blueprint for European Exascale Research Data and Compute Infrastructure

DevFeed: [SPECTRUM: A Strategic Framework and Technical Blueprint for European Exascale Research Data and Compute Infrastructure](<https://devfeed.tech/articles/spectrum-a-strategic-framework-and-technical-blueprint-for-european-exascale-research-data-and-compute-infrastructure-107595.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.40493>)

Author: Sergio Andreozzi, Stefano Bagnasco, Florian Berberich, Tommaso Boccali, Concezio Bozzi, Luis Cifuentes, Luca dell'Agnello, Ama\\"el Ellien, Chiara Ferrari, Luciano Gaido, Maria Girone, Hanno Holties, H

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [astro-ph-im](<https://devfeed.tech/tags/astro-ph-im.md>), [astronomy](<https://devfeed.tech/tags/astronomy.md>), [blueprint](<https://devfeed.tech/tags/blueprint.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [framework](<https://devfeed.tech/tags/framework.md>), [funding](<https://devfeed.tech/tags/funding.md>), [impact](<https://devfeed.tech/tags/impact.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [innovation](<https://devfeed.tech/tags/innovation.md>), [investment](<https://devfeed.tech/tags/investment.md>), [radio-astronomy](<https://devfeed.tech/tags/radio-astronomy.md>), [research](<https://devfeed.tech/tags/research.md>), [roadmap](<https://devfeed.tech/tags/roadmap.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spectrum](<https://devfeed.tech/tags/spectrum.md>), [strategic-research](<https://devfeed.tech/tags/strategic-research.md>), [technical](<https://devfeed.tech/tags/technical.md>), [use-cases](<https://devfeed.tech/tags/use-cases.md>)

### AI overview

SPECTRUM addresses the coming exabyte-scale data demands of HiLumi LHC, SKA Observatory and LOFAR 2.0 with two linked planning documents. Its Technical Blueprint maps a European compute and data continuum, consolidates gaps across High Energy Physics, Radio Astronomy and related use cases, and recommends actions in eight technical areas. Its Strategic Research, Innovation and Deployment Agenda sets thirteen priorities across four pillars, connects them to investment areas and European funding instruments, and phases them against the facilities' timelines. The paper summarizes both documents, their consultative method and the SRIDA's expected impact.

### Source excerpt

arXiv:2610.40493v1 Announce Type: new Abstract: The HiLumi LHC, the SKA Observatory and LOFAR 2.0 will take High Energy Physics (HEP) and Radio Astronomy (RA) towards exabyte-scale data within the next decade, beyond the capacity of today's infrastructure. The SPECTRUM project responded with two documents built on the same evidence. A Technical Blueprint describes a European compute and data continuum as a capability map, consolidates the gaps found in HEP, RA and related use cases, and recommends technical actions in eight areas. A Strategic Research, Innovation and Deployment Agenda (SRIDA) organises thirteen priorities in four pillars, maps them to investment areas and European funding instruments, and phases them over a roadmap set against the timelines of the three instruments. This paper, by editors and contributors of both documents, summarises them, their consultative method and the impact the SRIDA expects.

## Vast and Cloudera build a unified data foundation for enterprise AI

DevFeed: [Vast and Cloudera build a unified data foundation for enterprise AI](<https://devfeed.tech/articles/vast-and-cloudera-build-a-unified-data-foundation-for-enterprise-ai-106532.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/10/29/vast-and-cloudera-unified-data-foundation-for-enterprise-ai-evolve26/>)

Author: Victoria Gayton

Published: 2026-10-29T17:10:33Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [NVIDIA DGX](<https://devfeed.tech/topics/nvidia-dgx.md>), [data-governance](<https://devfeed.tech/topics/data-governance.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cloudera](<https://devfeed.tech/tags/cloudera.md>), [cloudera-anywhere-cloud](<https://devfeed.tech/tags/cloudera-anywhere-cloud.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [demand](<https://devfeed.tech/tags/demand.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [evolve26](<https://devfeed.tech/tags/evolve26.md>), [evolve2626eventpage](<https://devfeed.tech/tags/evolve2626eventpage.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [infra](<https://devfeed.tech/tags/infra.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [isv](<https://devfeed.tech/tags/isv.md>), [itamar-ankorion](<https://devfeed.tech/tags/itamar-ankorion.md>), [jeremy-reams](<https://devfeed.tech/tags/jeremy-reams.md>), [large](<https://devfeed.tech/tags/large.md>), [mission](<https://devfeed.tech/tags/mission.md>), [news](<https://devfeed.tech/tags/news.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [paul-gillin](<https://devfeed.tech/tags/paul-gillin.md>), [performance](<https://devfeed.tech/tags/performance.md>), [providers](<https://devfeed.tech/tags/providers.md>), [rebecca-knight](<https://devfeed.tech/tags/rebecca-knight.md>), [scale](<https://devfeed.tech/tags/scale.md>), [thecube](<https://devfeed.tech/tags/thecube.md>), [unified-data-foundation](<https://devfeed.tech/tags/unified-data-foundation.md>), [vast-data](<https://devfeed.tech/tags/vast-data.md>)

### AI overview

Vast and Cloudera describe a joint approach to enterprise AI that aims to support demanding workloads while limiting separate data copies. Vast's platform lets applications access the same file through different storage protocols; paired with Cloudera Anywhere Cloud data services, this is presented as a unified foundation for varied workloads. The companies say keeping data flowing to expensive, parallelized GPUs can improve their utilization. Cloudera is also expanding its AI ecosystem, with joint engineering and blueprints intended to help customers explore deployments and use cases. The excerpt does not provide measured performance or ROI results.

### Source excerpt

A unified data foundation can help enterprises support growing workloads without multiplying the systems and copies needed to serve them. Customer demand for AI infrastructure is pushing technology providers to work more closely together. Supporting large, mission-critical applications requires both scale and performance, according to Itamar Ankorion (pictured, left), head of strategic ISV alliances at [...] The post Vast and Cloudera build a unified data foundation for enterprise AI appeared first on SiliconANGLE.

## Production AI Doesn't Just Need to Work. It Needs to Scale

DevFeed: [Production AI Doesn't Just Need to Work. It Needs to Scale](<https://devfeed.tech/articles/production-ai-doesn-t-just-need-to-work-it-needs-to-scale-105911.md>)

Original publisher: [Read original article](<https://blog.everpuredata.com/perspectives/production-ai-doesnt-just-need-to-work-it-needs-to-scale/>)

Author: Cameron Bahar

Published: 2026-10-28T13:00:00Z

Content type: article

Language: en

Sources: [Everpure](<https://devfeed.tech/sources/everpure.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Service fabric](<https://devfeed.tech/topics/service-fabric.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [site-reliability-engineering](<https://devfeed.tech/topics/site-reliability-engineering.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-factories](<https://devfeed.tech/tags/ai-factories.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [context](<https://devfeed.tech/tags/context.md>), [context-management](<https://devfeed.tech/tags/context-management.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [performance](<https://devfeed.tech/tags/performance.md>), [perspectives](<https://devfeed.tech/tags/perspectives.md>), [production-ai](<https://devfeed.tech/tags/production-ai.md>), [proof-of-concept](<https://devfeed.tech/tags/proof-of-concept.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Production AI infrastructure must handle concurrency, constant data movement, growing context demands, and failures under load; a successful pilot does not show how the system will behave across thousands of GPUs and multiple teams. The excerpt warns that metadata can bottleneck workloads even when bandwidth appears sufficient, and that checkpoint bursts, tenant contention, and recovery can reduce GPU utilization. It recommends evaluating the system as a fleet under production pressure, including metadata capacity, checkpoint effects, data reuse, and recovery behavior. Everpure says FlashBlade//EXA uses a disaggregated metadata core and consolidates file and object workloads while preserving native access paths and semantics. STN reports up to 100 GB/s on FlashBlade//EXA versus about 35 GB/s on its previous architecture, and scaling to 192 nodes without finding a limit; these are vendor-reported examples, not general benchmarks.

### Source excerpt

Production AI Doesn't Just Need to Work. It Needs to Scale by Everpure Blog Discover why production AI needs more than a proof of concept and how FlashBlade//EXA helps deliver predictable performance as workloads scale. The post Production AI Doesn't Just Need to Work. It Needs to Scale appeared first on Everpure Blog.

## Quality-Driven Energy-Efficient Big Data Aggregation in WBANs

DevFeed: [Quality-Driven Energy-Efficient Big Data Aggregation in WBANs](<https://devfeed.tech/articles/quality-driven-energy-efficient-big-data-aggregation-in-wbans-104145.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2206.05699>)

Author: Amit Samanta, Tri Gia Nguyen

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [big-data-infrastructure](<https://devfeed.tech/tags/big-data-infrastructure.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [communications](<https://devfeed.tech/tags/communications.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [delay](<https://devfeed.tech/tags/delay.md>), [devices](<https://devfeed.tech/tags/devices.md>), [energy](<https://devfeed.tech/tags/energy.md>), [medical-data](<https://devfeed.tech/tags/medical-data.md>), [medical-research](<https://devfeed.tech/tags/medical-research.md>), [network](<https://devfeed.tech/tags/network.md>), [qos](<https://devfeed.tech/tags/qos.md>), [quality](<https://devfeed.tech/tags/quality.md>), [simulation](<https://devfeed.tech/tags/simulation.md>)

### AI overview

DEBA is a quality-driven, energy-efficient aggregation approach for cloud-assisted Wireless Body Area Networks (WBANs). It addresses large medical data volumes from low-powered sensor nodes with heterogeneous Quality-of-Service requirements across intra-BAN (Phase I) and inter-BAN (Phase II) communications. The abstract reports that simulations show improved network efficiency, measured by aggregation delay and cost, compared with existing schemes; it gives no quantitative results or further implementation details.

### Source excerpt

arXiv:2206.05699v1 Announce Type: cross Abstract: In the Internet-of-Things (IoT) era, the development of Wireless Body Area Networks (WBANs) and their applications in big data infrastructure has gotten a lot of attention from the medical research community. Since sensor nodes are low-powered devices that require heterogeneous Quality-of-Service (QoS), managing large amounts of medical data is critical in WBANs. Therefore, effectively aggregating a large volume of medical data is important. In this context, we propose a quality-driven and energy-efficient big data aggregation approach for cloud-assisted WBANs. For both intra-BAN (Phase I) and inter-BAN (Phase II) communications, the aggregation approach is cost-effective. Extensive simulation results show that the proposed approach DEBA greatly improves network efficiency in terms of aggregation delay and cost as compared to existing schemes.

## LSM-Vec: A Memory-Efficient Disk-Based System for Large-Scale Dynamic Vector Search

DevFeed: [LSM-Vec: A Memory-Efficient Disk-Based System for Large-Scale Dynamic Vector Search](<https://devfeed.tech/articles/lsm-vec-a-memory-efficient-disk-based-system-for-large-scale-dynamic-vector-search-101498.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2505.17152>)

Author: Dingheng Mo, Shurui Zhong, Yingzhuo Qian, Siqiang Luo

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [ann](<https://devfeed.tech/topics/ann.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ann](<https://devfeed.tech/tags/ann.md>), [approximate-nearest-neighbor](<https://devfeed.tech/tags/approximate-nearest-neighbor.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [graph](<https://devfeed.tech/tags/graph.md>), [hnsw](<https://devfeed.tech/tags/hnsw.md>), [index](<https://devfeed.tech/tags/index.md>), [indexes](<https://devfeed.tech/tags/indexes.md>), [lsm-tree](<https://devfeed.tech/tags/lsm-tree.md>), [memory](<https://devfeed.tech/tags/memory.md>), [pages](<https://devfeed.tech/tags/pages.md>), [queries](<https://devfeed.tech/tags/queries.md>), [query](<https://devfeed.tech/tags/query.md>), [rag](<https://devfeed.tech/tags/rag.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>), [system](<https://devfeed.tech/tags/system.md>), [vector](<https://devfeed.tech/tags/vector.md>), [vector-search](<https://devfeed.tech/tags/vector-search.md>)

### AI overview

LSM-Vec separates HNSW graph updates from vector storage to reduce disk and memory costs for dynamic, large-scale vector search. A graph-oriented LSM-tree absorbs edge changes out of place, while vectors reside in a separate paged store. Compact in-memory tree sketches route queries to nearby disk entries, and HNSW-based reordering places vectors evaluated together on nearby pages to reduce scattered reads. On streaming SIFT and SPACEV workloads up to 100 million vectors, the authors report up to 6.2x less memory and 7.6x less disk than FreshDiskANN and SPFresh, and up to 15x fewer bytes written per insertion. At 100 million vectors, LSM-Vec maintains stable tail latency during updates, reported at 2.3-3.8x below SPFresh at comparable recall, while SPFresh's index outgrows memory and FreshDiskANN exhausts it.

### Source excerpt

arXiv:2505.17152v3 Announce Type: replace Abstract: RAG and AI retrieval have made vector approximate nearest-neighbor (ANN) search a key retrieval operation in modern data systems. These workloads are both large and dynamic. Vector collections grow until index state no longer fits in DRAM and must reside on disk, and every new record arrives as an update the index must absorb while queries continue to run. Existing dynamic designs on disk leave two architecture-level gaps. First, graph connections and vector data follow different access and update paths, yet many disk-based indexes store them as one physical record, so a graph-only update rewrites vector data it does not modify. Second, keeping an index fresh gives memory a role beyond caching, because these systems either hold recent updates in in-memory indexes or reserve memory to maintain a mutable disk layout. We present LSM-Vec, a disk-based dynamic vector search system built on a decoupled architecture. LSM-Vec stores the HNSW base layer in a graph-oriented LSM-tree that absorbs fine-grained edge changes out of place, and keeps vector data in a separate paged store. To keep this design query-efficient within a small memory budget, we design in-memory tree-based sketches that route each query to an entry close to its target on disk, so the resident hierarchy stays compact. We further introduce HNSW-based reordering, which places vector data that queries evaluate together on nearby disk pages and reduces scattered reads. On streaming SIFT and SPACEV workloads with up to a hundred million vectors, LSM-Vec requires up to 6.2$\times$ less memory and 7.6$\times$ less disk than FreshDiskANN and SPFresh, and writes up to 15$\times$ fewer bytes per insertion. At the hundred-million scale, where SPFresh's index outgrows memory and FreshDiskANN exhausts it, LSM-Vec keeps its tail latency stable throughout the update stream, 2.3--3.8$\times$ below SPFresh's at comparable recall.

## Scale reservoir uncertainty analysis with agile reservoir modeling

DevFeed: [Scale reservoir uncertainty analysis with agile reservoir modeling](<https://devfeed.tech/articles/scale-reservoir-uncertainty-analysis-with-agile-reservoir-modeling-107677.md>)

Original publisher: [Read original article](<https://azure.microsoft.com/en-us/blog/scale-reservoir-uncertainty-analysis-with-agile-reservoir-modeling/>)

Author: Sverre Brandsberg Dahl

Published: 2026-10-26T16:00:00Z

Content type: article

Language: en

Sources: [Microsoft Azure](<https://devfeed.tech/sources/microsoft-azure-blog.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>), [Azure](<https://devfeed.tech/topics/azure.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [account](<https://devfeed.tech/tags/account.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agile](<https://devfeed.tech/tags/agile.md>), [ai](<https://devfeed.tech/tags/ai.md>), [all](<https://devfeed.tech/tags/all.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [azure](<https://devfeed.tech/tags/azure.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [data](<https://devfeed.tech/tags/data.md>), [faster](<https://devfeed.tech/tags/faster.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-azure-data-manager-for-energy](<https://devfeed.tech/tags/microsoft-azure-data-manager-for-energy.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [need](<https://devfeed.tech/tags/need.md>), [scale](<https://devfeed.tech/tags/scale.md>), [storage](<https://devfeed.tech/tags/storage.md>), [teams](<https://devfeed.tech/tags/teams.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

SLB's generally available agile reservoir modeling (ARM), integrated with Azure Data Manager for Energy (ADME), distributes reservoir realization generation and simulation across elastic Azure high-performance computing. Shared Azure Files storage supports concurrent access, while compute and storage capacity, throughput, and IOPS can be scaled independently for ensemble workloads. Petrel connects engineers' uncertainty workflows to governed ADME data; ARM orchestrates supported SLB simulator runs, including Intersect and Eclipse; and ADME's OSDU-aligned services support data management, search, discovery, and reuse. Teams can analyze results in the reservoir engineering workspace or return them to Petrel. The article says this integration can reduce disconnected data copies and make ensemble studies repeatable, while entitlement-based access, Microsoft Entra ID, and OSDU legal tags provide access and compliance controls. It also describes future OSDU community work toward more cloud-native interoperability as ongoing, not a current capability.

### Source excerpt

As subsurface data grows in scale and complexity, teams need a faster, more connected way to transform uncertainty into actionable decisions. The post Scale reservoir uncertainty analysis with agile reservoir modeling appeared first on Microsoft Azure Blog.

## Data Enablement for AI: A Practitioner's Guide to 6 Advanced Architectural Patterns

DevFeed: [Data Enablement for AI: A Practitioner's Guide to 6 Advanced Architectural Patterns](<https://devfeed.tech/articles/data-enablement-for-ai-a-practitioner-s-guide-to-6-advanced-architectural-patterns-99143.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/data-enablement-for-ai-a-practitioners-guide-to-6-advanced-architectural-patterns/>)

Author: Partha Sarkar

Published: 2026-10-25T12:00:00Z

Content type: article

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [enterprise data protection](<https://devfeed.tech/topics/enterprise-data-protection.md>), [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [advanced](<https://devfeed.tech/tags/advanced.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-systems](<https://devfeed.tech/tags/agentic-systems.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-data](<https://devfeed.tech/tags/ai-data.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [context-engineering](<https://devfeed.tech/tags/context-engineering.md>), [data](<https://devfeed.tech/tags/data.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-enablement](<https://devfeed.tech/tags/data-enablement.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [event-driven-architecture](<https://devfeed.tech/tags/event-driven-architecture.md>), [governed-ai](<https://devfeed.tech/tags/governed-ai.md>), [guide](<https://devfeed.tech/tags/guide.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [practitioner](<https://devfeed.tech/tags/practitioner.md>), [production](<https://devfeed.tech/tags/production.md>), [semantic-layer](<https://devfeed.tech/tags/semantic-layer.md>)

### AI overview

The guide argues that production enterprise AI depends on a governed, fresh data foundation, not prompt engineering alone. It describes six patterns: CDC-driven updates with content hashing and deletion tombstones; hierarchical parsing that preserves document structure and table relationships; a certified semantic layer that turns natural-language requests into typed intent and deterministically compiled SQL; an in-memory feature store for fast operational state; source-permission metadata enforced through pre-query retrieval filtering; and tiered data curation with deduplication and synthetic questions to improve retrieval. Each pattern has costs and requirements: streaming and indexing infrastructure, GPU-based parsing, maintained metric catalogs, in-memory storage, filtered vector engines, or evaluated small models. The guide warns against batch-only indexing, raw-table querying, post-retrieval security checks, blind text splitting, and storing rapidly changing operational data in vector indices.

### Source excerpt

Beyond naive chunk-and-embed: Six production architectures for turning enterprise lakehouses into governed AI platforms. The post Data Enablement for AI: A Practitioner's Guide to 6 Advanced Architectural Patterns appeared first on Towards Data Science.

## Why Modern Data Lakehouses Fail at Scale: The Metadata Problem Nobody Talks About

DevFeed: [Why Modern Data Lakehouses Fail at Scale: The Metadata Problem Nobody Talks About](<https://devfeed.tech/articles/why-modern-data-lakehouses-fail-at-scale-the-metadata-problem-nobody-talks-about-98982.md>)

Original publisher: [Read original article](<https://hackernoon.com/why-modern-data-lakehouses-fail-at-scale-the-metadata-problem-nobody-talks-about?source=rss>)

Author: FNU Aditya Ranjan

Published: 2026-10-24T17:00:02Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [applications](<https://devfeed.tech/tags/applications.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [delta](<https://devfeed.tech/tags/delta.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [files](<https://devfeed.tech/tags/files.md>), [growth](<https://devfeed.tech/tags/growth.md>), [hackernoon-top-story](<https://devfeed.tech/tags/hackernoon-top-story.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [metadata-management](<https://devfeed.tech/tags/metadata-management.md>), [modern](<https://devfeed.tech/tags/modern.md>), [modern-data](<https://devfeed.tech/tags/modern-data.md>), [modern-data-lakehouses](<https://devfeed.tech/tags/modern-data-lakehouses.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [small](<https://devfeed.tech/tags/small.md>)

### AI overview

Lakehouse performance can degrade as file counts, partition fragmentation, and metadata grow, even when object storage scales. Small files increase metadata handling, object operations, scheduling, and scan overhead; compaction can improve layout but costs compute and I/O. Iceberg describes table state through metadata and manifests, while Delta Lake uses a transaction log and checkpoints, so each requires lifecycle maintenance. Partitioning helps pruning only when it fits real workloads; overly granular partitions can add more file and metadata overhead than benefit. Snapshot expiration and orphan cleanup must preserve reader, rollback, and concurrency requirements. The excerpt proposes comparing no proactive optimization, fixed-schedule maintenance, and workload-aware triggers using matched workloads and configurations, measuring latency, planning time, operations, rewrite volume, storage, and cost. This framework is proposed, not experimentally validated.

### Source excerpt

Explore why modern data lakehouses slow down at scale and how metadata growth, small files, partitioning, and maintenance affect Iceberg and Delta Lake.

## Three Ways of Storing and Accessing Lots of Images in Python

DevFeed: [Three Ways of Storing and Accessing Lots of Images in Python](<https://devfeed.tech/articles/three-ways-of-storing-and-accessing-lots-of-images-in-python-98865.md>)

Original publisher: [Read original article](<https://realpython.com/storing-images-in-python/>)

Author: Rebecca Stone

Published: 2026-10-24T14:00:00Z

Content type: tutorial

Language: en

Sources: [Real Python](<https://devfeed.tech/sources/real-python.md>)

Topics: [Python](<https://devfeed.tech/topics/python.md>), [Image](<https://devfeed.tech/topics/image.md>), [cifar-10](<https://devfeed.tech/topics/cifar-10.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [pip](<https://devfeed.tech/topics/pip.md>), [OpenCV](<https://devfeed.tech/topics/opencv.md>)

Tags: [compare](<https://devfeed.tech/tags/compare.md>), [files](<https://devfeed.tech/tags/files.md>), [images](<https://devfeed.tech/tags/images.md>), [python](<https://devfeed.tech/tags/python.md>), [reading](<https://devfeed.tech/tags/reading.md>), [writing](<https://devfeed.tech/tags/writing.md>)

### AI overview

On the CIFAR-10 experiments described, reading and writing many images was much faster with LMDB and HDF5 than with separate PNG files; HDF5 started slower but edged ahead of LMDB at larger image counts. For a single image, all three methods were quick, and PNG was slightly faster to read. LMDB stores serialized key-value records in a memory-mapped database and supports multiple readers but only one writer; it requires advance `map_size` planning and may need larger storage for larger images. HDF5 stores homogeneous arrays in datasets, supports indexed and range access, and may be easier to extend, though concurrent access can be sequential without a parallel file system. Both used more disk space than PNG in the sample, and results depend on data size, operating system, and access patterns. The article recommends planning storage around dataset structure and workload; no method is universally best.

### Source excerpt

Compare three ways of storing images in Python, as PNG files on disk, in LMDB, and in HDF5, with timings for reading and writing lots of images.

## Podcast #155: Micron's 245TB SSD and What Extreme Density Enables at Scale

DevFeed: [Podcast #155: Micron's 245TB SSD and What Extreme Density Enables at Scale](<https://devfeed.tech/articles/podcast-155-micron-s-245tb-ssd-and-what-extreme-density-enables-at-scale-93947.md>)

Original publisher: [Read original article](<https://www.storagereview.com/podcast/podcast-155-micron-245tb-ssd-extreme-density-bill-bollengier>)

Author: Harold Fritts

Published: 2026-10-21T21:21:34Z

Content type: article

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [bill](<https://devfeed.tech/tags/bill.md>), [drive](<https://devfeed.tech/tags/drive.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [first](<https://devfeed.tech/tags/first.md>), [global](<https://devfeed.tech/tags/global.md>), [ion](<https://devfeed.tech/tags/ion.md>), [linkedin](<https://devfeed.tech/tags/linkedin.md>), [live](<https://devfeed.tech/tags/live.md>), [management](<https://devfeed.tech/tags/management.md>), [micron](<https://devfeed.tech/tags/micron.md>), [ocp](<https://devfeed.tech/tags/ocp.md>), [october](<https://devfeed.tech/tags/october.md>), [podcast](<https://devfeed.tech/tags/podcast.md>), [power](<https://devfeed.tech/tags/power.md>), [product](<https://devfeed.tech/tags/product.md>), [product-management](<https://devfeed.tech/tags/product-management.md>), [scale](<https://devfeed.tech/tags/scale.md>), [space](<https://devfeed.tech/tags/space.md>), [ssd](<https://devfeed.tech/tags/ssd.md>), [summit](<https://devfeed.tech/tags/summit.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Micron's 245TB 6600 ION targets the warm storage tier now held largely by hard drives, increasing capacity per bay while reducing space and power needs. The excerpt reports Micron estimates up to 86% better storage efficiency per watt and about 921 MWh saved annually for a one-exabyte deployment versus hard drives. The drive uses QLC NAND and PCIe Gen5; Micron reserves Gen6 for high-performance drives because bandwidth at this capacity would demand more power. Its 16K indirection unit keeps mapping-table and DRAM needs in check while avoiding host software changes. The speakers describe tiering across tape, hard drives, TLC and QLC flash, and emphasize matching storage to workload, network bandwidth and service needs. They do not predict that flash will replace all hard drives.

### Source excerpt

Brian hosted Bill Bollengier, Senior Director of Product Management at Micron, for a live podcast on LinkedIn on October 15, with Bill calling in from the floor of the OCP Global Summit. The subject was the Micron 6600 ION 245TB SSD and what a quarter petabyte per drive bay does to floor space, power budgets, The post Podcast #155: Micron's 245TB SSD and What Extreme Density Enables at Scale appeared first on StorageReview.com.

## The bridge to agent-scale Git infrastructure

DevFeed: [The bridge to agent-scale Git infrastructure](<https://devfeed.tech/articles/the-bridge-to-agent-scale-git-infrastructure-92954.md>)

Original publisher: [Read original article](<https://github.blog/engineering/architecture-optimization/the-bridge-to-agent-scale-git-infrastructure/>)

Author: Eleanor White

Published: 2026-10-21T16:00:00Z

Content type: article

Language: en

Sources: [GitHub Blog](<https://devfeed.tech/sources/github-engineering.md>), [GitHub Engineering](<https://devfeed.tech/sources/github.md>)

Topics: [Git](<https://devfeed.tech/topics/git.md>), [Azure](<https://devfeed.tech/topics/azure.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Infrastructure as a Service (IaaS)](<https://devfeed.tech/topics/infrastructure-as-a-service-iaas.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2025](<https://devfeed.tech/tags/2025.md>), [agent](<https://devfeed.tech/tags/agent.md>), [architecture-optimization](<https://devfeed.tech/tags/architecture-optimization.md>), [azure](<https://devfeed.tech/tags/azure.md>), [blog](<https://devfeed.tech/tags/blog.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [copy](<https://devfeed.tech/tags/copy.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [first](<https://devfeed.tech/tags/first.md>), [git](<https://devfeed.tech/tags/git.md>), [github](<https://devfeed.tech/tags/github.md>), [here](<https://devfeed.tech/tags/here.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [object](<https://devfeed.tech/tags/object.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [over](<https://devfeed.tech/tags/over.md>), [post](<https://devfeed.tech/tags/post.md>), [reference](<https://devfeed.tech/tags/reference.md>), [repositories](<https://devfeed.tech/tags/repositories.md>), [scale](<https://devfeed.tech/tags/scale.md>), [storage](<https://devfeed.tech/tags/storage.md>), [technical-deep-dive](<https://devfeed.tech/tags/technical-deep-dive.md>)

### AI overview

GitHub says per-reference write quorums enabled its busiest repositories to move to Azure despite roughly 25 ms cross-region round trips. The old repository-wide lock made each write wait for the slowest healthy replica; serializing only updates to the same reference lets independent branch updates proceed concurrently. In realistic internal tests, writes to different references achieved 3-4x higher throughput at lower latency. To scale reads separately, GitHub adds Azure Blob Storage as a virtual quorum replica: writes are acknowledged only after becoming durable there, while ephemeral read replicas warm caches from storage and serve traffic without joining the quorum. The added durability step costs some write latency, and background reconciliation limits which read workloads virtual replicas currently support. GitHub identifies quorum-bound write throughput and reconciliation for broader read support as remaining challenges.

### Source excerpt

We moved a copy of 99.99999% of GitHub's repositories to Azure. Here's how per-reference quorums will bring over the rest, and how object storage lets reads scale independently. The post The bridge to agent-scale Git infrastructure appeared first on The GitHub Blog.

## StarMind: Challenges and Opportunities in Orbital Computing

DevFeed: [StarMind: Challenges and Opportunities in Orbital Computing](<https://devfeed.tech/articles/starmind-challenges-and-opportunities-in-orbital-computing-93849.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.26916>)

Author: Zisheng Gong, Zhaoyu Liu, Houtianfu Wang, O. Tansel Baydas, Liu Cao, Xia Xu, Ozgur B. Akan

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [resource orchestration](<https://devfeed.tech/topics/resource-orchestration.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [computing](<https://devfeed.tech/tags/computing.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [e2e](<https://devfeed.tech/tags/e2e.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [federated](<https://devfeed.tech/tags/federated.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [large](<https://devfeed.tech/tags/large.md>), [latency](<https://devfeed.tech/tags/latency.md>), [layers](<https://devfeed.tech/tags/layers.md>), [links](<https://devfeed.tech/tags/links.md>), [mission](<https://devfeed.tech/tags/mission.md>), [model](<https://devfeed.tech/tags/model.md>)

### AI overview

StarMind proposes a five-layer framework and six standardizable interfaces for coordinating distributed computing across heterogeneous orbital systems. In a representative SAR/InSAR case study, resource-aware task placement and gateway selection reduce end-to-end latency to 41 minutes, compared with 61 minutes for localized onboard processing and 146 minutes for ground processing. The improvement comes from reducing contact waits and urgent Earth downlinks, not from faster orbital processors. The approach is more effective with large source datasets, compact outputs, and sufficient optical inter-satellite link capacity; otherwise, conventional processing may be preferable. The article also identifies joint task partitioning and cross-domain orchestration, hierarchical federated intelligence, and large model-enabled intelligence as research opportunities. This overview reflects the supplied abstract.

### Source excerpt

arXiv:2610.26916v1 Announce Type: new Abstract: Space-based artificial intelligence (AI) is moving beyond isolated onboard processing toward constellation-scale computing over optical inter-satellite links (OISLs). Yet current infrastructures remain fragmented, making end-to-end (E2E) coordination across heterogeneous and multi-vendor systems difficult. This article examines the StarMind vision and develops a reference framework for distributed orbital computing. It organizes the system into five layers and defines six standardizable interfaces that link mission intent to resource conditions and trustworthy result delivery. The framework also clarifies the operational constraints that determine whether orbital cooperation is worthwhile. In a representative synthetic aperture radar/interferometric SAR (SAR/InSAR) case study, the framework reduces E2E latency to 41-min, compared with 61-min for localized onboard computing and 146-min for ground-based processing. This gain does not rely on faster processors in orbit. Instead, it comes from reducing contact waiting and urgent Earth downlink traffic through resource-aware task placement and gateway selection. Feasibility analysis shows that the proposed framework is more effective when source datasets are large, output products are compact, and optical inter-satellite link capacity is sufficient. When these conditions are absent, conventional processing may remain preferable. Finally, we discuss research opportunities in joint task partitioning and cross-domain orchestration, hierarchical federated intelligence, and large model-enabled intelligence to advance an interoperable space-ground computing continuum.

## How Fortune 500 Companies Build Data Platforms

DevFeed: [How Fortune 500 Companies Build Data Platforms](<https://devfeed.tech/articles/how-fortune-500-companies-build-data-platforms-89724.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-fortune-500-companies-build-data-platforms?source=rss>)

Author: Sunilkumar Reddy Eraganeni

Published: 2026-10-18T11:13:22Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [data-platforms](<https://devfeed.tech/topics/data-platforms.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [ai](<https://devfeed.tech/tags/ai.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [bad-data](<https://devfeed.tech/tags/bad-data.md>), [batch](<https://devfeed.tech/tags/batch.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakehouse-architecture](<https://devfeed.tech/tags/data-lakehouse-architecture.md>), [data-lineage-and-quality](<https://devfeed.tech/tags/data-lineage-and-quality.md>), [data-mesh-architecture](<https://devfeed.tech/tags/data-mesh-architecture.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-platform-architecture](<https://devfeed.tech/tags/data-platform-architecture.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [data-team](<https://devfeed.tech/tags/data-team.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [enterprise-data-engineering](<https://devfeed.tech/tags/enterprise-data-engineering.md>), [enterprise-data-governance](<https://devfeed.tech/tags/enterprise-data-governance.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [fortune-500-technology](<https://devfeed.tech/tags/fortune-500-technology.md>), [governance](<https://devfeed.tech/tags/governance.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

The article describes common patterns in large enterprise data platforms, including layered architectures, lakehouses, batch and streaming pipelines, data quality checks, data mesh ownership, and governance. It emphasizes that architectural choices depend on team needs and that there is no single correct design.

### Source excerpt

Explore how large enterprises build scalable data platforms using lakehouses, batch and streaming pipelines, data mesh, governance, and layered architectures.

## Scale enterprise analytics by running Cloudera Data Platform with OpenShift Virtualization

DevFeed: [Scale enterprise analytics by running Cloudera Data Platform with OpenShift Virtualization](<https://devfeed.tech/articles/scale-enterprise-analytics-by-running-cloudera-data-platform-with-openshift-virtualization-83990.md>)

Original publisher: [Read original article](<https://www.redhat.com/en/blog/scale-enterprise-analytics-running-cloudera-data-platform-openshift-virtualization>)

Author: Patrick Farley; Firas Yasin; Kuldeep Sahu

Published: 2026-10-09T00:00:00Z

Content type: article

Language: en

Sources: [Red Hat Blog](<https://devfeed.tech/sources/red-hat-blog.md>)

Topics: [virtualization](<https://devfeed.tech/topics/virtualization.md>), [rhel](<https://devfeed.tech/topics/rhel.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [big-data](<https://devfeed.tech/tags/big-data.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [red-hat-openshift](<https://devfeed.tech/tags/red-hat-openshift.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [virtualization](<https://devfeed.tech/tags/virtualization.md>)

### AI overview

Red Hat and Cloudera validated Cloudera Data Platform on Red Hat OpenShift Virtualization, describing a hybrid architecture that runs Cloudera Base in virtual machines and Cloudera Data Services on bare-metal OpenShift. The article outlines the 110-node benchmark, deployment considerations such as memory and storage sizing, and unified operations, security, and governance across virtual machines and containers.

### Source excerpt

IT leaders face a frustrating dilemma when scaling big data: Should they force engineering teams to rewrite applications running on traditional virtual machines (VMs) for containers, or maintain separate, expensive infrastructure environments? Red Hat OpenShift lets organizations modernize VMs alongside containers--without forcing immediate application refactoring.To help organizations facing this dilemma, Cloudera has validated Cloudera Data Platform (CDP) on Red Hat OpenShift Virtualization at scale--testing deployments across more than 100 virtualized nodes. On the data side, organization

## NetApp's Novus tackles metadata bottlenecks in AI data infrastructure

DevFeed: [NetApp's Novus tackles metadata bottlenecks in AI data infrastructure](<https://devfeed.tech/articles/netapp-s-novus-tackles-metadata-bottlenecks-in-ai-data-infrastructure-65247.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/10/05/netapps-novus-targets-ai-data-infrastructure-zettabyte-scale-netappinsight/>)

Author: Sloane Kali Faye

Published: 2026-10-05T22:58:23Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Data Infrastructure](<https://devfeed.tech/topics/data-infrastructure.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [data centers](<https://devfeed.tech/topics/data-centers.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-data-engine](<https://devfeed.tech/tags/ai-data-engine.md>), [ai-data-infrastructure](<https://devfeed.tech/tags/ai-data-infrastructure.md>), [christophe-bertrand](<https://devfeed.tech/tags/christophe-bertrand.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [data](<https://devfeed.tech/tags/data.md>), [enterprise-storage](<https://devfeed.tech/tags/enterprise-storage.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-clusters](<https://devfeed.tech/tags/gpu-clusters.md>), [gunna-marripudi](<https://devfeed.tech/tags/gunna-marripudi.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [linux](<https://devfeed.tech/tags/linux.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [nasdaq-ntap](<https://devfeed.tech/tags/nasdaq-ntap.md>), [netapp-inc](<https://devfeed.tech/tags/netapp-inc.md>), [netapp-insight](<https://devfeed.tech/tags/netapp-insight.md>), [netapp-insight-2026](<https://devfeed.tech/tags/netapp-insight-2026.md>), [netapp-novus-data](<https://devfeed.tech/tags/netapp-novus-data.md>), [netapp-s-ai-data-engine](<https://devfeed.tech/tags/netapp-s-ai-data-engine.md>), [netappinsight](<https://devfeed.tech/tags/netappinsight.md>), [news](<https://devfeed.tech/tags/news.md>), [novus-data-director](<https://devfeed.tech/tags/novus-data-director.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [ontap](<https://devfeed.tech/tags/ontap.md>), [parallel-file-systems](<https://devfeed.tech/tags/parallel-file-systems.md>), [peak-aio](<https://devfeed.tech/tags/peak-aio.md>), [pnfs](<https://devfeed.tech/tags/pnfs.md>), [production](<https://devfeed.tech/tags/production.md>), [production-ai](<https://devfeed.tech/tags/production-ai.md>), [rebecca-knight](<https://devfeed.tech/tags/rebecca-knight.md>), [storage](<https://devfeed.tech/tags/storage.md>), [syam-nair](<https://devfeed.tech/tags/syam-nair.md>), [thecube](<https://devfeed.tech/tags/thecube.md>)

### AI overview

NetApp presents Novus as an AI data infrastructure architecture designed to reduce metadata bottlenecks while scaling GPU workloads and making enterprise data accessible to AI models and agents. Its Data Director separates metadata from stored data, enabling independent scaling and a unified view across ONTAP clusters. NetApp says its AI Data Engine can discover, classify and vectorize enterprise data, while planned pNFS technology is intended to work alongside existing systems.

### Source excerpt

AI data infrastructure must support growing GPU clusters while making enterprise data usable for models and agents. NetApp Inc. is tying intelligent data infrastructure to production AI through a unified approach to storage and data management. Fragmented data and governance designed around manually applied policies can prevent enterprises from realizing returns on AI investments, according [...] The post NetApp's Novus tackles metadata bottlenecks in AI data infrastructure appeared first on SiliconANGLE.

[Next page](<https://devfeed.tech/topics/big-data-infrastructure.md?cursor=WyIyMDI2LTEwLTA1VDIyOjU4OjIzKzAwOjAwIiwgIjFiYWUwZWQxLTM3M2ItNDhkOC04MTk0LTgyN2ZlNzU4ZWZkYyJd>)