# big-data

Big data refers to extensive datasets characterized by volume, variety, velocity, and/or variability that require scalable architectures for efficient storage, manipulation, and analysis.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap

DevFeed: [Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap](<https://devfeed.tech/articles/hitachi-vantara-bets-smarter-storage-can-close-ai-s-pilot-to-production-gap-144137.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/19/apache-iceberg-intelligent-storage-help-ai-reach-production-sc26/>)

Author: Kelly Knight

Published: 2026-11-19T18:39:55Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [accelerated-computing](<https://devfeed.tech/tags/accelerated-computing.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data-centers](<https://devfeed.tech/tags/ai-data-centers.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-supercomputers](<https://devfeed.tech/tags/ai-supercomputers.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [arun-prasad-raman](<https://devfeed.tech/tags/arun-prasad-raman.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [compute-express-link](<https://devfeed.tech/tags/compute-express-link.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [data](<https://devfeed.tech/tags/data.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [data-scientists](<https://devfeed.tech/tags/data-scientists.md>), [database](<https://devfeed.tech/tags/database.md>), [energy](<https://devfeed.tech/tags/energy.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [etl](<https://devfeed.tech/tags/etl.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gemma-allen](<https://devfeed.tech/tags/gemma-allen.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [high-performance-computing](<https://devfeed.tech/tags/high-performance-computing.md>), [high-performance-storage](<https://devfeed.tech/tags/high-performance-storage.md>), [high-speed-networking](<https://devfeed.tech/tags/high-speed-networking.md>), [hitachi-iq](<https://devfeed.tech/tags/hitachi-iq.md>), [hitachi-vantara](<https://devfeed.tech/tags/hitachi-vantara.md>), [hpc](<https://devfeed.tech/tags/hpc.md>), [hpc-infrastructure](<https://devfeed.tech/tags/hpc-infrastructure.md>), [hpc-unites](<https://devfeed.tech/tags/hpc-unites.md>), [jeff-lundberg](<https://devfeed.tech/tags/jeff-lundberg.md>), [john-furrier](<https://devfeed.tech/tags/john-furrier.md>), [memory-pooling](<https://devfeed.tech/tags/memory-pooling.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [parallel-file-systems](<https://devfeed.tech/tags/parallel-file-systems.md>), [pilot](<https://devfeed.tech/tags/pilot.md>), [production](<https://devfeed.tech/tags/production.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [siliconangle](<https://devfeed.tech/tags/siliconangle.md>), [storage](<https://devfeed.tech/tags/storage.md>), [thecube](<https://devfeed.tech/tags/thecube.md>)

### AI overview

The excerpt identifies fragmented, ungoverned data and access controls as obstacles to moving AI projects into production, and describes Hitachi Vantara's approach of adding classification and governance directly to storage. Its VSP One object store can flag files containing personal information so organizations can exclude them from models or restrict cross-border replication. For analytics, the company describes storing CT scan metadata in Apache Iceberg and querying it directly--for example, to find abdominal scans from the past year--without a separate ETL process or database. These are company representatives' examples; the excerpt does not provide independent performance results.

### Source excerpt

Enterprises pushing AI from pilot to production are learning that data, not GPUs, is the hardest part, and open table formats such as Apache Iceberg are part of the answer. Demand for AI infrastructure shows few signs of slowing, but much of the information companies need is still scattered, ungoverned and out of reach. That [...] The post Hitachi Vantara bets smarter storage can close AI's pilot-to-production gap appeared first on SiliconANGLE.

## Architectural Design Decisions for Data Ingestion in Big Data Architectures

DevFeed: [Architectural Design Decisions for Data Ingestion in Big Data Architectures](<https://devfeed.tech/articles/architectural-design-decisions-for-data-ingestion-in-big-data-architectures-142161.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23297>)

Author: Yikang Huang, Stefano Fossati, Indika Kumara, Damian A. Tamburri, Willem-Jan van den Heuvel

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-ingestion-pipeline](<https://devfeed.tech/tags/data-ingestion-pipeline.md>), [data-ingestion-pipelines](<https://devfeed.tech/tags/data-ingestion-pipelines.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [decision](<https://devfeed.tech/tags/decision.md>), [design](<https://devfeed.tech/tags/design.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [experts](<https://devfeed.tech/tags/experts.md>), [framework](<https://devfeed.tech/tags/framework.md>), [functional](<https://devfeed.tech/tags/functional.md>), [options](<https://devfeed.tech/tags/options.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

The paper presents a framework to help data engineers and architects choose designs for data ingestion pipelines based on functional and non-functional requirements. A systematic review of 42 gray literature articles and qualitative analysis produced a catalog of 8 architectural decisions, 59 options, and 36 selection drivers. Semi-structured expert interviews assessed the framework's completeness, simplicity, understandability, operational feasibility, and usefulness. The supplied text reports that these qualities were evaluated, but does not give the interview findings or name the decisions and options.

### Source excerpt

arXiv:2611.23297v1 Announce Type: new Abstract: Introduction: Data ingestion is the first critical step in a data architecture, in which data from various sources is moved to storage systems, such as data lakes and data warehouses, for analysis and insight generation. When developing a data ingestion pipeline, practitioners (e.g., data engineers and architects) must understand the data ingestion requirements (functional and non-functional) and select appropriate design options to support them. Objective: To help practitioners systematically develop data ingestion processes, this paper aims to compile a catalog of data ingestion design decisions, their options, and the selection drivers. Method: We conducted a systematic gray literature review on data ingestion, analyzed the selected literature using qualitative data analysis methods, and conducted semi-structured interviews with experts to assess the framework's practicality and utility. Results: From 42 gray literature articles, we identified 8 architectural design decisions, 59 decision options, and 36 decision drivers to guide practitioners in the design and implementation of data ingestion pipelines. The framework was evaluated through expert interviews, assessing its completeness, simplicity, understandability, operational feasibility, and usefulness.

## Fivetran Makes Big Strides in India | Blog | Fivetran

DevFeed: [Fivetran Makes Big Strides in India | Blog | Fivetran](<https://devfeed.tech/articles/fivetran-makes-big-strides-in-india-blog-fivetran-141314.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/fivetran-makes-big-strides-in-india>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [2021](<https://devfeed.tech/tags/2021.md>), [2022](<https://devfeed.tech/tags/2022.md>), [about](<https://devfeed.tech/tags/about.md>), [addition](<https://devfeed.tech/tags/addition.md>), [after](<https://devfeed.tech/tags/after.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [annual-recurring-revenue](<https://devfeed.tech/tags/annual-recurring-revenue.md>), [aws](<https://devfeed.tech/tags/aws.md>), [blog](<https://devfeed.tech/tags/blog.md>), [country](<https://devfeed.tech/tags/country.md>), [data](<https://devfeed.tech/tags/data.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [india](<https://devfeed.tech/tags/india.md>), [staff](<https://devfeed.tech/tags/staff.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Within 11 months of launching its product in India in January 2021, Fivetran reported $1 million in local ARR, ninefold revenue growth and more than twice as many customers. It cited over 180 connectors for centralizing data in warehouses such as Snowflake. In one example, Cars24 connected MySQL data to Snowflake and had a test database running in two days; the company said an internal pipeline would have required five or six engineers for up to four months. Fivetran also reported reaching 100 employees in India and opening an AWS data center alongside its in-country GCP data center. Following its HVR integration, it said it could serve large enterprises needing high-volume, real-time, multidirectional synchronization from sources such as Oracle and SAP.

### Source excerpt

With 100+ staff and dedicated in-country GCP and AWS data centers, Fivetran is growing rapidly in the country.

## Three Must-Haves for Your Modern Retail Data Stack | Blog | Fivetran

DevFeed: [Three Must-Haves for Your Modern Retail Data Stack | Blog | Fivetran](<https://devfeed.tech/articles/three-must-haves-for-your-modern-retail-data-stack-blog-fivetran-141344.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/three-must-haves-for-modern-retail-data-stack>)

Author: Raman Kaler

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [analytics stack](<https://devfeed.tech/topics/analytics-stack.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [act](<https://devfeed.tech/tags/act.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [application-development](<https://devfeed.tech/tags/application-development.md>), [blog](<https://devfeed.tech/tags/blog.md>), [culture](<https://devfeed.tech/tags/culture.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [invest](<https://devfeed.tech/tags/invest.md>), [modern](<https://devfeed.tech/tags/modern.md>), [quickly](<https://devfeed.tech/tags/quickly.md>), [retail](<https://devfeed.tech/tags/retail.md>), [retailers](<https://devfeed.tech/tags/retailers.md>), [stack](<https://devfeed.tech/tags/stack.md>)

### AI overview

A modern retail data stack combines automated connectors, a cloud warehouse and visual analytics to make current information available for faster retail decisions. The article describes Fivetran connectors as syncing data from applications, databases, events and files, adapting to source changes and requiring no maintenance; Snowflake as providing governed access to centralized data; and Tableau as helping users explore performance. It cites a dashboard using more than 500 million transactions and 3 billion inventory records. Effective use also depends on data access, transparency and literacy, supported by coordination and a plan. Retail data needs vary by sector, including personalization, inventory and supply-chain optimization, consumer trend analysis, and ecommerce demand forecasting.

### Source excerpt

When retailers invest in a culture of data, they gain the ability to quickly navigate shifting customer priorities.

## PersonaHate: A Scalable Persona-Based Data Synthesis Pipeline for Hate Speech Analysis

DevFeed: [PersonaHate: A Scalable Persona-Based Data Synthesis Pipeline for Hate Speech Analysis](<https://devfeed.tech/articles/personahate-a-scalable-persona-based-data-synthesis-pipeline-for-hate-speech-analysis-141476.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21951>)

Author: Xinyu Zhang, Ziqing Yang, Michael Backes, Yang Zhang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [managing large-scale text data in AI](<https://devfeed.tech/topics/managing-large-scale-text-data-in-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [annotation](<https://devfeed.tech/tags/annotation.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [extension](<https://devfeed.tech/tags/extension.md>), [generators](<https://devfeed.tech/tags/generators.md>), [identity](<https://devfeed.tech/tags/identity.md>), [language](<https://devfeed.tech/tags/language.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llama-3-1](<https://devfeed.tech/tags/llama-3-1.md>), [llms](<https://devfeed.tech/tags/llms.md>), [meta](<https://devfeed.tech/tags/meta.md>), [openai](<https://devfeed.tech/tags/openai.md>), [performance](<https://devfeed.tech/tags/performance.md>), [persona](<https://devfeed.tech/tags/persona.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [speech](<https://devfeed.tech/tags/speech.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>)

### AI overview

PersonaHate uses 457K personas from two real-world sources to prompt eight LLMs to generate hate speech across 34 identity groups, then applies multi-judge annotation. The pipeline produces 791K samples and a 67K group-balanced training set, and is designed to extend to new groups, generators, and emerging targets. Training detectors from scratch or fine-tuning existing models yielded competitive results on four external benchmarks. A 184M-parameter DeBERTa-v3 model achieved 77.5% average F1 and 79.4% average accuracy, exceeding OpenAI's Moderation API on HateBenchSet and MHS. Fine-tuning Llama-3.1-8B raised average F1 from 70.7% to 78.4%; it surpassed LlamaGuard-3 on three of four benchmarks.

### Source excerpt

arXiv:2611.21951v1 Announce Type: new Abstract: Hate speech remains a persistent and evolving threat on online platforms. The widespread use of large language models (LLMs) further intensifies this problem by making hateful content easier and cheaper to generate at scale. Although stronger hate speech detectors continue to be proposed, their performance remains uneven across target identity groups and different forms of hateful expression, partly because existing training data provides only limited coverage of these cases. Thus, we introduce PersonaHate, a scalable persona-driven data synthesis pipeline for constructing diverse hate speech datasets. PersonaHate begins with a large persona pool of 457K personas drawn from two real-world sources. Conditioned on selected personas from this pool, it prompts eight LLMs to generate hate speech across 34 identity groups and performs multi-judge annotation to obtain reliable labels. The resulting corpus contains 791K samples, from which we curate a 67K group-balanced training set. Our modular design also supports extension to new identity groups, generators, and emerging hate targets. Built upon this dataset, we evaluate two detector training strategies: training from scratch and further fine-tuning. Results show that all trained models achieve competitive performance across four widely used external benchmarks. Notably, a 184M-parameter DeBERTa-v3 model trained on PersonaHate reaches 77.5% average F1 and 79.4% average accuracy, outperforming OpenAI's Moderation API on HateBenchSet and MHS. Furthermore, fine-tuning Llama-3.1-8B raises its average F1 from 70.7% to 78.4%, a gain of 7.7 percentage points, surpassing LlamaGuard-3, Meta's official safety model based on the same Llama-3.1-8B architecture, on three of four benchmarks. These results show that PersonaHate can substantially improve the robustness and cross-dataset generalization of hate speech detectors.

## The importance of unstructured data for AI

DevFeed: [The importance of unstructured data for AI](<https://devfeed.tech/articles/the-importance-of-unstructured-data-for-ai-140261.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/the-importance-of-unstructured-data-for-ai>)

Author: Charles Wang

Published: 2026-11-18T04:00:45Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-platforms](<https://devfeed.tech/topics/data-platforms.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data-architecture](<https://devfeed.tech/topics/data-architecture.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [automated](<https://devfeed.tech/tags/automated.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [unstructured-data](<https://devfeed.tech/tags/unstructured-data.md>)

### AI overview

Generative AI can extract meaning from unstructured text, images, code, audio and video, while many enterprise questions also require structured data for reporting and quantitative analysis. The article says unstructured data makes up an estimated 80-90% of organizational data, but is harder to standardize, govern and check for compliance than structured data. It presents automated integration as a way to centralize both types in a warehouse or data lake: Fivetran's file connectors ingest PDFs, images and documents alongside structured sources, and its Connector SDK supports building connectors for unsupported sources. The article also notes that LLMs can struggle with purely computational questions, so they may need integration with systems specialized in mathematical reasoning. These are the article's claims about Fivetran and AI use; it does not provide an independent product comparison.

### Source excerpt

Much of your organization's most important data is unstructured. Automated data integration allows you to access it, and AI to analyze it.

## Benchmarked: A data pipeline throughput analysis

DevFeed: [Benchmarked: A data pipeline throughput analysis](<https://devfeed.tech/articles/benchmarked-a-data-pipeline-throughput-analysis-140199.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/benchmarked-a-data-pipeline-throughput-analysis>)

Author: Eric O'Connor

Published: 2026-11-18T04:00:44Z

Content type: news

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [elt](<https://devfeed.tech/topics/elt.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [details](<https://devfeed.tech/tags/details.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [testing](<https://devfeed.tech/tags/testing.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Fivetran's excerpt describes a HammerDB TPROC-C benchmark for historical database loads, using an approximately 1 TB dataset with rows widened to about 1 KB to approximate customer data. The test measures throughput as dataset size divided by elapsed sync time; the stated full sync to Snowflake took 1 hour 30 minutes to 1 hour 40 minutes, but the excerpt omits the numeric throughput results and detailed configuration. Fivetran says its sync process uses checkpoints to resume after interruptions and splits large tables into chunks to limit transaction duration. It attributes performance improvements to greater parallelism and optimized warehouse loading patterns.

### Source excerpt

Looking under the hood and load testing Fivetran's data pipelines.

## NetBreaker: Benchmarking Network Dismantling across Objectives, Scales, and Toplogoies

DevFeed: [NetBreaker: Benchmarking Network Dismantling across Objectives, Scales, and Toplogoies](<https://devfeed.tech/articles/netbreaker-benchmarking-network-dismantling-across-objectives-scales-and-toplogoies-138851.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19298>)

Author: Wenchuan Yang, Tianci Bu, Suoyi Tan, Changjun Fan, Zijie Zhou, Xin Lu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [computational complexity](<https://devfeed.tech/topics/computational-complexity.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [learning](<https://devfeed.tech/tags/learning.md>), [network](<https://devfeed.tech/tags/network.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [scalability](<https://devfeed.tech/tags/scalability.md>)

### AI overview

NetBreaker benchmarks 30 network-dismantling methods--18 non-learning and 12 learning-based--on 185 networks across eight domains, ranging from 14 to 1.69 million nodes. It converts varied method outputs into node-removal sequences and evaluates them under a shared protocol across target-based and trajectory-based effectiveness, structural diagnostics, and applicability and computational efficiency. The reported experiments find that learning-based methods do not consistently outperform strong non-learning approaches, with effectiveness, scalability, and efficiency varying across network conditions and evaluation perspectives.

### Source excerpt

arXiv:2611.19298v1 Announce Type: new Abstract: Network dismantling seeks to fragment a network with as few node removals as possible, which is widely applied in vulnerability analysis, epidemic containment, and infrastructure protection. Although learning-based methods have become increasingly prominent, their progress remains difficult to assess because existing studies differ in method coverage, datasets, evaluation criteria, and experimental protocols. Thus, we introduce NetBreaker, a comprehensive benchmark designed for the learning era of network dismantling. NetBreaker organizes 30 methods (18 non-learning and 12 learning-based) under a common taxonomy and evaluates them on 185 networks from eight unified domains, ranging from 14 to 1.69 million nodes. To enable fair comparison, heterogeneous method outputs are converted into node removal sequences and assessed under a unified execution protocol. The evaluation covers four complementary dimensions: target-based effectiveness, trajectory-based effectiveness, structural diagnostics, and applicability & computational efficiency. Extensive experiments show that current learningbased methods do not consistently outperform strong non-learning approaches, while exposing substantial differences in effectiveness, scalability, and efficiency across network conditions and evaluation perspectives. NetBreaker provides a reproducible basis for measuring progress and developing reliable, scalable, and computationally efficient network dismantling methods.

## Dell Exascale Adds 32-Drive Liquid-Cooled Storage for AI Racks, and a 512-Core Air-Cooled PowerEdge M8825

DevFeed: [Dell Exascale Adds 32-Drive Liquid-Cooled Storage for AI Racks, and a 512-Core Air-Cooled PowerEdge M8825](<https://devfeed.tech/articles/dell-exascale-adds-32-drive-liquid-cooled-storage-for-ai-racks-and-a-512-core-air-cooled-poweredge-m8825-136078.md>)

Original publisher: [Read original article](<https://www.storagereview.com/news/dell-exascale-liquid-cooled-storage-poweredge-m8825-openmanage-rack-scale-sc26>)

Author: Harold Fritts

Published: 2026-11-16T15:40:37Z

Content type: news

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [dell](<https://devfeed.tech/topics/dell.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [NVMe](<https://devfeed.tech/topics/nvme.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [virtual machines](<https://devfeed.tech/topics/virtual-machines.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-factory](<https://devfeed.tech/tags/ai-factory.md>), [air](<https://devfeed.tech/tags/air.md>), [core](<https://devfeed.tech/tags/core.md>), [data](<https://devfeed.tech/tags/data.md>), [dell](<https://devfeed.tech/tags/dell.md>), [dell-ai-factory](<https://devfeed.tech/tags/dell-ai-factory.md>), [dell-technologies](<https://devfeed.tech/tags/dell-technologies.md>), [drive](<https://devfeed.tech/tags/drive.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-storage](<https://devfeed.tech/tags/enterprise-storage.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [liquid](<https://devfeed.tech/tags/liquid.md>), [news](<https://devfeed.tech/tags/news.md>), [nvme](<https://devfeed.tech/tags/nvme.md>), [platform](<https://devfeed.tech/tags/platform.md>), [poweredge](<https://devfeed.tech/tags/poweredge.md>), [software](<https://devfeed.tech/tags/software.md>), [software-defined-storage](<https://devfeed.tech/tags/software-defined-storage.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Dell's SC26 updates add two Dell Exascale storage options for AI racks: a fanless 2-OU direct-liquid-cooled system with up to 32 NVMe drives, and a 1U air-cooled system with up to 20. Dell expects both in early 2027; block support for the liquid-cooled platform is planned for the first half of 2027. The 1U system's claimed 67% drive-density gain compares with prior 2U Exascale hardware. Other announcements include an Agentic Data Gateway that centrally governs agent access to data, tools, and models inside the customer environment; OpenManage Rack Scale, which manages up to 100,000 devices; and Fabric Manager for network-fabric automation. Dell's performance and efficiency figures are vendor or lab results. The air-cooled, two-socket PowerEdge M8825 offers up to 512 cores, has preliminary specifications, and is targeted for the first half of 2027. The excerpt also describes AMD GPU configurations, validated AI solutions, and services, with availability varying by offering.

### Source excerpt

Dell Technologies has announced a round of Dell AI Factory updates at SC26, led by two new hardware options for Dell Exascale, its software-defined storage for AI racks: a 2-OU direct-liquid-cooled platform with up to 32 NVMe drives and a 1U air-cooled platform with up to 20. The news also covers an Agentic Data Gateway The post Dell Exascale Adds 32-Drive Liquid-Cooled Storage for AI Racks, and a 512-Core Air-Cooled PowerEdge M8825 appeared first on StorageReview.com.

## HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets

DevFeed: [HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets](<https://devfeed.tech/articles/histogramtools-for-efficient-data-analysis-and-distribution-representation-in-large-data-sets-132665.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2504.00001>)

Author: Shubham Malhotra

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [applications](<https://devfeed.tech/tags/applications.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [error](<https://devfeed.tech/tags/error.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [mapreduce](<https://devfeed.tech/tags/mapreduce.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [r](<https://devfeed.tech/tags/r.md>), [storage](<https://devfeed.tech/tags/storage.md>), [visualization](<https://devfeed.tech/tags/visualization.md>)

### AI overview

HistogramTools extends R's built-in histogram features for large-scale and distributed data analysis. It supports serializing histograms with Protocol Buffers, merging and modifying them, and manipulating bins. The abstract also describes distance measures, quantile approximation, and methods to estimate and visualize information loss, including error in CDFs derived from histograms. The package is presented as suited to MapReduce settings where compact storage and data sharing matter; the supplied text does not provide performance results or implementation details.

### Source excerpt

arXiv:2504.00001v2 Announce Type: replace Abstract: Histograms provide a powerful means of summarizing large data sets by representing their distribution in a compact, binned form. The HistogramTools R package enhances R built-in histogram functionality, offering advanced methods for manipulating and analyzing histograms, especially in large-scale data environments. Key features include the ability to serialize histograms using Protocol Buffers for distributed computing tasks, tools for merging and modifying histograms, and techniques for measuring and visualizing information loss in histogram representations. The package is particularly suited for environments utilizing MapReduce, where efficient storage and data sharing are critical. This paper presents various methods of histogram bin manipulation, distance measures, quantile approximation, and error estimation in cumulative distribution functions (CDFs) derived from histograms. Visualization techniques and efficient storage representations are also discussed alongside applications for large data processing and distributed computing tasks.

## TSseek: Pattern-Based Search for Distributed Time Series Datasets

DevFeed: [TSseek: Pattern-Based Search for Distributed Time Series Datasets](<https://devfeed.tech/articles/tsseek-pattern-based-search-for-distributed-time-series-datasets-132672.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.09824>)

Author: Xiaoshuai Li, Khalid Alnuaim, Mohamed Y. Eltabakh, Elke A. Rundensteiner

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Time Series](<https://devfeed.tech/topics/time-series.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>)

Tags: [cs-db](<https://devfeed.tech/tags/cs-db.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engine](<https://devfeed.tech/tags/engine.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [index](<https://devfeed.tech/tags/index.md>), [language](<https://devfeed.tech/tags/language.md>), [matching](<https://devfeed.tech/tags/matching.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [order](<https://devfeed.tech/tags/order.md>), [pattern](<https://devfeed.tech/tags/pattern.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [plan](<https://devfeed.tech/tags/plan.md>), [processing](<https://devfeed.tech/tags/processing.md>), [queries](<https://devfeed.tech/tags/queries.md>), [query](<https://devfeed.tech/tags/query.md>), [range](<https://devfeed.tech/tags/range.md>), [reading](<https://devfeed.tech/tags/reading.md>), [recall](<https://devfeed.tech/tags/recall.md>), [search](<https://devfeed.tech/tags/search.md>), [time-series](<https://devfeed.tech/tags/time-series.md>)

### AI overview

TSseek searches distributed time series for patterns involving trends, value ranges, wildcards, and constraints on changes, duration, position, order, and spacing. It builds TSseek-X from error-bounded line-segment approximations and their slope and range metadata. Query planning uses selectivity estimates; spatial probes find candidates while retaining every true match, then exact refinement checks candidates against original readings. Across four datasets of up to 200 million series (25.6 billion readings), TSseek reports a 20x speedup over a distributed exact full scan using the same matcher. In whole-matching comparisons, adapted baselines reached at most 57% recall, while TSseek returned the exact answer set.

### Source excerpt

arXiv:2606.09824v5 Announce Type: replace Abstract: In applications such as household energy monitoring and ECG analysis, analysts may search for time series that satisfy constraints on trends, value ranges, and durations. To support such queries at scale, we propose TSseek, a distributed processing engine for time series pattern queries. Its regex-inspired language enables users to compose trends, value ranges, and wildcards with constraints on changes between readings and the duration, position, order, and spacing of patterns. During indexing, TSseek approximates each stored time series with error-bounded line segments, records their slope and value-range metadata, and builds TSseek-X, a distributed spatial index over the segments. At query time, TSseek rewrites the query and uses selectivity estimates to plan the index lookup. During execution, index lookup uses spatial probes derived from query constraints over reading positions and values to select candidate series from TSseek-X while guaranteeing to retain every true match. Finally, exact refinement checks each candidate against the full query using its original readings. Experiments on four datasets of up to 200 million time series (25.6 billion readings) show 20x speedup over a distributed exact full scan using the same matcher. In the whole-matching comparison, the adapted baselines each reach at most 57% recall, while TSseek returns the exact answer set.

## Buckets guide to what's in store with AWS Storage at re:Invent 2026

DevFeed: [Buckets guide to what's in store with AWS Storage at re:Invent 2026](<https://devfeed.tech/articles/buckets-guide-to-what-s-in-store-with-aws-storage-at-re-invent-2026-131282.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/buckets-guide-to-whats-in-store-with-aws-storage-at-reinvent-2026/>)

Author: Sean White

Published: 2026-11-13T00:36:07Z

Content type: article

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Amazon Elastic Block Store (Amazon EBS)](<https://devfeed.tech/topics/amazon-elastic-block-store-amazon-ebs.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Network Attached Storage (NAS)](<https://devfeed.tech/topics/network-attached-storage-nas.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Persistence](<https://devfeed.tech/topics/persistence.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon GuardDuty](<https://devfeed.tech/topics/amazon-guardduty.md>), [sharding](<https://devfeed.tech/topics/sharding.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-elastic-block-store-amazon-ebs](<https://devfeed.tech/tags/amazon-elastic-block-store-amazon-ebs.md>), [amazon-elastic-file-system-efs](<https://devfeed.tech/tags/amazon-elastic-file-system-efs.md>), [amazon-fsx](<https://devfeed.tech/tags/amazon-fsx.md>), [amazon-fsx-for-lustre](<https://devfeed.tech/tags/amazon-fsx-for-lustre.md>), [amazon-fsx-for-netapp-ontap](<https://devfeed.tech/tags/amazon-fsx-for-netapp-ontap.md>), [amazon-fsx-for-openzfs](<https://devfeed.tech/tags/amazon-fsx-for-openzfs.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-backup](<https://devfeed.tech/tags/aws-backup.md>), [aws-cloud-storage](<https://devfeed.tech/tags/aws-cloud-storage.md>), [aws-re-invent](<https://devfeed.tech/tags/aws-re-invent.md>), [aws-storage](<https://devfeed.tech/tags/aws-storage.md>), [re-invent](<https://devfeed.tech/tags/re-invent.md>), [storage](<https://devfeed.tech/tags/storage.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

The AWS Storage track at re:Invent 2026 offers 79 sessions across 107 time slots, with roughly three quarters of slots interactive and 73 sessions at level 300 or higher. Topics span AI agent storage, performance, data lakes, migration, security, and recovery. The guide contrasts services for particular needs: S3 Express One Zone for latency-sensitive object access, FSx for Lustre for high throughput, EBS for low-latency block storage, and EFS/FSx for file workloads. Sessions include hands-on workshops, chalk talks, and builders' sessions; these are not recorded, while breakouts are posted to AWS Events YouTube after the event. Reserved seats are open to registered attendees, though walk-up seating may be available; many sessions run twice. The event is in Las Vegas November 30-December 4, 2026, and attendees should allow at least 30 minutes between venues.

### Source excerpt

Wondering what is in store at re:Invent this year? A lot. I would know. Storing things is my whole job. Every great story starts with a single object. This one starts with me. Hi, I am Buckets. The official AWS Storage mascot, self-appointed guardian of all things durable, scalable, and correctly permissioned. At re:Invent, I [...]

## NFL and AWS Big Data Bowl will bridge the gap from college to the pros

DevFeed: [NFL and AWS Big Data Bowl will bridge the gap from college to the pros](<https://devfeed.tech/articles/nfl-and-aws-big-data-bowl-will-bridge-the-gap-from-college-to-the-pros-127783.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/media/nfl-and-aws-big-data-bowl-will-bridge-the-gap-from-college-to-the-pros/>)

Author: Ari Entin

Published: 2026-11-11T00:04:01Z

Content type: release

Language: en

Sources: [AWS for M&E Blog](<https://devfeed.tech/sources/aws-for-m-e-blog.md>)

Topics: [Sports](<https://devfeed.tech/topics/sports.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [college](<https://devfeed.tech/tags/college.md>), [competition](<https://devfeed.tech/tags/competition.md>), [data](<https://devfeed.tech/tags/data.md>), [industries](<https://devfeed.tech/tags/industries.md>), [media-entertainment](<https://devfeed.tech/tags/media-entertainment.md>), [sports](<https://devfeed.tech/tags/sports.md>), [tracking](<https://devfeed.tech/tags/tracking.md>)

### AI overview

The 2027 Big Data Bowl will test whether college production and NFL Combine tracking can help explain rookie NFL performance. Its dataset covers players from the 2023-2025 draft classes at five position groups, with x-y movement tracking from standardized Combine drills and NFL games plus drill and play context. The competition encourages position-specific analysis, such as comparing route breaks, agility, or first-step quickness with early in-game performance, while recognizing that some drills may show no predictive signal. Successful methods could inform the NFL's Draft IQ tools, with AWS helping scale competition code for production; prior Big Data Bowl work has also produced metrics used in broadcasts and team research. The article presents this as exploratory: predictive value has not been systematically established, and null findings are considered useful.

### Source excerpt

The ninth annual Big Data Bowl, the NFL's flagship analytics competition, is taking a new direction. For the first time, participants will work with data that spans the full arc of a player's journey into the NFL, combining college statistics, player tracking data from the NFL Scouting Combine, and Next Generation (Next Gen) Stats from their rookie seasons. The challenge is to find the connections that link what a player does before the league to what they do after they arrive. Join the 2027 Big Data Bowl and start exploring the data.

## Too Big to Inspect? Ethically Auditing Large Datasets Through Small Samples

DevFeed: [Too Big to Inspect? Ethically Auditing Large Datasets Through Small Samples](<https://devfeed.tech/articles/too-big-to-inspect-ethically-auditing-large-datasets-through-small-samples-125665.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10399>)

Author: Giovana Marchetti, Jhessica Silva, Tiago Torrent, Carlos Caetano, Sandra Avila

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [ai regulation](<https://devfeed.tech/topics/ai-regulation.md>)

Tags: [accountability](<https://devfeed.tech/tags/accountability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-ethics](<https://devfeed.tech/tags/ai-ethics.md>), [ai-regulation](<https://devfeed.tech/tags/ai-regulation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [automated](<https://devfeed.tech/tags/automated.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [datasheets](<https://devfeed.tech/tags/datasheets.md>), [ethics](<https://devfeed.tech/tags/ethics.md>), [human-resources](<https://devfeed.tech/tags/human-resources.md>), [industry](<https://devfeed.tech/tags/industry.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [models](<https://devfeed.tech/tags/models.md>), [problems](<https://devfeed.tech/tags/problems.md>), [process](<https://devfeed.tech/tags/process.md>), [small](<https://devfeed.tech/tags/small.md>)

### AI overview

The paper finds that random samples can support qualitative ethical audits of large datasets without examining every item. It applied four AI Ethics Tools--Datasheets for Datasets, Data Statements, Data Nutrition Labels, and Data Cards--to a 1% sample of Framed Multi30K, and Datasheets for Datasets to two 0.00006% samples of relaion2B-en-research-safe. The authors report that these audits surfaced ethical problems, structural biases, and technical inconsistencies with manageable human and computational resources. The tools' questions prompted different perspectives, but analysis and findings depended fundamentally on human judgment. The abstract describes these case studies as evidence that small-sample audits are feasible; it does not establish that sampling captures every issue in a dataset.

### Source excerpt

arXiv:2611.10399v1 Announce Type: cross Abstract: Big techs and researchers increasingly claim that their training datasets are too large to inspect for bias and ethical problems. However, in the race to build ever-large models, they have relied on automated web scraping and no human curation. Datasets built this way absorb the stereotypes, exclusionary norms, and historical prejudices of their sources, and the models trained on them can reproduce and amplify those patterns. Because manually auditing billions of samples is humanly impossible, we need practical, effective methods to identify biases in these datasets. This paper studies the feasibility of ethically auditing large datasets through small samples and proposes a practical method for identifying structural biases by applying AI Ethics Tools (AIETs) to random samples. We applied Datasheets for Datasets, Data Statements, Data Nutrition Labels, and Data Cards to a 1% sample of Framed Multi30K, and Datasheets for Datasets to two 0.00006% samples of relaion2B-en-research-safe. Our results show that evaluating small samples with AIETs can reveal ethical problems, structural biases, and technical inconsistencies. Although industry practice assumes that identifying problems requires analyzing the entire dataset, we show that a random subset supports a viable qualitative analysis without overwhelming computational or human resources. Demonstrating that a dataset is biased does not require examining every instance: if a bias appears in a random sample, it is a flaw of the creation process. The AIETs' questions acted as provocateurs, evoking different perspectives and analysis methods for the same dataset, but the analysis and its results depended fundamentally on the human factor. We hope these findings contribute to urgent public discussions about ethics, transparency, accountability, and AI regulation by offering a practical pathway for responsibly auditing massive datasets.

## GARDEN: Efficient GPU-Accelerated Billion-Scale GNN Training via Caching-Sampling-Batching Co-Design

DevFeed: [GARDEN: Efficient GPU-Accelerated Billion-Scale GNN Training via Caching-Sampling-Batching Co-Design](<https://devfeed.tech/articles/garden-efficient-gpu-accelerated-billion-scale-gnn-training-via-caching-sampling-batching-co-design-126718.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11043>)

Author: Qingrui Zhu, Lei Liu, Yushu Cai, Kai Sheng, Zhiwen Chen, Xin He

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [caching](<https://devfeed.tech/tags/caching.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [gnn](<https://devfeed.tech/tags/gnn.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-accelerated](<https://devfeed.tech/tags/gpu-accelerated.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scale](<https://devfeed.tech/tags/scale.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

GARDEN targets GPU memory and data-transfer bottlenecks in training GNNs on billion-edge graphs by jointly designing caching, sampling, and batching. Its cache allocator combines offline reuse profiling with online hotness tracking to manage feature and adjacency caching within a fixed GPU memory budget. A grouping-based sampler reorders neighborhood accesses to improve warp occupancy and hide latency on power-law graphs, while a degree classifier combines mini-batches into degree-homogeneous groups for coalesced transfers and higher GPU utilization. The abstract reports end-to-end speedups of 4.54x over DGL-UVA, 1.27x over DUCATI, and 2.97x over FastGL on real-world billion-scale graphs, with model convergence and accuracy maintained.

### Source excerpt

arXiv:2611.11043v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) have shown impressive performance in diverse tasks such as social network analysis and recommendation systems. However, scaling their training to billion-edge graphs is hindered by the memory wall: GPU memory capacity falls short of the massive storage requirements for node features and topology. Sampling-based GNN training frameworks mitigate this by streaming subgraphs from CPU to GPU, but they incur prohibitive PCIe data transfers and non-coalesced memory accesses, resulting in bandwidth underutilization and low compute efficiency. We propose GARDEN, a GPU-accelerated sampling-based system that co-designs caching, sampling, and batching to eliminate these bottlenecks. GARDEN introduces three key innovations: (1) an Adaptive High-Fidelity Cache Allocator (AHFCA) that combines offline reuse profiling and online hotness tracking to dynamically cache features and adjacency lists within a fixed GPU memory budget, avoiding static allocation waste; (2) a Grouping-based Parallel Sampler that reorders stochastic neighborhood accesses(GPS) to maximize warp occupancy and hide latency on power-law graphs; and (3) a Super-batch Degree Classifier(SDC) that fuses mini-batches into degree-homogeneous groups, enabling cache-friendly coalesced transfers and high GPU utilization. Evaluations on real-world billion-scale graphs demonstrate that GARDEN achieves 4.54x, 1.27x, and 2.97x end-to-end training speedups over DGL-UVA, DUCATI, and FastGL, respectively, while maintaining model convergence and accuracy.

## LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

DevFeed: [LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization](<https://devfeed.tech/articles/llm-watermarking-as-big-data-provenance-a-deployment-oriented-systematization-126893.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.10103>)

Author: Huy Phan, Kieu Dang, Ojaswi Dulal, Aiham AL Shukairi, Abby Shine, Chase Garner, Phung Lai

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [attribution](<https://devfeed.tech/tags/attribution.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [data-provenance](<https://devfeed.tech/tags/data-provenance.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [detection](<https://devfeed.tech/tags/detection.md>), [llm](<https://devfeed.tech/tags/llm.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [watermarking](<https://devfeed.tech/tags/watermarking.md>)

### AI overview

The paper systematizes LLM watermarking as a provenance tool for large-scale data ecosystems, organizing methods by insertion point, verification authority, operational state, and threat model. It maps them to online generation, streaming detection, transformation pipelines, and ecosystem governance, using the five Vs of big data as a deployment lens. The abstract identifies recurring constraints: some methods require generation-time access, add inference overhead, or are sensitive to downstream transformations; calibration and lifecycle compatibility are also incompletely reported. It proposes workload-specific evaluation of statistical reliability, throughput, utility, robustness, and governance, addressing needs that sample-level detectability and robustness evaluations may leave underexamined, including population-scale false positives, multi-key attribution, and long-term verification.

### Source excerpt

arXiv:2607.10103v3 Announce Type: replace Abstract: As large language models (LLMs) generate and transform content at scale, reliable attribution becomes an increasingly important requirement for big data provenance. Although numerous watermarking methods have been proposed, existing evaluations predominantly emphasize sample-level detectability and robustness, leaving deployment requirements such as serving efficiency, population-scale false positives, multi-key attribution, and long-term verification less systematically examined. This paper presents a deployment-oriented systematization of LLM watermarking through the five Vs of big data: Volume, Velocity, Variety, Veracity, and Value. We classify representative watermarking methods across four dimensions: insertion point, verification authority, operational state, and adversarial threat model. We further map these methods to four deployment workloads: online generation, streaming detection, transformation pipelines, and ecosystem governance. Our synthesis identifies recurring limitations, including dependencies on generation-time access, additional inference overhead, sensitivity to downstream transformations, and incomplete reporting of calibration and lifecycle compatibility. Based on these findings, we propose workload-specific evaluation requirements covering statistical reliability, throughput, utility, robustness, and governance. This framework connects watermark design choices to practical deployment constraints and provides guidance for selecting and evaluating provenance mechanisms in large-scale data ecosystems.

## CoVault: Secure, Scalable Analytics of Personal Data

DevFeed: [CoVault: Secure, Scalable Analytics of Personal Data](<https://devfeed.tech/articles/covault-secure-scalable-analytics-of-personal-data-126675.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2208.03784>)

Author: Roberta De Viti, Isaac Sheff, Noemi Glaeser, Baltasar Dinis, Rodrigo Rodrigues, Bobby Bhattacharjee, Anwar Hithnawi, Deepak Garg, Peter Druschel

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>), [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [000](<https://devfeed.tech/tags/000.md>), [2pc](<https://devfeed.tech/tags/2pc.md>), [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [awareness](<https://devfeed.tech/tags/awareness.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [can](<https://devfeed.tech/tags/can.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [core](<https://devfeed.tech/tags/core.md>), [country](<https://devfeed.tech/tags/country.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data](<https://devfeed.tech/tags/data.md>), [datacenter](<https://devfeed.tech/tags/datacenter.md>), [environments](<https://devfeed.tech/tags/environments.md>), [example](<https://devfeed.tech/tags/example.md>), [personal](<https://devfeed.tech/tags/personal.md>), [personal-data](<https://devfeed.tech/tags/personal-data.md>)

### AI overview

CoVault combines server-aided multi-party computation (MPC) with trusted execution environments (TEEs) to colocate MPC parties in one datacenter while preserving the stated security level, then scales MPC horizontally across available datacenter resources. The authors identify secure horizontal scaling under a strong threat model as a gap in analytics for personal data. They report scaling the DualEx two-party computation protocol to continuous epidemic analytics for a country of 80 million people--about 11.85 billion records per day--using one core pair per 30,000 people. The supplied text is an abstract and does not detail the threat model or evaluation methods.

### Source excerpt

arXiv:2208.03784v4 Announce Type: replace-cross Abstract: There is growing awareness that the analysis of personal data, such as individuals' mobility, financial, and health data, can provide significant benefits to society. However, liberal societies have so far refrained from such analytics, arguably due to the lack of secure analytics platforms that scale to billions of records while operating in a very strong threat model. We contend that one fundamental gap here is the lack of an architecture that can scale (actively-)secure multi-party computation (MPC) horizontally without weakening security. To bridge this gap, we present CoVault, an analytics platform that leverages server-aided MPC and trusted execution environments (TEEs) to colocate the MPC parties in a single datacenter without reducing security, and scales MPC horizontally to the datacenter's available resources. CoVault scales well empirically. For example, CoVault can scale the DualEx 2PC protocol to perform epidemic analytics for a country of 80M people (about 11.85B data records/day) on a continuous basis using one core pair for every 30,000 people.

## ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining

DevFeed: [ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining](<https://devfeed.tech/articles/zarrfm-a-canonical-tensor-storage-architecture-for-eeg-foundation-model-pretraining-125330.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11149>)

Author: Dheeraj Chintapalli, Sunkalp Chandra

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [sharding](<https://devfeed.tech/topics/sharding.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [canonical](<https://devfeed.tech/tags/canonical.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [eeg](<https://devfeed.tech/tags/eeg.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [storage](<https://devfeed.tech/tags/storage.md>), [storage-architecture](<https://devfeed.tech/tags/storage-architecture.md>)

### AI overview

ZarrFM stores heterogeneous EEG corpora in a single Zarr v3 hierarchy, complementing BIDS with indexed metadata, canonical channel spaces, explicit masks, stable subject splits, verified event tables, and block-aligned sharding. In tests on real PhysioNet recordings and synthetic corpora, its two-level index reduced metadata queries across 200 recordings from 601 to 14. Preserving native signal codes used 22.7 MB/hour versus 42.4 MB/hour for EDF, and block-aligned sampling reached 3,131 windows/s, up to 11 times Parquet's rate. Reads were bit-identical across independent Zarr implementations, and failure-injected concurrent appends recovered without data loss. These reported results suggest a compact, interoperable storage option for EEG pretraining; the excerpt does not provide broader evaluation details.

### Source excerpt

arXiv:2611.11149v1 Announce Type: new Abstract: Statistical models for brain-computer interfaces (BCIs) increasingly require large, heterogeneous EEG corpora, yet existing datasets vary in sampling rates, channel layouts, window lengths, and storage formats, creating substantial preprocessing and I/O overhead. We propose ZarrFM, an execution-oriented representation that complements BIDS by storing a corpus as a single Zarr v3 hierarchy with indexed metadata, canonical channel spaces, explicit masks, hash-stable subject splits, verified event tables, and block-aligned sharding. On real PhysioNet recordings and synthetic corpora, we compare ZarrFM with EDF, FIF, NPY, HDF5, and Parquet. Across 200 recordings, its two-level index reduces metadata queries from 601 requests to 14. Preserving native signal codes reduces storage from 42.4 MB/hour in EDF to 22.7 MB/hour, while block-aligned sampling reaches 3,131 windows/s, up to 11 times faster than Parquet. Reads remain bit-identical across independent Zarr implementations, and concurrent failure-injected appends recover without data loss. These results show that ZarrFM can provide a compact, interoperable, and execution-efficient storage layer for large-scale EEG pretraining.

## Hedging Erasure-Coded Reads in HDFS

DevFeed: [Hedging Erasure-Coded Reads in HDFS](<https://devfeed.tech/articles/hedging-erasure-coded-reads-in-hdfs-127005.md>)

Original publisher: [Read original article](<https://product.hubspot.com/blog/hedging-erasure-coded-reads-in-hdfs>)

Author: Ray Mattingly

Published: 2026-11-09T21:30:20Z

Content type: article

Language: en

Sources: [HubSpot](<https://devfeed.tech/sources/hubspot.md>)

Topics: [hdfs](<https://devfeed.tech/topics/hdfs.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [circuit-breaker](<https://devfeed.tech/tags/circuit-breaker.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [erasure-coding](<https://devfeed.tech/tags/erasure-coding.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [hubspot](<https://devfeed.tech/tags/hubspot.md>)

### AI overview

HubSpot added latency-triggered hedged reads for erasure-coded HDFS: when a cell read exceeds a cluster threshold, the HDFS client reconstructs it from other stripe cells and races that work against the original read. The approach preserves erasure coding's stated 50% storage savings versus triple replication while reducing dependence on a slow DataNode. On the case-study cluster, slow-read host-minutes fell 98%; hedges won 90% of the time and occurred about once per million reads. Each hedge costs six additional reads plus decoding, so thresholds tune the latency and resource tradeoff. A per-RegionServer circuit breaker disables hedging during broader slowdowns, with a separate cap on hedge rate. For HBase Gets, the timer also covers connection setup; during two reported DataNode failures, this avoided the over-five-second stalls seen when only data transfer was hedged.

### Source excerpt

Written by Ray Mattingly, Staff Engineering Manager, Big Data @ HubSpot

## Minimum-Sufficient Knowledge and Computation for Wind Energy Predictive Analysis

DevFeed: [Minimum-Sufficient Knowledge and Computation for Wind Energy Predictive Analysis](<https://devfeed.tech/articles/minimum-sufficient-knowledge-and-computation-for-wind-energy-predictive-analysis-123279.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08558>)

Author: Purbak Sengupta, Aparna S. varde, Emil J{\\o}rgensen Njor

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [Sustainable Computing](<https://devfeed.tech/topics/sustainable-computing.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [apply](<https://devfeed.tech/tags/apply.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [computing](<https://devfeed.tech/tags/computing.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cyber](<https://devfeed.tech/tags/cyber.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [edge](<https://devfeed.tech/tags/edge.md>), [energy](<https://devfeed.tech/tags/energy.md>), [event](<https://devfeed.tech/tags/event.md>), [external](<https://devfeed.tech/tags/external.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [gate](<https://devfeed.tech/tags/gate.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [sufficient](<https://devfeed.tech/tags/sufficient.md>)

### AI overview

Minimum-Sufficient Knowledge and Computation (MSKC) routes each decision through the least costly option that meets risk and deployment constraints, escalating when cheaper routes are insufficient. In a wind ramp-event prediction proof of concept using public data from two wind farms, the study applies a state-aware gate to local, relational, and physics-informed routes. At a 35.9% relational-knowledge injection rate, adaptive routing was statistically indistinguishable from always-on injection on the reported metrics while reducing knowledge calls by 64%. The evidence is limited to the reported metrics and this proof-of-concept setting.

### Source excerpt

arXiv:2611.08558v1 Announce Type: new Abstract: Resource-constrained cyber-physical decision systems often apply fixed sensing, knowledge, and computational pipelines despite state-dependent information needs. We introduce Minimum-Sufficient Knowledge and Computation (MSKC), a resource-aware computing principle: select the least-costly route meeting decision-risk and deployment constraints, escalating only when cheaper routes are insufficient. As proof of concept, we isolate adaptive knowledge routing, the decision underlying Knowledge-Guided Machine Learning (KGML), which conventionally injects external knowledge unconditionally rather than only where needed. We instantiate this for wind ramp-event predictive analysis, an edge-oriented decision-support task, evaluating local, relational, and physics-informed routes with a state-aware gate on publicly available real data from two wind farms. We establish conditions for feasibility preservation, cost dominance, and strict expected savings over always-on routing. At a 35.9% relational-knowledge injection rate, adaptive routing is statistically indistinguishable from always-on injection on the reported metrics while reducing knowledge calls by 64%, supporting sustainable, Green-IT-aligned deployment. These results support MSKC's premise that uniformly maximal intelligence is unnecessary. This offers intelligent data analysis for big data streams: the analysis itself decides, per state, how much knowledge and computation the data warrant.

## Logical Scalability in Big Data Architectures: Impact of Data Modeling on the Database Layer

DevFeed: [Logical Scalability in Big Data Architectures: Impact of Data Modeling on the Database Layer](<https://devfeed.tech/articles/logical-scalability-in-big-data-architectures-impact-of-data-modeling-on-the-database-layer-124231.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09346>)

Author: Rosario Napoli, Mark Adrian Gambito, Antonio Celesti, Maria Fazio

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [Database](<https://devfeed.tech/topics/database.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [database-scalability](<https://devfeed.tech/topics/database-scalability.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>)

Tags: [architectures](<https://devfeed.tech/tags/architectures.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [database](<https://devfeed.tech/tags/database.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-coordination](<https://devfeed.tech/tags/distributed-coordination.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [graph](<https://devfeed.tech/tags/graph.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [layer](<https://devfeed.tech/tags/layer.md>), [logical](<https://devfeed.tech/tags/logical.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machines](<https://devfeed.tech/tags/machines.md>), [misalignment](<https://devfeed.tech/tags/misalignment.md>), [model](<https://devfeed.tech/tags/model.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-analysis](<https://devfeed.tech/tags/performance-analysis.md>), [scalability](<https://devfeed.tech/tags/scalability.md>)

### AI overview

The abstract reports that choosing a database model suited to the data and query patterns is more important for performance than adding machines alone. It compares Document, Graph, and Multi-Model databases across workloads, queries, and dataset scales, and finds that model misalignment can keep performance bottlenecked despite more computational resources. Scaling out an inadequate model can perform no better than, or worse than, a single machine because distributed coordination adds overhead. The supplied text does not provide detailed measurements or per-paradigm results.

### Source excerpt

arXiv:2611.09346v1 Announce Type: new Abstract: In modern Big Data architectures, scalability is often addressed through infrastructural expansion, involving vertical or horizontal scaling to manage growing data volumes. However, this hardware-centric perspective overlooks the logical scalability of the storage layer. A central challenge lies in how data modeling choices affect the storage layer of big data architectures. Specifically, when the data model is misaligned with the underlying query patterns, performance bottlenecks emerge and persist regardless of the available computational resources. This paper investigates the concept of logical scalability by conducting a comparative performance analysis of Document, Graph and Multi-Model database paradigms under varying workloads, queries and dataset scales. Our results demonstrate that performance degradation caused by model misalignment cannot be mitigated by additional computational resources alone. Increasing the number of machines while retaining a structurally inadequate data model yields performance equivalent to or worse than a single-machine configuration, as distributed coordination overhead compounds the inefficiencies of the underlying logical model. We demonstrate that efficiency is maximized when the logical structure of the database aligns with the inherent nature of the data and the intended analysis scope, a condition that proves to be more decisive than any infrastructural scaling strategy.

## Towards Scalable Diagnostic-Driven Whole-Slide Image Retrieval for Pathology Archives

DevFeed: [Towards Scalable Diagnostic-Driven Whole-Slide Image Retrieval for Pathology Archives](<https://devfeed.tech/articles/towards-scalable-diagnostic-driven-whole-slide-image-retrieval-for-pathology-archives-120271.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07935>)

Author: Jingsong Liu, Youssef Wally Han Li, Weiwei Zhou, Jing Dai, Elisabeth Wetzer, Peter J. Sch\\"uffler

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Medical imaging](<https://devfeed.tech/topics/medical-imaging.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [archive](<https://devfeed.tech/tags/archive.md>), [bank](<https://devfeed.tech/tags/bank.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [framework](<https://devfeed.tech/tags/framework.md>), [image](<https://devfeed.tech/tags/image.md>), [latency](<https://devfeed.tech/tags/latency.md>), [precision](<https://devfeed.tech/tags/precision.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [scalability](<https://devfeed.tech/tags/scalability.md>)

### AI overview

CLEAR-WSI uses fixed-length whole-slide image embeddings to retrieve visually similar pathology cases without depending on consistent metadata or manual annotations. It refines candidates using either predicted or ground-truth diagnostic labels, depending on the retrieval setting, and adds newly processed cases to an expanding memory bank. Across four retrieval tasks on two public datasets and one private clinical dataset, it improved Precision@5 over the strongest evaluated baseline, with reported gains of 3.10 on CAMELYON16, 2.75 on BRACS-1, 1.95 on BRACS-2, and 3.75 on X-HER2. The abstract also reports ablation and latency studies, but provides no further details on their findings.

### Source excerpt

arXiv:2611.07935v1 Announce Type: cross Abstract: The growing adoption of digital pathology has rapidly expanded collections of hematoxylin-and-eosin whole-slide images, yet efficient navigation through these archives remains challenging, as metadata is often inconsistent or unavailable. Content-based image retrieval addresses this challenge by identifying similar cases directly from visual representations instead of manual annotations or metadata. Existing retrieval systems have demonstrated promising performance but largely rely on handcrafted indexing pipelines and predefined sampling strategies, constraining scalability and generalization. Here, we present CLEAR-WSI (Constant-Length Embedding for Automatic Retrieval of Whole-Slide Images) as a scalable, diagnostic-driven slide-retrieval framework, validated on both public and clinical data. Using a fixed-length slide embedding, CLEAR-WSI refines retrieved candidates with either a predicted diagnostic label or the ground-truth label, depending on the retrieval setting. Newly processed cases are subsequently added to the expanding memory bank, allowing the retrieval archive to grow over time. On a comprehensive benchmark consisting of two public and one private clinical datasets representing four retrieval tasks, CLEAR-WSI improves Precision@5 over the strongest evaluated baseline, with gains on CAMELYON16 (3.10), BRACS-1 (2.75), BRACS-2 (1.95), and X-HER2 (3.75). Additional ablation and latency studies further examine the contribution of each component and highlight its potential for scalable deployment.

## Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale

DevFeed: [Knowledge Graph Lakehouse: A Cloud-Native Architecture for Managing Enterprise Knowledge Graphs at Scale](<https://devfeed.tech/articles/knowledge-graph-lakehouse-a-cloud-native-architecture-for-managing-enterprise-knowledge-graphs-at-scale-115964.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04899>)

Author: Bashar Ahmad, Christoph G. Schuetz, Loris Bozzato

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Microservices](<https://devfeed.tech/topics/microservices.md>), [Big data infrastructure](<https://devfeed.tech/topics/big-data-infrastructure.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [cloud-native-architecture](<https://devfeed.tech/tags/cloud-native-architecture.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

The Knowledge Graph Lakehouse constructs application-specific contextualized knowledge graphs on demand, rather than maintaining one monolithic graph. It indexes incoming documents along configurable context dimensions and stores them in a data lake; independently deployable microservices use configuration for context dimensions and document formats, keeping the architecture domain-independent. The authors report a proof of concept evaluated in experiments on a distributed cluster, but the supplied abstract gives no performance results.

### Source excerpt

arXiv:2611.04899v1 Announce Type: new Abstract: Modern data architectures for integrating heterogeneous data sources to discover actionable knowledge include the data warehouse, the data lake, and the data lakehouse, which combines the flexibility of the data lake with the structured data management capabilities of the data warehouse through metadata, indexes, and caching. Enterprise knowledge graphs (KGs) offer an alternative approach to such integration for the purposes of data analysis, but handling them at scale must address the Three Vs of big data: volume, velocity, and variety. Contextualized KGs (CKGs) meet part of that challenge by enriching facts with contextual information, yet many applications require only a context-specific subset of that knowledge. We propose the Knowledge Graph Lakehouse (KG Lakehouse), a cloud-native microservice architecture that adapts the data lakehouse paradigm for management of CKGs at scale. The KG Lakehouse indexes incoming source documents along configurable context dimensions and stores these documents in a data lake, from which application-specific CKGs are constructed on demand instead of maintaining a single, monolithic KG. We realize the proposed architecture as a proof of concept, with independently deployable services built from established cloud technology; context dimensions and format-specific document handling enter as configuration, leaving the architecture independent of the application domain. We evaluate the implemented system in experiments on a distributed cluster deployment.

## SCANO: Restoring Boundary-Condition Completeness in Sampled Meshes with a Sliced-Cross-Attention Neural Operator for 3D Thermal Fields

DevFeed: [SCANO: Restoring Boundary-Condition Completeness in Sampled Meshes with a Sliced-Cross-Attention Neural Operator for 3D Thermal Fields](<https://devfeed.tech/articles/scano-restoring-boundary-condition-completeness-in-sampled-meshes-with-a-sliced-cross-attention-neural-operator-for-3d-thermal-fields-116961.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04603>)

Author: Tianyu Li, Zhiwei Cao, Qingang Zhang, Ruihang Wang, Binyang Song, Yonggang Wen

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [3D](<https://devfeed.tech/topics/3d.md>), [big-data](<https://devfeed.tech/topics/big-data.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [aggregates](<https://devfeed.tech/tags/aggregates.md>), [all](<https://devfeed.tech/tags/all.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [anchor](<https://devfeed.tech/tags/anchor.md>), [approach](<https://devfeed.tech/tags/approach.md>), [area](<https://devfeed.tech/tags/area.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [condition](<https://devfeed.tech/tags/condition.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fields](<https://devfeed.tech/tags/fields.md>), [neural](<https://devfeed.tech/tags/neural.md>), [operator](<https://devfeed.tech/tags/operator.md>), [physics-comp-ph](<https://devfeed.tech/tags/physics-comp-ph.md>), [thermal](<https://devfeed.tech/tags/thermal.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

SCANO addresses boundary-condition distortions that arise when neural operators are trained on subsampled million-cell meshes and used for whole meshes or selected regions. The abstract identifies incomplete or sample-dependent boundary coverage, face weighting by count instead of area, and compression of boundary inputs as sources of inconsistency. Its backbone-agnostic Boundary Quadrature Attention aggregates typed boundary faces with area weighting into slice tokens read by query points at every layer; Anchored Global State estimates global state from anchors independent of the sampled points. On controlled and public benchmarks, BQA reduces whole-mesh error by up to 15-fold, and the two modules together nearly halve the best original backbone's whole-mesh error. AGS aligns regional and whole-mesh predictions, with regional error cut at least tenfold for most backbones on the public benchmark. On the data-centre thermal simulation dataset and other reported evaluations, SCANO predicts a room-scale field in under one second, over 100 times faster than whole-room anchor self-attention.

### Source excerpt

arXiv:2611.04603v1 Announce Type: new Abstract: Training neural operators on million-cell meshes can exceed GPU memory limits. Models are therefore trained on subsamples and deployed on whole meshes or selected regions. However, subsampling can distort boundary information, degrading accuracy. We ask when boundary conditioning under sampling stays consistent with whole-mesh training, and how to aggregate and inject boundary data. Representing the solution as an integral of boundary data against type-dependent measures, we show that subsampling distorts this conditioning in three ways: incomplete boundary coverage or a sample-dependent global state, face weighting by count rather than area, and compression of boundary inputs into a low-dimensional vector. This analysis yields four measurable consistency conditions checkable before training. We satisfy them with two backbone-agnostic modules. Boundary Quadrature Attention (BQA) aggregates all boundary faces, typed and area-weighted, into slice tokens that every query point reads at every layer. Anchored Global State (AGS) estimates the global state from anchors independent of the sampled points. Adding both to a scalable message-passing and slice-attention backbone yields SCANO. We evaluate the approach on controlled and public benchmarks and a data-centre thermal simulation dataset (DCTS) containing million-cell meshes. On both benchmarks, BQA improves every backbone, reducing whole-mesh error up to 15-fold. On all three datasets, AGS aligns regional with whole-mesh predictions, cutting regional error at least tenfold for most backbones on the public benchmark. Together, the modules nearly halve the best original backbone's whole-mesh error on both benchmarks. SCANO predicts a room-scale field in under one second, over 100 times faster than whole-room anchor self-attention.

[Next page](<https://devfeed.tech/topics/big-data.md?cursor=WyIyMDI2LTExLTA0VDA1OjAwOjAwKzAwOjAwIiwgImFhZGFmMWY3LWMxZGUtNDkxYS1hNDZiLWVjMDgwOGIyZTFiNiJd>)