# Data science and engineering

An interdisciplinary field combining computer science, statistics, information science, and engineering to develop techniques and systems for collecting, managing, integrating, analyzing, and using data.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Stop Removing Stopwords: How an Inherited Preprocessing Default Distorts Legal Text-as-Data

DevFeed: [Stop Removing Stopwords: How an Inherited Preprocessing Default Distorts Legal Text-as-Data](<https://devfeed.tech/articles/stop-removing-stopwords-how-an-inherited-preprocessing-default-distorts-legal-text-as-data-144926.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.19153>)

Author: Gregory M. Dickinson

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [database](<https://devfeed.tech/tags/database.md>), [f1](<https://devfeed.tech/tags/f1.md>), [information-retrieval](<https://devfeed.tech/tags/information-retrieval.md>), [law](<https://devfeed.tech/tags/law.md>), [legal](<https://devfeed.tech/tags/legal.md>), [stop](<https://devfeed.tech/tags/stop.md>), [text](<https://devfeed.tech/tags/text.md>), [tf-idf](<https://devfeed.tech/tags/tf-idf.md>)

### AI overview

An exhaustive single-word ablation found no detectable benefit from stopword removal in two Supreme Court opinion classification tasks: ideological direction (7,668 opinions; baseline F1 about 0.68) and constitutional versus non-constitutional law type (7,001; about 0.92). The study tested roughly 18,500 words per task. Generic stoplists fell below the no-removal baseline in all twelve held-out tests; task-specific stoplists changed held-out F1 by only +0.0023 for ideology (95% CI −0.0124 to +0.0170) and +0.0001 for law type (−0.0082 to +0.0085). The authors argue that, in interpretable legal text-as-data pipelines, inherited preprocessing choices should be evaluated against the downstream objective because reshaping available features may alter the signal researchers aim to recover.

### Source excerpt

arXiv:2609.19153v3 Announce Type: replace-cross Abstract: Empirical legal scholarship increasingly treats judicial text as data, and much of it still runs on sparse, interpretable pipelines -- TF-IDF features and linear classifiers -- because the textual feature is often the object of study rather than a means to prediction. Yet these pipelines inherit preprocessing defaults from mid-century information retrieval that were never validated against classification accuracy. The most entrenched of these is stopword removal. This study introduces an exhaustive single word ablation that measures the effect of a preprocessing step directly against the downstream objective, and applies it to stopword removal as the hardest case to dislodge. Matching Supreme Court Database labels to Caselaw Access Project opinion texts, the study examines two binary tasks that bracket F1 headroom, ideological direction (no-removal baseline F1 ~0.68) and constitutional versus non-constitutional law type (~0.92), across 7,668 and 7,001 opinions. For each task, the ablation removes each of about 18,500 candidate words in turn, and a task-specific stoplist is built from the measurements. Common generic stoplists fall below the no-removal baseline in all twelve held-out tests. The task-specific stoplists move held-out F1 by +0.0023 (95% CI [-0.0124, +0.0170]) on ideology and by +0.0001 ([-0.0082, +0.0085]) on law type. Neither task shows a detectable benefit from removal, and supplemental analysis finds that word-level statistics predict removal effects poorly, because true removal effects differ by less than the measurement can register, leaving little to predict. The method generalizes to any inherited preprocessing default; the result is a caution specific to interpretable legal text-as-data, where a step that reshapes which features a model sees can distort the doctrinal and ideological signal the research is meant to recover. The burden of proof sits with removal.

## Neural Conditional Gaussian Model: Efficient Data Assimilation with Sequential Uncertainty Quantification for Nonlinear Dynamics

DevFeed: [Neural Conditional Gaussian Model: Efficient Data Assimilation with Sequential Uncertainty Quantification for Nonlinear Dynamics](<https://devfeed.tech/articles/neural-conditional-gaussian-model-efficient-data-assimilation-with-sequential-uncertainty-quantification-for-nonlinear-dynamics-145185.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26463>)

Author: Chuanqi Chen, Nan Chen

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-driven](<https://devfeed.tech/tags/data-driven.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [encoder-decoder](<https://devfeed.tech/tags/encoder-decoder.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [math-ds](<https://devfeed.tech/tags/math-ds.md>), [model](<https://devfeed.tech/tags/model.md>), [neural](<https://devfeed.tech/tags/neural.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>)

### AI overview

Neural CGM learns a dynamical model by mapping physical states into a latent space with conditional Gaussian dynamics, enabling closed-form data-assimilation updates there. Nonlinear encoder-decoder mappings represent physical-space dynamics; a Gaussian decoder maps sequential latent posterior means and covariances back to physical variables using local moment propagation and learned corrections. These posterior statistics also enter a Gaussian negative log-likelihood during training, so the model learns from both estimated states and uncertainty. It can forecast and simulate without explicit governing equations. Benchmark experiments report accurate estimation and forecasting with substantially lower online assimilation cost than conventional nonlinear data-assimilation methods. The supplied abstract does not quantify these results.

### Source excerpt

arXiv:2611.26463v1 Announce Type: new Abstract: Data assimilation (DA) in partially observed nonlinear systems requires repeated estimation of both hidden states and their uncertainty. Conventional nonlinear filtering can be computationally expensive and often relies on an explicit dynamical model. In this paper, we develop the Neural Conditional Gaussian Model (Neural CGM), a data-driven dynamical model that maps unobserved physical states into a learned latent space, where the dynamics of the latent states follow a conditional Gaussian structure. This structure allows closed-form DA updates in the latent space, while nonlinear encoder-decoder mappings retain the flexibility needed to represent complex dynamics in physical space. A key feature of Neural CGM is its treatment of uncertainty in DA. The latent posterior mean and covariance are mapped to the physical variables through a Gaussian decoder that combines local moment propagation with learned nonlinear corrections. The resulting physical-space posterior statistics depend on the sequentially updated latent distribution rather than on a separate post-processing estimate. The same statistics enter a Gaussian negative log-likelihood during training, so both the posterior mean and uncertainty affect the learned model. Because Neural CGM learns the dynamical model itself, it can also be used for state forecasting and simulation without explicit knowledge of the governing equations. Numerical experiments on benchmark nonlinear dynamical systems show accurate state estimation and forecasting while substantially reducing online assimilation cost relative to conventional nonlinear DA methods. Neural CGM is therefore suited to repeated state estimation, uncertainty quantification, and prediction from partial observations.

## TabPFN: One Model to Rule Them All?

DevFeed: [TabPFN: One Model to Rule Them All?](<https://devfeed.tech/articles/tabpfn-one-model-to-rule-them-all-144854.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2505.20003>)

Author: Qiong Zhang, Yan Shuo Tan, Qinglong Tian, Pengfei Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [application](<https://devfeed.tech/tags/application.md>), [bayesian-inference](<https://devfeed.tech/tags/bayesian-inference.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [tabpfn](<https://devfeed.tech/tags/tabpfn.md>), [tabular-data](<https://devfeed.tech/tags/tabular-data.md>)

### AI overview

The paper interprets TabPFN, a transformer for tabular regression and classification, as approximate Bayesian inference and examines its potential relevance to statistics. The authors report that out-of-the-box TabPFN sometimes outperforms specialized state-of-the-art methods in semi-supervised parameter estimation, prediction under covariate shift, and heterogeneous treatment effect estimation. They suggest one explanation is its ability to adapt to both nonparametric and parametric structure; in some experiments, it outperforms LASSO even when LASSO's assumptions are correctly specified. The evidence provided is an abstract, so it does not detail experimental setups or results.

### Source excerpt

arXiv:2505.20003v3 Announce Type: replace Abstract: Hollmann et al. (Nature 637 (2025) 319-326) recently introduced TabPFN, a transformer-based deep learning model for regression and classification on tabular data, which they claim "outperforms all previous methods on datasets with up to 10,000 samples by a wide margin, using substantially less training time." Furthermore, they have called TabPFN a "foundation model" for tabular data, as it can support "data generation, density estimation, learning reusable embeddings and fine-tuning". In this paper, we provide a tailored explanation of how TabPFN works for a statistics audience, by emphasizing its interpretation as approximate Bayesian inference. We then explore the significance of TabPFN to the field of statistics: We show that an out-of-the-box application of TabPFN can sometimes outperform specialized state-of-the-art methods for semi-supervised parameter estimation, prediction under covariate shift, and heterogeneous treatment effect estimation. As a partial explanation for the predictive effectiveness of TabPFN, we show that it can simultaneously adapt to both nonparametric structure and parametric structure, for instance, sometimes outperforming LASSO even when assumptions are correctly specified. All experiments can be reproduced using the code provided at https://github.com/qinglong-tian/tabpfn_study (https://github.com/qinglong-tian/tabpfn_study).

## An Explainable and Scalable Hierarchical Tensor Framework for Integrating Multi-Omics and Digital Pathomics

DevFeed: [An Explainable and Scalable Hierarchical Tensor Framework for Integrating Multi-Omics and Digital Pathomics](<https://devfeed.tech/articles/an-explainable-and-scalable-hierarchical-tensor-framework-for-integrating-multi-omics-and-digital-pathomics-145543.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25584>)

Author: Ahmad Hussein, Ali Anaissi, Mukesh Prasad, Ali Braytee

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [ggml](<https://devfeed.tech/topics/ggml.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [clustering](<https://devfeed.tech/topics/clustering.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [background](<https://devfeed.tech/tags/background.md>), [based](<https://devfeed.tech/tags/based.md>), [cancer](<https://devfeed.tech/tags/cancer.md>), [combines](<https://devfeed.tech/tags/combines.md>), [complete](<https://devfeed.tech/tags/complete.md>), [compress](<https://devfeed.tech/tags/compress.md>), [copy](<https://devfeed.tech/tags/copy.md>), [crispr](<https://devfeed.tech/tags/crispr.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [digital](<https://devfeed.tech/tags/digital.md>), [dna](<https://devfeed.tech/tags/dna.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [essential](<https://devfeed.tech/tags/essential.md>), [example](<https://devfeed.tech/tags/example.md>), [framework](<https://devfeed.tech/tags/framework.md>), [q-bio-gn](<https://devfeed.tech/tags/q-bio-gn.md>)

### AI overview

HIT-Fuse combines six molecular and pathology modalities to identify cancer subgroups and support survival prediction despite missing data. It compresses each modality with an autoencoder, applies two-stage CP tensor decomposition, and reconstructs missing latent modalities from clinical covariates and observed omics; a mathematical traceback provides global and local attributions. Across four TCGA cohorts, it reports significant survival stratification in glioma, breast, and kidney cancer with complete or reconstructed modalities. Cohort expansion produced cohort-specific gains, including a 56.4% relative increase in weighted F1 for glioma. Biological checks supported decoded drivers across all four cohorts, though validation coverage varied: 71-92% of decoded miRNAs had miRTarBase support and 77-84% of selected genes were covered in DepMap.

### Source excerpt

arXiv:2611.25584v1 Announce Type: cross Abstract: Background: Precision oncology requires integration of heterogeneous molecular and pathology data, yet many fusion models offer limited biologically grounded interpretability and struggle with missing modalities. Methods: We present Hierarchical Interpretable Tensor Fusion (HIT-Fuse), integrating six modalities: somatic copy number variation, DNA methylation, RNA sequencing, microRNA expression, RPPA protein abundance, and vision transformer-derived pathomics across four TCGA cohorts (glioma, lung adenocarcinoma, breast, and kidney cancer). Modality-specific autoencoders compress high-dimensional inputs, followed by two-stage CP tensor decomposition. Missing latent modalities are reconstructed using clinical covariates and observed omics to expand cohorts and retrain downstream models. A mathematical traceback mechanism enables global and local xAI attributions. Results: HIT-Fuse identifies distinct latent subgroups with significant survival stratification in glioma, breast, and kidney cohorts, both with complete and reconstructed modalities. Cohort expansion yields cohort-specific gains, including a 56.4% relative increase in weighted F1 for glioma. Biological validation against DepMap CRISPR screens, miRTarBase and RPPA annotations supports decoded drivers across all four cohorts. Essential gene rates range from 6.2% in glioma to 7.6% in lung cancer; example drivers include ANAPC11, SNRPD1, PLK1 and RPL14. Of decoded miRNAs, 71-92% have miRTarBase regulatory support, while 77-84% of selected genes are covered in DepMap. Conclusions: HIT-Fuse combines hierarchical tensor fusion, missing-modality reconstruction and traceable attributions to support interpretable multi-omics and digital pathology-based cancer prognosis across heterogeneous cohorts.

## How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial

DevFeed: [How to Generate Synthetic Healthcare Data Using Synthea: A Hands On Tutorial](<https://devfeed.tech/articles/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial-144721.md>)

Original publisher: [Read original article](<https://www.freecodecamp.org/news/how-to-generate-synthetic-healthcare-data-using-synthea-a-hands-on-tutorial/>)

Author: Lakshmi Mahabaleshwara

Published: 2026-11-19T22:00:42Z

Content type: tutorial

Language: en

Sources: [Free Code Camp](<https://devfeed.tech/sources/freecodecamp-programming-tutorials-python-javascript-git-more.md>)

Topics: [Tutorial](<https://devfeed.tech/topics/tutorial.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [applications](<https://devfeed.tech/tags/applications.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [dataengineering](<https://devfeed.tech/tags/dataengineering.md>), [generate](<https://devfeed.tech/tags/generate.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [healthcareai](<https://devfeed.tech/tags/healthcareai.md>), [healthcaredata](<https://devfeed.tech/tags/healthcaredata.md>), [healthtech](<https://devfeed.tech/tags/healthtech.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetichealthcaredata](<https://devfeed.tech/tags/synthetichealthcaredata.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Synthea simulates longitudinal patient records, making it useful for prototyping and testing healthcare data workflows without using real patient histories. The tutorial shows how to build it with Java 17 or newer, generate about 100 patients, enable CSV export explicitly, and explore linked patient, encounter, condition, observation, medication, and procedure files with Python and Pandas. A fixed seed helps reproduce runs; version, configuration, and reference date also affect results. Default population overflow can produce more records than requested (115 in the described run); disabling it targets the requested count but may change how closely survivor demographics match population targets. The tutorial demonstrates basic relationship and missing-value checks and reconstructing a patient's timeline. Synthetic data is not necessarily representative, clinically accurate, unbiased, or suitable evidence for real-world model performance, so validate concepts, data quality, and representativeness before use.

### Source excerpt

Healthcare generates huge amounts of data every day, from patient visits and lab results to diagnoses, medications, and medical imaging. For developers and data engineers building healthcare applicati

## Neoadjuvant chemotherapy response prediction using pretreatment diffusion and contrast-enhanced magnetic resonance imaging with clinical variables

DevFeed: [Neoadjuvant chemotherapy response prediction using pretreatment diffusion and contrast-enhanced magnetic resonance imaging with clinical variables](<https://devfeed.tech/articles/neoadjuvant-chemotherapy-response-prediction-using-pretreatment-diffusion-and-contrast-enhanced-magnetic-resonance-imaging-with-clinical-variables-142564.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.26105>)

Author: Pablo Garc\\'ia Marcos, Paula Puerta Gonz\\'alez, Guillermo Lorenzo, H\\'ector G\\'omez, Covadonga del Camino, Ad\\'an Rodr\\'iguez, Ignacio Pel\\'aez, Angel Rio-Alvarez, V\\'ictor M. Gonz\\'alez

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Medical imaging](<https://devfeed.tech/topics/medical-imaging.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [cancer](<https://devfeed.tech/tags/cancer.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [efficientnet](<https://devfeed.tech/tags/efficientnet.md>), [external](<https://devfeed.tech/tags/external.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [imaging](<https://devfeed.tech/tags/imaging.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [response](<https://devfeed.tech/tags/response.md>), [variables](<https://devfeed.tech/tags/variables.md>)

### AI overview

A pretreatment model combining apparent diffusion coefficient maps, dynamic contrast-enhanced MRI, and HR/HER2 subtype achieved a mean AUC of 0.86 for predicting pathological complete response to neoadjuvant chemotherapy. On the public ACRIN 6698/I-SPY2 multicenter dataset, stratified five-fold cross-validation yielded mean AUCs of 0.79 for ADC maps alone and 0.74 for DCE-MRI alone; adding HR/HER2 subtype raised these to 0.83 and 0.81. Of the clinical variables evaluated, only HR/HER2 subtype improved average AUC and was retained. The model uses pretrained EfficientNet-B0 image encoders with late fusion of clinical information. External validation is still required before clinical use.

### Source excerpt

arXiv:2609.26105v2 Announce Type: replace Abstract: Prediction of pathological complete response before neoadjuvant chemotherapy may facilitate more tailored therapeutic planning for breast cancer patients. This work proposes a deep-learning model for pretreatment data only, combining apparent diffusion coefficient maps, dynamic contrast-enhanced magnetic resonance imaging, and clinical variables. The study uses the public ACRIN 6698/I-SPY2 multicenter dataset. The architecture employs EfficientNet-B0 pretrained encoders for image feature extraction and late fusion with clinical information. Multiple clinical variables were evaluated, including age, race, histological type, HR/HER2 subtype, SBR grade, and maximum diameter. Only HR/HER2 subtype improved the average area under the receiver operating characteristic curve (AUC) and was retained in the final model. Using stratified five-fold cross-validation, standalone apparent diffusion coefficient maps achieved a mean AUC of 0.79, whereas dynamic contrast-enhanced magnetic resonance imaging achieved 0.74. Adding HR/HER2 subtype improved performance to 0.83 and 0.81, respectively. The final configuration, using both imaging modalities and HR/HER2 subtype, achieved an AUC of 0.86. These results support pretreatment multimodal learning for response prediction, although external validation is required before clinical use.

## Integrating AI Coding Agents Into Scientific Research

DevFeed: [Integrating AI Coding Agents Into Scientific Research](<https://devfeed.tech/articles/integrating-ai-coding-agents-into-scientific-research-142179.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23334>)

Author: Rory Barnes

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [version-control](<https://devfeed.tech/topics/version-control.md>), [trust](<https://devfeed.tech/topics/trust.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [accessible](<https://devfeed.tech/tags/accessible.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-development](<https://devfeed.tech/tags/agentic-development.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [also](<https://devfeed.tech/tags/also.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [archive](<https://devfeed.tech/tags/archive.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [astro-ph-im](<https://devfeed.tech/tags/astro-ph-im.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [research](<https://devfeed.tech/tags/research.md>), [scientific-research](<https://devfeed.tech/tags/scientific-research.md>)

### AI overview

AI coding agents can speed up scientific analysis and figure generation, but errors may damage a researcher's computer, increase validation effort, and weaken confidence in results. The paper proposes the six-level PROOF scale, which classifies AI-assisted numerical experiments by provenance, reproducibility, openness, oversight, and falsifiability. Its levels progress from locally self-consistent, version-controlled, human-verified results to a curated public archive with files, change history, prompt transcripts, independent agent regeneration, and cryptographically verifiable attestations. The open-source vaibify package supports this workflow: Docker containers isolate agents, while a GUI lets researchers prompt command-line agents, monitor local and remote file changes, and work toward PROOF Level 3, described as reproducible. The authors suggest these methods may reduce verification time and help communicate trust in AI-assisted science.

### Source excerpt

arXiv:2611.23334v1 Announce Type: new Abstract: New software tools have emerged that enable AI agents to quickly develop code that analyzes data and generates figures, but their mistakes can also harm a researcher's computer, require extensive resources for validation, and cause the results to be received more skeptically. These concerns could be mitigated if scientific software is developed in a secure environment with a system for monitoring file changes and well-defined standards for handling AI agents. This paper first describes how coding agents change eleven aspects of generating, verifying, and reproducing numerical results. A six-level scale is then proposed for classifying AI-assisted numerical experiments by their provenance, reproducibility, openness, oversight, and falsifiability (PROOF). The scale begins with locally self-consistent, version-controlled, human-verified results and culminates in a publicly accessible and curated archive of the files, change history, prompt transcripts, independent agent regeneration of the analysis, and multiple cryptographically verifiable attestations of regeneration by independent researchers. These aspects of AI-assisted scientific software development have been synthesized into a new open source software package called vaibify that optimizes the agentic development of computational experiments ascending the PROOF Ladder. Docker containers sequester the agents while a GUI enables researchers to prompt command line agents, monitor the current status of local and remote files, and drive a project to PROOF Level 3 (reproducible). The methods described in this paper could help researchers reduce the time required to verify agent-written software and efficiently communicate trust in AI-assisted science.

## Architectural Design Decisions for Data Ingestion in Big Data Architectures

DevFeed: [Architectural Design Decisions for Data Ingestion in Big Data Architectures](<https://devfeed.tech/articles/architectural-design-decisions-for-data-ingestion-in-big-data-architectures-142161.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23297>)

Author: Yikang Huang, Stefano Fossati, Indika Kumara, Damian A. Tamburri, Willem-Jan van den Heuvel

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-architecture](<https://devfeed.tech/tags/data-architecture.md>), [data-engineers](<https://devfeed.tech/tags/data-engineers.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-ingestion-pipeline](<https://devfeed.tech/tags/data-ingestion-pipeline.md>), [data-ingestion-pipelines](<https://devfeed.tech/tags/data-ingestion-pipelines.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [decision](<https://devfeed.tech/tags/decision.md>), [design](<https://devfeed.tech/tags/design.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [experts](<https://devfeed.tech/tags/experts.md>), [framework](<https://devfeed.tech/tags/framework.md>), [functional](<https://devfeed.tech/tags/functional.md>), [options](<https://devfeed.tech/tags/options.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

The paper presents a framework to help data engineers and architects choose designs for data ingestion pipelines based on functional and non-functional requirements. A systematic review of 42 gray literature articles and qualitative analysis produced a catalog of 8 architectural decisions, 59 options, and 36 selection drivers. Semi-structured expert interviews assessed the framework's completeness, simplicity, understandability, operational feasibility, and usefulness. The supplied text reports that these qualities were evaluated, but does not give the interview findings or name the decisions and options.

### Source excerpt

arXiv:2611.23297v1 Announce Type: new Abstract: Introduction: Data ingestion is the first critical step in a data architecture, in which data from various sources is moved to storage systems, such as data lakes and data warehouses, for analysis and insight generation. When developing a data ingestion pipeline, practitioners (e.g., data engineers and architects) must understand the data ingestion requirements (functional and non-functional) and select appropriate design options to support them. Objective: To help practitioners systematically develop data ingestion processes, this paper aims to compile a catalog of data ingestion design decisions, their options, and the selection drivers. Method: We conducted a systematic gray literature review on data ingestion, analyzed the selected literature using qualitative data analysis methods, and conducted semi-structured interviews with experts to assess the framework's practicality and utility. Results: From 42 gray literature articles, we identified 8 architectural design decisions, 59 decision options, and 36 decision drivers to guide practitioners in the design and implementation of data ingestion pipelines. The framework was evaluated through expert interviews, assessing its completeness, simplicity, understandability, operational feasibility, and usefulness.

## Where Should Physics Enter? Information, Identifiability and Calibration in Probabilistic Sea Surface Temperature Forecasting

DevFeed: [Where Should Physics Enter? Information, Identifiability and Calibration in Probabilistic Sea Surface Temperature Forecasting](<https://devfeed.tech/articles/where-should-physics-enter-information-identifiability-and-calibration-in-probabilistic-sea-surface-temperature-forecasting-141849.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23129>)

Author: Parmitha Vangapandu, Jonnakuti Pavan Kumar, Sathwik Narkedimilli, MSVPJ Sathvik, T. M. Balakrishnan Nair, Santanu Bhattacharya

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [forecast](<https://devfeed.tech/tags/forecast.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [network](<https://devfeed.tech/tags/network.md>), [neural](<https://devfeed.tech/tags/neural.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [physics](<https://devfeed.tech/tags/physics.md>), [physics-informed](<https://devfeed.tech/tags/physics-informed.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [surface](<https://devfeed.tech/tags/surface.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [transport](<https://devfeed.tech/tags/transport.md>), [using](<https://devfeed.tech/tags/using.md>), [v2](<https://devfeed.tech/tags/v2.md>), [variance](<https://devfeed.tech/tags/variance.md>), [velocity](<https://devfeed.tech/tags/velocity.md>)

### AI overview

In these Bay of Bengal SST experiments, learned advection-diffusion priors did not reliably improve forecasts or recover currents. On real data, a learned-velocity residual offered no reliable lead-15 CRPS benefit, while GLORYS12-current transport performed worse at leads 1 and 5; reanalysis advection scarcely tracked daily OISST changes. In the synthetic benchmark, supplied currents helped, but input channels without transport recovered 92% of transport's lead-5 gain, and a frozen-current numerical forecast performed similarly. Across models, lead-dependent miscalibration was the common issue: fitting one variance multiplier per lead on validation data brought 90% coverage to 0.85-0.91 and reduced day-1 CRPS by 13-21%. Fine-tuning on 15-day rollouts corrected long-lead coverage at the cost of short-lead skill. The excerpt also withdraws an earlier 81% CRPS reduction as based on an uncontrolled comparison.

### Source excerpt

arXiv:2611.23129v1 Announce Type: new Abstract: Advection-diffusion priors are a common way to make neural sea surface temperature (SST) forecasts physics-informed. We test where such a prior must enter a heteroscedastic ConvLSTM to help, using daily NOAA OISST v2.1 for the Bay of Bengal (1982-2024) and a synthetic benchmark on the same grid with known currents of controlled strength. (i) On real data, a soft advection-diffusion residual with a learned velocity gives no reliable benefit (lead-15 CRPS -0.007 degrees C, 95% interval [-0.022, +0.008]); a plain temporal-smoothness penalty matches it, and equivalence within 3% is not established either. (ii) Learned velocities do not recover the currents, whether placed in the loss, in an explicit transport step, or in a transport step without neural correction: they correlate with the reference no better than a shuffled-date null, even on synthetic data where knowing the currents would improve day-15 forecasts by 16-23%. (iii) Supplied currents help on synthetic data, but mostly as information: giving them to the network as input channels, without transport, recovers 92% of the transport design's lead-5 gain, and a frozen-current numerical forecast does as well as either. (iv) With GLORYS12 reanalysis currents on real data, the transport design is 6% and 4% worse at leads 1 and 5; reanalysis advection explains essentially none of the daily OISST change (correlation 0.02), and frozen-current transport is worse than persistence. (v) The defect that matters is lead-dependent miscalibration shared by every model. One variance multiplier per lead, fitted on validation data, brings 90% coverage to 0.85-0.91 at all leads and lowers day-1 CRPS by 13-21%; fine-tuning on 15-day rollouts fixes long-lead coverage but costs short-lead skill. This version withdraws an earlier claim of an 81% CRPS reduction that came from an uncontrolled comparison.

## Solve B2C Attribution Challenges With Fivetran and Braze | Blog | Fivetran

DevFeed: [Solve B2C Attribution Challenges With Fivetran and Braze | Blog | Fivetran](<https://devfeed.tech/articles/solve-b2c-attribution-challenges-with-fivetran-and-braze-blog-fivetran-140257.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/solve-b-2-c-attribution-challenges-with-fivetran-and-braze>)

Author: Fivetran

Published: 2026-11-18T02:46:08Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Data Mining & Modeling](<https://devfeed.tech/topics/data-mining-modeling.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [advertising](<https://devfeed.tech/tags/advertising.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [braze](<https://devfeed.tech/tags/braze.md>), [campaign](<https://devfeed.tech/tags/campaign.md>), [cloud-data-warehouse](<https://devfeed.tech/tags/cloud-data-warehouse.md>), [connectors](<https://devfeed.tech/tags/connectors.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [customer-engagement](<https://devfeed.tech/tags/customer-engagement.md>), [data](<https://devfeed.tech/tags/data.md>), [ecommerce](<https://devfeed.tech/tags/ecommerce.md>), [facebook-ads](<https://devfeed.tech/tags/facebook-ads.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [funnel](<https://devfeed.tech/tags/funnel.md>), [google-ads](<https://devfeed.tech/tags/google-ads.md>), [marketing](<https://devfeed.tech/tags/marketing.md>)

### AI overview

The article's approach to B2C attribution is to centralize engagement, advertising and transaction data in a cloud warehouse for analysis. Braze provides detailed customer behavior across devices, channels and touchpoints; Fivetran connectors replicate and normalize that data, with in-warehouse transformations for SQL or BI analysis. The article says ad connectors for services such as Google Ads and Facebook Ads use predefined schemas and recommended configurations, account for restatements and attribution windows, and employ techniques to address sampling error. Payment connectors such as Square, Stripe and Recurly can add conversion data, helping teams connect campaign activity to outcomes and inform ad-spend allocation. The excerpt describes this as an approach, not evidence of measured attribution accuracy or results.

### Source excerpt

In today's marketing environment, B2C attribution is a huge challenge. Fivetran connectors make it easier by effortlessly combining transactional, behavioral and campaign data.

## $\\beta$-number and Its Applications to Data Science

DevFeed: [$\\beta$-number and Its Applications to Data Science](<https://devfeed.tech/articles/beta-number-and-its-applications-to-data-science-138865.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19379>)

Author: Pengxu Wen, Ziwei Nie, Liwen Zou, Ying Cao, Xiaoping Yang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Data Science](<https://devfeed.tech/topics/data-science.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [image-classification](<https://devfeed.tech/tags/image-classification.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [metric](<https://devfeed.tech/tags/metric.md>), [principal-component-analysis](<https://devfeed.tech/tags/principal-component-analysis.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>)

### AI overview

The paper defines the β-number of a set as the minimum average p-th-power distance of its points to a d-dimensional plane, using it to quantify oscillation and identify a subspace for dimensionality reduction. In discrete settings with p=2, the selected plane coincides with the PCA plane, making β-number a generalization of PCA. The authors report bounds and a scaling law, describe algorithms for continuous and discrete settings, and test results numerically in low dimensions. They also demonstrate potential applications in dimensionality reduction and image classification; the abstract gives no performance details.

### Source excerpt

arXiv:2611.19379v1 Announce Type: new Abstract: In this paper, we introduce a novel metric, $\beta$-number, designed to measure the "$p$-th order $d$-dimensional oscillation degree" of a set. For a given set $K$ in the Euclidean space $\mathbb{R}^n$, $\beta$-number is computed by identifying an optimal $d$-dimensional plane $L$ that minimizes the average $p$-th order distance of all points in $K$ to $L$. This metric serves as a versatile tool in data science, functioning both as a dimensionality reduction technique to find the optimal $p$-th order $d$-dimensional subspace and as a measure to characterize data oscillation for classification. We conduct a thorough exploration of $\beta$-number through theoretical analysis and numerical experiments. Initially, we establish the upper and lower bounds of $\beta$-number for any set $K$, along with its scaling law. We then elucidate the relationships between $\beta$-number and its related parameters. A key finding of this study is that in discrete settings, when $p=2$, the $d$-dimensional plane determined by $\beta$-number coincides with that obtained via Principal Component Analysis (PCA), positioning $\beta$-number as a generalization of PCA. Furthermore, we propose two algorithms for calculating $\beta$-number in both continuous and discrete settings. We validate our theoretical results through a series of numerical experiments in low-dimensional cases. Finally, we demonstrate the potential of $\beta$-number in practical data analysis through a dimensionality reduction task and an image classification task.

## Unlocking the Duality between Flow and Field Matching

DevFeed: [Unlocking the Duality between Flow and Field Matching](<https://devfeed.tech/articles/unlocking-the-duality-between-flow-and-field-matching-138739.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.02261>)

Author: Daniil Shlenskii, Alexander Varlamov, Nazar Buzun, Alexander Korotin

Published: 2026-11-17T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Development](<https://devfeed.tech/topics/ai-development.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [conditional-probability](<https://devfeed.tech/tags/conditional-probability.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [duality](<https://devfeed.tech/tags/duality.md>), [field](<https://devfeed.tech/tags/field.md>), [flow](<https://devfeed.tech/tags/flow.md>), [flow-matching](<https://devfeed.tech/tags/flow-matching.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [general](<https://devfeed.tech/tags/general.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [maps](<https://devfeed.tech/tags/maps.md>), [matching](<https://devfeed.tech/tags/matching.md>), [models](<https://devfeed.tech/tags/models.md>), [orientation](<https://devfeed.tech/tags/orientation.md>), [physics](<https://devfeed.tech/tags/physics.md>)

### AI overview

Conditional Flow Matching (CFM) and forward-only Interaction Field Matching (IFM) induce the same family of data-space dynamics, with explicit constructions connecting the two. CFM defines conditional probability paths in data space, while IFM defines physics-inspired interaction fields in an augmented space. General IFM is more expressive: even Electrostatic Field Matching can produce orientation-reversing transport maps with positive probability, which CFM flows cannot. The authors propose using the probabilistic and geometric views together and exploring field-based transport beyond the forward-only case.

### Source excerpt

arXiv:2602.02261v2 Announce Type: replace Abstract: Conditional Flow Matching (CFM) unifies conventional generative paradigms such as diffusion and flow-based models. Interaction Field Matching (IFM) is a recently proposed framework that generalizes Electrostatic Field Matching (EFM), rooted in Poisson Flow Generative Models (PFGM). While both frameworks define generative dynamics, they start from different objects: CFM specifies a conditional probability path in data space, whereas IFM specifies a physics-inspired interaction field in an augmented data space. This raises a basic question: are CFM and IFM genuinely different, or are they two descriptions of the same underlying dynamics? We show that they coincide for a natural subclass of IFM, which we call forward-only IFM. Specifically, we give explicit constructions in both directions and show that CFM and forward-only IFM induce the same family of data-space dynamics. General IFM, however, is strictly more expressive: already in EFM, its sampling rule yields transport maps that reverse orientation with positive probability, something no CFM flow can do. Our findings suggest using the probabilistic and geometric views jointly when developing generative models, and exploring field-based transport beyond the forward-only regime.

## Parameterization method of reservoir properties for ensemble-based data assimilation using intermediate latent space of StyleGAN

DevFeed: [Parameterization method of reservoir properties for ensemble-based data assimilation using intermediate latent space of StyleGAN](<https://devfeed.tech/articles/parameterization-method-of-reservoir-properties-for-ensemble-based-data-assimilation-using-intermediate-latent-space-of-stylegan-135009.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.39626>)

Author: Marcio A. Sampaio, Paulo H. Ranazzi, Martin J. Blunt

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [case-studies](<https://devfeed.tech/tags/case-studies.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [deep-learning-models](<https://devfeed.tech/tags/deep-learning-models.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [linear](<https://devfeed.tech/tags/linear.md>), [matching](<https://devfeed.tech/tags/matching.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [results](<https://devfeed.tech/tags/results.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [vae](<https://devfeed.tech/tags/vae.md>)

### AI overview

In two 2D reservoir cases, StyleGAN2 parameterization in its intermediate w-space produced better data-assimilation results than using the traditional z-space. The study compares VAE-GAN, Latent Diffusion, and StyleGAN models for ensemble-based history matching of three-facies categorical and continuous properties. All were reported as efficient, while StyleGAN2 stood out for geological realism and data matching. The authors attribute w-space's advantage to its greater linearity and disentanglement, which better suits ESMDA's linear updates and helps keep updated vectors near realistic geological patterns. Results were assessed with geostatistical and history-matching metrics; the evidence covers only these two 2D cases.

### Source excerpt

arXiv:2609.39626v2 Announce Type: replace-cross Abstract: Ensemble smoothers are the most successful and efficient techniques currently available for history matching. However, because these methods rely on Gaussian assumptions, their performance is severely degraded when the prior geology is described in terms of complex facies distributions (non-Gaussian). In this way, for these methods, we need to apply efficient parameterization techniques. Currently, the most efficient methods for performing parameterization are deep learning models. However, given the variety of existing deep learning models, studies have not identified which is most suitable for use with ensemble-based methods, although some important models had already been evaluated. Based on a recent literature review, the most promising models selected were VAE-GAN, Latent Diffusion, and StyleGAN models. As a novel aspect of this work, data assimilation with the second generation of StyleGAN (StyleGAN2) model was performed using the latent z-space and intermediate w-space, separately. They were applied in two 2D case studies: one categorical (three facies) and the other continuous. The results demonstrated that all three models are highly efficient, with the StyleGAN2 model standing out for generating samples with geological realism and achieving excellent data matching in the cases studied. Our findings show that performing data assimilation with StyleGAN2 using the intermediate space (w-space) yielded better results than the traditional application in the latent space (z-space). This is due to the fact that ESMDA uses linear updates and the w-space is much more linear and disentangled than the highly entangled z-space, thereby ensuring that the updated vectors remain close to realistic geological patterns. These results were validated using main geostatistical and history matching metrics.

## ReCo: When to Relocate Sensor Kits under Deployment Constraints -- A NILM Case Study

DevFeed: [ReCo: When to Relocate Sensor Kits under Deployment Constraints -- A NILM Case Study](<https://devfeed.tech/articles/reco-when-to-relocate-sensor-kits-under-deployment-constraints-a-nilm-case-study-135718.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.35075>)

Author: Haokun Chen, Yu Tong, Yehai Chen

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [IoT sensor data forecasting](<https://devfeed.tech/topics/iot-sensor-data-forecasting.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [appliances](<https://devfeed.tech/tags/appliances.md>), [background](<https://devfeed.tech/tags/background.md>), [climate](<https://devfeed.tech/tags/climate.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [counts](<https://devfeed.tech/tags/counts.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [days](<https://devfeed.tech/tags/days.md>), [decision](<https://devfeed.tech/tags/decision.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [feature](<https://devfeed.tech/tags/feature.md>), [field](<https://devfeed.tech/tags/field.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [framework](<https://devfeed.tech/tags/framework.md>), [reco](<https://devfeed.tech/tags/reco.md>), [sensor](<https://devfeed.tech/tags/sensor.md>)

### AI overview

With limited sensor kits, a collection deadline, and downtime at each move, ReCo (Relocation by Coverage gain) helps decide whether to keep collecting at a home or relocate for non-intrusive load monitoring (NILM). It counts new operating regimes in a joint target-background feature space, forecasts each home's future data gain, and compares staying with moving each night after accounting for downtime. In replayed Plegma deployments across two kit counts and two downtime costs, ReCo outperformed fixed-dwell and count-based schedules and a threshold rule on the same metric in every setting. The reported advantage was not explained by collecting more days alone and was consistent with allocating collection days to more valuable homes and periods.

### Source excerpt

arXiv:2609.35075v2 Announce Type: replace Abstract: Many sensing tasks obtain training labels only by deploying instruments in the field. With a limited number of sensor kits, a collection deadline, and measurement downtime at every move, the collector must repeatedly decide whether to stay at the current site or relocate. We study this decision in non-intrusive load monitoring (NILM), which estimates the power drawn by individual appliances from a home's main meter and is trained on data from homes temporarily fitted with appliance-level sub-meters. In NILM, appliance usage varies with the appliance, season and climate, and the value of new data depends on how diverse the combinations of target operation and background load are. To address this, we propose a constraint-based relocation framework and instantiate it for NILM as ReCo (Relocation by Coverage gain). ReCo counts new operating regimes in a joint target--background feature space, forecasts each home's future gain from the data collected so far, and each night weighs the gain of staying against the gain of moving elsewhere after the downtime. In replayed deployments on the Plegma dataset under two kit counts and two downtime costs, ReCo outperforms fixed-dwell and count-based schedules and a threshold rule using the same metric in every setting. Its advantage is not explained by collecting more days alone and is consistent with allocating the days to more valuable homes and periods.

## HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets

DevFeed: [HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets](<https://devfeed.tech/articles/histogramtools-for-efficient-data-analysis-and-distribution-representation-in-large-data-sets-132665.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2504.00001>)

Author: Shubham Malhotra

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [applications](<https://devfeed.tech/tags/applications.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [error](<https://devfeed.tech/tags/error.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [mapreduce](<https://devfeed.tech/tags/mapreduce.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [r](<https://devfeed.tech/tags/r.md>), [storage](<https://devfeed.tech/tags/storage.md>), [visualization](<https://devfeed.tech/tags/visualization.md>)

### AI overview

HistogramTools extends R's built-in histogram features for large-scale and distributed data analysis. It supports serializing histograms with Protocol Buffers, merging and modifying them, and manipulating bins. The abstract also describes distance measures, quantile approximation, and methods to estimate and visualize information loss, including error in CDFs derived from histograms. The package is presented as suited to MapReduce settings where compact storage and data sharing matter; the supplied text does not provide performance results or implementation details.

### Source excerpt

arXiv:2504.00001v2 Announce Type: replace Abstract: Histograms provide a powerful means of summarizing large data sets by representing their distribution in a compact, binned form. The HistogramTools R package enhances R built-in histogram functionality, offering advanced methods for manipulating and analyzing histograms, especially in large-scale data environments. Key features include the ability to serialize histograms using Protocol Buffers for distributed computing tasks, tools for merging and modifying histograms, and techniques for measuring and visualizing information loss in histogram representations. The package is particularly suited for environments utilizing MapReduce, where efficient storage and data sharing are critical. This paper presents various methods of histogram bin manipulation, distance measures, quantile approximation, and error estimation in cumulative distribution functions (CDFs) derived from histograms. Visualization techniques and efficient storage representations are also discussed alongside applications for large data processing and distributed computing tasks.

## Support-Constrained Offline Policy Improvement for Pediatric ECMO from Limited Clinical Data

DevFeed: [Support-Constrained Offline Policy Improvement for Pediatric ECMO from Limited Clinical Data](<https://devfeed.tech/articles/support-constrained-offline-policy-improvement-for-pediatric-ecmo-from-limited-clinical-data-133180.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17134>)

Author: Fateme Golivand Darvishvand, Saurabh Mathur, Nikhilesh Prabhakar, Michael A. Skinner, Ameet Soni, Neel Shah, Ethan L Sanford, Lakshmi Raman, Sriraam Natarajan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [000](<https://devfeed.tech/tags/000.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [actions](<https://devfeed.tech/tags/actions.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balance](<https://devfeed.tech/tags/balance.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [can](<https://devfeed.tech/tags/can.md>), [children](<https://devfeed.tech/tags/children.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [imitation](<https://devfeed.tech/tags/imitation.md>), [imitation-learning](<https://devfeed.tech/tags/imitation-learning.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [offline](<https://devfeed.tech/tags/offline.md>), [policy](<https://devfeed.tech/tags/policy.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

Support-Constrained Fitted Q-Iteration (FQI-SC) cautiously improves offline policies for pediatric ECMO by restricting optimization to empirically supported state-action pairs and defaulting to clinician behavior where data are sparse. Evaluated on 78 trajectories spanning over 6,000 hours, it improved the reported Balance Score by 38% versus standard FQI, raised estimated value by 1.28 FQE units versus imitation learning, and reduced disagreement with clinician actions by 71% versus FQI. These are reported offline evaluation results using Fitted Q-Evaluation and behavioral disagreement; the supplied abstract does not establish clinical outcomes.

### Source excerpt

arXiv:2611.17134v1 Announce Type: new Abstract: Managing children on ECMO (Extracorporeal Membrane Oxygenation) requires balancing short-term physiological stability with the long-term goal of preventing neurological injury. Standard reinforcement learning can propose risky, out-of-distribution actions, while imitation learning is limited to reproducing clinician behavior. We introduce Support-Constrained Fitted Q-Iteration (FQI-SC), an offline reinforcement learning framework that cautiously improves upon expert behavior by restricting optimization to empirically supported state-action pairs and defaulting to the clinician baseline in low-data regions. On 78 pediatric trajectories totaling over 6,000 hours of clinical data, we evaluate our framework using Fitted Q-Evaluation (FQE) and a behavioral disagreement. Additionally, we formalize the performance-safety trade-off by introducing a unified Balance Score ($S(\pi)$). Our empirical evaluation shows that support-constrained policy improves the Balance Score by 38\% relative to FQI, improves estimated value by 1.28 FQE units over imitation learning, and reduces disagreement with clinician actions by 71\% relative to FQI.

## Privacy-Preserving Range Counting over Local Graph Statistics

DevFeed: [Privacy-Preserving Range Counting over Local Graph Statistics](<https://devfeed.tech/articles/privacy-preserving-range-counting-over-local-graph-statistics-132701.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16483>)

Author: Gengda Zhao, Yizhang He, Dong Wen, Kai Wang, Ruijia Wu, Jianwei Wang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Differential Privacy](<https://devfeed.tech/topics/differential-privacy.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversary](<https://devfeed.tech/tags/adversary.md>), [aes](<https://devfeed.tech/tags/aes.md>), [bias](<https://devfeed.tech/tags/bias.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [graph](<https://devfeed.tech/tags/graph.md>), [local](<https://devfeed.tech/tags/local.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [range](<https://devfeed.tech/tags/range.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

The paper proposes an edge-local-DP framework for estimating ranges of vertex statistics while limiting disclosure of sensitive graph edges: each vertex randomizes its report before sending it to an untrusted curator. It combines scalar statistic estimators, directly privatized range indicators, and adaptive estimator selection. Its Bounded Conditional Range Inversion (BCRI) uses reusable bounded geometric reports for unbiased range estimates; a boundary-conditioned version has expected L2 loss no greater than the unbounded inverse, and histogram preprocessing enables constant-time queries. The reports also yield a nonnegative unbiased variance estimate. Adaptive Estimator Selection (AES) privately predicts bias, variance, and covariance to combine BCRI with a conditional estimator from the same transcript. The abstract reports experiments showing accuracy, efficiency, and workload scalability across diverse graph tasks.

### Source excerpt

arXiv:2611.16483v1 Announce Type: new Abstract: Range counting over vertex statistics supports contact analysis, community profiling, and reputation screening. However, releasing exact range counts can disclose sensitive edges to an adversary with background knowledge. We study this problem under edge local differential privacy (edge LDP), where each vertex randomizes its report before sending it to an untrusted curator. We develop a layered framework that connects statistic estimation, direct range estimation, and adaptive estimator selection. Statistic-estimation methods provide complementary scalar estimators with explicit loss analyses, while indicator-estimation methods privatize range indicators directly. We next propose Bounded Conditional Range Inversion (BCRI), which derives unbiased range estimates from reusable bounded geometric reports through an explicit local inverse. Its boundary-conditioned estimator achieves no larger expected L2 loss than the unbounded inverse, and histogram preprocessing supports constant-time queries. The same reports also provide a nonnegative, unbiased variance estimate for every answer. At the center of the framework, Adaptive Estimator Selection (AES) privately predicts bias, variance, and covariance to combine BCRI with a conditional estimator from the same transcript. Experiments on diverse real-world graph tasks demonstrate the accuracy, efficiency, and workload scalability of the proposed framework.

## Daily Emergency Call Volume Forecasting using Univariate and Multivariate TimeGPT

DevFeed: [Daily Emergency Call Volume Forecasting using Univariate and Multivariate TimeGPT](<https://devfeed.tech/articles/daily-emergency-call-volume-forecasting-using-univariate-and-multivariate-timegpt-125419.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11613>)

Author: Feriel Fass, Djemel Ziou

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [mlp](<https://devfeed.tech/topics/mlp.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [application](<https://devfeed.tech/tags/application.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [best](<https://devfeed.tech/tags/best.md>), [better](<https://devfeed.tech/tags/better.md>), [boosting](<https://devfeed.tech/tags/boosting.md>), [call](<https://devfeed.tech/tags/call.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [daily](<https://devfeed.tech/tags/daily.md>), [data](<https://devfeed.tech/tags/data.md>), [emergency](<https://devfeed.tech/tags/emergency.md>), [error](<https://devfeed.tech/tags/error.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [grid](<https://devfeed.tech/tags/grid.md>), [large](<https://devfeed.tech/tags/large.md>), [long-short-term-memory](<https://devfeed.tech/tags/long-short-term-memory.md>), [volume](<https://devfeed.tech/tags/volume.md>)

### AI overview

On 2,101 daily emergency call observations, TimeGPT did not outperform simpler established methods overall: SARIMAX performed best in the univariate setting, while Poisson regression and Negative Binomial regression performed best in the multivariate setting. The study compared one-step-ahead and seven-day forecasts, adding temporal, weather, demographic, and socioeconomic variables in multivariate models. It used chronological training, validation, and test splits, with validation-based grid search and early stopping when applicable. The findings suggest large-scale pretraining did not guarantee better accuracy for this application; the excerpt does not provide metric values or results for each forecast horizon.

### Source excerpt

arXiv:2611.11613v1 Announce Type: new Abstract: This study investigates the potential of TimeGPT, a pretrained foundation model for timeseries, for forecasting daily emergency call volumes. Real-world emergency call data, comprising 2,101 daily observations, are used to evaluate both univariate and multivariate forecasting settings under one-step-ahead and seven-day multi-step-ahead scenarios. The multivariate forecasting incorporates temporal, meteorological, demographic, and socioeconomic variables. A chronological training, validation, and test split is adopted to prevent information leakage, while hyperparameters are selected using the validation set through grid search and early stopping when applicable. Forecasting performance is evaluated using multiple error metrics. Its forecasting performance is compared with several established forecasting approaches, including Seasonal Autoregressive Integrated Moving Average with Exogenous Regressors (SARIMAX), Poisson regression (PR), Negative Binomial regression (NBR), Extreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Long Short-Term Memory (LSTM). The results show that SARIMAX achieves the best overall performance in the univariate setting. In the multivariate setting, NBR and PR provide best performance. These findings indicate that the large-scale pretraining of TimeGPT do not necessarily translate into superior forecasting accuracy for emergency call volumes. Frugal methods trained solely on the available data remain better for this application.

## Confronting Uncertainty in Mechanism-AI Coupled Dynamical Systems via Optimal Transport and Information Geometry: Part I. Distribution Learning

DevFeed: [Confronting Uncertainty in Mechanism-AI Coupled Dynamical Systems via Optimal Transport and Information Geometry: Part I. Distribution Learning](<https://devfeed.tech/articles/confronting-uncertainty-in-mechanism-ai-coupled-dynamical-systems-via-optimal-transport-and-information-geometry-part-i-distribution-learning-125220.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10563>)

Author: Pengfei Song, Yanni Xiao, Shuangshuang Yin

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [bayesian-inference](<https://devfeed.tech/tags/bayesian-inference.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [math-ds](<https://devfeed.tech/tags/math-ds.md>), [math-mg](<https://devfeed.tech/tags/math-mg.md>), [math-pr](<https://devfeed.tech/tags/math-pr.md>), [numerical-methods](<https://devfeed.tech/tags/numerical-methods.md>), [optimal-transport](<https://devfeed.tech/tags/optimal-transport.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [uncertainty-quantification](<https://devfeed.tech/tags/uncertainty-quantification.md>)

### AI overview

The paper formulates uncertainty quantification for mechanism-AI coupled dynamical systems as minimizing an energy functional over probability densities with a Riemannian metric, evaluating loss through a differential equation. It describes gradient flows under Wasserstein and Fisher-Rao metrics and numerical methods based on abstract adjoint sensitivity analysis. The formulation connects Bayesian inference methods for dynamical systems; numerical experiments compare posterior approximation and predictive uncertainty under a common computational budget. The abstract also frames the approach geometrically in relation to dynamical modeling and the evolution of heterogeneity.

### Source excerpt

arXiv:2611.10563v1 Announce Type: new Abstract: Mechanism-AI coupled dynamical systems are increasingly used to study complex biological systems. However, uncertainty associated with noisy, sparse, and partially observed data limits their reliability. This paper presents a systematic mathematical formulation for treating these uncertainties from the perspective of optimal transport and information geometry. We formulate the uncertainty quantification problem as the minimization of an energy functional on a probability density space equipped with a Riemannian metric, with the loss evaluated through a differential equation, and give a unified description of the gradient flows of the energy functional under specific metrics, such as the Wasserstein and Fisher-Rao metrics. Furthermore, we present corresponding numerical methods for solving these gradient flows using abstract adjoint sensitivity analysis. The formulation connects a range of Bayesian inference methods for dynamical systems; numerical experiments compare posterior approximation and predictive uncertainty under a common computational budget. Moreover, it also provides a geometric perspective on dynamical modeling and the study of heterogeneity evolution.

## Information-Theoretic Generalization for Set-Input Optimization-Valued Objectives

DevFeed: [Information-Theoretic Generalization for Set-Input Optimization-Valued Objectives](<https://devfeed.tech/articles/information-theoretic-generalization-for-set-input-optimization-valued-objectives-125181.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10362>)

Author: Futoshi Futami, Masahiro Fujisawa

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [concept-learning](<https://devfeed.tech/topics/concept-learning.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [batch](<https://devfeed.tech/tags/batch.md>), [certificates](<https://devfeed.tech/tags/certificates.md>), [convex](<https://devfeed.tech/tags/convex.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [index](<https://devfeed.tech/tags/index.md>), [information](<https://devfeed.tech/tags/information.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [risk](<https://devfeed.tech/tags/risk.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [selector](<https://devfeed.tech/tags/selector.md>), [square](<https://devfeed.tech/tags/square.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper develops information-theoretic generalization bounds for batch-level criteria that optimize over auxiliary witnesses, thresholds, or reweightings, including multicalibration, CVaR, and Cressie-Read DRO. In the balanced supersample setting, the expected optimized validation-training gap is bounded using an objective-dependent sub-Gaussian parameter and the square root of prediction-based conditional mutual information between the learned prediction table and selector vector. The authors combine this with fixed-predictor population-validation comparisons and establish sub-Gaussianity through stability and convex L2-Lipschitz certificates, including a stability-based single-index refinement. The supplied abstract does not give the concrete bound formulas.

### Source excerpt

arXiv:2611.10362v1 Announce Type: cross Abstract: Modern machine learning is increasingly evaluated using criteria such as multicalibration, CVaR, and Cressie-Read DRO, which target subgroup reliability, tail risk, and distributional robustness rather than averages of fixed per-example losses. Existing information-theoretic generalization bounds for pointwise losses do not directly apply, because these empirical criteria depend on the entire batch and involve a post-hoc auxiliary optimization over witnesses, thresholds, or reweightings. We formalize this structure as set-input optimization-valued objectives. In the balanced supersample setting, we bound the expected optimized validation-training gap in terms of an objective-dependent sub-Gaussian parameter and the square root of the prediction-based conditional mutual information (fCMI) between the learned supersample prediction table and the selector vector. Combining this with fixed-predictor population-validation comparisons yields unified algorithm-dependent generalization bounds. We verify sub-Gaussianity of the selector gap using stability and convex L2-Lipschitz certificates, obtaining concrete bounds for multicalibration, CVaR, and Cressie-Read DRO, with a stability-based single-index refinement.

## LoStBench: Benchmarking Longitudinal Studies Data Integration

DevFeed: [LoStBench: Benchmarking Longitudinal Studies Data Integration](<https://devfeed.tech/articles/lostbench-benchmarking-longitudinal-studies-data-integration-125663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10242>)

Author: Pratik Pokharel, Oliver Kennedy, Abhi Ramtel, Poonam Kumari, Jeff Good, Katarzyna Kordas, Marianthi Markatou, Andrew Talal

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [matching](<https://devfeed.tech/tags/matching.md>)

### AI overview

Longitudinal surveys change over time, leaving heterogeneous data that must be matched before analysis. LoStBench addresses this challenge with a medium-scale, subject-matter-expert-annotated benchmark built from the nearly eight-decade American National Election Studies dataset and its canonical expert-curated schema matching. The paper adapts existing schema matching algorithms and evaluates state-of-the-art methods, noting that long prose survey questions provide a stronger matching signal than the short attribute names and data instances many tools rely on. The study reports algorithm limitations, proposes improvements for some, and identifies remaining challenges.

### Source excerpt

arXiv:2611.10242v1 Announce Type: new Abstract: A longitudinal study observes the same subjects or variables, repeatedly over an extended period of time, allowing researchers to examine changes, trends, and relationships as they develop. Such studies typically evolve over their life cycles, as improved understanding of the study's domain, or external factors trigger the need for revisions. The resulting heterogeneity among collected data must be resolved before the survey's results may be analyzed. Although this problem is a form of schema matching, it is a form of schema matching that differs significantly from prior workloads explored by the database community. For instance, rather than the short attribute names and data instances that many schema matching tools rely on, a stronger signal in longitudinal studies arises from longer prose questions that these tools are less equipped to handle. This paper documents the joint efforts of a team of computer scientists and domain experts to adapt existing schema matching algorithms to our new use case of longitudinal data integration. We present a case study in the context of a representative workload: The American National Election Studies (ANES) dataset, a nearly eight-decade survey with publicly available data and a canonical domain-expert curated schema matching. Using ANES, we develop LoStBench (Longitudinal Studies Benchmark for data integration), a medium-scale, subject-matter-expert-annotated benchmark. Using LoStBench, we conduct a comprehensive empirical study evaluating state-of-the-art schema matching algorithms, and expose limitations of these algorithms when applied in the context of longitudinal studies. We propose improvements that address some of these limitations, and identify key challenges to motivate future data integration work to address the others.

## Motivators and Demotivators of Integrating Large Language Models in Software Engineering Education: An Exploratory Familiarity Allocation Framework

DevFeed: [Motivators and Demotivators of Integrating Large Language Models in Software Engineering Education: An Exploratory Familiarity Allocation Framework](<https://devfeed.tech/articles/motivators-and-demotivators-of-integrating-large-language-models-in-software-engineering-education-an-exploratory-familiarity-allocation-framework-127185.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09393>)

Author: Maryam Khan, Jussi Kasurinen, Muhammad Azeem Akbar, Estefan\\'ia Mart\\'in-Barroso

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Learning](<https://devfeed.tech/topics/learning.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [ai](<https://devfeed.tech/tags/ai.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bayes](<https://devfeed.tech/tags/bayes.md>), [critical-thinking](<https://devfeed.tech/tags/critical-thinking.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [education](<https://devfeed.tech/tags/education.md>), [framework](<https://devfeed.tech/tags/framework.md>), [governance](<https://devfeed.tech/tags/governance.md>), [intellectual-property](<https://devfeed.tech/tags/intellectual-property.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [logistic-regression](<https://devfeed.tech/tags/logistic-regression.md>), [plagiarism](<https://devfeed.tech/tags/plagiarism.md>), [problem-solving](<https://devfeed.tech/tags/problem-solving.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [software-engineering-education](<https://devfeed.tech/tags/software-engineering-education.md>), [survey](<https://devfeed.tech/tags/survey.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

A survey of 126 stakeholders rated 19 motivators and demotivators of integrating large language models into software engineering education. Programming and debugging assistance, personalized learning, and AI as a learning partner ranked highest among motivators; plagiarism and intellectual property concerns, over-reliance, and reduced critical thinking ranked highest among demotivators. Naive Bayes and Logistic Regression each achieved 0.461 accuracy, below the majority-class baseline, so their probabilities served as exploratory relative scores in a Genetic Algorithm rather than individual predictions. Global allocation emphasized integrity and governance factors, while local allocation showed additional pedagogical and risk patterns. These are exploratory scenarios, not validated priorities or evidence of educational effectiveness; classroom and institutional validation is needed.

### Source excerpt

arXiv:2605.09393v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly influencing software engineering practice and education, yet little research has examined their systematic integration using stakeholder evidence and quantitative decision-support models. This study presents an exploratory familiarity-allocation framework for analyzing motivating and demotivating factors in LLM integration in software engineering education. Building on our earlier literature-based taxonomy, we evaluated 19 factors (nine motivators and ten demotivators) through a survey of 126 stakeholders from multiple countries. Naive Bayes and Logistic Regression were used to examine associations between factor ratings and high self-reported LLM familiarity. Both models reached an accuracy of 0.461, below the majority-class baseline, so their probabilities were used as exploratory relative scoring functions rather than individual-level predictions. These estimates were fed into a Genetic Algorithm with a defined allocation score to support Global Factor Allocation (GC), Local Factor Allocation (LC), and category-level Delta-Fitness analyses. The most highly rated motivators were Programming Assistance and Debugging Support, Personalized and Adaptive Learning, and AI as a Learning Partner. The most highly rated demotivators were Plagiarism and Intellectual Property Concerns, Over-Reliance on AI in Learning, and Reduced Critical Thinking and Problem-Solving. Under the fitted models and allocation assumptions, GC assigned higher levels to integrity- and governance-related factors, while LC revealed further pedagogical and risk-related patterns within categories. These configurations are exploratory scenarios, not validated implementation priorities or evidence of educational effectiveness, and classroom and institutional validation is needed before the framework can guide practice.

## Beyond Node Attributes: A Controlled Evaluation and Ablation of Local Interaction Statistics for Ethereum Phishing Detection

DevFeed: [Beyond Node Attributes: A Controlled Evaluation and Ablation of Local Interaction Statistics for Ethereum Phishing Detection](<https://devfeed.tech/articles/beyond-node-attributes-a-controlled-evaluation-and-ablation-of-local-interaction-statistics-for-ethereum-phishing-detection-125350.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11263>)

Author: Thanh Dat Tran, Duy Loi Duong, Duy Luu Hoang, Minh Huy Tran, Vo Minh Thu Le, Nhat Tung Le

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Ethereum](<https://devfeed.tech/topics/ethereum.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [Statistics](<https://devfeed.tech/topics/statistics.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [edge](<https://devfeed.tech/tags/edge.md>), [ethereum](<https://devfeed.tech/tags/ethereum.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [false-positive-rate](<https://devfeed.tech/tags/false-positive-rate.md>), [graph](<https://devfeed.tech/tags/graph.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [phishing](<https://devfeed.tech/tags/phishing.md>), [phishing-detection](<https://devfeed.tech/tags/phishing-detection.md>), [static](<https://devfeed.tech/tags/static.md>), [statistics](<https://devfeed.tech/tags/statistics.md>), [temporal](<https://devfeed.tech/tags/temporal.md>)

### AI overview

On one static, transductive benchmark of 2,763 labeled Ethereum accounts, adding 12 unlabeled local interaction statistics to HEIG-Phish's 14 source features improved test Macro-F1 across six model families. The controlled comparison held partitions, configurations, seeds, thresholds, and sampled neighborhoods constant; all model pairs improved on each of 20 partitions, with lower false-positive rates and corrected 95% confidence intervals above zero. A Random Forest ablation attributed most of the gain to the incoming transfer-edge sum; it did not establish an added benefit from the other 11 statistics once that feature was included. The findings show stability across re-partitions of this benchmark, not temporal or cross-dataset generalization.

### Source excerpt

arXiv:2611.11263v1 Announce Type: new Abstract: Methods for detecting phishing accounts on Ethereum are usually evaluated as complete pipelines that differ in features, architecture, and training protocol, which makes the contribution of interaction information hard to isolate. This paper evaluates the Unlabeled Feature Extension (UFE), a block of 12 local interaction statistics computed from each account's incoming and outgoing call and transfer edges without reading class labels. UFE is appended to the 14 source features of HEIG-Phish and tested with six model families: Logistic Regression, Random Forest, LightGBM, Graph Attention Network, GraphSAGE, and Meta-IFD. Within each pair of Baseline and Plus variants, the data partition, model configuration, random seed, decision threshold, and sampled neighborhoods are identical; only the input feature block changes. Across 20 predetermined stratified partitions of the same 2,763 labeled externally owned accounts, UFE increased test Macro-F1 by 1.0 to 9.1 percentage points across the six models. Every pair improved on all 20 partitions, all Nadeau-Bengio corrected 95% confidence intervals lie above zero, and the false-positive rate decreased for all six models. An exact-edge Random Forest ablation over 10 predetermined partitions associates most of the gain with the incoming transfer-edge sum attribute; once this feature is present, an incremental contribution of the remaining 11 UFE columns was not established. The results support reporting feature-augmented baselines alongside graph models, but they show stability under re-partitioning of one static, transductive benchmark rather than temporal or cross-dataset generalization.

## Ensemble Distributionally Robust Bayesian Optimisation

DevFeed: [Ensemble Distributionally Robust Bayesian Optimisation](<https://devfeed.tech/articles/ensemble-distributionally-robust-bayesian-optimisation-123134.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.07565>)

Author: Tigran Ramazyan, Denis Derkach

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [behaviour](<https://devfeed.tech/tags/behaviour.md>), [computer](<https://devfeed.tech/tags/computer.md>), [context](<https://devfeed.tech/tags/context.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [model](<https://devfeed.tech/tags/model.md>), [novel](<https://devfeed.tech/tags/novel.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [results](<https://devfeed.tech/tags/results.md>), [robust](<https://devfeed.tech/tags/robust.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [under](<https://devfeed.tech/tags/under.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

The paper proposes Ensemble Distributionally Robust Bayesian Optimisation for zeroth-order optimisation when the context distribution is uncertain. It uses an ensemble surrogate to reduce reliance on any single model and is reported to remain computationally tractable with continuous context. The authors report sublinear regret bounds that improve on current state-of-the-art results, with empirical behaviour aligned with those guarantees. The supplied text is an abstract excerpt and does not describe the algorithm's implementation or experiments in detail.

### Source excerpt

arXiv:2605.07565v3 Announce Type: replace Abstract: We study zeroth-order optimisation under context distributional uncertainty, a setting commonly tackled using Bayesian optimisation (BO). A prevailing strategy to make BO more robust to the complex and noisy nature of data is to employ an ensemble as the surrogate model, thereby mitigating the weaknesses of any single model. In this study, we propose a novel algorithm for Ensemble Distributionally Robust Bayesian Optimisation that remains computationally tractable while managing continuous context. We obtain theoretical sublinear regret bounds, improving current state-of-the-art results. We show that our method's empirical behaviour aligns with its theoretical guarantees.

[Next page](<https://devfeed.tech/topics/data-science-and-engineering.md?cursor=WyIyMDI2LTExLTA5VDA1OjAwOjAwKzAwOjAwIiwgImNmMDBlYmU1LWNjZTgtNDViZi1iY2JjLWQ2MWYzMjNiMzdjMyJd>)