# bert

BERT is a language representation model that pre-trains bidirectional representations from Transformers.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## RePAIR: Predictive Self-Supervised Representation Learning in Chess

DevFeed: [RePAIR: Predictive Self-Supervised Representation Learning in Chess](<https://devfeed.tech/articles/repair-predictive-self-supervised-representation-learning-in-chess-144905.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.11860>)

Author: Christoph Koller, Johannes F\\"urnkranz, Timo Bertram

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Self-supervised learning](<https://devfeed.tech/topics/self-supervised-learning.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [bert](<https://devfeed.tech/topics/bert.md>)

Tags: [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [bert](<https://devfeed.tech/tags/bert.md>), [chess](<https://devfeed.tech/tags/chess.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [representation-learning](<https://devfeed.tech/tags/representation-learning.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [space](<https://devfeed.tech/tags/space.md>)

### AI overview

RePAIR learns compact representations of consecutive chess positions by masking large portions of latent-state sequences, then using a lightweight predictor to repair the gaps in a lower-dimensional embedding space. The authors report that its encoder clusters meaningful chess concepts, and that reconstructing masked board states captures piece movements without reinforcement learning. They also use trajectories through the representation space to inspect games. These findings are reported for chess; the supplied excerpt does not establish performance on other sequential data.

### Source excerpt

arXiv:2606.11860v2 Announce Type: replace Abstract: In this paper, we introduce Representation Prediction via Autoencoding using Iterative Refinement (RePAIR) - a novel self-supervised representation learning architecture that synthesizes Masked Autoencoders (MAE), Joint Embedding Predictive Architectures (JEPA), and Bidirectional Encoder Representations from Transformers (BERT). We demonstrate how it can be used to encode objects in sequential data like consecutive chess positions into compact yet meaningful representations. The basic principle of the architecture is to mask large portions of a sequence of latent states, similar to BERT and MAE. Then, we apply a lightweight Predictor to the latent representations that repairs gaps in the sequence in a lower-dimensional embedding space akin to JEPA. Our experiments in the domain of chess show that the Encoder refines the board representations such that meaningful chess concepts emerge clustered in the latent space. Furthermore, reconstructions of the masked board states show that the model is able to reason about the piece movements without relying on costly reinforcement learning methods. Lastly, we find that the resulting representation space allows for quick and intuitive dissections of chess games by observing the game path trajectories in this semantically rich space.

## Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection

DevFeed: [Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection](<https://devfeed.tech/articles/implementing-binary-transformers-with-approximate-memories-with-permutation-invariance-enabled-selective-pies-protection-142645.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23358>)

Author: Zhen Gao, Xiao Yin, Kexin Liang, Pedro Reviriego, Shanshan Liu, Fabrizio Lombardi

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [bert](<https://devfeed.tech/topics/bert.md>), [ecc ram](<https://devfeed.tech/topics/ecc-ram.md>)

Tags: [bert](<https://devfeed.tech/tags/bert.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [devices](<https://devfeed.tech/tags/devices.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [error](<https://devfeed.tech/tags/error.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [permutation](<https://devfeed.tech/tags/permutation.md>), [permutations](<https://devfeed.tech/tags/permutations.md>), [power](<https://devfeed.tech/tags/power.md>), [protection](<https://devfeed.tech/tags/protection.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [recover](<https://devfeed.tech/tags/recover.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The study evaluates how memory bit errors affect binary Transformers, using BERT as a case study. It reports large accuracy losses only at high bit error rates, such as 5% or 10%. The proposed PIES scheme uses Transformers' invariance to row and column permutations to identify critical weights and selectively protect them. Simulations indicate that PIES recovers most performance loss at error rates up to 10%, with substantially lower overhead than traditional error correction schemes. These findings suggest binary Transformers may be feasible with approximate memory and aggressive voltage scaling on low-power edge devices.

### Source excerpt

arXiv:2611.23358v1 Announce Type: cross Abstract: Quantization is an effective technique to reduce memory storage, computation complexity and power for Transformers, and the binarization of the weights is important for meeting these requirements. Approximate memory can further reduce the power for memory, but it also introduces errors. Current research for binary Transformers focuses on task performance, but the model dependable operation against errors on the memory has not been well studied. In this paper, we first evaluate the dependability of binary Transformers with the Bidirectional Encoder Representations from Transformers (BERT) as a case study. A large accuracy loss occurs only at high Bit Error Rates (e.g. BER = 5% or 10%). Then we propose a novel Permutation Invariance Enabled Selective (PIES) scheme to protect the binary weights at very low overhead. This scheme exploits the invariance of Transformers to permutations in the rows and columns to group the critical weights and perform selective protection on those weights. Simulation results show that PIES can recover most of the performance loss caused by bit errors at BERs as high as 10%, at an overhead significantly lower than traditional error correction schemes. These results confirm the feasibility of running binary Transformers using approximate memory with aggressive voltage scaling on resource-limited low power edge devices. The code for PIES is released on github (https://github.com/YangZhutian/binary-transformer-prefix-repair).

## Zero-Dimensional Attention Topology for Linguistic Acceptability

DevFeed: [Zero-Dimensional Attention Topology for Linguistic Acceptability](<https://devfeed.tech/articles/zero-dimensional-attention-topology-for-linguistic-acceptability-141765.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22312>)

Author: Saeed AlMarri (Khalifa University)

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [bert](<https://devfeed.tech/tags/bert.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [persistent-homology](<https://devfeed.tech/tags/persistent-homology.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [topology](<https://devfeed.tech/tags/topology.md>)

### AI overview

The fixed H₀ attention representation did not show a predictive advantage over simpler attention statistics for linguistic acceptability. It summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. In the retrospective evaluation, its out-of-domain Matthews correlation was 0.438, versus 0.508 for attention statistics and 0.545 for BERT. The analysis found that the representation retains token-count information while discarding some directional attention differences. The authors also report that uncertainty estimates from resampling BERT predictions depend on the sampling unit. Attention-feature scores are retained aggregate measurements, while BERT predictions were rescored and length controls refitted from accompanying records.

### Source excerpt

arXiv:2611.22312v1 Announce Type: cross Abstract: Topological summaries of Transformer attention provide sentence representations for linguistic acceptability classification. We examine a zero-dimensional persistent-homology ($H_0$) representation that summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. The retrospective evaluation uses 8,517 downstream training sentences and 521 in-domain and 497 out-of-domain CoLA validation sentences after data-integrity filtering. The recorded out-of-domain Matthews correlation coefficients are 0.438 for the topological representation and 0.508 for attention statistics, compared with 0.545 for BERT. Attention-feature results are retained aggregate measurements; BERT predictions are rescored and length controls are refitted from the accompanying records. Analytical results show that the representation retains token-count information and discards a class of directional attention differences. Source resampling of BERT predictions also demonstrates the sensitivity of uncertainty estimates to the sampling unit. The reported results do not demonstrate a predictive advantage for this fixed topological representation over simpler attention summaries. They identify representation properties and evaluation controls relevant to subsequent studies of attention topology.

## A Comparative Evaluation of BERT-Based and LLM-Based Models for Indic Sentence Similarity

DevFeed: [A Comparative Evaluation of BERT-Based and LLM-Based Models for Indic Sentence Similarity](<https://devfeed.tech/articles/a-comparative-evaluation-of-bert-based-and-llm-based-models-for-indic-sentence-similarity-138863.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19356>)

Author: Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary, Pranav Kulkarni, Raviraj Joshi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bert](<https://devfeed.tech/tags/bert.md>), [compare](<https://devfeed.tech/tags/compare.md>), [construction](<https://devfeed.tech/tags/construction.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [development](<https://devfeed.tech/tags/development.md>), [difference](<https://devfeed.tech/tags/difference.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [embeddinggemma](<https://devfeed.tech/tags/embeddinggemma.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [indic](<https://devfeed.tech/tags/indic.md>), [llm](<https://devfeed.tech/tags/llm.md>), [models](<https://devfeed.tech/tags/models.md>), [similarity](<https://devfeed.tech/tags/similarity.md>)

### AI overview

Across the evaluated Indic sentence similarity benchmarks, the top BERT-based encoder outperformed the top LLM-based embedding model, though the gap differed by benchmark. On translated STS Benchmark development sets in ten Indic languages, IndicSBERT led in every language (macro-average Pearson correlation 0.843); BGE-M3 scored 0.792, while Harrier-270M was the strongest LLM-based model at 0.751. On the native Marathi MahaSTS test set, IndicSBERT scored 0.901 and Bhasha-Embed-v0 0.900; EmbeddingGemma led LLM-based models at 0.871. The results do not show uniform gains from LLM-derived embeddings and point to language coverage, task supervision, and benchmark construction as relevant evaluation factors.

### Source excerpt

arXiv:2611.19356v1 Announce Type: cross Abstract: The relative effectiveness of BERT-based encoders and large language model (LLM)-based embeddings for Indic semantic textual similarity remains insufficiently characterized. We compare eight BERT-based and five LLM-based models on the 1,500-pair STS Benchmark development set translated into ten Indic languages and on the native Marathi MahaSTS test set. IndicSBERT achieves the highest Pearson correlation in all ten translated language sets, with a macro-average of 0.843, followed by BGE-M3 at 0.792. Harrier-270M is the strongest LLM-based model on translated STS-B, with a macro-average of 0.751. On MahaSTS, IndicSBERT and Bhasha-Embed-v0 achieve Pearson correlations of 0.901 and 0.900, respectively, while EmbeddingGemma leads the LLM-based models at 0.871. The highest-scoring BERT-based model thus exceeds the highest-scoring LLM-based model in both settings, although the magnitude of the difference varies across benchmarks. These findings indicate that LLM-derived embeddings do not uniformly improve Indic sentence similarity over established encoders and motivate evaluation that accounts for language coverage, task supervision, and benchmark construction.

## Does Deep Knowledge Tracing Learn Semantic Relations Between Knowledge Components?

DevFeed: [Does Deep Knowledge Tracing Learn Semantic Relations Between Knowledge Components?](<https://devfeed.tech/articles/does-deep-knowledge-tracing-learn-semantic-relations-between-knowledge-components-133111.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16323>)

Author: Kia Karbasi, Mahdi Saiedi, Kevin Hong, Gregory Pottie

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [bert](<https://devfeed.tech/topics/bert.md>)

Tags: [2017](<https://devfeed.tech/tags/2017.md>), [align](<https://devfeed.tech/tags/align.md>), [area](<https://devfeed.tech/tags/area.md>), [bert](<https://devfeed.tech/tags/bert.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [components](<https://devfeed.tech/tags/components.md>), [cosine](<https://devfeed.tech/tags/cosine.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deep](<https://devfeed.tech/tags/deep.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fine](<https://devfeed.tech/tags/fine.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [information](<https://devfeed.tech/tags/information.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [learn](<https://devfeed.tech/tags/learn.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

DKT's learned relations between knowledge components align only weakly and inconsistently with text-derived semantic relations, and adding pretrained text representations does not consistently improve on standard DKT. The study tests nine representation pipelines: it compares DKT relation graphs with cosine-similarity structures, then evaluates semantic information in DKT. In reduced-data ASSISTments 2017 experiments, semantic initialization has higher test AUC than the baseline at eight of nine nested sample sizes. KC-supervised fine-tuned BERT shows small positive AUC differences in some matched comparisons, but their stability across runs is not established. The results suggest semantic information alone does not explain DKT's learned relations; the excerpt identifies behavioral, temporal, structural, or psychometric signals as possible influences.

### Source excerpt

arXiv:2611.16323v1 Announce Type: new Abstract: Deep Knowledge Tracing (DKT) is a knowledge tracing (KT) model that updates a shared latent state from sequences of interactions, each pairing a knowledge component (KC) with a response. An interaction with one KC can therefore affect predictions for other KCs. We ask: Are the KC relations learned by DKT semantic? We construct text-derived semantic KC representations using nine pipelines to test whether the conclusions hold across alternative ways of constructing those representations. The first test compares DKT-derived KC relation graphs with semantic relation structures induced by pairwise cosine similarities among the KC representations. Across the evaluated datasets, embedding models, and pipelines, the two structures align only weakly and inconsistently. The second test introduces semantic information into DKT; the pretrained text-derived representations do not consistently outperform the standard DKT baseline. In the reduced-data experiments, semantic initialization yields a higher test area under the ROC curve (AUC) than the baseline at eight of nine nested sample sizes for ASSISTments 2017. KC-supervised fine-tuned BERT produces small positive AUC differences in some matched comparisons, but the study does not establish their stability across runs. These effects remain limited and inconsistent. The findings suggest that behavioral, temporal, structural, or psychometric signals may also shape DKT's learned relations. DKT-derived relations are not primarily semantic: the tested text-derived semantic structures do not strongly recover the DKT-derived relation structure.

## When Handshakes Go Dark: An Adaptive Semantic-Sequential Fusion Framework for Encrypted Traffic Classification

DevFeed: [When Handshakes Go Dark: An Adaptive Semantic-Sequential Fusion Framework for Encrypted Traffic Classification](<https://devfeed.tech/articles/when-handshakes-go-dark-an-adaptive-semantic-sequential-fusion-framework-for-encrypted-traffic-classification-126803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12442>)

Author: Tong Guan, Yunchun Li, Minyuan Xiao, Mingyuan Xia, Yuchen Zhao, Wei Li

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Internet Traffic](<https://devfeed.tech/topics/internet-traffic.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [1-3](<https://devfeed.tech/tags/1-3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [application](<https://devfeed.tech/tags/application.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [availability](<https://devfeed.tech/tags/availability.md>), [based](<https://devfeed.tech/tags/based.md>), [bert](<https://devfeed.tech/tags/bert.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [classification](<https://devfeed.tech/tags/classification.md>), [client](<https://devfeed.tech/tags/client.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [encrypted](<https://devfeed.tech/tags/encrypted.md>), [encrypted-traffic](<https://devfeed.tech/tags/encrypted-traffic.md>), [framework](<https://devfeed.tech/tags/framework.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [handshakes](<https://devfeed.tech/tags/handshakes.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

FLARE addresses encrypted traffic classification when TLS 1.3, QUIC, and ECH hide information that could connect packet bytes to applications. It pre-trains a BERT-based packet encoder to reconstruct masked tokens from field-structured context, freezes that encoder during fine-tuning, then uses a gate to combine protocol-semantic features with directional packet lengths. A flow-level encoder models dependencies between packets for classification. Across eight datasets, including strongly encrypted traffic scenarios, FLARE reportedly outperforms baselines; on QUIC website fingerprinting it exceeds the strongest pre-trained baseline by 10.92 percentage points. The supplied abstract does not provide further methodological details or evaluation caveats.

### Source excerpt

arXiv:2611.12442v1 Announce Type: new Abstract: Encrypted traffic classification is essential for network security monitoring and traffic management. Self-supervised pre-training can learn reusable representations from large-scale unlabeled traffic, but TLS 1.3, QUIC, and privacy mechanisms such as Encrypted Client Hello (ECH) increasingly conceal application, handshake, and service-identifying information. Observable upper-layer bytes are therefore dominated by ciphertext, weakening the link between raw-byte patterns and application semantics and limiting model generalization. Moreover, the availability and discriminative value of observable information vary across protocols, protection profiles, and tasks, complicating classification across diverse scenarios. To address these challenges, we propose FLARE, a Field-aware Learning and Adaptive Representation Fusion framework for encrypted traffic classification. FLARE pre-trains a BERT-based packet encoder through Masked Field Prediction, which reconstructs masked tokens from field-structured context. During downstream fine-tuning, the parameters of the pre-trained packet encoder remain frozen, and a packet-level gated module adaptively combines protocol-semantic representations with directional packet-length representations. A flow-level encoder then models inter-packet dependencies for classification. We evaluate FLARE on eight datasets spanning diverse encrypted-traffic tasks, including CipherSpectrum and VisQUIC for strongly encrypted scenarios. FLARE achieves superior performance over baselines across the eight datasets, exceeding the strongest pre-trained baseline by 10.92 percentage points on QUIC website fingerprinting. These results demonstrate the effectiveness of field-aware pre-training and adaptive semantic-sequential fusion under constrained observability.

## Bridging Structure and Semantics: Knowledge Graph Verification via Triple-to-Text Alignment

DevFeed: [Bridging Structure and Semantics: Knowledge Graph Verification via Triple-to-Text Alignment](<https://devfeed.tech/articles/bridging-structure-and-semantics-knowledge-graph-verification-via-triple-to-text-alignment-120248.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07780>)

Author: Ziang Cui

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [bert](<https://devfeed.tech/topics/bert.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [bert](<https://devfeed.tech/tags/bert.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [errors](<https://devfeed.tech/tags/errors.md>), [graph](<https://devfeed.tech/tags/graph.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [semantic-drift](<https://devfeed.tech/tags/semantic-drift.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [text](<https://devfeed.tech/tags/text.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

GenVer verifies knowledge graph triples by pairing each serialized triple with a generated natural-language representation. A sequence-to-sequence generator is trained on selected triple-text pairs, and a BERT-based classifier combines features from the triple and its generated text. On four FactFB variants built from FB15k with perturbed negative triples, it achieved the highest classification accuracy among the evaluated baselines (88.56%-91.52%). The reported case analysis also found semantic drift and high-confidence errors, so the results suggest generated text can help classification but do not eliminate verification mistakes.

### Source excerpt

arXiv:2611.07780v1 Announce Type: cross Abstract: Semantic accuracy is an important dimension of knowledge graph quality. Graph-based scoring methods exploit relational structure, while textual representations allow classifiers to draw on knowledge encoded by pre-trained language models (PLMs). We present GenVer, a generative verifier that scores triples using both their serialized form and generated natural language representations (NLRs). A triple-text alignment procedure selects training pairs for a sequence-to-sequence generator; a BERT-based classifier then combines features from each triple and its generated NLR. On four FactFB variants, constructed from FB15k using perturbed negative triples, GenVer obtains the highest classification accuracy among the evaluated baselines, ranging from 88.56% to 91.52%. The experiments suggest that generated representations can help triple classification, while the case analysis also exposes semantic drift and high-confidence errors.

## DELBERT-2: Pretrained Fingerprint Language Models for DEL Protein Binder Prediction

DevFeed: [DELBERT-2: Pretrained Fingerprint Language Models for DEL Protein Binder Prediction](<https://devfeed.tech/articles/delbert-2-pretrained-fingerprint-language-models-for-del-protein-binder-prediction-114358.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00281>)

Author: Bing Hu, Sun Sun, Shaik Salman Basha, Anita Layton, Helen Chen

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [bert](<https://devfeed.tech/topics/bert.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fingerprint](<https://devfeed.tech/tags/fingerprint.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [ood](<https://devfeed.tech/tags/ood.md>), [pr](<https://devfeed.tech/tags/pr.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [prioritization](<https://devfeed.tech/tags/prioritization.md>), [self-supervised-learning](<https://devfeed.tech/tags/self-supervised-learning.md>), [virtual](<https://devfeed.tech/tags/virtual.md>)

### AI overview

DELBERT-2 uses a ModernBERT encoder trained with masked language modelling on 2.5 million AIRCHECK DEL molecules to turn sparse molecular fingerprints into token sequences. Across six protein targets and three out-of-distribution tests of chemical novelty, cross-library transfer, and compositional generalization, it improves PR-AUC and NDCG@1000 over LightGBM ensembles and transformers trained from scratch, with larger gains in more stringent settings. At top-100, its enrichment factor is 13.28 ± 3.72 versus 12.36 ± 3.75 without pretraining (7.4% higher; p=0.040), and 13 times random selection. The evidence is limited to the abstract and does not provide further methodological or result details.

### Source excerpt

arXiv:2611.00281v1 Announce Type: new Abstract: We present DELBERT-2, a pretrained fingerprint language model that converts sparse molecular fingerprints into unified token sequences using a ModernBERT encoder trained with masked language modelling on 2.5M molecules from the AIRCHECK DEL corpus. We evaluate across six targets (WDR91, WDR12, SETDB1, PLCZ1, LRRK2, DCAF7) under three out-of-distribution (OOD) protocols: hierarchical cluster splits probe chemical novelty, library splits test cross-library transfer, and building-block splits assess compositional generalization. DELBERT-2 consistently improves PR-AUC and NDCG@1000 relative to LightGBM ensemble baselines and transformers trained from scratch, with the largest gains in stringent OOD regimes. DELBERT-2 achieves 13.28+/-3.72 enrichment factor at top-100 vs. 12.36+/-3.75 for no-pretraining (p=0.040), representing 7.4% improvement and 13x enrichment over random selection. These results demonstrate that fingerprint-centric self-supervised learning effectively improves hit prioritization under distribution shift, enabling practical DEL virtual screening. Code is available at \hyperlink{https://github.com/GenerativeDrugDiscovery/DELBERT-2}{DELBERT-2}.

## GenoBERT: A Language Model for Accurate Genotype Imputation

DevFeed: [GenoBERT: A Language Model for Accurate Genotype Imputation](<https://devfeed.tech/articles/genobert-a-language-model-for-accurate-genotype-imputation-107883.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.00058>)

Author: Lei Huang, Chuan Qiu, Kuan-Jui Su, Anqi Liu, Yun Gong, Weiqiang Lin, Lindong Jiang, Chen Zhao, Meng Song, Jeffrey Deng, Qing Tian, Zhe Luo, Ping Gong, Hui Shen, Chaoyang Zhang, Hong-Wen Deng

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [bert](<https://devfeed.tech/topics/bert.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bias](<https://devfeed.tech/tags/bias.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [genomics](<https://devfeed.tech/tags/genomics.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [panel](<https://devfeed.tech/tags/panel.md>), [performance](<https://devfeed.tech/tags/performance.md>), [q-bio-gn](<https://devfeed.tech/tags/q-bio-gn.md>), [range](<https://devfeed.tech/tags/range.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [sufficient](<https://devfeed.tech/tags/sufficient.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

GenoBERT imputes missing phased genotypes without a reference panel, using a transformer with self-attention to capture short- and long-range linkage disequilibrium. In tests on the Louisiana Osteoporosis Study and 1000 Genomes Project across ancestry groups and 5-50% missingness, it had the highest overall accuracy among the four named baselines: Beagle5.4, SCDA, BiU-Net, and STICI. The abstract reports r² around 0.98 with up to 25% missing and above 0.90 at 50% missing. It also reports consistent gains across ancestries and resilience to small samples and weak LD. LD-decay analyses supported a 128-SNP context window (about 100 Kb) for local correlations. These findings are based on the abstract; it does not provide detailed methods or per-group results.

### Source excerpt

arXiv:2604.00058v2 Announce Type: replace-cross Abstract: Genotype imputation enables dense variant coverage for genome-wide association and risk-prediction studies, yet conventional reference-panel methods remain limited by ancestry bias and reduced rare-variant accuracy. We present Genotype Bidirectional Encoder Representations from Transformers (GenoBERT), a transformer-based, reference-free framework that tokenizes phased genotypes and uses a self-attention mechanism to capture both short- and long-range linkage disequilibrium (LD) dependencies. Benchmarking on two independent datasets including the Louisiana Osteoporosis Study (LOS) and the 1000 Genomes Project (1KGP) across ancestry groups and multiple genotype missingness levels (5-50%) shows that GenoBERT achieves the highest overall accuracy compared to four baseline methods (Beagle5.4, SCDA, BiU-Net, and STICI). At practical sparsity levels (up to 25% missing), GenoBERT attains high overall imputation accuracy ($r^2 approx 0.98$) across datasets, and maintains robust performance ($r^2 > 0.90$) even at 50% missingness. Experimental results across different ancestries confirm consistent gains across datasets, with resilience to small sample sizes and weak LD. A 128-SNP (single-nucleotide polymorphism) context window (approximately 100 Kb) is validated through LD-decay analyses as sufficient to capture local correlation structures. By eliminating reference-panel dependence while preserving high accuracy, GenoBERT provides a scalable and robust solution for genotype imputation and a foundation for downstream genomic modeling.

## Tracing Contextual Token Representations Through Transformer Layers

DevFeed: [Tracing Contextual Token Representations Through Transformer Layers](<https://devfeed.tech/articles/tracing-contextual-token-representations-through-transformer-layers-100885.md>)

Original publisher: [Read original article](<https://hackernoon.com/tracing-contextual-token-representations-through-transformer-layers?source=rss>)

Author: Lian Kim

Published: 2026-10-26T17:22:30Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [vector embeddings in AI](<https://devfeed.tech/topics/vector-embeddings-in-ai.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [mlp](<https://devfeed.tech/topics/mlp.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [Temperature settings in AI](<https://devfeed.tech/topics/temperature-settings-in-ai.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2019](<https://devfeed.tech/tags/2019.md>), [2020](<https://devfeed.tech/tags/2020.md>), [2023](<https://devfeed.tech/tags/2023.md>), [2025](<https://devfeed.tech/tags/2025.md>), [22](<https://devfeed.tech/tags/22.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [absolute](<https://devfeed.tech/tags/absolute.md>), [accounting](<https://devfeed.tech/tags/accounting.md>), [activation](<https://devfeed.tech/tags/activation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [classification](<https://devfeed.tech/tags/classification.md>), [contextual-embeddings](<https://devfeed.tech/tags/contextual-embeddings.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [hackernoon-top-story](<https://devfeed.tech/tags/hackernoon-top-story.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [intent-classification](<https://devfeed.tech/tags/intent-classification.md>), [layers](<https://devfeed.tech/tags/layers.md>), [pythia-160m](<https://devfeed.tech/tags/pythia-160m.md>), [structured](<https://devfeed.tech/tags/structured.md>), [token](<https://devfeed.tech/tags/token.md>), [token-embeddings](<https://devfeed.tech/tags/token-embeddings.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [transformer-models](<https://devfeed.tech/tags/transformer-models.md>)

### AI overview

In a six-context Pythia-160M experiment, the same token `Ġlight` starts from an identical embedding, then shows a within-group distinction by recorded state 3 that persists through intermediate layers. Direction-based separation shrinks at the final state, while raw Euclidean separation grows alongside vector norms; the six selected contexts do not isolate word sense or establish semantic understanding. A separate banking-intent comparison shows interface-level trade-offs: on 320 eight-category messages, JEV had the highest strict accuracy and full label validity but the longest mean client latency; Claude was fastest but produced 22 invalid labels, and a post hoc normalization improved its score without surpassing JEV. The systems use different underlying models, runs, and billing units, so results do not isolate interface effects or establish direct cost or production-latency comparisons. Together, the experiments show why representation metrics and application outputs should be evaluated against the specific task and output contract.

### Source excerpt

Explore two AI experiments investigating contextual token representations, Transformer hidden states, and the accuracy of structured classification outputs.

## Analysing Austrian Parliamentary Speeches by Predicting Party Affiliations

DevFeed: [Analysing Austrian Parliamentary Speeches by Predicting Party Affiliations](<https://devfeed.tech/articles/analysing-austrian-parliamentary-speeches-by-predicting-party-affiliations-99627.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.32515>)

Author: Sarah Sulollari, Vanja Karan, Benjamin Roth

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>)

Tags: [2018](<https://devfeed.tech/tags/2018.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bert](<https://devfeed.tech/tags/bert.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [insights](<https://devfeed.tech/tags/insights.md>), [methodology](<https://devfeed.tech/tags/methodology.md>), [model](<https://devfeed.tech/tags/model.md>), [paper](<https://devfeed.tech/tags/paper.md>), [party](<https://devfeed.tech/tags/party.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [quality](<https://devfeed.tech/tags/quality.md>), [release](<https://devfeed.tech/tags/release.md>), [set](<https://devfeed.tech/tags/set.md>), [speaker](<https://devfeed.tech/tags/speaker.md>), [speech](<https://devfeed.tech/tags/speech.md>)

### AI overview

The paper uses a fine-tuned BERT model to assign each Austrian National Council speech a probability distribution over parties, then aggregates predictions by speaker and party and evaluates their quality. It treats party misassignments as signals for studying speech patterns: parties governing together become more similar, opposition parties move further from governing parties, and members who switch parties usually adapt their speech characteristics only after the switch. The authors release party-affiliation predictions for speeches from 1996 to 2018.

### Source excerpt

arXiv:2610.32515v1 Announce Type: cross Abstract: This paper examines speech characteristics of the Austrian National Council members (CMs). We introduce analysis methodology based on a fine-tuned BERT-Model which assigns to every speech a probability distribution over Austrian parties. These are then aggregated at the speaker and party level. We perform an extensive evaluation to ensure the quality of the generated distributions prior to further analysis. Instead of considering mis-assignments of predicted parties as a performance issue, we leverage them to derive insights about the CMs and political developments. Our findings show that parties that govern together become more similar in their speech characteristics, while, analogously, parties that are in opposition move further away from governing parties. Moreover, CMs who switch party usually adapt their speech characteristics only after the switch occurs. Finally, we publicly release a data set containing party affiliation predictions for the speeches of the Austrian National Council from 1996 to 2018.

## Patronizing and Condescending Language Detection in English and Mandarin

DevFeed: [Patronizing and Condescending Language Detection in English and Mandarin](<https://devfeed.tech/articles/patronizing-and-condescending-language-detection-in-english-and-mandarin-99899.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31808>)

Author: Xinyi Vicky Xiang, Catherine Qingyun Wang, Paula Cort\\'es Lemos

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Language Engineering](<https://devfeed.tech/topics/language-engineering.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [bert](<https://devfeed.tech/tags/bert.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [detection](<https://devfeed.tech/tags/detection.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [english](<https://devfeed.tech/tags/english.md>), [language](<https://devfeed.tech/tags/language.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>)

### AI overview

The paper proposes automated detection of patronizing and condescending language (PCL), which can belittle vulnerable groups and reinforce stereotypes, yet is difficult to identify even for human judges. It compares a classifier using static word embeddings and a convolutional neural network with one using pre-trained BERT embeddings fine-tuned for PCL. The BERT approach performed considerably better, reaching an F1 score of 0.528 on the development set.

### Source excerpt

arXiv:2610.31808v1 Announce Type: new Abstract: Patronizing and condescending language (PCL) is a form of harmful language that belittles and assumes pity for vulnerable groups, reinforcing negative societal stereotypes. However, unlike overt hate speech, it is often difficult to detect, even for human judges. This paper proposes a system for the automated detection of PCL. Our first approach uses static word embeddings and a convolutional neural network for classification, while our second approach uses pre-trained BERT embeddings fine-tuned for the PCL task. Our second approach was considerably more successful, and achieved an F1 score of 0.528 over the development set.

## The Model Was Never the Bottleneck: What Shipping a Text Classifier Into a Government Office Taught

DevFeed: [The Model Was Never the Bottleneck: What Shipping a Text Classifier Into a Government Office Taught](<https://devfeed.tech/articles/the-model-was-never-the-bottleneck-what-shipping-a-text-classifier-into-a-government-office-taught-59359.md>)

Original publisher: [Read original article](<https://hackernoon.com/the-model-was-never-the-bottleneck-what-shipping-a-text-classifier-into-a-government-office-taught?source=rss>)

Author: Vladimir Beskorovainyi

Published: 2026-09-24T06:17:41Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [bert](<https://devfeed.tech/topics/bert.md>), [data](<https://devfeed.tech/topics/data.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [bert](<https://devfeed.tech/tags/bert.md>), [government](<https://devfeed.tech/tags/government.md>), [govtech](<https://devfeed.tech/tags/govtech.md>), [human-in-the-loop-ai](<https://devfeed.tech/tags/human-in-the-loop-ai.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [model](<https://devfeed.tech/tags/model.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [queue](<https://devfeed.tech/tags/queue.md>), [testing](<https://devfeed.tech/tags/testing.md>), [text](<https://devfeed.tech/tags/text.md>), [text-classification](<https://devfeed.tech/tags/text-classification.md>)

### AI overview

A government office built a text classifier to sort citizen appeals and route them to departments. Although BERT was the most accurate model tested, the team shipped a smaller Word2Vec+LSTM system. Load testing showed that human review, not model computation, was the main bottleneck.

### Source excerpt

We shipped a Word2Vec+LSTM over a more accurate BERT, then load testing showed the model was 0.3% of the wall-clock time. The queue was five human reviewers.