# tokenization

Tokenization splits input text into tokens and maps tokens to IDs in a model vocabulary.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use

DevFeed: [The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use](<https://devfeed.tech/articles/the-vocabulary-tax-safely-trimming-unused-tokens-from-llms-for-on-device-use-137513.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19913>)

Author: Sattaya Singkul

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [english](<https://devfeed.tech/tags/english.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [generative](<https://devfeed.tech/tags/generative.md>), [hub](<https://devfeed.tech/tags/hub.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [llms](<https://devfeed.tech/tags/llms.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [safely](<https://devfeed.tech/tags/safely.md>), [tax](<https://devfeed.tech/tags/tax.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trimming](<https://devfeed.tech/tags/trimming.md>)

### AI overview

The study measures the storage cost of unused vocabulary rows across 1,244 Hugging Face generative and speech models. For Thai and English, this "vocabulary tax" is a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B; Gemma 4 E2B incurs 1.93 GB. Restricting rows to the deployment's writing systems is exact, while selecting tokens from user text can remove more but may break BPE merges and alter how new text is split. The authors preserve merge dependencies with "merge closure," adding 5.9% more rows, and use a Good-Turing certificate based only on trimming text to estimate split changes and needed sample size. In reported evaluations, certified trims met a 2% target without test data; across six models and 11 languages, passing trims removed a median 89.7% of rows with bits per character rising by at most 2.0%. The provided evidence is an abstract and does not establish results beyond the described evaluations.

### Source excerpt

arXiv:2611.19913v1 Announce Type: cross Abstract: Open large language models (LLMs) and speech models ship vocabularies of up to 262k tokens, and every user downloads the tables those tokens index (the input embedding, an untied output head and, in recent on-device models, per-layer embeddings), whatever languages they need. We call the bytes in rows a deployment never uses the vocabulary tax and measure it, without downloading weights, for 1,244 generative and speech-recognition models on the Hugging Face Hub. For a Thai and English deployment it is a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B, and 1.93 GB in Gemma 4 E2B: it falls hardest on the small models that edge devices run. Keeping only the rows of the deployment's writing systems is exact. Keeping the tokens seen in the user's own text removes much more, a median 29.0% below 0.5B, but the usual way of doing it breaks BPE tokenizers: it drops the intermediate tokens that merges build on, so new text is split differently 1.9 times as often as the standard check reports. We repair this with merge closure, at a cost of 5.9% more rows, and add a trim certificate: a Good-Turing estimate, from the trimming text alone, of how much new text the trimmed tokenizer will split differently (correlation 0.996 over 300 settings in 22 languages) and of how much more text a target needs. Without test data our method never exceeds a 2% target, while prior keep rules checked on test data exceed it in 9-13 of 44 settings. On six models of 270M to 3B parameters in 11 languages, trimmed models keep their predictions: where the certificate passes, a median 89.7% of rows is removed and bits per character rise by at most 2.0%. The census, the trimming and the certificate are released as an open Python package, and the per-model tax for 41 languages as an open dataset.

## A Backdoor Trigger Is Legible in Fine-Tune Weights

DevFeed: [A Backdoor Trigger Is Legible in Fine-Tune Weights](<https://devfeed.tech/articles/a-backdoor-trigger-is-legible-in-fine-tune-weights-138854.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19304>)

Author: Snigdha Chandan Khilar

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>)

Tags: [backdoor](<https://devfeed.tech/tags/backdoor.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [static](<https://devfeed.tech/tags/static.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trigger](<https://devfeed.tech/tags/trigger.md>)

### AI overview

The authors report a static statistic that can identify a backdoor trigger token from a fine-tuned model checkpoint using weights alone. It requires no forward pass of the suspect model, data, labeled population of poisoned models, or prior knowledge of a backdoor, and ranks one trigger-phrase token highest in the vocabulary. The supplied excerpt gives no further evaluation details.

### Source excerpt

arXiv:2611.19304v1 Announce Type: new Abstract: A backdoored language model behaves normally until a trigger phrase appears in its input. Every published method for recovering such a trigger runs the model. We show the trigger is legible in the weights alone: a static statistic reads a fine-tuned checkpoint with no forward pass of the suspect, no data, no labelled population of poisoned models and no prior knowledge that a backdoor exists, and names one of the tokens of the trigger phrase as the single highest-ranked candidate in the vocabulary.

## Tokenization: Hardness and Bicriteria Approximation Algorithms

DevFeed: [Tokenization: Hardness and Bicriteria Approximation Algorithms](<https://devfeed.tech/articles/tokenization-hardness-and-bicriteria-approximation-algorithms-132091.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15074>)

Author: Atieh Bastan, Xiaoling He, A. Pavan, N. V. Vinodchandran

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [approximation-algorithms](<https://devfeed.tech/tags/approximation-algorithms.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cc](<https://devfeed.tech/tags/cs-cc.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

Binary Tokenization selects up to K substrings from a corpus of binary strings to minimize the total tokens needed to segment each document; the single symbols 0 and 1 are always available. For total corpus length n and maximum document length M, the abstract proves that no polynomial-time n^(1−ε)-approximation exists for any fixed ε in (0,1), unless P=NP. It also gives a polynomial-time bicriteria algorithm with γ=√M log n: it uses at most γK vocabulary entries and at most γ times the optimum token count under budget K. In the storage-budget variant, where selected substring lengths sum to at most B, the stated bicriteria factors are both O(log n log B) for 2≤B<n.

### Source excerpt

arXiv:2611.15074v1 Announce Type: cross Abstract: We study Binary Tokenization: given a corpus of binary strings over $\{0,1\}$ and a vocabulary budget $K$, select at most $K$ substrings (called vocabulary) to minimize the total number of tokens needed to represent the corpus. The primitive symbols $0$ and $1$ are always available, and each document must be segmented separately. Letting $n$ denote the total corpus length and $M$ for the maximum document length, we prove that, for every fixed $\varepsilon\in(0,1)$, Binary Tokenization problem does not have polynomial-time $n^{1-\varepsilon}$-approximation algorithms unless $\mathsf{P}=\mathsf{NP}$. To complement the hardness result, we give a polynomial-time $(\gamma,\gamma)$-bicriteria algorithm, with $\gamma = \sqrt{M}\log n$: The algorithm outputs a vocabulary of size at most $\gamma K$ with number of tokens at most $\gamma {\rm OPT}_K$, where ${\rm OPT}_K$ is the optimal number of tokens under the original budget $K$. We also study a storage-budget variant, in which the sum of the selected substrings' lengths is bounded by $B$. For $2\le B

## Optimal Single-History Tokenization Disambiguation for Provably Secure Linguistic Steganography

DevFeed: [Optimal Single-History Tokenization Disambiguation for Provably Secure Linguistic Steganography](<https://devfeed.tech/articles/optimal-single-history-tokenization-disambiguation-for-provably-secure-linguistic-steganography-133342.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17091>)

Author: Feng Jiang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [steganography](<https://devfeed.tech/tags/steganography.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

Exact, error-free extraction through a text channel constrains a single-history receiver to prefix-free token candidates at each step, requiring it to forgo the stop entropy--the uncertainty over whether a prefix ends at a token. The paper's coin projection fixes those stop decisions using shared randomness, reaches that bound, works with any distribution coder in one model call per token, preserves ordinary sampling, and refines SyncPool for every key. In English, Chinese, and Japanese, ambiguous text is reported at no more than 11 per 1,000 tokens, while stop entropy accounts for 7.2-9.0% of entropy per step. Coin projection retains 91-93% of string entropy for the coder across vocabulary sizes; with the full vocabulary, it converts 2.16-6.85 times as much entropy into message bits as SyncPool and recovers every completed message.

### Source excerpt

arXiv:2611.17091v1 Announce Type: cross Abstract: Provably secure linguistic steganography assumes that the receiver recovers the sender's tokens, but a text channel delivers only text, and ambiguous tokenizations break extraction. Existing fixes change the distribution, synchronize whole pools of tokens at a capacity that collapses with large vocabularies, or track several token histories. We prove that if extraction is exact and error-free across full-support language models, a single-history receiver is confined to a prefix-free candidate set at every step -- even if it reads the entire future text and the sender constrains later steps -- and must therefore give up the stop entropy, the entropy of deciding whether each prefix token ends there. Fixing these stop decisions on the prefix forest with shared randomness yields coin projection: a closed-form scheme that attains the bound, works with any distribution coder in one model call per token, preserves ordinary sampling, and refines SyncPool for every key. Across English, Chinese and Japanese, ambiguous text is rare (at most 11 in 1,000 tokens), yet the stop entropy at word and morpheme prefixes takes 7.2-9.0% of the entropy at every step; SyncPool's further loss comes from space and single-byte UTF-8 hubs that merge subtrees while carrying almost no stop uncertainty. Coin projection keeps 91-93% of the string entropy for the coder at every vocabulary size; with the full vocabulary, it turns 2.16-6.85 times as much entropy into message bits as SyncPool and recovers every completed message.

## One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data

DevFeed: [One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data](<https://devfeed.tech/articles/one-byte-one-token-byteifying-a-27b-hybrid-attention-llm-with-self-generated-data-128738.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13607>)

Author: Liwei Zhang, ZhiMing Tong

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Encoding](<https://devfeed.tech/topics/encoding.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [entries](<https://devfeed.tech/tags/entries.md>), [failure](<https://devfeed.tech/tags/failure.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [llm](<https://devfeed.tech/tags/llm.md>), [one](<https://devfeed.tech/tags/one.md>), [token](<https://devfeed.tech/tags/token.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The study converts Qwen3.8-27B from subword tokens to one-byte tokens by shrinking its vocabulary to 384 entries, keeping other weights unchanged, and continuing training on 4.77 GB of supervised bytes. The two-stage curriculum uses dialogue and behavioral data generated by the original model, plus encyclopedia text for knowledge reading; training took 178.6 hours on eight H200 GPUs. On several benchmarks, performance was statistically close to the original, while scores fell 4-6 points on C-Eval, thinking-mode GSM8K, and non-thinking IFEval. Without thinking, the byte model improved results on the reported character-level tasks, but lost some Chinese character lookup knowledge, which thinking did not restore. Throughput is a major cost: Chinese characters per second was about one quarter of the original, and 64K-byte prefill was 6.5 times slower. The authors also report generation failure modes and fixes.

### Source excerpt

arXiv:2611.13607v1 Announce Type: cross Abstract: Byte-level BPE fragments Chinese text (for Qwen3.8 on Chinese Wikipedia, 3.84 bytes per token on average; 64% of tokens are single characters) and hides spelling from the model. We ask whether an already-trained 27B subword model can be converted directly into a pure byte-level model (one byte, one token), without a patching structure or a local encoder, while keeping most of its ability. Starting from Qwen3.8-27B (48 Gated DeltaNet layers + 16 full-attention layers), we shrink the vocabulary from 248,320 to 384 entries (the 256 byte symbols already in the BPE vocabulary plus special ids), leave every other weight untouched, and continue training in a two-stage curriculum (encoding first, then reading) on 4.77 GB of supervised bytes: all dialogue/behavioral supervision is generated on-policy by the original model itself (character dictionary, word dictionary, general dialogue, coding sessions), and knowledge reading uses encyclopedia text (54% of supervised bytes). Training takes 178.6 h on 8xH200. On the same GPU and serving stack, the byte model's gap to the original falls within the paired bootstrap interval on MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K, and it loses 4-6 points on C-Eval, thinking-mode GSM8K and non-thinking IFEval. Without thinking it leads on every character-level task (CUTE 75.2 -> 84.0, EXECUTE-zho 81.6 -> 93.9, our Chinese character suite 54.4 -> 64.0), but lookup knowledge about Chinese characters such as pronunciation and stroke count drops, and thinking does not recover it. The cost is throughput: per-step speed is identical, but each step yields 3-5x less output; Chinese characters per second is about 1/4 of the original, and prefill at 64K bytes is 6.5x slower. We also document four byte-model failure modes (greedy runaway, sampling breaking UTF-8, ineffective penalty terms on a small vocabulary, repetition) and their fixes.

## A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English

DevFeed: [A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English](<https://devfeed.tech/articles/a-multi-word-tokenizer-for-204-languages-that-needs-fewer-tokens-than-gpt-4o-s-including-on-english-128249.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13059>)

Author: Johin Johny, Ankur Ghosh, Shervin Sam

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Compression](<https://devfeed.tech/topics/compression.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [english](<https://devfeed.tech/tags/english.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [languages](<https://devfeed.tech/tags/languages.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

On FLORES-200, Havon v4 uses fewer tokens than GPT-4o's o200k_base on 161 of 204 languages at the same 200,019-token vocabulary size, including all 25 Indian languages (median 0.74x GPT-4o's count); it also uses 6.1% fewer on English. The tokenizer adds a multi-word SuperBPE training stage to v3. Results vary by language and vocabulary size: gains are largest for English, Hindi and Romance languages, while Khmer, Burmese and Tibetan gain little and lose subword merges. Havon v5 improves code compression over v4, using 1.03-1.22x GPT-4o's tokens versus v4's 1.11-1.29x, while retaining reported gains on Indian languages and English. These are compression measurements; the authors have not evaluated models trained with v4 or v5.

### Source excerpt

arXiv:2611.13059v1 Announce Type: cross Abstract: Languages of India need 1.6 to 14 times as many tokens as English on GPT-4o's tokenizer for the same content, which makes them more expensive and slower to serve. We train tokenizers from scratch on human-written text in about 200 languages. Havon v3 is a byte-level BPE tokenizer with 262,144 tokens and a pre-tokenizer that keeps combining marks with their letters and chunks scripts written without spaces. On FLORES-200 devtest it needs fewer tokens than GPT-4o's o200k_base on all 25 languages of India (median 0.74 of GPT-4o's count) and on 172 of 204 languages, but 7% more on English. Havon v4 adds a second, multi-word training stage (SuperBPE) to v3. At exactly GPT-4o's vocabulary size of 200,019 tokens, v4 needs 6.1% fewer tokens than GPT-4o on English and fewer on 161 of 204 languages and all 25 languages of India (median 0.74); at the same size, v3 needs 9.4% more on English and wins on 133. With 262,144 tokens, v4 needs 9.6% fewer tokens on English and fewer on 182 of 204 languages, and with 524,288 tokens it needs fewer on 203 of 204. The gains are largest for English, Hindi and the Romance languages; scripts without spaces, such as Khmer, Burmese and Tibetan, gain little and lose subword merges. Havon v5 replaces v4's Python-only code sample with permissively licensed code in 15 languages: on code it needs 1.03 to 1.22 times GPT-4o's tokens, against 1.11 to 1.29 for v4, while keeping all 25 languages of India (median 0.74, all significant under a sentence-level bootstrap) and English (0.94). These are compression results only: we have not yet evaluated a model trained with v4 or v5.

## Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South

DevFeed: [Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South](<https://devfeed.tech/articles/inherited-weights-measuring-the-tokenization-premium-and-the-sovereignty-cost-of-language-models-for-the-global-south-128312.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13441>)

Author: Paulo Camara (LUA Vision)

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [digital sovereignty](<https://devfeed.tech/topics/digital-sovereignty.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [premium](<https://devfeed.tech/tags/premium.md>), [sovereignty](<https://devfeed.tech/tags/sovereignty.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

Fine-tuning may leave key model properties inherited from a small set of base families, including vocabulary, cultural alignment, memorized data and licensing. Across 31 languages and seven tokenizers, tokenization premiums reached 10x for Amharic with older tokenizers and 6.8x with the newest commercial one; Indigenous South American languages saw 1.6x-2.7x premiums under global tokenizers. A Portuguese-built vocabulary reduced premiums for 11 of 13 tested Latin-script Global South languages compared with every global tokenizer tested, by about 24% for Guarani and Quechua. The paper also proposes an eight-test black-box sovereignty audit for buyers without model-weight access, and argues that energy and liability constraints favor accuracy per joule, calibration and selective-risk metrics over raw accuracy. Preliminary internal measurements from a proprietary architecture are reported with limitations; data and measurement code are included.

### Source excerpt

arXiv:2611.13441v1 Announce Type: cross Abstract: Most language models deployed outside the United States and China are fine-tuned descendants of a small number of pretrained families. We argue that fine-tuning leaves the decisive properties of a model inherited from its base: vocabulary, knowledge, cultural alignment, memorized data and license. We contribute (i) an updated, reproducible measurement of the tokenization premium across 31 languages and 7 tokenizers released in 2023-2024, on 27 articles of the Universal Declaration of Human Rights, with explicit correction for unknown-token collapse. Premiums reach 10x for Amharic in older tokenizers and remain at 6.8x in the newest commercial one; Indigenous South American languages pay between 1.6x and 2.7x under global tokenizers. A vocabulary built from Portuguese lowers the premium for 11 of 13 Latin-script Global South languages relative to every global tokenizer tested, by about 24% for Guarani and Quechua. (ii) An eight-test black-box sovereignty audit that a buyer can run without access to weights. (iii) A regime argument: under energy and liability constraints, accuracy per joule, calibration and selective risk should replace raw accuracy as design objectives, with metrics that abstention cannot inflate. We close with preliminary internal measurements from a proprietary architecture built around these objectives, reported with their limitations. Data and measurement code are included.

## How to Run an Embedding Model on Bare Metal

DevFeed: [How to Run an Embedding Model on Bare Metal](<https://devfeed.tech/articles/how-to-run-an-embedding-model-on-bare-metal-135929.md>)

Original publisher: [Read original article](<https://qdrant.tech/blog/embeddings-on-bare-metal/>)

Author: info@qdrant.tech (Andrey Vasnetsov)

Published: 2026-11-11T01:00:00Z

Content type: tutorial

Language: en

Sources: [Qdrant Blog on Qdrant - Vector Search Engine](<https://devfeed.tech/sources/qdrant-blog-on-qdrant-vector-search-engine.md>)

Topics: [Local AI](<https://devfeed.tech/topics/local-ai.md>), [rust-ai](<https://devfeed.tech/topics/rust-ai.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [approximate-nearest-neighbor-search](<https://devfeed.tech/tags/approximate-nearest-neighbor-search.md>), [bare-metal](<https://devfeed.tech/tags/bare-metal.md>), [bert](<https://devfeed.tech/tags/bert.md>), [bits](<https://devfeed.tech/tags/bits.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [edge](<https://devfeed.tech/tags/edge.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embedding-model](<https://devfeed.tech/tags/embedding-model.md>), [embeddinggemma](<https://devfeed.tech/tags/embeddinggemma.md>), [embeddinggemma-2](<https://devfeed.tech/tags/embeddinggemma-2.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [fasttext](<https://devfeed.tech/tags/fasttext.md>), [hnsw](<https://devfeed.tech/tags/hnsw.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [image-search](<https://devfeed.tech/tags/image-search.md>), [inference](<https://devfeed.tech/tags/inference.md>), [knn-algorithm](<https://devfeed.tech/tags/knn-algorithm.md>), [matching](<https://devfeed.tech/tags/matching.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [qdrant](<https://devfeed.tech/tags/qdrant.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [recommender-system](<https://devfeed.tech/tags/recommender-system.md>), [saas](<https://devfeed.tech/tags/saas.md>), [search](<https://devfeed.tech/tags/search.md>), [simaes-networks](<https://devfeed.tech/tags/simaes-networks.md>), [similarity](<https://devfeed.tech/tags/similarity.md>), [size](<https://devfeed.tech/tags/size.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [vector](<https://devfeed.tech/tags/vector.md>), [vector-search](<https://devfeed.tech/tags/vector-search.md>), [vector-search-engine](<https://devfeed.tech/tags/vector-search-engine.md>), [vectors](<https://devfeed.tech/tags/vectors.md>), [word2vec](<https://devfeed.tech/tags/word2vec.md>)

### AI overview

A static embedding model can run on an ESP32-S3 without an operating system: the article reports fitting `potion-base-2M` on a board with 16MB flash and 8MB PSRAM using `no_std` Rust. Its weights and tokenizer vocabulary are baked into firmware at build time. Runtime inference tokenizes with a custom WordPiece implementation, looks up token vectors, mean-pools them, and optionally L2-normalizes; it avoids transformer matrix multiplication and dependencies such as PyTorch and Hugging Face `tokenizers`. Because `no_std` lacks filesystem access and `HashMap`, the build script extracts tensor metadata and encodes the sorted vocabulary for binary-search lookup. A heap allocator from `esp-alloc` supports `Vec`. The reported throughput is about 1.2 ms for 10-12 tokens, slower than the article's MacBook comparison, but demonstrated on a chip with far less memory. The approach is specific to small static embedding models.

### Source excerpt

Running vector search on the edge is something we've been working towards here at Qdrant. The main problem is that vector search is resource-hungry, starting with the embedding step: EmbeddingGemma-2, a small modern model, takes up 310MB of memory when quantized to 8 bits, while Qwen3 Embedding 4B, a larger one, needs 4.28GB. Size is just one side of the coin: transformer-based inference involves expensive computations and often relies on heavy dependencies like PyTorch (or libtorch).

## Does Pose Encoding Matter? Analyzing Geometric Representations for Discrete Motion Vocabulary Learning

DevFeed: [Does Pose Encoding Matter? Analyzing Geometric Representations for Discrete Motion Vocabulary Learning](<https://devfeed.tech/articles/does-pose-encoding-matter-analyzing-geometric-representations-for-discrete-motion-vocabulary-learning-126070.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11093>)

Author: Amir Kimiyaie, Nahal Mafi, Hamed Tabkhi

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [VAE](<https://devfeed.tech/topics/vae.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [cells](<https://devfeed.tech/tags/cells.md>), [code](<https://devfeed.tech/tags/code.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-dm](<https://devfeed.tech/tags/cs-dm.md>), [design](<https://devfeed.tech/tags/design.md>), [dimension](<https://devfeed.tech/tags/dimension.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [effect](<https://devfeed.tech/tags/effect.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [family](<https://devfeed.tech/tags/family.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [full](<https://devfeed.tech/tags/full.md>), [function](<https://devfeed.tech/tags/function.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [generation](<https://devfeed.tech/tags/generation.md>), [learning](<https://devfeed.tech/tags/learning.md>), [motion](<https://devfeed.tech/tags/motion.md>)

### AI overview

The input pose encoding strongly affects how efficiently human motion can be discretized: rotation and root-relative position encodings continue to turn larger codebooks into effective vocabulary, while full kinematic encoding levels off at a few hundred effective cells and heatmaps at fewer than sixty. The study trains VQ-VAE tokenizers with spherical codebooks across code sizes, temporal strides, and dimensions, measuring utilization and entropy-based effective vocabulary alongside reconstruction. The smallest code dimension yields the highest utilization for three of four encodings, and temporal-stride effects vary by encoding. Encodings of the same pose family can differ by an order of magnitude in effective vocabulary, a distinction reconstruction loss alone does not show. These results are reported in the supplied abstract.

### Source excerpt

arXiv:2611.11093v1 Announce Type: cross Abstract: Compressing continuous human motion into a finite vocabulary of discrete tokens is now the first stage of many motion understanding and generation pipelines. Vector quantization partitions a latent space into Voronoi cells around a finite codebook, but the geometry of that latent space is inherited from the input pose encoding, and it is unclear which encoding admits an efficient discretization. We study four encodings that place skeletal motion in different geometric spaces: Gaussian joint heatmaps on the pixel lattice, which we show to be a morphological dilation of the joint point set by a quadratic structuring function; root-relative joint positions in Euclidean space; local joint rotations on $\SO$ through a continuous 6D embedding; and a redundant full kinematic encoding. For each, we train VQ-VAE tokenizers whose codebooks lie on the unit hypersphere, varying codebook size, temporal stride, and code dimension. Beyond reconstruction, we measure codebook utilization and the effective vocabulary size, i.e., the number of equiprobable spherical Voronoi cells with the same entropy as the observed token distribution. Rotation and position encodings keep converting additional codewords into effective vocabulary, whereas the full kinematic encoding saturates at a few hundred effective cells and heatmaps at fewer than sixty. The smallest code dimension gives the highest utilization for three of the four encodings, and the effect of temporal stride depends on the encoding. Encodings of the same pose family can thus yield effective vocabularies differing by an order of magnitude, a difference that reconstruction loss alone does not reveal, which points to input geometry as a first-order design choice in motion tokenization.

## MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling

DevFeed: [MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling](<https://devfeed.tech/articles/moto-mixture-of-tokenizers-towards-fair-multilingual-language-modeling-125309.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11027>)

Author: G\\"ul Sena Alt{\\i}nta\\c{s}, Colin Raffel

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [budget](<https://devfeed.tech/tags/budget.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [english](<https://devfeed.tech/tags/english.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [information-density](<https://devfeed.tech/tags/information-density.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [languages](<https://devfeed.tech/tags/languages.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [modular](<https://devfeed.tech/tags/modular.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scripts](<https://devfeed.tech/tags/scripts.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [tokenizers](<https://devfeed.tech/tags/tokenizers.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

MoTo combines dedicated per-language BPE tokenizers into a shared superset vocabulary, deduplicating tokens into common IDs while allowing each tokenizer its own normalization, pre-tokenization rules, and vocabulary budget. The abstract reports retained English performance despite allocating English significantly fewer entries, alongside gains in 18 of 21 languages across six scripts. Improvements were most pronounced for morphologically complex and non-Latin-script languages. These findings suggest modular tokenization may offer an extensible alternative to a single monolithic tokenizer for multilingual parity.

### Source excerpt

arXiv:2611.11027v1 Announce Type: cross Abstract: Large language models rely on a single tokenizer that is chosen at training time and fixed thereafter. Most tokenizers are trained on English-dominant corpora, making them under-serve morphologically rich and non-Latin-script languages, where they can produce fragmented representations, longer sequence lengths, and reduced information density during training. We propose Mixture of Tokenizers MoTo, a modular tokenization framework that trains dedicated per-language BPE tokenizers and composes them into a unified superset vocabulary, where tokens shared across languages are deduplicated into a common ID space. Each tokenizer maintains its own normalization policy, pre-tokenization rules, and vocabulary budget. Despite allocating significantly fewer entries to English, MoTo retains English performance while delivering consistent gains across 18 out of 21 typologically diverse languages spanning six scripts, with improvements most pronounced on morphologically complex and non-Latin-script languages that are systematically underserved by English-dominant tokenizer design. Our results suggest that modular subword tokenization is a practical and extensible alternative to monolithic tokenization for multilingual parity.

## Phrase-Level Speculative Decoding on a Frozen Target

DevFeed: [Phrase-Level Speculative Decoding on a Frozen Target](<https://devfeed.tech/articles/phrase-level-speculative-decoding-on-a-frozen-target-125556.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12464>)

Author: Madhusudhana Naidu

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [target](<https://devfeed.tech/tags/target.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

The paper replaces target-token speculation with phrase-level drafts: a small MLP reads the frozen target's hidden state and predicts one- to three-word entries in a separate vocabulary. Greedy longest-match expansion converts each phrase to target tokens, and a prefix-tree mask lets the target verify candidates in one pass. The target and tokenizer remain frozen; accepting only candidates matching the target's argmax preserves greedy output, except for floating-point tie-breaking. On one A5000 at batch 1 against HuggingFace generate(), a 21.94M-parameter head reports 2.391x speedup on GSM8K, 2.444x on MATH-500, and 1.544x on MT-Bench. An 8,000-entry, 3.434M-parameter head retains 92.3% of the larger head's GSM8K tokens per pass and is reported as 64x smaller than EAGLE-3's drafter. Against EAGLE-3 on the same prompts and baseline, the larger phrase head beats tree-16 on both math sets but trails tree-60 on every set.

### Source excerpt

arXiv:2611.12464v1 Announce Type: cross Abstract: Speculative decoders raise throughput by banking more target tokens per forward pass, and Medusa, EAGLE-3, DFlash and DSpark all do it the same way: more candidates that survive verification. All of them draft in the target's own vocabulary, so one node of the candidate tree is one target token, and the tokenizer makes that unit look like a constant. It is not. BPE's pretokenizer splits on whitespace before merges are learned, so no multi-word string can ever become a single token: in Qwen3-4B, "United States of America" is four tokens and no adjacent pair of them exists as one entry, so a phrase is out of reach of any method drafting in the target's vocabulary. We change the unit. A small MLP head reads the frozen target's hidden state and predicts a one- to three-word phrase as one entry in a separate draft vocabulary; greedy longest-match expansion maps it back into the target's own tokens, and N candidates are verified in one forward pass behind a prefix-tree mask, so the change is orthogonal to tree depth and width. The target and its tokenizer stay frozen and a candidate is accepted only while it matches the target's own argmax, so the served output is identical to greedy decoding up to floating-point tie-breaking. On one A5000 at batch 1, against HuggingFace generate(), a 21.94M head (0.545% of Qwen3-4B) reaches 2.391x on GSM8K, 2.444x on MATH-500 and 1.544x on MT-Bench. Because the head's output layer is one row of 257 parameters per entry, the draft vocabulary size is the parameter budget: an 8,000-entry head is 3.434M parameters, 0.0854% of the target and 64x smaller than EAGLE-3's drafter, and keeps 92.3% of the larger head's GSM8K tokens per pass. With EAGLE-3 run on same prompts and baseline, our 21.94M head is ahead of tree-16 on both maths sets and behind tree-60 on every set.

## Tokenizer Myths and Where To Find Them

DevFeed: [Tokenizer Myths and Where To Find Them](<https://devfeed.tech/articles/tokenizer-myths-and-where-to-find-them-116918.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04215>)

Author: Rylan Schaeffer, Sanmi Koyejo

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Chinchilla scaling law](<https://devfeed.tech/topics/chinchilla-scaling-law.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [myths](<https://devfeed.tech/tags/myths.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [tokenizers](<https://devfeed.tech/tags/tokenizers.md>)

### AI overview

The paper challenges three claims about tokenizer evaluation. Bits-Per-Byte (BPB), the product of tokens per byte and bits per token, does not cancel out tokenizer or vocabulary effects: even untrained models show preferences among tokenizers. Lower BPB also does not reliably indicate a better model: untrained models can favor certain vocabulary sizes while accuracy stays at chance, and BPB correlates weakly with downstream performance for models trained on the same data. The authors find that unigram-normalized loss is not tokenizer-agnostic; after correcting a confounder, its reported correlation with BPB drops from 0.989 to 0.057, and the metrics select different vocabularies at each model size. They argue that no metric can simultaneously reward only the true distribution, score identical predictions equally, and tie all untrained models. Metric choice therefore reflects the research question.

### Source excerpt

arXiv:2611.04215v1 Announce Type: cross Abstract: Tokenization is a fundamental component of modern language models: before a model sees data, its tokenizer has already determined how the data appears. We identify and challenge three myths in the tokenization research literature. Myth #1: Bits-Per-Byte (BPB) is agnostic to the tokenizer and/or the vocabulary size: BPB is tokens per byte times bits per token, and the two factors do not cancel; using untrained models, we show that BPB prefers some tokenizers over others: the model has learned nothing, but the tokenizer--model pair has. Myth #2: lower BPB means a better model: On untrained models, BPB strongly prefers some vocabulary sizes, even though accuracy remains at chance, and on real models trained on the same data, BPB correlates weakly with downstream performance. Myth #3: different "tokenizer-agnostic" metrics yield consistent results: The "unigram-normalized loss" is not tokenizer-agnostic; its reported correlation of 0.989 with BPB comes from a confounder and falls to 0.057 once fixed; from the same data the two metrics select different vocabularies at each model size. We present a tokenizer trilemma: no metric can score only the true distribution best, give the same predictions the same score, and tie all untrained models, so choosing a metric is choosing a research question. We conclude with a caution for reading the tokenization literature and a conjecture for future work.

## Beyond Semantic IDs: Atomic Creator Tokens for Large-Scale Generative Recommendation

DevFeed: [Beyond Semantic IDs: Atomic Creator Tokens for Large-Scale Generative Recommendation](<https://devfeed.tech/articles/beyond-semantic-ids-atomic-creator-tokens-for-large-scale-generative-recommendation-115012.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00323>)

Author: Zhuosheng Liu, Yihan Wu, Ziyun Xu, Xiaoxuan Liu, Longfei Yun, Kevin Han, Mingze Gao, Pengfei Wang, Junfeng Pan, Wenjie Fu, Haoran Liu

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [personalization](<https://devfeed.tech/topics/personalization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [ai-experiments](<https://devfeed.tech/topics/ai-experiments.md>)

Tags: [atomic](<https://devfeed.tech/tags/atomic.md>), [cold-start-problem](<https://devfeed.tech/tags/cold-start-problem.md>), [creators](<https://devfeed.tech/tags/creators.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [generative](<https://devfeed.tech/tags/generative.md>), [ids](<https://devfeed.tech/tags/ids.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [latency](<https://devfeed.tech/tags/latency.md>), [live](<https://devfeed.tech/tags/live.md>), [model](<https://devfeed.tech/tags/model.md>), [recall](<https://devfeed.tech/tags/recall.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [test](<https://devfeed.tech/tags/test.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [top](<https://devfeed.tech/tags/top.md>), [top-k](<https://devfeed.tech/tags/top-k.md>), [training](<https://devfeed.tech/tags/training.md>), [variable](<https://devfeed.tech/tags/variable.md>)

### AI overview

Atomic Creator Tokens (ACT) assign each creator a unique vocabulary token trained end-to-end, avoiding the collisions and frozen-representation gradient limit of Semantic IDs (SIDs). In a controlled production-data comparison where only the token layer varied, ACT improved creator-level Recall@15 by 19.8 points (1.84x) over a matched SID baseline, remained ahead with a SID codebook 64x larger, and had about 2.7x lower per-user latency. On KuaiRand, ACT was the only method reported to beat global popularity, provided each creator token had enough training examples. A live A/B test was directionally positive on every reported metric against SID-enhanced and classic production rankers.

### Source excerpt

arXiv:2611.00323v1 Announce Type: new Abstract: Recommending creators that a user does not yet follow is a cold-start problem: from hundreds of thousands of popular creators and only a short trail of prior engagements, the system must predict the next creator worth surfacing. The prevailing generative recommendation recipe tokenizes each creator into a \emph{Semantic ID} (SID) by quantizing a frozen embedding and trains a language model to generate SIDs. We show that this SID layer, not the language model, is the binding constraint. The creator-to-SID map is many-to-one, so colliding creators must be resolved by a popularity tie-break that caps creator-level recall and favors already-popular creators, and the frozen quantizer stops the training gradient from reaching the creator representation. We propose Atomic Creator Token (ACT), which gives every creator its own vocabulary token trained end-to-end. The map is bijective, so collisions are impossible and inference is a direct top-$k$ over creator logits. In a controlled comparison on production data in which the token layer is the only variable, ACT improves creator-level Recall@15 by 19.8 points ($1.84\times$) over a matched SID baseline, still leads when the SID codebook is enlarged $64\times$, and serves at ${\sim}2.7\times$ lower per-user latency. The advantage carries over to the public KuaiRand dataset, where ACT is the only method that beats a global-popularity baseline, provided each creator token sees enough training examples. In a live A/B test, ACT is directionally positive on every reported metric against both the SID-enhanced and the classic production rankers.

## Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization

DevFeed: [Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization](<https://devfeed.tech/articles/certified-splitting-decidability-and-anchor-supply-for-maximal-munch-tokenization-113767.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.08854>)

Author: Nicklas Nidh\\"ogg

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [content defined chunking](<https://devfeed.tech/topics/content-defined-chunking.md>), [Regular expression](<https://devfeed.tech/topics/regular-expression.md>), [Finite-state machine](<https://devfeed.tech/topics/finite-state-machine.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [anchor](<https://devfeed.tech/tags/anchor.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audits](<https://devfeed.tech/tags/audits.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [budget](<https://devfeed.tech/tags/budget.md>), [campaign](<https://devfeed.tech/tags/campaign.md>), [certificate](<https://devfeed.tech/tags/certificate.md>), [certification](<https://devfeed.tech/tags/certification.md>), [commit](<https://devfeed.tech/tags/commit.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [covering](<https://devfeed.tech/tags/covering.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-fl](<https://devfeed.tech/tags/cs-fl.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [decidability](<https://devfeed.tech/tags/decidability.md>), [either](<https://devfeed.tech/tags/either.md>), [follow](<https://devfeed.tech/tags/follow.md>), [free](<https://devfeed.tech/tags/free.md>), [input](<https://devfeed.tech/tags/input.md>), [inside](<https://devfeed.tech/tags/inside.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

The paper defines certified byte windows whose occurrences identify safe starting points for independent maximal-munch scans, preserving sequential token boundaries without speculation or a fixup pass. It decides certification from the token set using two methods: an online method with a completeness cutoff for literal vocabularies, and an offline armed-run verifier for arbitrary regular token sets. It also derives bounds on boundary movement after edits, gives exact conditions for delimiter-based cuts, and presents tools to measure anchor-free spans and detect boundary differences between token sets. The abstract says the results have executable checks and that a threaded scanner in the munch library reproduced the sequential boundary stream on an archived campaign corpus.

### Source excerpt

arXiv:2610.08854v2 Announce Type: replace-cross Abstract: A certificate is a window of bytes with an origin inside it such that, wherever the window occurs in a completely tokenizable input, the token covering the window's final byte begins at that origin. Certification is decided from the token set alone, and a certificate's occurrences mark where work may soundly begin. This paper decides certification and applies it to cuts, edits and audits. It is decided two ways, online with a completeness cutoff for literal vocabularies and offline through an armed-run verifier for arbitrary regular token sets. Either route, under a declared window budget, yields the anchor inventory, with a counterexample on each refutation. Cutting a completely tokenizable input at certified anchors and scanning the chunks independently reproduces the sequential segmentation, with no speculation or fixup pass. A tokenizability-preserving byte substitution moves boundaries only strictly between the certified anchors witnessed in the unchanged bytes on either side of it. For tokens of length at most L, an edit at p moves no boundary at or below p - L + 1. Cutting at every occurrence of a delimiter is sound exactly when it occurs only at token starts, and cutting after every occurrence exactly when it occurs only at token ends. Two instruments follow. The anchor-free span of a finite certified inventory is decided bounded or not with its exact supremum, three for the UTF-8 shape; and a differential auditor decides whether two token sets place different boundaries on an input both tokenize, with a differing input. Every theorem carries an executable check. Every measurement is a named program's output or arithmetic on such outputs, and each program ships with the paper or is pinned by commit in the munch library. That library's threaded scanner splits the archived campaign corpus at certified anchors and reproduces the sequential boundary stream byte for byte.

## Context-Aware Subtitle Translation on CPU: Deployment-Driven Training, Evaluation, and Latency Design

DevFeed: [Context-Aware Subtitle Translation on CPU: Deployment-Driven Training, Evaluation, and Latency Design](<https://devfeed.tech/articles/context-aware-subtitle-translation-on-cpu-deployment-driven-training-evaluation-and-latency-design-97952.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.09957>)

Author: Tsz-To Wong, Shyang-En Weng

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [machine-translation](<https://devfeed.tech/topics/machine-translation.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [browser](<https://devfeed.tech/tags/browser.md>), [context](<https://devfeed.tech/tags/context.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [free](<https://devfeed.tech/tags/free.md>), [google-translate](<https://devfeed.tech/tags/google-translate.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [open](<https://devfeed.tech/tags/open.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [taiwan](<https://devfeed.tech/tags/taiwan.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [training](<https://devfeed.tech/tags/training.md>), [translation](<https://devfeed.tech/tags/translation.md>)

### AI overview

The paper reports an English-to-Traditional-Chinese (Taiwan) subtitle translator designed to run on a viewer's CPU within about 500 ms per subtitle. Its Sotto models use task-specific vocabulary and several training steps, including continued pretraining, supervised fine-tuning, distillation, and preference optimization. Since subtitle references can be unreliable, the authors compare translations pairwise with Google Translate, using two LLM judges who assess each pair in both orders. On 1,820 held-out subtitles, Sotto-1.7B has a 0.576 win rate against Google Translate, beats four open translation models, and performs slightly below a 27B model. Idle-time prefill reduces mean CPU end-to-end latency by 31%, while the task-specific vocabulary represents the same Chinese text with 28% fewer tokens. The deployed Sotto-0.6B remains within 500 ms for every subtitle in the reported latency test.

### Source excerpt

arXiv:2607.09957v3 Announce Type: replace-cross Abstract: We study English-to-Traditional-Chinese (Taiwan) subtitle translation for a free browser extension, where the model must run on the viewer's CPU in about 500 ms per subtitle. We train two models built on the LMT-60 base models, Sotto-1.7B and the deployed Sotto-0.6B, with a task-specific vocabulary, continued pretraining, supervised fine-tuning, weight interpolation, logit distillation and on-policy preference optimization. Because reference translations of subtitles are unreliable, we evaluate by pairwise comparison against one fixed baseline, Google Translate, with each pair judged in both orders by a panel of two LLM judges. On 1,820 held-out subtitles, Sotto-1.7B has a win rate of 0.576 against the baseline and wins head-to-head against four open translation models, and is slightly below a 27B model. We split latency into prefill and decoding: idle-time prefill, which prefills the known part of the prompt while waiting for the next subtitle, cuts mean end-to-end latency by 31% on a CPU, and the task-specific vocabulary encodes the same Chinese text in 28% fewer tokens. The deployed model stays within 500 ms for every subtitle in our latency test. The code is available at https://github.com/aiden1020/localsubs .

## Multi-Token Prediction and Alternative Tokenization Improve Learning Latent Multi-Hop Graph Search at Scale

DevFeed: [Multi-Token Prediction and Alternative Tokenization Improve Learning Latent Multi-Hop Graph Search at Scale](<https://devfeed.tech/articles/multi-token-prediction-and-alternative-tokenization-improve-learning-latent-multi-hop-graph-search-at-scale-97067.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31439>)

Author: Arvid Frydenlund, FuTe Wong, Katherine Lambert, Frank Rudzicz

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [graph](<https://devfeed.tech/tags/graph.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [scale](<https://devfeed.tech/tags/scale.md>), [search](<https://devfeed.tech/tags/search.md>), [token](<https://devfeed.tech/tags/token.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

For shortest-path generation on unseen graphs, the authors report that scaling failures stem from both the supervision signal and tokenization. They establish and empirically demonstrate that multi-token prediction (MTP) improves learning, with benefits they describe as topologically agnostic. They also introduce alternative tokenizations and report that their effects are orthogonal to MTP. Together, these methods reportedly enable consistent, stable learning at graph scales beyond prior limits. The supplied text is an abstract and does not provide implementation details or quantitative results.

### Source excerpt

arXiv:2610.31439v1 Announce Type: new Abstract: Given an unseen graph, a start node, $s$, and an end node, $t$, can a language model generate the shortest path from $s$ to $t$? Prior work has found that models fail to learn this task as the graphs scale. We show that this failure has two causes: the supervision signal and the tokenization. First, we theoretically establish and empirically demonstrate the effectiveness of multi-token prediction (MTP), and show that the benefits of MTP are topologically agnostic. Second, we introduce alternative tokenizations and show that the two causes are orthogonal. Using these methods, we break through the scaling wall observed by prior work, consistently and stably learning to solve the task at scales far beyond previous limits.

## MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version

DevFeed: [MoVoC: Morphology-Aware Subword Construction for Geez Script Languages Corrected version](<https://devfeed.tech/articles/movoc-morphology-aware-subword-construction-for-geez-script-languages-corrected-version-97893.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2509.08812>)

Author: Hailay Kidu Teklehaymanot, Dren Fazlija, Wolfgang Nejdl

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [markup](<https://devfeed.tech/topics/markup.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [approach](<https://devfeed.tech/tags/approach.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [languages](<https://devfeed.tech/tags/languages.md>), [precision](<https://devfeed.tech/tags/precision.md>), [resource](<https://devfeed.tech/tags/resource.md>), [results](<https://devfeed.tech/tags/results.md>), [script](<https://devfeed.tech/tags/script.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [tokenizers](<https://devfeed.tech/tags/tokenizers.md>)

### AI overview

MoVoC-Tok combines supervised morpheme segmentation with Byte Pair Encoding (BPE) tokens to build subword vocabularies that better preserve morphological boundaries in Ge'ez-script languages. The authors release 169,806 morpheme-annotated records across four languages and morpheme-aware vocabularies for two. On words held out from the morphology resource used to constrain each language's tokenizer, MoVoC-Tok achieves the highest MorphScore in three of four languages and the highest morpheme-boundary precision in two. Its clearest gains are for Tigre, where it is applied cross-lingually despite not being trained on that language. The abstract withdraws machine translation results from the published version; data, tokenizers, and evaluation code are publicly available.

### Source excerpt

arXiv:2509.08812v2 Announce Type: replace-cross Abstract: Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Ge'ez script. To address this, we present Morpheme-aware Subword Vocabulary Construction (MoVoC) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we release morpheme-annotated data for four Ge'ez script languages (169,806 records) and a morpheme-aware vocabulary for two of them. Scoring every comparison on words held out from the morpheme resource that constrains the MoVoC-Tok applied to that language, MoVoC-Tok has the highest MorphScore in three of four languages and the highest morpheme boundary precision in two of four, with its largest and most clearly resolved gains on Tigre, a language it was not trained on and to which it is applied cross-lingually. The machine translation results reported in the published version are withdrawn. Data, tokenizers and evaluation code are publicly available.

## Rethinking Molecular Text Representations for LLMs: An Empirical Study

DevFeed: [Rethinking Molecular Text Representations for LLMs: An Empirical Study](<https://devfeed.tech/articles/rethinking-molecular-text-representations-for-llms-an-empirical-study-96849.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.03057>)

Author: Arun Raja, Garrett M. Morris, Kian Ming A. Chai

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Computational Chemistry / Materials Science](<https://devfeed.tech/topics/computational-chemistry-materials-science.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [audits](<https://devfeed.tech/tags/audits.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [chemistry](<https://devfeed.tech/tags/chemistry.md>), [cml](<https://devfeed.tech/tags/cml.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [llms](<https://devfeed.tech/tags/llms.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [routing](<https://devfeed.tech/tags/routing.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [specialization](<https://devfeed.tech/tags/specialization.md>), [structured](<https://devfeed.tech/tags/structured.md>), [study](<https://devfeed.tech/tags/study.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

MolRepBench benchmarks 16 LLMs on eight chemical tasks and finds that molecular representation strongly affects performance, with no single format best across tasks. CML and MolJSON lead on structural tasks, while IUPAC leads on semantic tasks and molecule retrieval for all 16 models; IUPAC also yields the highest fraction of correct molecule generations in LLM-as-a-judge evaluation. Chemistry-specialized models perform well with SMILES but degrade substantially on structured formats, so SMILES-only evaluation may not reflect broader competence. Tokenization, probing and attention analyses find representation-specific internal processing, including more attention across molecular spans for structured formats. The authors recommend task-aware representation routing and caution against representation-invariant evaluation.

### Source excerpt

arXiv:2606.03057v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.

## How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

DevFeed: [How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications](<https://devfeed.tech/articles/how-specializing-the-vocabulary-of-slms-can-cut-inference-cost-by-77-in-real-applications-93065.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specialization>)

Author: Bernardo França de Aguiar; Gabriel Pimenta de Freitas Cardoso; Bruno Duarte; Francisco de Almeida Rocha Alves

Published: 2026-10-21T17:35:36Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [7-4](<https://devfeed.tech/tags/7-4.md>), [applications](<https://devfeed.tech/tags/applications.md>), [cost](<https://devfeed.tech/tags/cost.md>), [domain](<https://devfeed.tech/tags/domain.md>), [essay](<https://devfeed.tech/tags/essay.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [general](<https://devfeed.tech/tags/general.md>), [inference](<https://devfeed.tech/tags/inference.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [production](<https://devfeed.tech/tags/production.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [slms](<https://devfeed.tech/tags/slms.md>), [source](<https://devfeed.tech/tags/source.md>), [split](<https://devfeed.tech/tags/split.md>), [text](<https://devfeed.tech/tags/text.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

In a production Portuguese essay grader built on Qwen3-4B, replacing the general-purpose tokenizer with VocabX reduced average responses from 350 to 101 tokens and measured inference cost by 77%, while the bench score rose from 0.703 to 0.810. The comparison held the base model, data, training recipe and seed constant; tests used 500 requests on an RTX 5090 with vLLM. VocabX selects vocabulary changes by estimated serving cost, accounting for generated text being more expensive than input in this setup, and combines subword rebuilding with multiword macro tokens for repeated spans such as JSON structures. It also estimates gains for candidates missed by a sample. A fused macro that included grades was more compact but hurt quality and saved less cost; separating grade tokens improved both outcomes. Across five public domains and three model families, VocabX compressed more than the reference methods in all 15 combinations, though the paper is still under review and its results may change.

### Source excerpt

*Models specialized for a domain or a task by fine-tuning general open-source models, such as Qwen and Gemma, inherit a vocabulary built for many domains and general applications.* *In an essay grader we run in production, that vocabulary split the text the model writes most into many tokens.

## Discrete Neural Interlingua for Ontology-Agnostic EHR Modeling

DevFeed: [Discrete Neural Interlingua for Ontology-Agnostic EHR Modeling](<https://devfeed.tech/articles/discrete-neural-interlingua-for-ontology-agnostic-ehr-modeling-94234.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.27047>)

Author: Tong Ding, Caiwei Tian, Dongmin Bang, Ming Y. Lu, Sophia J. Wagner, Andrew Zhang, Rowland Pettit, Long Phi Le, Faisal Mahmood

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [linking](<https://devfeed.tech/tags/linking.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [neural](<https://devfeed.tech/tags/neural.md>), [ontology](<https://devfeed.tech/tags/ontology.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [systems](<https://devfeed.tech/tags/systems.md>), [text](<https://devfeed.tech/tags/text.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

MEDIATOR addresses the mismatch between clinical concepts expressed through different codes, local identifiers, synonyms, and text by mapping text-described concepts into a shared vector-quantized codebook. Trained with synonym supervision and information from concept descriptions, code co-occurrence, and relation graphs, it supports open-vocabulary EHR tokenization without manual mappings or a fixed vocabulary. With an EHR Transformer, it improves mean AUPRC over the strongest prior tokenizer by 4.0 points on 20 MIMIC-III tasks and by 8.6 points on 14 zero-shot eICU tasks, using all eICU events where a closed-vocabulary tokenizer used 13%. Under identical inputs, MEDIATOR still leads on two reported transfer tasks, suggesting gains from representation as well as coverage. The abstract attributes in-domain accuracy mainly to the text encoder and transfer robustness and entity-linking quality to MEDIATOR's learned objectives.

### Source excerpt

arXiv:2610.27047v1 Announce Type: cross Abstract: EHR foundation models promise scalable clinical prediction across health systems, yet their deployment is constrained by a tokenization bottleneck. The same clinical concept can appear as different ontology codes, local identifiers, synonyms, or free-text descriptions, which standard EHR tokenizers treat as unrelated symbols. A model trained on MIMIC-III with a closed-vocabulary tokenizer can use only 13% of eICU events. We introduce MEDIATOR, a discrete ontology-agnostic tokenizer that maps any clinical concept expressed as text into a shared vector-quantized codebook, turning EHR harmonization into open-vocabulary tokenization without manual mappings or a fixed vocabulary. Trained with synonym supervision and guided by enriched concept descriptions, code co-occurrence, and relation graphs, MEDIATOR collapses equivalent expressions while organizing distinct concepts in a medically meaningful discrete space. Used as the token interface of an EHR Transformer, MEDIATOR improves mean AUPRC over the strongest prior tokenizer by 4.0 points (8% relative) on 20 MIMIC-III tasks and by 8.6 points (29% relative) in zero-shot transfer to 14 eICU tasks, where it uses all events. The transfer gain combines coverage and representation, as MEDIATOR remains the best tokenizer under identical inputs (+1.0 and +1.9 points on operational and subcohort-mortality tasks). Same-encoder controls show that the text encoder drives in-domain accuracy, while the learned objectives of MEDIATOR drive transfer robustness and entity-linking quality. MEDIATOR also enables transferable attribution across hospitals.

## GenRecs Tokenization: Transforming Ads Recommendation at Meta with Stable Model Vocabulary

DevFeed: [GenRecs Tokenization: Transforming Ads Recommendation at Meta with Stable Model Vocabulary](<https://devfeed.tech/articles/genrecs-tokenization-transforming-ads-recommendation-at-meta-with-stable-model-vocabulary-90425.md>)

Original publisher: [Read original article](<https://engineering.fb.com/2026/10/19/ml-applications/genrecs-tokenization-transforming-ads-recommendation-at-meta-with-stable-model-vocabulary/>)

Author: Carl Hu; Justin Johnson; Parish Aggarwal; Kaushik Rangadurai; Senthil Manickavelu; Tianwei Zhang

Published: 2026-10-19T16:00:47Z

Content type: article

Language: en

Sources: [Engineering at Meta](<https://devfeed.tech/sources/engineering-at-meta.md>), [Meta ML Applications](<https://devfeed.tech/sources/meta-ml-applications.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [personalization](<https://devfeed.tech/topics/personalization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [llms](<https://devfeed.tech/tags/llms.md>), [meta](<https://devfeed.tech/tags/meta.md>), [ml-applications](<https://devfeed.tech/tags/ml-applications.md>), [model](<https://devfeed.tech/tags/model.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

Meta describes GenRecs Tokenization, a semantic representation for ads and other items that replaces individual item IDs with sequences of tokens from a stable vocabulary. It combines item content with engagement signals so recommendation models can generalize to new or low-engagement items. Meta reports a 0.35% conversion lift when the tokenizer was paired with its Generative Ads Model and Adaptive Ranking Model, based on internal data measured in February 2026.

### Source excerpt

Generative Recommenders (GenRecs) represent a paradigm shift from scoring every ad to reasoning about ad content and user preferences together in order to predict the best outcome for advertisers. Leveraging the world knowledge built into LLMs makes ad matching more intelligent and precise, compounding gains for advertisers. GenRecs Tokenization is the foundational data representation that [...] Read More... The post GenRecs Tokenization: Transforming Ads Recommendation at Meta with Stable Model Vocabulary appeared first on Engineering at Meta.

## UltraBERT Embeddings: Whole Documents Versus Chunks for Retrieval

DevFeed: [UltraBERT Embeddings: Whole Documents Versus Chunks for Retrieval](<https://devfeed.tech/articles/ultrabert-fits-32k-tokens-in-one-vector-rip-chunking-90009.md>)

Original publisher: [Read original article](<https://qdrant.tech/blog/ultrabert-embed/>)

Author: info@qdrant.tech (Andrey Vasnetsov)

Published: 2026-10-19T00:00:00Z

Content type: article

Language: en

Sources: [Qdrant Blog on Qdrant - Vector Search Engine](<https://devfeed.tech/sources/qdrant-blog-on-qdrant-vector-search-engine.md>)

Topics: [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [Qdrant](<https://devfeed.tech/topics/qdrant.md>), [vector embeddings in AI](<https://devfeed.tech/topics/vector-embeddings-in-ai.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [approximate-nearest-neighbor-search](<https://devfeed.tech/tags/approximate-nearest-neighbor-search.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bert](<https://devfeed.tech/tags/bert.md>), [compression](<https://devfeed.tech/tags/compression.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [fasttext](<https://devfeed.tech/tags/fasttext.md>), [hnsw](<https://devfeed.tech/tags/hnsw.md>), [image-search](<https://devfeed.tech/tags/image-search.md>), [knn-algorithm](<https://devfeed.tech/tags/knn-algorithm.md>), [latency](<https://devfeed.tech/tags/latency.md>), [matching](<https://devfeed.tech/tags/matching.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [qdrant](<https://devfeed.tech/tags/qdrant.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [recommender-system](<https://devfeed.tech/tags/recommender-system.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [saas](<https://devfeed.tech/tags/saas.md>), [simaes-networks](<https://devfeed.tech/tags/simaes-networks.md>), [similarity](<https://devfeed.tech/tags/similarity.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [vector-search](<https://devfeed.tech/tags/vector-search.md>), [vector-search-engine](<https://devfeed.tech/tags/vector-search-engine.md>), [vectors](<https://devfeed.tech/tags/vectors.md>), [word2vec](<https://devfeed.tech/tags/word2vec.md>)

### AI overview

Tests of UltraBERT-Embed-150M in Qdrant found that 512-token chunks generally retrieved document details better than whole-document vectors, though whole-document vectors performed better for queries about an entire episode and for finding passkeys. Combining both representations helped only slightly on two tasks and performed worse on others. The article also reports that 1-bit TurboQuant compression reduced chunk-vector storage for 200,000 articles to about 140 MB, with quality and rescoring latency tradeoffs.

### Source excerpt

Every retrieval pipeline starts with the same call: how big is a chunk? Too small, and a chunk loses its context. Too big, and the embedding model cuts it off. DMIS Lab's UltraBERT-Embed-150M moves that cutoff to 32K tokens. It's a 151M-parameter dense embedding model built on UltraBERT, an English encoder trained for 32K-token inputs. Most documents fit in one pass, and you get back one 768-dimension vector. So is chunking dead? We tested it in Qdrant: one vector per document against 512-token chunks, the two together, and both compressed.

## How Large Language Models Actually Work: Tokens, Attention and Transformers Explained With Code

DevFeed: [How Large Language Models Actually Work: Tokens, Attention and Transformers Explained With Code](<https://devfeed.tech/articles/how-large-language-models-actually-work-tokens-attention-and-transformers-explained-with-code-85799.md>)

Original publisher: [Read original article](<https://ihatereading.in/blogs/how-large-language-models-work-tokens-attention-transformers>)

Author: Shrey Vijayvargiya

Published: 2026-10-12T10:00:00Z

Content type: tutorial

Language: en

Sources: [iHateReading](<https://devfeed.tech/sources/ihatereading.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [attention-is-all-you-need-explained](<https://devfeed.tech/tags/attention-is-all-you-need-explained.md>), [backend](<https://devfeed.tech/tags/backend.md>), [blockchain](<https://devfeed.tech/tags/blockchain.md>), [coding](<https://devfeed.tech/tags/coding.md>), [development](<https://devfeed.tech/tags/development.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [how-chatgpt-works](<https://devfeed.tech/tags/how-chatgpt-works.md>), [how-do-large-language-models-work](<https://devfeed.tech/tags/how-do-large-language-models-work.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm-working-principle](<https://devfeed.tech/tags/llm-working-principle.md>), [next-token-prediction](<https://devfeed.tech/tags/next-token-prediction.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [product](<https://devfeed.tech/tags/product.md>), [programming](<https://devfeed.tech/tags/programming.md>), [react](<https://devfeed.tech/tags/react.md>), [self-attention-code-example](<https://devfeed.tech/tags/self-attention-code-example.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [tokenization-embeddings-attention](<https://devfeed.tech/tags/tokenization-embeddings-attention.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [transformer-architecture-explained](<https://devfeed.tech/tags/transformer-architecture-explained.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [typescript](<https://devfeed.tech/tags/typescript.md>), [web-development](<https://devfeed.tech/tags/web-development.md>)

### AI overview

This guide explains how large language models generate text, tracing the process from tokenization and embeddings through self-attention, transformer blocks, sampling, caching, and training. Runnable Python examples illustrate the core mechanisms and explain practical implications for people building with LLM APIs and products.

### Source excerpt

A beginner-to-intermediate guide to how LLMs work under the hood: tokenization, embeddings, self-attention from 'Attention Is All You Need', transformer blocks...

## ttok 1.0

DevFeed: [ttok 1.0](<https://devfeed.tech/articles/ttok-1-0-77519.md>)

Original publisher: [Read original article](<https://simonwillison.net/2026/Oct/9/ttok/>)

Author: Simon Willison

Published: 2026-10-09T00:34:43Z

Content type: release

Language: en

Sources: [Simon Willison's Weblog](<https://devfeed.tech/sources/simon-willison-s-weblog.md>)

Topics: [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-2-271](<https://devfeed.tech/tags/ai-2-271.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [generative-ai-2-013](<https://devfeed.tech/tags/generative-ai-2-013.md>), [gpt-5](<https://devfeed.tech/tags/gpt-5.md>), [gpt-6](<https://devfeed.tech/tags/gpt-6.md>), [llms](<https://devfeed.tech/tags/llms.md>), [llms-1-979](<https://devfeed.tech/tags/llms-1-979.md>), [openai](<https://devfeed.tech/tags/openai.md>), [openai-474](<https://devfeed.tech/tags/openai-474.md>), [projects](<https://devfeed.tech/tags/projects.md>), [projects-556](<https://devfeed.tech/tags/projects-556.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [tokenization-15](<https://devfeed.tech/tags/tokenization-15.md>)

### AI overview

The ttok 1.0 release changes the default tokenizer from GPT-4 to GPT-5/GPT-6. An experiment reported matching token counts across seven GPT models on 31 fixtures, though OpenAI has not confirmed that GPT-6 uses the same tokenizer as GPT-5.

### Source excerpt

Release: ttok 1.0 I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead! I figured switching the default was a reasonable excuse to finally ship a 1.0. OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same: All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus. Tags: projects, ai, openai, generative-ai, llms, tokenization

[Next page](<https://devfeed.tech/topics/tokenization.md?cursor=WyIyMDI2LTEwLTA5VDAwOjM0OjQzKzAwOjAwIiwgIjA5ZGM0NzhiLWQ5NWQtNDlmMi05YmZjLTMwYmY4YmU2YjVmNCJd>)