# One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data

DevFeed: [One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data](<https://devfeed.tech/articles/one-byte-one-token-byteifying-a-27b-hybrid-attention-llm-with-self-generated-data-128738.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13607>)

Author: Liwei Zhang, ZhiMing Tong

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [tokenization](<https://devfeed.tech/topics/tokenization.md>), [Encoding](<https://devfeed.tech/topics/encoding.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [entries](<https://devfeed.tech/tags/entries.md>), [failure](<https://devfeed.tech/tags/failure.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [llm](<https://devfeed.tech/tags/llm.md>), [one](<https://devfeed.tech/tags/one.md>), [token](<https://devfeed.tech/tags/token.md>), [with](<https://devfeed.tech/tags/with.md>)

## AI overview

The study converts Qwen3.8-27B from subword tokens to one-byte tokens by shrinking its vocabulary to 384 entries, keeping other weights unchanged, and continuing training on 4.77 GB of supervised bytes. The two-stage curriculum uses dialogue and behavioral data generated by the original model, plus encyclopedia text for knowledge reading; training took 178.6 hours on eight H200 GPUs. On several benchmarks, performance was statistically close to the original, while scores fell 4-6 points on C-Eval, thinking-mode GSM8K, and non-thinking IFEval. Without thinking, the byte model improved results on the reported character-level tasks, but lost some Chinese character lookup knowledge, which thinking did not restore. Throughput is a major cost: Chinese characters per second was about one quarter of the original, and 64K-byte prefill was 6.5 times slower. The authors also report generation failure modes and fixes.

## Source excerpt

arXiv:2611.13607v1 Announce Type: cross Abstract: Byte-level BPE fragments Chinese text (for Qwen3.8 on Chinese Wikipedia, 3.84 bytes per token on average; 64% of tokens are single characters) and hides spelling from the model. We ask whether an already-trained 27B subword model can be converted directly into a pure byte-level model (one byte, one token), without a patching structure or a local encoder, while keeping most of its ability. Starting from Qwen3.8-27B (48 Gated DeltaNet layers + 16 full-attention layers), we shrink the vocabulary from 248,320 to 384 entries (the 256 byte symbols already in the BPE vocabulary plus special ids), leave every other weight untouched, and continue training in a two-stage curriculum (encoding first, then reading) on 4.77 GB of supervised bytes: all dialogue/behavioral supervision is generated on-policy by the original model itself (character dictionary, word dictionary, general dialogue, coding sessions), and knowledge reading uses encyclopedia text (54% of supervised bytes). Training takes 178.6 h on 8xH200. On the same GPU and serving stack, the byte model's gap to the original falls within the paired bootstrap interval on MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K, and it loses 4-6 points on C-Eval, thinking-mode GSM8K and non-thinking IFEval. Without thinking it leads on every character-level task (CUTE 75.2 -> 84.0, EXECUTE-zho 81.6 -> 93.9, our Chinese character suite 54.4 -> 64.0), but lookup knowledge about Chinese characters such as pronunciation and stroke count drops, and thinking does not recover it. The cost is throughput: per-step speed is identical, but each step yields 3-5x less output; Chinese characters per second is about 1/4 of the original, and prefill at 64K bytes is 6.5x slower. We also document four byte-model failure modes (greedy runaway, sampling breaking UTF-8, ineffective penalty terms on a small vocabulary, repetition) and their fixes.