# Tokenization in Transformers v5: Simpler, Clearer, and More Modular

DevFeed: [Tokenization in Transformers v5: Simpler, Clearer, and More Modular](<https://devfeed.tech/articles/tokenization-in-transformers-v5-simpler-clearer-and-more-modular-7520.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/tokenizers>)

Author: Ita Zaporozhets; Aritra Roy Gosthipaty; Arthur Zucker; Sergio Paniego; merve; Pedro Cuenca

Published: 2025-12-18T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [AI, ML & Data Engineering](<https://devfeed.tech/topics/ai-ml-data-engineering.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [backend](<https://devfeed.tech/tags/backend.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [natural-language-processing](<https://devfeed.tech/tags/natural-language-processing.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [rust](<https://devfeed.tech/tags/rust.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>), [tokenizers](<https://devfeed.tech/tags/tokenizers.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [v5](<https://devfeed.tech/tags/v5.md>)

## AI overview

A practical guide to the Transformers v5 tokenizer redesign. It explains tokenization for LLMs, separates tokenizer architecture from trained vocabulary, and covers modular processing stages and algorithms such as BPE.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.