# Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

DevFeed: [Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese](<https://devfeed.tech/articles/homebrewed-slms-autoregressive-and-masked-diffusion-language-models-built-at-home-for-portuguese-108924.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.40533>)

Author: Pedro Carvalho Brom

Published: 2026-10-30T04:00:00Z

Content type: tutorial

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Tutorial](<https://devfeed.tech/topics/tutorial.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [building](<https://devfeed.tech/tags/building.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [compute](<https://devfeed.tech/tags/compute.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [continuation](<https://devfeed.tech/tags/continuation.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [domain](<https://devfeed.tech/tags/domain.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [legal](<https://devfeed.tech/tags/legal.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [minutes](<https://devfeed.tech/tags/minutes.md>), [models](<https://devfeed.tech/tags/models.md>), [slms](<https://devfeed.tech/tags/slms.md>)

## AI overview

The tutorial shows how to train an 11-12M-parameter Brazilian Portuguese language model from scratch on one consumer GPU, with a tokenizer, tested Python package, and two notebooks. On Brazilian prose and federal laws, autoregressive training peaked at 64 epochs before memorizing the training text, while masked diffusion improved through 256 epochs; their best test likelihoods differed by 0.06 bits per byte, within the preregistered minimum effect. On held-out and post-training laws, the model matched a 6-gram Kneser-Ney floor on likelihood over the same windows but lost on continuation. It beat that floor at tamper detection and filled cross-references more often than the floor and 160M/630M public models only in decrees, a genre absent from training; on balanced-group blank filling, it tied the 630M model per item. For laws, public models using roughly 500 to 100,000 times more compute led at tamper detection and continuation. The preregistered blind human reading is not included in this version.

## Source excerpt

arXiv:2610.40533v1 Announce Type: cross Abstract: Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It proposes no new method. It releases a tested Python package and two notebooks in Brazilian Portuguese with autoregressive and masked diffusion generation on one network, a preregistered comparison protocol, legal tasks built from text alone with a Kneser-Ney floor and public yardsticks, reference numbers for models of 11 to 12M parameters, and a catalogue of defects that only measurement revealed. On Brazilian prose and federal laws, at minutes per run, the autoregressive mode peaks at 64 epochs over the same text and then memorises it, while masked diffusion improves up to 256. At their best points on test, the diffusion bound exceeds the autoregressive likelihood by 0.06 bits per byte, within the preregistered minimum effect. On held-out laws and on acts published after training, the model ties a 6-gram floor in likelihood on the same windows, and it loses to the floor at continuing those acts. It beats the floor at detecting tampered text, and fills cross-references more often than the floor and public models of 160M and 630M parameters, only in decrees, a genre absent from training, where those scorers stay near or below chance, and in filling the blank only when each group weighs the same, since per item it ties the 630M model. On laws, the public models, trained with about 5x10^2 to 10^5 times its compute, lead in tampered text and continuation. The blind human reading is preregistered and not part of this version.