# Mini-R1: Reproduce Deepseek R1 "aha moment" a RL tutorial

DevFeed: [Mini-R1: Reproduce Deepseek R1 "aha moment" a RL tutorial](<https://devfeed.tech/articles/mini-r1-reproduce-deepseek-r1-aha-moment-a-rl-tutorial-7418.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/open-r1/mini-r1-contdown-game>)

Author: Philipp Schmid

Published: 2025-01-31T10:29:40Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [deepseek](<https://devfeed.tech/topics/deepseek.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Nvidia](<https://devfeed.tech/topics/nvidia.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [development](<https://devfeed.tech/tags/development.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [open](<https://devfeed.tech/tags/open.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [train](<https://devfeed.tech/tags/train.md>), [training](<https://devfeed.tech/tags/training.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

## AI overview

This tutorial reproduces a small version of DeepSeek-R1's reported reinforcement-learning "aha moment" by training an open model with Group Relative Policy Optimization (GRPO) to solve the Countdown numbers game. It covers self-verification, search behavior, distributed training with DeepSpeed and vLLM, and a four-NVIDIA-H100-GPU setup.

## Source excerpt

The release of Deepseek R1 shocked the industry. Why? Well, DeepSeek-R1 is an open model that rivals OpenAI's o1 in complex reasoning tasks, introduced using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach. They not only released the model, but also a research paper on how they did it. In the paper they described an "aha moment" when using pure RL to train the model.