# Differential Transformer V2

DevFeed: [Differential Transformer V2](<https://devfeed.tech/articles/differential-transformer-v2-7349.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/microsoft/diff-attn-v2>)

Author: Li Dong

Published: 2026-01-20T03:20:57Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [GitHub](<https://devfeed.tech/topics/github.md>)

Tags: [comparison](<https://devfeed.tech/tags/comparison.md>), [github](<https://devfeed.tech/tags/github.md>), [llm](<https://devfeed.tech/tags/llm.md>), [speed](<https://devfeed.tech/tags/speed.md>), [techniques](<https://devfeed.tech/tags/techniques.md>)

## AI overview

Differential Transformer V2 compares with V1 and modifies grouped-query attention by doubling query heads while keeping the number of key-value heads unchanged. The design targets memory-bound LLM decoding, maintains compatibility with standard Transformer attention speed, avoids custom attention kernels, and reduces output-projection parameters and floating-point operations under an equal-query-dimension comparison.

## Source excerpt

We compare DIFF V2 with DIFF V1 below: (For simplicity, we omit the batch dimension and assume that both the input and output of the following are three-dimensional tensors . Heads belonging to the same GQA group are arranged contiguously in the output) Note DIFF V2 subtracts two heads that are in the same GQA group, which means they share the same key and value. This is crucial to performance. See design ablations section and Github code.