# DiScoFormer: One transformer for density and score, across distributions

DevFeed: [DiScoFormer: One transformer for density and score, across distributions](<https://devfeed.tech/articles/discoformer-one-transformer-for-density-and-score-across-distributions-7083.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/allenai/discoformer>)

Author: Kyle Wiggers

Published: 2026-06-29T18:02:48Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AI, ML & Data Engineering](<https://devfeed.tech/topics/ai-ml-data-engineering.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

## AI overview

DiScoFormer is a transformer model for estimating a distribution's density and score from a set of data points in one forward pass, without retraining for each distribution. It uses cross-attention and coupled density and score output heads, with their mathematical relationship providing a label-free consistency loss.

## Source excerpt

Many problems in machine learning and the sciences come down to the same task: you have a collection of data points and want to recover the distribution they came from--which values are common, and which are rare. Pinning down that distribution means estimating two quantities: the distribution's density and, more useful as dimensionality grows, its score. The density is the smooth version of a histogram--high where points cluster and low where they're scarce.