# Beyond Compressing Everything: Contrastive Memory Learning for Selective Context Compression

DevFeed: [Beyond Compressing Everything: Contrastive Memory Learning for Selective Context Compression](<https://devfeed.tech/articles/beyond-compressing-everything-contrastive-memory-learning-for-selective-context-compression-132229.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16384>)

Author: Sikui Zhang, Haowei Liu, Ziyun Gan, Chunfeng Yuan, Guangze Gao, Nian Liu, Boyu Cai, Bo Li, Xiaolei Lv, Jun Gao, Bing Li, Weiming Hu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [compression](<https://devfeed.tech/tags/compression.md>), [context-compression](<https://devfeed.tech/tags/context-compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [learning](<https://devfeed.tech/tags/learning.md>), [memory](<https://devfeed.tech/tags/memory.md>)

## AI overview

CoMem compresses context selectively for each query, aiming to retain relevant evidence within a limited number of memory slots. It trains document-and-query-conditioned memories with Contrastive Compressor Learning and mines hard negatives by pairing one document with different queries, encouraging distinct evidence selection for each. It also enables bidirectional attention among memory slots to address information-flow limits from causal masking. The abstract reports that CoMem outperforms existing context compression methods across compression ratios on question answering and RULER-NIAH benchmarks; it provides no numerical results or further evaluation details.

## Source excerpt

arXiv:2611.16384v1 Announce Type: cross Abstract: As large language models (LLMs) advance toward increasingly long inputs, soft context compression offers a promising approach to efficient inference by converting long inputs into compact memory slots. Under limited memory budgets, effective context compression remains challenging because task-relevant evidence varies substantially across queries, ranging from localized evidence to multiple distant spans or information dispersed throughout the full document. Instead of compressing everything indiscriminately, we propose CoMem, a query-guided selective context compression framework that learns memory representations conditioned on both the document and the query. CoMem develops Contrastive Compressor Learning to achieve query-sensitive yet context-preserving memory representations. To inject a strong prior over query-dependent evidence distributions, we devise a same-context hard negative mining strategy by pairing a single document with multiple queries, requiring the compressor to selectively retain information according to each query's distinct evidence needs. To address the restricted information flow imposed by causal masking, we further enable bidirectional attention among memory slots. Experiments demonstrate that CoMem consistently outperforms existing context compression methods across compression ratios on question answering and RULER-NIAH benchmarks. Our code is available at https://github.com/scar-on/CoMem.