# SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

DevFeed: [SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories](<https://devfeed.tech/articles/slackbench-benchmarking-agents-on-collaborative-projects-grounded-in-real-code-repositories-118332.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05512>)

Author: Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [academic](<https://devfeed.tech/tags/academic.md>), [academic-research](<https://devfeed.tech/tags/academic-research.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-harnesses](<https://devfeed.tech/tags/agent-harnesses.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [best](<https://devfeed.tech/tags/best.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [projects](<https://devfeed.tech/tags/projects.md>), [repositories](<https://devfeed.tech/tags/repositories.md>)

## AI overview

SlackBench evaluates agents on collaborative research and engineering work that requires reasoning across workplace conversations and changing repository state. Built from privacy-preserving reenactments of academic projects, it provides reproducible environments with multi-party dialogue, evolving decisions, and related GitHub code and Google Docs. Its 140 queries cover cross-source search, abstention on unanswerable or false-premise questions, goal-directed summarization, and inference from long conversations. Tasks can require finding relevant channels and threads, checking repository state, and reconciling stale or conflicting evidence. Across 33 frontier models and agent harnesses, the top system scored around 80%, while most scored below 66%. The abstract presents the benchmark as an extensible way to evaluate agents in collaborative settings.

## Source excerpt

arXiv:2611.05512v1 Announce Type: new Abstract: AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement. SlackBench thus provides an extensible framework for evaluating agents in collaborative work environments. The SlackBench dataset is available at https://huggingface.co/datasets/HuanzhiMao/SlackBench .