# OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues

DevFeed: [OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues](<https://devfeed.tech/articles/openre-bench-a-benchmark-harness-for-multi-agent-requirements-engineering-with-mined-github-issues-129172.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13925>)

Author: Haowei Cheng, Milhan Kim, Bowen Jiang, Weixing Zhang, Yuhong Fu, Zhixu Liu, Naoyasu Ubayashi, Hironori Washizaki

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [api](<https://devfeed.tech/tags/api.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [docker](<https://devfeed.tech/tags/docker.md>), [docker-image](<https://devfeed.tech/tags/docker-image.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [environment](<https://devfeed.tech/tags/environment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [feature](<https://devfeed.tech/tags/feature.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [github](<https://devfeed.tech/tags/github.md>), [github-issue](<https://devfeed.tech/tags/github-issue.md>), [github-issues](<https://devfeed.tech/tags/github-issues.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [requirements-engineering](<https://devfeed.tech/tags/requirements-engineering.md>)

## AI overview

OpenRE-Bench provides a shared, reproducible way to evaluate multi-agent requirements-engineering frameworks, which otherwise differ in agent roles, negotiation policies, outputs, runtimes, and metrics. Its run harness standardizes inputs, phase artifacts, and metrics; a single adapter lets a framework join without changing cases or other adapters. A reproducible GitHub issue-mining pipeline and versioned case suite support corpus growth and stable comparisons. In the demonstration, 3,000 issues from five projects yielded five mined cases, combined with five hand-built cases in v1.0; MARE and iReDev (paper-faithful reimplementations) and QUARE were run under one protocol. The release includes a pinned environment, Docker image, frozen API snapshots, 651 tests, and 1,396 filtered issues behind the mined cases.

## Source excerpt

arXiv:2611.13925v1 Announce Type: new Abstract: OpenRE-Bench is an open-source benchmarking infrastructure for large language model (LLM) based multi-agent requirements-engineering (RE) frameworks. Such frameworks are increasingly common, but they use different agent roles, negotiation policies, and output shapes, and are typically evaluated on their own runtimes, metrics, and small sets of curated system descriptions, which makes fair, reproducible comparison difficult. OpenRE-Bench addresses this with four reusable pieces: a shared run harness with one input, phase-artifact, and metric contract; an adapter mechanism through which a framework joins by implementing a single adapter, leaving cases, metrics, and other adapters untouched; a reproducible GitHub issue-mining pipeline that grows the corpus from real feature requests; and a versioned case suite. As a demonstration, we mine 3,000 fetched issues across five projects (three languages, five domains) into five cases, package them with five hand-built cases as the compact v1.0 suite, and run three heterogeneous frameworks (MARE and iReDev as paper-faithful reimplementations, plus QUARE) under one protocol. OpenRE-Bench originated as the evaluation back-end of our QUARE study and has since been reused by two further studies of our group. It is released with a pinned environment, a Docker image, frozen API snapshots, a 651-test suite, and the full pool of 1,396 filtered issues behind the mined cases.