# ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

DevFeed: [ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation](<https://devfeed.tech/articles/abacus-a-benchmark-for-image-editing-models-to-assess-object-category-and-counting-under-scene-conservation-94402.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.28316>)

Author: Aymen Lassoued, Angelique Loesch, Romaric Audigier

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [image editing](<https://devfeed.tech/topics/image-editing.md>), [Image](<https://devfeed.tech/topics/image.md>), [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [counts](<https://devfeed.tech/tags/counts.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [image-editing](<https://devfeed.tech/tags/image-editing.md>), [models](<https://devfeed.tech/tags/models.md>), [object](<https://devfeed.tech/tags/object.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

## AI overview

ABACUS evaluates image editors' ability to add, remove, or change an exact number of objects while preserving the rest of a scene. Rather than relying on holistic similarity scores or VLM counting, its protocol checks edits instance by instance against panoptic ground truth confirmed by detector consensus; a VLM is used only to confirm categories in rare ambiguous crops. The authors report substantial numerical editing failures that increase with scene complexity and requested object count, errors that prior benchmarks may overlook or misjudge.

## Source excerpt

arXiv:2610.28316v1 Announce Type: cross Abstract: Instruction-based image editing models can now add, remove a specified number of objects or change their category while preserving the scene. Yet these models often fail to produce the correct number, and these failures grow sharply as scenes become more complex and the requested count increases. Current benchmarks largely miss this problem: they score edits with holistic similarity metrics or with vision-language model (VLM) judges that are themselves poor at counting, so numerical errors go unnoticed or are graded incorrectly. As a result, the true numerical ability of today's editing models has never been accurately measured. We address this issue with ABACUS, a new benchmark, evaluation protocol and dedicated scoring metric for numerical image editing. Our protocol is judge-free: no VLM or MLLM scores an edit or counts objects (a VLM only confirms the category of rare ambiguous crops). Instead, it verifies each edit at the instance level against panoptic ground truth confirmed by detector consensus, checking both that the exact requested count was achieved and that the rest of the scene was preserved, across the add, remove and change operations. By evaluating leading image editors, our method reveals substantial numerical editing failures that worsen with increasing scene complexity and object count; failures that prior benchmarks often overlooked or misjudged.