# Grab Bench: Evaluating AI on Grab-shaped production work

DevFeed: [Grab Bench: Evaluating AI on Grab-shaped production work](<https://devfeed.tech/articles/grab-bench-evaluating-ai-on-grab-shaped-production-work-1248.md>)

Original publisher: [Read original article](<https://engineering.grab.com/grab-bench-evaluating-ai>)

Author: Christian Coffrant

Published: 2026-08-12T00:00:00Z

Content type: article

Language: en

Sources: [Grab Tech](<https://devfeed.tech/sources/grab-tech.md>)

Topics: [Human-AI evaluation](<https://devfeed.tech/topics/human-ai-evaluation.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code](<https://devfeed.tech/tags/code.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [product](<https://devfeed.tech/tags/product.md>), [production](<https://devfeed.tech/tags/production.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sql](<https://devfeed.tech/tags/sql.md>)

## AI overview

Grab Bench is a configurable evaluation harness for AI systems performing production-shaped tasks. It evaluates failures in SQL generation, tool calling, profile updates, and coding agents using task plugins, row-level records, deterministic scorers, or LLM judges.

## Source excerpt

Introduction What worried us wasn't the hallucination, it was the subtle plausibility. Answers an engineer could easily read past and accept: a right-looking Structured Query Language (SQL) query, a plausible tool call, an innocent profile update, or a patch that satisfied the surface tests. When we analyzed the row-level failures, a clear pattern emerged: SQL generation: kept the query shape but changed the underlying metric. Tool calling: selected the right tool family but drifted on parameters. Profile updates: cited every event instead of only the evidence that supported the claim. Coding agents: passed visible tests while missing a hidden stateful invariant. Grab Bench bridges this exact gap. Grab Bench is a configurable eval (evaluation) harness for artificial intelligence (AI) systems on Grab-shaped work. It runs model providers through task plugins, records one row per case/model pair, and uses deterministic scorers or large language model (LLM) judges depending on the task. We treat the eval like software: version it, run baselines, keep score records, and make the failure modes visible enough for a team to debug. This write-up focuses on the design choices behind that work. The problem: plausible is not correct Public leaderboards are still useful; we read them too. They just answer a different question. A product team needs to know whether a model can preserve a metric definition, obey an internal tool contract, stay cautious with weak evidence, or make a code change without breaking behaviour hidden from the prompt. The hard part is that real examples are rarely reusable as-is. Production traces, schemas, user records, and internal workflows need protection. So the benchmark has to preserve the shape of the work without depending on the work itself. That constraint shaped Grab Bench from the beginning. Some surfaces stay internal. Others use synthetic or redacted cases. Either way, the case has to keep the thing that makes the work hard: metric faithfuln