# Teaching Sidekick to say no: automated data curation with LLM judge consensus

DevFeed: [Teaching Sidekick to say no: automated data curation with LLM judge consensus](<https://devfeed.tech/articles/teaching-sidekick-to-say-no-automated-data-curation-with-llm-judge-consensus-1616.md>)

Original publisher: [Read original article](<https://shopify.engineering/sidekick-curation>)

Author: Shuang Xie

Published: 2026-06-15T19:30:00Z

Content type: article

Language: en

Sources: [Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering.md>), [Shopify Engineering - Shopify Engineering](<https://devfeed.tech/sources/shopify-engineering-shopify-engineering.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [foundation-models](<https://devfeed.tech/topics/foundation-models.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [feature-engineering](<https://devfeed.tech/tags/feature-engineering.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [shopify](<https://devfeed.tech/tags/shopify.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

Shopify Engineering describes how Sidekick's production training data failed to teach refusal behavior because it contained only successful merchant queries. The article presents automated data curation using consensus among LLM judges to identify blind spots and improve an AI assistant built from an outer planner and specialized skill models.

## Source excerpt

Production training data only captures successful queries; it can't teach a model when to say no. We built an automated curation pipeline using LLM judge consensus to close that gap.