# llm-as-a-judge

Published articles for llm-as-a-judge.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## LLM-судья для нейроразбора резюме на hh

DevFeed: [LLM-судья для нейроразбора резюме на hh](<https://devfeed.tech/articles/llm-hh-30669.md>)

Original publisher: [Read original article](<https://habr.com/ru/companies/hh/articles/1050174/>)

Author: Mentalitet (hh.ru)

Published: 2026-06-22T05:31:17Z

Content type: article

Language: ru

Sources: [HeadHunter RU](<https://devfeed.tech/sources/headhunter-ru.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-quality](<https://devfeed.tech/tags/ai-quality.md>), [eval](<https://devfeed.tech/tags/eval.md>), [hh-ru](<https://devfeed.tech/tags/hh-ru.md>), [hr](<https://devfeed.tech/tags/hr.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-10b5a83db3c5](<https://devfeed.tech/tags/llm-10b5a83db3c5.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [llm-as-judge](<https://devfeed.tech/tags/llm-as-judge.md>), [quality-assurance](<https://devfeed.tech/tags/quality-assurance.md>), [tag-61cd5a476b1d](<https://devfeed.tech/tags/tag-61cd5a476b1d.md>), [tag-a939ae104bb0](<https://devfeed.tech/tags/tag-a939ae104bb0.md>), [tag-b6914c0b0244](<https://devfeed.tech/tags/tag-b6914c0b0244.md>), [tag-d96e238f1419](<https://devfeed.tech/tags/tag-d96e238f1419.md>), [tag-e7e38deac50c](<https://devfeed.tech/tags/tag-e7e38deac50c.md>)

### AI overview

This article describes how hh.ru designed an LLM judge for its resume-analysis feature in an AI assistant. It explains why reliable evaluation requires its own rubrics, datasets, quality metrics, operating-cost considerations, and controls for hallucinations and interpretation errors. The system assesses candidates against criteria relevant to a specific position, which may be extracted from a vacancy or defined by an employer.

### Source excerpt

Создать LLM-судью легко. Гораздо сложнее сделать так, чтобы его оценкам можно было доверять. Мы убедились в этом на практике при разработке нейроразбора резюме для ИИ-помощника hh.ru. Быстро выяснилось, что хороший LLM-судья -- это отдельный продукт со своими рубриками, датасетами, метриками качества и стоимостью эксплуатации. Меня зовут Женя Орлов, я LLM Eval Lead. В этой статье расскажу, как мы проектировали систему оценки для нейроразбора резюме, почему отказались от наивных подходов и какие выводы сделали по ходу разработки. Читать далее

## Using LLM-as-a-Judge For Evaluation: A Complete Guide

DevFeed: [Using LLM-as-a-Judge For Evaluation: A Complete Guide](<https://devfeed.tech/articles/using-llm-as-a-judge-for-evaluation-a-complete-guide-18793.md>)

Original publisher: [Read original article](<https://hamel.dev/blog/posts/llm-judge/>)

Author: Hamel Husain

Published: 2024-10-29T07:00:00Z

Content type: tutorial

Language: en

Sources: [Hamel Husain](<https://devfeed.tech/sources/hamel-husain.md>)

Topics: [LLMs](<https://devfeed.tech/topics/llms.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>)

Tags: [evals](<https://devfeed.tech/tags/evals.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [guide](<https://devfeed.tech/tags/guide.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [llms](<https://devfeed.tech/tags/llms.md>)

### AI overview

A step-by-step guide to evaluating AI outputs with LLM-as-a-judge. It argues against uncalibrated multi-score metrics and recommends involving a principal domain expert through a technique called "Critique Shadowing."

### Source excerpt

Earlier this year, I wrote Your AI product needs evals. Many of you asked, "How do I get started with LLM-as-a-judge?" This guide shares what I've learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in Data Ever spend weeks building an AI system, only to realize you have no idea if it's actually working? You're not alone. I've noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs: Too Many Metrics: Creating numerous measurements that become unmanageable. Arbitrary Scoring Systems: Using uncalibrated scales (like 1-5) across multiple dimensions, where the difference between scores is unclear and subjective. What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently. Ignoring Domain Experts: Not involving the people who understand the subject matter deeply. Unvalidated Metrics: Using measurements that don't truly reflect what matters to the users or the business. The result? Teams end up buried under mountains of metrics or data they don't trust and can't use. Progress grinds to a halt. Everyone gets frustrated. For example, it's not uncommon for me to see dashboards that look like this: An illustrative example of a bad eval dashboard Tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process (I'll discuss why later). In this post, I'll show you how to avoid these pitfalls. The solution is to use a technique that I call "Critique Shadowing". Here's how to do it, step by step. Step 1: Find The Principal Domain Expert In most organizations there is usually one (maybe two) key individuals whose judgment is crucial for the success of your AI product. These are the people with deep domain expertise or represent your target users. Identifying and involving this Principal Domain Expert early in the process is critical. Why is finding the right domain expert so important? They Set the Standard: This person not only defi