# Hamel Husain Notes on applied AI engineering, machine learning, and data science. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Design AI Products for Verification Before Building Evals DevFeed: [Design AI Products for Verification Before Building Evals]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2026-06-29T07:00:00Z Content type: opinion Language: en Sources: [Hamel Husain]() Topics: [Artificial Intelligence](), [data](), [dashboards](), [SQL]() Tags: [ai](), [ai-evals](), [data-agents](), [evals](), [interface](), [llms](), [techniques](), [verification]() ### AI overview The article argues that products described as difficult to evaluate often make their outputs difficult for users to verify. Using AI data agents as an example, it recommends providing checkable artifacts--such as source comparisons, precise metric definitions, breakdowns, SQL, and uncertainty notes--before focusing on eval design. ### Source excerpt For the past 3 years, AI evals have been my professional focus.1 The most common objection I hear to evals is "our product is hard to eval". This objection is a product smell. Artifacts that are hard for you to verify are often hard for users too. In the worst case, users have to redo the work from scratch to verify the output. More importantly, designing your product for ease of verification should come before building evals. In this post, I'll walk through three products I advised on that faced this issue. I'll also show before and after sketches to demonstrate design principles. After these examples, I'll discuss how to apply this general pattern to your product. Example 1: the AI data agent Almost every company I've worked with builds an internal AI data agent. You ask it a business question, like what was net revenue for Product A last quarter, and it finds relevant data sources, runs the queries, and provides an answer. The goal of this agent is to reduce dependency on data analysts. A common mistake when building AI data agents is to make the answer the only output, as illustrated below. Data Agent What was net revenue for Product A last quarter? Net revenue for Product A last quarter was $4.21M. Ask anything about your business...➤ Since the only output is the answer, there is nothing here to check. In the sketch above, the user has no way to verify the answer beyond redoing work.2 A better design is to provide the user with checkable artifacts, informed by how a domain expert might validate the output. Here are techniques I use to validate metrics as a data scientist: Compare the quantity and any intermediate calculations against a trusted source, like a vetted dashboard or report, or a similar analysis a colleague has already vetted.3 Confirm the metric definition precisely. A number like net revenue can include or exclude things like returns and discounts. Sanity-check a related quantity. If I can't verify the number directly, I pull a related number that s ## The Revenge of the Data Scientist DevFeed: [The Revenge of the Data Scientist]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2026-03-26T07:00:00Z Content type: opinion Language: en Sources: [Hamel Husain]() Topics: [Data Science](), [Large Language Model](), [AI Development](), [foundation-models](), [SIEM, Security, Observability]() Tags: [ai](), [codex](), [data-science](), [evals](), [llm](), [llms](), [observability](), [openai](), [opinion]() ### AI overview An opinion article argues that data science remains central to AI development even as foundation-model APIs let teams integrate AI without relying on data scientists and machine learning engineers for every project. It emphasizes experimentation, generalization testing, debugging stochastic systems, metrics, and observability in AI harnesses. ### Source excerpt Is the heyday of the data scientist over? The Harvard Business Review once called it "The Sexiest Job of the 21st Century."1 In tech, data scientist roles were often among the best paid.2 The job also demanded an unusual mix of skills: Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. -- JosH100 (@josh_wills) May 3, 2012 In addition to creating a high-barrier to entry, these skills enabled data scientists to build predicitive models, measure casuality and find patterns in data. Of these, predicitive modeling paid best. Companies later peeled that work off into a new title: Machine Learning Engineer ("MLE").3 For years, shipping AI meant keeping data scientists and MLEs on the critical path. With LLMs, this stopped being the default. Foundation-model APIs now allow teams to integrate AI independently. Getting cut out of the loop rattled data scientists and MLEs I know. If the company no longer needs you to ship AI, it is fair to wonder whether the job still has the same upside. The harsher story people tell themselves: unless you are pretraining at a foundation-model lab, you are not where the action is. In my opinion, training models was never most of the job. The bulk of the work is setting up experiments to test how well the AI generalizes to unseen data, debugging stochastic systems, and designing good metrics. Calling an LLM over an API does not make this work go away. I recently gave a talk titled "The Revenge of the Data Scientist" at PyAI Conf to make that case with examples rather than assertion alone. Below is an annotated version of that presentation. The Harness Is Data Science OpenAI published a blog post on harness engineering that I recommend reading. They describe how Codex worked on a software project for months, autonomously, with agents developing code bounded by a harness of tests and specifications. One detail in that blog post's description of the harness i ## Evals Skills for Coding Agents DevFeed: [Evals Skills for Coding Agents]() Original publisher: [Read original article]() Author: Shreya Shankar Published: 2026-03-02T08:00:00Z Content type: release Language: en Sources: [Hamel Husain]() Topics: [AI-assisted coding](), [Artificial Intelligence](), [coding]() Tags: [agents](), [ai](), [ai-evals](), [audit](), [coding-agents](), [evals](), [review](), [skills](), [traces]() ### AI overview Shreya Shankar and the author publish evals skills for AI product evaluations. The collection routes users to skills for auditing eval pipelines, discovering errors from traces, generating synthetic test inputs, designing and validating LLM judges, and evaluating RAG quality. ### Source excerpt Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1. Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you've looked at your data. These skills help you avoid common mistakes we've seen helping 50+ companies and teaching students in our AI Evals course. Why skills for evals There are many easily avoidable footguns in evals. These skills help you avoid them. evals-start is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two: eval-audit, if you already have an eval pipeline. It inspects your setup and recommends next steps. 2 error-discovery, if you have traces but haven't analyzed them yet. It builds a customized annotation interface and helps you sample traces intelligently. Shreya does a live walkthrough of using this skill here. The skills Install the skills: npx skills add https://github.com/ai-evals-course/evals-skills Then give your agent this prompt: Run the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. If you're experienced with evals, skip the router and pick the skill you need: Skill What it does evals-start Entry point. Routes to the skill that matches your situation eval-audit Audit an eval pipeline and surface problems with prioritized severity error-discovery Build a review app, select diverse samples, and organize your notes into failure modes generate-synthetic-data Create diverse synthetic test inputs using dimension-based tuple generation write-judge-prompt Design LLM-as-Judge evaluators for subjective quality criteria validate-evaluator Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction evaluate-rag Evaluate retrieval and generation quality in RAG pipel ## Why I Stopped Using nbdev DevFeed: [Why I Stopped Using nbdev]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2026-01-18T08:00:00Z Content type: opinion Language: en Sources: [Hamel Husain]() Topics: [Artificial Intelligence](), [ai-coding](), [jupyter](), [Python](), [Development]() Tags: [ai](), [ai-coding](), [developer](), [jupyter](), [jupyter-notebooks](), [nbdev](), [python](), [software](), [tools]() ### AI overview The author explains why they stopped using nbdev, a literate-programming development environment built around Jupyter notebooks. They argue that AI coding tools work less effectively with nbdev's notebook-to-library workflow, while AI can now help understand and maintain documentation separately from code. The author concludes that disciplined development and documentation depend more on developer effort than on the tooling environment. ### Source excerpt Programmers love to proclaim they've found the best tool. Paul Graham called Lisp his "secret weapon." DHH described Ruby as "a magical glove that just fit my brain perfectly." Pieter Levels ships million-dollar products with vanilla PHP and jQuery. These declarations aren't about the languages themselves. They're about developers finding tools that fit how they think. When the environment clicks, you move fast. I had that experience with nbdev, a development environment for literate programming that I helped build and maintain1. I created hundreds of projects with it and was one of its biggest proponents. Today, I no longer use it. AI coding tools changed the trade-offs. Fighting the AI The beauty of nbdev is its workflow. You write code, documentation and tests in one source of truth: Jupyter notebooks. Afterwards, these notebooks are transpiled into a Python library and documentation website. This workflow is idiosyncratic. AI coding tools, trained on vast amounts of conventional source code, get confused. They struggle to differentiate between editing the notebook and editing the final source code. It feels like fighting the AI instead of working with it. I write software to solve problems, not to write code. I want to work in an environment where AI has the highest chance of success. With nbdev, I was swimming upstream. Some argue that AI tools encourage lazy thinking: that without guardrails, developers skip the hard work of breaking problems into steps. But thinking step-by-step is a human skill. Notebooks don't force you to write clean code. AI tools don't force you to think carefully. Discipline comes from the developer, not the environment. Tools Don't Matter As Much As I Thought A central promise of literate programming is better documentation. By keeping code and docs in one place, you reduce the chance they become stale. Strangely, many nbdev projects lacked sufficient documentation for my taste. Sometimes, this helped me learn a codebase by contributin ## Selecting The Right AI Evals Tool DevFeed: [Selecting The Right AI Evals Tool]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2025-10-01T07:00:00Z Content type: article Language: en Sources: [Hamel Husain]() Topics: [Artificial Intelligence](), [AI Engineering](), [Developer experience](), [LangChain](), [phoenix]() Tags: [ai](), [ai-engineering](), [ai-evals](), [analysis](), [developer-experience](), [evals](), [langchain](), [phoenix](), [quality]() ### AI overview The article examines how to select AI evaluation tools, arguing that no single tool is best for every team. It compares approaches from LangSmith, Braintrust, and Arize Phoenix through a shared assignment and highlights workflow, developer experience, SDK ergonomics, documentation, integrations, and human-in-the-loop support as selection criteria. ### Source excerpt Over the past year, I've focused heavily on AI Evals, both in my consulting work and teaching. A question I get constantly is, "What's the best tool for evals?". I've always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately. Having used many of the popular eval tools, I can genuinely say that no single one is superior in every dimension. The "best" tool depends on your team's skillset, technical stack, and maturity. Instead of a feature-by-feature comparison, I think it's more valuable to show you how a panel of data scientists skilled in evals assesses these tools. As part of my AI Evals course, we had three of the most dominant vendors--Langsmith, Braintrust, and Arize Phoenix complete the same homework assignment. This gave us a unique opportunity to see how they tackle the exact same challenge. We recorded the entire process and live commentary, which is available below. We think this might be helpful in learning about the kinds of things you should consider when selecting a tool for your team. Thanks to Shreya Shankar and Bryan Bischof for serving as the panelists (alongside me). Langsmith With Harrison Chase, CEO of LangChain. Braintrust With Wayde Gilliam, former developer relations at Braintrust. Arize Phoenix With SallyAnn DeLucia, Technical AI Product Leader at Arize. Criteria for Assessing AI Evals Tools Here are themes that consistently surfaced during our review. 1. Workflow and Developer Experience Reducing friction is more important than any single feature. Concretely, you should be mindful of the time it takes to go from observing a failure to iterating on a solution. For example, we appreciated the ability to go from viewing a single trace to experimenting with that same trace in a playground. For some teams with data-science backgrounds, a note ## AI Evals: Common Questions About Model and Product Evaluation DevFeed: [AI Evals: Common Questions About Model and Product Evaluation]() Original publisher: [Read original article]() Author: Shreya Shankar Published: 2025-05-28T07:00:00Z Content type: tutorial Language: en Sources: [Hamel Husain]() Topics: [Artificial Intelligence](), [Benchmark](), [Code]() Tags: [ai](), [ai-evals](), [benchmark](), [code](), [evals](), [llms](), [retrieval](), [tools]() ### AI overview This FAQ explains AI evaluations as tests for determining whether an AI system meets user and business goals. It distinguishes model benchmarks from product evaluations, which measure a specific product across its model, prompts, retrieval, tools, and application code. ### Source excerpt This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. For a guided path through the rest of our evals work, use the AI evals topic hub. 👉 Want to learn more about AI Evals? Check out our AI Evals course. It's a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈 Getting Started & Fundamentals Q: What are AI Evals? AI evals are tests that tell you whether an AI system is doing what you want. They give your team feedback when the product drifts from user needs or business goals. The failures they catch also become data you can use to improve the system. More formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or structured review. Most AI products need several evals because they can fail in different ways. When you hear the word "evals," it usually refers to one of two things: model benchmarks or product evals. Model benchmarks Model benchmarks compare general-purpose models on shared tasks. Model providers publish these benchmark results when they release new models. Common examples include GPQA Diamond for graduate-level science reasoning, Terminal-Bench for agents doing complex work in command-line environments, and MMLU for knowledge and reasoning across a wide range of subjects. These scores can help you choose a promising model as a starting point. To assess quality on your own tasks you need product evals, which we discuss next. Product evals Product evals measure whether your specific AI product does what you want it to do. They turn your judgment about what a good product experience looks like into metrics you can track. Product evals encompass all components of your product, including the model, prompts, retrieval, tools, and application code. This flavor of ## A Field Guide to Improving AI Products Through Measurement and Iteration DevFeed: [A Field Guide to Improving AI Products Through Measurement and Iteration]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2025-03-24T07:00:00Z Content type: tutorial Language: en Sources: [Hamel Husain]() Topics: [AI Development](), [Artificial Intelligence](), [synthetic-data]() Tags: [ai](), [ai-development](), [analysis](), [dashboards](), [evaluation](), [experiments](), [guide](), [llms](), [synthetic-data]() ### AI overview This field guide argues that AI teams should prioritize measurement and iteration over tools and frameworks. It presents error analysis as a high-value practice and discusses data viewers, domain experts, synthetic data, evaluation trust, and experiment-focused roadmaps. ### Source excerpt Most AI teams focus on the wrong things. Here's a common scene from my consulting work: AI TEAM Here's our agent architecture - we've got RAG here, a router there, and we're using this new framework for... ME [Holding up my hand to pause the enthusiastic tech lead.] "Can you show me how you're measuring if any of this actually works?" ... Room goes quiet This scene has played out dozens of times over the last two years. Teams invest weeks building complex AI systems, but can't tell me if their changes are helping or hurting. This isn't surprising. With new tools and frameworks emerging weekly, it's natural to focus on tangible things we can control - which vector database to use, which LLM provider to choose, which agent framework to adopt. But after helping 30+ companies build AI products, I've discovered the teams who succeed barely talk about tools at all. Instead, they obsess over measurement and iteration. In this post, I'll show you exactly how these successful teams operate. You'll learn: How error analysis consistently reveals the highest-ROI improvements Why a simple data viewer is your most important AI investment How to empower domain experts (not just engineers) to improve your AI Why synthetic data is more effective than you think How to maintain trust in your evaluation system Why your AI roadmap should count experiments, not features I'll explain each of these topics with real examples. While every situation is unique, you'll see patterns that apply regardless of your domain or team size. Let's start by examining the most common mistake I see teams make - one that derails AI projects before they even begin. 1. The Most Common Mistake: Skipping Error Analysis The "tools first" mindset is the most common mistake in AI development. Teams get caught up in architecture diagrams, frameworks, and dashboards while neglecting the process of actually understanding what's working and what isn't. One client proudly showed me this evaluation dashboard: The kind of das ## Building an Audience Through Technical Writing: Strategies and Mistakes DevFeed: [Building an Audience Through Technical Writing: Strategies and Mistakes]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2024-11-30T08:00:00Z Content type: tutorial Language: en Sources: [Hamel Husain]() Topics: [Learning](), [Artificial Intelligence]() Tags: [building](), [consistency](), [developers](), [how-to](), [marketing](), [personal-experience](), [technical](), [writing]() ### AI overview An advice article on building an audience through technical writing. It recommends thoughtfully engaging with other people's work, adding meaningful insights, publishing consistently, and participating in discussions rather than relying only on promotional content. ### Source excerpt People often find me through my writing on AI and tech. This creates an interesting pattern. Nearly every week, vendors reach out asking me to write about their products. While I appreciate their interest and love learning about new tools, I reserve my writing for topics that I have personal experience with. One conversation last week really stuck with me. A founder confided, "We can write the best content in the world, but we don't have any distribution." This hit home because I used to think the same way. Let me share what works for reaching developers. Companies and individuals alike often skip the basics when trying to grow their audience. These are proven approaches I've seen succeed, both in my work and in others' efforts to grow their audience in the AI space. 1. Build on Great Work Here's something surprising: few people take the time to thoughtfully engage with others' work in our field. But when you do, amazing things happen naturally. For example, here are some recent posts I've enjoyed that present opportunities to engage with others: Shreya Shankar's DocETL Eugene Yan's work on AlignEval Ben Claive's work on rerankers Jeremy Howard's work on llms.txt In the above examples, you could share how their ideas connect with what you've built. You could add additional case studies and real-world insights. If you deeply engage with someone's work and add your insights, they often share your content with their audience. Not because you asked, but because you've added something meaningful to their work. Swyx has written a great post on how to do this effectively. The key is authenticity. Don't do this just for marketing--do it because you're genuinely interested in learning from others and building on their ideas. It's not hard to find things to be excited about. I'm amazed by how few people take this approach. It's both effective and fun. 2. Show Up Consistently I see too many folks blogging or posting once every few months and wondering why they're not getting tr ## Using LLM-as-a-Judge For Evaluation: A Complete Guide DevFeed: [Using LLM-as-a-Judge For Evaluation: A Complete Guide]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2024-10-29T07:00:00Z Content type: tutorial Language: en Sources: [Hamel Husain]() Topics: [LLMs](), [Artificial Intelligence](), [dashboards]() Tags: [evals](), [evaluation](), [guide](), [llm](), [llm-as-a-judge](), [llms]() ### AI overview A step-by-step guide to evaluating AI outputs with LLM-as-a-judge. It argues against uncalibrated multi-score metrics and recommends involving a principal domain expert through a technique called "Critique Shadowing." ### Source excerpt Earlier this year, I wrote Your AI product needs evals. Many of you asked, "How do I get started with LLM-as-a-judge?" This guide shares what I've learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in Data Ever spend weeks building an AI system, only to realize you have no idea if it's actually working? You're not alone. I've noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs: Too Many Metrics: Creating numerous measurements that become unmanageable. Arbitrary Scoring Systems: Using uncalibrated scales (like 1-5) across multiple dimensions, where the difference between scores is unclear and subjective. What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently. Ignoring Domain Experts: Not involving the people who understand the subject matter deeply. Unvalidated Metrics: Using measurements that don't truly reflect what matters to the users or the business. The result? Teams end up buried under mountains of metrics or data they don't trust and can't use. Progress grinds to a halt. Everyone gets frustrated. For example, it's not uncommon for me to see dashboards that look like this: An illustrative example of a bad eval dashboard Tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process (I'll discuss why later). In this post, I'll show you how to avoid these pitfalls. The solution is to use a technique that I call "Critique Shadowing". Here's how to do it, step by step. Step 1: Find The Principal Domain Expert In most organizations there is usually one (maybe two) key individuals whose judgment is crucial for the success of your AI product. These are the people with deep domain expertise or represent your target users. Identifying and involving this Principal Domain Expert early in the process is critical. Why is finding the right domain expert so important? They Set the Standard: This person not only defi ## An Open Course on LLMs, Led by Practitioners DevFeed: [An Open Course on LLMs, Led by Practitioners]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2024-07-29T07:00:00Z Content type: release Language: en Sources: [Hamel Husain]() Topics: [LLMs](), [Tutorial](), [Fine-tuning](), [Retrieval Augmented Generation (RAG)](), [Prompt Engineering](), [Data Science](), [MLOps](), [recommendation systems]() Tags: [course](), [courses](), [data-science](), [fine-tuning](), [free](), [guide](), [llms](), [machine-learning](), [mlops](), [open](), [rag](), [recommendation-systems](), [tutorial]() ### AI overview The article announces Mastering LLMs, a free, open course of workshops and talks led by more than 25 industry practitioners. It covers applied topics including evaluations, retrieval-augmented generation, fine-tuning, application development, and prompt engineering, and is intended for technical professionals with basic LLM experience. ### Source excerpt Today, we are releasing Mastering LLMs, a set of workshops and talks from practitioners on topics like evals, retrieval-augmented-generation (RAG), fine-tuning and more. This course is unique because it is: Taught by 25+ industry veterans who are experts in information retrieval, machine learning, recommendation systems, MLOps and data science. We discuss how this prior art can be applied to LLMs to give you a meaningful advantage. Focused on applied topics that are relevant to people building AI products. Free and open to everyone . We have organized and annotated the talks from our popular paid course.1 This is a survey course for technical ICs (including engineers and data scientists) who have some experience with LLMs and need guidance on how to improve AI products. Speakers include Jeremy Howard, Sophia Yang, Simon Willison, JJ Allaire, Wing Lian, Mark Saroufim, Jane Xu, Jason Liu, Emmanuel Ameisen, Hailey Schoelkopf, Johno Whitaker, Zach Mueller, John Berryman, Ben Clavié, Abhishek Thakur, Kyle Corbitt, Ankur Goyal, Freddy Boulton, Jo Bergum, Eugene Yan, Shreya Shankar, Charles Frye, Hamel Husain, Dan Becker and more Getting The Most Value From The Course Prerequisites The course assumes basic familiarity with LLMs. If you do not have any experience, we recommend watching A Hacker's Guide to LLMs. We also recommend the tutorial Instruction Tuning llama2 if you are interested in fine-tuning 2. Navigating The Material The course has over 40 hours of content. To help you navigate this, we provide: Organization by subject area: evals, RAG, fine-tuning, building applications and prompt engineering. Chapter summaries: quickly peruse topics in each talk and skip ahead Notes, slides, and resources: these are resources used in the talk, as well as resources to learn more. Many times we have detailed notes as well! To get started, navigate to this page and explore topics that interest you. Feel free to skip sections that aren't relevant to you. We've organized the talks ## Debugging AI With Adversarial Validation DevFeed: [Debugging AI With Adversarial Validation]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2024-04-12T07:00:00Z Content type: tutorial Language: en Sources: [Hamel Husain]() Topics: [MLOps](), [Machine learning](), [datasets](), [Fine-tuning](), [Training AI Models]() Tags: [ai](), [dataset](), [datasets](), [debugging](), [evaluation](), [fine-tuning](), [llms](), [mlops](), [model](), [validation]() ### AI overview This article explains how to use Adversarial Validation to detect drift between datasets such as training, evaluation, and production data. It describes labeling two datasets, training a binary classifier to distinguish them, and inspecting feature importance or SHAP values to investigate detected differences. It also notes that failing to detect drift does not prove that no drift exists. ### Source excerpt For years, I've relied on a straightforward method to identify sudden changes in model inputs or training data, known as "drift." This method, Adversarial Validation1, is both simple and effective. The best part? It requires no complex tools or infrastructure. Examples where drift can cause bugs in your AI: Your data for evaluations are materially different from the inputs your model receives in production, causing your evaluations to be misleading. Updates to prompts, functions, RAG, and similar elements aren't incorporated into your fine-tuning or training data, leading to unexpected model behavior in production. No matter how careful you are, bugs can still slip through the cracks. A a high ROI activity is to routinely audit all your AI/ML projects for drift. How It Works WarningUncool Warning This method is so simple that it might seem uncool. You aren't going to impress any data scientists. Despite this, it's too valuable to ignore. This slide from my talk on MLOps tools explains the technique behind Adversarial Validation2: Slide The process is as follows: Collect two datasets to compare. For example: Training data from two different fine-tuning runs Training data vs. evaluation data Training data vs. production data (organized into the same format) Data from two different time-periods Create features from the dataset. A basic example that creates features from tokens is illustrated here.3 Give dataset #1 a label of 0 and dataset #2 a label of 1. Fit a binary classifier (random forest, logistic regression, etc) to discriminate between the two datasets. If the classifier demonstrates sufficient predictive power (ex: AUC >=0.60), we know there is drift. If you used an interpretable model (like logistic regression, random forest, etc.), you can inspect feature importance metrics to understand the root cause of the drift. If you use a more complex model (like a neural network), you can use SHAP values or other methods to understand what is causing the drift. I rec ## Your AI Product Needs Evals DevFeed: [Your AI Product Needs Evals]() Original publisher: [Read original article]() Author: Hamel Husain Published: 2024-03-29T07:00:00Z Content type: article Language: en Sources: [Hamel Husain]() Topics: [Artificial Intelligence](), [Large Language Model](), [Software Engineering](), [Prompt Engineering](), [Logging](), [Fine-tuning]() Tags: [ai](), [code](), [data](), [debugging](), [evals](), [fine-tuning](), [language-models](), [llms](), [logging](), [prompt-engineering](), [software-engineering](), [tests]() ### AI overview This article argues that robust evaluation systems are essential for building and improving LLM-powered AI products. It recommends combining quality evaluation, debugging through logging and data inspection, and system changes such as prompt engineering, fine-tuning, and code changes. It also introduces a case study involving Lucy, Rechat's real estate AI assistant. ### Source excerpt Motivation I started working with language models five years ago when I led the team that created CodeSearchNet, a precursor to GitHub CoPilot. Since then, I've seen many successful and unsuccessful approaches to building LLM products. I've found that unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems. I originally wrote this while helping companies build domain-specific AI products. I hope companies can save thousands of dollars in consulting fees by reading this post carefully. This post outlines my thoughts on building evaluation systems for LLMs-powered AI products. Iterating Quickly == Success Like software engineering, success with AI hinges on how fast you can iterate. You must have processes and tools for: Evaluating quality (ex: tests). Debugging issues (ex: logging & inspecting data). Changing the behavior or the system (prompt eng, fine-tuning, writing code) Many people focus exclusively on #3 above, which prevents them from improving their LLM products beyond a demo.1 Doing all three activities well creates a virtuous cycle differentiating great from mediocre AI products (see the diagram below for a visualization of this cycle). If you streamline your evaluation process, all other activities become easy. This is very similar to how tests in software engineering pay massive dividends in the long term despite requiring up-front investment. To ground this post in a real-world situation, I'll walk through a case study in which we built a system for rapid improvement. I'll primarily focus on evaluation as that is the most critical component. Case Study: Lucy, A Real Estate AI Assistant Rechat is a SaaS application that allows real estate professionals to perform various tasks, such as managing contracts, searching for listings, building creative assets, managing appointments, and more. The thesis of Rechat is that you can do everything in one place rather than having to context switch between many diff