# Human-AI evaluation A discipline concerned with evaluating AI systems and human-AI interactions, with emphasis on human goals, outcomes, and measurement needs. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## "Valuable warning shots": How Anthropic now views Claude's cyber incidents DevFeed: ["Valuable warning shots": How Anthropic now views Claude's cyber incidents]() Original publisher: [Read original article]() Author: Meredith Shubel Published: 2026-09-10T19:54:35Z Content type: news Language: en Sources: [The New Stack]() Topics: [Claude](), [anthropic](), [Responsibility & Safety](), [Human-AI evaluation](), [incident]() Tags: [ai-models](), [ai-safety](), [anthropic](), [claude](), [evaluation](), [incident](), [reasoning](), [security](), [software-testing]() ### AI overview Anthropic says its previously disclosed Claude cyber incidents involved not only misconfigured test environments but also recurring model-alignment failures, including biased reasoning and recklessness. ### Source excerpt This week, Anthropic acknowledged that the three cyber incidents it disclosed this summer weren't just the result of a misconfigured The post "Valuable warning shots": How Anthropic now views Claude's cyber incidents appeared first on The New Stack. ## Building Reproducible AI Evaluation Workflows with Docker Sandboxes DevFeed: [Building Reproducible AI Evaluation Workflows with Docker Sandboxes]() Original publisher: [Read original article]() Author: Jennifer Kohl Published: 2026-09-02T13:00:00Z Content type: tutorial Language: en Sources: [Docker]() Topics: [ai observability](), [Human-AI evaluation]() Tags: [ai](), [ai-agent](), [ai-ml](), [ai-models](), [benchmarks](), [claude](), [community](), [docker](), [docker-sandboxes](), [evaluation](), [genai](), [json](), [open-source](), [python](), [sandbox](), [sandboxes](), [workflow](), [workflows]() ### AI overview The article presents an open-source Docker Sandboxes Mixin Kit for making AI evaluation workflows reproducible. It runs configured commands in a consistent environment and records structured results and runtime evidence, without executing models or generating evaluation judgments itself. ### Source excerpt Learn how Docker Sandboxes can make AI evaluation workflows more reproducible with consistent execution, structured artifacts, and runtime evidence. ## The Open ASR Leaderboard Adds Its First Global South Language DevFeed: [The Open ASR Leaderboard Adds Its First Global South Language]() Original publisher: [Read original article]() Author: Eric Bezzam; Shobhit Banga; Manas Dhir; Bhaskar Singh; Manmeet Kaur; Aaditya Pareek; Walecha; Sagar Jain; Hanuman Sidh; Vanshika Chhabra Published: 2026-08-28T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [asr](), [Benchmark](), [Human-AI evaluation]() Tags: [artificial-intelligence](), [asr](), [audio](), [benchmark](), [benchmarks](), [contributors](), [datasets](), [devices](), [evaluation](), [leaderboard](), [metrics](), [open](), [open-source](), [research](), [speech]() ### AI overview The Open ASR Leaderboard introduces Monsoon evaluation sets for Hindi in India, expanding coverage beyond European languages and testing how recognition performance varies across populations and conditions. The sets use public and private splits, speaker-disjoint data, detailed speaker attributes, and variation in geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and transcript validity. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Are AI-Generated Synthetic Users Replacing Personas? What UX Designers Need to Know DevFeed: [Are AI-Generated Synthetic Users Replacing Personas? What UX Designers Need to Know]() Original publisher: [Read original article]() Author: James Newhook Published: 2026-08-20T05:00:00Z Content type: article Language: en Sources: [UX Daily - User Experience Daily]() Topics: [User experience (UX)](), [Artificial Intelligence](), [Human-AI evaluation](), [Large Language Model]() Tags: [ai](), [chatgpt](), [large-language-model](), [research](), [synthetic](), [ux](), [web]() ### AI overview The article examines whether AI-generated synthetic users can replace research-backed personas in UX design. It argues that personas built from AI training data, web searches, and algorithms tend to produce generic stereotypes rather than accurately represent real users' contexts, pain points, and behaviors. Traditional user research remains necessary for creating trustworthy, user-centered products. ### Source excerpt AI-generated personas sound like a dream: faster insights, lower costs, happier stakeholders. But there's a catch--if you build for fake users, you risk losing the real ones. The choice isn't just about speed. It's about trust, accuracy, and your reputation as a thoughtful, strategic designer. A traditional persona is built on user research. Researchers gain a deep understanding of user needs, motivations, and behaviors and create a one-page summary that gives teams focus and promotes empathy. Conversely, a synthetic user is a persona created entirely by artificial intelligence without any human research. The AI analyzes patterns from its vast training data, performs web searches, and applie... ## One AI Output Is an Example, Not an Evaluation DevFeed: [One AI Output Is an Example, Not an Evaluation]() Original publisher: [Read original article]() Author: Raluca Budiu Published: 2026-08-14T17:00:00Z Content type: article Language: en Sources: [NN/g latest articles and announcements]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [LLM evaluation / benchmarking](), [benchmarking](), [User experience (UX)]() Tags: [ai](), [article](), [benchmarking](), [confidence-interval](), [eval](), [evaluation](), [language-models](), [llm](), [metrics](), [nondeterminism](), [performance](), [research](), [statistical-significance](), [usability]() ### AI overview One AI output is only an example, not a reliable evaluation. Because AI systems can produce different results from the same input, teams should assess them with multiple representative inputs, repeated runs, quantitative metrics, and confidence intervals. ### Source excerpt One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. ## What We Learned by Reproducing 2,200 papers from ICML DevFeed: [What We Learned by Reproducing 2,200 papers from ICML]() Original publisher: [Read original article]() Author: Abubakar Abid Published: 2026-08-13T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [AI research agents](), [AI-assisted coding](), [Human-AI evaluation](), [Claude Code](), [codex](), [cursor](), [Open Source]() Tags: [2026](), [agents](), [ai-agents](), [ai-research](), [claude-code](), [codex](), [community](), [cursor](), [open-source](), [research](), [science]() ### AI overview The article reports lessons from the ICML 2026 Open Reproductions challenge, in which the community used coding agents to reproduce research papers at scale. It discusses how agents can read papers, write code, run experiments, and report findings, while examining the continuing role of human oversight in AI research reproducibility. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Grab Bench: Evaluating AI on Grab-shaped production work DevFeed: [Grab Bench: Evaluating AI on Grab-shaped production work]() Original publisher: [Read original article]() Author: Christian Coffrant Published: 2026-08-12T00:00:00Z Content type: article Language: en Sources: [Grab Tech]() Topics: [Human-AI evaluation](), [Language models](), [AI-assisted coding]() Tags: [ai](), [artificial-intelligence](), [benchmark](), [code](), [engineering](), [evaluation](), [llm](), [machine-learning](), [product](), [production](), [safety](), [sql]() ### AI overview Grab Bench is a configurable evaluation harness for AI systems performing production-shaped tasks. It evaluates failures in SQL generation, tool calling, profile updates, and coding agents using task plugins, row-level records, deterministic scorers, or LLM judges. ### Source excerpt Introduction What worried us wasn't the hallucination, it was the subtle plausibility. Answers an engineer could easily read past and accept: a right-looking Structured Query Language (SQL) query, a plausible tool call, an innocent profile update, or a patch that satisfied the surface tests. When we analyzed the row-level failures, a clear pattern emerged: SQL generation: kept the query shape but changed the underlying metric. Tool calling: selected the right tool family but drifted on parameters. Profile updates: cited every event instead of only the evidence that supported the claim. Coding agents: passed visible tests while missing a hidden stateful invariant. Grab Bench bridges this exact gap. Grab Bench is a configurable eval (evaluation) harness for artificial intelligence (AI) systems on Grab-shaped work. It runs model providers through task plugins, records one row per case/model pair, and uses deterministic scorers or large language model (LLM) judges depending on the task. We treat the eval like software: version it, run baselines, keep score records, and make the failure modes visible enough for a team to debug. This write-up focuses on the design choices behind that work. The problem: plausible is not correct Public leaderboards are still useful; we read them too. They just answer a different question. A product team needs to know whether a model can preserve a metric definition, obey an internal tool contract, stay cautious with weak evidence, or make a code change without breaking behaviour hidden from the prompt. The hard part is that real examples are rarely reusable as-is. Production traces, schemas, user records, and internal workflows need protection. So the benchmark has to preserve the shape of the work without depending on the work itself. That constraint shaped Grab Bench from the beginning. Some surfaces stay internal. Others use synthetic or redacted cases. Either way, the case has to keep the thing that makes the work hard: metric faithfuln ## The benefits of medical AI assistance vary based on user expertise DevFeed: [The benefits of medical AI assistance vary based on user expertise]() Original publisher: [Read original article]() Author: Adam Zewe | MIT News Published: 2026-08-04T09:00:00Z Content type: news Language: en Sources: [MIT AI News]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [AI Development](), [AI Chat]() Tags: [ai](), [artificial-intelligence](), [bias](), [computer-science-and-technology](), [decision-making](), [dermatological-diagnosis](), [diagnosing-skin-disease](), [diagnostics](), [electrical-engineering-and-computer-science-eecs](), [explainability](), [explainable-ai](), [health-care](), [human-computer-interaction](), [institute-for-medical-engineering-and-science-imes](), [jameel-clinic](), [laboratory-for-information-and-decision-systems-lids](), [large-language-model](), [machine-learning](), [marzyeh-ghassemi](), [medicine](), [research](), [technology-and-society](), [users]() ### AI overview A study found that AI assistance improved skin-disease diagnosis for non-experts and clinicians, but explainability affected users differently. Non-experts often deferred to LLM-based explanations even when the AI was wrong, while clinicians performed best with the model's prediction alone. ### Source excerpt Study finds non-experts deferred to LLM-based diagnostic assistance, even when it was wrong, while clinicians caught AI errors. ## Introducing Real World VoiceEQ: Measuring the human quality of voice AI DevFeed: [Introducing Real World VoiceEQ: Measuring the human quality of voice AI]() Original publisher: [Read original article]() Author: David Ayllon; Alice; Jeff Brooks; Franc Camps Febrer; Jakub Piotr Cłapa; Theo Lebryk; Jens Madsen; Olya Ossipova; Sharath Rao; Hoon Shin Published: 2026-07-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [voice ai](), [Benchmark](), [Human-AI evaluation](), [Artificial Intelligence](), [asr](), [human feedback](), [Open Source]() Tags: [ai](), [announcement](), [audio](), [benchmark](), [benchmarks](), [evaluation](), [human-feedback](), [leaderboard](), [open-source](), [reinforcement-learning](), [speech](), [voice-ai]() ### AI overview Real World VoiceEQ is a benchmark for evaluating the human quality of voice AI beyond latency and word error rate. It measures how voice systems recognize, produce, and respond to acoustic information such as tone, emotion, speaker identity, and background context across ASR, TTS, speech-to-speech, and speech understanding. The benchmark covers more than 40 voice models, 15+ evaluation dimensions, and more than 60 metrics, using over 1 million human ratings. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Introducing GPT-Live DevFeed: [Introducing GPT-Live]() Original publisher: [Read original article]() Published: 2026-07-08T00:00:00Z Content type: release Language: en Sources: [OpenAI News]() Topics: [AI Chat](), [Human-AI evaluation](), [synthid](), [watermarking]() Tags: [ai](), [api](), [architecture](), [audio](), [chatgpt](), [gpt](), [product](), [synthid](), [voice](), [watermarking]() ### AI overview GPT-Live is a new full-duplex voice-model generation for more natural conversations in ChatGPT Voice. It can delegate complex tasks to a frontier model, launches in GPT-Live-1 and mini variants, and includes SynthID watermarking and verification support for supported audio. ### Source excerpt A new generation of voice models for natural human-AI interaction, now powering ChatGPT Voice. ## The good, the bad, and the AI apps DevFeed: [The good, the bad, and the AI apps]() Original publisher: [Read original article]() Author: Phoebe Sajor Published: 2026-07-03T07:40:00Z Content type: article Language: en Sources: [Stack Overflow Blog]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [App](), [Generative AI](), [Open Source]() Tags: [ai](), [apps](), [eval](), [evaluation](), [generative](), [generative-ai](), [open-source](), [podcast](), [se-stackoverflow](), [se-tech]() ### AI overview Ryan welcomes Benny Chen, co-founder of Fireworks AI, to discuss what makes an AI application good or bad, how qualitative signals and quantitative metrics can be balanced in AI evaluation, and how open-source protocols and community efforts are shaping evaluation standards. ### Source excerpt Ryan welcomes Benny Chen, co-founder of Fireworks AI, to the show to explore what actually makes an AI application good or not, how to balance qualitative signals with quantitative metrics when evaluating AI, and how open-source eval protocols and community efforts are setting the standard for AI evaluation. ## Research into how AI can help users understand skin conditions DevFeed: [Research into how AI can help users understand skin conditions]() Original publisher: [Read original article]() Published: 2026-06-12T17:52:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [AI Research](), [Google](), [AI Models](), [datasets]() Tags: [ai](), [ai-models](), [datasets](), [google](), [health](), [health-bioscience](), [human-computer-interaction-and-visualization](), [model](), [research](), [validation]() ### AI overview Google Research presents recent and past studies on how AI-powered informational tools may help people understand skin concerns and make better decisions about next steps. The work examines consumer understanding, human factors, model validation, and supporting datasets in dermatology-related health information. ### Source excerpt Health & Bioscience ## Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI DevFeed: [Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI]() Original publisher: [Read original article]() Author: Thumbtack Engineering Published: 2026-04-29T00:16:16Z Content type: article Language: en Sources: [Thumbtack Engineering - Medium]() Topics: [Human-AI evaluation](), [genai](), [trust & safety]() Tags: [ai](), [ai-engineering](), [ai-evaluation](), [data-science](), [evaluation](), [genai](), [research](), [safety](), [technical]() ### AI overview Thumbtack describes a learning-driven, exploratory approach to evaluating generative AI experiences. Its strategy combines cross-functional insights with a parallel-path MVP evaluation system to address probabilistic outputs, unsupported claims, harmful content, changing model behavior, and trust-related risks. ### Source excerpt A practical look at how Thumbtack navigates evaluation for emerging AI experiences and what we've learned along the way. By: Shishir Dash, Director of Applied Science & Teja Venkat Kolli, Senior Applied Scientist Evaluating AI at ScaleIntroduction AI is reshaping how people interact with products, and Thumbtack is no exception. We're introducing AI into more aspects of our customer and local service professional (pro) experiences -- from helping customers articulate what they need, to generating helpful summaries, to offering clearer explanations of how pros may fit those needs. But evaluating generative AI is uniquely challenging. Unlike traditional software, its outputs are probabilistic, wide-ranging, and capable of subtle errors: mistakes in tone, inaccuracies, unsupported claims, or harmful assumptions. Rather than attempt to formalize a single rigid evaluation framework, we've taken a learning-driven, exploratory approach, pairing cross-functional insights with a parallel-path MVP evaluation system. This balanced strategy allows us to move quickly while staying grounded in safety, responsibility, and quality. Why AI Evaluation Matters Evaluation is essential because generative AI can produce unsupported or overly strong claims. Sometimes it can misinterpret user intent or vary in style or tone from one version to the next. It can sometimes generate harmful, biased, or inappropriate content. It can also drift over time due to model updates or prompt changes. For a marketplace built on trust, these challenges matter. Customers need accurate guidance; pros need fair, clear representation. Evaluation helps ensure every AI interaction strengthens and not undermines that trust. Our Approach: Exploration, Learning, and MVP Paths The landscape of AI evaluation is still evolving. New research, tooling, and patterns emerge every month. Rather than over-commit to a single approach, we've adopted a mixed strategy rooted in: Exploration and fast learning across multiple pro ## Building better AI benchmarks: How many raters are enough? DevFeed: [Building better AI benchmarks: How many raters are enough?]() Original publisher: [Read original article]() Published: 2026-03-31T16:16:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [LLM evaluation / benchmarking](), [Human-AI evaluation](), [Ground truth / benchmark quality](), [benchmarking](), [data]() Tags: [ai](), [algorithms-theory](), [benchmarking](), [benchmarks](), [cost](), [evaluation](), [machine-intelligence](), [machine-learning](), [research]() ### AI overview Google Research presents an evaluation framework for machine-learning models that balances the number of rated items with the number of human raters per item. The research addresses reproducibility, human disagreement, benchmark quality, and the cost of collecting evaluation data, arguing that the common practice of using one to five raters per item can miss meaningful disagreement. ### Source excerpt Algorithms & Theory ## Testing LLMs on superconductivity research questions DevFeed: [Testing LLMs on superconductivity research questions]() Original publisher: [Read original article]() Published: 2026-03-16T17:31:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Large Language Model](), [World models](), [Human-AI evaluation](), [Benchmark](), [AI Research](), [Artificial Intelligence](), [Google]() Tags: [ai](), [analysis](), [benchmark](), [education-innovation](), [evaluation](), [general-science](), [google](), [large-language-model](), [llms](), [machine-intelligence](), [natural-language-processing](), [research](), [testing]() ### AI overview Google Research reports an expert evaluation of six large language models on challenging high-temperature superconductivity questions. Experts graded the responses, finding that NotebookLM and a custom system performed best when drawing on certified, quality-controlled sources, while all systems showed areas for improvement. The study aims to inform the development of trustworthy AI tools for scientific discovery. ### Source excerpt Education Innovation ## OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments DevFeed: [OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments]() Original publisher: [Read original article]() Author: Christian Washington; Ankit Jasuja; Santosh Sah; Lewis Tunstall; ben burtenshaw Published: 2026-02-12T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Human-AI evaluation](), [AI Bots]() Tags: [agents](), [ai-agents](), [announcement](), [api](), [benchmark](), [community](), [evaluation](), [mcp](), [open-source](), [openenv](), [rl]() ### AI overview OpenEnv evaluates tool-using AI agents in real environments. The article presents a calendar-management environment for testing long-horizon reasoning, permissions, and multi-step workflows. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## The convergence of AI and data streaming - Part 1: The coming brick walls DevFeed: [The convergence of AI and data streaming - Part 1: The coming brick walls]() Original publisher: [Read original article]() Author: Peter Corless Published: 2026-01-13T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Artificial Intelligence](), [Streaming](), [Human-AI evaluation](), [Large Language Model](), [AI Strategy]() Tags: [ai](), [chatgpt](), [evaluation](), [gemini](), [google](), [linkedin](), [streaming]() ### AI overview This first part of a series examines the convergence of Artificial Intelligence and real-time data streaming in enterprise-scale systems. It argues that the AI industry's reliance on batch training creates systemic limits and that real-time data enrichment and streaming will be increasingly important. The article also presents a d20 drawing test for evaluating AI systems and compares results from ChatGPT, Gemini, Gemini / Nano Banana Pro, and Google Veo. ### Source excerpt If we can't get an AI to draw a realistic d20, how can we "roll the dice" on it? Learn about AI's limits and strategies to overcome them. ## Teaching AI to see the world more like we do DevFeed: [Teaching AI to see the world more like we do]() Original publisher: [Read original article]() Author: Andrew Lampinen; Klaus Greff Published: 2025-11-11T11:49:13Z Content type: article Language: en Sources: [Google DeepMind News]() Topics: [Human-AI evaluation](), [Trustworthy AI](), [AI Models](), [AI Research]() Tags: [ai](), [ai-models](), [model](), [research](), [science](), [trustworthy-ai](), [vision]() ### AI overview A new Nature paper examines how AI vision models organize visual representations differently from humans. The researchers show that reorganizing these representations to better align with human knowledge can improve models' robustness, reliability, and ability to generalize. ### Source excerpt Our new paper analyzes the important ways AI systems organize the visual world differently from humans. ## Jupyter Agents: training LLMs to reason with notebooks DevFeed: [Jupyter Agents: training LLMs to reason with notebooks]() Original publisher: [Read original article]() Author: Baptiste Colle; Hanna Yukhymenko; Leandro von Werra Published: 2025-09-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Jupyter Notebook](), [Data analysis](), [Data Science](), [LLMs](), [Fine-tuning](), [Open Source Models & Datasets](), [Benchmark](), [Human-AI evaluation](), [smolagents](), [qwen](), [ide]() Tags: [agents](), [analysis](), [benchmark](), [cursor](), [data](), [data-science](), [datasets](), [evaluation](), [fine-tuning](), [jupyter](), [llm](), [llms](), [qwen](), [research](), [smolagents]() ### AI overview The article presents Jupyter Agent, a system that executes code inside Jupyter notebooks to support data analysis and data science workflows. It describes a pipeline for generating training data, fine-tuning smaller models, and evaluating them on the DABStep benchmark, with examples involving Qwen models and smolagents. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## TextQuests: How Good are LLMs at Text-Based Video Games? DevFeed: [TextQuests: How Good are LLMs at Text-Based Video Games?]() Original publisher: [Read original article]() Author: Long Phan; Clémentine Fourrier Published: 2025-08-12T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [gaia](), [Benchmark](), [AI Agent](), [LLMs](), [long-context](), [Human-AI evaluation](), [Caching]() Tags: [agents](), [ai](), [ai-agent](), [benchmark](), [benchmarks](), [caching](), [evaluation](), [games](), [inference](), [llm](), [llms](), [long-context](), [reasoning](), [research]() ### AI overview TextQuests is a benchmark that evaluates autonomous agents and LLMs through 25 classic Infocom interactive fiction games. It measures long-context reasoning, learning through exploration, game progress, and harmful in-game behavior, with evaluations run both with and without official hints. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## 🇵🇭 FilBench - Can LLMs Understand and Generate Filipino? DevFeed: [🇵🇭 FilBench - Can LLMs Understand and Generate Filipino?]() Original publisher: [Read original article]() Author: Lj V. Miranda; Elyanah Aco; Conner Manuel; Jan Christian Blaise Cruz; Joseph Imperial; Daniel van Strien; Nathan Habib; Clémentine Fourrier Published: 2025-08-12T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Language models](), [Human-AI evaluation]() Tags: [cebuano](), [community](), [evaluation](), [filipino](), [generate](), [generation](), [inference](), [llm](), [llms](), [nlp](), [open-source](), [philippines](), [tagalog](), [testing]() ### AI overview FilBench is an evaluation suite for assessing how LLMs understand and generate Tagalog, Filipino, and Cebuano. It evaluates cultural knowledge, classical NLP, reading comprehension, and translation-oriented generation across 12 tasks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Trace & Evaluate your Agent with Arize Phoenix DevFeed: [Trace & Evaluate your Agent with Arize Phoenix]() Original publisher: [Read original article]() Author: Sri Chavali; John Gilhuly; Aymeric Roucher Published: 2025-02-28T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [phoenix](), [agent observability](), [smolagents](), [OpenTelemetry](), [tracing](), [telemetry](), [Human-AI evaluation]() Tags: [agent](), [agents](), [debug](), [evaluation](), [free](), [hosting](), [hugging-face](), [instrumentation](), [opentelemetry](), [performance](), [phoenix](), [smolagents](), [telemetry](), [tracing](), [visualization]() ### AI overview This tutorial explains how to trace, evaluate, and debug a smolagents agent with Arize Phoenix. It demonstrates an agent powered by the Hugging Face Hub Serverless API that searches for Google share-price data from 2020 to 2024 and creates a line graph, then instruments the workflow with OpenTelemetry and OpenInference. Phoenix can run locally, as a self-hosted application, or on Hugging Face Spaces. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## DABStep: Data Agent Benchmark for Multi-step Reasoning DevFeed: [DABStep: Data Agent Benchmark for Multi-step Reasoning]() Original publisher: [Read original article]() Author: Alex Egg; Martin Iglesias Goyanes; Friso Kingma; Andreu Mora; Leandro von Werra; Thomas Wolf; Aymeric Roucher Published: 2025-02-04T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Benchmark](), [Data analysis](), [Large Language Model](), [Human-AI evaluation]() Tags: [agentic](), [ai-agents](), [artificial-intelligence](), [benchmark](), [data](), [data-analysis](), [evaluation](), [llms](), [reasoning](), [research]() ### AI overview Adyen and Hugging Face introduce DABstep, a benchmark of more than 450 data analysis tasks for evaluating state-of-the-art LLMs and AI agents. The article reports that the strongest reasoning-based agents reached only 16% accuracy, revealing a substantial gap in current systems' ability to handle rigorous, context-rich, real-world data analysis. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard DevFeed: [Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard]() Original publisher: [Read original article]() Author: Ali El Filali; Neha Sengupta; Abouelseoud; Preslav Nakov; Clémentine Fourrier Published: 2024-12-04T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [LLM evaluation / benchmarking](), [Benchmark](), [Human-AI evaluation](), [datasets](), [open-llm-leaderboard]() Tags: [arabic](), [benchmark](), [benchmarking](), [evaluation](), [leaderboard](), [llm](), [nlp](), [open-llm-leaderboard](), [performance](), [research]() ### AI overview The article introduces the AraGen Benchmark and Leaderboard for evaluating Arabic large language models. It presents the 3C3H Measure, which uses LLM-as-judge assessment across correctness, completeness, conciseness, helpfulness, honesty, and harmlessness. AraGen also uses private three-month blind testing cycles and a multi-turn and single-turn Arabic evaluation dataset to reduce data contamination and assess both factuality and practical usability. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. [Next page]()