# Key takeaways from the PyAI conference on AI evaluation, software design, and open-source maintenance

DevFeed: [Key takeaways from the PyAI conference on AI evaluation, software design, and open-source maintenance](<https://devfeed.tech/articles/learnings-from-the-pyai-conference-21748.md>)

Original publisher: [Read original article](<http://blog.pamelafox.org/2026/03/learnings-from-pyai-conference.html>)

Author: Pamela Fox (noreply@blogger.com)

Published: 2026-03-12T06:40:00Z

Content type: article

Language: en

Sources: [Pamela Fox](<https://devfeed.tech/sources/pamela-fox.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Python](<https://devfeed.tech/topics/python.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [MCP](<https://devfeed.tech/topics/mcp.md>), [SDKs](<https://devfeed.tech/topics/sdks.md>), [Pydantic](<https://devfeed.tech/topics/pydantic.md>), [FastAPI](<https://devfeed.tech/topics/fastapi.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-evals](<https://devfeed.tech/tags/ai-evals.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [conference](<https://devfeed.tech/tags/conference.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [fastapi](<https://devfeed.tech/tags/fastapi.md>), [github](<https://devfeed.tech/tags/github.md>), [maintainers](<https://devfeed.tech/tags/maintainers.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [python](<https://devfeed.tech/tags/python.md>), [sdk](<https://devfeed.tech/tags/sdk.md>)

## AI overview

The article summarizes lessons from PyAI conference sessions on evaluating AI systems, designing Python software for maintainability by coding agents, and handling AI-generated pull requests in open-source projects. It recommends validating LLM judges with labeled data and conventional evaluation metrics, using clearer software abstractions, and developing systems to triage low-quality contributions.

## Source excerpt

I recently spoke at the PyAI conference, put on by the good folks at Prefect and Pydantic, and I learnt so much from the talks I attended. Here are my top takeaways from the sessions that I watched: AI Evals Pitfalls Hamel Husain 📺 Watch the video recording | 📊 View slides Hamel cautioned against blindly using automated evaluation frameworks and built-in evaluators (like helpfulness and coherence). Instead, we should adopt a data science approach to evaluation: explore the data, discover what's actually breaking, identify the most important metric, and iterate as new data comes in. We shouldn't just trust an LLM-as-a-judge to be given accurate scores. Instead, we should validate it like we would validate a ML classifier- with labeled data, train/dev/test splits, and precision/recall metrics. LLM-judges should always give pass/fail results, instead of 1-5 scores, so that there's no ambiguity in their judgment. When generating synthetic data, first come up with dimensions (such as persona), generate combinations based off dimensions, and convert those into realistic queries. Hamel created evals-skills, a collection of skills for coding agents that can be run against evaluation pipelines to find issues like poorly designed LLM-judges. Build Reasonable Software Jeremiah Lowin (FastMCP/Prefect) 📺 Watch the video recording Write your Python programs in a way that coding agents can reason about them, so that they can more easily maintain and build them. For example, FastMCP v2 SDK was not well designed (bad abstractions) so a new CodeMod feature required 4,000 lines of code. In the new FastMCP v3 SDK (same functional API, different abstractions backing it), the same feature only required 500 lines of code. To make Python FastMCP servers more Pythonic, Jeremiah is developing a new package for MCP apps which includes the most common UIs (forms/tables/charts), called PreFab: https://github.com/PrefectHQ/prefab Panel: Open Source in the Age of AI Guido van Rossum (CPython), Sa