# An Engineer's Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent...

DevFeed: [An Engineer's Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent...](<https://devfeed.tech/articles/an-engineer-s-guide-to-better-ai-skills-implementing-a-testing-process-to-optimize-agent-1226.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/an-engineers-guide-to-better-ai-skills-implementing-a-testing-process-to-optimize-agent-a000c9c9abcd?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-05-12T16:01:00Z

Content type: article

Language: en

Sources: [Pinterest Engineering Blog - Medium](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [Agent Skill](<https://devfeed.tech/topics/agent-skill.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [codex](<https://devfeed.tech/topics/codex.md>), [Bash](<https://devfeed.tech/topics/bash.md>), [iOS](<https://devfeed.tech/topics/ios.md>), [JSON](<https://devfeed.tech/topics/json.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-skill](<https://devfeed.tech/tags/agent-skill.md>), [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [bash](<https://devfeed.tech/tags/bash.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [code](<https://devfeed.tech/tags/code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [debug](<https://devfeed.tech/tags/debug.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [engineering-culture](<https://devfeed.tech/tags/engineering-culture.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [openai](<https://devfeed.tech/tags/openai.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [programming](<https://devfeed.tech/tags/programming.md>), [skills](<https://devfeed.tech/tags/skills.md>), [testing](<https://devfeed.tech/tags/testing.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

## AI overview

An engineer's guide to testing AI skill invocation in repositories and custom skills. It describes a Bash-based harness that runs positive and negative prompts, captures verbose logs, parses JSON-streamed output, and measures invocation reliability across an internal Codex fork and Claude Code.

## Source excerpt

An Engineer's Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent Performance in Any Repository or Skill Author: Daniel Reed The tech industry is currently seeing a massive overhaul in the way we work and many are enjoying the benefits of AI agents, particularly when automating engineer workflows and serving domain-specific knowledge. However, relying on agents to consistently invoke a custom skill can be surprisingly unreliable at times. When adopting a new skill intended to help agents write code for Pinterest's iOS architecture (I'll call it rx-mvvm) we discovered that sometimes our knowledge skill wasn't being loaded into our agents. To address this, we conducted a series of tests on Pin-agent (an internal fork of OpenAI's Codex) and Claude Code to quantify the reliability of skill invocation and identify some best practices to maximize performance. This was a direct result of observing agents struggling to meet the skills bar during architectural reviews. We found that by applying different techniques we could track and drastically improve skill invocation rates on both tested agents. How to Build A Skill Test Harness Building a reliable test harness for agent skill invocation requires three key components working in concert. The Core Tool is a Bash script that orchestrates automated testing by piping prompts to your agent and capturing verbose output logs. The core execution is simple: if echo "$prompt" | claude --print --verbose --output-format stream-json > "$log_file" 2>&1; then command_success=true fi The script runs all test cases in sequence, collecting logs for later analysis. We ran the entire suite multiple times to account for the nondeterministic nature of agents. Prompts were categorized into two categories defined as arrays: Positive Cases -- 15 prompts covering the full spectrum of skill domains: CORE_PROMPTS=( "load the rx-mvvm-architecture skill" "check if this follows rx-mvvm patterns" # ... 13 more cases ) Negative Case