# Pinterest Engineering Blog - Medium Inventive engineers building the first visual discovery engine, 300 billion ideas and counting. - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Evolving Pinterest's Embedding Retrieval Platform DevFeed: [Evolving Pinterest's Embedding Retrieval Platform]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-11T15:01:03Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [Architecture & Design](), [Algorithms](), [Scalability](), [Latency](), [IO]() Tags: [ann](), [cost](), [embedding](), [embeddings](), [engineering](), [infrastructure](), [latency](), [memory](), [models](), [pinterest](), [platform](), [quantization](), [real-time](), [retrieval](), [scalability](), [search]() ### AI overview Pinterest describes evolving its Manas embedding-retrieval platform to address the cost, scale, and flexibility challenges of serving billions of embeddings. The excerpt covers ANN search, vector quantization, and SSD-based serving. ### Source excerpt Authors: Bowen Zhou | Staff Software Engineer; Shan Gao | Senior Software Engineer; Jingwen Hu | Software Engineer II; Wenjiang Chu | Staff Software Engineer The Billion-Embedding Challenge At Pinterest, the "signal" is our lifeblood. Whether it's a home decor enthusiast finding the perfect rug or a fashion seeker discovering a new aesthetic, our discovery engine relies on understanding deep semantic relationships to help our users find inspirations. Over the last few years, the explosive growth of embedding-based retrieval has fundamentally transformed how we surface these signals -- and at the heart of that transformation is Manas, Pinterest's in-house distributed search platform. Embedding Retrieval is one of the core capabilities of Manas, supporting multiple approximate nearest neighbor search algorithms, hybrid queries with both token and embedding clauses, as well as real-time updates to ensure fresh contents become searchable within seconds. Deployed on over 80 clusters and serving billions of embeddings, Manas embedding retrieval powers all major product surfaces at Pinterest including Home Feed, Search, Related Pins, Ads, and Notifications. However, as our corpus scales toward tens of billions of embeddings and our models capture increasingly complex interactions, we face mounting challenges around cost efficiency, scalability, and flexibility. On the infrastructure side, traditional ANN algorithms like HNSW are notoriously memory-hungry -- they require the entire index to reside in RAM to maintain low query latency, making cost grow linearly with corpus size. On the modeling side, the classic two-tower retrieval paradigm is too restrictive: it reduces each candidate to a single embedding and scores relevance through a simple dot product, leaving little room to express richer, context-dependent notions of similarity. To tackle these challenges, our team has been evolving Manas's embedding retrieval stack across three fronts: Quantization. We reduce the memor ## Building Pinterest's VLM Serving Stack on NVIDIA Dynamo DevFeed: [Building Pinterest's VLM Serving Stack on NVIDIA Dynamo]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-10T23:08:16Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [vlm](), [Inference](), [model-deployment](), [Orchestration](), [Latency](), [Caching](), [Open Source Models & Datasets]() Tags: [blackwell](), [cache](), [dynamo](), [embeddings](), [engineering](), [inference](), [latency](), [llm](), [multimodal](), [multimodal-ai](), [nvidia](), [nvidia-blackwell](), [open-source](), [orchestration](), [performance](), [pinterest](), [vllm](), [vlm](), [vlm-serving]() ### AI overview Pinterest describes its VLM serving stack built on NVIDIA Blackwell GPUs and NVIDIA Dynamo. The stack addresses multimodal inference demands such as image processing, variable prefill costs, KV-cache pressure, routing, and cache offloading. ### Source excerpt Lei Pan | Senior Software Engineer; Salina Wu | Senior Software Engineer; Cristian Lopez | Software Engineer I; Guangtong Bai | Staff Software Engineer; Soam Acharya | Principal Engineer; Saurabh Vishwas Joshi | Principal Engineer; Chia-Wei Chen | Staff Software Engineer; Ambud Sharma | Principal Engineer Why VLM Serving Matters at Pinterest Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more. This direction also reflects Pinterest's broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance. Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest's VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today's most demanding AI workloads -- including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity -- that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that g ## Becoming an AI Team DevFeed: [Becoming an AI Team]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-01T15:01:06Z Content type: opinion Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [AI Strategy](), [AI Development](), [code productivity](), [Refactoring]() Tags: [ai](), [developer-productivity](), [engineering](), [engineering-culture](), [infrastructure](), [pinterest](), [refactoring]() ### AI overview The article argues that becoming an AI team requires changing team ownership, planning, operating models, and roles--not merely adding AI tools. It describes AI-assisted automation, analysis, and prototyping as ways to improve infrastructure reliability, cost efficiency, and developer productivity. ### Source excerpt John Grass | Sr. Manager, Engineering A Fundamental Transformation An AI team is fundamentally more than just a group whose members incorporate AI tools into their existing workflows. The journey to becoming an AI team necessitates a fundamental and comprehensive paradigm shift in how the team defines ownership, engages in strategic planning, and, most critically, executes on its core goals and objectives. This transformation is not merely an addition of new technology; it is a restructuring of the team's operating model, philosophy, and individual roles. Becoming an AI team requires a holistic shift: team members transition from routine, manual execution to empowered, AI-augmented strategists and problem-solvers. In an AI-centric environment, every single member is significantly empowered, not only through access to cutting-edge AI tools and sophisticated models but through an expanded scope of responsibility and influence. These new capabilities allow individuals to automate routine tasks, accelerate data analysis, and rapidly prototype solutions, freeing up cognitive resources for higher-level, more strategic thinking. The expectation shifts from simply completing tasks to orchestrating intelligent systems and focusing on solving problems that were previously intractable. Consequently, the very nature of the work for both individual contributors and managers will be fundamentally different from anything that has come before. At Pinterest, this isn't theoretical. Our infrastructure teams sit at the core of a product that serves billions of Pins, boards, ads, and real-time signals. That reality has forced us to treat "becoming an AI team" as an operational necessity, not a side project as we cannot keep scaling reliability, cost efficiency, and developer productivity using only traditional playbooks. AI Team Capabilities The advent of accessible Artificial Intelligence (AI) tools has fundamentally raised the performance ceiling for what development teams can accomp ## Scaling Conditional Learned Retrieval for Pinterest Home Feed DevFeed: [Scaling Conditional Learned Retrieval for Pinterest Home Feed]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-08-26T14:01:05Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [recommendation systems](), [Embeddings](), [Machine Learning & Artificial Intelligence]() Tags: [blog](), [embedding](), [eng-culture](), [engineering](), [infrastructure](), [pinner-experience](), [pinterest](), [ranking](), [retrieval]() ### AI overview Pinterest describes scaling Conditional Learned Retrieval for home-feed candidate generation. The approach extends two-tower retrieval with explicit contexts that produce multiple user embeddings for different interests and retrieval use cases. ### Source excerpt Devin Kreuzer | Sr. Machine Learning Engineer; Yichi Wang | Machine Learning Engineer I; Sujan Reddy Ale | Machine Learning Engineer I; Zelun Wang | Sr. Machine Learning Engineer; Hongtao Lin | Sr. Machine Learning Engineer; Piyush Maheshwari | Staff Machine Learning Engineer Pinterest home feed candidate generation is a large-scale User-to-Pin retrieval problem. A common approach is a two-tower model: a user tower encodes the user, an item tower encodes candidate Pins, and approximate nearest neighbor search retrieves Pins close to the user embedding. But Pinterest users often have multiple intentions at once -- planning a renovation, saving recipes, exploring fashion, or organizing travel ideas. A single retrieval embedding can struggle to capture this diversity. Conditional Learned Retrieval, or CLR, extends the two-tower setup by conditioning the user tower on an explicit retrieval context. Instead of producing only one user embedding, CLR can generate condition-aware embeddings that reflect different aspects of a user's interests while still grounding retrieval in the user's overall behavior. Prior Pinterest work studied this formulation in two settings. The RecSys'24 paper: Bootstrapping Conditional Retrieval for User-to-Item Recommendations described how to bootstrap conditional retrieval by constructing training data for (user, condition) -> item retrieval from existing user-item and condition signals, and applied it to interest-based notifications. The KDD'25 paper: Synergizing Implicit and Explicit User Interests: A Multi-Embedding Retrieval Framework at Pinterest placed Conditional Retrieval within a broader multi-embedding retrieval framework for home feed, where explicit interest conditions complement implicit interests extracted from user behavior. In this blog, we describe how CLR evolved from early interest-conditioned retrieval into a broader retrieval system for Pinterest home feed. We focus on three areas: expanding CLR to support more retrieval us ## Pinner Progression: Better Use-Case Representation Driving Weekly Active User Growth at Pinterest DevFeed: [Pinner Progression: Better Use-Case Representation Driving Weekly Active User Growth at Pinterest]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-07-27T16:01:02Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [recommendation systems](), [Data Science](), [data]() Tags: [engineering](), [growth](), [interest-exploration](), [metrics](), [personalization](), [pinterest](), [recommendation-system](), [retention](), [retrieval](), [understanding-user]() ### AI overview Pinterest introduces Pinner Progression, a recommendation-system program that uses persistent User Interest Clusters to improve discovery and make retention a first-class objective alongside engagement. ### Source excerpt Part 1 of 2 Authors Personalization (Homefeed): Yuke Yan, Chuxi Wang, Andreanne Lemay, Olafur Gudmundsson, Anna Kiyantseva, Krystal Benitez, Jongho Kim, Jiacong He, Rahul Goutam, James Li, Dylan Wang User Understanding: Simin Li, Sufyan Suliman, Yingjian Ding, Hongbo Deng Data Science: Armando Ordorica, Yan Chen, Ellie Zhang, Karim Wahba Introduction Pinterest's mission is to help people discover the inspiration to create a life they love. Our recommendation system serves hundreds of millions of users, surfacing billions of Pins across interests ranging from home renovation to meal planning to wedding decor. The home feed , where much of that discovery happens, is powered by a multi-stage pipeline spanning retrieval, lightweight scoring, ranking, and re-ranking [1][2][3]. Most of our prior work on this pipeline has been optimized for engagement: clicks, saves, downloads, closeups. These are strong signals of immediate relevance, and optimizing for them has driven significant gains across the system [4][5]. The problem is that engagement and retention are different things. A user can save ten sourdough recipes today and churn next month anyway. All we did was feed them more of what they already liked: we never helped them find something new for next time. This post is one of two that introduces Pinner Progression, a program that reframes the home feed recommendation system around retention as a first-class objective. Our core insight is that by augmenting sequential, action-by-action user understanding with holistic, persistent use-case representation, we can reliably anticipate the user's next moves and start to serve recommendations that ignite their serendipitous discovery. In this post, we introduce the key use-case representation signal: User Interest Clusters (UICs) -- and describe its construction, integration into the recommendation stack, and impact on engagement on retention metrics. A follow-up will cover how we predict unseen UICs and conduct systematic us ## Securing Infrastructure at Scale: Introducing Pinterest's Resource Provisioner Pipeline (RPP) DevFeed: [Securing Infrastructure at Scale: Introducing Pinterest's Resource Provisioner Pipeline (RPP)]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-07-22T16:01:02Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [Infrastructure as code](), [Terraform](), [CI/CD](), [GitHub Actions](), [Amazon Web Services](), [GitHub](), [AWS IAM](), [Security](), [Provisioning](), [Amazon S3](), [Kubernetes](), [networking]() Tags: [aws](), [ci-cd](), [cloud-infrastructure](), [engineering](), [github-actions](), [iam](), [infrastructure](), [infrastructure-as-code](), [kubernetes](), [networking](), [pinterest](), [pull-request](), [s3](), [security](), [terraform]() ### AI overview Pinterest's Resource Provisioner Pipeline (RPP) is a proprietary Terraform execution engine for securely managing infrastructure changes across a distributed, multi-repository environment. The article describes its centralized GitHub Actions workflow, compliance controls, role chaining, and management of AWS infrastructure including IAM, networking, S3, compute, and Kubernetes resources. ### Source excerpt Ammar Ekbote | Senior Software Engineer Chan Kim | Senior Software Engineer Managing Infrastructure as Code (IaC) across a massive organization comes with a unique set of security and logistical challenges, particularly when operating within a distributed, multi-repository architecture. At Pinterest, we designed the Resource Provisioner Pipeline (RPP), our specialized, proprietary Terraform execution engine to safely manage both critical and non-critical infrastructure changes. In this post, we will look under the hood of the first iteration of the RPP system. We will explore how it established compliance and provides robust security guardrails for our global AWS operations by utilizing dual controls, centralized GitHub Actions execution, and a secure role-chaining mechanism. If your engineering team is looking to add strict guardrails to a GitHub workflow-driven Terraform setup, our architecture might serve as a helpful blueprint. What is RPP? RPP delivers a secure, standardized CI/CD workflow for deploying and managing Pinterest's foundational AWS infrastructure. Today, the system manages hundreds of Terraform workspaces that collectively govern tens of thousands of resources. This includes: Security Policies: Managing critical IAM roles and access policies. Networking Infrastructure: Configuring VPCs, Security Groups, Load Balancers, and DNS. Storage & Compute: Provisioning S3 buckets (including access controls) and Kubernetes clusters. The RPP GitHub WorkflowThe Multi-Repo Challenge Pinterest's Terraform code is currently distributed across multiple repositories, each owned and maintained by a completely different team. While we have an ongoing initiative to consolidate these into a unified mono-repository, RPP acts as a vital bridge that securely supports this legacy multi-repo structure. Centralized Execution Model To maintain absolute control over infrastructure states, RPP enforces a centralized execution model for all code changes: Workflow Invocation: The ## Achieving Near-Linear Training Scalability for Pinterest's Foundation Models DevFeed: [Achieving Near-Linear Training Scalability for Pinterest's Foundation Models]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-25T16:01:02Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [Training AI Models](), [NCCL](), [networking]() Tags: [aws](), [distributed-training](), [embedding](), [engineering](), [foundation-models](), [gpu](), [machine-learning](), [nccl](), [pinterest](), [scalability](), [training]() ### AI overview Pinterest describes optimizing multi-node distributed training for an embedding-heavy foundation ranking model. The reported improvements raise scaling from 1.13x to 2.0x at two nodes and from 1.21x to 3.9x at four nodes, with 7.5x scaling at eight nodes. ### Source excerpt Sheng Huang | Software Engineer, AI Platform; Pong Eksombatchai | Machine Learning Engineer, Applied Sciences; Saurabh Vishwas Joshi | Software Engineer, AI Platform; Gaurav Arora | Software Engineer, AI Platform; Karthik Anantha Padmanabhan | Engineering Director, AI Platform At Pinterest, foundation models power recommendations for over 600 million monthly active users. Our latest Foundation Model (ACM RecSys 2025) pre-trains on two years of user activity data and is deployed into Home feed and Related Pins ranking, the platform's two most important recommendation systems. Multi-node distributed training is the key to unlocking the next level of that capacity.ยน But when we first attempted multi-node training, adding a second machine made training 5x slower, producing a scaling factor of roughly 0.2x. Enabling AWS Elastic Fabric Adapter (EFA) for OS-bypass networking fixed the networking layer and recovered a viable baseline, but scaling was still poor: 1.13x at 2 nodes and 1.21x at 4 nodes. Three extra nodes, 3x more GPUs, 3x more cost, yet only 21% more throughput. This post describes how we took 2-node scaling from 1.13x to 2.0x and 4-node scaling from 1.21x to 3.9x (97.5% of ideal), then extended to 8 nodes at 7.5x. The larger models this unlocked have driven significant engagement gains across Pinterest's recommendation surfaces. Figure 1: Training scalability before and after optimization. Left: before EFA and optimization, adding a second node degraded throughput to 0.2x of single-node. Right: after optimization, scaling is near-linear across 2, 4, and 8 nodes, with 8-node reaching 7.5x (93.75% of ideal).Background Training scalability measures whether adding more resources yields proportionally more throughput. Training efficiency measures how much throughput you extract from the same resources. This post focuses on scalability. Our Foundation Ranking Model is embedding-heavy: approximately 99% of parameters reside in embedding tables, with the dense transf ## Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework DevFeed: [Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-24T18:01:01Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [schema-evolution](), [Apache Iceberg](), [Automation](), [Databases](), [Apache Spark](), [Kafka](), [Pull Request](), [Code generation](), [datasets](), [monitor](), [Provisioning]() Tags: [automation](), [code-generation](), [datasets](), [engineering](), [icebergs](), [kafka](), [monitor](), [pinterest](), [process](), [schema-evolution](), [software](), [spark](), [storage]() ### AI overview Pinterest describes an automated schema-evolution framework for its CDC-based database ingestion platform, which uses Kafka, Flink, Spark, and Iceberg. The framework propagates supported schema changes across the stack, uses pull-request-based versioning and auditing, targets SLA-based consistency between online and offline schemas, and provides recovery paths for unsupported or ambiguous changes. ### Source excerpt Yisheng Zhou | Software Engineer II Liang Mou | Sr Staff Software Engineer Gabriel Raphael Garcia Montoya | Staff Software Engineer Istvan Podor | Staff Software Engineer Introduction In the first post of this series, we introduced Pinterest's next-generation CDC-based ingestion platform built on Kafka, Flink, Spark, and Iceberg. In production, upstream schemas are constantly evolving, and in a distributed CDC pipeline, schema is not just metadata -- it is a cross-system contract spanning ingestion, transformation, storage, and historical backfill. A schema change that is not handled carefully can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline representations. This post walks through how we make schema evolution safe in practice: the onboarding model it builds on, the changes we support and the tradeoffs we accept, how updates propagate across the stack, and how we monitor and recover when things go wrong. We end with where we're heading next -- zero-gap schema evolution. Background & Motivation Schema evolution was particularly challenging in our environment because our ingestion pipeline is composed of multiple tightly related stages, each of which depends on schema in a different way. A single table onboarding flow spans CDC source configuration, Kafka provisioning, Flink and Spark code generation, Iceberg CDC and base table creation, and historical bootstrap -- all driven by the same schema. Once a source schema changes, that change must be reflected across code, metadata, and storage in a controlled and auditable way. Without a unified automation framework, manual updates across these layers increase the risk of drift, failed deployments, and inconsistent datasets. We therefore needed a way to make schema evolution safe, repeatable, and scalable. Our Solution: SLA-Based Automated Schema Evolution To address this problem, we built an automated schema evolution framework on top of our CDC ingestion platform, providing: au ## Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use DevFeed: [Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-05-21T16:01:00Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [data](), [recommendation systems](), [datasets](), [Machine learning](), [Embeddings](), [Feature Engineering](), [Inference](), [Latency](), [systems]() Tags: [data](), [data-infrastructure](), [datasets](), [embeddings](), [engineering](), [feature-engineering](), [inference](), [latency](), [machine-learning](), [pinterest](), [production](), [recommendation-system](), [recommendation-systems](), [retrieval](), [systems](), [train](), [transformers]() ### AI overview Pinterest describes a redesign of its user-sequence platform for ranking, retrieval, and recommendation workloads. The article explains how enriched event sequences support training datasets, offline analysis, online inference, and latency-sensitive production use cases, with goals of reducing cost, improving extensibility, and simplifying debugging. ### Source excerpt Authors (listed alphabetically) Ads Feature Engineering Infra team: Ajay Venkatakrishnan, Le Zhang Core ML Infra team: Eric Shang, Pihui Wei ML Data team: Connor Votroubek, Yi He User Understanding team: Camilo Munoz, Simin Li If you work on ranking, retrieval, or recommendation systems, you've probably asked for some version of the same thing: "Give me the last N meaningful actions this user took, with the right enrichments, in a format that's easy to train and serve ML models." On paper, that sounds simple. In practice, "user sequences" often become one of the most expensive and fragile parts of the ML data stack. They end up powering everything from training datasets to offline analysis and online inference, so they need to be fresh and complete at the same time. They must remain consistent as you add new events and enrichments. And they have to do all of this while serving latency-sensitive production workloads. This article walks through how we redesigned our user-sequence platform to make these sequences cheaper to run, faster to extend, and easier to debug, while still supporting demanding production use cases. What We Mean by "User Sequence" In this context, a user sequence is an ordered list of recent, relevant events for a user, along with the enrichments (signals) attached to each event. Here, enrichments mean all the extra signals we attach to raw events, so they're useful for models: embeddings (for example, Pin or query representations), contextual features (such as surface, device, or country), and derived attributes or counters that describe how the user interacted with a piece of content over time. A concrete example helps. Imagine a sequence made up of the last 500 engagements a user had with Pinterest Pins. Each event in that sequence might carry a timestamp, an action type, the surface where the action occurred, and a handful of embedding features or categorical attributes. As a data primitive, user sequences are powerful. They capture temporal b ## An Engineer's Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent... DevFeed: [An Engineer's Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent...]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-05-12T16:01:00Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [Agent Skill](), [Artificial Intelligence](), [Claude Code](), [codex](), [Bash](), [iOS](), [JSON]() Tags: [agent](), [agent-skill](), [agentic-coding](), [agents](), [ai](), [ai-agents](), [analysis](), [architecture](), [bash](), [claude-code](), [code](), [codex](), [debug](), [engineering](), [engineering-culture](), [how-to](), [openai](), [pinterest](), [programming](), [skills](), [testing](), [workflows]() ### AI overview An engineer's guide to testing AI skill invocation in repositories and custom skills. It describes a Bash-based harness that runs positive and negative prompts, captures verbose logs, parses JSON-streamed output, and measures invocation reliability across an internal Codex fork and Claude Code. ### Source excerpt An Engineer's Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent Performance in Any Repository or Skill Author: Daniel Reed The tech industry is currently seeing a massive overhaul in the way we work and many are enjoying the benefits of AI agents, particularly when automating engineer workflows and serving domain-specific knowledge. However, relying on agents to consistently invoke a custom skill can be surprisingly unreliable at times. When adopting a new skill intended to help agents write code for Pinterest's iOS architecture (I'll call it rx-mvvm) we discovered that sometimes our knowledge skill wasn't being loaded into our agents. To address this, we conducted a series of tests on Pin-agent (an internal fork of OpenAI's Codex) and Claude Code to quantify the reliability of skill invocation and identify some best practices to maximize performance. This was a direct result of observing agents struggling to meet the skills bar during architectural reviews. We found that by applying different techniques we could track and drastically improve skill invocation rates on both tested agents. How to Build A Skill Test Harness Building a reliable test harness for agent skill invocation requires three key components working in concert. The Core Tool is a Bash script that orchestrates automated testing by piping prompts to your agent and capturing verbose output logs. The core execution is simple: if echo "$prompt" | claude --print --verbose --output-format stream-json > "$log_file" 2>&1; then command_success=true fi The script runs all test cases in sequence, collecting logs for later analysis. We ran the entire suite multiple times to account for the nondeterministic nature of agents. Prompts were categorized into two categories defined as arrays: Positive Cases -- 15 prompts covering the full spectrum of skill domains: CORE_PROMPTS=( "load the rx-mvvm-architecture skill" "check if this follows rx-mvvm patterns" # ... 13 more cases ) Negative Case