# Helpshift Engineering blog for Helpshift - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## From Pass/Fail to Confidence Levels: The Evolution of QA in the AI Era DevFeed: [From Pass/Fail to Confidence Levels: The Evolution of QA in the AI Era]() Original publisher: [Read original article]() Author: Gayatri Panganti Published: 2026-07-22T06:16:20Z Content type: article Language: en Sources: [Helpshift]() Topics: [Testing](), [Software Testing](), [AI Agent](), [Large Language Model]() Tags: [agents](), [ai](), [ai-agents](), [ai-ready-qa-mindset](), [beyond-traditional-qa](), [bug](), [configuration](), [consistency](), [qa](), [qa-beyond-pass-or-fail](), [software-testing](), [testing](), [trust]() ### AI overview This article explains how quality assurance for AI features differs from traditional deterministic software testing. It argues that testing should assess behavior, consistency, safety, policy adherence, and trust across varied prompts and contexts rather than relying only on binary pass/fail results. ### Source excerpt For decades, software testing followed a familiar pattern. A feature either worked or it didn't. A button click resulted in an expected outcome. An API returned a predefined response. A workflow either passed or failed. Large Language Models (LLMs) have changed that equation. 🤖 Modern AI agents don't simply execute the instructions- they interpret, reason, generate and adapt. The same prompt may produce different responses across executions while still being technically correct. As a result, Quality Assurance is evolving from validating deterministic outputs to evaluating confidence in AI features and behaviour. That question is no longer : >> Did the feature work? It has become: >> Can we trust how the AI behaves across a wide range of real world situations? Traditional QA Vs AI QA Traditional applications are deterministic. If the same input is provided ten times, the output should be identical ten times. -- -- -- Input -> Business Logic -> Expected Output -- -- -- AI systems behave differently. Same Prompt -> Correct Answer -> Different (but still correct) Answer -> Hallucination -> Policy Violation -> Escalation -> Rejection/Resolution Every one of these outcomes may be observed from the same feature depending upon the context, prompt wording, conversation history, model behaviour or configuration. Accordingly AI bug can be classified as, Prompt Issue / Model Behaviour / Product Bug / Configuration Issue This does not mean the AI feature is broken, it means the testing strategy must evolve. -- -- -- Pass/Fail Is No Longer Enough Traditional QA measures correctness. AI QA measures behaviour. Instead of asking whether a test passed, QA teams increasingly evaluate questions such as, Was the response helpful? Was it aligned with the expected response? Did it follow the configured instructions? Was sensitive information protected? Was the response safe and comply with organisation policies? Did it performed the configured actions correctly? Was the escalation triggered only w ## Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management DevFeed: [Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management]() Original publisher: [Read original article]() Author: Manav Mehta Published: 2026-06-18T07:11:45Z Content type: article Language: en Sources: [Helpshift]() Topics: [Data Quality](), [incident management](), [Monitoring](), [DataOps]() Tags: [data-engineering](), [data-quality](), [email](), [incident-management](), [monitoring](), [notifications](), [observability](), [pipelines](), [slack](), [snowflake]() ### AI overview This article describes the design of a centralized alerting and incident management framework for data quality and pipeline monitoring. The framework uses Snowflake's native Alerting capabilities with Email, Slack, and Splunk On-Call integrations to detect issues, notify the appropriate engineers, escalate critical incidents, and provide actionable context. ### Source excerpt Before We Knew Better As our data platform grew, so did the number of pipelines, scheduled tasks, and data quality checks running every day. While Snowflake provided a reliable platform for storing and processing data, operational monitoring was fragmented across multiple systems. Data quality failures were often discovered only after downstream reports showed inconsistencies. Pipeline issues sometimes required engineers to manually inspect logs, query tables, and trace execution paths before identifying the root cause. The challenge wasn't detecting failures -- we already had mechanisms to identify them. The real challenge was ensuring the right people were notified quickly, with enough context to take action. Questions during on-call incidents were often similar: Did the pipeline fail or was data simply delayed? Which validation check triggered the alert? Who should respond to the issue? How can we ensure critical failures don't get missed overnight? As the number of pipelines increased, manually monitoring these failures became increasingly difficult. We needed a centralized alerting framework. What We Actually Needed Our goal wasn't simply to send more notifications. We wanted a system that could: Detect data quality issues automatically Notify engineers through channels they already use Escalate critical incidents to on-call responders Provide actionable context instead of generic failure messages Scale across multiple pipelines and monitoring use cases Most importantly, we wanted to keep the solution as close to the data platform as possible. Since our monitoring logic already lived in Snowflake, it made sense for the alerting framework to live there as well. The Architecture We Chose To address these challenges, we designed a centralized notification and incident management framework using Snowflake's native Alerting capabilities, combined with Email, Slack, and Splunk On-Call integrations. Rather than introducing another monitoring platform, we chose to build ## Detecting Customer Churn with Structured Metrics and Behavioral Transitions DevFeed: [Detecting Customer Churn with Structured Metrics and Behavioral Transitions]() Original publisher: [Read original article]() Author: Mithil Oswal Published: 2026-06-17T09:05:24Z Content type: article Language: en Sources: [Helpshift]() Topics: [data](), [Framework](), [Support](), [Machine learning](), [Feature Engineering]() Tags: [analytics](), [case-study](), [churn](), [churn-analysis](), [churn-prediction](), [churn-rate](), [customer-churn](), [data](), [engineering](), [framework](), [metric](), [models]() ### AI overview This case study describes a Churn Intelligence Framework based on structured descriptive analytics rather than relying primarily on black-box predictive models. It uses revenue and support-ticket volume dynamics across current and previous periods to classify customers as attrited, declining, growing, new, or stable, with the goal of identifying multi-period decline before full attrition. ### Source excerpt Photo by Deng Xiang on UnsplashOne in a Million Ways to Detect Customer Churn -- Powered by Pure Metric EngineeringA real-world case study on building a Churn Intelligence Framework using revenue dynamics, structured KPI design, and behavioral transitions.😯 Wow, Churn Prediction sounds impressivePhoto by Ksenia Yakovleva on Unsplash Until you realize that most ML models struggle in production. -- Data fluctuates 🔢 -- Features change 💱 -- Stakeholders don't trust black-box outputs ⬛ Teams jump into feature engineering and classification algorithms, chasing accuracy scores -- while the business still lacks a clear behavioral definition of decline. The result? Black-box probabilities that stakeholders don't trust and Customer Success teams don't know how to act on. Predictive models attempt to forecast an outcome. But churn isn't just an outcome. It's a progression.So then what's new here? We use fundamentally structured descriptive analytics, aka real numbers. The objective was not to build another dashboard. It was to introduce intelligence into the existing reporting system. Specifically, the framework was designed to: Understand churn behavior structurally rather than as a single percentage metric or a boolean value. Track support ticket growth and decline across relative time periods, recognizing that volume trends directly influence revenue stability. Enable proactive client retention by identifying multi-period decline before full attrition occurs. Setting the ground❗At Helpshift, we used support ticket volume dynamics as a proxy for revenue. ➡ Definitions:- Start Date (filter) = Report start date that defines the Current Period End Date (filter) = Report end date that defines the Current Period Current Period = Revenue / issue volume for the current period timeframe Previous Period = Revenue / issue volume for the previous period timeframe, where Previous Period has been calculated by pulling back the "Current Period" dates by X days, where X is the difference betwe ## Migrating from a Monolithic Orchestrator to Apache Airflow DevFeed: [Migrating from a Monolithic Orchestrator to Apache Airflow]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2026-06-03T09:20:17Z Content type: article Language: en Sources: [Helpshift]() Topics: [airflow](), [Orchestration](), [Clojure](), [Python](), [Monitoring]() Tags: [apache-airflow](), [architecture](), [aws](), [clojure](), [data-engineering](), [etl](), [monitoring](), [monolithic-architecture](), [orchestration](), [python](), [software-architecture]() ### AI overview The article describes a migration from a monolithic Clojure-based scheduler to Apache Airflow. The legacy system coupled event pipelines, dependencies, and an EMR step, making workflows difficult to understand, maintain, monitor, and debug. The authors report that Airflow's Python-based DAGs made dependencies, retries, and scheduling more readable and helped improve onboarding and maintainability. ### Source excerpt Photo by Corinne Kutz on UnsplashBefore we knew better Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain. Understanding how a workflow executed often meant looking through multiple files, configurations and database tables. For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior. Limitations of our legacy design We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function. Legacy Workflow This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster. We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue. Photo by Tim Gouw on Unsplash There was no single place to answer basic questions like: What runs first? What happens if this step fails? How do I re-run just one part safely? The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That's when we realized we needed a better way. What we actually needed Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable. Simpler onboarding, less mental overhead Our existing step scheduler was built in Clojure and c ## Agile vs. Waterfall QA: A Comparative Guide DevFeed: [Agile vs. Waterfall QA: A Comparative Guide]() Original publisher: [Read original article]() Author: Pankaj Dusane Published: 2026-01-09T12:06:15Z Content type: comparison Language: en Sources: [Helpshift]() Topics: [Agile](), [Testing](), [Development](), [Self-organizing Team](), [Software Engineering]() Tags: [agile](), [agile-testing](), [guide](), [qa](), [software-development](), [testing](), [vs](), [waterfall-testing]() ### AI overview This comparative guide explains how quality assurance differs between Agile and Waterfall software development. It describes Waterfall QA as a dedicated testing phase after development, with fixed requirements and less flexibility, while Agile integrates QA throughout iterative development with continuous testing, feedback, and collaboration. ### Source excerpt Quality Assurance (QA) is a critical aspect of software development, ensuring that the final product meets the desired standards and functions as intended. While QA practices are integral to all development methodologies, the approach to QA can vary significantly depending on whether a project follows the Agile or Waterfall methodology. In this blog post, we'll compare these two methodologies, focusing on their impact on QA processes, workflows, and outcomes. What is Waterfall QA? The Waterfall model is a linear and sequential approach to software development. In this methodology, each phase -- from requirements gathering to design, development, testing, and deployment -- is completed before moving to the next. QA in the Waterfall model is typically conducted during a dedicated testing phase after development is complete. Key Characteristics of Waterfall QA: Late Involvement: QA is often involved only after development is finished. Fixed Scope: Testing is based on a predefined set of requirements and test cases. Extensive Testing: Since QA occurs after development, it often involves extensive end-to-end testing. Minimal Flexibility: Changes to requirements or functionality after testing begins can be costly and disruptive. Predictability: The linear nature of the process makes timelines and deliverables more predictable. Advantages of Waterfall QA: Clear documentation and test plans. Structured processes provide clarity on responsibilities and timelines. Suitable for projects with well-defined requirements and minimal expected changes. Challenges of Waterfall QA: Delayed feedback loops can prolong the discovery of critical issues. Limited ability to adapt to changes or new requirements. Potential for larger defects due to the lack of continuous testing. What is Agile QA? Agile methodology emphasizes flexibility, collaboration, and iterative development. Unlike Waterfall, Agile integrates QA throughout the development lifecycle, ensuring continuous testing and feedback ## From Data to Insight: Helpshift's Journey with ML Observability DevFeed: [From Data to Insight: Helpshift's Journey with ML Observability]() Original publisher: [Read original article]() Author: Sujit Singh Published: 2025-11-26T14:00:15Z Content type: article Language: en Sources: [Helpshift]() Topics: [Machine learning](), [observability](), [telemetry](), [Inference](), [model-deployment](), [generalization in machine learning]() Tags: [analytics](), [artificial-intelligence](), [inference](), [machine-learning](), [metrics](), [model-monitoring](), [monitoring](), [observability]() ### AI overview Helpshift describes its journey toward building a custom machine learning observability solution. The article explains ML observability, outlines system, inference, and model monitoring, discusses limitations of existing tools, and introduces an approach based on "Wide Events." ### Source excerpt Introduction In an age where artificial intelligence (AI) and machine learning (ML) are integral to almost every aspect of our lives, ensuring the effectiveness, fairness, and reliability of ML models is paramount. Observability plays a crucial role in maintaining the performance of these models, allowing us to detect and resolve issues promptly. At Helpshift, we recognized the need for robust ML observability to keep our models running smoothly and efficiently. This blog post explores our journey in building a custom ML observability solution tailored to our specific needs. We'll delve into the concept of ML observability, discuss the limitations of existing tools, and share how we implemented our own solution based on the idea of "Wide Events." Understanding ML Observability ML observability is the ability to monitor and understand the performance, behavior, and outputs of machine learning models in real-time. It enables us to proactively identify potential issues and anomalies, facilitating timely interventions and mitigating risks. ML observability encompasses several key components: System Monitoring: Tracking the performance of the infrastructure where ML services are deployed, including metrics like CPU and memory usage, network traffic, disk space, and service performance. Inference Monitoring: Evaluating and auditing the real-time performance of deployed ML models in production by tracking incoming requests and the accuracy of model predictions. Model Monitoring: Observing the long-term accuracy of ML models by monitoring key metrics such as accuracy, precision, recall, and F1-score, and detecting any drift over time. Why Observability Matters If you've ever played Age of Empires, you know how crucial it is to explore the map to manage resources proactively and strategize effectively. Similarly, ML observability is about exploring properties and patterns not determined in advance. It allows us to be proactive in debugging and improving our systems, ensuring ## How QA Validates Productivity Analytics Data Across Environments DevFeed: [How QA Validates Productivity Analytics Data Across Environments]() Original publisher: [Read original article]() Author: Gayatri Panganti Published: 2025-11-26T13:55:02Z Content type: article Language: en Sources: [Helpshift]() Topics: [Testing](), [data](), [Kafka](), [dashboards](), [JSON]() Tags: [analytics](), [dashboards](), [data-validation-testing](), [json](), [kafka](), [kafka-consumer](), [logging](), [metric](), [qa](), [testing](), [time]() ### AI overview The article describes a primarily manual QA process for a productivity analytics framework. It uses simulations, event validation through Kafka, metric checks with Metabase, timestamp recalculations, and testing across sandbox, staging, and production environments. ### Source excerpt Every accurate metric is backed by countless validations, events checks and integrity tests in the background.Introduction Quality Assurance in the data-driven systems extends beyond UI validation and backend verification. Such systems rely heavily on data precision and accuracy. A recent QA focused on validating a productivity analytics framework, ensuring that every event, metric and data flow accurately represented real-world user behaviour. The process was primarily manual, involving live simulations, event validation and detailed metric verification across environment which emphasised logic and data accuracy over automation. Simulating Real-World Scenarios Extensive simulations were conducted for multiple user roles as Agent, Supervisor, Admin and Super admin. Each governed by specific dashboard permissions and access rules. Actions tested included : Logging in and logging out from the dashboard Marking presence states such as Available, Online, Or Away Switching between workzones Navigating through dashboards These test conditions generated diverse event streams used to verify how accurately the system captured and processed state transitions. Parallel Event Validation Through Kafka Real-time validation was a key aspect of this QA process. Event streams were observed directly through kafka consoles, enabling verification of generated events and their payloads, data structures, and JSON fields as actions were executed. Each dashboard or SDK action was simulated while the corresponding kafka stream was monitored in parallel, confirming that events triggered correctly and carried accurate data information. Testing covered sandbox, staging, and production environments, ensuring reliability and consistency across all setups. Data Verification with Metabase and Calculations After events were processed by the analytics pipeline, metabase queries were used to validate computed metrics against expected outcomes from simulations. To cross-check results, timestamps (ex. ## Why Mentorship Matters? Beyond Tasks and Deadlines DevFeed: [Why Mentorship Matters? Beyond Tasks and Deadlines]() Original publisher: [Read original article]() Author: Gayatri Panganti Published: 2025-11-26T10:36:42Z Content type: article Language: en Sources: [Helpshift]() Topics: [Learning](), [meetings]() Tags: [alignment](), [async](), [communication](), [growth](), [growth-mindset](), [mentorship](), [organizational-culture](), [quality](), [team]() ### AI overview The article explains how mentorship supports skill growth, confidence, accountability, alignment between individual and team goals, and a culture of feedback. It recommends regular check-ins, clear goals, reflection, and flexible async communication during demanding project work. ### Source excerpt In every growing organisation, mentorship quietly powers progress. It's not just about reviewing work or assigning tasks but it's about helping people discover their potential, learn faster, and build confidence in their expertise. Strong mentorship programs help create a culture where learning is continuous, team alignment improves , and quality becomes a shared mindset rather than an individual effort. 🎯 Why Mentorship Is Important?1. Skill Growth Happens Faster Learning becomes faster and more meaningful with the right guidance. Mentorship helps bridge gaps that could otherwise take years to overcome whether it's technical depth, communication, or accountability. 2. Builds Confidence and Accountability When someone knows there's support and guidance available, they are more likely to take ownership, ask questions, and explore solutions without fear of failure. Mentorship creates a safe environment for experimentation and growth. 3. Aligns Individual and Team Goals Most of the times individual OKRs (Objectives and Key Results) often feel abstract until someone helps translate them into personal growth plans. Effective mentorship bridges that gap ensuring skill development aligns with both personal and organizational success metrics. 4. Encourages a Culture of Feedback Mentorship promotes open communication. Each session becomes an opportunity to exchange constructive feedback and insights. Over time, this fosters a culture of transparency and continuous improvement across the team. 🧭 What Effective Mentorship Looks Like A good mentorship rhythm involves consistent check-ins, clear goals, and space for reflection. Regular discussions/sync-ups on personal OKRs help mentees focus on targeted skill improvement. Weekly or biweekly meetings provide structure for reviewing progress and addressing blockers. During high-priority feature or project work, flexibility is key. Short async updates or quick syncs can keep the mentorship active without adding pressure. The intent ## How Data Powers Agent Productivity DevFeed: [How Data Powers Agent Productivity]() Original publisher: [Read original article]() Author: Poorva Patil Published: 2025-10-06T04:22:31Z Content type: article Language: en Sources: [Helpshift]() Topics: [data](), [dashboards]() Tags: [analytics](), [apache-spark](), [big-data](), [customers](), [data](), [data-analysis](), [metrics](), [support]() ### AI overview This article describes how a customer support team developed custom metrics to measure agent productivity more accurately. The metrics aim to distinguish productive work from idle or merely available time and support staffing, scheduling, and performance decisions. ### Source excerpt As a data engineer, I used to see metrics as just numbers on a dashboard -- until I realized they're the lens through which customers view and run their operations. In customer support, for example, agent productivity metrics aren't just figures, they're actionable insights that drive efficiency, shape staffing decisions, and directly impact customer satisfaction. These aren't just charts -- they help customers understand the value we provide, how well things are working, and what decisions to make next. Realizing this changed how I think about building analytics. ➡💡The Question That Shifted Our Perspective In customer support, how well the team works really matters. It affects how much the company spends, how happy the customers are, and how the team feels about their work. Support managers often ask: Are we staffed correctly for the volume we're handling? Are agents spending their time productively? How can we optimize scheduling and performance? When we began our Agent Workforce Management project, we already had a few standard metrics in place like online time, login time, and available time. These told us when agents were present -- but not what they were actually doing. Customers weren't asking "Are our agents online?" They were asking "How productive are our agents?" And truthfully, we didn't have a good answer. There was no visibility into how much time was being spent on real work versus idle time. No way to differentiate between being "available" and being "productive". This made it hard for teams to identify gaps, support high performers, or spot patterns that needed attention. This project was all about answering that question in the right way. 🔍📊 Custom Metrics We Built We designed a set of new metrics that give a clearer picture of how agents spend their time. These metrics give us a deeper understanding of how time is actually being spent, helping us move beyond assumptions and focus on what really drives productivity. Engagement Metrics These show how ## Using Clojure channels to increase throughput DevFeed: [Using Clojure channels to increase throughput]() Original publisher: [Read original article]() Author: Abhinav Dubey Published: 2025-05-28T10:07:12Z Content type: tutorial Language: en Sources: [Helpshift]() Topics: [Clojure](), [Concurrency](), [Kafka](), [benchmarking](), [async](), [Grafana]() Tags: [benchmarking](), [channel](), [clojure](), [concurrency](), [core-async](), [grafana](), [kafka](), [multithreading](), [parallelism]() ### AI overview This tutorial explains how to increase throughput in a Clojure-based Kafka consumer by using core.async channels and multiple worker threads. It compares a single-threaded baseline with configurations using three and ten workers, reporting shorter processing times for 1,000 events in the described benchmark. ### Source excerpt When building systems that process large volumes of messages synchronously, performance bottlenecks can quickly become a challenge specially with single-threaded designs. In this post, we'll look at how leveraging worker threads in a Clojure-based Kafka consumer can significantly boost throughput & reduce total processing time. Using simple concurrency primitives, it's possible to achieve parallelism & scale gracefully, all while keeping the codebase clean & maintainable. We'll start with a baseline, introduce worker threads using Clojure's core.async & measure the impact. Setup & Context Kafka & Zookeeper For observability: Grafana Kafka producer: A simple script that sends messages to a Kafka topic at a configurable rate (messages per minute) for a fixed duration. After each event is pushed, a counter metric is emitted Kafka consumer: A simple script that listens to a topic & consumes messages & simulates processing time finding square-root of a number (henceforth, assume that it takes ~1 second to find the square root) . A counter metric is emitted after processing each message The Baseline: Single-Threaded Consumer If each message takes t seconds to process & there are n messages, total processing time becomes n x t seconds. This provides a clean baseline to evaluate the impact of using channel moving forward. Adding workers with core.asyncValues are conveyed on queue-like channels. By default channels require producer and consumer to rendezvous for the transfer of a value through the channel https://clojuredocs.org/clojure.core.async To improve throughput, we introduce parallelism using Clojure's core.async channels. Messages from Kafka are fed into a channel, & multiple worker threads read from this channel to process messages concurrently Here, we used >!! (blocking put) &