# Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs

DevFeed: [Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs](<https://devfeed.tech/articles/using-llms-to-analyze-spark-sql-plans-a-practical-approach-to-debugging-long-running-jobs-19740.md>)

Original publisher: [Read original article](<https://medium.com/expedia-group-tech/using-llms-to-analyze-spark-sql-plans-a-practical-approach-to-debugging-long-running-jobs-35eace7eeec4?source=rss----38998a53046f---4>)

Author: Sagibhuvana

Published: 2026-06-30T11:01:01Z

Content type: article

Language: en

Sources: [Expedia](<https://devfeed.tech/sources/expedia.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [debugging](<https://devfeed.tech/topics/debugging.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [MCP](<https://devfeed.tech/topics/mcp.md>)

Tags: [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [innovation](<https://devfeed.tech/tags/innovation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [mcp-server](<https://devfeed.tech/tags/mcp-server.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [spark](<https://devfeed.tech/tags/spark.md>)

## AI overview

This article describes an automated workflow that uses large language models to analyze Spark SQL physical plans, identify performance anti-patterns, and suggest fixes for long-running jobs. It uses an open-source Spark MCP server to provide SQL execution metadata, plans, stage and task metrics, shuffle, spill, and I/O statistics.

## Source excerpt

Expedia Group Technology -- InnovationUsing large language models to reveal bottlenecks in Spark SQL execution plansPhoto by Luis del Río If you've ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you. Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale. A query that runs fine on day one can slow to a crawl as data grows, joins get wider, and aggregations become more nested. Suddenly, jobs take hours instead of minutes, clusters need to be over-provisioned, and costs climb with no clear explanation. When things go wrong, the situation gets worse. Debugging Spark SQL often means wading through deeply nested physical plans, fragmented UI metrics, and stage-level statistics that don't clearly point to the root cause. Engineers are left guessing whether the issue is a bad join strategy, skewed data, excessive shuffling, or memory pressure -- turning optimization into a time-consuming, trial-and-error exercise. So we built an automated, LLM-powered workflow that reads Spark SQL plans, flags performance anti-patterns, and suggests concrete fixes. Here's why we built it, how it works, and what we learned after running it on real workloads. Why we even needed this Across teams we kept bumping into the same pain points: Slow or failing jobs that needed deep Spark expertise to debug SQL that didn't behave the way people assumed Skew, spills, or partition issues hiding beneath the surface Tuning efforts that were reactive and inconsistent At some point, the obvious question came up: Can an LLM look at a Spark SQL plan and tell you what's likely to go wrong -- where cost will spike, where execution will drag, and where seemingly harmless SQL hides real operational risk? Turns out: yes How we approached it1. Bring in real context with the Spark MCP server We used an open-source Spark MCP server that exposes: SQL exec