WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=0RNNfxpdbQk NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/0RNNfxpdbQk.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.560 --> 00:00:02.480 Hi. s2 00:00:02.480 --> 00:00:04.720 My name is Draško Proferović. s3 00:00:04.720 --> 00:00:07.320 I'm a staff engineer at Pinterest. s4 00:00:07.320 --> 00:00:16.040 Today, I'll cover Medic for Apache Spark, which is our agentic diagnostics tool built to troubleshoot Spark failures. s5 00:00:16.040 --> 00:00:25.840 We'll dive into why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next. s6 00:00:26.240 --> 00:00:28.680 A bit of background about myself. s7 00:00:28.680 --> 00:00:33.200 I had the opportunity to work at a few companies under the data platform org. s8 00:00:33.200 --> 00:00:38.280 Despite many differences between those companies, there's at least one similarity. s9 00:00:38.280 --> 00:00:46.520 The high bar for providing quality support to partner teams who rely on the infrastructure owned by the data platform org. s10 00:00:46.520 --> 00:00:54.920 I'm sure I'm not alone when I say that the support rotation feels like a never-ending stream of questions or problems to resolve. s11 00:00:54.960 --> 00:01:02.680 Moreover, it's easy to forget how difficult it is to troubleshoot Spark or any distributed system for that matter. s12 00:01:02.680 --> 00:01:07.680 This is particularly true for anyone just getting started with the framework. s13 00:01:07.720 --> 00:01:12.840 The other challenge with supporting a load-bearing system comes down to ambiguous priorities. s14 00:01:12.840 --> 00:01:20.320 Do you focus on helping one team with their failing job, or do you unblock another team with a looming deadline? s15 00:01:20.320 --> 00:01:28.040 It's not always straightforward to rank these asks, but as humans, we often have to decide how we'll spend our time. s16 00:01:28.040 --> 00:01:30.440 The same is not true for LLMs. s17 00:01:30.440 --> 00:01:35.320 We can easily scale out knowledge and capabilities on demand. s18 00:01:36.040 --> 00:01:41.160 Our vision for a diagnostics agent was to ask it simply, "Why did a job fail?" s19 00:01:41.160 --> 00:01:47.480 and get back a deep research document which provides evidence on the root cause of the failure. s20 00:01:47.760 --> 00:01:54.120 The agent would also need to provide suggested fixes that are grounded in the context of the job. s21 00:01:54.120 --> 00:02:04.600 Needless to say, this agent would need to be available on all the surfaces where our users operate today, like Slack or the Airflow UI to name a few. s22 00:02:05.560 --> 00:02:13.560 We started by exposing our data resources by way of the model context protocol as a way to connect them to the LLMs. s23 00:02:13.560 --> 00:02:22.440 At this point, we could start an LLM conversation with the MCP tools enabled and ask the model to reason about our Spark job. s24 00:02:22.440 --> 00:02:28.560 This worked in practice, but it required a lot of careful prompting from the human operator. s25 00:02:29.520 --> 00:02:36.280 We extended our prototype by creating a single reasoning and acting agent, React for short. s26 00:02:36.280 --> 00:02:49.080 The agent was given a single prompt, which embodied the problem-solving approach it would take, how to structure the responses as a report, and specific examples to common failure patterns. s27 00:02:49.080 --> 00:02:55.440 At this point, we had enough capabilities to start trialing the solution with our beta users. s28 00:02:55.440 --> 00:03:00.840 From those early trials, we found a lot of shortcomings with our solution. s29 00:03:00.920 --> 00:03:03.720 Prompt tuning became unsustainable. s30 00:03:03.720 --> 00:03:10.840 One prompt had to do everything, and adding detail in one area degraded the behavior in another. s31 00:03:10.840 --> 00:03:13.519 Response quality was inconsistent. s32 00:03:13.519 --> 00:03:18.080 Sometimes analysis was shallow or other times too verbose. s33 00:03:18.080 --> 00:03:21.959 We lacked controls to keep the agent on track. s34 00:03:21.959 --> 00:03:25.760 And we often hit context window issues for production jobs. s35 00:03:25.760 --> 00:03:35.000 As an example, large tool outputs from logs would click quickly consume tokens and brought a halt to the agent's reasoning. s36 00:03:35.400 --> 00:03:42.239 Lastly, our end-to-end testing strategy up to this point relied on manual tests from production. s37 00:03:42.239 --> 00:03:47.120 This felt anecdotal since production data would be retentioned away. s38 00:03:47.120 --> 00:03:52.480 Overall, it was hard to know if changes broke earlier wins. s39 00:03:53.120 --> 00:03:57.840 To improve the system, we invested in observability and testability. s40 00:03:57.840 --> 00:04:01.760 We used open telemetry to publish traces to LangFuse. s41 00:04:01.760 --> 00:04:10.959 And by viewing the agent's execution as a waterfall diagram of steps, we could better understand the cause of lower quality responses. s42 00:04:11.440 --> 00:04:18.720 The reliance on manual end-to-end testing highlighted the need for a more reliable and scalable solution. s43 00:04:18.720 --> 00:04:23.560 We built an end-to-end test harness to snapshot production state. s44 00:04:23.560 --> 00:04:28.280 And we could codify expectations as offline evaluations. s45 00:04:28.760 --> 00:04:33.960 Lastly, this allowed us to tune our prompt based on the results. s46 00:04:34.280 --> 00:04:38.000 In practice, the end-to-end test harness is simple. s47 00:04:38.000 --> 00:04:45.680 In record mode, the agent calls real downstream systems, and tool responses are captured as fixtures. s48 00:04:45.680 --> 00:04:49.760 These are then saved to the file system and checked in as code. s49 00:04:49.760 --> 00:04:59.040 Playback mode, the agent runs against fixtures instead of production data, but this time performs the analysis and generates the report. s50 00:04:59.040 --> 00:05:04.640 The test suite then grades the report based on the offline evals we have authored. s51 00:05:05.240 --> 00:05:11.200 For example, an offline eval might check for a limit of three suggested fixes. s52 00:05:11.200 --> 00:05:19.240 The eval would score lower if the agent provided too many fixes towards managing the verbosity of the final report. s53 00:05:19.400 --> 00:05:25.160 Our end-to-end tests allowed us to quantify quality instead of relying on intuition. s54 00:05:25.160 --> 00:05:32.000 And as we grew our test coverage, we gained confidence that improvements did not introduce regressions. s55 00:05:32.240 --> 00:05:38.560 Once we had the testing coverage in place, we invested deeper in our handling of logs. s56 00:05:39.440 --> 00:05:43.760 Logs are noisy, and many exceptions we see in logs are benign. s57 00:05:43.760 --> 00:05:49.400 So, simply focusing the last exception may not always be suitable. s58 00:05:49.400 --> 00:05:56.280 Initially, we kept it simple with a heuristics based approach using regex to filter out certain exceptions. s59 00:05:56.280 --> 00:05:58.280 But, this didn't scale well. s60 00:05:58.280 --> 00:06:02.080 Instead, we built the exception classifier pipeline. s61 00:06:02.080 --> 00:06:13.000 The core idea was we would learn which exceptions commonly appear in successful jobs, treat those as likely red herrings, and filter them out in the future analysis. s62 00:06:13.240 --> 00:06:23.320 The agent would fingerprint and cluster exceptions, then rank them based on content relevance, and how recently they occurred compared to the termination of the job. s63 00:06:23.560 --> 00:06:28.880 The agent stopped consuming logs directly, and instead was given two MCP tools. s64 00:06:28.880 --> 00:06:36.360 Get the top K truncated exceptions, or get full log details for a specific exception. s65 00:06:36.640 --> 00:06:46.560 This resulted in an improvement to our signal-to-noise ratio, and reduced the chance of the LLM to anchor its investigation with a misleading exception. s66 00:06:47.840 --> 00:06:54.240 Like logs, we found that we could improve the overall quality by investing in how we handle metrics. s67 00:06:54.240 --> 00:06:58.000 Raw time series metrics are not context window friendly. s68 00:06:58.000 --> 00:07:05.760 Simply feeding the raw data to an LLM works in the small scale, but fails for long-running jobs in production. s69 00:07:05.760 --> 00:07:09.919 Not to mention, it's horribly token inefficient. s70 00:07:09.919 --> 00:07:15.800 The approach we took was to perform metrics analysis in a quarantine sub-agent. s71 00:07:15.800 --> 00:07:23.120 We would convert the raw time series data into graphs, which are then collaged into an final image. s72 00:07:23.120 --> 00:07:33.440 The appearance of which isn't much different than a Grafana dashboard, albeit with annotations that we found useful, like calling out the min and max values. s73 00:07:33.440 --> 00:07:46.000 The image is then attached the LLM conversation, and we would prompt the model to reason about patterns in the the Images worked better because we could guarantee how many input tokens s74 00:07:46.000 --> 00:07:52.000 would be used for analyzing any given Spark job irrespective of its duration. s75 00:07:52.280 --> 00:08:07.120 Examples of useful signals we would be able to get included executors dropping down to zero or near zero, long plateaus or bottlenecks, effectively any resource behavior inconsistent with healthy progress. s76 00:08:07.200 --> 00:08:17.280 Our sub agent would summarize its findings and return the results back to the parent agent, thereby ensuring the context window is kept healthy. s77 00:08:17.680 --> 00:08:21.120 Lastly, we overhauled our agent harness. s78 00:08:21.120 --> 00:08:26.440 We went from a single react agent with multi-agent architecture. s79 00:08:27.040 --> 00:08:31.960 This was accomplished by building on top of LangGraph's deep agent library. s80 00:08:31.960 --> 00:08:36.919 Each agent now had a dedicated prompt and a subset of MCP tools. s81 00:08:36.919 --> 00:08:45.880 Meanwhile, the deep agent library itself provided built-in tools to keep the agent on track, like a to-do list or a virtual file system. s82 00:08:45.880 --> 00:08:53.000 This approach mirrors what we come to expect from our coding tools like Cloud Code or Codex, to name a few. s83 00:08:53.040 --> 00:08:57.480 We could finally decompose our single prompt into specialized roles. s84 00:08:57.480 --> 00:09:10.839 This refactor provided clear separation of each agent and it made it easier for developers to maintain each prompt separately while performing focused testing on the system using our end-to-end test harness. s85 00:09:10.839 --> 00:09:19.520 A pleasant consequence of this architecture was that the effort to expand the scope of the project was as simple as adding a new prompt. s86 00:09:19.520 --> 00:09:27.200 And this is how we were able to extend the Medic to also help users optimize their Spark SQL jobs. s87 00:09:28.040 --> 00:09:40.320 Our workflow started starts with the user's request entering the system, whereupon intent is classified as either requiring a simple answer to a question or a deep diagnostic session. s88 00:09:40.320 --> 00:09:46.240 If it's the latter, the triage agent determines the Spark job's life cycle state. s89 00:09:46.240 --> 00:09:52.680 And if it's failed, it'll use a subset of tools to generate a set of failure hypotheses. s90 00:09:52.680 --> 00:09:59.160 Each hypothesis is researched in parallel where we gather evidence to validate it. s91 00:09:59.160 --> 00:10:03.280 These research agents then return a score and a root cause. s92 00:10:03.280 --> 00:10:15.000 The supervisor selects the highest confidence root cause and invokes the healer agent to offer remediations based on runbooks ingested into our vector database. s93 00:10:15.120 --> 00:10:22.560 Lastly, the supervisor agent would assemble the final report ensuring proper formatting. s94 00:10:23.720 --> 00:10:31.840 The multi-agent architecture proved very effective offering us the greatest control over the system's behavior. s95 00:10:31.920 --> 00:10:39.760 Enhancements to log handling led to substantial reduction in inaccurate root causes. s96 00:10:40.240 --> 00:10:50.720 We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle compared to the reasoning and acting agent paradigm. s97 00:10:50.839 --> 00:10:59.200 What we're experimenting with now is incorporating user feedback from prior sessions to automatically improve the agent. s98 00:10:59.200 --> 00:11:07.040 Lastly, we see a broader opportunity to apply this pattern to other distributed systems like Flink and Trino. s99 00:11:08.080 --> 00:11:13.800 Medic for Apache Spark project was made possible by the hard work from these contributors. s100 00:11:13.800 --> 00:11:16.280 Thank you for your time.