Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest

AI Engineer · 11 min · 100 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Hi.
2. 00:02My name is Draško Proferović.
3. 00:04I'm a staff engineer at Pinterest.
4. 00:07Today, I'll cover Medic for Apache Spark, which is our agentic diagnostics tool built to troubleshoot Spark failures.
5. 00:16We'll dive into why we built the Medic, the journey from prototype to the current architecture, lessons learned along the way, and what's next.
6. 00:26A bit of background about myself.
7. 00:28I had the opportunity to work at a few companies under the data platform org.
8. 00:33Despite many differences between those companies, there's at least one similarity.
9. 00:38The high bar for providing quality support to partner teams who rely on the infrastructure owned by the data platform org.
10. 00:46I'm sure I'm not alone when I say that the support rotation feels like a never-ending stream of questions or problems to resolve.
11. 00:54Moreover, it's easy to forget how difficult it is to troubleshoot Spark or any distributed system for that matter.
12. 01:02This is particularly true for anyone just getting started with the framework.
13. 01:07The other challenge with supporting a load-bearing system comes down to ambiguous priorities.
14. 01:12Do you focus on helping one team with their failing job, or do you unblock another team with a looming deadline?
15. 01:20It's not always straightforward to rank these asks, but as humans, we often have to decide how we'll spend our time.
16. 01:28The same is not true for LLMs.
17. 01:30We can easily scale out knowledge and capabilities on demand.
18. 01:36Our vision for a diagnostics agent was to ask it simply, "Why did a job fail?"
19. 01:41and get back a deep research document which provides evidence on the root cause of the failure.
20. 01:47The agent would also need to provide suggested fixes that are grounded in the context of the job.
21. 01:54Needless to say, this agent would need to be available on all the surfaces where our users operate today, like Slack or the Airflow UI to name a few.
22. 02:05We started by exposing our data resources by way of the model context protocol as a way to connect them to the LLMs.
23. 02:13At this point, we could start an LLM conversation with the MCP tools enabled and ask the model to reason about our Spark job.
24. 02:22This worked in practice, but it required a lot of careful prompting from the human operator.
25. 02:29We extended our prototype by creating a single reasoning and acting agent, React for short.
26. 02:36The agent was given a single prompt, which embodied the problem-solving approach it would take, how to structure the responses as a report, and specific examples to common failure patterns.
27. 02:49At this point, we had enough capabilities to start trialing the solution with our beta users.
28. 02:55From those early trials, we found a lot of shortcomings with our solution.
29. 03:00Prompt tuning became unsustainable.
30. 03:03One prompt had to do everything, and adding detail in one area degraded the behavior in another.
31. 03:10Response quality was inconsistent.
32. 03:13Sometimes analysis was shallow or other times too verbose.
33. 03:18We lacked controls to keep the agent on track.
34. 03:21And we often hit context window issues for production jobs.
35. 03:25As an example, large tool outputs from logs would click quickly consume tokens and brought a halt to the agent's reasoning.
36. 03:35Lastly, our end-to-end testing strategy up to this point relied on manual tests from production.
37. 03:42This felt anecdotal since production data would be retentioned away.
38. 03:47Overall, it was hard to know if changes broke earlier wins.
39. 03:53To improve the system, we invested in observability and testability.
40. 03:57We used open telemetry to publish traces to LangFuse.
41. 04:01And by viewing the agent's execution as a waterfall diagram of steps, we could better understand the cause of lower quality responses.
42. 04:11The reliance on manual end-to-end testing highlighted the need for a more reliable and scalable solution.
43. 04:18We built an end-to-end test harness to snapshot production state.
44. 04:23And we could codify expectations as offline evaluations.
45. 04:28Lastly, this allowed us to tune our prompt based on the results.
46. 04:34In practice, the end-to-end test harness is simple.
47. 04:38In record mode, the agent calls real downstream systems, and tool responses are captured as fixtures.
48. 04:45These are then saved to the file system and checked in as code.
49. 04:49Playback mode, the agent runs against fixtures instead of production data, but this time performs the analysis and generates the report.
50. 04:59The test suite then grades the report based on the offline evals we have authored.
51. 05:05For example, an offline eval might check for a limit of three suggested fixes.
52. 05:11The eval would score lower if the agent provided too many fixes towards managing the verbosity of the final report.
53. 05:19Our end-to-end tests allowed us to quantify quality instead of relying on intuition.
54. 05:25And as we grew our test coverage, we gained confidence that improvements did not introduce regressions.
55. 05:32Once we had the testing coverage in place, we invested deeper in our handling of logs.
56. 05:39Logs are noisy, and many exceptions we see in logs are benign.
57. 05:43So, simply focusing the last exception may not always be suitable.
58. 05:49Initially, we kept it simple with a heuristics based approach using regex to filter out certain exceptions.
59. 05:56But, this didn't scale well.
60. 05:58Instead, we built the exception classifier pipeline.
61. 06:02The core idea was we would learn which exceptions commonly appear in successful jobs, treat those as likely red herrings, and filter them out in the future analysis.
62. 06:13The agent would fingerprint and cluster exceptions, then rank them based on content relevance, and how recently they occurred compared to the termination of the job.
63. 06:23The agent stopped consuming logs directly, and instead was given two MCP tools.
64. 06:28Get the top K truncated exceptions, or get full log details for a specific exception.
65. 06:36This resulted in an improvement to our signal-to-noise ratio, and reduced the chance of the LLM to anchor its investigation with a misleading exception.
66. 06:47Like logs, we found that we could improve the overall quality by investing in how we handle metrics.
67. 06:54Raw time series metrics are not context window friendly.
68. 06:58Simply feeding the raw data to an LLM works in the small scale, but fails for long-running jobs in production.
69. 07:05Not to mention, it's horribly token inefficient.
70. 07:09The approach we took was to perform metrics analysis in a quarantine sub-agent.
71. 07:15We would convert the raw time series data into graphs, which are then collaged into an final image.
72. 07:23The appearance of which isn't much different than a Grafana dashboard, albeit with annotations that we found useful, like calling out the min and max values.
73. 07:33The image is then attached the LLM conversation, and we would prompt the model to reason about patterns in the the Images worked better because we could guarantee how many input tokens
74. 07:46would be used for analyzing any given Spark job irrespective of its duration.
75. 07:52Examples of useful signals we would be able to get included executors dropping down to zero or near zero, long plateaus or bottlenecks, effectively any resource behavior inconsistent with healthy progress.
76. 08:07Our sub agent would summarize its findings and return the results back to the parent agent, thereby ensuring the context window is kept healthy.
77. 08:17Lastly, we overhauled our agent harness.
78. 08:21We went from a single react agent with multi-agent architecture.
79. 08:27This was accomplished by building on top of LangGraph's deep agent library.
80. 08:31Each agent now had a dedicated prompt and a subset of MCP tools.
81. 08:36Meanwhile, the deep agent library itself provided built-in tools to keep the agent on track, like a to-do list or a virtual file system.
82. 08:45This approach mirrors what we come to expect from our coding tools like Cloud Code or Codex, to name a few.
83. 08:53We could finally decompose our single prompt into specialized roles.
84. 08:57This refactor provided clear separation of each agent and it made it easier for developers to maintain each prompt separately while performing focused testing on the system using our end-to-end test harness.
85. 09:10A pleasant consequence of this architecture was that the effort to expand the scope of the project was as simple as adding a new prompt.
86. 09:19And this is how we were able to extend the Medic to also help users optimize their Spark SQL jobs.
87. 09:28Our workflow started starts with the user's request entering the system, whereupon intent is classified as either requiring a simple answer to a question or a deep diagnostic session.
88. 09:40If it's the latter, the triage agent determines the Spark job's life cycle state.
89. 09:46And if it's failed, it'll use a subset of tools to generate a set of failure hypotheses.
90. 09:52Each hypothesis is researched in parallel where we gather evidence to validate it.
91. 09:59These research agents then return a score and a root cause.
92. 10:03The supervisor selects the highest confidence root cause and invokes the healer agent to offer remediations based on runbooks ingested into our vector database.
93. 10:15Lastly, the supervisor agent would assemble the final report ensuring proper formatting.
94. 10:23The multi-agent architecture proved very effective offering us the greatest control over the system's behavior.
95. 10:31Enhancements to log handling led to substantial reduction in inaccurate root causes.
96. 10:40We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle compared to the reasoning and acting agent paradigm.
97. 10:50What we're experimenting with now is incorporating user feedback from prior sessions to automatically improve the agent.
98. 10:59Lastly, we see a broader opportunity to apply this pattern to other distributed systems like Flink and Trino.
99. 11:08Medic for Apache Spark project was made possible by the hard work from these contributors.
100. 11:13Thank you for your time.