Why We Killed Our Multi-Agent Pipeline — Subbiah Sethuraman and Abhilash Asokan, ZS Associates

AI Engineer · 15 min · 258 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12Uh okay, so I'm Subbiah.
3. 00:14I had AI engineering at ZS.
4. 00:16Uh I'm Abhilash.
5. 00:17I'm director of AI
6. 00:20So, ZS, we are a tech firm.
7. 00:21We work with many of the top companies in the world including a lot of the top pharmas actually.
8. 00:25Um so, today's talk I think we wanted to as I think already introduced, right?
9. 00:29We wanted to talk from our experience, right?
10. 00:31Building multi-agent pipelines.
11. 00:33Uh what are the mistakes we did, right?
12. 00:35And what did we learn and how did we fix them?
13. 00:38So, um so, I'm going to orient it more on pharma commercial domain and for people in the room probably who are not aware quickly um pharma has two main functions.
14. 00:46One is R&D, right?
15. 00:47The drug discovery, right?
16. 00:49And the clinical trials part.
17. 00:50And then there's a commercial.
18. 00:51How do you take a drug to a patient basically?
19. 00:54And and within commercial there are different functions.
20. 00:58Like once you create a drug, right?
21. 00:59Um what is the performance of a brand?
22. 01:01How is the drug performing in different markets, right?
23. 01:04Uh then there are things around your field force, your reps.
24. 01:07How effectively are they engaging uh right?
25. 01:09With everyone?
26. 01:10There are things around patient journey.
27. 01:11How a patient are adopting a drug, right?
28. 01:14I think if there's any therapy switch which is happening.
29. 01:16So, as you can think about there is a lot of analytics which really happens in a commercial domain.
30. 01:21And how do typically analysts work?
31. 01:25So, there are four steps, right?
32. 01:26What analysts do, right?
33. 01:28So, first there is always something called signal detection, right?
34. 01:31So, signal can be in something like okay, the prescriptions what a doctor is writing.
35. 01:35Maybe is there is there a drop in the prescription?
36. 01:37So, that's a signal.
37. 01:39So, once you got a signal, the second thing what an analyst does is why is the signal really failing?
38. 01:44What is the reason for it?
39. 01:45So, is it like there is a competitor drug which has come in?
40. 01:48Because of that is it is it reducing?
41. 01:50Is it is it because maybe a payer coverage for the drug has reduced?
42. 01:53Or maybe the the reps on the ground does not proper uh they are actually not taking the benefits to the doctors.
43. 02:00And once you arrive at the reason, the next step becomes, "Okay, what is the action do you take?"
44. 02:05So, if reps Suppose if reps the coverage is not good in a particular region, should do we have to increase that?
45. 02:12And once you do that, what is the what is the outlook, right?
46. 02:15Is my brand is my sales performance is is it going to improve, right?
47. 02:18So, these are the four things which happens.
48. 02:21Now, for some of the top farmers, what we have done is how do we in an agentic way, right?
49. 02:26I think how do we actually mimic this this analytics work?
50. 02:29So, what we did, we built agents for every step, right?
51. 02:33Signal detection, we said, "Okay, we'll have an agent for signal detection.
52. 02:36It'll identify identify the signals for me."
53. 02:38Second, what is the root cause, right?
54. 02:40For the signals, right?
55. 02:42So, in this case, we have two agents.
56. 02:43One, we call it a source localization.
57. 02:45So, for example, if my sales is dropping at a national level, is it because it's dropping at say a particular region, or is it dropping for a payer?
58. 02:53So, we need to understand that.
59. 02:54We are We are trying to identify the source of it.
60. 02:57And then, once we understand what is the real reason, right?
61. 02:59I think the the sales performance has gone down.
62. 03:03That's another agent, the driver attribution agent.
63. 03:05And then, the last step is your synthesis, right?
64. 03:07So, depending on the cause, now what is the action you have to take and what is the outlook, like typically how an analyst used to do.
65. 03:13And all of this, we used to have an orchestrator agent which connects all these agents together.
66. 03:18So, now, what happened once we had the system, what did it generate, right?
67. 03:22It generated something an information packet like this, right?
68. 03:25It clearly tells you the signal, right?
69. 03:27So, first it says, "Maybe my brand's prescriptions have dropped 18% in some territory, right?
70. 03:32In some timeframe, maybe 4 weeks."
71. 03:34It then tells you the reason.
72. 03:36Why did it Why did it drop, right?
73. 03:38The reason it says because a payer actually, right?
74. 03:41The coverage for this drug has actually they moved it to a lower tier.
75. 03:45So, for patient it's expensive actually to to buy this drug.
76. 03:49The action it says, "Okay, because doctors are writing less prescriptions, maybe send more sales reps to talk to doctors, right?
77. 03:57And increase the number of prescriptions you're writing.
78. 04:00And then if you take this action, maybe your outlook your sales performance is going to increase.
79. 04:04So, all of this looks good high level, but then if you if you look at at it closely, it's not very coherent, right?
80. 04:11The cause is right.
81. 04:12It identified the right cause, right?
82. 04:14Because patients can't afford the drug drug, right?
83. 04:17But the action it said it didn't really focus on the payer part, the insurance part of it, right?
84. 04:22It just said reps, right?
85. 04:24Uh focus more reps actually, right?
86. 04:26And then the outlook because the action is wrong, the outlook is not going to match.
87. 04:30So, at each level if you see, it has actually derived the right fact, but then there is no single agent which is owning which understands the end-to-end picture basically.
88. 04:40So, why did this happen?
89. 04:41Like why did it fail, right?
90. 04:44So, obviously it's not the LLM which failed, right?
91. 04:46It's the way how we split the work, right?
92. 04:48Because we tried mimicking the analyst behavior, and we did it.
93. 04:52The first first key issue is like a language model is actually determining your signals.
94. 04:58So, signals like things like your sales drop is a simple information which you can use statistical methods to actually go and fetch this information.
95. 05:05You don't need a language model actually, right?
96. 05:07To fetch this information.
97. 05:09Second is as your multi agents, there is a lot of context hand off which is happening, and context is actually getting lost at each of these hand offs.
98. 05:17So, for example, the driver attribution agent is actually determining the right cause, but then the next agent, the synthesis agent, is actually is not able to understand why is that, right?
99. 05:26The the payers are finding the drug to be expensive.
100. 05:29It's not understanding the weightage of an insurance coverage going going down.
101. 05:33So, that is a key information, right?
102. 05:35Which is getting lost.
103. 05:37And the last big piece is there is no shared understanding of the business domain knowledge for all these agents.
104. 05:42All these agents don't understand metrics, right?
105. 05:45So, things like TRX, the number of transaction the number of prescriptions which a doctor writes, right?
106. 05:50What is the relationship between them?
107. 05:51Why does it go up or down?
108. 05:53So, those are the the reasons.
109. 05:55So, then what did we do?
110. 05:57Uh so, I called Abhilash to come and solve for this.
111. 06:01Uh thank you, Sufiyan.
112. 06:03So, we had like three problems.
113. 06:05Uh so, we are thinking how are we going to solve this?
114. 06:08Uh our first instinct was we'll go back to the drawing board.
115. 06:12Uh we'll start designing it again.
116. 06:13So, maybe the topology was wrong.
117. 06:15The skills were wrong.
118. 06:16The tools were wrong.
119. 06:18Or maybe the maybe we have to dis- define a better handoff.
120. 06:22Design a better handoff for a better schema between agents.
121. 06:26But we took a step back.
122. 06:27We didn't do any of that.
123. 06:29We like all of us, we went back to Cloud Code.
124. 06:33So, we opened a very plain empty directory.
125. 06:38I ran Cloud Code.
126. 06:39Then give it just bash and the database.
127. 06:43Then give it an actual signal which we identified.
128. 06:46Then started observing what it is doing.
129. 06:49So, while we look at what code does, we are able to figure out fix for all the three issues we discussed.
130. 06:55The first the first part.
131. 06:58So, what was happening is the agent was looking at data and deciding on a signal.
132. 07:05Sometimes it applied some statistical methods.
133. 07:07Sometimes it barely look at the data and say this is a signal.
134. 07:10Sometimes it's actually a signal.
135. 07:11Sometimes it's a noise.
136. 07:14This is something we don't want an agent to do.
137. 07:16This is a completely deterministic workflow.
138. 07:18So, we separated it out from the agentic system.
139. 07:22So, we built a pure deterministic workflow with different statistical methods.
140. 07:26We put guardrails.
141. 07:27We put thresholds.
142. 07:28We put prioritization.
143. 07:30Everything happened before the agent even kick starts.
144. 07:34So, we ran an automated pipeline which scans through the data.
145. 07:38Identify signals for each of the KPI.
146. 07:40Is anything happening with that?
147. 07:42Any anomalies which is happening?
148. 07:43Any trend which is very uh any trend which is breaking.
149. 07:47Based on that, we identified a signal, we put it on a queue.
150. 07:50The moment a signal comes to the queue, the agent wakes up.
151. 07:53So, the agent's job is to investigate, not to identify.
152. 07:59The second part.
153. 08:01So, mainly the issue which we previously what Subair was mentioning, there's no coherence in the output which the agents produced.
154. 08:10So, we started consolidating.
155. 08:11We look at how cloud code operates.
156. 08:13It's able to do a lot of operations, so we started designing around that.
157. 08:18So, it's repeatedly writing a function and querying database.
158. 08:21So, we give it a tool for that.
159. 08:25So, this consolidated the entire process into a single agent.
160. 08:28That doesn't mean that we didn't do parallelism.
161. 08:31We still do parallelism.
162. 08:33What we removed is, do we need distributed reasoning?
163. 08:36We didn't but the judgment to be distributed between agents.
164. 08:39That we wanted to consolidate to a single agent.
165. 08:42So, that what exactly what we did.
166. 08:44Then occasionally we observed cloud code, it's launching sub-agents dynamically for a very particular focused task.
167. 08:52So, we did the same thing.
168. 08:53Because if suppose you want to understand if rapid activity in a particular region, that's an investigation which you need to run.
169. 08:59That you can still delegate to a sub-agent.
170. 09:02You can get back the uh results back, not the reasoning or the judgment.
171. 09:06That is still controlled by the by the main agent, but the investigation and a part of it we did we're delegating it to a sub-agent.
172. 09:13So, these are some of the things we kept based on observing what cloud was doing cloud code was doing.
173. 09:18So, that give us a more lighter architecture than what was initially there.
174. 09:23But still it doesn't solve the problem.
175. 09:24That's why you see a knowledge graph in the diagram.
176. 09:28So, it still doesn't have the business context.
177. 09:31It still didn't doesn't understand all the entities, the domain, the KPIs, how do they relate to each other?
178. 09:42So, that's something we wanted to solve for because the agent was looking at data, looking at tables, then trying to infer the relationship.
179. 09:49That which was not scalable.
180. 09:50And it often produce relationship which which is not actually exist in the data.
181. 09:57So, what we did.
182. 09:59So, we've been in this working in this field for a lot lot whole whole lot of years.
183. 10:04So, we've been doing this for our clients.
184. 10:06So, we had a lot of domain experts who understand the former domain very well.
185. 10:09We're doing commercial analytics for the clients.
186. 10:11So, we start start with them, start building a knowledge graph.
187. 10:14We start start trying to map out the domain.
188. 10:19So, we are able to build a knowledge graph.
189. 10:21We are able to identify different entities, their relationship between them.
190. 10:25So, if you see, there are like geographic entities.
191. 10:27There are payers.
192. 10:29How How geographic entities connect to a payer or an account?
193. 10:33How does this connect to a brand?
194. 10:35Then from brand or payer, how does it go to a KPI?
195. 10:38How does a KPI relate to like a uh secondary KPI, tertiary KPI?
196. 10:43How does one KPI drives another KPI?
197. 10:45So, we started mapping out all of this information and then creating our our knowledge graph.
198. 10:51Once you have the knowledge graph, then we wanted let the agent to navigate the knowledge graph.
199. 10:56So, the knowledge graph is not just something the agent looks up for data.
200. 11:01It is a control plane for the agent.
201. 11:04So, what do you mean by that?
202. 11:06So, the knowledge graphs dictates what the agent can look into, what path it can take, what investigation hypothesis uh it can evaluate.
203. 11:17So, for example, if you when you do this analysis, the root of the problem, say something like your TRX is declining at a national level.
204. 11:24This could be when you do source localization.
205. 11:26So, that's our uh contact for how to find the where.
206. 11:31It might happen within a region.
207. 11:32It could be concentrated in a territory.
208. 11:34It would be concentrated in a combination of a territory or a pair or an account.
209. 11:38So, there are whole lot of dimensions which the agent needs to evaluate.
210. 11:42Now, it's a lot of permission permutation combination.
211. 11:45So, the knowledge graph guides the agent on how to find where this is concentrated.
212. 11:50Then the why why part of it.
213. 11:53Why it is happening?
214. 11:54When you figure out that okay, something is concentrated at let's say the decline is concentrated at a particular region.
215. 11:59Now, you need to figure out the why.
216. 12:01This is where where the KPIs and the relations come comes in.
217. 12:05Once the agent is able to narrow down the where, then it's able to go and figure out the uh figure out the why part of it.
218. 12:13Which KPIs is driving what?
219. 12:16Graph access the control surface.
220. 12:18So, agent every edge is a hypothesis.
221. 12:21So, the agent can go and evaluate that hypothe- hypothesis.
222. 12:25Uh it doesn't go outside of this.
223. 12:27So, that gives a more bounded surface for the agent to investigate.
224. 12:34So, like everyone was mentioning about loop, so we also build a loop.
225. 12:38So, what the agent does?
226. 12:40So, the agent first start with an entity.
227. 12:44It goes to the graph.
228. 12:46It looks at the neighborhood of the Then it it figures out the edges.
229. 12:51So, it's got some hypothesis.
230. 12:53It'll go It'll go back to the original data.
231. 12:56Evaluate that hypothesis.
232. 12:59Look at the actual numbers.
233. 13:01Then reason over it.
234. 13:02Then it will it will either find it find it contradicting or it finding supporting the the evidence supporting the hypothesis.
235. 13:09If it's supporting it, it started traversing through the graph.
236. 13:13So, this repeats this until it run out of all the hypothesis or it's able to figure out the root cause.
237. 13:20So, this this concludes this run.
238. 13:23This runs and it'll be able to figure out the uh root cause of the problem.
239. 13:28So, once you build this, so maybe like after like 50 plus turns, a whole lot of tokens, it's able to produce something an analyst was able to produce maybe in three or four weeks in like maybe 20 30 minutes.
240. 13:41Thank you.
241. 13:44So, just to wrap it up, key takeaways.
242. 13:48First thing is I think we should not be introducing human constraints or design constraints into architecture.
243. 13:53I think let the architecture be derived actually.
244. 13:56Number one.
245. 13:57Second, I think any complex workflows will have deterministic parts and agentic parts.
246. 14:01Don't let agents actually run the deterministic part, right?
247. 14:04So, I think we need to break break that off.
248. 14:08The third, need to have one agent, right?
249. 14:11Which owns the reasoning end to end, right?
250. 14:13This agent can actually take the call to use sub agents, tools, skills to actually spawn off other other tasks basically, right?
251. 14:19But you need one agent to own the the reasoning.
252. 14:23And the last, probably the most important, I think graph cannot be treated just as a lookup layer.
253. 14:28I think graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically.
254. 14:36Okay, with that, I think thank you.
255. 14:38Thanks.
256. 14:39Thanks for attending this.
257. 14:40[applause]
258. 14:57[music]