Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

AI Engineer · 110 min · 1107 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00Morning.
2. 00:01[music]
3. 00:14Thanks everyone for being for being here and welcome to AIE.
4. 00:18I guess first first session for everyone and for us as well.
5. 00:22So h we wanted to introduce ourselves and tell you about what we're going to discuss.
6. 00:28We think we need to stop talking with spreadsheets.
7. 00:31So, this is all going to be about building a financial analyst harness that you can actually trust.
8. 00:37We are Franchesco Filipe.
9. 00:39A little bit of a quick introduction about me.
10. 00:42I'm originally from Venezuela.
11. 00:43I lived in the UK for about seven years and recently moved to Madrid.
12. 00:47H I work at data and I work at a bank as well with Felipe.
13. 00:53Erh.
14. 00:54I like just finding the best eeries in any city that I visit.
15. 00:57So if you have any recommendations for New York, I'm more than happy to take them and just travel around.
16. 01:02And the reason my hobbies sound so vanilla is because I just spend most of my time building stuff and nerding out at the expense of my personal time.
17. 01:11So that's how you end up with a list like that.
18. 01:14Uh and then nice.
19. 01:16All right.
20. 01:17Um happy to meet everyone.
21. 01:19Um also quick intro.
22. 01:20I'm Philipe.
23. 01:21I'm originally Brazilian.
24. 01:22Um, I also moved to the UK.
25. 01:24It's been a year and a half now.
26. 01:27So, it's been a fun journey.
27. 01:28Um, life turned pretty much upside down.
28. 01:30But, yeah, it's good to be it's good to be there.
29. 01:33Um, I like to tell a story that before going to to decide whether I wanted to do it in university, I used to play piano for like literally seven hours a day.
30. 01:41Um, and the natural decision there was to then study electrical engineering because who wouldn't, right?
31. 01:48Um, I ended up at some point landing in in data because someone showed me something that at the time called uh Google Data Studio.
32. 01:55Now it's Looker Studio, but now they went back.
33. 01:57I don't I don't even know what's going on.
34. 01:59Um, and then my first dashboard looked like a papyrus of how vertical it was.
35. 02:03You had to scroll scroll scroll scroll scroll down to see something.
36. 02:06Um, but yeah, the the rest is history.
37. 02:08So I worked at a lot of um startups and small teams and scaleups.
38. 02:13Um and then eventually went to a big company at New Bank.
39. 02:17Maybe some of you do know um and then before going back to to Monzo which is I would say a middle turn between a scale up and a big enterprise company but it's growing uh so fast for now.
40. 02:28Um and yeah most of my time currently has been working at Monzo with building internal AIdriven solutions for internal productivity primarily for the data team.
41. 02:38So, we basically want to get our data team better by um building with AI.
42. 02:43Um and yeah, when I'm not talking to robots and agents, uh you can probably find me climbing somewhere um either in nature or in a gym.
43. 02:51Um and some starting like points of why are we bringing this for you today?
44. 02:57I think um both Franchesco and I worked in data teams in financial services.
45. 03:02So again, Monzo, New Bank and a few other companies in the in the sector.
46. 03:06Um, we're now pushing what AI can do there.
47. 03:08Our team at Monzo has been um, working a lot with with adopting AI from an impact perspective.
48. 03:15We don't want to be like token maxing.
49. 03:17We don't want to be spending a lot of money if that doesn't bring us like concrete business value.
50. 03:22Um, so the adoption is very very deliberate and we've been seeing very good results on that.
51. 03:28And on the flip side, this is something also something we've been we've been doing, Franchesco particularly is applying AI to make our lives easier.
52. 03:35Um and again not through the approach of I'm going to set up an open client instance nothing against it um and just give it connections to everything but actually what are the things that I
53. 03:46care about in my life that I can um get the help of AI to to u make easier.
54. 03:53So this is why we say we're not representing Monzo um in this example because we're going to use the knowledge we bring from Monzo but uh build with you an example where we can um actually see that in practice without having to discuss a lot of domain knowledge.
55. 04:08Um but if you are curious about the things we are building at Monzo we have a breakout session tomorrow um where we're going to talk about agent chip which is our inhouse
56. 04:17remote agent capability.
57. 04:20Um so with that Franchesco back to you.
58. 04:22Cool.
59. 04:22So, as a reminder, today the goal is trust and it may seem a little bit counterintuitive because it feels very wishy-washy and this is a workshop and you're supposed to be coding really hard for two hours.
60. 04:36H but you're going to see we're going to give it a little bit of a spin.
61. 04:40Uh because I think we're past the point of having to spend hours like grinding through your keyboard uh to get something that you can trust.
62. 04:48So, let's start with the domain.
63. 04:51Finance is tailored.
64. 04:53You have your numbers.
65. 04:54You have your strategy.
66. 04:57Sometimes you don't really have a strategy.
67. 05:00And there is the knowledge that you have on finance, on your personal situation and so on or generally misconceptions.
68. 05:07H when we talk about a financial analyst, there is a very specific job description.
69. 05:12We are being a little bit generous with that job description.
70. 05:15So it's going all the way from doing research for investments, investment firms, banks and so on all the way to like personal use cases as well.
71. 05:24But finance is finance, right?
72. 05:26So right now the gap on producing something, getting hold of your finances, doing an analysis is not really speed.
73. 05:35H I'm pretty sure some of you may have some background tasks running with like claude or chpt getting some stuff done.
74. 05:43So that's not the bottleneck.
75. 05:45H we believe that based on the experience we had over the last few months, the bottleneck now is how it connects with us.
76. 05:55Um so trust is the thing we cannot take for granted and transparently because we're talking about money specifically.
77. 06:05Erh this is one of the highest stakes topics uh we need to invest in making sure that there is trust built into the system and transparently if money is not high stakes enough for you and you can like oneshot your money problems your budgeting your financing everything else h find me after the workshop I will buy you
78. 06:26in into some investment ideas so yeah let's chat later please um so where everything starts is from the goal which is earning the trust.
79. 06:38So then you can truly delegate the boring work to multiple agents, your swarms, your factories, whatever.
80. 06:45But how you do trust the results that you're getting.
81. 06:49Erh, I'm going to namerop a few things because honestly none of them are new.
82. 06:56You're probably already grinding thinking well you buy trust by h setting up traces or doing loops or any of the other keywords we have.
83. 07:05So unit testing that gets old planning with your agent we know that that's good.
84. 07:11Uh doing evals for sure traces and observability loops and graphs like I can keep naming components of how you think you can start building trust but they are basically
85. 07:23components.
86. 07:25Erh what we want to discuss today is actually the design principles on how to achieve that trust.
87. 07:31So building a hardness that actually h works for money matters h requires an in im immense amount of trust.
88. 07:40So we're not going to be discussing specific technologies.
89. 07:43We're not going to read you the readme of a particular product and we will assume you know some of the concepts that have been listed so far.
90. 07:52Um and mainly this is going to be a slightly different workshop because we want you to feel how the principles work in reality.
91. 08:02So we have built a hardness today around a very simple example and we want you to try it out but don't focus on oh if I type this then I get the response so much quicker and the UI is so slick.
92. 08:16It spoiler it won't be.
93. 08:18Erh but more importantly we want you to feel how it may feel like having trust in an agent that has been built with a very specific hardness.
94. 08:28Uh so again this is just one very simple implementation an example h you can roast me later if you think this is too top level but we can discuss that later.
95. 08:39So trust and I'm probably going to run out of times I can say trust without going insane but we think you need to build it from the ground up right.
96. 08:48So you have a shared domain, the whole thing about a plan, making sure that you and the model are talking in the same terms.
97. 08:56And it sounds more trivial than it actually is.
98. 08:59Then consistency and pretty sure everyone in the room will be on the same page, but it's making sure that the numbers come from a correct place.
99. 09:09So rather than a language model trying to process a lot of arithmetic, it will do some tool calling.
100. 09:15It will write us some scripts.
101. 09:16It will try to come up with a way that the results are uh consistent.
102. 09:22We also want evidence where basically show the receipts, right?
103. 09:27You can follow back and trace back the decisions and what led to them.
104. 09:33Accountability is probably a very spicy one and maybe the mo most abstract of them, but ultimately even if you're running a 100 agents, the results of what they do,
105. 09:46you're still accountable for them.
106. 09:48So that's something we take very seriously at work.
107. 09:51And we think it's a mindset shift that when people make it, they start building systems that evolve with them and not black boxes that they just throw problems at.
108. 10:02hoping that something sticks.
109. 10:05And finally, verification.
110. 10:07So, we want to make sure that there is a separate point of view, another perspective.
111. 10:13H I would say I'm guilty of thinking I have the best idea in the world probably like three times a year and then only to have Filipe tell me,
112. 10:22please don't implement that.
113. 10:23Please, please, please.
114. 10:25And I I Oops, one second.
115. 10:27And I would just compliment on on this bit that I think two things.
116. 10:31The first one is we have a very generous window here to to work together.
117. 10:35So um it will be very nice to learn from your folks's experience as well as we build through this principles.
118. 10:42So if this resonates with any work you've done or any particularly any challenges you've faced um when working with agents feel absolutely free to to raise that and then we can we can build it uh together.
119. 10:54Um and just dunning uh doubling down on something that Franchesco said this we're not saying this is the model you won't say us going to to the big stage and saying
120. 11:04forget everything you know this is how to do it this is a model a way to approach it uh that has been working for us in a number of use cases I'll share a few examples at Monzo later and Francisco has a few other examples as well
121. 11:18um but yeah again there's not a single silver bullet to to solve every single problem we And also feel feel free to challenge us.
122. 11:27If you think actually trust overrated just throw a 100 agents at a problem eventually will get solved.
123. 11:35Happy to hear the success stories.
124. 11:37H but from repetition from experience and a lot of different use cases at work and outside.
125. 11:42We do think you need to build a system that gives you the confidence that things are going in the right direction and not just that things look correct.
126. 11:53So yeah, trust.
127. 11:54Happy days.
128. 11:55Um, speaking of trust, I'm here to confess that I have an ulterior motive with today's workshop.
129. 12:04Uh, so breaking your trust immediately.
130. 12:06What is it?
131. 12:07Less than five minutes.
132. 12:09You are here to help me, Francesco, to plan my wedding.
133. 12:16So nobody's super excited, but you will get excited.
134. 12:20uh specifically just to figure out some some stuff around the finances of paying for a wedding.
135. 12:26Um and because it's a conference, I want to make it as cringey as possible.
136. 12:30So, show of hands.
137. 12:32Uh who has had a wedding?
138. 12:34Who's gotten married?
139. 12:37Okay, awesome.
140. 12:38Number of guests under 50.
141. 12:42Yeah.
142. 12:43Okay.
143. 12:43No, still show of hands.
144. 12:44It's still like cringey mode.
145. 12:45Okay, cool.
146. 12:47About 100.
147. 12:49Yeah.
148. 12:50Nice.
149. 12:51How about 150?
150. 12:53Okay.
151. 12:54Nice.
152. 12:56200.
153. 12:57More.
154. 12:57All right.
155. 12:58Yeah.
156. 12:58Yeah.
157. 12:58Big wedding.
158. 12:59Nice.
159. 13:01This is probably going to resonate, but it sounds like it's going to be a very simple calculation, very simple process, and the wedding is going to go by nicely and smoothly.
160. 13:11And for the people who got married, probably you know that is not the case.
161. 13:15Um, and even running the calculations and figuring out how things got so expensive so quickly.
162. 13:22It's not trivial.
163. 13:24This is a little bit of storytelling.
164. 13:26And hopefully my girlfriend doesn't fiance doesn't see this, but we were doing some calculations and she was like, "No, yeah, but the cost per like dinner is this much."
165. 13:37And I'm like, "Yeah, and you have another two variable costs on top of that.
166. 13:41So that's not the number."
167. 13:43And anyway, budgeting for this has been a challenge to underelling it a little bit.
168. 13:49I wouldn't recommend doing it, but you know, I'm already on the train.
169. 13:54Erh, so today you're going to give me a hand with the wedding unpaid labor or you can do your own plan.
170. 14:03H So we put a few things in a repo that you're going to get access to.
171. 14:06So you if you want to pick your own adventure, do that.
172. 14:10wouldn't recommend because I do need help, but whatever you choose.
173. 14:15So, we'll give you a few minutes uh for you to set this up.
174. 14:21Just either scan the QR code or go to my uh repo and there will be some instructions on how to set it up.
175. 14:33We did not plan a background music like an elevator music for this moment.
176. 14:38Yeah, let us know if you have any any trouble setting up.
177. 14:44Also, if anyone sings, it would be a good time to
178. 14:47karaoke live.
179. 14:51By the way, you don't need to execute anything just yet.
180. 14:54Just make sure you have the repo in your preferred um setup and then we'll guide you through the the execution.
181. 15:01Cool.
182. 15:02Uh, so you just cloned a virus.
183. 15:06No, just kidding.
184. 15:06But a lot of vioded stuff in there.
185. 15:10The structure does make sense.
186. 15:12So we have a guide on how we start for today's workshop.
187. 15:16H, we have the example most important thing of the talk today.
188. 15:20So helping me figure out my wedding.
189. 15:22H, also the harness itself, the thing we build to show you how the principles are applied.
190. 15:28And then finally, there is a folder where everything you build that drifts away from what we're doing is going to get stored in.
191. 15:36So again, it's a little bit of a pick your own adventure uh with the added risk that we may not be able to help you if you get stuck, but you can
192. 15:46um and everything you do today, you can take it home and you can check what you did and what you like, what you didn't.
193. 15:53And particularly I was going to say particularly the harness itself talking to to the jokes we were just making about like this is as far as this was built.
194. 16:00Um again the the principles we're sharing here the principles that are built here we were discussing just yesterday is like uh this can feel like we are showing you a table to
195. 16:12teach you how to build a table if that makes sense.
196. 16:15Uh but we really do we really do believe that the again building is not the hard thing.
197. 16:21The hard thing is knowing how we want to approach the problem.
198. 16:24So that's why we're bringing it like this.
199. 16:25But we're gonna open the open the look under the hood a bit um at times just to show you what's going on um behind the the principles we're applying.
200. 16:36And pretty much like Filipe was saying, it can feel akin to saying like, "Hey Claude, please build me a $1 million company."
201. 16:48If you don't know exactly what are the principles and the decisions you're making in order to make something happen, doesn't matter how many times do you try, you're likely going to be just
202. 16:58leaving yourself up for the randomness of something good happening by accident.
203. 17:05So we want to anchor the principles again because they build on top of each other and we think there is a nice sequence that kind of makes logical sense.
204. 17:16So building that shared domain plan mode however you want to call it.
205. 17:20Then setting up consistency making sure that whatever can run or should run reliably does then having the evidence to back up where the decisions are coming from what information is coming through if things are being tested and so on and putting that front and center.
206. 17:37Then the accountability how do you keep a good enough grasp of what the machine is doing?
207. 17:44You don't want to go full blackbox mode because then your accountability is gone.
208. 17:49And that has happened to many of us in this room, I'm sure.
209. 17:52But the other side is also not very realistic where oh for every change you make send me the diff.
210. 18:00I'm going to review it line by line for sure.
211. 18:03Nobody has done that ever.
212. 18:04Uh and then finally verification.
213. 18:06So in data and in finance uh you often need to have a separate point of reference a counterfactual that you can check against h and ignoring that part of the equation just leads to you running things on your own biases.
214. 18:22So your results are always going to be skewed.
215. 18:25There's some shortsidedness that plays into that.
216. 18:28So these principles are just to try to funnel our efforts into where is most useful to get something that we can actually trust.
217. 18:38Um heads up again something will break today surely just call us to you.
218. 18:44We will try to fix it with you.
219. 18:46H some of it is also overengineered.
220. 18:49This is not a session on good UI UX design.
221. 18:53Okay.
222. 18:53So this is just to show you how it could feel, show you the potential.
223. 18:58Uh by no means we're building a product.
224. 19:02Um and yeah, just raise your hand.
225. 19:04Something's not working out, you're a little bit confused, raise your hand.
226. 19:08We're going to float around the room.
227. 19:10Um so you may want to check if this works.
228. 19:15Try and run that command.
229. 19:22Yeah.
230. 19:22And I think at this point we can just go around if things are not working for anyone and give everyone five minutes to get going with this bit.
231. 19:31You should see the UI opening if everything goes successful with the setup works success in the terminal as well.
232. 19:43Cool.
233. 19:43And that second command there will open a very well-designed UI for you to do some stuff.
234. 19:51Here is the example we're going to show you because this is kind of like layered, you know, like an onion or an ogre.
235. 19:59We're going to show you an example hardness being run on an example use case.
236. 20:05If you want to run that command at the bottom, you will see kind of like a more fleshed out example.
237. 20:11But if you stay on the just running the second thing and just use the UI, it may feel weird that I'm asking you to use an UI to see how a hardness behaves,
238. 20:21but that's the point.
239. 20:22The point is to show you how it feels in should feel in the real world.
240. 20:26Um, should I leave it here for a moment?
241. 20:29Yeah, I think so.
242. 20:30Yeah.
243. 20:30Okay.
244. 20:31Okay.
245. 20:31I hear less typing.
246. 20:33I'm going to assume you've typed whichever command you wanted to type.
247. 20:37Erh, so let's go through the actual steps and the principles of the hardness and the share domain is the first one.
248. 20:45Can I see little show of hands of who opened the UI and hasn't done anything else with it?
249. 20:53Cool, cool, cool, cool, cool.
250. 20:54All right, so um long story short, you have a chat window on your right right hand side and it has nothing essentially.
251. 21:03Erh, if you want to follow through an example of a financial worry, some analysis you want to do, that's the place to start.
252. 21:11And basically, we want to walk from the goal we want to achieve and just try our best to give an explanation.
253. 21:18You're probably doing this daily with your agents anyway, but focus on understanding that you're likely not going to have the same context as the agent.
254. 21:29Um, wait, can I go back and forward just in case?
255. 21:33Okay.
256. 21:33Yeah.
257. 21:33Yeah.
258. 21:33Demo.
259. 21:34Um, so I wanted to explain something and I think most people don't realize the fact that what the model knows is not necessarily what you think.
260. 21:48So unless you're very rich and you have a lot of models trained for you, h you're like likely your definition of certain things are not going to be the exact
261. 21:58as what the majority of the population defines.
262. 22:02So if you're looking up for a certain term, the agent may the model may be smart enough to know well this sits in this space but it's not necessarily the thing that you meant.
263. 22:13So going back and forth with your uh model really makes a difference.
264. 22:18And the hardness is supposed to encourage you if you're trying it out yourself to dig deeper to challenge some of your assumptions to ask you questions you may not knew
265. 22:28you needed to to ask yourself.
266. 22:30So if you want to do Yeah.
267. 22:37So let me do the same thing that you folks are doing, but I'm going to do harness example.
268. 22:53Let me go back here for a second.
269. 22:57Just make sure I have everything Is anyone completely lost or are good?
270. 23:20Okay, maybe could you give him a hand
271. 23:23while I I I run everything.
272. 23:26And for the folks that arrived later, I will give you the QR code as well for the so if anyone lost that you can you can pick it up again.
273. 23:43Okay.
274. 23:44Um let me show you two things while I while we we let you go with it.
275. 23:49So this is what you should be seeing when you run with a wedding example, right?
276. 23:55Um we're we're going to explain in a bit um what's going on here.
277. 24:00If you are choosing just to show you as well, if you are choosing to uh pick your own pick your own direction, um you're going to see something like
278. 24:11this where it's empty.
279. 24:14And then what you can do is as Francesca is going to run through the wedding example, you can say something you want to to use a harness for as well.
280. 24:23So things like I don't know plan your own wedding, plan a trip, um plan a new investment in your company or in your personal life, doesn't matter.
281. 24:31Um anything that relates to financial the financial life that we can um implement the same principles.
282. 24:38If you ever need uh again just opening the hood a bit.
283. 24:41If you ever need to switch between one and another like the the wedding example or your own um you can just delete the in the my and var folder
284. 24:52you're going to have a database that's basically what's storing this.
285. 24:56So you can just delete it.
286. 24:59Oops.
287. 25:03And then once you open the the UI again with the previous command, you're going to get the example.
288. 25:10Do you want to open the clean one?
289. 25:12Yes, I can.
290. 25:12Sure.
291. 25:13Yeah.
292. 25:13So, appreciating that you're literally you're literally being given uh a blank canvas.
293. 25:20We want to disambiguate a little bit.
294. 25:21So, if you open the wedding example, that's a work through example that has all the elements.
295. 25:27But let's say this is where we're starting, right?
296. 25:30And I have a problem.
297. 25:32Do you have a dictation thing there?
298. 25:34Um, I do.
299. 25:35Um,
300. 25:36okay.
301. 25:37Let's see if this is working.
302. 25:38Wait a second.
303. 25:40So, basically on that like chat window on the uh right hand side, you are it's a familiar interface, right?
304. 25:49So, how would you go about getting some help from an AI agent for whatever specific problem?
305. 25:54Just start with a chat.
306. 25:55You start with a a problem and you start to describe what you want to achieve.
307. 25:59So, if I were to do it, I'll say I have a wedding and I need to make three payments.
308. 26:08One for $6,000, another one for an unknown amount, uh, and another one for the rest of the money.
309. 26:16And also, uh, some of my costs are fixed.
310. 26:20I know what I'm going to spend on a DJ and I don't know decorations, but I don't know how many people are going to attend the wedding.
311. 26:31And that's the amount of money I'm going to pay per h per dinner per seat.
312. 26:36So, I don't even know how to go about making sure that I can pay in time in time.
313. 26:42On time, on time on time.
314. 26:44Um, I'll say let me just cut the first bit.
315. 26:48Cool.
316. 26:50So if you want to run it for a problem that you actually care a little bit more about uh you will start there and all I'm doing is planning with the agent.
317. 27:02Most of the harnesses today come with some notion that triages that you're trying to plan for a problem and it will try to come up with a reasonable idea some set of questions to give some more clarity.
318. 27:13The harness itself is what guides the sort of questions that you will get asked and the sort of rigor that the plan needs to have before implementing anything.
319. 27:23The idea is that it doesn't go immediately into a yes man mode and just implements whatever crazy idea you came up with that is very unclear but that helps you drill down and dig into what it needs to do.
320. 27:38So that's the principle of getting a shared context.
321. 27:44Quick one before you you continue.
322. 27:46Is the size okay for the back of the room?
323. 27:48The size of the text.
324. 27:49Yes.
325. 27:50Okay.
326. 27:50Cool.
327. 27:50Thank you.
328. 27:54Yeah.
329. 27:54Sorry, I'm reading because this is like live.
330. 27:59Okay.
331. 28:00From the one I have.
332. 28:02Okay.
333. 28:03So maybe I can keep adding clarifying information here.
334. 28:07And I'm going to skip ahead, but something like I have no savings.
335. 28:13It will come from my pocket.
336. 28:16My mom may give me like 5k.
337. 28:19Uh, I know the price per a person going to the wedding is around 150 and I'm paying 500 for the for the flowers.
338. 28:38I need to buy my suit.
339. 28:41I'm just adding a bunch of like complexity and being disorganized on purpose because the point of the agent is that it doesn't let go the fact that you're being vague
340. 28:53um in not providing the information ahead of time.
341. 28:57The whole point of this is that it's really guardrail for you to for to forcing yourself to actually go through um every single data point that it needs to come up with a plan.
342. 29:09I think for the next one we just build whatever comes up to try and move on.
343. 29:14Propose the plan.
344. 29:15Now,
345. 29:16is anyone trying it out with a problem you have personally?
346. 29:23Yes.
347. 29:23Okay, cool.
348. 29:25Do you want to tell us a little bit about it?
349. 29:30Oh, fair, fair, fair.
350. 29:31Nice.
351. 29:37Okay.
352. 29:37It's asking a lot of questions and I'm
353. 29:39No. Yeah.
354. 29:39I I accepted the plan.
355. 29:41Yeah.
356. 29:41So, can you go back to the seated example?
357. 29:46Yes, I can.
358. 29:47Um,
359. 29:48so to save you a lot of the back and forth back and forth here because it's not very high signal for any of you.
360. 29:54There's a couple of takeaways.
361. 29:56One is having that self-awareness and forcing the hardness to remember that your domain knowledge and whatever the agent knows is not going to be the same.
362. 30:09You want to find that kind of like vin diagram superposition as much as you can.
363. 30:15H but it takes a lot of back and forth and that's often where most of the time is spent.
364. 30:20If you design a really good harness, it will guide you by the hand on coming up with a really good plan, implementing things well, uh, and knowing what components are important and which ones aren't.
365. 30:33Uh, if your harness is basically like, yeah, say yes to everything I say and do your best to implement it, then you're going to get variable results.
366. 30:42So, this is the seated example.
367. 30:44The plan accepted thing at the top means essentially that I already went through the back and forth for around 10 minutes or whatever.
368. 30:53Uh and the agent feel it knows enough about my problem to break it down in steps.
369. 30:58So what you have in the diagram over there is the all the steps that it considers as decision points in a decision tree if you may to actually make a decision
370. 31:14and if I if I may um I think it's it's as we said in the beginning it's nice to look under the hood a bit just to give you an idea and you can of course explore this by yourself or with your preferred agents
371. 31:26help.
372. 31:27Um but when we are when we are talking about everything that Franchesco is describing um this also comes from the instructions that we have inside the harness right so
373. 31:40this is where we have the instructions of the agent performing the interview and what it must end up with so this is the materialization of the principle built beyond the harness right like built inside the harness I mean
374. 31:55if you want to go back to um to the UI.
375. 31:58So this is hopefully going to start making more sense as we as we progress in the example.
376. 32:04But we added a couple of quality of life things where if you want to know what elements of this whole UI that is so beautifully designed h makes sense for whatever
377. 32:17uh principle we're discussing.
378. 32:19You just need to hover down at the bottom and it will be broken down.
379. 32:23So you see when we did the planning the planning the sharing the domain context it came up with a plan is highlighted at the top on the right and then it came up with all the nodes and the connections in between the other decisions
380. 32:39and what we need to start scoping from here.
381. 32:42So should we move on?
382. 32:46Yes.
383. 32:48Is anyone still doing any planning with the chat?
384. 32:56Cool.
385. 33:05Okay.
386. 33:05Yeah.
387. 33:05So, we had a few things here.
388. 33:07If you wanted to uh play around and peruse the uh harness yourself, you could for instance go on the step three in the UI like in the little diagram on the
389. 33:21left hand side.
390. 33:22Oops, sorry.
391. 33:23I was going to show
392. 33:24too many tabs.
393. 33:25You could click on like step no
394. 33:27step three in the Yeah.
395. 33:30And it will give you a little bit of a breakdown of where the information came from where why the model thinks this is the node you need to make a decision of
396. 33:41uh including if it's making any assumptions based on the language you were using during your planning with it.
397. 33:49So for instance the agent did you say this I could point with this
398. 33:54uh if you press and hold that yeah
399. 33:57awesome
400. 33:58not AI
401. 34:00so there's there's quite a few things in the plan h that the agent put together that were proposed by it so you can start dissecting a little bit and understanding like
402. 34:10actually this came from me this is a fact that I think I know or this is something the agent inferred based on my language.
403. 34:17If you scroll up a little bit just on the same Yeah.
404. 34:20more more more more there.
405. 34:24Yeah, sure.
406. 34:25Uh here there was some back and forth because I was being vague with the language I was using and it made an assumption that because I was using this keyword here,
407. 34:37I meant this.
408. 34:39Again, this is a pretty obvious example, but you can play around with clicking through, seeing things around.
409. 34:45If we go back to the presentation, there's some annotation because again, this is all about grounding the principles into something that feels like it was designed for building trust.
410. 34:58So you have some notation.
411. 34:59Again, not the best UI design in the world, but knowing if things came from you, from the agent, or is it still an open point and you need to make a decision on
412. 35:09and you can tell how the plan has been built.
413. 35:13You have another thing you can try.
414. 35:15You can go back into step two uh and actually tell the agent if we click on the node there.
415. 35:23Yeah.
416. 35:24And you can actually tell the agent after you have sorry uh you can just tell the agent like well actually you're making some assumptions about this step that are not correct
417. 35:36and then go a little bit back and forth.
418. 35:39Erh, I will still do the show of hands thing, but because of reinforcement and engagement, uh, who has been in a chat session with over like well over 10 turns, so back and forth and back and forth at least 10 times
419. 35:57where at some point you said, let let me start from the beginning like this is not making any sense.
420. 36:02Some stuff we already discussed like it's well gone.
421. 36:06Who has had that feeling on a regular basis?
422. 36:10Yeah.
423. 36:11So, that is a consequence of the design of the harnesses we have today where they're designed for a linear discussion and to keep everything in the context as one
424. 36:23big conversation.
425. 36:25Whereas here the way we designed it is supposed to break it down so you can discuss individual elements without necessarily making things you know more complicated in the grander scheme of things.
426. 36:38Yeah.
427. 36:41Erh cool.
428. 36:43So I will let you play around with it a little bit but I do want to maybe go a little bit further.
429. 36:49So I will introduce the consistency principle and this build on the shared domain erh because you already have a plan and the agent is very capable to understand well these pieces of your plan
430. 37:07are just a piece of code or just a calculation spreadsheet like it will always give you a consistent result is a deterministic element of your whole plan.
431. 37:17Erh and based on that you need to implement code.
432. 37:22You need to implement something that will give you the same result every time.
433. 37:25Erh similar show of hands thing.
434. 37:28Has anyone tried to do like budgeting or any financial calculation in the chat and how did that go?
435. 37:38Like are you confident on the results?
436. 37:41You Yeah, a little bit.
437. 37:42Temperature maybe like very well um not too well.
438. 37:47pretty bad.
439. 37:49Okay.
440. 37:51So, so all right.
441. 37:53So, I would love to know if anyone wants to volunteer their story, but what they saw and why they think it happened.
442. 38:01But to me, a couple of times I thought I was giving all the necessary information.
443. 38:08The the agent gave me back a formula.
444. 38:10It was even formatted as a formula.
445. 38:12You know, the kind of like italics formula thing that looks really nice.
446. 38:16I was like knows what he's talking about.
447. 38:19And then the result was like I'm not a calculator and I know that result is wrong.
448. 38:24So from the get-go I was like nah start from scratch.
449. 38:27H anyone has any other like examples
450. 38:29consistency right?
451. 38:30So a lot of times the results look good reasonable but you run the same thing again that's something.
452. 38:45Yeah.
453. 38:45So that's basically the inspiration on the principle.
454. 38:48Sorry, what do you want to share?
455. 38:49Yeah.
456. 38:49So it goes to the point that every time it might
457. 39:07Yeah.
458. 39:08Uh and and that's a great a great um example to say part of the benefit why we are talking about building a harness and not just plugging these principles in a set of skills or instructions
459. 39:22to an existing harness such as like codeex cloud code or anything else like when we when we do it the non-deterministic way which is to give the agent a bunch of instructions that
460. 39:34um say the right things but still rely on the agent agent.
461. 39:38Uh, make no mistakes, please do the right thing.
462. 39:41Um, that's that's the point.
463. 39:42We're relying on a bunch of things happening in the background and basically getting a nondeterministic result every time.
464. 39:49So, I think the value here from building the harness again, this is looking a bit into the under the hood and like from principles we we applied at at Monzo as well is mixing the non-determinism
465. 40:02with some deterministic behavior that we want to control.
466. 40:06um when when relevant.
467. 40:10I think you mentioned that for this demo a lot of this harness was quite I get this point right for production stuff like what's the difference between you making calculations
468. 40:24in in chat versus you asking the model to code up calculations for you in code.
469. 40:30So I have a couple of personal examples on doing forecast where you can try and do forecast yourself by hand or come up with a spreadsheet with a lot of fancy
470. 40:42um calculations and you will likely get a few things right and get a decent forecast.
471. 40:47You can also just get a library from a trusted uh repo and that has everything baked in.
472. 40:54You can also write some code in a more modular fashion that is well unit tested, you're doing end to end testing, you're doing all the like good software um engineering practices
473. 41:05that we know how to do to ensure that that piece of code is going to give you a good result.
474. 41:11Um, so I think the different Oh, sorry.
475. 41:15The difference lies a little bit on the fact that if you're just leaving it to the agent to decide how to implement things and when they need to do it in code versus when it like you should just build something on the side,
476. 41:31you're going to have a little bit of a big miss ratio like hit or miss.
477. 41:36Yeah.
478. 41:36Yeah.
479. 41:36I I guess what I'm asking is if this is why quoted no good engineering practices were applied.
480. 41:42I get this part.
481. 41:42Do you have like a product that that was tested?
482. 41:46We we I'm going to do the the uh ad again.
483. 41:50Uh tomorrow we're going to have a breakout session in the morning where we're going to talk about uh what we build at Monzo that applies some of these principles.
484. 41:59uh but yeah I think the what is behind when we say like the harness being voded I think it's more about the implementation detail of like abstractions methods the actual code
485. 42:10uh but the principles of mixing the determinism and the non-determinism are the same regardless if that makes sense
486. 42:18thank you
487. 42:21uh and another thing another thing that signals that this is a a good principle to consider on over the last what year or so most of the agents have started to do more aggressive tool calling and building their own tools to solve calculations in a more deterministic way
488. 42:42and honestly agents are just getting better right so we could just wait and then all the problems are solved the one challenge I have with that mindset is that h it will create stuff
489. 42:54on the on the go like in flight so Yes, you may get the right code for the right problem at an at a point in time, but maybe the
490. 43:03time number 100, you run it, turns out is inventing a new piece of code that didn't need to be reinvented.
491. 43:10uh and other examples non-workrelated that I can share err are around like doing exactly that like I have a I don't want to say strict but semi-strict close of budget every month and I categorize everything in my statements
492. 43:27I left it to the agent to decide how to do the categorization and how to do the aggregates and everything else and the refunds and expenses and at some point I was like no no no this needs to be a script because I'm gettinglight slightly different results every time and I'm spending more money more money and time
493. 43:44uh wrangling with the agent to get it to give me the results that I wanted that it got right three times already but didn't get it right in the fourth
494. 43:53uh so I don't want to repeat it again but make sure that we run the calculations you write the calculation once and then you run it every time you need it
495. 44:05um and the model is able to pick the tools is you just need to make sure that your hardness makes it very strict that it's not going to do any arithmetic
496. 44:16or try to solve these problems by just parsing tokens.
497. 44:21Um so what can happen if you don't make sure that your calculations and everything that is deterministic runs in code they're in the thickest in the model's head.
498. 44:33So it's gone once you start new session the new code for every new question and or new session and it becomes a mess and same question different answer non-determination
499. 44:44again you're asking the stoastic machine to produce a consistent output so you're likely not going to get it.
500. 44:50Um, so here's some examples of things that have a way to be calculated, right?
501. 44:59Like I'm sure if you ask 10 different financial services companies, they will have a slightly different way of doing these things, but the general formula, the general idea is the same.
502. 45:11So you could realistically export a library, write it once, update it, and these things are not new.
503. 45:18they saw from like software good practices.
504. 45:22So demo again if we just want to go to the seated example h in the can you highlight this?
505. 45:34Okay.
506. 45:37So again just using this h you can see how the nodes in your decision making process um some of them needed to be implemented as code and I think the one of the easiest way to check is if it's a calculation likely it has to be implemented as code right
507. 45:55uh as well as some testing unit testing golden examples etc. the you should build trust with your agent when you're building a harness like this by making sure that the information you care about about its implementation
508. 46:10it's in front of you.
509. 46:12Another example of how this has looked in the wild is giving you versions of the code.
510. 46:18Your your code is more likely versioned.
511. 46:21If anything change with the version of the code and you're trying to run the same process, the agent should be able to pick it up and surface that information to you and say actually this code change you probably
512. 46:32even though the function is the same you probably shouldn't trust uh what's coming out of this because the whatever calculation h is different now.
513. 46:41So as you're working with the with the harness and you accepted the plan, you have a nice graph like this being built, you will get something at the top saying, do you want me to build the code?
514. 46:56H and it will do it for you.
515. 46:59And again, it's all guided.
516. 47:00That's the point of a hardness that it guides things.
517. 47:04Erh, is anyone building any of the steps on their plan?
518. 47:13Is that okay?
519. 47:14Yeah.
520. 47:14Yeah.
521. 47:15Cool.
522. 47:17I switch to focus.
523. 47:19Okay.
524. 47:20Yeah.
525. 47:20[laughter]
526. 47:21Fair enough.
527. 47:22Fair.
528. 47:22Fair.
529. 47:28Cool.
530. 47:28So if you're following on the wedding example, you can try a few things.
531. 47:34Click on step one, then code, and then the test and run the tests now.
532. 47:47Tests here.
533. 47:48Yeah.
534. 47:5295.
535. 47:55Sorry, what time is it?
536. 47:569:54.
537. 47:58[laughter]
538. 47:58Okay, good.
539. 47:59Um, but yeah, the the the principle here and just going back to try and tie everything up together, you shouldn't rely on you remembering to run the test.
540. 48:10You shouldn't rely on you remembering that, hey, this this sounds like I can h create some code to solve it every time.
541. 48:19The agent should guide you through it.
542. 48:21And that's one of the principles we baked into building this.
543. 48:25Uh, as an aside, I think we were telling you, uh, there's a little extra file in the repo that says build yours or something.
544. 48:35Um, it essentially has all the principles baked in in order and it will guide you on building a
545. 48:41[snorts]
546. 48:42similar hardness.
547. 48:45It's built your own.
548. 48:48There you go.
549. 48:52Yeah.
550. 48:53So yeah, it will guide you on building something that feels similar to the harness we're showing you that prioritizes trust building mechanisms over just giving you something that looks correct.
551. 49:06So you can try this.
552. 49:07The
553. 49:07[snorts]
554. 49:07test should be passing.
555. 49:09You will notice that there was no thinking, perusing, all the verbs that agents put when they're loading something.
556. 49:19Yeah.
557. 49:20because these are just tests that need to run in your environment.
558. 49:24Don't overengineer it.
559. 49:28Funny that I'm saying that, but it's easier if all the components of your harness just work together as they should and not based on like hopes and dreams and magic glue from an agent orchestrating
560. 49:41things together.
561. 49:42I think we had a question.
562. 49:58Erh, so if we Yeah, I mean we can do that, but I'd rather not do a reading session.
563. 50:07Basically, it it all goes about h figuring out what you actually care about and what are the functional requirements for a functioning harness.
564. 50:16H I'm sprinkling some of that through the conversation.
565. 50:19Again, need to avoid oversharing a little bit because of IP.
566. 50:23Uh but you want to make sure that the objectives of each of the steps when you're building the parts that will make up the behavior of your hardness have an intent.
567. 50:35The intent of one of the first components is gathering as much information as possible to come up with a plan.
568. 50:41And that is written in here as well.
569. 50:44So this is basically instructions for instructions.
570. 50:46It's the IKEA guide.
571. 50:49Um it is supposed to guide you very nicely on these are the things you should care about but but then you can pick uh your preference on how they get implemented.
572. 51:03Question.
573. 51:04So how is this different from let's say the specdriven development expected something like that which is also linear from starting from all the questions from you starting from idea all the way to the end
574. 51:18and how is it different to this one is different to that
575. 51:22do you mind if I come to that um I I want to mention two things that I think are are relevant here um I think the first one is
576. 51:30when we're building a harness for a specific type of use case it's a bit different from the coding assistant world of agents because with the coding assistance we usually
577. 51:39the code is a means to an end right like we we're building something that performs an action.
578. 51:45It's a product, is a an app, whatever that is.
579. 51:48Um, in this case, the agent is helping us like analyze, solve a problem and work us through that.
580. 51:54So the the gigantic spec for this like let's say for Franchesco's wedding things will change like it will evolve it will change and then if we go back to the everchanging
581. 52:08spec we are relying again on the determin on the non-determinism of the agent to interpret that that can be very very very um robust and perform this the the correct thing.
582. 52:21So the harness essentially breaks down what would be the spec in opinionated sections if that makes sense.
583. 52:30Um and I think the second thing to complement with that and is actually something I wanted to to like ground on our experience from data.
584. 52:37I don't know how many of you work in data engineering analytics or anything like that.
585. 52:43I think something else that we bring from this word in these principles is the idea that um metric definitions and dimension definitions always change like in our experience we've like
586. 52:55[snorts]
587. 52:55had many many many times and I'm sure many of you as well um that conversation were is this number correct?
588. 53:02I don't know.
589. 53:02It sounds like it is, but we're never sure and then if we're never sure, is the number correct?
590. 53:08Um, I think this is also coming from that perspective is that to the point where we trust the agent as well as we trust a dashboard or a metric or a visualization.
591. 53:18Um, we've reached alignment with it and the harness can continue guiding us into that alignment continuing to be true over time.
592. 53:27But premises will change, source data will change, context will change.
593. 53:31And then having the harness able to adapt to that is like recognizing that this change will be a thing um rather than having to every time pro prompt again and go back to the process.
594. 53:46Does that make sense?
595. 53:49Yep.
596. 53:54Follow question on that.
597. 53:55any complnt this perspective because most most workable thing you are developing something even though it's subject to change through different iterations you are developing something with an AI agent can be part of this flow as well
598. 54:14absolutely use that uh this part this harness to do that it
599. 54:20absolutely and this like under the hood this is essentially what's happening as well.
600. 54:27So when we have when we're creating um the if you if you look under the hood in the my folder, you're going to have everything that is saved.
601. 54:41So you can think of this as the spec that's behind the harness.
602. 54:46So absolutely like the same principle applies here in the in the specdriven development for again coding agents is usually used.
603. 54:52you have something that is the source of um definition for the thing being built.
604. 54:58In this case, it's the same thing.
605. 55:00The difference is that we surface that through opinionated like uh workflows rather than a single um looking document if that makes sense.
606. 55:11All right, cool.
607. 55:12So, that's another thing you can try out.
608. 55:18um you can try and change how a calculation works.
609. 55:23Uh and you're likely going to get some push back from the harness.
610. 55:27Just to reiterate, the point of the harness is to guide you and force you to do the right thing.
611. 55:32And the principles are here to help you make the right decision every time.
612. 55:37So again if you go try and change uh multiplication for a an aggregation you may just go make the quick edit save the file success you sabot attach the analysis
613. 55:49but when you run the test you won't be able to actually reach any conclusion of the analysis you wanted to do h because the hardness is going to stop you by its own like definitions and the way it was built.
614. 56:02Um, should we move on evidence?
615. 56:08Er, evidence is the whole thing I was saying before about the receipts, having the receipts and being able to show that well, yeah, this is where this came from.
616. 56:18This is correct because of in doing that causality relationship.
617. 56:24So it builds on the second principle as since you have code that is tested as well as everything else h you have a trail that you can follow backwards.
618. 56:35So you have a way to substantiate the decisions you're making and you have the evidence and it's important in this principle because a principle is something you need to interpret right and I don't want to leave you with a very vague
619. 56:50ambitional thing like oh yeah do evidence it's awesome but what is what about evidence is something that you need to take care of going back to being data professionals this is something that bites us on a weekly basis.
620. 57:07Erh, not all information is the same.
621. 57:10If you really want to present a piece of analysis or you went to do some financial analysis on a potential investment for your firm, you need to show where your numbers came from
622. 57:23and be able to attribute your decision to the body of evidence you had.
623. 57:28Erh so since we can show where everything came from uh there's some mistakes we want to avoid where everything is treated as a fact.
624. 57:41So oh yeah this number you gave me for the number of um I don't know going back to my wedding the number of people invited is a fact.
625. 57:49Turns out my fiance invited three more people didn't tell me.
626. 57:53Now my whole model is wrong.
627. 57:56uh and I'm trying to reduce it to a very simple example but I think you can imagine how having a wrong assumption somewhere and not knowing where it came from can be fatal for any financial analysis.
628. 58:08Um so let's say in the world of a gentic building it's very easy to assume something or treat something as a gueststimate.
629. 58:21erh the model can't tell what is a hard truth and what is an estimation.
630. 58:27So you need to enforce that through the design of your hardness.
631. 58:31Erh, and another example where this kind of like falls fall into pieces really quickly is imagine you have a data person, an analyst, something coming to you and being like, "No, no, no. Here's the result.
632. 58:44Trust me, I run the numbers."
633. 58:46Do you trust that as a Yeah, I see some like no. I don't think anyone here will want to see all the logs and the math behind a single number.
634. 58:58But just having the trace to figure out, okay, I can follow back this decision increased increases confidence for us humans quite a bit.
635. 59:08Erh, so we have some like different examples here.
636. 59:13Erh, unstructured thoughts in a chat needs to be labeled as an interpretation.
637. 59:18the agent reasoning or pulling information from memory needs to be treated as an assumption and then if I submit a bank statement what is there to discuss like that is a fact that's how much I spent
638. 59:31etc if I take like go back again to being a data person data labeling and making sure that you have a curated data set for either training something or running any process is paramount and this is basically the same thing you just need to assign the right level of
639. 59:52importance to the evidence you're gathering and make sure that you gather as much as as much of it as you can.
640. 59:59Um, this is a really funny one, but evidence also goes into the code.
641. 60:06So, it's not just where do you get this number from, but how did you get this number?
642. 60:13And I had I have an anecdote where someone was using the master file.
643. 60:18In every company there's a master file and is favorited in the share drive and whatever.
644. 60:24And that person puts some numbers in there and everything looked fine.
645. 60:27The master file had some assumptions of what range of values were reasonable.
646. 60:34That person didn't know because the file changed.
647. 60:36They didn't do anything wrong.
648. 60:37They just didn't know the file changed.
649. 60:39Uh and then the results were entirely wrong.
650. 60:42And the same will happen with deterministic code you build with your harness.
651. 60:47If you don't have a way to having a traceability even for changes you make in your code, you're leaving yourself uh vulnerable for blind spots on the process change.
652. 60:59The inputs were the same.
653. 61:01Erh skipping ahead a bit.
654. 61:08So if we're following from the example, can you open any of them?
655. 61:17Yes.
656. 61:19The evidence is highlighted.
657. 61:21Erh, and again, you can see like the areas that you care about through the example by just like clicking on the bottom and then seeing the bits that get highlighted.
658. 61:34But this is how the flow of information goes through and what parts are proposed by an assistant like you can show the receipts of where where this came from.
659. 61:43Looking at other implementations of financial analysts in the industry you get traceability down to the data set like you know which uh connector or which supplier of data gave you the information that whatever Apple
660. 62:00had this price at this point of the day.
661. 62:02if you're doing something about trading or investing in stocks and shares.
662. 62:07So, this is extremely important.
663. 62:09I don't think we need to spend any more time here unless anyone has any questions.
664. 62:16Oh, good.
665. 62:18Cool.
666. 62:23So, here's a few things you can try.
667. 62:26Um, so you can add a little bit of a a curveball to the agent.
668. 62:33Just typing in the chat and you will see how some of those evidence lines or some of the inputs that go into the rest of your process change and it it's able to trace that it came from you correcting some fact
669. 62:49in the chat.
670. 62:54And another example is h if you want to go and make a change and click on the underlying figure that changed h it should show you all the way back
671. 63:06where this came from and why it matters for the rest of your calculation.
672. 63:12I'm going to say something that might feel repetitive.
673. 63:16Um but again we are we are this this harness is an example of how to apply the principles but I I'm sure everyone is thinking about your use cases your companies the agency you're building the agency you're using and how like to to to connect the dots and to to I don't know put the wires together.
674. 63:33I think this is an an example we can think about when our users our and customers of the agents that we're building are using it and again the trust that they have in your product in your agent and in the output is what will drive
675. 63:50usage adoption improvement and so on through the loops that we mentioned through tracing evolves and so on.
676. 63:57So again, I think um what's underlying here is when you're building a a harness, make sure that you can um make this happen to uh your customers that are using your users that are using so that they trust the results that your agent is giving them.
677. 64:15Again, maybe repetitive, but like making sure we uh draw get everything together as we go.
678. 64:25Before we move on, anyone has any example of a time where having the evidence trail would have saved you a lot of pain?
679. 64:39I see nodding like painful acknowledgement, but if anything you want to share, more than happy to to hear you out.
680. 64:47Otherwise, we can carry on.
681. 64:51basically every single time you try to do something.
682. 64:54And so I'm playing with this.
683. 64:57I really like the idea of being able to show the receipts and everything.
684. 65:02That's basically every single conversation I have with an agent, whether I'm coding or like working on a deck or something.
685. 65:08I'm like, "Where did this number come from?"
686. 65:11Like, "Well, we talked about this blah blah blah blah blah."
687. 65:14I'm like, "It would be nice if I could just be able to show me specifically like the paper I like the that you use a presentation as an example
688. 65:27because I had an example myself where I was trying as hard as I could to not have to resize text boxes in Google Slides for a good like month and a half.
689. 65:40So I was forcing my way into having something that could help me build slides.
690. 65:46and I had a really nice harness that would like dig into the problem, get all the relevant documents and help me highlight what are the key points that I wanted to present
691. 65:56to the team.
692. 65:57And turns out in my conversations, I never caught the fact that when I ask for a modification and give an example, the model will fixate on the example so hard that it will put it everywhere in my presentation.
693. 66:14I didn't pick it up.
694. 66:16So, you can imagine I was showing some colorful example to senior leadership and they were like, "Where did you get that number and that assumption?"
695. 66:25And I was like, "That's crazy.
696. 66:28I don't know where that came from.
697. 66:29That's crazy."
698. 66:31But yeah, it tends to happen.
699. 66:32It's really annoying.
700. 66:33And that trace of consequence can really hurt the trust that you have in your harness and the results it produces.
701. 66:41Let me carry on.
702. 66:43So another principle is on accountability.
703. 66:46Uh and this builds on the evidence because you only make a decision when you have some facts when you have some point of reference.
704. 66:57Erh well I do want to say that but we all have that friend who speaks very confidently about something they don't know.
705. 67:04So little bit of a counter example, but generally I think if there's one takeaway I would love for everyone here to feel aligned with is we are ultimately the responsible people, the accountable people for whatever
706. 67:22AI agents do, right?
707. 67:24Um and there's a funny thing where people discussing that AI is going to um you know we're not going to have any jobs because of AI.
708. 67:34I always go to back to the fact that I get higher, I get paid partially because I am accountable for the results that I produce for the company, not for me sitting in a computer and typing some stuff.
709. 67:47So there is an element of the human experience where the whole trust and accountability and working together as a team matters a lot.
710. 67:56And how do you keep accountability high?
711. 67:58You do need to be able to see under the hood, inside the black box, etc. Erh, I don't know who said this.
712. 68:08I loved it.
713. 68:09I never able to quote it.
714. 68:11But agents face no real world consequences.
715. 68:15You do.
716. 68:16So I I love the show of hands thing, but it probably is the only resource I have.
717. 68:21Maybe that's why.
718. 68:23Has anyone seen either written or been told?
719. 68:27Well, I don't know.
720. 68:28That's what Claude gave me.
721. 68:30[laughter]
722. 68:32Yeah,
723. 68:33that feels great, right?
724. 68:35Like, oh, that makes sense.
725. 68:37Yeah.
726. 68:37Yeah.
727. 68:37Everything is forgiven.
728. 68:38Don't worry.
729. 68:40We face the consequences of what agents do.
730. 68:43So, having accountability mechanisms in our harnesses reduces a lot the amount of stress you need to go through.
731. 68:53Just going in your head like, "But do you think it made the wrong assumption here?"
732. 68:57Whatever.
733. 68:58Erh, so you want to reduce ambiguity and not remove it.
734. 69:06Erh, I'm pretty sure a lot of people here will remember fondly the days prior to auto mode.
735. 69:13H, who wants to go back to before when auto mode existed and you had to approve every single change by tapping enter?
736. 69:20Yeah.
737. 69:22Yeah.
738. 69:22Lovely times.
739. 69:24I still do it.
740. 69:27Fair enough.
741. 69:30[laughter]
742. 69:31Fair enough.
743. 69:36And from all the let's say 100 taps you did, how many of them you failed where?
744. 69:42I mean, come on.
745. 69:43This is a oneline change versus this is important.
746. 69:47Thank god I was looking at this.
747. 69:48Yeah, I just pref
748. 70:08and and that's basic that's basically part of the hardness like the design is intentional.
749. 70:14So you review tasks instead of individual changes.
750. 70:18And we're just providing you with a a reminder that when you're designing a harness, you want to make sure that you include on purpose those gates to make sure that you can still be accountable for what is being produced.
751. 70:31Um so make sure that any judgment call, any disambiguation that is important, not the trivial ones, right?
752. 70:39H you can get some hold of it still.
753. 70:44H what happens without it?
754. 70:45I don't know.
755. 70:46The stoastic machine said it was fine.
756. 70:48Amazing answer.
757. 70:50Approving every calculation.
758. 70:52The other side of the spectrum doesn't feel great.
759. 70:55H or reviewing every PR line by line that your agents ship.
760. 70:59Yeah, that it's not really well written, right?
761. 71:03But yeah, it leads to a lot of either a lot of toil and unnecessary work if it's not designed um in a way that you know makes sense for the use cases.
762. 71:15Erh.
763. 71:18So what is the main takeaway on how to design a good mechanisms for accountability?
764. 71:24Few gates with very high signal.
765. 71:27Your example of the tasks and breaking it down in a way that you can reason about the the way they connect.
766. 71:35It's a really good example on where people do h reviews only on functional tests being passed.
767. 71:44anything prior to that.
768. 71:45I think you can be a little bit handwavy with yeah sure this looks about right but you wouldn't push anything into production that hasn't passed uh all your tests right so you put some gates in place but those gates need to be very high signal
769. 72:01um
770. 72:01and uh to to complement I think we're using some examples in the coding assistant world and and again this is not how we for instance approach things at at Monzo because of like a different context text, but when we this is bringing to to our hardness.
771. 72:17I think we use the AI coding examples because they're close to us and they're the tools many of us are using on a on a daily basis.
772. 72:24Uh but when we apply here is the the same thing that claude code does when they have a feature that allows you to gauge how much you want to be involved in the process.
773. 72:35When we are building our financial harnesses, we can also guide our users in the in the same way.
774. 72:43That is pretty spot on.
775. 72:46Um, and again, just going back to the domain with money, there's some stuff you can leave the agent to figure out.
776. 72:55For instance, if you have a few possible values for something, you're like, well, do three scenarios.
777. 73:02It's probably going to be fine.
778. 73:04But you want to review them side to side before making any decision.
779. 73:07You wouldn't hand over entirely the keys to the kingdom to something that you don't necessarily trust is going to have your best interest in mind.
780. 73:16Not that it will have an interest at mine, but um so this is a it's very important that you know that bad decisions carry forward into your other decisions.
781. 73:32So the gates you design need to be fit for purpose.
782. 73:36You need to know the use case really well in order to be able to put the gates where it matters.
783. 73:42Otherwise, you're just adding before any decision, ask me and then it becomes very low signal again.
784. 73:49Um and well for code you don't necessarily need to ask because the tests will speak for themselves.
785. 73:55That's where the receipts come in handy because you only need to check that everything is passing and you will have a lot of confidence because of it.
786. 74:04So if you go back to the demo, funnily enough, this plan is not very high on accountability.
787. 74:13The only step is I will check everything that we calculated all the scenarios whatever and I will make a decision on how much I need to save every month.
788. 74:24I mean still nice because no money is moving out of my bank account anywhere but yeah there's a accountability mechanism I feel confident that that will gate any anything else
789. 74:36I think as a as a as an extra example like in this this challenge and decision here likely as we let's say we accept this and this is now marked as a decision we made
790. 74:48if we ever use this information like the 20k landing on the same day as payment three as something that influences the final decision.
791. 74:57This definitely comes with the with the accountability.
792. 75:00Yes.
793. 75:10Go up.
794. 75:11No, I actually like I'm improvising a little bit.
795. 75:13But if you scroll up a little bit,
796. 75:16h this is like again not not priding myself of any good design at all, anything the contrary.
797. 75:23But h you see we have a little bit of a weird structure there.
798. 75:28Looks like a tree a little bit with something branching out joining back in.
799. 75:32That's another accountability mechanism as Philip is saying where there's some decisions the model will either assume and just try to verify with you like hey is this right?
800. 75:44Is this the number you want to use here?
801. 75:46Or just try to figure out things how they fit into the overall um system.
802. 75:53We thought a good way to showcase that you need to design a harness that gives you the accountability is making sure that you can trace all the decisions that have been made,
803. 76:03who made them uh and have that sense of well this was something that the agent assumed and I let carry on in the rest of my flow or it's something that
804. 76:16I provided a revised input to.
805. 76:20And the reason again I'm a very visual person that could be symptom of something I don't know uh but the reason I think is kind of like a not a straight line conversation but rather branching out and then branching back in is that when you're chatting with an agent
806. 76:35again for financial purposes you may reason about something after a decision has been made and you are well in your right to correct information and add more context.
807. 76:48So, I think it's important that when you're creating an a harness that you not only build those gates for validating things, making sure that you're still accountable, but also for correcting the course
808. 77:01and making sure that you can do so safely and it's not like, oh, because I typed this sentence in the chat, now all my analysis is gone.
809. 77:10H that has happened to me many many times trying to solve a problem and maybe I was like dispersed dictating to my phone I said something in the middle of it and then the whole analysis is gone because I can't really
810. 77:25take away that one sentence and please keep everything else as it was h so this gives us another means to be accountable for what goes into the system for financial analysis and what
811. 77:42Do you want to move to the slides?
812. 77:46Yes.
813. 77:48Okay, cool.
814. 77:48So, uh, a few things we can try.
815. 77:51Erh, you can go and and hover into the chip that guest that says how many guests we have in the wedding example.
816. 78:00And you can add that that will add it to the chat and you can say, "Oh, actually, it's not this number.
817. 78:06H we will get no fewer than 150 guests for instance.
818. 78:10Erh that will get your harness to rethink the plan see the consequences of that input into other stuff and suggest a either a modification of your plan or just change the input whatever.
819. 78:25The important part here is you build the feedback mechanism on purpose and you have some judgment over making sure that there is some consequential decision making on the fact that this input may change your whole analysis.
820. 78:41H you need to be on top of making sure that those changes make sense.
821. 78:49The other thing you can try is on if you click on step five erh you can open up a new branch conversation like a branched conversation I think is the button on the chat at the bottom
822. 79:03here.
823. 79:04Yeah.
824. 79:04Yeah.
825. 79:04So you can open a new branch.
826. 79:08Has anyone used the slash
827. 79:12btw by the way thing on cloud?
828. 79:17How has been your experience using it?
829. 79:20Sometimes it works, sometimes it doesn't.
830. 79:24doesn't.
831. 79:24And it feels like you're doing an aside, but not really an aside aside.
832. 79:28It becomes a part of the context, right?
833. 79:33But do you get a choice?
834. 79:35Like maybe I have a stupid question in the middle of the conversation.
835. 79:38No, I don't have a choice.
836. 79:39[laughter]
837. 79:39If I ask a question, then it becomes the part of the session.
838. 79:42Yeah.
839. 79:43Yeah.
840. 79:44So in here, what we're trying to do, and going back to why we're doing it like this, we want to show you how it could feel if we design something
841. 79:54with the intent of, well, this really needs to be an aside.
842. 79:58I have too many stupid questions when I'm chatting with AI.
843. 80:01So many.
844. 80:02And it's shameful, but I end up having to open a separate chat because I don't trust it's not going to contaminate what I'm doing.
845. 80:08And then I will take more time just being like, "Hey, this is a stupid question.
846. 80:12please don't use for anything else is the please don't make any mistakes thing Philippe was saying
847. 80:17h doesn't work quite well.
848. 80:19So whatever you design if you have a feedback mechanism make sure that you tailor it to how you actually want to use it.
849. 80:27Erh oops cool and then as for h the last point the last principle will be verification.
850. 80:41So you want a second pair of eyes and this builds on the principle of accountability because you are accountable for that review as well.
851. 80:54Erh, and maybe that's too abstract, but has anyone been in a situation where someone h passed the blame on a bad decision to you or anyone else just on the basis of
852. 81:10No, but Franchesco, review it.
853. 81:14That happens.
854. 81:15That happens.
855. 81:16So, um, we were having a funny conversation like a month ago where a few a few people are starting to have thoughts that the burden of something being good or correct with VIP coding
856. 81:31ends up being in the hands of the reviewer in a team that feels really demoralizing because you're just being given a bunch of stuff that sure looks about right, but then it's your responsibility
857. 81:44responsibility ultimately to make sure that it is right.
858. 81:48This whole principle of verification is just building more feedback loops so you don't have blind spots but without relegating any of the accountability.
859. 81:58Ultimately, this is just extra information from a different source to avoid biases and avoid you getting stuck on your own thinking and your own processes.
860. 82:09But it doesn't remove any of the responsibility or accountability.
861. 82:15Um yeah, funny enough, expert experts often do not check every cell.
862. 82:22And this is more on like my experience working with financial finance teams in data.
863. 82:30A they could open a spreadsheet and just look at the you know a few cells they know will give you a ballpark total and if it doesn't feel right they will let you know.
864. 82:41It's like, how did we spend this much money on marketing?
865. 82:44I don't think that's right.
866. 82:46Uh, and it's a little bit of a gut reaction.
867. 82:49You don't need a lot of effort.
868. 82:51You don't need to like redo your plan entirely or challenge every decision you're made, but you do want to have a separate point of reference.
869. 82:59Erh, I think I I had an example, but I don't remember what was it was something about a tweet that got really famous about Elon Musk having I don't know how many trillion dollars
870. 83:15and someone saying like, well, he could give a million dollars to every person in the world and still have money.
871. 83:22And he was like, wait, what?
872. 83:25That doesn't feel right.
873. 83:26Like you read it and you're like, wait, what?
874. 83:28Looks plausible.
875. 83:29And then you're like, "No, no, that's not right."
876. 83:32And that needed need that needed for you to look into the calculations the person made is like a little bit gut feeling.
877. 83:40Um, so just having something review and give you a rough answer or some challenge if something doesn't look right is extremely helpful.
878. 83:49Sorry.
879. 83:57supply.
880. 84:06That is such a good question.
881. 84:07Uh I was trying to come up with a examples that feel more down to earth, but I think that's a very conversation to have.
882. 84:16A good argument is if you're getting golden examples of how the data should look like, why would your process give you something different to that?
883. 84:26Like why don't you just check with your golden examples?
884. 84:29And this is something I I wanted to abstract away from the world of finance.
885. 84:35You have a lot of sources of information, a lot of them.
886. 84:38You always have another point of reference outside of your data.
887. 84:42So if some sources you're using are claiming I don't know whatever fact a number you have the internet you should always try and enhance the signal you're getting by looking out
888. 84:58look at all the resources that are not the ones that I'm using and you avoid that h feedback loop where you only have the information you have available you only have your golden examples
889. 85:09and everything looks right to me Always try to get a little bit creative when building a harness to try and guide which other uh points of reference it could use to help you make a better decision or to know if something is not quite right.
890. 85:25Does that
891. 85:26Yeah.
892. 85:38And I guess we generally try to like work whoever it is to like understand what would be the best set of things that agent should use the best answer.
893. 85:50So trying to understand how that would have been the optimal to give you another example that could be useful.
894. 86:05I think the golden examples give you your ground truth for making sure that you are within the right space but no domain is isolated.
895. 86:16So what happened to me a lot of times when I was a BI analyst in some like company is that I thought what I wrote was really good.
896. 86:27So my query was good.
897. 86:29The number I got seemed reasonable and I was checking everything and nothing looked off.
898. 86:34And then when I presented the dashboard to some stakeholder immediately went like that's not correct and I was like what do you mean?
899. 86:43Do you know SQL more than I do?
900. 86:46Did you do do the analysis with me?
901. 86:48And turns out this person was from a separate team and they just had an idea of how many whatever sales were made in that year.
902. 86:57So they knew instinctively that number can't be right because it means my information in this other place is incorrect.
903. 87:06So it's not if we translate that to the agentic world the golden examples I will give my agent in my hardness will be based on like biases and selection bias and so on
904. 87:18based on the domain and I picked them for the domain and I think they're good for the domain but when you look at a slightly tangential domain you may find out that
905. 87:30actually this is nonsensical like this number is wrong just by looking at this other thing next to me I I don't even need to look again into my calculations.
906. 87:40Cool.
907. 87:42Erh.
908. 87:43So there is a concept called thing slicing where you have thin slices of your anecdotal experience and you know what things feel right or feel wrong based on your experience in your domain.
909. 88:01Whatever analysis you're doing.
910. 88:03the model will have an average of everyone else's experiences.
911. 88:08So when you're building your hardness, you need to make sure that you can uh have a feedback mechanism so that you can verify from what the model thinks it knows
912. 88:22all the way to what you think you know.
913. 88:25And if you flip that convoluted sentence I just said it's the it's how we started the whole framework and all the principles where you're trying to get to that
914. 88:37shared domain shared context.
915. 88:40You're essentially translating your language into the training data set for the model.
916. 88:46You're just working with the model to be like okay this is how I understand the problem or how my company does this.
917. 88:53and then the model is helping like okay this is how would I fit it in my training data set here we're doing the opposite we're just saying okay I think we got to a point where you can make a reasonable judgment over this domain
918. 89:07but let's have a feedback loop so I can tell you from my experience if this looks entirely off or if it looks reasonable so um yes it's probably fine may look nothing like yours
919. 89:22I think is a way good way to describe it like yeah it looks about right and then turns out your company does care about that 5% difference it makes a world of a difference when you're building hardness for your team that you're aware that that 5%
920. 89:38it will be a deal breaker for the whole team.
921. 89:42Um there's some stuff that happens if you don't have um a way to do this.
922. 89:48So checking everything line by line or checking nothing or you know no sense of what about right looks like and if you don't have that point of reference your harness won't have it either.
923. 90:00So you need to build those kind of like external validation points.
924. 90:07I'm going to skip through some of this.
925. 90:13And for the verification, it's those notes in the H decision as well where we're adding uh an external point of reference to some of the stuff the model thought it knew.
926. 90:37It's kind of satisfac satisfactory when the thing just like
927. 90:41the branch go comes back.
928. 90:42Yeah, exactly.
929. 90:44And again, I think it creates a a trace to to um be able to like the the thin slice and the quick instinctive look also comes from from this because you understand the thought process
930. 90:58that the agent went through and not only the the output only.
931. 91:06I go back.
932. 91:07Yes.
933. 91:08So, as with the other examples, if you're still following along, uh there's a few things you can try.
934. 91:15So, you can open a branch uh on a tagged challenge.
935. 91:20That's basically a separate agent that we tasked with reviewing the plan, reviewing the assumptions, and then doing some healthy challenge.
936. 91:29Does this read reasonable?
937. 91:32Um so just click it open it up and then you will see how just Philippe did it will join back into the trunk and actually change the plan.
938. 91:42So there is a consequence and we think this is another important part of being intentional with the design of your harnesses.
939. 91:54We could have left it as LLM as a judge.
940. 91:57So for every decision, for every plan, whatever, spin another uh agent and then do a review and call it a day.
941. 92:06The problem is if there's no feedback loop, our financial decision or decision whatsoever doesn't improve at all.
942. 92:12So that's part of the design, the taste you need to build h to create fitforpurpose harnesses for whatever use cases you have.
943. 92:22Then the other thing you can try is opening another and just clicking on dismiss.
944. 92:27And we did that on purpose because sometimes as I was saying before some conversations don't actually change anything or not material to your process and you will know better.
945. 92:39So you need to have that kind of like point of reference for validating things.
946. 92:44Um so just to summarize and leave this on the screen for a moment.
947. 92:49Um, if you're still playing around with the harness, uh, the framework for building really good harnesses that you can trust with financial data with money, it's built on these five principles.
948. 93:03So having a shared domain, having consistency, showing the receipts, keeping your accountability, and making sure that you create intentional loops for verification outside of what you think, you know.
949. 93:18Erh, what?
950. 93:24No confetti.
951. 93:25Disappointing.
952. 93:27Okay.
953. 93:28So, there's a few cases where we think it's important to define.
954. 93:34Oh, sorry.
955. 93:35You had your hand up.
956. 93:37Yes.
957. 93:38So, I just confirm my understanding.
958. 93:41You brought us a framework like this five principles.
959. 93:45You can build your harness upon it.
960. 93:47And you also brought one harness as an example.
961. 93:51So, I'm I'm thinking about when should I build harness, how generic or how specific the harness should be?
962. 94:01Because you solve problems with the specific harness and like for example, how many harnesses do you have in your work?
963. 94:10It's just one and you solve everything that you have for this in this harness or you have multiple more specific harness.
964. 94:19How generic and how often do you build one?
965. 94:22Like how much effort do you put in building one?
966. 94:25you make one every month or you make one every six months.
967. 94:29That's a that's a great question and there's no clear answer of course.
968. 94:32Um I think it's we could we could trace as you said like the two ends of the spectrum as I just use clo code or or codeex for everything and I just embed everything into
969. 94:45skills instructions so on so forth and on the other end of the spectrum is for every small new thing I want to do I'll build it from the ground up and like of course that will get tiring at some point because like you're spending more time creating your tools than actually using the tools sometimes
970. 95:02Um again there's no clear answer.
971. 95:04I think it comes to um when you feel the use case is worth enough.
972. 95:10So you want reproducibility, you want it to feel few the results you are getting using the pre-built harnesses are very frequently getting it wrong because the value you get from building the harness
973. 95:26is you get more control again with the deterministic bit together with the nondeterminism.
974. 95:33Right?
975. 95:34Um the reason why we're talking about the the principles in the case in the um scenario of a financial analysis is because if we do this type of use case
976. 95:46in a harness that doesn't have it we'll probably have some of the f pitfalls that Franchesco talks through each of the steps.
977. 95:54So then in this case it makes sense for um to avoid these pitfalls we are seeing over and over and over again to build the thing.
978. 96:03But again, I think it comes down to how many users you believe will go through that use case, how critical is the problem that you were seeing.
979. 96:12So like if if there's a non-b businessiness critical um task that you were solving, you were probably fine to have a human in the loop reviewing the outputs that you get and steering the agent even if that takes a few extra turns.
980. 96:26But if you're putting this in the hands of a uh internal user, for instance, let's say you were building something for your team of credit analysts, um you're going to have 50 people using something
981. 96:38to support key company decisions.
982. 96:41Yeah, you absolutely don't want a lot of variability on that.
983. 96:46So again, there's no clearcut I would say, but these are the things I would consider in in making the decision.
984. 96:53Maybe I can help with uh the counterfactual.
985. 96:56So, we think this is a good recipe specifically for financial harnesses because there's a lot at stake.
986. 97:04You want a lot of feedback loops.
987. 97:06There's very specific things about those use cases that somehow we're representing here even at the personal level or the company level.
988. 97:16But imagine you want to do you want to use AI just to produce like social media content and maybe one of your use cases is just flooding Instagram and Tik Tok with AI generated videos like everyone is doing.
989. 97:31There's some steps here that you probably don't care that much.
990. 97:34Evidence gathering.
991. 97:37What for?
992. 97:38H verification or accountability even less so.
993. 97:42So this is specifically geared towards like financial use cases and it can be extended to different levels of it but everything hinges on the fact that is a trust system.
994. 97:56H this recipe may not be as useful if your use cases are not necessarily like that and in terms of how many harnesses you want to build.
995. 98:05I think Filipe said it better, but you can easily fall victim for doing a lot of axe sharpening and not actually using the axe.
996. 98:17H So, at some point, if you're trying to use something and you're not getting the right results, you can try adding a couple of skills, a couple of commands, a couple of extra things, still not getting the results or the experience you want.
997. 98:31You're finding yourself copy pasting text from one chat to another, opening a new session.
998. 98:36It feels convoluted harness.
999. 98:39So you all the repeated things that you had to do three five times.
1000. 98:44Okay.
1001. 98:45Just once and you design it around it.
1002. 98:49Yes.
1003. 98:50Yes.
1004. 98:51Some more related question.
1005. 98:52So for your wedding planning task, right?
1006. 98:55I think one of like one of the naive way of looking at this is like maybe agent right like agents, venue agents.
1007. 99:04Uh, have you have you have you tried that and what's the outcome or like your opinion on that?
1008. 99:09I will start by confessing that I have the most expensive cloth subscription and I do max it out and not in a good way as in Anthropic is definitely on the on the red because of me.
1009. 99:23I tried research everything for all the venue, the flowers, the wedding, do the planning, do the different like scenarios for how much we will have to pay etc. And what I found is that
1010. 99:36in many cases it was good enough to give me the list of the top five photographers in Madrid with the new web search with like a refined h tooling etc.
1011. 99:46Worked like a charm.
1012. 99:47didn't have to do anything else just spinning and having comparisons and even reaching out to them but for some more nuance use cases where I do want to have more control and predictability and this is linking to the financial nature of them
1013. 100:01I did find myself being like I need to build something robust I'm not recalculating this again uh and to to make the pain very real I remember myself one day because I was in the car and I was like I can dictate some stuff.
1014. 100:19Voice mode and then I was driving and it was about an hour drive back and forth, you know, an hour and an hour and I re had I had the same conversation three times in that t in that trench
1015. 100:32just because every time I was trying to do some financially related or adjacent planning, something will go wrong and I'll be like so yes, that helps.
1016. 100:47So getting back to your example of being a BI right like um I don't think back in the day there were people who were validating your SQL queries right
1017. 101:00um so this seems like we are it's like next level of SQL queries right now you're building tools because harnesses are easier to build and stuff have you found
1018. 101:12ways to kind of make this point clear to leadership ship in in conversations that like because because to me it feels every time there is an AI there is an expectation that it will be 100% correct
1019. 101:25like you see what I'm getting at
1020. 101:28so is your question about managing the expectations so that when people do agentic self surf
1021. 101:36it's not enginic sales more so like let's say you build this and and you're talking to your senior leadership to make a financial decision in your company Right?
1022. 101:46And they start drilling you and drilling you and and the moment that you say, "Oh, I use AI and harnesses, they're like, well, this has to be 200% correct or something like you know what I'm saying?
1023. 101:56Have you found the way to to kind of make sure that they understand this is still SQL queries?"
1024. 102:02Bit of a spoiler,
1025. 102:04but there's a talk tomorrow.
1026. 102:06[laughter]
1027. 102:08I don't know.
1028. 102:09Do you have any?
1029. 102:10[clears throat]
1030. 102:10Yeah.
1031. 102:10Yeah.
1032. 102:10Yeah.
1033. 102:11No, I think um it's with the AI first world, we sometimes as you were saying forget that what's that that's only a two and a means to an end
1034. 102:25be before like what's what's going on right so I think what we would have noticed and again from Monzo's experience we've been approaching it from an highly impactbased um point of view so
1035. 102:39it's not about we should use AI for this because AI can something something something.
1036. 102:45It's like what's the outcome we're looking for here?
1037. 102:48What how do we measure success here?
1038. 102:50Are we looking for less uh back and forth with a BI team?
1039. 102:55Uh we don't even have a BI team, but you get you get what I mean.
1040. 102:57Um to validate a new dashboard.
1041. 103:01Okay.
1042. 103:01So, we can track that and actually see if AI can support us in in generating that outcome.
1043. 103:07um are we looking for um less I don't know internal ask tickets because the number looks wrong.
1044. 103:16Okay, so let's track that and then the approach that we do the way we do that doesn't really matter to a point if that makes sense because it will be again a tool that can get us to that
1045. 103:29um to that result probably faster with better guard rails and so on and so forth.
1046. 103:36So I think it's less it's less about gaining trust on the implementation on the AI itself but more about aligning on what's the outcome we're trying to drive with the AI.
1047. 103:51I think we had another one there.
1048. 103:55My question is more you have show a harness that has some complexity right you have a UI you have databases say but I have also build processes that invoke some of these steps but they are all at skill
1049. 104:11right that it ask you questions it runs the number based on a Python script and so on so my question is more like where you see the boundaries between like having a skill that does all these creating like a separate tool or
1050. 104:26have have you tried that?
1051. 104:27Absolutely.
1052. 104:27Yeah, absolutely.
1053. 104:29Um I think it it again very similarly to to what we were saying about your user base and the reproducibility you want is when this becomes critical enough that the tradeoffs we are getting with the
1054. 104:43nondeterministic approach are are not uh we're not willing to have them.
1055. 104:48I usually describe the the skill command like the the promptbased approach as hey please follow all of this and make no mistakes please please please like again because this is what's going on right we are we are giving a a lot of organized
1056. 105:04very well organized instructions to a machine and hoping that it it will perform these instructions agents are good enough nowadays they will probably perform your instructions 99% of the time.
1057. 105:18Uh, but what about that 1%?
1058. 105:21What about an edge case that makes things don't work?
1059. 105:26What about if you distribute this to your internal users and your internal user has a instruction in their user level cloud.mmd file that goes against what you are telling your agent to do.
1060. 105:40You see my point is like um there are many when you when you start distributing it to a number of users and use cases things can get messy in places you are not able to predict
1061. 105:54and then you can find yourself I we have a at Moz we have an internal like contribution for agent extensions where multiple teams can well I've come across something that works with me let me share it internally
1062. 106:07um so like plugins skills and so on and very frequent ly we when reviewing the instructions we see things trying to cover for if the user tries to do this
1063. 106:17tell the user back that this skill does not support that and you see like it's again trying to cover for many things that with the determinism and when I say determinism is the backend code like the python the whatever we are using
1064. 106:31uh that the harness provides that out of the like out of the box is something we can cover or
1065. 106:43anyone has any other comments, questions.
1066. 106:49All right, cool.
1067. 106:52Where is the thingy?
1068. 106:53[laughter]
1069. 106:54Never mind.
1070. 106:55Never mind.
1071. 106:55Never mind.
1072. 107:00Cool, cool, cool, cool.
1073. 107:01Skipping ahead.
1074. 107:01Skipping ahead.
1075. 107:02Ah, fun.
1076. 107:04Erh, I wanted to make a joke and I completely missed it for clicking this thing too quickly.
1077. 107:09But finance is funny.
1078. 107:12And I say funny because I find it to be a super interesting domain with a the added advantage that back in the day finance felt like just working with spreadsheets,
1079. 107:26very like boring work, adding numbers here and there, formulas, whatever.
1080. 107:31If you ask any accountant is all about creativity and interpretation because they will come up with the weirdest things and I think in that level of ambiguity AI is really good
1081. 107:44really really good you have some bits that are the spreadsheet and then you realize some of the cash flow didn't quite make sense because some of the revenue was attributed a month later and you're like ah creativity.
1082. 107:57So because we are in this space of finance and depending on where is your mind mindset at you could look at a problem as something very simple that you can just get an agent to fix within 10 minutes
1083. 108:11or you could look at it as it actually is which is a mixture of things that need to run reliably 100% of the time and a lot of interpretation and freedom of
1084. 108:23creativity in between the lines.
1085. 108:26So yeah, sorry, very poorly delivered joke, but that's what I wanted to go through.
1086. 108:30Uh, and the bottom line being the design of your harnesses needs to be intentional to increase trust.
1087. 108:41Don't assume that oh because this is running on AI is going to be what was the the quote 200% accurate.
1088. 108:49It will likely not unless you design all the mechanisms so that whatever process runs you know you can trust the results to a degree.
1089. 108:59Um and yeah models are going to get better.
1090. 109:03We have seen the last six months nine months in saying I'm getting a little bit of whiplash even with you know how fast things are going but they still have to earn the trust.
1091. 109:13it's not a 100% right and in when we're talking about money 1% wrong sometimes is a lot of money.
1092. 109:21So yeah, we kind of skipped into the questions, but if anyone has anything else,
1093. 109:26yeah, just really quick, like in the UI, how do you demonstrate trust?
1094. 109:29Do I like you just ask it?
1095. 109:32Show me how you calculated this or does it show you the code?
1096. 109:36Can you show an example?
1097. 109:37Like how do you prove it?
1098. 109:38The intent was that it was a aggregation of everything.
1099. 109:42So all the steps in the framework should have failed like, oh, but I know where this is coming from, but I know this is running as code, but I know this is tested.
1100. 109:51So it should have been an experience to kind of like immerse into this feels trustworthy.
1101. 109:57But again, it's one implementation we came up with.
1102. 110:00So not not super direct.
1103. 110:02Yeah.
1104. 110:02But I think the the takeaway is there won't be a single thing that will work to earn the user's trust at any point.
1105. 110:10So I think that's for some people the verification will be more important.
1106. 110:14Some others are a bit more skeptic and they want to see the actual calculations and numbers and so on.
1107. 110:19So I think again that the process guides and covers for a multitude of u of possible possibilities.