WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=e9-br9hveRI

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/e9-br9hveRI.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:01.309
Morning.

s2
00:00:01.309 --> 00:00:03.309
[music]

s3
00:00:14.639 --> 00:00:18.640
Thanks everyone for being for being here and welcome to AIE.

s4
00:00:18.640 --> 00:00:22.240
I guess first first session for everyone and for us as well.

s5
00:00:22.240 --> 00:00:28.160
So h we wanted to introduce ourselves and tell you about what we're going to discuss.

s6
00:00:28.160 --> 00:00:31.679
We think we need to stop talking with spreadsheets.

s7
00:00:31.679 --> 00:00:37.920
So, this is all going to be about building a financial analyst harness that you can actually trust.

s8
00:00:37.920 --> 00:00:39.520
We are Franchesco Filipe.

s9
00:00:39.520 --> 00:00:42.079
A little bit of a quick introduction about me.

s10
00:00:42.079 --> 00:00:43.920
I'm originally from Venezuela.

s11
00:00:43.920 --> 00:00:47.920
I lived in the UK for about seven years and recently moved to Madrid.

s12
00:00:47.920 --> 00:00:53.440
H I work at data and I work at a bank as well with Felipe.

s13
00:00:53.440 --> 00:00:54.079
Erh.

s14
00:00:54.079 --> 00:00:57.760
I like just finding the best eeries in any city that I visit.

s15
00:00:57.760 --> 00:01:02.960
So if you have any recommendations for New York, I'm more than happy to take them and just travel around.

s16
00:01:02.960 --> 00:01:11.760
And the reason my hobbies sound so vanilla is because I just spend most of my time building stuff and nerding out at the expense of my personal time.

s17
00:01:11.760 --> 00:01:14.720
So that's how you end up with a list like that.

s18
00:01:14.720 --> 00:01:16.640
Uh and then nice.

s19
00:01:16.640 --> 00:01:17.280
All right.

s20
00:01:17.280 --> 00:01:19.280
Um happy to meet everyone.

s21
00:01:19.280 --> 00:01:20.640
Um also quick intro.

s22
00:01:20.640 --> 00:01:21.200
I'm Philipe.

s23
00:01:21.200 --> 00:01:22.799
I'm originally Brazilian.

s24
00:01:22.799 --> 00:01:24.640
Um, I also moved to the UK.

s25
00:01:24.640 --> 00:01:27.040
It's been a year and a half now.

s26
00:01:27.040 --> 00:01:28.640
So, it's been a fun journey.

s27
00:01:28.640 --> 00:01:30.880
Um, life turned pretty much upside down.

s28
00:01:30.880 --> 00:01:33.439
But, yeah, it's good to be it's good to be there.

s29
00:01:33.439 --> 00:01:41.920
Um, I like to tell a story that before going to to decide whether I wanted to do it in university, I used to play piano for like literally seven hours a day.

s30
00:01:41.920 --> 00:01:48.399
Um, and the natural decision there was to then study electrical engineering because who wouldn't, right?

s31
00:01:48.399 --> 00:01:55.759
Um, I ended up at some point landing in in data because someone showed me something that at the time called uh Google Data Studio.

s32
00:01:55.759 --> 00:01:57.520
Now it's Looker Studio, but now they went back.

s33
00:01:57.520 --> 00:01:59.280
I don't I don't even know what's going on.

s34
00:01:59.280 --> 00:02:03.520
Um, and then my first dashboard looked like a papyrus of how vertical it was.

s35
00:02:03.520 --> 00:02:06.560
You had to scroll scroll scroll scroll scroll down to see something.

s36
00:02:06.560 --> 00:02:08.879
Um, but yeah, the the rest is history.

s37
00:02:08.879 --> 00:02:13.120
So I worked at a lot of um startups and small teams and scaleups.

s38
00:02:13.120 --> 00:02:17.280
Um and then eventually went to a big company at New Bank.

s39
00:02:17.280 --> 00:02:28.319
Maybe some of you do know um and then before going back to to Monzo which is I would say a middle turn between a scale up and a big enterprise company but it's growing uh so fast for now.

s40
00:02:28.319 --> 00:02:38.480
Um and yeah most of my time currently has been working at Monzo with building internal AIdriven solutions for internal productivity primarily for the data team.

s41
00:02:38.480 --> 00:02:43.040
So, we basically want to get our data team better by um building with AI.

s42
00:02:43.040 --> 00:02:51.360
Um and yeah, when I'm not talking to robots and agents, uh you can probably find me climbing somewhere um either in nature or in a gym.

s43
00:02:51.360 --> 00:02:57.920
Um and some starting like points of why are we bringing this for you today?

s44
00:02:57.920 --> 00:03:02.560
I think um both Franchesco and I worked in data teams in financial services.

s45
00:03:02.560 --> 00:03:06.400
So again, Monzo, New Bank and a few other companies in the in the sector.

s46
00:03:06.400 --> 00:03:08.959
Um, we're now pushing what AI can do there.

s47
00:03:08.959 --> 00:03:15.680
Our team at Monzo has been um, working a lot with with adopting AI from an impact perspective.

s48
00:03:15.680 --> 00:03:17.680
We don't want to be like token maxing.

s49
00:03:17.680 --> 00:03:22.400
We don't want to be spending a lot of money if that doesn't bring us like concrete business value.

s50
00:03:22.400 --> 00:03:28.000
Um, so the adoption is very very deliberate and we've been seeing very good results on that.

s51
00:03:28.000 --> 00:03:35.599
And on the flip side, this is something also something we've been we've been doing, Franchesco particularly is applying AI to make our lives easier.

s52
00:03:35.599 --> 00:03:46.640
Um and again not through the approach of I'm going to set up an open client instance nothing against it um and just give it connections to everything but actually what are the things that I

s53
00:03:46.640 --> 00:03:53.360
care about in my life that I can um get the help of AI to to u make easier.

s54
00:03:53.360 --> 00:04:08.720
So this is why we say we're not representing Monzo um in this example because we're going to use the knowledge we bring from Monzo but uh build with you an example where we can um actually see that in practice without having to discuss a lot of domain knowledge.

s55
00:04:08.720 --> 00:04:17.840
Um but if you are curious about the things we are building at Monzo we have a breakout session tomorrow um where we're going to talk about agent chip which is our inhouse

s56
00:04:17.840 --> 00:04:20.160
remote agent capability.

s57
00:04:20.160 --> 00:04:22.320
Um so with that Franchesco back to you.

s58
00:04:22.320 --> 00:04:22.800
Cool.

s59
00:04:22.800 --> 00:04:36.560
So, as a reminder, today the goal is trust and it may seem a little bit counterintuitive because it feels very wishy-washy and this is a workshop and you're supposed to be coding really hard for two hours.

s60
00:04:36.560 --> 00:04:40.080
H but you're going to see we're going to give it a little bit of a spin.

s61
00:04:40.080 --> 00:04:48.160
Uh because I think we're past the point of having to spend hours like grinding through your keyboard uh to get something that you can trust.

s62
00:04:48.160 --> 00:04:51.120
So, let's start with the domain.

s63
00:04:51.120 --> 00:04:53.280
Finance is tailored.

s64
00:04:53.280 --> 00:04:54.960
You have your numbers.

s65
00:04:54.960 --> 00:04:57.360
You have your strategy.

s66
00:04:57.360 --> 00:05:00.560
Sometimes you don't really have a strategy.

s67
00:05:00.560 --> 00:05:07.600
And there is the knowledge that you have on finance, on your personal situation and so on or generally misconceptions.

s68
00:05:07.600 --> 00:05:12.720
H when we talk about a financial analyst, there is a very specific job description.

s69
00:05:12.720 --> 00:05:15.280
We are being a little bit generous with that job description.

s70
00:05:15.280 --> 00:05:24.720
So it's going all the way from doing research for investments, investment firms, banks and so on all the way to like personal use cases as well.

s71
00:05:24.720 --> 00:05:26.960
But finance is finance, right?

s72
00:05:26.960 --> 00:05:35.840
So right now the gap on producing something, getting hold of your finances, doing an analysis is not really speed.

s73
00:05:35.840 --> 00:05:43.440
H I'm pretty sure some of you may have some background tasks running with like claude or chpt getting some stuff done.

s74
00:05:43.440 --> 00:05:45.600
So that's not the bottleneck.

s75
00:05:45.600 --> 00:05:55.919
H we believe that based on the experience we had over the last few months, the bottleneck now is how it connects with us.

s76
00:05:55.919 --> 00:06:05.919
Um so trust is the thing we cannot take for granted and transparently because we're talking about money specifically.

s77
00:06:05.919 --> 00:06:26.880
Erh this is one of the highest stakes topics uh we need to invest in making sure that there is trust built into the system and transparently if money is not high stakes enough for you and you can like oneshot your money problems your budgeting your financing everything else h find me after the workshop I will buy you

s78
00:06:26.880 --> 00:06:38.960
in into some investment ideas so yeah let's chat later please um so where everything starts is from the goal which is earning the trust.

s79
00:06:38.960 --> 00:06:45.759
So then you can truly delegate the boring work to multiple agents, your swarms, your factories, whatever.

s80
00:06:45.759 --> 00:06:49.360
But how you do trust the results that you're getting.

s81
00:06:49.360 --> 00:06:56.080
Erh, I'm going to namerop a few things because honestly none of them are new.

s82
00:06:56.080 --> 00:07:05.919
You're probably already grinding thinking well you buy trust by h setting up traces or doing loops or any of the other keywords we have.

s83
00:07:05.919 --> 00:07:11.199
So unit testing that gets old planning with your agent we know that that's good.

s84
00:07:11.199 --> 00:07:23.840
Uh doing evals for sure traces and observability loops and graphs like I can keep naming components of how you think you can start building trust but they are basically

s85
00:07:23.840 --> 00:07:25.199
components.

s86
00:07:25.199 --> 00:07:31.360
Erh what we want to discuss today is actually the design principles on how to achieve that trust.

s87
00:07:31.360 --> 00:07:40.160
So building a hardness that actually h works for money matters h requires an in im immense amount of trust.

s88
00:07:40.160 --> 00:07:43.440
So we're not going to be discussing specific technologies.

s89
00:07:43.440 --> 00:07:52.319
We're not going to read you the readme of a particular product and we will assume you know some of the concepts that have been listed so far.

s90
00:07:52.319 --> 00:08:02.560
Um and mainly this is going to be a slightly different workshop because we want you to feel how the principles work in reality.

s91
00:08:02.560 --> 00:08:16.879
So we have built a hardness today around a very simple example and we want you to try it out but don't focus on oh if I type this then I get the response so much quicker and the UI is so slick.

s92
00:08:16.879 --> 00:08:18.639
It spoiler it won't be.

s93
00:08:18.639 --> 00:08:28.479
Erh but more importantly we want you to feel how it may feel like having trust in an agent that has been built with a very specific hardness.

s94
00:08:28.479 --> 00:08:39.200
Uh so again this is just one very simple implementation an example h you can roast me later if you think this is too top level but we can discuss that later.

s95
00:08:39.200 --> 00:08:48.000
So trust and I'm probably going to run out of times I can say trust without going insane but we think you need to build it from the ground up right.

s96
00:08:48.000 --> 00:08:56.720
So you have a shared domain, the whole thing about a plan, making sure that you and the model are talking in the same terms.

s97
00:08:56.720 --> 00:08:59.920
And it sounds more trivial than it actually is.

s98
00:08:59.920 --> 00:09:09.200
Then consistency and pretty sure everyone in the room will be on the same page, but it's making sure that the numbers come from a correct place.

s99
00:09:09.200 --> 00:09:15.760
So rather than a language model trying to process a lot of arithmetic, it will do some tool calling.

s100
00:09:15.760 --> 00:09:16.959
It will write us some scripts.

s101
00:09:16.959 --> 00:09:22.000
It will try to come up with a way that the results are uh consistent.

s102
00:09:22.000 --> 00:09:27.360
We also want evidence where basically show the receipts, right?

s103
00:09:27.360 --> 00:09:33.040
You can follow back and trace back the decisions and what led to them.

s104
00:09:33.040 --> 00:09:46.560
Accountability is probably a very spicy one and maybe the mo most abstract of them, but ultimately even if you're running a 100 agents, the results of what they do,

s105
00:09:46.560 --> 00:09:48.640
you're still accountable for them.

s106
00:09:48.640 --> 00:09:51.200
So that's something we take very seriously at work.

s107
00:09:51.200 --> 00:10:02.640
And we think it's a mindset shift that when people make it, they start building systems that evolve with them and not black boxes that they just throw problems at.

s108
00:10:02.640 --> 00:10:05.120
hoping that something sticks.

s109
00:10:05.120 --> 00:10:07.120
And finally, verification.

s110
00:10:07.120 --> 00:10:13.920
So, we want to make sure that there is a separate point of view, another perspective.

s111
00:10:13.920 --> 00:10:22.880
H I would say I'm guilty of thinking I have the best idea in the world probably like three times a year and then only to have Filipe tell me,

s112
00:10:22.880 --> 00:10:23.920
please don't implement that.

s113
00:10:23.920 --> 00:10:25.120
Please, please, please.

s114
00:10:25.120 --> 00:10:27.680
And I I Oops, one second.

s115
00:10:27.680 --> 00:10:31.920
And I would just compliment on on this bit that I think two things.

s116
00:10:31.920 --> 00:10:35.519
The first one is we have a very generous window here to to work together.

s117
00:10:35.519 --> 00:10:42.720
So um it will be very nice to learn from your folks's experience as well as we build through this principles.

s118
00:10:42.720 --> 00:10:54.399
So if this resonates with any work you've done or any particularly any challenges you've faced um when working with agents feel absolutely free to to raise that and then we can we can build it uh together.

s119
00:10:54.399 --> 00:11:04.880
Um and just dunning uh doubling down on something that Franchesco said this we're not saying this is the model you won't say us going to to the big stage and saying

s120
00:11:04.880 --> 00:11:18.720
forget everything you know this is how to do it this is a model a way to approach it uh that has been working for us in a number of use cases I'll share a few examples at Monzo later and Francisco has a few other examples as well

s121
00:11:18.720 --> 00:11:27.680
um but yeah again there's not a single silver bullet to to solve every single problem we And also feel feel free to challenge us.

s122
00:11:27.680 --> 00:11:35.040
If you think actually trust overrated just throw a 100 agents at a problem eventually will get solved.

s123
00:11:35.040 --> 00:11:37.120
Happy to hear the success stories.

s124
00:11:37.120 --> 00:11:42.959
H but from repetition from experience and a lot of different use cases at work and outside.

s125
00:11:42.959 --> 00:11:53.600
We do think you need to build a system that gives you the confidence that things are going in the right direction and not just that things look correct.

s126
00:11:53.600 --> 00:11:54.720
So yeah, trust.

s127
00:11:54.720 --> 00:11:55.600
Happy days.

s128
00:11:55.600 --> 00:12:04.320
Um, speaking of trust, I'm here to confess that I have an ulterior motive with today's workshop.

s129
00:12:04.320 --> 00:12:06.399
Uh, so breaking your trust immediately.

s130
00:12:06.399 --> 00:12:07.120
What is it?

s131
00:12:07.120 --> 00:12:09.680
Less than five minutes.

s132
00:12:09.680 --> 00:12:16.480
You are here to help me, Francesco, to plan my wedding.

s133
00:12:16.480 --> 00:12:20.639
So nobody's super excited, but you will get excited.

s134
00:12:20.639 --> 00:12:26.560
uh specifically just to figure out some some stuff around the finances of paying for a wedding.

s135
00:12:26.560 --> 00:12:30.560
Um and because it's a conference, I want to make it as cringey as possible.

s136
00:12:30.560 --> 00:12:32.160
So, show of hands.

s137
00:12:32.160 --> 00:12:34.720
Uh who has had a wedding?

s138
00:12:34.720 --> 00:12:37.279
Who's gotten married?

s139
00:12:37.279 --> 00:12:38.639
Okay, awesome.

s140
00:12:38.639 --> 00:12:42.160
Number of guests under 50.

s141
00:12:42.720 --> 00:12:43.040
Yeah.

s142
00:12:43.040 --> 00:12:43.279
Okay.

s143
00:12:43.279 --> 00:12:44.480
No, still show of hands.

s144
00:12:44.480 --> 00:12:45.920
It's still like cringey mode.

s145
00:12:45.920 --> 00:12:47.920
Okay, cool.

s146
00:12:47.920 --> 00:12:49.920
About 100.

s147
00:12:49.920 --> 00:12:50.320
Yeah.

s148
00:12:50.320 --> 00:12:51.760
Nice.

s149
00:12:51.760 --> 00:12:53.839
How about 150?

s150
00:12:53.839 --> 00:12:54.560
Okay.

s151
00:12:54.560 --> 00:12:56.399
Nice.

s152
00:12:56.399 --> 00:12:57.279
200.

s153
00:12:57.279 --> 00:12:57.839
More.

s154
00:12:57.839 --> 00:12:58.240
All right.

s155
00:12:58.240 --> 00:12:58.480
Yeah.

s156
00:12:58.480 --> 00:12:58.720
Yeah.

s157
00:12:58.720 --> 00:12:59.440
Big wedding.

s158
00:12:59.440 --> 00:13:01.440
Nice.

s159
00:13:01.920 --> 00:13:11.600
This is probably going to resonate, but it sounds like it's going to be a very simple calculation, very simple process, and the wedding is going to go by nicely and smoothly.

s160
00:13:11.600 --> 00:13:15.839
And for the people who got married, probably you know that is not the case.

s161
00:13:15.839 --> 00:13:22.639
Um, and even running the calculations and figuring out how things got so expensive so quickly.

s162
00:13:22.639 --> 00:13:24.399
It's not trivial.

s163
00:13:24.399 --> 00:13:26.079
This is a little bit of storytelling.

s164
00:13:26.079 --> 00:13:37.360
And hopefully my girlfriend doesn't fiance doesn't see this, but we were doing some calculations and she was like, "No, yeah, but the cost per like dinner is this much."

s165
00:13:37.360 --> 00:13:41.920
And I'm like, "Yeah, and you have another two variable costs on top of that.

s166
00:13:41.920 --> 00:13:43.120
So that's not the number."

s167
00:13:43.120 --> 00:13:49.760
And anyway, budgeting for this has been a challenge to underelling it a little bit.

s168
00:13:49.760 --> 00:13:54.160
I wouldn't recommend doing it, but you know, I'm already on the train.

s169
00:13:54.160 --> 00:14:03.040
Erh, so today you're going to give me a hand with the wedding unpaid labor or you can do your own plan.

s170
00:14:03.040 --> 00:14:06.720
H So we put a few things in a repo that you're going to get access to.

s171
00:14:06.720 --> 00:14:10.399
So you if you want to pick your own adventure, do that.

s172
00:14:10.399 --> 00:14:15.040
wouldn't recommend because I do need help, but whatever you choose.

s173
00:14:15.040 --> 00:14:20.480
So, we'll give you a few minutes uh for you to set this up.

s174
00:14:21.600 --> 00:14:30.880
Just either scan the QR code or go to my uh repo and there will be some instructions on how to set it up.

s175
00:14:33.120 --> 00:14:38.560
We did not plan a background music like an elevator music for this moment.

s176
00:14:38.560 --> 00:14:42.959
Yeah, let us know if you have any any trouble setting up.

s177
00:14:44.000 --> 00:14:47.199
Also, if anyone sings, it would be a good time to

s178
00:14:47.199 --> 00:14:49.920
karaoke live.

s179
00:14:51.360 --> 00:14:54.079
By the way, you don't need to execute anything just yet.

s180
00:14:54.079 --> 00:15:01.519
Just make sure you have the repo in your preferred um setup and then we'll guide you through the the execution.

s181
00:15:01.519 --> 00:15:02.000
Cool.

s182
00:15:02.000 --> 00:15:06.079
Uh, so you just cloned a virus.

s183
00:15:06.079 --> 00:15:06.800
No, just kidding.

s184
00:15:06.800 --> 00:15:10.480
But a lot of vioded stuff in there.

s185
00:15:10.480 --> 00:15:12.959
The structure does make sense.

s186
00:15:12.959 --> 00:15:16.639
So we have a guide on how we start for today's workshop.

s187
00:15:16.639 --> 00:15:20.320
H, we have the example most important thing of the talk today.

s188
00:15:20.320 --> 00:15:22.880
So helping me figure out my wedding.

s189
00:15:22.880 --> 00:15:28.560
H, also the harness itself, the thing we build to show you how the principles are applied.

s190
00:15:28.560 --> 00:15:36.160
And then finally, there is a folder where everything you build that drifts away from what we're doing is going to get stored in.

s191
00:15:36.160 --> 00:15:46.000
So again, it's a little bit of a pick your own adventure uh with the added risk that we may not be able to help you if you get stuck, but you can

s192
00:15:46.000 --> 00:15:53.040
um and everything you do today, you can take it home and you can check what you did and what you like, what you didn't.

s193
00:15:53.040 --> 00:16:00.160
And particularly I was going to say particularly the harness itself talking to to the jokes we were just making about like this is as far as this was built.

s194
00:16:00.160 --> 00:16:12.160
Um again the the principles we're sharing here the principles that are built here we were discussing just yesterday is like uh this can feel like we are showing you a table to

s195
00:16:12.160 --> 00:16:15.040
teach you how to build a table if that makes sense.

s196
00:16:15.040 --> 00:16:21.040
Uh but we really do we really do believe that the again building is not the hard thing.

s197
00:16:21.040 --> 00:16:24.000
The hard thing is knowing how we want to approach the problem.

s198
00:16:24.000 --> 00:16:25.839
So that's why we're bringing it like this.

s199
00:16:25.839 --> 00:16:36.160
But we're gonna open the open the look under the hood a bit um at times just to show you what's going on um behind the the principles we're applying.

s200
00:16:36.160 --> 00:16:48.399
And pretty much like Filipe was saying, it can feel akin to saying like, "Hey Claude, please build me a $1 million company."

s201
00:16:48.959 --> 00:16:58.560
If you don't know exactly what are the principles and the decisions you're making in order to make something happen, doesn't matter how many times do you try, you're likely going to be just

s202
00:16:58.560 --> 00:17:05.039
leaving yourself up for the randomness of something good happening by accident.

s203
00:17:05.039 --> 00:17:16.160
So we want to anchor the principles again because they build on top of each other and we think there is a nice sequence that kind of makes logical sense.

s204
00:17:16.160 --> 00:17:20.640
So building that shared domain plan mode however you want to call it.

s205
00:17:20.640 --> 00:17:37.360
Then setting up consistency making sure that whatever can run or should run reliably does then having the evidence to back up where the decisions are coming from what information is coming through if things are being tested and so on and putting that front and center.

s206
00:17:37.360 --> 00:17:44.960
Then the accountability how do you keep a good enough grasp of what the machine is doing?

s207
00:17:44.960 --> 00:17:49.440
You don't want to go full blackbox mode because then your accountability is gone.

s208
00:17:49.440 --> 00:17:52.960
And that has happened to many of us in this room, I'm sure.

s209
00:17:52.960 --> 00:18:00.240
But the other side is also not very realistic where oh for every change you make send me the diff.

s210
00:18:00.240 --> 00:18:03.200
I'm going to review it line by line for sure.

s211
00:18:03.200 --> 00:18:04.799
Nobody has done that ever.

s212
00:18:04.799 --> 00:18:06.960
Uh and then finally verification.

s213
00:18:06.960 --> 00:18:22.880
So in data and in finance uh you often need to have a separate point of reference a counterfactual that you can check against h and ignoring that part of the equation just leads to you running things on your own biases.

s214
00:18:22.880 --> 00:18:25.200
So your results are always going to be skewed.

s215
00:18:25.200 --> 00:18:28.000
There's some shortsidedness that plays into that.

s216
00:18:28.000 --> 00:18:38.080
So these principles are just to try to funnel our efforts into where is most useful to get something that we can actually trust.

s217
00:18:38.080 --> 00:18:44.799
Um heads up again something will break today surely just call us to you.

s218
00:18:44.799 --> 00:18:46.799
We will try to fix it with you.

s219
00:18:46.799 --> 00:18:49.280
H some of it is also overengineered.

s220
00:18:49.280 --> 00:18:53.120
This is not a session on good UI UX design.

s221
00:18:53.120 --> 00:18:53.919
Okay.

s222
00:18:53.919 --> 00:18:58.960
So this is just to show you how it could feel, show you the potential.

s223
00:18:58.960 --> 00:19:02.559
Uh by no means we're building a product.

s224
00:19:02.640 --> 00:19:04.960
Um and yeah, just raise your hand.

s225
00:19:04.960 --> 00:19:08.000
Something's not working out, you're a little bit confused, raise your hand.

s226
00:19:08.000 --> 00:19:10.480
We're going to float around the room.

s227
00:19:10.480 --> 00:19:15.200
Um so you may want to check if this works.

s228
00:19:15.200 --> 00:19:18.160
Try and run that command.

s229
00:19:22.080 --> 00:19:22.400
Yeah.

s230
00:19:22.400 --> 00:19:31.200
And I think at this point we can just go around if things are not working for anyone and give everyone five minutes to get going with this bit.

s231
00:19:31.200 --> 00:19:39.280
You should see the UI opening if everything goes successful with the setup works success in the terminal as well.

s232
00:19:43.360 --> 00:19:43.679
Cool.

s233
00:19:43.679 --> 00:19:51.600
And that second command there will open a very well-designed UI for you to do some stuff.

s234
00:19:51.600 --> 00:19:59.440
Here is the example we're going to show you because this is kind of like layered, you know, like an onion or an ogre.

s235
00:19:59.440 --> 00:20:05.760
We're going to show you an example hardness being run on an example use case.

s236
00:20:05.760 --> 00:20:11.200
If you want to run that command at the bottom, you will see kind of like a more fleshed out example.

s237
00:20:11.200 --> 00:20:21.280
But if you stay on the just running the second thing and just use the UI, it may feel weird that I'm asking you to use an UI to see how a hardness behaves,

s238
00:20:21.280 --> 00:20:22.400
but that's the point.

s239
00:20:22.400 --> 00:20:26.880
The point is to show you how it feels in should feel in the real world.

s240
00:20:26.880 --> 00:20:29.600
Um, should I leave it here for a moment?

s241
00:20:29.600 --> 00:20:30.320
Yeah, I think so.

s242
00:20:30.320 --> 00:20:30.559
Yeah.

s243
00:20:30.559 --> 00:20:31.280
Okay.

s244
00:20:31.280 --> 00:20:31.600
Okay.

s245
00:20:31.600 --> 00:20:33.280
I hear less typing.

s246
00:20:33.280 --> 00:20:37.600
I'm going to assume you've typed whichever command you wanted to type.

s247
00:20:37.600 --> 00:20:45.280
Erh, so let's go through the actual steps and the principles of the hardness and the share domain is the first one.

s248
00:20:45.280 --> 00:20:52.880
Can I see little show of hands of who opened the UI and hasn't done anything else with it?

s249
00:20:53.520 --> 00:20:54.960
Cool, cool, cool, cool, cool.

s250
00:20:54.960 --> 00:21:03.600
All right, so um long story short, you have a chat window on your right right hand side and it has nothing essentially.

s251
00:21:03.600 --> 00:21:11.360
Erh, if you want to follow through an example of a financial worry, some analysis you want to do, that's the place to start.

s252
00:21:11.360 --> 00:21:18.799
And basically, we want to walk from the goal we want to achieve and just try our best to give an explanation.

s253
00:21:18.799 --> 00:21:29.360
You're probably doing this daily with your agents anyway, but focus on understanding that you're likely not going to have the same context as the agent.

s254
00:21:29.360 --> 00:21:33.039
Um, wait, can I go back and forward just in case?

s255
00:21:33.039 --> 00:21:33.120
Okay.

s256
00:21:33.120 --> 00:21:33.280
Yeah.

s257
00:21:33.280 --> 00:21:33.520
Yeah.

s258
00:21:33.520 --> 00:21:34.080
Demo.

s259
00:21:34.080 --> 00:21:48.640
Um, so I wanted to explain something and I think most people don't realize the fact that what the model knows is not necessarily what you think.

s260
00:21:48.640 --> 00:21:58.480
So unless you're very rich and you have a lot of models trained for you, h you're like likely your definition of certain things are not going to be the exact

s261
00:21:58.480 --> 00:22:02.400
as what the majority of the population defines.

s262
00:22:02.400 --> 00:22:13.440
So if you're looking up for a certain term, the agent may the model may be smart enough to know well this sits in this space but it's not necessarily the thing that you meant.

s263
00:22:13.440 --> 00:22:18.080
So going back and forth with your uh model really makes a difference.

s264
00:22:18.080 --> 00:22:28.159
And the hardness is supposed to encourage you if you're trying it out yourself to dig deeper to challenge some of your assumptions to ask you questions you may not knew

s265
00:22:28.159 --> 00:22:30.799
you needed to to ask yourself.

s266
00:22:30.799 --> 00:22:34.000
So if you want to do Yeah.

s267
00:22:37.679 --> 00:22:47.760
So let me do the same thing that you folks are doing, but I'm going to do harness example.

s268
00:22:53.120 --> 00:22:57.200
Let me go back here for a second.

s269
00:22:57.200 --> 00:23:20.720
Just make sure I have everything Is anyone completely lost or are good?

s270
00:23:20.720 --> 00:23:23.600
Okay, maybe could you give him a hand

s271
00:23:23.600 --> 00:23:26.159
while I I I run everything.

s272
00:23:26.159 --> 00:23:43.120
And for the folks that arrived later, I will give you the QR code as well for the so if anyone lost that you can you can pick it up again.

s273
00:23:43.600 --> 00:23:44.320
Okay.

s274
00:23:44.320 --> 00:23:49.039
Um let me show you two things while I while we we let you go with it.

s275
00:23:49.039 --> 00:23:55.200
So this is what you should be seeing when you run with a wedding example, right?

s276
00:23:55.200 --> 00:24:00.000
Um we're we're going to explain in a bit um what's going on here.

s277
00:24:00.000 --> 00:24:11.840
If you are choosing just to show you as well, if you are choosing to uh pick your own pick your own direction, um you're going to see something like

s278
00:24:11.840 --> 00:24:14.000
this where it's empty.

s279
00:24:14.000 --> 00:24:23.200
And then what you can do is as Francesca is going to run through the wedding example, you can say something you want to to use a harness for as well.

s280
00:24:23.200 --> 00:24:31.600
So things like I don't know plan your own wedding, plan a trip, um plan a new investment in your company or in your personal life, doesn't matter.

s281
00:24:31.600 --> 00:24:38.480
Um anything that relates to financial the financial life that we can um implement the same principles.

s282
00:24:38.480 --> 00:24:41.919
If you ever need uh again just opening the hood a bit.

s283
00:24:41.919 --> 00:24:52.960
If you ever need to switch between one and another like the the wedding example or your own um you can just delete the in the my and var folder

s284
00:24:52.960 --> 00:24:56.080
you're going to have a database that's basically what's storing this.

s285
00:24:56.080 --> 00:24:59.679
So you can just delete it.

s286
00:24:59.679 --> 00:25:01.679
Oops.

s287
00:25:03.360 --> 00:25:10.000
And then once you open the the UI again with the previous command, you're going to get the example.

s288
00:25:10.000 --> 00:25:12.000
Do you want to open the clean one?

s289
00:25:12.000 --> 00:25:12.960
Yes, I can.

s290
00:25:12.960 --> 00:25:13.200
Sure.

s291
00:25:13.200 --> 00:25:13.520
Yeah.

s292
00:25:13.520 --> 00:25:20.000
So, appreciating that you're literally you're literally being given uh a blank canvas.

s293
00:25:20.000 --> 00:25:21.919
We want to disambiguate a little bit.

s294
00:25:21.919 --> 00:25:27.120
So, if you open the wedding example, that's a work through example that has all the elements.

s295
00:25:27.120 --> 00:25:30.720
But let's say this is where we're starting, right?

s296
00:25:30.720 --> 00:25:32.159
And I have a problem.

s297
00:25:32.159 --> 00:25:34.000
Do you have a dictation thing there?

s298
00:25:34.000 --> 00:25:35.039
Um, I do.

s299
00:25:35.039 --> 00:25:36.159
Um,

s300
00:25:36.159 --> 00:25:37.679
okay.

s301
00:25:37.679 --> 00:25:38.720
Let's see if this is working.

s302
00:25:38.720 --> 00:25:40.799
Wait a second.

s303
00:25:40.799 --> 00:25:49.679
So, basically on that like chat window on the uh right hand side, you are it's a familiar interface, right?

s304
00:25:49.679 --> 00:25:54.880
So, how would you go about getting some help from an AI agent for whatever specific problem?

s305
00:25:54.880 --> 00:25:55.760
Just start with a chat.

s306
00:25:55.760 --> 00:25:59.760
You start with a a problem and you start to describe what you want to achieve.

s307
00:25:59.760 --> 00:26:08.960
So, if I were to do it, I'll say I have a wedding and I need to make three payments.

s308
00:26:08.960 --> 00:26:16.960
One for $6,000, another one for an unknown amount, uh, and another one for the rest of the money.

s309
00:26:16.960 --> 00:26:20.960
And also, uh, some of my costs are fixed.

s310
00:26:20.960 --> 00:26:31.120
I know what I'm going to spend on a DJ and I don't know decorations, but I don't know how many people are going to attend the wedding.

s311
00:26:31.120 --> 00:26:36.559
And that's the amount of money I'm going to pay per h per dinner per seat.

s312
00:26:36.559 --> 00:26:42.159
So, I don't even know how to go about making sure that I can pay in time in time.

s313
00:26:42.159 --> 00:26:44.240
On time, on time on time.

s314
00:26:44.240 --> 00:26:48.400
Um, I'll say let me just cut the first bit.

s315
00:26:48.400 --> 00:26:50.000
Cool.

s316
00:26:50.000 --> 00:27:02.240
So if you want to run it for a problem that you actually care a little bit more about uh you will start there and all I'm doing is planning with the agent.

s317
00:27:02.240 --> 00:27:13.440
Most of the harnesses today come with some notion that triages that you're trying to plan for a problem and it will try to come up with a reasonable idea some set of questions to give some more clarity.

s318
00:27:13.440 --> 00:27:23.440
The harness itself is what guides the sort of questions that you will get asked and the sort of rigor that the plan needs to have before implementing anything.

s319
00:27:23.440 --> 00:27:38.960
The idea is that it doesn't go immediately into a yes man mode and just implements whatever crazy idea you came up with that is very unclear but that helps you drill down and dig into what it needs to do.

s320
00:27:38.960 --> 00:27:44.159
So that's the principle of getting a shared context.

s321
00:27:44.480 --> 00:27:46.240
Quick one before you you continue.

s322
00:27:46.240 --> 00:27:48.080
Is the size okay for the back of the room?

s323
00:27:48.080 --> 00:27:49.840
The size of the text.

s324
00:27:49.840 --> 00:27:50.320
Yes.

s325
00:27:50.320 --> 00:27:50.640
Okay.

s326
00:27:50.640 --> 00:27:50.960
Cool.

s327
00:27:50.960 --> 00:27:53.120
Thank you.

s328
00:27:54.640 --> 00:27:54.960
Yeah.

s329
00:27:54.960 --> 00:27:58.960
Sorry, I'm reading because this is like live.

s330
00:27:59.840 --> 00:28:00.240
Okay.

s331
00:28:00.240 --> 00:28:02.799
From the one I have.

s332
00:28:02.799 --> 00:28:03.279
Okay.

s333
00:28:03.279 --> 00:28:07.120
So maybe I can keep adding clarifying information here.

s334
00:28:07.120 --> 00:28:13.440
And I'm going to skip ahead, but something like I have no savings.

s335
00:28:13.440 --> 00:28:16.080
It will come from my pocket.

s336
00:28:16.080 --> 00:28:19.600
My mom may give me like 5k.

s337
00:28:19.600 --> 00:28:38.240
Uh, I know the price per a person going to the wedding is around 150 and I'm paying 500 for the for the flowers.

s338
00:28:38.240 --> 00:28:41.120
I need to buy my suit.

s339
00:28:41.520 --> 00:28:53.039
I'm just adding a bunch of like complexity and being disorganized on purpose because the point of the agent is that it doesn't let go the fact that you're being vague

s340
00:28:53.039 --> 00:28:57.279
um in not providing the information ahead of time.

s341
00:28:57.279 --> 00:29:08.880
The whole point of this is that it's really guardrail for you to for to forcing yourself to actually go through um every single data point that it needs to come up with a plan.

s342
00:29:09.360 --> 00:29:14.960
I think for the next one we just build whatever comes up to try and move on.

s343
00:29:14.960 --> 00:29:15.600
Propose the plan.

s344
00:29:15.600 --> 00:29:16.000
Now,

s345
00:29:16.000 --> 00:29:22.320
is anyone trying it out with a problem you have personally?

s346
00:29:23.360 --> 00:29:23.760
Yes.

s347
00:29:23.760 --> 00:29:25.120
Okay, cool.

s348
00:29:25.120 --> 00:29:28.640
Do you want to tell us a little bit about it?

s349
00:29:30.080 --> 00:29:31.120
Oh, fair, fair, fair.

s350
00:29:31.120 --> 00:29:33.120
Nice.

s351
00:29:37.120 --> 00:29:37.440
Okay.

s352
00:29:37.440 --> 00:29:39.039
It's asking a lot of questions and I'm

s353
00:29:39.039 --> 00:29:39.440
No. Yeah.

s354
00:29:39.440 --> 00:29:41.279
I I accepted the plan.

s355
00:29:41.279 --> 00:29:41.679
Yeah.

s356
00:29:41.679 --> 00:29:46.000
So, can you go back to the seated example?

s357
00:29:46.000 --> 00:29:47.279
Yes, I can.

s358
00:29:47.279 --> 00:29:48.480
Um,

s359
00:29:48.480 --> 00:29:54.799
so to save you a lot of the back and forth back and forth here because it's not very high signal for any of you.

s360
00:29:54.799 --> 00:29:56.880
There's a couple of takeaways.

s361
00:29:56.880 --> 00:30:09.760
One is having that self-awareness and forcing the hardness to remember that your domain knowledge and whatever the agent knows is not going to be the same.

s362
00:30:09.760 --> 00:30:15.039
You want to find that kind of like vin diagram superposition as much as you can.

s363
00:30:15.039 --> 00:30:20.480
H but it takes a lot of back and forth and that's often where most of the time is spent.

s364
00:30:20.480 --> 00:30:33.679
If you design a really good harness, it will guide you by the hand on coming up with a really good plan, implementing things well, uh, and knowing what components are important and which ones aren't.

s365
00:30:33.679 --> 00:30:42.480
Uh, if your harness is basically like, yeah, say yes to everything I say and do your best to implement it, then you're going to get variable results.

s366
00:30:42.480 --> 00:30:44.799
So, this is the seated example.

s367
00:30:44.799 --> 00:30:53.039
The plan accepted thing at the top means essentially that I already went through the back and forth for around 10 minutes or whatever.

s368
00:30:53.039 --> 00:30:58.799
Uh and the agent feel it knows enough about my problem to break it down in steps.

s369
00:30:58.799 --> 00:31:14.480
So what you have in the diagram over there is the all the steps that it considers as decision points in a decision tree if you may to actually make a decision

s370
00:31:14.480 --> 00:31:26.960
and if I if I may um I think it's it's as we said in the beginning it's nice to look under the hood a bit just to give you an idea and you can of course explore this by yourself or with your preferred agents

s371
00:31:26.960 --> 00:31:27.760
help.

s372
00:31:27.760 --> 00:31:40.159
Um but when we are when we are talking about everything that Franchesco is describing um this also comes from the instructions that we have inside the harness right so

s373
00:31:40.159 --> 00:31:55.360
this is where we have the instructions of the agent performing the interview and what it must end up with so this is the materialization of the principle built beyond the harness right like built inside the harness I mean

s374
00:31:55.360 --> 00:31:58.960
if you want to go back to um to the UI.

s375
00:31:58.960 --> 00:32:04.320
So this is hopefully going to start making more sense as we as we progress in the example.

s376
00:32:04.320 --> 00:32:17.679
But we added a couple of quality of life things where if you want to know what elements of this whole UI that is so beautifully designed h makes sense for whatever

s377
00:32:17.679 --> 00:32:19.519
uh principle we're discussing.

s378
00:32:19.519 --> 00:32:23.039
You just need to hover down at the bottom and it will be broken down.

s379
00:32:23.039 --> 00:32:39.600
So you see when we did the planning the planning the sharing the domain context it came up with a plan is highlighted at the top on the right and then it came up with all the nodes and the connections in between the other decisions

s380
00:32:39.600 --> 00:32:42.720
and what we need to start scoping from here.

s381
00:32:42.720 --> 00:32:46.720
So should we move on?

s382
00:32:46.720 --> 00:32:48.320
Yes.

s383
00:32:48.320 --> 00:32:53.360
Is anyone still doing any planning with the chat?

s384
00:32:56.080 --> 00:32:58.080
Cool.

s385
00:33:05.120 --> 00:33:05.360
Okay.

s386
00:33:05.360 --> 00:33:05.679
Yeah.

s387
00:33:05.679 --> 00:33:07.120
So, we had a few things here.

s388
00:33:07.120 --> 00:33:21.360
If you wanted to uh play around and peruse the uh harness yourself, you could for instance go on the step three in the UI like in the little diagram on the

s389
00:33:21.360 --> 00:33:22.320
left hand side.

s390
00:33:22.320 --> 00:33:23.360
Oops, sorry.

s391
00:33:23.360 --> 00:33:24.000
I was going to show

s392
00:33:24.000 --> 00:33:25.440
too many tabs.

s393
00:33:25.440 --> 00:33:27.919
You could click on like step no

s394
00:33:27.919 --> 00:33:30.000
step three in the Yeah.

s395
00:33:30.000 --> 00:33:41.840
And it will give you a little bit of a breakdown of where the information came from where why the model thinks this is the node you need to make a decision of

s396
00:33:41.840 --> 00:33:49.360
uh including if it's making any assumptions based on the language you were using during your planning with it.

s397
00:33:49.360 --> 00:33:54.399
So for instance the agent did you say this I could point with this

s398
00:33:54.399 --> 00:33:57.440
uh if you press and hold that yeah

s399
00:33:57.440 --> 00:33:58.559
awesome

s400
00:33:58.559 --> 00:34:00.000
not AI

s401
00:34:00.000 --> 00:34:10.800
so there's there's quite a few things in the plan h that the agent put together that were proposed by it so you can start dissecting a little bit and understanding like

s402
00:34:10.800 --> 00:34:17.280
actually this came from me this is a fact that I think I know or this is something the agent inferred based on my language.

s403
00:34:17.280 --> 00:34:20.560
If you scroll up a little bit just on the same Yeah.

s404
00:34:20.560 --> 00:34:24.560
more more more more there.

s405
00:34:24.560 --> 00:34:25.440
Yeah, sure.

s406
00:34:25.440 --> 00:34:37.839
Uh here there was some back and forth because I was being vague with the language I was using and it made an assumption that because I was using this keyword here,

s407
00:34:37.839 --> 00:34:39.599
I meant this.

s408
00:34:39.599 --> 00:34:45.599
Again, this is a pretty obvious example, but you can play around with clicking through, seeing things around.

s409
00:34:45.599 --> 00:34:58.000
If we go back to the presentation, there's some annotation because again, this is all about grounding the principles into something that feels like it was designed for building trust.

s410
00:34:58.000 --> 00:34:59.520
So you have some notation.

s411
00:34:59.520 --> 00:35:09.440
Again, not the best UI design in the world, but knowing if things came from you, from the agent, or is it still an open point and you need to make a decision on

s412
00:35:09.440 --> 00:35:13.599
and you can tell how the plan has been built.

s413
00:35:13.599 --> 00:35:15.680
You have another thing you can try.

s414
00:35:15.680 --> 00:35:23.040
You can go back into step two uh and actually tell the agent if we click on the node there.

s415
00:35:23.040 --> 00:35:24.160
Yeah.

s416
00:35:24.160 --> 00:35:36.000
And you can actually tell the agent after you have sorry uh you can just tell the agent like well actually you're making some assumptions about this step that are not correct

s417
00:35:36.000 --> 00:35:39.040
and then go a little bit back and forth.

s418
00:35:39.040 --> 00:35:57.119
Erh, I will still do the show of hands thing, but because of reinforcement and engagement, uh, who has been in a chat session with over like well over 10 turns, so back and forth and back and forth at least 10 times

s419
00:35:57.119 --> 00:36:02.320
where at some point you said, let let me start from the beginning like this is not making any sense.

s420
00:36:02.320 --> 00:36:06.000
Some stuff we already discussed like it's well gone.

s421
00:36:06.000 --> 00:36:10.640
Who has had that feeling on a regular basis?

s422
00:36:10.640 --> 00:36:11.040
Yeah.

s423
00:36:11.040 --> 00:36:23.680
So, that is a consequence of the design of the harnesses we have today where they're designed for a linear discussion and to keep everything in the context as one

s424
00:36:23.680 --> 00:36:25.760
big conversation.

s425
00:36:25.760 --> 00:36:38.480
Whereas here the way we designed it is supposed to break it down so you can discuss individual elements without necessarily making things you know more complicated in the grander scheme of things.

s426
00:36:38.480 --> 00:36:40.480
Yeah.

s427
00:36:41.599 --> 00:36:43.839
Erh cool.

s428
00:36:43.839 --> 00:36:49.599
So I will let you play around with it a little bit but I do want to maybe go a little bit further.

s429
00:36:49.599 --> 00:37:07.200
So I will introduce the consistency principle and this build on the shared domain erh because you already have a plan and the agent is very capable to understand well these pieces of your plan

s430
00:37:07.200 --> 00:37:17.680
are just a piece of code or just a calculation spreadsheet like it will always give you a consistent result is a deterministic element of your whole plan.

s431
00:37:17.680 --> 00:37:22.240
Erh and based on that you need to implement code.

s432
00:37:22.240 --> 00:37:25.760
You need to implement something that will give you the same result every time.

s433
00:37:25.760 --> 00:37:28.320
Erh similar show of hands thing.

s434
00:37:28.320 --> 00:37:38.880
Has anyone tried to do like budgeting or any financial calculation in the chat and how did that go?

s435
00:37:38.880 --> 00:37:41.280
Like are you confident on the results?

s436
00:37:41.280 --> 00:37:42.720
You Yeah, a little bit.

s437
00:37:42.720 --> 00:37:47.280
Temperature maybe like very well um not too well.

s438
00:37:47.280 --> 00:37:49.520
pretty bad.

s439
00:37:49.520 --> 00:37:51.040
Okay.

s440
00:37:51.040 --> 00:37:53.280
So, so all right.

s441
00:37:53.280 --> 00:38:01.760
So, I would love to know if anyone wants to volunteer their story, but what they saw and why they think it happened.

s442
00:38:01.760 --> 00:38:08.480
But to me, a couple of times I thought I was giving all the necessary information.

s443
00:38:08.480 --> 00:38:10.960
The the agent gave me back a formula.

s444
00:38:10.960 --> 00:38:12.880
It was even formatted as a formula.

s445
00:38:12.880 --> 00:38:16.640
You know, the kind of like italics formula thing that looks really nice.

s446
00:38:16.640 --> 00:38:19.200
I was like knows what he's talking about.

s447
00:38:19.200 --> 00:38:24.240
And then the result was like I'm not a calculator and I know that result is wrong.

s448
00:38:24.240 --> 00:38:27.119
So from the get-go I was like nah start from scratch.

s449
00:38:27.119 --> 00:38:29.920
H anyone has any other like examples

s450
00:38:29.920 --> 00:38:30.880
consistency right?

s451
00:38:30.880 --> 00:38:39.680
So a lot of times the results look good reasonable but you run the same thing again that's something.

s452
00:38:45.040 --> 00:38:45.599
Yeah.

s453
00:38:45.599 --> 00:38:48.079
So that's basically the inspiration on the principle.

s454
00:38:48.079 --> 00:38:49.440
Sorry, what do you want to share?

s455
00:38:49.440 --> 00:38:49.680
Yeah.

s456
00:38:49.680 --> 00:38:59.839
So it goes to the point that every time it might

s457
00:39:07.359 --> 00:39:08.640
Yeah.

s458
00:39:08.640 --> 00:39:22.400
Uh and and that's a great a great um example to say part of the benefit why we are talking about building a harness and not just plugging these principles in a set of skills or instructions

s459
00:39:22.400 --> 00:39:34.320
to an existing harness such as like codeex cloud code or anything else like when we when we do it the non-deterministic way which is to give the agent a bunch of instructions that

s460
00:39:34.320 --> 00:39:38.320
um say the right things but still rely on the agent agent.

s461
00:39:38.320 --> 00:39:41.440
Uh, make no mistakes, please do the right thing.

s462
00:39:41.440 --> 00:39:42.960
Um, that's that's the point.

s463
00:39:42.960 --> 00:39:49.359
We're relying on a bunch of things happening in the background and basically getting a nondeterministic result every time.

s464
00:39:49.359 --> 00:40:02.320
So, I think the value here from building the harness again, this is looking a bit into the under the hood and like from principles we we applied at at Monzo as well is mixing the non-determinism

s465
00:40:02.320 --> 00:40:06.720
with some deterministic behavior that we want to control.

s466
00:40:06.720 --> 00:40:09.839
um when when relevant.

s467
00:40:10.160 --> 00:40:24.079
I think you mentioned that for this demo a lot of this harness was quite I get this point right for production stuff like what's the difference between you making calculations

s468
00:40:24.079 --> 00:40:30.560
in in chat versus you asking the model to code up calculations for you in code.

s469
00:40:30.560 --> 00:40:42.400
So I have a couple of personal examples on doing forecast where you can try and do forecast yourself by hand or come up with a spreadsheet with a lot of fancy

s470
00:40:42.400 --> 00:40:47.760
um calculations and you will likely get a few things right and get a decent forecast.

s471
00:40:47.760 --> 00:40:54.079
You can also just get a library from a trusted uh repo and that has everything baked in.

s472
00:40:54.079 --> 00:41:05.440
You can also write some code in a more modular fashion that is well unit tested, you're doing end to end testing, you're doing all the like good software um engineering practices

s473
00:41:05.440 --> 00:41:11.599
that we know how to do to ensure that that piece of code is going to give you a good result.

s474
00:41:11.599 --> 00:41:15.520
Um, so I think the different Oh, sorry.

s475
00:41:15.520 --> 00:41:31.280
The difference lies a little bit on the fact that if you're just leaving it to the agent to decide how to implement things and when they need to do it in code versus when it like you should just build something on the side,

s476
00:41:31.280 --> 00:41:36.319
you're going to have a little bit of a big miss ratio like hit or miss.

s477
00:41:36.319 --> 00:41:36.480
Yeah.

s478
00:41:36.480 --> 00:41:36.640
Yeah.

s479
00:41:36.640 --> 00:41:42.000
I I guess what I'm asking is if this is why quoted no good engineering practices were applied.

s480
00:41:42.000 --> 00:41:42.960
I get this part.

s481
00:41:42.960 --> 00:41:46.800
Do you have like a product that that was tested?

s482
00:41:46.800 --> 00:41:50.880
We we I'm going to do the the uh ad again.

s483
00:41:50.880 --> 00:41:59.359
Uh tomorrow we're going to have a breakout session in the morning where we're going to talk about uh what we build at Monzo that applies some of these principles.

s484
00:41:59.359 --> 00:42:10.319
uh but yeah I think the what is behind when we say like the harness being voded I think it's more about the implementation detail of like abstractions methods the actual code

s485
00:42:10.319 --> 00:42:18.240
uh but the principles of mixing the determinism and the non-determinism are the same regardless if that makes sense

s486
00:42:18.240 --> 00:42:20.480
thank you

s487
00:42:21.280 --> 00:42:42.400
uh and another thing another thing that signals that this is a a good principle to consider on over the last what year or so most of the agents have started to do more aggressive tool calling and building their own tools to solve calculations in a more deterministic way

s488
00:42:42.400 --> 00:42:54.079
and honestly agents are just getting better right so we could just wait and then all the problems are solved the one challenge I have with that mindset is that h it will create stuff

s489
00:42:54.079 --> 00:43:03.760
on the on the go like in flight so Yes, you may get the right code for the right problem at an at a point in time, but maybe the

s490
00:43:03.760 --> 00:43:10.400
time number 100, you run it, turns out is inventing a new piece of code that didn't need to be reinvented.

s491
00:43:10.400 --> 00:43:27.920
uh and other examples non-workrelated that I can share err are around like doing exactly that like I have a I don't want to say strict but semi-strict close of budget every month and I categorize everything in my statements

s492
00:43:27.920 --> 00:43:44.400
I left it to the agent to decide how to do the categorization and how to do the aggregates and everything else and the refunds and expenses and at some point I was like no no no this needs to be a script because I'm gettinglight slightly different results every time and I'm spending more money more money and time

s493
00:43:44.400 --> 00:43:53.839
uh wrangling with the agent to get it to give me the results that I wanted that it got right three times already but didn't get it right in the fourth

s494
00:43:53.839 --> 00:44:05.440
uh so I don't want to repeat it again but make sure that we run the calculations you write the calculation once and then you run it every time you need it

s495
00:44:05.440 --> 00:44:16.800
um and the model is able to pick the tools is you just need to make sure that your hardness makes it very strict that it's not going to do any arithmetic

s496
00:44:16.800 --> 00:44:21.680
or try to solve these problems by just parsing tokens.

s497
00:44:21.680 --> 00:44:33.359
Um so what can happen if you don't make sure that your calculations and everything that is deterministic runs in code they're in the thickest in the model's head.

s498
00:44:33.359 --> 00:44:44.720
So it's gone once you start new session the new code for every new question and or new session and it becomes a mess and same question different answer non-determination

s499
00:44:44.720 --> 00:44:50.720
again you're asking the stoastic machine to produce a consistent output so you're likely not going to get it.

s500
00:44:50.720 --> 00:44:59.920
Um, so here's some examples of things that have a way to be calculated, right?

s501
00:44:59.920 --> 00:45:11.200
Like I'm sure if you ask 10 different financial services companies, they will have a slightly different way of doing these things, but the general formula, the general idea is the same.

s502
00:45:11.200 --> 00:45:18.400
So you could realistically export a library, write it once, update it, and these things are not new.

s503
00:45:18.400 --> 00:45:22.160
they saw from like software good practices.

s504
00:45:22.160 --> 00:45:34.800
So demo again if we just want to go to the seated example h in the can you highlight this?

s505
00:45:34.800 --> 00:45:36.800
Okay.

s506
00:45:37.040 --> 00:45:55.280
So again just using this h you can see how the nodes in your decision making process um some of them needed to be implemented as code and I think the one of the easiest way to check is if it's a calculation likely it has to be implemented as code right

s507
00:45:55.280 --> 00:46:10.960
uh as well as some testing unit testing golden examples etc. the you should build trust with your agent when you're building a harness like this by making sure that the information you care about about its implementation

s508
00:46:10.960 --> 00:46:12.400
it's in front of you.

s509
00:46:12.400 --> 00:46:18.560
Another example of how this has looked in the wild is giving you versions of the code.

s510
00:46:18.560 --> 00:46:21.440
Your your code is more likely versioned.

s511
00:46:21.440 --> 00:46:32.480
If anything change with the version of the code and you're trying to run the same process, the agent should be able to pick it up and surface that information to you and say actually this code change you probably

s512
00:46:32.480 --> 00:46:41.760
even though the function is the same you probably shouldn't trust uh what's coming out of this because the whatever calculation h is different now.

s513
00:46:41.760 --> 00:46:56.720
So as you're working with the with the harness and you accepted the plan, you have a nice graph like this being built, you will get something at the top saying, do you want me to build the code?

s514
00:46:56.720 --> 00:46:59.040
H and it will do it for you.

s515
00:46:59.040 --> 00:47:00.480
And again, it's all guided.

s516
00:47:00.480 --> 00:47:04.400
That's the point of a hardness that it guides things.

s517
00:47:04.400 --> 00:47:12.960
Erh, is anyone building any of the steps on their plan?

s518
00:47:13.119 --> 00:47:14.400
Is that okay?

s519
00:47:14.400 --> 00:47:14.960
Yeah.

s520
00:47:14.960 --> 00:47:15.280
Yeah.

s521
00:47:15.280 --> 00:47:17.280
Cool.

s522
00:47:17.839 --> 00:47:19.520
I switch to focus.

s523
00:47:19.520 --> 00:47:20.000
Okay.

s524
00:47:20.000 --> 00:47:20.767
Yeah.

s525
00:47:20.767 --> 00:47:21.680
[laughter]

s526
00:47:21.680 --> 00:47:22.319
Fair enough.

s527
00:47:22.319 --> 00:47:22.640
Fair.

s528
00:47:22.640 --> 00:47:24.640
Fair.

s529
00:47:28.480 --> 00:47:28.960
Cool.

s530
00:47:28.960 --> 00:47:34.960
So if you're following on the wedding example, you can try a few things.

s531
00:47:34.960 --> 00:47:42.319
Click on step one, then code, and then the test and run the tests now.

s532
00:47:47.440 --> 00:47:48.720
Tests here.

s533
00:47:48.720 --> 00:47:50.720
Yeah.

s534
00:47:52.560 --> 00:47:54.560
95.

s535
00:47:55.520 --> 00:47:56.880
Sorry, what time is it?

s536
00:47:56.880 --> 00:47:58.103
9:54.

s537
00:47:58.103 --> 00:47:58.319
[laughter]

s538
00:47:58.319 --> 00:47:59.359
Okay, good.

s539
00:47:59.359 --> 00:48:10.800
Um, but yeah, the the the principle here and just going back to try and tie everything up together, you shouldn't rely on you remembering to run the test.

s540
00:48:10.800 --> 00:48:19.359
You shouldn't rely on you remembering that, hey, this this sounds like I can h create some code to solve it every time.

s541
00:48:19.359 --> 00:48:21.200
The agent should guide you through it.

s542
00:48:21.200 --> 00:48:25.040
And that's one of the principles we baked into building this.

s543
00:48:25.040 --> 00:48:35.119
Uh, as an aside, I think we were telling you, uh, there's a little extra file in the repo that says build yours or something.

s544
00:48:35.119 --> 00:48:41.988
Um, it essentially has all the principles baked in in order and it will guide you on building a

s545
00:48:41.988 --> 00:48:42.000
[snorts]

s546
00:48:42.000 --> 00:48:45.119
similar hardness.

s547
00:48:45.119 --> 00:48:48.880
It's built your own.

s548
00:48:48.880 --> 00:48:51.200
There you go.

s549
00:48:52.720 --> 00:48:53.119
Yeah.

s550
00:48:53.119 --> 00:49:06.000
So yeah, it will guide you on building something that feels similar to the harness we're showing you that prioritizes trust building mechanisms over just giving you something that looks correct.

s551
00:49:06.000 --> 00:49:07.280
So you can try this.

s552
00:49:07.280 --> 00:49:07.534
The

s553
00:49:07.534 --> 00:49:07.599
[snorts]

s554
00:49:07.599 --> 00:49:09.280
test should be passing.

s555
00:49:09.280 --> 00:49:19.760
You will notice that there was no thinking, perusing, all the verbs that agents put when they're loading something.

s556
00:49:19.760 --> 00:49:20.960
Yeah.

s557
00:49:20.960 --> 00:49:24.559
because these are just tests that need to run in your environment.

s558
00:49:24.559 --> 00:49:27.520
Don't overengineer it.

s559
00:49:28.000 --> 00:49:41.200
Funny that I'm saying that, but it's easier if all the components of your harness just work together as they should and not based on like hopes and dreams and magic glue from an agent orchestrating

s560
00:49:41.200 --> 00:49:42.079
things together.

s561
00:49:42.079 --> 00:49:44.960
I think we had a question.

s562
00:49:58.880 --> 00:50:07.359
Erh, so if we Yeah, I mean we can do that, but I'd rather not do a reading session.

s563
00:50:07.359 --> 00:50:16.240
Basically, it it all goes about h figuring out what you actually care about and what are the functional requirements for a functioning harness.

s564
00:50:16.240 --> 00:50:19.280
H I'm sprinkling some of that through the conversation.

s565
00:50:19.280 --> 00:50:23.680
Again, need to avoid oversharing a little bit because of IP.

s566
00:50:23.680 --> 00:50:35.680
Uh but you want to make sure that the objectives of each of the steps when you're building the parts that will make up the behavior of your hardness have an intent.

s567
00:50:35.680 --> 00:50:41.920
The intent of one of the first components is gathering as much information as possible to come up with a plan.

s568
00:50:41.920 --> 00:50:44.400
And that is written in here as well.

s569
00:50:44.400 --> 00:50:46.960
So this is basically instructions for instructions.

s570
00:50:46.960 --> 00:50:49.839
It's the IKEA guide.

s571
00:50:49.839 --> 00:51:01.839
Um it is supposed to guide you very nicely on these are the things you should care about but but then you can pick uh your preference on how they get implemented.

s572
00:51:03.760 --> 00:51:04.800
Question.

s573
00:51:04.800 --> 00:51:18.400
So how is this different from let's say the specdriven development expected something like that which is also linear from starting from all the questions from you starting from idea all the way to the end

s574
00:51:18.400 --> 00:51:22.000
and how is it different to this one is different to that

s575
00:51:22.000 --> 00:51:30.160
do you mind if I come to that um I I want to mention two things that I think are are relevant here um I think the first one is

s576
00:51:30.160 --> 00:51:39.760
when we're building a harness for a specific type of use case it's a bit different from the coding assistant world of agents because with the coding assistance we usually

s577
00:51:39.760 --> 00:51:45.359
the code is a means to an end right like we we're building something that performs an action.

s578
00:51:45.359 --> 00:51:48.079
It's a product, is a an app, whatever that is.

s579
00:51:48.079 --> 00:51:54.880
Um, in this case, the agent is helping us like analyze, solve a problem and work us through that.

s580
00:51:54.880 --> 00:52:08.720
So the the gigantic spec for this like let's say for Franchesco's wedding things will change like it will evolve it will change and then if we go back to the everchanging

s581
00:52:08.720 --> 00:52:21.280
spec we are relying again on the determin on the non-determinism of the agent to interpret that that can be very very very um robust and perform this the the correct thing.

s582
00:52:21.280 --> 00:52:30.400
So the harness essentially breaks down what would be the spec in opinionated sections if that makes sense.

s583
00:52:30.400 --> 00:52:37.680
Um and I think the second thing to complement with that and is actually something I wanted to to like ground on our experience from data.

s584
00:52:37.680 --> 00:52:43.359
I don't know how many of you work in data engineering analytics or anything like that.

s585
00:52:43.359 --> 00:52:55.476
I think something else that we bring from this word in these principles is the idea that um metric definitions and dimension definitions always change like in our experience we've like

s586
00:52:55.476 --> 00:52:55.520
[snorts]

s587
00:52:55.520 --> 00:53:02.079
had many many many times and I'm sure many of you as well um that conversation were is this number correct?

s588
00:53:02.079 --> 00:53:02.720
I don't know.

s589
00:53:02.720 --> 00:53:08.160
It sounds like it is, but we're never sure and then if we're never sure, is the number correct?

s590
00:53:08.160 --> 00:53:18.720
Um, I think this is also coming from that perspective is that to the point where we trust the agent as well as we trust a dashboard or a metric or a visualization.

s591
00:53:18.720 --> 00:53:27.119
Um, we've reached alignment with it and the harness can continue guiding us into that alignment continuing to be true over time.

s592
00:53:27.119 --> 00:53:31.200
But premises will change, source data will change, context will change.

s593
00:53:31.200 --> 00:53:46.720
And then having the harness able to adapt to that is like recognizing that this change will be a thing um rather than having to every time pro prompt again and go back to the process.

s594
00:53:46.720 --> 00:53:49.040
Does that make sense?

s595
00:53:49.040 --> 00:53:51.040
Yep.

s596
00:53:54.160 --> 00:53:55.760
Follow question on that.

s597
00:53:55.760 --> 00:54:14.559
any complnt this perspective because most most workable thing you are developing something even though it's subject to change through different iterations you are developing something with an AI agent can be part of this flow as well

s598
00:54:14.559 --> 00:54:20.559
absolutely use that uh this part this harness to do that it

s599
00:54:20.559 --> 00:54:27.520
absolutely and this like under the hood this is essentially what's happening as well.

s600
00:54:27.520 --> 00:54:41.920
So when we have when we're creating um the if you if you look under the hood in the my folder, you're going to have everything that is saved.

s601
00:54:41.920 --> 00:54:46.400
So you can think of this as the spec that's behind the harness.

s602
00:54:46.400 --> 00:54:52.960
So absolutely like the same principle applies here in the in the specdriven development for again coding agents is usually used.

s603
00:54:52.960 --> 00:54:58.000
you have something that is the source of um definition for the thing being built.

s604
00:54:58.000 --> 00:55:00.079
In this case, it's the same thing.

s605
00:55:00.079 --> 00:55:09.680
The difference is that we surface that through opinionated like uh workflows rather than a single um looking document if that makes sense.

s606
00:55:11.359 --> 00:55:12.960
All right, cool.

s607
00:55:12.960 --> 00:55:18.400
So, that's another thing you can try out.

s608
00:55:18.400 --> 00:55:23.280
um you can try and change how a calculation works.

s609
00:55:23.280 --> 00:55:27.440
Uh and you're likely going to get some push back from the harness.

s610
00:55:27.440 --> 00:55:32.880
Just to reiterate, the point of the harness is to guide you and force you to do the right thing.

s611
00:55:32.880 --> 00:55:37.760
And the principles are here to help you make the right decision every time.

s612
00:55:37.760 --> 00:55:49.839
So again if you go try and change uh multiplication for a an aggregation you may just go make the quick edit save the file success you sabot attach the analysis

s613
00:55:49.839 --> 00:56:02.400
but when you run the test you won't be able to actually reach any conclusion of the analysis you wanted to do h because the hardness is going to stop you by its own like definitions and the way it was built.

s614
00:56:02.400 --> 00:56:08.799
Um, should we move on evidence?

s615
00:56:08.799 --> 00:56:18.480
Er, evidence is the whole thing I was saying before about the receipts, having the receipts and being able to show that well, yeah, this is where this came from.

s616
00:56:18.480 --> 00:56:24.160
This is correct because of in doing that causality relationship.

s617
00:56:24.160 --> 00:56:35.680
So it builds on the second principle as since you have code that is tested as well as everything else h you have a trail that you can follow backwards.

s618
00:56:35.680 --> 00:56:50.559
So you have a way to substantiate the decisions you're making and you have the evidence and it's important in this principle because a principle is something you need to interpret right and I don't want to leave you with a very vague

s619
00:56:50.559 --> 00:57:07.280
ambitional thing like oh yeah do evidence it's awesome but what is what about evidence is something that you need to take care of going back to being data professionals this is something that bites us on a weekly basis.

s620
00:57:07.280 --> 00:57:10.799
Erh, not all information is the same.

s621
00:57:10.799 --> 00:57:23.599
If you really want to present a piece of analysis or you went to do some financial analysis on a potential investment for your firm, you need to show where your numbers came from

s622
00:57:23.599 --> 00:57:28.319
and be able to attribute your decision to the body of evidence you had.

s623
00:57:28.319 --> 00:57:41.280
Erh so since we can show where everything came from uh there's some mistakes we want to avoid where everything is treated as a fact.

s624
00:57:41.280 --> 00:57:49.599
So oh yeah this number you gave me for the number of um I don't know going back to my wedding the number of people invited is a fact.

s625
00:57:49.599 --> 00:57:53.440
Turns out my fiance invited three more people didn't tell me.

s626
00:57:53.440 --> 00:57:56.240
Now my whole model is wrong.

s627
00:57:56.240 --> 00:58:08.799
uh and I'm trying to reduce it to a very simple example but I think you can imagine how having a wrong assumption somewhere and not knowing where it came from can be fatal for any financial analysis.

s628
00:58:08.799 --> 00:58:21.680
Um so let's say in the world of a gentic building it's very easy to assume something or treat something as a gueststimate.

s629
00:58:21.680 --> 00:58:27.359
erh the model can't tell what is a hard truth and what is an estimation.

s630
00:58:27.359 --> 00:58:31.040
So you need to enforce that through the design of your hardness.

s631
00:58:31.040 --> 00:58:44.640
Erh, and another example where this kind of like falls fall into pieces really quickly is imagine you have a data person, an analyst, something coming to you and being like, "No, no, no. Here's the result.

s632
00:58:44.640 --> 00:58:46.319
Trust me, I run the numbers."

s633
00:58:46.319 --> 00:58:58.720
Do you trust that as a Yeah, I see some like no. I don't think anyone here will want to see all the logs and the math behind a single number.

s634
00:58:58.720 --> 00:59:08.079
But just having the trace to figure out, okay, I can follow back this decision increased increases confidence for us humans quite a bit.

s635
00:59:08.079 --> 00:59:13.680
Erh, so we have some like different examples here.

s636
00:59:13.680 --> 00:59:18.720
Erh, unstructured thoughts in a chat needs to be labeled as an interpretation.

s637
00:59:18.720 --> 00:59:31.680
the agent reasoning or pulling information from memory needs to be treated as an assumption and then if I submit a bank statement what is there to discuss like that is a fact that's how much I spent

s638
00:59:31.680 --> 00:59:52.160
etc if I take like go back again to being a data person data labeling and making sure that you have a curated data set for either training something or running any process is paramount and this is basically the same thing you just need to assign the right level of

s639
00:59:52.240 --> 00:59:59.839
importance to the evidence you're gathering and make sure that you gather as much as as much of it as you can.

s640
00:59:59.839 --> 01:00:06.720
Um, this is a really funny one, but evidence also goes into the code.

s641
01:00:06.720 --> 01:00:13.280
So, it's not just where do you get this number from, but how did you get this number?

s642
01:00:13.280 --> 01:00:18.720
And I had I have an anecdote where someone was using the master file.

s643
01:00:18.720 --> 01:00:24.400
In every company there's a master file and is favorited in the share drive and whatever.

s644
01:00:24.400 --> 01:00:27.839
And that person puts some numbers in there and everything looked fine.

s645
01:00:27.839 --> 01:00:34.079
The master file had some assumptions of what range of values were reasonable.

s646
01:00:34.079 --> 01:00:36.720
That person didn't know because the file changed.

s647
01:00:36.720 --> 01:00:37.920
They didn't do anything wrong.

s648
01:00:37.920 --> 01:00:39.920
They just didn't know the file changed.

s649
01:00:39.920 --> 01:00:42.319
Uh and then the results were entirely wrong.

s650
01:00:42.319 --> 01:00:47.760
And the same will happen with deterministic code you build with your harness.

s651
01:00:47.760 --> 01:00:59.040
If you don't have a way to having a traceability even for changes you make in your code, you're leaving yourself uh vulnerable for blind spots on the process change.

s652
01:00:59.040 --> 01:01:01.920
The inputs were the same.

s653
01:01:01.920 --> 01:01:05.839
Erh skipping ahead a bit.

s654
01:01:08.799 --> 01:01:17.200
So if we're following from the example, can you open any of them?

s655
01:01:17.200 --> 01:01:19.200
Yes.

s656
01:01:19.520 --> 01:01:21.920
The evidence is highlighted.

s657
01:01:21.920 --> 01:01:34.799
Erh, and again, you can see like the areas that you care about through the example by just like clicking on the bottom and then seeing the bits that get highlighted.

s658
01:01:34.799 --> 01:01:43.680
But this is how the flow of information goes through and what parts are proposed by an assistant like you can show the receipts of where where this came from.

s659
01:01:43.680 --> 01:02:00.559
Looking at other implementations of financial analysts in the industry you get traceability down to the data set like you know which uh connector or which supplier of data gave you the information that whatever Apple

s660
01:02:00.559 --> 01:02:02.960
had this price at this point of the day.

s661
01:02:02.960 --> 01:02:07.359
if you're doing something about trading or investing in stocks and shares.

s662
01:02:07.359 --> 01:02:09.440
So, this is extremely important.

s663
01:02:09.440 --> 01:02:15.440
I don't think we need to spend any more time here unless anyone has any questions.

s664
01:02:16.000 --> 01:02:18.000
Oh, good.

s665
01:02:18.000 --> 01:02:20.000
Cool.

s666
01:02:23.920 --> 01:02:26.319
So, here's a few things you can try.

s667
01:02:26.319 --> 01:02:33.200
Um, so you can add a little bit of a a curveball to the agent.

s668
01:02:33.200 --> 01:02:49.520
Just typing in the chat and you will see how some of those evidence lines or some of the inputs that go into the rest of your process change and it it's able to trace that it came from you correcting some fact

s669
01:02:49.520 --> 01:02:52.079
in the chat.

s670
01:02:54.880 --> 01:03:06.640
And another example is h if you want to go and make a change and click on the underlying figure that changed h it should show you all the way back

s671
01:03:06.640 --> 01:03:11.040
where this came from and why it matters for the rest of your calculation.

s672
01:03:12.720 --> 01:03:16.799
I'm going to say something that might feel repetitive.

s673
01:03:16.799 --> 01:03:33.760
Um but again we are we are this this harness is an example of how to apply the principles but I I'm sure everyone is thinking about your use cases your companies the agency you're building the agency you're using and how like to to to connect the dots and to to I don't know put the wires together.

s674
01:03:33.760 --> 01:03:50.480
I think this is an an example we can think about when our users our and customers of the agents that we're building are using it and again the trust that they have in your product in your agent and in the output is what will drive

s675
01:03:50.480 --> 01:03:57.680
usage adoption improvement and so on through the loops that we mentioned through tracing evolves and so on.

s676
01:03:57.680 --> 01:04:15.920
So again, I think um what's underlying here is when you're building a a harness, make sure that you can um make this happen to uh your customers that are using your users that are using so that they trust the results that your agent is giving them.

s677
01:04:15.920 --> 01:04:24.640
Again, maybe repetitive, but like making sure we uh draw get everything together as we go.

s678
01:04:25.039 --> 01:04:38.079
Before we move on, anyone has any example of a time where having the evidence trail would have saved you a lot of pain?

s679
01:04:39.440 --> 01:04:47.920
I see nodding like painful acknowledgement, but if anything you want to share, more than happy to to hear you out.

s680
01:04:47.920 --> 01:04:50.799
Otherwise, we can carry on.

s681
01:04:51.680 --> 01:04:54.720
basically every single time you try to do something.

s682
01:04:54.720 --> 01:04:57.760
And so I'm playing with this.

s683
01:04:57.760 --> 01:05:02.559
I really like the idea of being able to show the receipts and everything.

s684
01:05:02.559 --> 01:05:08.640
That's basically every single conversation I have with an agent, whether I'm coding or like working on a deck or something.

s685
01:05:08.640 --> 01:05:11.200
I'm like, "Where did this number come from?"

s686
01:05:11.200 --> 01:05:14.160
Like, "Well, we talked about this blah blah blah blah blah."

s687
01:05:14.160 --> 01:05:27.839
I'm like, "It would be nice if I could just be able to show me specifically like the paper I like the that you use a presentation as an example

s688
01:05:27.839 --> 01:05:40.240
because I had an example myself where I was trying as hard as I could to not have to resize text boxes in Google Slides for a good like month and a half.

s689
01:05:40.240 --> 01:05:46.240
So I was forcing my way into having something that could help me build slides.

s690
01:05:46.240 --> 01:05:56.000
and I had a really nice harness that would like dig into the problem, get all the relevant documents and help me highlight what are the key points that I wanted to present

s691
01:05:56.000 --> 01:05:57.839
to the team.

s692
01:05:57.839 --> 01:06:14.480
And turns out in my conversations, I never caught the fact that when I ask for a modification and give an example, the model will fixate on the example so hard that it will put it everywhere in my presentation.

s693
01:06:14.480 --> 01:06:16.079
I didn't pick it up.

s694
01:06:16.079 --> 01:06:25.599
So, you can imagine I was showing some colorful example to senior leadership and they were like, "Where did you get that number and that assumption?"

s695
01:06:25.599 --> 01:06:28.400
And I was like, "That's crazy.

s696
01:06:28.400 --> 01:06:29.680
I don't know where that came from.

s697
01:06:29.680 --> 01:06:31.039
That's crazy."

s698
01:06:31.039 --> 01:06:32.400
But yeah, it tends to happen.

s699
01:06:32.400 --> 01:06:33.200
It's really annoying.

s700
01:06:33.200 --> 01:06:40.880
And that trace of consequence can really hurt the trust that you have in your harness and the results it produces.

s701
01:06:41.760 --> 01:06:43.440
Let me carry on.

s702
01:06:43.440 --> 01:06:46.720
So another principle is on accountability.

s703
01:06:46.720 --> 01:06:57.839
Uh and this builds on the evidence because you only make a decision when you have some facts when you have some point of reference.

s704
01:06:57.839 --> 01:07:04.960
Erh well I do want to say that but we all have that friend who speaks very confidently about something they don't know.

s705
01:07:04.960 --> 01:07:22.079
So little bit of a counter example, but generally I think if there's one takeaway I would love for everyone here to feel aligned with is we are ultimately the responsible people, the accountable people for whatever

s706
01:07:22.079 --> 01:07:24.240
AI agents do, right?

s707
01:07:24.240 --> 01:07:34.480
Um and there's a funny thing where people discussing that AI is going to um you know we're not going to have any jobs because of AI.

s708
01:07:34.480 --> 01:07:47.119
I always go to back to the fact that I get higher, I get paid partially because I am accountable for the results that I produce for the company, not for me sitting in a computer and typing some stuff.

s709
01:07:47.119 --> 01:07:56.319
So there is an element of the human experience where the whole trust and accountability and working together as a team matters a lot.

s710
01:07:56.319 --> 01:07:58.640
And how do you keep accountability high?

s711
01:07:58.640 --> 01:08:08.240
You do need to be able to see under the hood, inside the black box, etc. Erh, I don't know who said this.

s712
01:08:08.240 --> 01:08:09.200
I loved it.

s713
01:08:09.200 --> 01:08:11.520
I never able to quote it.

s714
01:08:11.520 --> 01:08:15.119
But agents face no real world consequences.

s715
01:08:15.119 --> 01:08:16.159
You do.

s716
01:08:16.159 --> 01:08:21.279
So I I love the show of hands thing, but it probably is the only resource I have.

s717
01:08:21.279 --> 01:08:23.199
Maybe that's why.

s718
01:08:23.199 --> 01:08:27.359
Has anyone seen either written or been told?

s719
01:08:27.359 --> 01:08:28.560
Well, I don't know.

s720
01:08:28.560 --> 01:08:30.827
That's what Claude gave me.

s721
01:08:30.827 --> 01:08:32.827
[laughter]

s722
01:08:32.880 --> 01:08:33.279
Yeah,

s723
01:08:33.279 --> 01:08:35.279
that feels great, right?

s724
01:08:35.279 --> 01:08:37.199
Like, oh, that makes sense.

s725
01:08:37.199 --> 01:08:37.520
Yeah.

s726
01:08:37.520 --> 01:08:37.759
Yeah.

s727
01:08:37.759 --> 01:08:38.640
Everything is forgiven.

s728
01:08:38.640 --> 01:08:40.239
Don't worry.

s729
01:08:40.239 --> 01:08:43.279
We face the consequences of what agents do.

s730
01:08:43.279 --> 01:08:53.600
So, having accountability mechanisms in our harnesses reduces a lot the amount of stress you need to go through.

s731
01:08:53.600 --> 01:08:57.759
Just going in your head like, "But do you think it made the wrong assumption here?"

s732
01:08:57.759 --> 01:08:58.799
Whatever.

s733
01:08:58.799 --> 01:09:06.799
Erh, so you want to reduce ambiguity and not remove it.

s734
01:09:06.799 --> 01:09:13.520
Erh, I'm pretty sure a lot of people here will remember fondly the days prior to auto mode.

s735
01:09:13.520 --> 01:09:20.960
H, who wants to go back to before when auto mode existed and you had to approve every single change by tapping enter?

s736
01:09:20.960 --> 01:09:22.239
Yeah.

s737
01:09:22.239 --> 01:09:22.560
Yeah.

s738
01:09:22.560 --> 01:09:24.880
Lovely times.

s739
01:09:24.880 --> 01:09:27.440
I still do it.

s740
01:09:27.920 --> 01:09:30.159
Fair enough.

s741
01:09:30.433 --> 01:09:31.520
[laughter]

s742
01:09:31.520 --> 01:09:33.679
Fair enough.

s743
01:09:36.799 --> 01:09:42.719
And from all the let's say 100 taps you did, how many of them you failed where?

s744
01:09:42.719 --> 01:09:43.600
I mean, come on.

s745
01:09:43.600 --> 01:09:47.199
This is a oneline change versus this is important.

s746
01:09:47.199 --> 01:09:48.719
Thank god I was looking at this.

s747
01:09:48.719 --> 01:09:51.600
Yeah, I just pref

s748
01:10:08.960 --> 01:10:14.719
and and that's basic that's basically part of the hardness like the design is intentional.

s749
01:10:14.719 --> 01:10:18.640
So you review tasks instead of individual changes.

s750
01:10:18.640 --> 01:10:31.760
And we're just providing you with a a reminder that when you're designing a harness, you want to make sure that you include on purpose those gates to make sure that you can still be accountable for what is being produced.

s751
01:10:31.760 --> 01:10:39.840
Um so make sure that any judgment call, any disambiguation that is important, not the trivial ones, right?

s752
01:10:39.840 --> 01:10:44.000
H you can get some hold of it still.

s753
01:10:44.000 --> 01:10:45.920
H what happens without it?

s754
01:10:45.920 --> 01:10:46.480
I don't know.

s755
01:10:46.480 --> 01:10:48.960
The stoastic machine said it was fine.

s756
01:10:48.960 --> 01:10:50.880
Amazing answer.

s757
01:10:50.880 --> 01:10:52.400
Approving every calculation.

s758
01:10:52.400 --> 01:10:55.600
The other side of the spectrum doesn't feel great.

s759
01:10:55.600 --> 01:10:59.760
H or reviewing every PR line by line that your agents ship.

s760
01:10:59.760 --> 01:11:03.360
Yeah, that it's not really well written, right?

s761
01:11:03.440 --> 01:11:15.360
But yeah, it leads to a lot of either a lot of toil and unnecessary work if it's not designed um in a way that you know makes sense for the use cases.

s762
01:11:15.679 --> 01:11:17.679
Erh.

s763
01:11:18.159 --> 01:11:24.480
So what is the main takeaway on how to design a good mechanisms for accountability?

s764
01:11:24.480 --> 01:11:27.600
Few gates with very high signal.

s765
01:11:27.600 --> 01:11:35.840
Your example of the tasks and breaking it down in a way that you can reason about the the way they connect.

s766
01:11:35.840 --> 01:11:44.159
It's a really good example on where people do h reviews only on functional tests being passed.

s767
01:11:44.159 --> 01:11:45.440
anything prior to that.

s768
01:11:45.440 --> 01:12:01.120
I think you can be a little bit handwavy with yeah sure this looks about right but you wouldn't push anything into production that hasn't passed uh all your tests right so you put some gates in place but those gates need to be very high signal

s769
01:12:01.120 --> 01:12:01.840
um

s770
01:12:01.840 --> 01:12:17.840
and uh to to complement I think we're using some examples in the coding assistant world and and again this is not how we for instance approach things at at Monzo because of like a different context text, but when we this is bringing to to our hardness.

s771
01:12:17.840 --> 01:12:24.239
I think we use the AI coding examples because they're close to us and they're the tools many of us are using on a on a daily basis.

s772
01:12:24.239 --> 01:12:35.360
Uh but when we apply here is the the same thing that claude code does when they have a feature that allows you to gauge how much you want to be involved in the process.

s773
01:12:35.360 --> 01:12:42.560
When we are building our financial harnesses, we can also guide our users in the in the same way.

s774
01:12:43.120 --> 01:12:46.159
That is pretty spot on.

s775
01:12:46.159 --> 01:12:55.679
Um, and again, just going back to the domain with money, there's some stuff you can leave the agent to figure out.

s776
01:12:55.679 --> 01:13:02.640
For instance, if you have a few possible values for something, you're like, well, do three scenarios.

s777
01:13:02.640 --> 01:13:04.000
It's probably going to be fine.

s778
01:13:04.000 --> 01:13:07.760
But you want to review them side to side before making any decision.

s779
01:13:07.760 --> 01:13:16.080
You wouldn't hand over entirely the keys to the kingdom to something that you don't necessarily trust is going to have your best interest in mind.

s780
01:13:16.080 --> 01:13:32.719
Not that it will have an interest at mine, but um so this is a it's very important that you know that bad decisions carry forward into your other decisions.

s781
01:13:32.719 --> 01:13:36.320
So the gates you design need to be fit for purpose.

s782
01:13:36.320 --> 01:13:42.000
You need to know the use case really well in order to be able to put the gates where it matters.

s783
01:13:42.000 --> 01:13:49.280
Otherwise, you're just adding before any decision, ask me and then it becomes very low signal again.

s784
01:13:49.280 --> 01:13:55.360
Um and well for code you don't necessarily need to ask because the tests will speak for themselves.

s785
01:13:55.360 --> 01:14:04.000
That's where the receipts come in handy because you only need to check that everything is passing and you will have a lot of confidence because of it.

s786
01:14:04.000 --> 01:14:13.920
So if you go back to the demo, funnily enough, this plan is not very high on accountability.

s787
01:14:13.920 --> 01:14:24.239
The only step is I will check everything that we calculated all the scenarios whatever and I will make a decision on how much I need to save every month.

s788
01:14:24.239 --> 01:14:36.480
I mean still nice because no money is moving out of my bank account anywhere but yeah there's a accountability mechanism I feel confident that that will gate any anything else

s789
01:14:36.480 --> 01:14:48.480
I think as a as a as an extra example like in this this challenge and decision here likely as we let's say we accept this and this is now marked as a decision we made

s790
01:14:48.480 --> 01:14:57.679
if we ever use this information like the 20k landing on the same day as payment three as something that influences the final decision.

s791
01:14:57.679 --> 01:15:00.640
This definitely comes with the with the accountability.

s792
01:15:00.640 --> 01:15:02.640
Yes.

s793
01:15:10.000 --> 01:15:11.040
Go up.

s794
01:15:11.040 --> 01:15:13.520
No, I actually like I'm improvising a little bit.

s795
01:15:13.520 --> 01:15:16.320
But if you scroll up a little bit,

s796
01:15:16.320 --> 01:15:23.280
h this is like again not not priding myself of any good design at all, anything the contrary.

s797
01:15:23.280 --> 01:15:28.080
But h you see we have a little bit of a weird structure there.

s798
01:15:28.080 --> 01:15:32.960
Looks like a tree a little bit with something branching out joining back in.

s799
01:15:32.960 --> 01:15:44.159
That's another accountability mechanism as Philip is saying where there's some decisions the model will either assume and just try to verify with you like hey is this right?

s800
01:15:44.159 --> 01:15:46.560
Is this the number you want to use here?

s801
01:15:46.560 --> 01:15:53.040
Or just try to figure out things how they fit into the overall um system.

s802
01:15:53.040 --> 01:16:03.600
We thought a good way to showcase that you need to design a harness that gives you the accountability is making sure that you can trace all the decisions that have been made,

s803
01:16:03.600 --> 01:16:16.080
who made them uh and have that sense of well this was something that the agent assumed and I let carry on in the rest of my flow or it's something that

s804
01:16:16.080 --> 01:16:20.000
I provided a revised input to.

s805
01:16:20.000 --> 01:16:35.360
And the reason again I'm a very visual person that could be symptom of something I don't know uh but the reason I think is kind of like a not a straight line conversation but rather branching out and then branching back in is that when you're chatting with an agent

s806
01:16:35.360 --> 01:16:48.000
again for financial purposes you may reason about something after a decision has been made and you are well in your right to correct information and add more context.

s807
01:16:48.000 --> 01:17:01.040
So, I think it's important that when you're creating an a harness that you not only build those gates for validating things, making sure that you're still accountable, but also for correcting the course

s808
01:17:01.040 --> 01:17:10.960
and making sure that you can do so safely and it's not like, oh, because I typed this sentence in the chat, now all my analysis is gone.

s809
01:17:10.960 --> 01:17:25.520
H that has happened to me many many times trying to solve a problem and maybe I was like dispersed dictating to my phone I said something in the middle of it and then the whole analysis is gone because I can't really

s810
01:17:25.520 --> 01:17:40.640
take away that one sentence and please keep everything else as it was h so this gives us another means to be accountable for what goes into the system for financial analysis and what

s811
01:17:42.800 --> 01:17:46.719
Do you want to move to the slides?

s812
01:17:46.719 --> 01:17:48.080
Yes.

s813
01:17:48.080 --> 01:17:48.719
Okay, cool.

s814
01:17:48.719 --> 01:17:51.360
So, uh, a few things we can try.

s815
01:17:51.360 --> 01:18:00.719
Erh, you can go and and hover into the chip that guest that says how many guests we have in the wedding example.

s816
01:18:00.719 --> 01:18:06.239
And you can add that that will add it to the chat and you can say, "Oh, actually, it's not this number.

s817
01:18:06.239 --> 01:18:10.080
H we will get no fewer than 150 guests for instance.

s818
01:18:10.080 --> 01:18:25.120
Erh that will get your harness to rethink the plan see the consequences of that input into other stuff and suggest a either a modification of your plan or just change the input whatever.

s819
01:18:25.120 --> 01:18:41.760
The important part here is you build the feedback mechanism on purpose and you have some judgment over making sure that there is some consequential decision making on the fact that this input may change your whole analysis.

s820
01:18:41.760 --> 01:18:47.440
H you need to be on top of making sure that those changes make sense.

s821
01:18:49.040 --> 01:19:03.679
The other thing you can try is on if you click on step five erh you can open up a new branch conversation like a branched conversation I think is the button on the chat at the bottom

s822
01:19:03.679 --> 01:19:04.159
here.

s823
01:19:04.159 --> 01:19:04.400
Yeah.

s824
01:19:04.400 --> 01:19:04.880
Yeah.

s825
01:19:04.880 --> 01:19:07.840
So you can open a new branch.

s826
01:19:08.239 --> 01:19:12.239
Has anyone used the slash

s827
01:19:12.239 --> 01:19:16.560
btw by the way thing on cloud?

s828
01:19:17.600 --> 01:19:20.719
How has been your experience using it?

s829
01:19:20.719 --> 01:19:24.080
Sometimes it works, sometimes it doesn't.

s830
01:19:24.080 --> 01:19:24.560
doesn't.

s831
01:19:24.560 --> 01:19:28.880
And it feels like you're doing an aside, but not really an aside aside.

s832
01:19:28.880 --> 01:19:32.719
It becomes a part of the context, right?

s833
01:19:33.440 --> 01:19:35.520
But do you get a choice?

s834
01:19:35.520 --> 01:19:38.159
Like maybe I have a stupid question in the middle of the conversation.

s835
01:19:38.159 --> 01:19:39.590
No, I don't have a choice.

s836
01:19:39.590 --> 01:19:39.840
[laughter]

s837
01:19:39.840 --> 01:19:42.800
If I ask a question, then it becomes the part of the session.

s838
01:19:42.800 --> 01:19:43.120
Yeah.

s839
01:19:43.120 --> 01:19:44.880
Yeah.

s840
01:19:44.880 --> 01:19:54.320
So in here, what we're trying to do, and going back to why we're doing it like this, we want to show you how it could feel if we design something

s841
01:19:54.320 --> 01:19:58.320
with the intent of, well, this really needs to be an aside.

s842
01:19:58.320 --> 01:20:01.199
I have too many stupid questions when I'm chatting with AI.

s843
01:20:01.199 --> 01:20:02.000
So many.

s844
01:20:02.000 --> 01:20:08.800
And it's shameful, but I end up having to open a separate chat because I don't trust it's not going to contaminate what I'm doing.

s845
01:20:08.800 --> 01:20:12.159
And then I will take more time just being like, "Hey, this is a stupid question.

s846
01:20:12.159 --> 01:20:17.920
please don't use for anything else is the please don't make any mistakes thing Philippe was saying

s847
01:20:17.920 --> 01:20:19.360
h doesn't work quite well.

s848
01:20:19.360 --> 01:20:27.600
So whatever you design if you have a feedback mechanism make sure that you tailor it to how you actually want to use it.

s849
01:20:27.600 --> 01:20:41.280
Erh oops cool and then as for h the last point the last principle will be verification.

s850
01:20:41.280 --> 01:20:54.239
So you want a second pair of eyes and this builds on the principle of accountability because you are accountable for that review as well.

s851
01:20:54.239 --> 01:21:10.400
Erh, and maybe that's too abstract, but has anyone been in a situation where someone h passed the blame on a bad decision to you or anyone else just on the basis of

s852
01:21:10.400 --> 01:21:13.520
No, but Franchesco, review it.

s853
01:21:14.159 --> 01:21:15.520
That happens.

s854
01:21:15.520 --> 01:21:16.480
That happens.

s855
01:21:16.480 --> 01:21:31.040
So, um, we were having a funny conversation like a month ago where a few a few people are starting to have thoughts that the burden of something being good or correct with VIP coding

s856
01:21:31.040 --> 01:21:44.719
ends up being in the hands of the reviewer in a team that feels really demoralizing because you're just being given a bunch of stuff that sure looks about right, but then it's your responsibility

s857
01:21:44.719 --> 01:21:48.880
responsibility ultimately to make sure that it is right.

s858
01:21:48.880 --> 01:21:58.560
This whole principle of verification is just building more feedback loops so you don't have blind spots but without relegating any of the accountability.

s859
01:21:58.560 --> 01:22:09.520
Ultimately, this is just extra information from a different source to avoid biases and avoid you getting stuck on your own thinking and your own processes.

s860
01:22:09.520 --> 01:22:14.080
But it doesn't remove any of the responsibility or accountability.

s861
01:22:15.040 --> 01:22:22.880
Um yeah, funny enough, expert experts often do not check every cell.

s862
01:22:22.880 --> 01:22:30.719
And this is more on like my experience working with financial finance teams in data.

s863
01:22:30.719 --> 01:22:41.440
A they could open a spreadsheet and just look at the you know a few cells they know will give you a ballpark total and if it doesn't feel right they will let you know.

s864
01:22:41.440 --> 01:22:44.560
It's like, how did we spend this much money on marketing?

s865
01:22:44.560 --> 01:22:46.080
I don't think that's right.

s866
01:22:46.080 --> 01:22:49.920
Uh, and it's a little bit of a gut reaction.

s867
01:22:49.920 --> 01:22:51.280
You don't need a lot of effort.

s868
01:22:51.280 --> 01:22:59.520
You don't need to like redo your plan entirely or challenge every decision you're made, but you do want to have a separate point of reference.

s869
01:22:59.520 --> 01:23:15.280
Erh, I think I I had an example, but I don't remember what was it was something about a tweet that got really famous about Elon Musk having I don't know how many trillion dollars

s870
01:23:15.280 --> 01:23:22.800
and someone saying like, well, he could give a million dollars to every person in the world and still have money.

s871
01:23:22.800 --> 01:23:25.360
And he was like, wait, what?

s872
01:23:25.360 --> 01:23:26.320
That doesn't feel right.

s873
01:23:26.320 --> 01:23:28.159
Like you read it and you're like, wait, what?

s874
01:23:28.159 --> 01:23:29.040
Looks plausible.

s875
01:23:29.040 --> 01:23:32.320
And then you're like, "No, no, that's not right."

s876
01:23:32.320 --> 01:23:40.560
And that needed need that needed for you to look into the calculations the person made is like a little bit gut feeling.

s877
01:23:40.560 --> 01:23:49.360
Um, so just having something review and give you a rough answer or some challenge if something doesn't look right is extremely helpful.

s878
01:23:49.360 --> 01:23:51.360
Sorry.

s879
01:23:57.520 --> 01:23:59.520
supply.

s880
01:24:06.080 --> 01:24:07.360
That is such a good question.

s881
01:24:07.360 --> 01:24:16.719
Uh I was trying to come up with a examples that feel more down to earth, but I think that's a very conversation to have.

s882
01:24:16.719 --> 01:24:26.480
A good argument is if you're getting golden examples of how the data should look like, why would your process give you something different to that?

s883
01:24:26.480 --> 01:24:29.440
Like why don't you just check with your golden examples?

s884
01:24:29.440 --> 01:24:35.199
And this is something I I wanted to abstract away from the world of finance.

s885
01:24:35.199 --> 01:24:38.560
You have a lot of sources of information, a lot of them.

s886
01:24:38.560 --> 01:24:42.800
You always have another point of reference outside of your data.

s887
01:24:42.800 --> 01:24:58.320
So if some sources you're using are claiming I don't know whatever fact a number you have the internet you should always try and enhance the signal you're getting by looking out

s888
01:24:58.320 --> 01:25:09.199
look at all the resources that are not the ones that I'm using and you avoid that h feedback loop where you only have the information you have available you only have your golden examples

s889
01:25:09.199 --> 01:25:25.360
and everything looks right to me Always try to get a little bit creative when building a harness to try and guide which other uh points of reference it could use to help you make a better decision or to know if something is not quite right.

s890
01:25:25.360 --> 01:25:26.880
Does that

s891
01:25:26.880 --> 01:25:28.880
Yeah.

s892
01:25:38.880 --> 01:25:50.159
And I guess we generally try to like work whoever it is to like understand what would be the best set of things that agent should use the best answer.

s893
01:25:50.159 --> 01:26:05.440
So trying to understand how that would have been the optimal to give you another example that could be useful.

s894
01:26:05.440 --> 01:26:16.320
I think the golden examples give you your ground truth for making sure that you are within the right space but no domain is isolated.

s895
01:26:16.320 --> 01:26:27.520
So what happened to me a lot of times when I was a BI analyst in some like company is that I thought what I wrote was really good.

s896
01:26:27.520 --> 01:26:29.040
So my query was good.

s897
01:26:29.040 --> 01:26:34.960
The number I got seemed reasonable and I was checking everything and nothing looked off.

s898
01:26:34.960 --> 01:26:43.760
And then when I presented the dashboard to some stakeholder immediately went like that's not correct and I was like what do you mean?

s899
01:26:43.760 --> 01:26:46.639
Do you know SQL more than I do?

s900
01:26:46.639 --> 01:26:48.719
Did you do do the analysis with me?

s901
01:26:48.719 --> 01:26:57.840
And turns out this person was from a separate team and they just had an idea of how many whatever sales were made in that year.

s902
01:26:57.840 --> 01:27:06.000
So they knew instinctively that number can't be right because it means my information in this other place is incorrect.

s903
01:27:06.000 --> 01:27:18.560
So it's not if we translate that to the agentic world the golden examples I will give my agent in my hardness will be based on like biases and selection bias and so on

s904
01:27:18.560 --> 01:27:30.000
based on the domain and I picked them for the domain and I think they're good for the domain but when you look at a slightly tangential domain you may find out that

s905
01:27:30.000 --> 01:27:39.199
actually this is nonsensical like this number is wrong just by looking at this other thing next to me I I don't even need to look again into my calculations.

s906
01:27:40.560 --> 01:27:42.320
Cool.

s907
01:27:42.320 --> 01:27:43.600
Erh.

s908
01:27:43.600 --> 01:28:01.679
So there is a concept called thing slicing where you have thin slices of your anecdotal experience and you know what things feel right or feel wrong based on your experience in your domain.

s909
01:28:01.679 --> 01:28:03.600
Whatever analysis you're doing.

s910
01:28:03.600 --> 01:28:08.560
the model will have an average of everyone else's experiences.

s911
01:28:08.560 --> 01:28:22.320
So when you're building your hardness, you need to make sure that you can uh have a feedback mechanism so that you can verify from what the model thinks it knows

s912
01:28:22.320 --> 01:28:25.520
all the way to what you think you know.

s913
01:28:25.520 --> 01:28:37.360
And if you flip that convoluted sentence I just said it's the it's how we started the whole framework and all the principles where you're trying to get to that

s914
01:28:37.360 --> 01:28:40.000
shared domain shared context.

s915
01:28:40.000 --> 01:28:46.400
You're essentially translating your language into the training data set for the model.

s916
01:28:46.400 --> 01:28:53.360
You're just working with the model to be like okay this is how I understand the problem or how my company does this.

s917
01:28:53.360 --> 01:29:07.199
and then the model is helping like okay this is how would I fit it in my training data set here we're doing the opposite we're just saying okay I think we got to a point where you can make a reasonable judgment over this domain

s918
01:29:07.199 --> 01:29:22.719
but let's have a feedback loop so I can tell you from my experience if this looks entirely off or if it looks reasonable so um yes it's probably fine may look nothing like yours

s919
01:29:22.960 --> 01:29:38.880
I think is a way good way to describe it like yeah it looks about right and then turns out your company does care about that 5% difference it makes a world of a difference when you're building hardness for your team that you're aware that that 5%

s920
01:29:38.880 --> 01:29:42.560
it will be a deal breaker for the whole team.

s921
01:29:42.560 --> 01:29:48.960
Um there's some stuff that happens if you don't have um a way to do this.

s922
01:29:48.960 --> 01:30:00.480
So checking everything line by line or checking nothing or you know no sense of what about right looks like and if you don't have that point of reference your harness won't have it either.

s923
01:30:00.480 --> 01:30:05.760
So you need to build those kind of like external validation points.

s924
01:30:07.120 --> 01:30:11.040
I'm going to skip through some of this.

s925
01:30:13.600 --> 01:30:29.600
And for the verification, it's those notes in the H decision as well where we're adding uh an external point of reference to some of the stuff the model thought it knew.

s926
01:30:37.280 --> 01:30:41.280
It's kind of satisfac satisfactory when the thing just like

s927
01:30:41.280 --> 01:30:42.800
the branch go comes back.

s928
01:30:42.800 --> 01:30:44.719
Yeah, exactly.

s929
01:30:44.719 --> 01:30:58.159
And again, I think it creates a a trace to to um be able to like the the thin slice and the quick instinctive look also comes from from this because you understand the thought process

s930
01:30:58.159 --> 01:31:03.360
that the agent went through and not only the the output only.

s931
01:31:06.159 --> 01:31:07.199
I go back.

s932
01:31:07.199 --> 01:31:08.880
Yes.

s933
01:31:08.880 --> 01:31:15.520
So, as with the other examples, if you're still following along, uh there's a few things you can try.

s934
01:31:15.520 --> 01:31:20.400
So, you can open a branch uh on a tagged challenge.

s935
01:31:20.400 --> 01:31:29.280
That's basically a separate agent that we tasked with reviewing the plan, reviewing the assumptions, and then doing some healthy challenge.

s936
01:31:29.280 --> 01:31:32.000
Does this read reasonable?

s937
01:31:32.000 --> 01:31:42.639
Um so just click it open it up and then you will see how just Philippe did it will join back into the trunk and actually change the plan.

s938
01:31:42.639 --> 01:31:54.320
So there is a consequence and we think this is another important part of being intentional with the design of your harnesses.

s939
01:31:54.480 --> 01:31:57.920
We could have left it as LLM as a judge.

s940
01:31:57.920 --> 01:32:06.080
So for every decision, for every plan, whatever, spin another uh agent and then do a review and call it a day.

s941
01:32:06.080 --> 01:32:12.320
The problem is if there's no feedback loop, our financial decision or decision whatsoever doesn't improve at all.

s942
01:32:12.320 --> 01:32:22.719
So that's part of the design, the taste you need to build h to create fitforpurpose harnesses for whatever use cases you have.

s943
01:32:22.719 --> 01:32:27.600
Then the other thing you can try is opening another and just clicking on dismiss.

s944
01:32:27.600 --> 01:32:39.040
And we did that on purpose because sometimes as I was saying before some conversations don't actually change anything or not material to your process and you will know better.

s945
01:32:39.040 --> 01:32:44.480
So you need to have that kind of like point of reference for validating things.

s946
01:32:44.480 --> 01:32:49.199
Um so just to summarize and leave this on the screen for a moment.

s947
01:32:49.199 --> 01:33:03.840
Um, if you're still playing around with the harness, uh, the framework for building really good harnesses that you can trust with financial data with money, it's built on these five principles.

s948
01:33:03.840 --> 01:33:18.080
So having a shared domain, having consistency, showing the receipts, keeping your accountability, and making sure that you create intentional loops for verification outside of what you think, you know.

s949
01:33:18.080 --> 01:33:24.239
Erh, what?

s950
01:33:24.239 --> 01:33:25.199
No confetti.

s951
01:33:25.199 --> 01:33:27.199
Disappointing.

s952
01:33:27.760 --> 01:33:28.159
Okay.

s953
01:33:28.159 --> 01:33:34.719
So, there's a few cases where we think it's important to define.

s954
01:33:34.719 --> 01:33:35.440
Oh, sorry.

s955
01:33:35.440 --> 01:33:37.199
You had your hand up.

s956
01:33:37.199 --> 01:33:38.719
Yes.

s957
01:33:38.719 --> 01:33:41.840
So, I just confirm my understanding.

s958
01:33:41.840 --> 01:33:45.280
You brought us a framework like this five principles.

s959
01:33:45.280 --> 01:33:47.679
You can build your harness upon it.

s960
01:33:47.679 --> 01:33:51.760
And you also brought one harness as an example.

s961
01:33:51.760 --> 01:34:01.040
So, I'm I'm thinking about when should I build harness, how generic or how specific the harness should be?

s962
01:34:01.040 --> 01:34:10.960
Because you solve problems with the specific harness and like for example, how many harnesses do you have in your work?

s963
01:34:10.960 --> 01:34:19.679
It's just one and you solve everything that you have for this in this harness or you have multiple more specific harness.

s964
01:34:19.679 --> 01:34:22.400
How generic and how often do you build one?

s965
01:34:22.400 --> 01:34:25.520
Like how much effort do you put in building one?

s966
01:34:25.520 --> 01:34:29.280
you make one every month or you make one every six months.

s967
01:34:29.280 --> 01:34:32.880
That's a that's a great question and there's no clear answer of course.

s968
01:34:32.880 --> 01:34:45.760
Um I think it's we could we could trace as you said like the two ends of the spectrum as I just use clo code or or codeex for everything and I just embed everything into

s969
01:34:45.760 --> 01:35:02.639
skills instructions so on so forth and on the other end of the spectrum is for every small new thing I want to do I'll build it from the ground up and like of course that will get tiring at some point because like you're spending more time creating your tools than actually using the tools sometimes

s970
01:35:02.639 --> 01:35:04.400
Um again there's no clear answer.

s971
01:35:04.400 --> 01:35:10.560
I think it comes to um when you feel the use case is worth enough.

s972
01:35:10.560 --> 01:35:26.480
So you want reproducibility, you want it to feel few the results you are getting using the pre-built harnesses are very frequently getting it wrong because the value you get from building the harness

s973
01:35:26.480 --> 01:35:33.760
is you get more control again with the deterministic bit together with the nondeterminism.

s974
01:35:33.760 --> 01:35:34.400
Right?

s975
01:35:34.400 --> 01:35:46.719
Um the reason why we're talking about the the principles in the case in the um scenario of a financial analysis is because if we do this type of use case

s976
01:35:46.719 --> 01:35:54.480
in a harness that doesn't have it we'll probably have some of the f pitfalls that Franchesco talks through each of the steps.

s977
01:35:54.480 --> 01:36:03.280
So then in this case it makes sense for um to avoid these pitfalls we are seeing over and over and over again to build the thing.

s978
01:36:03.280 --> 01:36:12.960
But again, I think it comes down to how many users you believe will go through that use case, how critical is the problem that you were seeing.

s979
01:36:12.960 --> 01:36:26.159
So like if if there's a non-b businessiness critical um task that you were solving, you were probably fine to have a human in the loop reviewing the outputs that you get and steering the agent even if that takes a few extra turns.

s980
01:36:26.159 --> 01:36:38.400
But if you're putting this in the hands of a uh internal user, for instance, let's say you were building something for your team of credit analysts, um you're going to have 50 people using something

s981
01:36:38.400 --> 01:36:41.840
to support key company decisions.

s982
01:36:41.840 --> 01:36:46.000
Yeah, you absolutely don't want a lot of variability on that.

s983
01:36:46.000 --> 01:36:53.679
So again, there's no clearcut I would say, but these are the things I would consider in in making the decision.

s984
01:36:53.679 --> 01:36:56.800
Maybe I can help with uh the counterfactual.

s985
01:36:56.800 --> 01:37:04.719
So, we think this is a good recipe specifically for financial harnesses because there's a lot at stake.

s986
01:37:04.719 --> 01:37:06.639
You want a lot of feedback loops.

s987
01:37:06.639 --> 01:37:16.159
There's very specific things about those use cases that somehow we're representing here even at the personal level or the company level.

s988
01:37:16.159 --> 01:37:31.440
But imagine you want to do you want to use AI just to produce like social media content and maybe one of your use cases is just flooding Instagram and Tik Tok with AI generated videos like everyone is doing.

s989
01:37:31.440 --> 01:37:34.480
There's some steps here that you probably don't care that much.

s990
01:37:34.480 --> 01:37:36.960
Evidence gathering.

s991
01:37:37.840 --> 01:37:38.880
What for?

s992
01:37:38.880 --> 01:37:42.239
H verification or accountability even less so.

s993
01:37:42.239 --> 01:37:56.560
So this is specifically geared towards like financial use cases and it can be extended to different levels of it but everything hinges on the fact that is a trust system.

s994
01:37:56.560 --> 01:38:05.840
H this recipe may not be as useful if your use cases are not necessarily like that and in terms of how many harnesses you want to build.

s995
01:38:05.840 --> 01:38:17.760
I think Filipe said it better, but you can easily fall victim for doing a lot of axe sharpening and not actually using the axe.

s996
01:38:17.760 --> 01:38:31.920
H So, at some point, if you're trying to use something and you're not getting the right results, you can try adding a couple of skills, a couple of commands, a couple of extra things, still not getting the results or the experience you want.

s997
01:38:31.920 --> 01:38:36.880
You're finding yourself copy pasting text from one chat to another, opening a new session.

s998
01:38:36.880 --> 01:38:39.440
It feels convoluted harness.

s999
01:38:39.440 --> 01:38:44.560
So you all the repeated things that you had to do three five times.

s1000
01:38:44.560 --> 01:38:45.199
Okay.

s1001
01:38:45.199 --> 01:38:49.520
Just once and you design it around it.

s1002
01:38:49.920 --> 01:38:50.239
Yes.

s1003
01:38:50.239 --> 01:38:51.199
Yes.

s1004
01:38:51.199 --> 01:38:52.719
Some more related question.

s1005
01:38:52.719 --> 01:38:55.840
So for your wedding planning task, right?

s1006
01:38:55.840 --> 01:39:04.000
I think one of like one of the naive way of looking at this is like maybe agent right like agents, venue agents.

s1007
01:39:04.000 --> 01:39:09.920
Uh, have you have you have you tried that and what's the outcome or like your opinion on that?

s1008
01:39:09.920 --> 01:39:23.199
I will start by confessing that I have the most expensive cloth subscription and I do max it out and not in a good way as in Anthropic is definitely on the on the red because of me.

s1009
01:39:23.199 --> 01:39:36.719
I tried research everything for all the venue, the flowers, the wedding, do the planning, do the different like scenarios for how much we will have to pay etc. And what I found is that

s1010
01:39:36.719 --> 01:39:46.960
in many cases it was good enough to give me the list of the top five photographers in Madrid with the new web search with like a refined h tooling etc.

s1011
01:39:46.960 --> 01:39:47.840
Worked like a charm.

s1012
01:39:47.840 --> 01:40:01.920
didn't have to do anything else just spinning and having comparisons and even reaching out to them but for some more nuance use cases where I do want to have more control and predictability and this is linking to the financial nature of them

s1013
01:40:01.920 --> 01:40:19.119
I did find myself being like I need to build something robust I'm not recalculating this again uh and to to make the pain very real I remember myself one day because I was in the car and I was like I can dictate some stuff.

s1014
01:40:19.119 --> 01:40:32.320
Voice mode and then I was driving and it was about an hour drive back and forth, you know, an hour and an hour and I re had I had the same conversation three times in that t in that trench

s1015
01:40:32.320 --> 01:40:45.440
just because every time I was trying to do some financially related or adjacent planning, something will go wrong and I'll be like so yes, that helps.

s1016
01:40:47.040 --> 01:41:00.560
So getting back to your example of being a BI right like um I don't think back in the day there were people who were validating your SQL queries right

s1017
01:41:00.719 --> 01:41:12.880
um so this seems like we are it's like next level of SQL queries right now you're building tools because harnesses are easier to build and stuff have you found

s1018
01:41:12.880 --> 01:41:25.679
ways to kind of make this point clear to leadership ship in in conversations that like because because to me it feels every time there is an AI there is an expectation that it will be 100% correct

s1019
01:41:25.679 --> 01:41:28.560
like you see what I'm getting at

s1020
01:41:28.560 --> 01:41:36.960
so is your question about managing the expectations so that when people do agentic self surf

s1021
01:41:36.960 --> 01:41:46.159
it's not enginic sales more so like let's say you build this and and you're talking to your senior leadership to make a financial decision in your company Right?

s1022
01:41:46.159 --> 01:41:56.560
And they start drilling you and drilling you and and the moment that you say, "Oh, I use AI and harnesses, they're like, well, this has to be 200% correct or something like you know what I'm saying?

s1023
01:41:56.560 --> 01:42:02.880
Have you found the way to to kind of make sure that they understand this is still SQL queries?"

s1024
01:42:02.880 --> 01:42:04.159
Bit of a spoiler,

s1025
01:42:04.159 --> 01:42:06.285
but there's a talk tomorrow.

s1026
01:42:06.285 --> 01:42:08.285
[laughter]

s1027
01:42:08.960 --> 01:42:09.360
I don't know.

s1028
01:42:09.360 --> 01:42:10.215
Do you have any?

s1029
01:42:10.215 --> 01:42:10.480
[clears throat]

s1030
01:42:10.480 --> 01:42:10.639
Yeah.

s1031
01:42:10.639 --> 01:42:10.800
Yeah.

s1032
01:42:10.800 --> 01:42:11.040
Yeah.

s1033
01:42:11.040 --> 01:42:25.840
No, I think um it's with the AI first world, we sometimes as you were saying forget that what's that that's only a two and a means to an end

s1034
01:42:25.840 --> 01:42:39.119
be before like what's what's going on right so I think what we would have noticed and again from Monzo's experience we've been approaching it from an highly impactbased um point of view so

s1035
01:42:39.119 --> 01:42:45.760
it's not about we should use AI for this because AI can something something something.

s1036
01:42:45.760 --> 01:42:48.800
It's like what's the outcome we're looking for here?

s1037
01:42:48.800 --> 01:42:50.639
What how do we measure success here?

s1038
01:42:50.639 --> 01:42:55.119
Are we looking for less uh back and forth with a BI team?

s1039
01:42:55.119 --> 01:42:57.920
Uh we don't even have a BI team, but you get you get what I mean.

s1040
01:42:57.920 --> 01:43:01.199
Um to validate a new dashboard.

s1041
01:43:01.199 --> 01:43:01.440
Okay.

s1042
01:43:01.440 --> 01:43:07.679
So, we can track that and actually see if AI can support us in in generating that outcome.

s1043
01:43:07.679 --> 01:43:16.159
um are we looking for um less I don't know internal ask tickets because the number looks wrong.

s1044
01:43:16.159 --> 01:43:29.360
Okay, so let's track that and then the approach that we do the way we do that doesn't really matter to a point if that makes sense because it will be again a tool that can get us to that

s1045
01:43:29.360 --> 01:43:36.080
um to that result probably faster with better guard rails and so on and so forth.

s1046
01:43:36.080 --> 01:43:50.960
So I think it's less it's less about gaining trust on the implementation on the AI itself but more about aligning on what's the outcome we're trying to drive with the AI.

s1047
01:43:51.920 --> 01:43:54.960
I think we had another one there.

s1048
01:43:55.119 --> 01:44:11.040
My question is more you have show a harness that has some complexity right you have a UI you have databases say but I have also build processes that invoke some of these steps but they are all at skill

s1049
01:44:11.040 --> 01:44:26.400
right that it ask you questions it runs the number based on a Python script and so on so my question is more like where you see the boundaries between like having a skill that does all these creating like a separate tool or

s1050
01:44:26.400 --> 01:44:27.520
have have you tried that?

s1051
01:44:27.520 --> 01:44:27.920
Absolutely.

s1052
01:44:27.920 --> 01:44:29.199
Yeah, absolutely.

s1053
01:44:29.199 --> 01:44:43.679
Um I think it it again very similarly to to what we were saying about your user base and the reproducibility you want is when this becomes critical enough that the tradeoffs we are getting with the

s1054
01:44:43.679 --> 01:44:48.719
nondeterministic approach are are not uh we're not willing to have them.

s1055
01:44:48.719 --> 01:45:04.719
I usually describe the the skill command like the the promptbased approach as hey please follow all of this and make no mistakes please please please like again because this is what's going on right we are we are giving a a lot of organized

s1056
01:45:04.719 --> 01:45:18.880
very well organized instructions to a machine and hoping that it it will perform these instructions agents are good enough nowadays they will probably perform your instructions 99% of the time.

s1057
01:45:18.880 --> 01:45:21.600
Uh, but what about that 1%?

s1058
01:45:21.600 --> 01:45:26.639
What about an edge case that makes things don't work?

s1059
01:45:26.639 --> 01:45:40.639
What about if you distribute this to your internal users and your internal user has a instruction in their user level cloud.mmd file that goes against what you are telling your agent to do.

s1060
01:45:40.639 --> 01:45:54.719
You see my point is like um there are many when you when you start distributing it to a number of users and use cases things can get messy in places you are not able to predict

s1061
01:45:54.719 --> 01:46:07.280
and then you can find yourself I we have a at Moz we have an internal like contribution for agent extensions where multiple teams can well I've come across something that works with me let me share it internally

s1062
01:46:07.280 --> 01:46:17.280
um so like plugins skills and so on and very frequent ly we when reviewing the instructions we see things trying to cover for if the user tries to do this

s1063
01:46:17.280 --> 01:46:31.119
tell the user back that this skill does not support that and you see like it's again trying to cover for many things that with the determinism and when I say determinism is the backend code like the python the whatever we are using

s1064
01:46:31.119 --> 01:46:40.159
uh that the harness provides that out of the like out of the box is something we can cover or

s1065
01:46:43.360 --> 01:46:47.360
anyone has any other comments, questions.

s1066
01:46:49.040 --> 01:46:51.840
All right, cool.

s1067
01:46:52.400 --> 01:46:53.833
Where is the thingy?

s1068
01:46:53.833 --> 01:46:54.639
[laughter]

s1069
01:46:54.639 --> 01:46:55.040
Never mind.

s1070
01:46:55.040 --> 01:46:55.440
Never mind.

s1071
01:46:55.440 --> 01:46:57.600
Never mind.

s1072
01:47:00.080 --> 01:47:01.199
Cool, cool, cool, cool.

s1073
01:47:01.199 --> 01:47:01.840
Skipping ahead.

s1074
01:47:01.840 --> 01:47:02.719
Skipping ahead.

s1075
01:47:02.719 --> 01:47:04.719
Ah, fun.

s1076
01:47:04.719 --> 01:47:09.760
Erh, I wanted to make a joke and I completely missed it for clicking this thing too quickly.

s1077
01:47:09.760 --> 01:47:12.560
But finance is funny.

s1078
01:47:12.560 --> 01:47:26.400
And I say funny because I find it to be a super interesting domain with a the added advantage that back in the day finance felt like just working with spreadsheets,

s1079
01:47:26.400 --> 01:47:31.440
very like boring work, adding numbers here and there, formulas, whatever.

s1080
01:47:31.440 --> 01:47:44.400
If you ask any accountant is all about creativity and interpretation because they will come up with the weirdest things and I think in that level of ambiguity AI is really good

s1081
01:47:44.400 --> 01:47:57.360
really really good you have some bits that are the spreadsheet and then you realize some of the cash flow didn't quite make sense because some of the revenue was attributed a month later and you're like ah creativity.

s1082
01:47:57.360 --> 01:48:11.440
So because we are in this space of finance and depending on where is your mind mindset at you could look at a problem as something very simple that you can just get an agent to fix within 10 minutes

s1083
01:48:11.440 --> 01:48:23.040
or you could look at it as it actually is which is a mixture of things that need to run reliably 100% of the time and a lot of interpretation and freedom of

s1084
01:48:23.040 --> 01:48:26.320
creativity in between the lines.

s1085
01:48:26.639 --> 01:48:30.880
So yeah, sorry, very poorly delivered joke, but that's what I wanted to go through.

s1086
01:48:30.880 --> 01:48:40.880
Uh, and the bottom line being the design of your harnesses needs to be intentional to increase trust.

s1087
01:48:41.119 --> 01:48:49.199
Don't assume that oh because this is running on AI is going to be what was the the quote 200% accurate.

s1088
01:48:49.199 --> 01:48:59.520
It will likely not unless you design all the mechanisms so that whatever process runs you know you can trust the results to a degree.

s1089
01:48:59.520 --> 01:49:03.119
Um and yeah models are going to get better.

s1090
01:49:03.119 --> 01:49:13.520
We have seen the last six months nine months in saying I'm getting a little bit of whiplash even with you know how fast things are going but they still have to earn the trust.

s1091
01:49:13.520 --> 01:49:21.760
it's not a 100% right and in when we're talking about money 1% wrong sometimes is a lot of money.

s1092
01:49:21.760 --> 01:49:26.080
So yeah, we kind of skipped into the questions, but if anyone has anything else,

s1093
01:49:26.080 --> 01:49:29.440
yeah, just really quick, like in the UI, how do you demonstrate trust?

s1094
01:49:29.440 --> 01:49:32.719
Do I like you just ask it?

s1095
01:49:32.719 --> 01:49:36.159
Show me how you calculated this or does it show you the code?

s1096
01:49:36.159 --> 01:49:37.040
Can you show an example?

s1097
01:49:37.040 --> 01:49:38.320
Like how do you prove it?

s1098
01:49:38.320 --> 01:49:42.800
The intent was that it was a aggregation of everything.

s1099
01:49:42.800 --> 01:49:51.199
So all the steps in the framework should have failed like, oh, but I know where this is coming from, but I know this is running as code, but I know this is tested.

s1100
01:49:51.199 --> 01:49:57.440
So it should have been an experience to kind of like immerse into this feels trustworthy.

s1101
01:49:57.440 --> 01:50:00.400
But again, it's one implementation we came up with.

s1102
01:50:00.400 --> 01:50:02.560
So not not super direct.

s1103
01:50:02.560 --> 01:50:02.800
Yeah.

s1104
01:50:02.800 --> 01:50:10.560
But I think the the takeaway is there won't be a single thing that will work to earn the user's trust at any point.

s1105
01:50:10.560 --> 01:50:14.639
So I think that's for some people the verification will be more important.

s1106
01:50:14.639 --> 01:50:19.520
Some others are a bit more skeptic and they want to see the actual calculations and numbers and so on.

s1107
01:50:19.520 --> 01:50:27.520
So I think again that the process guides and covers for a multitude of u of possible possibilities.