WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=z0sh8HyTrDo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/z0sh8HyTrDo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.600
Hello everyone.

s3
00:00:13.600 --> 00:00:19.120
Welcome to this session about your finance agent's bottleneck is you.

s4
00:00:19.120 --> 00:00:21.520
So, sorry for the rude title.

s5
00:00:21.520 --> 00:00:34.000
I don't mean to call the audience here the bottlenecks, but I'm here to talk about the harnesses that you guys are developing and using these internal harnesses to build your production agents.

s6
00:00:34.000 --> 00:00:44.120
So, my name is Siddhant Imani and I'm a data scientist at Auditoria AI and we build production agents for finance.

s7
00:00:44.120 --> 00:00:49.640
So, if you're a CFO in the audience, I would love to speak to you after the session.

s8
00:00:49.960 --> 00:00:55.680
This talk is in between the harness engineering track and AI for finance.

s9
00:00:55.680 --> 00:01:09.440
So, this talk is mostly about identifying the bottlenecks within your developer and if you're a developer yourself, how do you be 10x productive with the agent harnesses that you're using.

s10
00:01:10.760 --> 00:01:16.600
So, all of us have seen, you know, beautiful demos in this AI engineer's world fair.

s11
00:01:16.600 --> 00:01:27.720
But, once these demos are promoted to pilots and you start onboarding new customers, the agent has never seen these future data.

s12
00:01:27.720 --> 00:01:36.560
So, all of us know production bugs are very high and production guards built by the hour.

s13
00:01:36.560 --> 00:01:38.960
So, that's a hard fact.

s14
00:01:38.960 --> 00:01:41.600
And writing code is very easy.

s15
00:01:41.600 --> 00:01:48.440
So, shipping beautiful demos and showing it to a lot of people is very easy right nowadays.

s16
00:01:49.240 --> 00:01:51.360
So, what is the problem?

s17
00:01:51.360 --> 00:01:54.840
And why do these demos fail in production?

s18
00:01:54.840 --> 00:01:56.560
Is it the model?

s19
00:01:56.560 --> 00:01:58.080
Do you need a better model?

s20
00:01:58.080 --> 00:02:00.120
Fable 5, perhaps?

s21
00:02:00.120 --> 00:02:02.360
Or do you need faster GPUs?

s22
00:02:02.360 --> 00:02:05.360
Or do you need a better framework?

s23
00:02:05.360 --> 00:02:06.560
Maybe.

s24
00:02:06.560 --> 00:02:10.520
Or your RALF loops are not working properly.

s25
00:02:10.520 --> 00:02:13.200
So, what is the answer?

s26
00:02:13.200 --> 00:02:17.920
If you wait 3 and 1/2 months, we are awarded with a new model in the market.

s27
00:02:17.920 --> 00:02:20.400
So, we can easily swap models.

s28
00:02:20.400 --> 00:02:23.720
If you wait perhaps 1 year, we have new chips.

s29
00:02:23.720 --> 00:02:25.920
We have faster GPUs.

s30
00:02:25.920 --> 00:02:28.680
And again, writing code is easy.

s31
00:02:28.680 --> 00:02:31.720
So, we have new from frameworks every day.

s32
00:02:31.720 --> 00:02:35.280
So, you can swap your framework every now and then.

s33
00:02:35.280 --> 00:02:40.520
So, how do we in real time fix these production bugs?

s34
00:02:40.520 --> 00:02:43.360
The answer is your dev loop velocity.

s35
00:02:43.360 --> 00:02:46.520
The model capability increases very exponentially.

s36
00:02:46.520 --> 00:02:55.200
And the developers have to spend a lot of time every day to automate your developer loop.

s37
00:02:55.800 --> 00:02:59.760
So, I'm talking about four primitives here.

s38
00:02:59.760 --> 00:03:01.680
All of you need to think about loops.

s39
00:03:01.680 --> 00:03:06.720
And at the end of the session, I hope you can 10x your production code.

s40
00:03:07.840 --> 00:03:10.160
So, first we have sub agents.

s41
00:03:10.160 --> 00:03:14.200
Nowadays, whatever harness you're using, you can spawn new sub agents.

s42
00:03:14.200 --> 00:03:17.480
You can have a You can have an army of them.

s43
00:03:17.480 --> 00:03:21.400
And get work trees are your best friend.

s44
00:03:21.400 --> 00:03:24.360
So, think of work trees as isolated folders.

s45
00:03:24.360 --> 00:03:29.400
And inside these folders, the agent writes whatever code it's generating.

s46
00:03:29.400 --> 00:03:32.360
So, you want these work trees to be in parallel.

s47
00:03:32.360 --> 00:03:39.040
So, the sub agents are doing independent tasks and are not fighting over the same thing.

s48
00:03:39.360 --> 00:03:40.560
Second, we have skills.

s49
00:03:40.560 --> 00:03:43.519
These are your organization secret recipes.

s50
00:03:43.519 --> 00:03:56.840
So, make sure you have a lot of skills because these skills, once you start say giving it to your agents, the agents will always make sure to use the correct and proper workflows

s51
00:03:56.840 --> 00:04:00.160
to solve whatever production bug you're facing.

s52
00:04:00.160 --> 00:04:05.840
And of course, all of us have seen a lot of MCP tools being shipped into the market right now.

s53
00:04:05.840 --> 00:04:12.600
Everybody says we can, yeah, the agent can connect to whatever MCP tool and whatever third-party server there is.

s54
00:04:12.600 --> 00:04:17.680
And your client data can live in any system you want.

s55
00:04:17.720 --> 00:04:23.520
And at the end of the day, if you have a lot of sub-agents, you have a lot of work to orchestrate.

s56
00:04:23.520 --> 00:04:27.880
So, minimal UX is the key here.

s57
00:04:29.280 --> 00:04:31.080
Let's look at the sub-agents.

s58
00:04:31.080 --> 00:04:38.919
With you as the orchestrator, you can have, let's say, with 48 GB of RAM on your MacBook, you can have 50 active work trees.

s59
00:04:38.919 --> 00:04:43.720
That is 50 active sub-agents working independently on different tasks.

s60
00:04:43.720 --> 00:04:46.360
So, where do these tasks come from?

s61
00:04:46.360 --> 00:04:51.960
So, let's say the production software you're going to ship has a lot of bugs that your QA is reporting.

s62
00:04:51.960 --> 00:04:56.960
So, all the Jira tickets can be thought of in a separate different work tree.

s63
00:04:56.960 --> 00:05:05.520
So, different work trees are handled by a separate agent, and these agents can spawn multiple sub-agents to solve that particular task.

s64
00:05:05.520 --> 00:05:12.919
You don't want to queue up your tasks because the agent is will do that a lot better than you.

s65
00:05:15.480 --> 00:05:19.680
Let's look at, you know, um, an example harness.

s66
00:05:19.680 --> 00:05:35.080
What if the QA reports a lot of bug tickets, and somehow magically there is an agent which parses the requirements, does a root cause analysis, pulls all the traces, pulls all the logs,

s67
00:05:35.080 --> 00:05:42.200
puts all this in a separate work tree, does the TDD, does the, implements the fix.

s68
00:05:42.200 --> 00:05:47.960
Because it's in your local system, you have to do test scripts, local end-to-end testing.

s69
00:05:47.960 --> 00:05:49.560
You create a PR.

s70
00:05:49.560 --> 00:05:52.840
You submit the PR to your team for review.

s71
00:05:52.840 --> 00:05:56.760
And after review, you merge it into your master branch, let's say.

s72
00:05:56.760 --> 00:06:11.520
After merging it, obviously, you have to build a Docker image, deploy it into your development environment, test it, again ship build an image to your stage environment, test it, deploy it to stage.

s73
00:06:11.520 --> 00:06:14.200
And then you go back to the QA saying, "Here you go.

s74
00:06:14.200 --> 00:06:16.720
You can test it now."

s75
00:06:16.720 --> 00:06:29.440
So, I would like to ask a question in the audience, um at what points do you think the human contact is required in this, um steps 1 to 9?

s76
00:06:29.960 --> 00:06:40.720
So, I would say the human is only required at steps 1 and 9 because the in-between steps, the agent can do a lot better work.

s77
00:06:40.720 --> 00:06:44.440
There needs to be a human to see what work the agent is doing.

s78
00:06:44.440 --> 00:06:50.280
And then needs to be a human at the end to validate after the work is being shipped to stage.

s79
00:06:52.720 --> 00:06:57.000
And obviously, we need minimal UX because humans love minimal UX.

s80
00:06:57.000 --> 00:07:12.560
So, in the image, if um you can if you squint your eyes and see, the image shows um the production agent software that you're building, the project dashboards which shows all your Kubernetes services, pods, examples,

s81
00:07:12.560 --> 00:07:22.080
all the logs, system logs, all your Jira tickets, all your GitHub PRs, and maybe a cloud code session at the bottom.

s82
00:07:22.080 --> 00:07:30.160
So, this is basically a macOS widget, and you don't need to open multiple windows to do all of this work.

s83
00:07:30.160 --> 00:07:34.720
A developer does like variety of things in their software development life cycle.

s84
00:07:34.720 --> 00:07:38.760
So, you can use just this one widget to do a lot of things.

s85
00:07:38.760 --> 00:07:45.800
So, you can see from the graph also, the number of neck rotations to ship one change like reduces a lot drastically.

s86
00:07:45.800 --> 00:07:54.480
And I imagine all of you have like two to three monitors on your table and you just keep rotating your neck orchestrating these agents.

s87
00:07:56.880 --> 00:07:58.960
So, Auditoria works in finance.

s88
00:07:58.960 --> 00:08:03.480
So, there's a lot of regulation and policies happening in finance right now.

s89
00:08:03.480 --> 00:08:10.400
So, what does it look like for orchestrating a team of sub agents in the finance sector?

s90
00:08:10.400 --> 00:08:21.520
If we take AI out of the picture, usually what happens is you have a human auditor which reviews the code and you have a controller which signs off under your socks compliance.

s91
00:08:21.520 --> 00:08:28.560
And reviewing agent to agent, it it doesn't Where do you keep the accountability?

s92
00:08:28.560 --> 00:08:32.719
If something goes wrong in production, you can't say Cloud is doing this.

s93
00:08:32.719 --> 00:08:34.680
Something is wrong.

s94
00:08:34.680 --> 00:08:43.080
So, but let's say you have all these sub agents and you're using this harnesses to fix bugs in real time.

s95
00:08:43.080 --> 00:08:44.960
What is the bottleneck?

s96
00:08:44.960 --> 00:08:51.280
It becomes a human attention because you yourself have to orchestrate all these different tasks.

s97
00:08:52.120 --> 00:08:58.680
And moving fast and breaking things in sector in the finance sector is a lot different.

s98
00:09:00.680 --> 00:09:06.080
So, let's look at part two, which is removing yourself from the loop.

s99
00:09:06.080 --> 00:09:14.000
Till now I've been saying a human is required to see what the agent is doing and at the end also to validate what the agent has done.

s100
00:09:14.000 --> 00:09:26.480
But with the self-improvement of the agent and model capabilities these days, we get Fable 5 and Mythos 5 and GPT 5.6 also.

s101
00:09:26.480 --> 00:09:35.280
So, what does it look like when you have this recursive self-improvement in your internal developer developer harnesses?

s102
00:09:35.280 --> 00:09:41.000
So, all your production failures become input.

s103
00:09:41.000 --> 00:09:51.200
So, let's say you automate keep automating these cell developer harnesses every day and you ask the agent to upgrade itself essentially.

s104
00:09:51.200 --> 00:09:52.440
So, you do a task.

s105
00:09:52.440 --> 00:09:54.200
You let the loop run.

s106
00:09:54.200 --> 00:09:55.760
Let's say one or two days.

s107
00:09:55.760 --> 00:09:58.560
You solve five to six bug tickets.

s108
00:09:58.560 --> 00:10:04.000
And you just tell the agent to analyze all the bottlenecks in this process.

s109
00:10:04.000 --> 00:10:05.600
Make a list of them.

s110
00:10:05.600 --> 00:10:10.600
And somehow slowly keep removing these bottlenecks every day.

s111
00:10:10.600 --> 00:10:21.200
At the end of one month, let's say, you have a really nice self-automated loop where you just type in one sentence and just say fix this bug for me.

s112
00:10:21.200 --> 00:10:33.000
And the agent goes off, connects to all your database systems, fetches all the logs, traces, tickets, and ships it and migrates it migrates it to the Jira to QA pipeline.

s113
00:10:33.000 --> 00:10:39.640
And you can just book a vacation maybe or work from home.

s114
00:10:40.200 --> 00:10:47.440
And what does it look like internally and what happens when you stare less and ship more?

s115
00:10:47.440 --> 00:10:52.120
Nowadays, how many of you know you can give goals to your agents?

s116
00:10:52.120 --> 00:10:55.760
You can just set a goal and forget about it.

s117
00:10:56.000 --> 00:10:57.800
Anybody?

s118
00:10:57.800 --> 00:10:59.400
Nice.

s119
00:10:59.400 --> 00:11:02.960
Um so, what if you combine goals and loops?

s120
00:11:02.960 --> 00:11:08.720
You can just set a goal saying there is some data discrepancy in this report.

s121
00:11:08.720 --> 00:11:16.040
And in the production bug like the source data is not matching with what the agent has generated.

s122
00:11:16.040 --> 00:11:20.320
So, you can just set a goal to fix this, look into this, set a loop.

s123
00:11:20.320 --> 00:11:25.960
You can even close like close your laptop because you can do it from your phone nowadays.

s124
00:11:25.960 --> 00:11:38.480
And if you look at the last but one point, which is dreaming, um let's say a lot of are using your production software and they are doing the same type of patterns

s125
00:11:38.480 --> 00:11:41.480
and they're facing the same type of problems.

s126
00:11:41.480 --> 00:11:54.613
So, you let the agent dream like humans dream in the background so that it collects all the sessions that your customers are using and compacts it into a set of data points which your system can use and basically

s127
00:11:54.613 --> 00:11:54.920
[snorts]

s128
00:11:54.920 --> 00:11:57.320
upgrade yourself.

s129
00:11:57.480 --> 00:12:05.240
So, with a combination of all these features, basically you can essentially remove yourself out of the loop.

s130
00:12:05.240 --> 00:12:18.160
But, as I said before, the developers do a lot of variety things in their software development life cycle and sitting behind a desk from 9:00 to 5:00 and just writing code is not

s131
00:12:18.160 --> 00:12:20.560
valid anymore.

s132
00:12:22.040 --> 00:12:26.120
So, just an overview of what I've covered till now in the session.

s133
00:12:26.120 --> 00:12:30.560
You can have a team of sub agents working in parallel work trees.

s134
00:12:30.560 --> 00:12:36.800
You can have skills, your organizational secret recipes, your customers recipes.

s135
00:12:36.800 --> 00:12:39.320
You can give all of these to an agent.

s136
00:12:39.320 --> 00:12:42.680
Your agent can connect to whatever third-party server there is.

s137
00:12:42.680 --> 00:12:46.960
It can be a logging system, it can be an authentication gateway.

s138
00:12:46.960 --> 00:12:53.760
And you just compress all of this into one pane of glass because minimal UX is the key.

s139
00:12:53.760 --> 00:12:56.520
And you can set goals and loops for autonomy.

s140
00:12:56.520 --> 00:13:04.600
If you think this particular work can be done by the agent a lot better, you can just ship it to the agent.

s141
00:13:04.600 --> 00:13:12.400
Always have the human as a verifier, but not the throughput ceiling because human attention is very limited.

s142
00:13:13.280 --> 00:13:22.520
So, thank you for your time and thank you for your Thank you for I hope you um learned something from the session.

s143
00:13:22.520 --> 00:13:24.349
Thank you.

s144
00:13:24.349 --> 00:13:26.349
[applause]

s145
00:13:40.069 --> 00:13:41.560
[music]

s146
00:13:41.560 --> 00:13:43.560
Hey.