Your Finance Agent's Bottleneck Is You — Ramana Siddanth Emani, Auditoria AI

AI Engineer · 13 min · 146 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello everyone.
00:13Welcome to this session about your finance agent's bottleneck is you.
00:19So, sorry for the rude title.
00:21I don't mean to call the audience here the bottlenecks, but I'm here to talk about the harnesses that you guys are developing and using these internal harnesses to build your production agents.
00:34So, my name is Siddhant Imani and I'm a data scientist at Auditoria AI and we build production agents for finance.
00:44So, if you're a CFO in the audience, I would love to speak to you after the session.
00:49This talk is in between the harness engineering track and AI for finance.
00:55So, this talk is mostly about identifying the bottlenecks within your developer and if you're a developer yourself, how do you be 10x productive with the agent harnesses that you're using.
01:10So, all of us have seen, you know, beautiful demos in this AI engineer's world fair.
01:16But, once these demos are promoted to pilots and you start onboarding new customers, the agent has never seen these future data.
01:27So, all of us know production bugs are very high and production guards built by the hour.
01:36So, that's a hard fact.
01:38And writing code is very easy.
01:41So, shipping beautiful demos and showing it to a lot of people is very easy right nowadays.
01:49So, what is the problem?
01:51And why do these demos fail in production?
01:54Is it the model?
01:56Do you need a better model?
01:58Fable 5, perhaps?
02:00Or do you need faster GPUs?
02:02Or do you need a better framework?
02:05Maybe.
02:06Or your RALF loops are not working properly.
02:10So, what is the answer?
02:13If you wait 3 and 1/2 months, we are awarded with a new model in the market.
02:17So, we can easily swap models.
02:20If you wait perhaps 1 year, we have new chips.
02:23We have faster GPUs.
02:25And again, writing code is easy.
02:28So, we have new from frameworks every day.
02:31So, you can swap your framework every now and then.
02:35So, how do we in real time fix these production bugs?
02:40The answer is your dev loop velocity.
02:43The model capability increases very exponentially.
02:46And the developers have to spend a lot of time every day to automate your developer loop.
02:55So, I'm talking about four primitives here.
02:59All of you need to think about loops.
03:01And at the end of the session, I hope you can 10x your production code.
03:07So, first we have sub agents.
03:10Nowadays, whatever harness you're using, you can spawn new sub agents.
03:14You can have a You can have an army of them.
03:17And get work trees are your best friend.
03:21So, think of work trees as isolated folders.
03:24And inside these folders, the agent writes whatever code it's generating.
03:29So, you want these work trees to be in parallel.
03:32So, the sub agents are doing independent tasks and are not fighting over the same thing.
03:39Second, we have skills.
03:40These are your organization secret recipes.
03:43So, make sure you have a lot of skills because these skills, once you start say giving it to your agents, the agents will always make sure to use the correct and proper workflows
03:56to solve whatever production bug you're facing.
04:00And of course, all of us have seen a lot of MCP tools being shipped into the market right now.
04:05Everybody says we can, yeah, the agent can connect to whatever MCP tool and whatever third-party server there is.
04:12And your client data can live in any system you want.
04:17And at the end of the day, if you have a lot of sub-agents, you have a lot of work to orchestrate.
04:23So, minimal UX is the key here.
04:29Let's look at the sub-agents.
04:31With you as the orchestrator, you can have, let's say, with 48 GB of RAM on your MacBook, you can have 50 active work trees.
04:38That is 50 active sub-agents working independently on different tasks.
04:43So, where do these tasks come from?
04:46So, let's say the production software you're going to ship has a lot of bugs that your QA is reporting.
04:51So, all the Jira tickets can be thought of in a separate different work tree.
04:56So, different work trees are handled by a separate agent, and these agents can spawn multiple sub-agents to solve that particular task.
05:05You don't want to queue up your tasks because the agent is will do that a lot better than you.
05:15Let's look at, you know, um, an example harness.
05:19What if the QA reports a lot of bug tickets, and somehow magically there is an agent which parses the requirements, does a root cause analysis, pulls all the traces, pulls all the logs,
05:35puts all this in a separate work tree, does the TDD, does the, implements the fix.
05:42Because it's in your local system, you have to do test scripts, local end-to-end testing.
05:47You create a PR.
05:49You submit the PR to your team for review.
05:52And after review, you merge it into your master branch, let's say.
05:56After merging it, obviously, you have to build a Docker image, deploy it into your development environment, test it, again ship build an image to your stage environment, test it, deploy it to stage.
06:11And then you go back to the QA saying, "Here you go.
06:14You can test it now."
06:16So, I would like to ask a question in the audience, um at what points do you think the human contact is required in this, um steps 1 to 9?
06:29So, I would say the human is only required at steps 1 and 9 because the in-between steps, the agent can do a lot better work.
06:40There needs to be a human to see what work the agent is doing.
06:44And then needs to be a human at the end to validate after the work is being shipped to stage.
06:52And obviously, we need minimal UX because humans love minimal UX.
06:57So, in the image, if um you can if you squint your eyes and see, the image shows um the production agent software that you're building, the project dashboards which shows all your Kubernetes services, pods, examples,
07:12all the logs, system logs, all your Jira tickets, all your GitHub PRs, and maybe a cloud code session at the bottom.
07:22So, this is basically a macOS widget, and you don't need to open multiple windows to do all of this work.
07:30A developer does like variety of things in their software development life cycle.
07:34So, you can use just this one widget to do a lot of things.
07:38So, you can see from the graph also, the number of neck rotations to ship one change like reduces a lot drastically.
07:45And I imagine all of you have like two to three monitors on your table and you just keep rotating your neck orchestrating these agents.
07:56So, Auditoria works in finance.
07:58So, there's a lot of regulation and policies happening in finance right now.
08:03So, what does it look like for orchestrating a team of sub agents in the finance sector?
08:10If we take AI out of the picture, usually what happens is you have a human auditor which reviews the code and you have a controller which signs off under your socks compliance.
08:21And reviewing agent to agent, it it doesn't Where do you keep the accountability?
08:28If something goes wrong in production, you can't say Cloud is doing this.
08:32Something is wrong.
08:34So, but let's say you have all these sub agents and you're using this harnesses to fix bugs in real time.
08:43What is the bottleneck?
08:44It becomes a human attention because you yourself have to orchestrate all these different tasks.
08:52And moving fast and breaking things in sector in the finance sector is a lot different.
09:00So, let's look at part two, which is removing yourself from the loop.
09:06Till now I've been saying a human is required to see what the agent is doing and at the end also to validate what the agent has done.
09:14But with the self-improvement of the agent and model capabilities these days, we get Fable 5 and Mythos 5 and GPT 5.6 also.
09:26So, what does it look like when you have this recursive self-improvement in your internal developer developer harnesses?
09:35So, all your production failures become input.
09:41So, let's say you automate keep automating these cell developer harnesses every day and you ask the agent to upgrade itself essentially.
09:51So, you do a task.
09:52You let the loop run.
09:54Let's say one or two days.
09:55You solve five to six bug tickets.
09:58And you just tell the agent to analyze all the bottlenecks in this process.
10:04Make a list of them.
10:05And somehow slowly keep removing these bottlenecks every day.
10:10At the end of one month, let's say, you have a really nice self-automated loop where you just type in one sentence and just say fix this bug for me.
10:21And the agent goes off, connects to all your database systems, fetches all the logs, traces, tickets, and ships it and migrates it migrates it to the Jira to QA pipeline.
10:33And you can just book a vacation maybe or work from home.
10:40And what does it look like internally and what happens when you stare less and ship more?
10:47Nowadays, how many of you know you can give goals to your agents?
10:52You can just set a goal and forget about it.
10:56Anybody?
10:57Nice.
10:59Um so, what if you combine goals and loops?
11:02You can just set a goal saying there is some data discrepancy in this report.
11:08And in the production bug like the source data is not matching with what the agent has generated.
11:16So, you can just set a goal to fix this, look into this, set a loop.
11:20You can even close like close your laptop because you can do it from your phone nowadays.
11:25And if you look at the last but one point, which is dreaming, um let's say a lot of are using your production software and they are doing the same type of patterns
11:38and they're facing the same type of problems.
11:41So, you let the agent dream like humans dream in the background so that it collects all the sessions that your customers are using and compacts it into a set of data points which your system can use and basically
11:54[snorts]
11:54upgrade yourself.
11:57So, with a combination of all these features, basically you can essentially remove yourself out of the loop.
12:05But, as I said before, the developers do a lot of variety things in their software development life cycle and sitting behind a desk from 9:00 to 5:00 and just writing code is not
12:18valid anymore.
12:22So, just an overview of what I've covered till now in the session.
12:26You can have a team of sub agents working in parallel work trees.
12:30You can have skills, your organizational secret recipes, your customers recipes.
12:36You can give all of these to an agent.
12:39Your agent can connect to whatever third-party server there is.
12:42It can be a logging system, it can be an authentication gateway.
12:46And you just compress all of this into one pane of glass because minimal UX is the key.
12:53And you can set goals and loops for autonomy.
12:56If you think this particular work can be done by the agent a lot better, you can just ship it to the agent.
13:04Always have the human as a verifier, but not the throughput ceiling because human attention is very limited.
13:13So, thank you for your time and thank you for your Thank you for I hope you um learned something from the session.
13:22Thank you.
13:24[applause]
13:40[music]
13:41Hey.