WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=jJQoVkd5yLg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/jJQoVkd5yLg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.960 --> 00:00:16.400
So, I'm Avi, and I work on product here at Reducto.

s3
00:00:16.400 --> 00:00:22.920
And before I dive in right away, I wanted to give you guys some more context on what Reducto is.

s4
00:00:22.920 --> 00:00:34.680
Uh we're an agentic document platform, which basically means that we help teams turn messy, unstructured documents into data that your tools and your AI agents can understand.

s5
00:00:34.680 --> 00:00:41.080
I'll jump into the agents themselves in a second, but I'd like to talk about where we're at first really quick.

s6
00:00:41.080 --> 00:00:52.280
In the last 3 years, we've processed over 3 billion documents across customers like Harvey, Scale AI, and top five global tech companies and hedge funds.

s7
00:00:52.280 --> 00:01:04.400
Running at that scale means that we've seen basically every way a document can break in production, and we've iterated our product and our models to handle basically anything that you can throw at them.

s8
00:01:04.400 --> 00:01:11.000
And getting AI to work on a clean document with a frontier model is not super hard anymore.

s9
00:01:11.000 --> 00:01:16.320
We basically like you can throw something in the cloud, and you might get like 60% of the way there.

s10
00:01:16.320 --> 00:01:18.560
But the hard part is production.

s11
00:01:18.560 --> 00:01:27.560
Scanned documents, rotated pages, etc. And the model starts hallucinating because a large majority of enterprise data is unstructured.

s12
00:01:27.560 --> 00:01:31.840
And so Reducto is a layer that sits between your model and that data.

s13
00:01:31.840 --> 00:01:41.520
We handle all the complex layouts and post-processing, and that way we end up working with and building a lot of agents ourselves while designing these pipelines.

s14
00:01:41.520 --> 00:01:48.880
So, now that you know a little more about what we do, I'm going to dive into our main topic with something seemingly unrelated.

s15
00:01:48.880 --> 00:01:54.120
Um I visited my girlfriend this past weekend, and she gave me a sticker from the movie Mean Girls.

s16
00:01:54.120 --> 00:01:56.920
And basically, it was an agentified version of it.

s17
00:01:56.920 --> 00:01:58.520
So, it says, "Get in, loser.

s18
00:01:58.520 --> 00:02:00.600
We're building agent-first software."

s19
00:02:00.600 --> 00:02:04.800
And in true software engineer fashion, I had to put a sticker like that on my laptop.

s20
00:02:04.800 --> 00:02:09.200
And she got this from work, but she asked me like, "What is agent-first software?"

s21
00:02:09.200 --> 00:02:11.200
cuz she doesn't work in AI.

s22
00:02:11.200 --> 00:02:17.760
And as as I was like doing so or as I was thinking about it, I realized I have a lot of different answers.

s23
00:02:17.760 --> 00:02:23.600
Building a really good harness, building a really good API, writing or iterating on prompts.

s24
00:02:23.600 --> 00:02:28.840
All of these don't really um land on the agent being put first.

s25
00:02:28.840 --> 00:02:30.959
They're more so just optimizing the model.

s26
00:02:30.959 --> 00:02:35.640
Agent-first software is inherently an architecture problem.

s27
00:02:35.640 --> 00:02:45.160
And so, I think that structure, that architecture, really comes down to one question, and that is who pays for the agent being wrong.

s28
00:02:45.160 --> 00:02:48.560
And to that, I think there are like three buckets that you can put this into.

s29
00:02:48.560 --> 00:02:55.720
So, you have auto mode to start, where you optimize for autonomy, you write a prompt, and you let the agent do whatever it wants.

s30
00:02:55.720 --> 00:03:06.360
And when it messes up, which it will, you won't have any idea because it's really confident about what it's doing, and you're the one that's paying for it being wrong to your customers, your users, etc.

s31
00:03:06.360 --> 00:03:08.160
Then you have user-first.

s32
00:03:08.160 --> 00:03:18.760
This is where you optimize the human interface, give the user all the knobs, and then you build on an agent that can't really do much, which is basically just a chatbot experience.

s33
00:03:18.760 --> 00:03:27.000
You've both capped the user's understanding of the product because now they can't prompt it, and the agent's capability within as well.

s34
00:03:27.000 --> 00:03:33.360
And lastly, the bucket that I think wins here is agent experience in service of the user.

s35
00:03:33.360 --> 00:03:47.480
This is where the agent's capabilities are structured to give the model um rich and scoped capabilities where it can do things that the user really wants, aiding and enabling them, and even where the human can still verify the work that it's doing.

s36
00:03:47.480 --> 00:03:58.720
This framing is what turns agent experience into less of a question of the model and more of a question of context, capability, and overall self-improvement.

s37
00:03:58.960 --> 00:04:03.720
And here's how I discovered these buckets and came to that last point.

s38
00:04:03.720 --> 00:04:09.880
At Reducto, we've been building tools that basically automate end-to-end document work through pipelines.

s39
00:04:09.880 --> 00:04:16.320
Think that we first parse a document, then we classify it, or maybe split it into sections, and then extract from it.

s40
00:04:16.320 --> 00:04:22.080
Um and that's in use cases like invoicing, contract management, and all these things that require human input.

s41
00:04:22.080 --> 00:04:31.440
One thing that I noticed early on was that any of the users, especially our sales team at Reducto, was spending all their time in configuration instead of outcomes.

s42
00:04:31.440 --> 00:04:38.640
And that's a larger problem across the board because legacy interfaces were designed for kind of building that pipeline.

s43
00:04:38.640 --> 00:04:49.240
Meanwhile, in this kind of agentic work world, we're increasingly focused on the final deliverable and how can we get the user to that point as fast as possible.

s44
00:04:49.240 --> 00:04:52.760
So, the clear move to get here for us was an MCP server.

s45
00:04:52.760 --> 00:04:59.720
Let the agent do the heavy lifting and only burden the user with thinking about their use case and writing a prompt.

s46
00:04:59.720 --> 00:05:10.960
To get some validation for the direction, I threw together an MVP with Claude code, and every API endpoint that drove our app was given an MCP tool in one massive file.

s47
00:05:10.960 --> 00:05:12.560
I gave it a really pointed prompt.

s48
00:05:12.560 --> 00:05:15.280
I put it in the Slack, and honestly, it was pretty nice.

s49
00:05:15.280 --> 00:05:18.680
Um to some extent, it should have worked in that setting.

s50
00:05:18.680 --> 00:05:20.400
I knew exactly what I was doing.

s51
00:05:20.400 --> 00:05:26.520
I was building a really curated demo, and that's why the standalone demo told me nothing at all.

s52
00:05:26.520 --> 00:05:33.800
The real test happened when I gave the MCP to everyone on the team, and immediately it broke.

s53
00:05:33.800 --> 00:05:35.680
And here's how.

s54
00:05:35.680 --> 00:05:43.640
Someone would feed it a half-baked description from a customer call or maybe a Notion page, and the agent would just start going.

s55
00:05:43.640 --> 00:05:55.560
It'd build the entire workflow end-to-end, but very confidently, it would tell the user that everything it did was right, but neither the user nor the agent could explain what the workflow actually did.

s56
00:05:55.560 --> 00:05:59.920
And that is the root awakening of auto mode in production.

s57
00:05:59.920 --> 00:06:02.680
The agent never really said that it's not sure.

s58
00:06:02.680 --> 00:06:09.760
It handled with every tool that it had, and there was nowhere for a human to really validate or think about what was going on.

s59
00:06:09.760 --> 00:06:16.200
We built something that could communicate with our system, but we hadn't built something that could actually work with it.

s60
00:06:16.200 --> 00:06:24.840
And you'd think that the fix here is more detail on the system prompt or more model guidance, but realistically it's architecture.

s61
00:06:24.840 --> 00:06:30.600
The whole file that we used to talk to the MCP, I deleted the entire thing in one commit.

s62
00:06:30.600 --> 00:06:33.320
And what replaced it wasn't more tools.

s63
00:06:33.320 --> 00:06:39.040
It was mostly fewer tools, but they carried the structure of actual work.

s64
00:06:39.040 --> 00:06:53.000
So for example, instead of one create step of this workflow to uh call over and over, and just praying that the agent assembled right in the pipeline, I built tools that understood how our internal dashboard or our product works.

s65
00:06:53.000 --> 00:07:01.440
For example, a common use case with Reducto is to route our classify endpoint that classifies documents into an extract node.

s66
00:07:01.440 --> 00:07:05.080
And what that does is basically extract information downstream.

s67
00:07:05.080 --> 00:07:14.960
And instead of basically letting the model guess this or figure it out through the prompt, I turned it into a single classify to extract tool.

s68
00:07:14.960 --> 00:07:23.600
And after each step, the agent gets contextual guidance on what actually comes next, with certain tools even chaining automatically like this one.

s69
00:07:23.600 --> 00:07:29.720
Rather than the agent having to guess or think about what is right versus being told what's right.

s70
00:07:29.720 --> 00:07:34.000
Now the agent doesn't get to freelance the parts that we know are true.

s71
00:07:34.000 --> 00:07:38.960
And since agents are stateless, we even added a snapshot tool.

s72
00:07:38.960 --> 00:07:50.280
So this is one call that basically hands back the live state of our pipeline from the context of the entire flow to the errors that were surfaced with like really specific error codes.

s73
00:07:50.280 --> 00:08:04.160
And that way they can reference what's actually there instead of guessing variable names, hallucinating, etc. By moving context out of the prompt and into structure, the agent stopped guessing and started referencing.

s74
00:08:04.160 --> 00:08:18.720
And because the tools only let pipelines take the shape that we as the engineers on the product wanted, the human could follow exactly what happened and we could explain the agent's outputs to downstream users of this MCP tool.

s75
00:08:18.720 --> 00:08:30.160
So, to avoid auto mode, you don't hand your agent an API, you hand it tools that carry the overall expertise such as the right patterns, the state, and the validators

s76
00:08:30.160 --> 00:08:35.360
so the correct path also becomes the simplest path that the MCP can take.

s77
00:08:35.360 --> 00:08:37.440
So, we can build good pipelines now.

s78
00:08:37.440 --> 00:08:38.800
We're done, right?

s79
00:08:38.800 --> 00:08:40.200
Not really.

s80
00:08:40.200 --> 00:08:46.400
Thinking back to auto mode, the agent was confidently wrong and no one could see it.

s81
00:08:46.400 --> 00:08:49.000
Um that what we just talked about fixed that for building.

s82
00:08:49.000 --> 00:08:53.560
The tools made the structure really legible and the human could follow what the agent did.

s83
00:08:53.560 --> 00:08:56.000
But now the problem showed up at runtime.

s84
00:08:56.000 --> 00:09:00.560
When the outcome itself that the agent is trying to build goes wrong, who can tell?

s85
00:09:00.560 --> 00:09:09.720
For example, if we had a really bad schema or maybe the user didn't really specify the actual use case, how can we tell that the agent did something wrong?

s86
00:09:09.720 --> 00:09:12.040
And with agents it goes wrong quietly.

s87
00:09:12.040 --> 00:09:23.839
So, instead of a seg fault or some actual error, um the pipeline runs across 100 documents and maybe it misses scan on page 30 or maybe the user didn't specify what types of documents

s88
00:09:23.839 --> 00:09:26.600
were going into the pipeline.

s89
00:09:26.600 --> 00:09:30.480
And basically both of these things are the same failure.

s90
00:09:30.480 --> 00:09:38.440
The agent itself is very confidently wrong and the uncertainty that the agent has is completely invisible.

s91
00:09:38.440 --> 00:09:46.560
So, everything that we did to fix this basically came down to one uh change, which was making uncertainty really legible.

s92
00:09:46.560 --> 00:09:50.600
In our system or with Reducto, uh failures carry information.

s93
00:09:50.600 --> 00:10:02.240
So, with Extract, we output confidence scores, bounding boxes, all of this information about what went wrong, and we were able to build tools that feed that information back into the model,

s94
00:10:02.240 --> 00:10:08.560
which actually helped optimize the extract schema over time, and get more accurate document outputs.

s95
00:10:08.560 --> 00:10:11.480
But, that's just the outcome uncertainty.

s96
00:10:11.480 --> 00:10:14.920
The agent also has to surface its own uncertainty.

s97
00:10:14.920 --> 00:10:24.920
When it hits something ambiguous, um two ways to read a field or a document type that it hasn't seen, it shouldn't guess, and it shouldn't just pause exactly where it is.

s98
00:10:24.920 --> 00:10:30.320
And rather, it should ask, "I can read this two ways," and tell the user, "What did you mean?"

s99
00:10:30.320 --> 00:10:47.160
And so, basically in our evals, we started failing the agent for building any shaky assumptions, and we gave our harness uh prompt guidance to use the ask user question tools, etc., that Cloud Code and other like frontier harnesses offer, instead of guessing.

s100
00:10:47.160 --> 00:10:49.839
So, we rewarded asking instead of guessing.

s101
00:10:49.839 --> 00:10:53.320
And judgment doesn't live in the agent alone.

s102
00:10:53.320 --> 00:11:03.520
This entire pipeline is still overseen by the user, and the confidence and the reasoning of the agent was then outputted as basically prompt guidance in the MCP.

s103
00:11:03.520 --> 00:11:14.240
So, now every time you make something new, it explains to the user what it did and why, so the user can then review it, inherently acting in service of the user.

s104
00:11:14.240 --> 00:11:21.480
The agent basically satisfies what you wrote, and not the um outcome that you wanted.

s105
00:11:21.480 --> 00:11:32.000
So, what happens if you make the uncertainty basically a bit more legible, and you keep the final judgment where it can actually be accountable in the hands of the human,

s106
00:11:32.000 --> 00:11:44.360
then that one clarifying question does the work of all three, giving the model context, capability, and also adding a human in the loop to actually build that judgment and verify things as well.

s107
00:11:44.360 --> 00:11:53.200
So, going back to kind of all of these failure modes overall, we had initially an auto mode that nothing nobody could inspect.

s108
00:11:53.200 --> 00:11:56.760
We had wrong answers that basically arrived in silence.

s109
00:11:56.760 --> 00:12:01.880
And then an agent that passed the test by reward hacking or hiding what it did.

s110
00:12:01.880 --> 00:12:10.640
And underneath, they're basically one shape where the agent had more autonomy and more access than the human could verify.

s111
00:12:10.640 --> 00:12:14.800
And that intelligence lives in the construction overall.

s112
00:12:14.800 --> 00:12:26.560
So, building the pipeline, generating the schema, and pulling in the right context is where the agent is more most powerful and where a mistake is the cheapest because the human is right there.

s113
00:12:26.560 --> 00:12:35.520
By building for that interface, we were able to also have user satisfaction go up with the MCP because they could really see what was going on.

s114
00:12:35.520 --> 00:12:44.560
And by using Reducto for the document layer, we can guarantee accurate parses and extractions for parts of the pipeline that we can't trust agents with.

s115
00:12:44.560 --> 00:12:53.200
Where the agent operates, it earns trust by being verifiable, correct, and building trust in service of the user.

s116
00:12:53.200 --> 00:12:59.560
And the whole conversation we had about legibility overall is what inherently pays off.

s117
00:12:59.560 --> 00:13:10.000
Agents are just intelligence in a box and the harness that you surround them with is what provides them with the ability to deliver these outcomes downstream.

s118
00:13:10.000 --> 00:13:23.920
And so, once we reach this point, we focus on instrumentation where every MCP session, the prompts that were sent from the model were logged, and we even asked like our sales team and other people on the team to basically export their cloud code sessions

s119
00:13:23.920 --> 00:13:26.680
to understand how how was behaving.

s120
00:13:26.680 --> 00:13:32.640
Where humans overrode the agent, where confidence of the model ran low, and where it asked questions.

s121
00:13:32.640 --> 00:13:41.160
And because we added all this instrumentation, um it flowed back into the long-term guidance that the MCP had overall.

s122
00:13:41.160 --> 00:13:47.480
So, the system got better at building pipelines by building more pipelines without us rewriting by hand.

s123
00:13:47.480 --> 00:13:53.240
This then became a self-improving loop per org or per user that was building these pipelines.

s124
00:13:53.240 --> 00:14:05.160
The foundation became overall the same decision where we kept the model very guided and very inspectable, and the information around it became everything else that actually guided the model.

s125
00:14:05.160 --> 00:14:17.000
And so, one thing that kind of told me that we were doing something right with these improvements is I never like explicitly told the sales team to use the agent to build pipelines,

s126
00:14:17.000 --> 00:14:22.000
but rather I just exposed it in our Cloud Enterprise as product tools.

s127
00:14:22.000 --> 00:14:27.280
And the sales team and various other people on the team started basically wiring it into their workflows.

s128
00:14:27.280 --> 00:14:37.640
So, before they had a customer call, they would use CircleBack or Slack and basically learn about the customer and then use the MCP to basically ask them questions about the customer

s129
00:14:37.640 --> 00:14:41.080
and then build a pipeline for that customer downstream.

s130
00:14:41.080 --> 00:14:53.000
So, when an agent's experience was built really in service of the users that were using it, we weren't pushing people directly toward it and rather they reached for it and actually started using it as a company-wide

s131
00:14:53.000 --> 00:14:54.000
resource.

s132
00:14:54.000 --> 00:15:05.520
And that's where they also started to trust the output enough to put this in front of high-value customers because the demo didn't just work, but the human actually said, "Hey, I built this.

s133
00:15:05.520 --> 00:15:06.680
Come check it out."

s134
00:15:06.680 --> 00:15:09.240
And it actually paid off in these sales calls.

s135
00:15:09.240 --> 00:15:13.960
And I hope the Slack reactions can also help gauge that a little bit.

s136
00:15:13.960 --> 00:15:25.320
So, overall, a few months ago I would have told you that agent first is means that you have the best model, the best inference speed or intelligence of the model.

s137
00:15:25.320 --> 00:15:33.640
And with models getting better from closed source to open source, from expensive to cheap, I think it now means almost the opposite of what it sounds.

s138
00:15:33.640 --> 00:15:36.480
To some extent, the agent doesn't go first.

s139
00:15:36.480 --> 00:15:45.440
And to build for agent experience means you have to consider the agent being in the loop with the people who have to answer for the work downstream.

s140
00:15:45.440 --> 00:15:49.000
So, it's not just what does the agent connect to?

s141
00:15:49.000 --> 00:15:54.280
Everyone's agent connects to everything nowadays, and Claude can make a new connector for you immediately.

s142
00:15:54.280 --> 00:16:03.160
But, regard Beyond that, it's when your agent is really confidently wrong, who finds out and how fast did they find out?

s143
00:16:03.160 --> 00:16:10.000
If you build for that, then you can build for something that people will actually trust with real work.

s144
00:16:10.000 --> 00:16:11.840
And yeah, that's about it.

s145
00:16:11.840 --> 00:16:12.880
Thank you, guys.

s146
00:16:12.880 --> 00:16:17.480
And also, we're at Reducto's at a booth P8, and we are hiring for a lot of positions.

s147
00:16:17.480 --> 00:16:21.200
So, please come check it out at reductoai.notcurious.

s148
00:16:21.560 --> 00:16:23.720
Thank you.

s149
00:16:36.921 --> 00:16:38.921
[music]