The Future Is Domain-Specific Agents - Justin Schroeder, StandardAgents

AI Engineer · 30 min · 390 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:02Okay, so I'm going to be talking about domain-specific agents and why I really think that they are going to play an unbelievably important role in the future of AI
00:14and in the future of how we build agents.
00:16To get started real quick, my name's Justin Schrader.
00:19Uh you can find me on X at JP Schrader.
00:23And um I work at a small company called Standard Agents, which nobody's heard of right now cuz we're still kind of in stealth mode.
00:29Um after this talk, if you're interested, feel free to reach out to me and uh I can let you know a little bit more.
00:37Mostly, I'm known for doing a lot of different open-source projects.
00:41Uh Dmux, which is a great multiplexer for all of your coding agents.
00:45Uh ArrowJS, which is sort of like a UI framework, sort of like React for um the agentic era.
00:51A bunch more that I won't get into, but you know, maybe check them out if you're interested.
00:56Okay.
00:57I think we can all agree that the moment in time that we are in is very similar to the Industrial Revolution.
01:04Um in fact, it might be like an accelerated Industrial Revolution.
01:07Maybe it's a bigger deal, but it's certainly not smaller.
01:10I probably don't need to convince you of that if you're listening to one of these talks.
01:14Um but that is the moment we find us find ourselves in.
01:17So, I actually think it's helpful to go back and sort of look at what was the key catalyst of the Industrial Revolution.
01:24And ultimately, it was that we learned how to harness energy with machines.
01:29We learned how to harness energy with machines.
01:32And what's interesting is that in this next era, we are essentially learning to harness intelligence with agents.
01:41And agents, I think, can be thought of a little bit like the machine of yesteryear.
01:46It's the thing that is going to use the intelligence.
01:50Not so much us, but the agents.
01:55What's interesting about that is I bet if I was in an actual room with you guys and and we all put up our hands, I bet a lot of you when I say what is an agent instantly
02:04have examples that pop into mind, but also probably can't pull out a definition immediately.
02:10Some of you maybe can, um but the reality is that we haven't even coalesced on a definition of what an agent is, even though we're well into the agentic era at this point.
02:23And I think that's kind of interesting.
02:25Um here's my definition.
02:26You can feel free to agree with it or not, but agents are deterministic software that harness the non-deterministic results produced by models in pursuit of some desired objective.
02:39Now, deterministic software might make you think more like a harness, and I actually think the distinction between an agent and a harness is really pedantic, not very helpful, um and for the most part, in most cases,
02:55you can just conflate the two.
02:56A harness is an agent and an agent is a harness, okay?
02:59And for the for the purposes of this talk, we're going to go ahead and just move forward with that.
03:03I think you could probably make some good arguments for why one is the other and vice versa, but really not important right now.
03:11Now, if you did have some examples pop to mind, they might have been like Claude or Codex, um you know, OpenClaude, Hermes.
03:20But you know what's interesting is I bet if you went out onto, you know, the the streets of corporate America in any city, maybe not San Francisco, but any city in America,
03:30and you asked somebody just in an office building, could you name an agent by name?
03:38I think some people are going to get Claude.
03:42Some people might get Codex.
03:45And that's about it.
03:46I don't think hardly anybody's going to be getting OpenClaude or Hermes.
03:50Uh Uh, and and really even Claude, I don't know that people would even know that that's an agent.
03:55These things are not well understood.
03:58And yet, what's so crazy is everybody is building agents.
04:03I have a real estate agency down the street that's building agents.
04:07I know in like independent private insurance brokers building their own agents.
04:13I know Fortune 500 companies, lots of them, building their own custom agents.
04:17Everybody is trying to build their own custom agents.
04:22And I know people don't believe me, uh, but go talk to them.
04:25Just go talk to people.
04:27They are trying to build custom agents.
04:29And I can't help but wonder why.
04:32Nobody seems to be asking this question, why?
04:34There's already AI everywhere.
04:36You can get on ChatGPT all the way down to some open-source model from China on some, you know, rickety website.
04:43There's everything in between, but still people want to build custom agents.
04:47And ultimately, it comes down to integration.
04:49Businesses want their data properly integrated into AI.
04:54They They believe, and are probably right, that if they appropriately leverage AI, they're going to have these dramatic gains in their business and so on and so forth.
05:04So, they need to figure out how to get integrated.
05:06And building their own custom agents is obviously a way to do that.
05:11And it's one of the first ways that they discover, um, as a mechanism for doing it.
05:16The problem, though, is that agents are really hard.
05:20You have to take very, very careful care of the agentic loop and make sure that it's properly orchestrated.
05:27There are a ton of different provider abstractions you need to think about.
05:32Um, fortunately, there's some good tools coming out around that, you know, like the the Vercel AI SDK is great.
05:38Um, durable execution, you need to make sure if there's faults, we can pick back up.
05:43These are relatively hard problems, um, especially if you're thinking about it at scale.
05:48And the reality is there's just tons more.
05:50There's all kinds of validations and stop conditions and so on and so forth.
05:54And so what often happens is people do try to build their own custom agents, and they sort of work as a demo, but but not much more than that.
06:01Um, and really it turns out that it's an absolute nightmare for people.
06:06Um, building robust agents is just hard.
06:09And if you go talk to anybody in an IT department, they are pulling their hair out because there are so many different concerns.
06:18Um, there's no defined way to build an agent right now.
06:20Like, actually no defined way.
06:22The closest thing maybe is, uh, Eve that just came out from Vercel is maybe like the closest thing.
06:29Um, but in reality, everybody's kind of coming up with their own way to do it.
06:34Um, telemetry and observability on these agents is unbelievably hard, especially at scale.
06:38Like, if you want to know exactly what is getting transmitted on every single step of every single turn of your agent, so that way you can diagnose it and fine-tune it and make sure things aren't going off the rails,
06:50that is very hard to do.
06:52Uh, agents are also not portable.
06:54So, if I do get a good agent working, if I've managed to climb to the top of, you know, this mountain and I've got a good agent that's finally working well, well, it works well on my machine.
07:05[laughter]
07:06But if I try to pass it off to somebody else, there's a very high likelihood that between all of the environment variable configurations and and system requirements and and runtimes,
07:16there's a good chance it's not going to run on that person's machine.
07:20And they're not composable.
07:21So, even if I get, you know, a really good chatbot working for my university, the chances that I'm going to then be able to reuse that for another thing
07:31is very, very low.
07:32I can't just easily share that.
07:35So, what often happens is after a short pursuit towards agents, people kind of back away and say, "Okay, fine.
07:41No more agents, no agents.
07:43Instead, we're going to do the MCP thing.
07:46We've heard about this, it works."
07:48And sure enough, Model Context Protocol, it does work.
07:52And really, it it works pretty well to take, you know, your corporate information like Zillow's information and then shove that into one of these really large pre-existing agents, something like Claude or ChatGPT, which I would consider a large general-purpose
08:09agent.
08:10Um and it and it sort of works like that.
08:12Uh and and it works okay.
08:14But if you take a look, this is actually from the MCP website, and if you take a look at what is supported in MCP clients around the world, you will notice that only one of these columns is actually filled out all the way down.
08:30And that, of course, is tools.
08:33So, MCP has become a de facto tool distribution mechanism for agents.
08:40So, if I need to get my company's tools into that other agent, then MCP's a good way to do that.
08:48It has not proven to be great at providing other value yet.
08:55And frankly, tools are just not enough.
08:57You know, I I I I like to joke that we didn't land a man on the moon by giving one guy a ton of tools.
09:05That's not a realistic way to get a really large project done.
09:10So, uh you know, maybe MCP's not the way, but aha, we have skills.
09:16We have skills, and skills are great.
09:18Um I I I actually do enjoy skills.
09:21I'm sure you do, too.
09:22We install them all the time for all kinds of things.
09:24And fundamentally, what a skill is is a markdown file, which basically works as documentation.
09:30Now, interestingly, there's lots of research out there that shows that if you use very many of these, it actually makes your agent substantially worse.
09:38But, they do work as documentation for various complex things.
09:44So, you know, back to the analogy of a man getting to the moon, it's a little bit like just giving this guy, you know, a ton of documentation.
09:51And the documentation's going to help, but it's not the fundamental problem.
09:56So, what's the fundamental problem?
09:58Okay.
09:59Let's build up a basic agent stack here.
10:02Let's start with a model.
10:03All agents start with a model.
10:05Big one, small one, doesn't matter.
10:07They Then, you have something like a system prompt on top of that, which tells the model what its role in the grand universe is, sort of like its its life objective.
10:19Then, we have tools, the things that it can actually do, the effects it can take.
10:24And then, skills would be layered on top of that.
10:27And then, MCP would be layered on top of that.
10:30And then, finally, you have all the messages from the conversation.
10:34That is roughly the stack of information that gets passed along within the runtime of an agent.
10:41And if you take a look here, almost all of it is context.
10:48Basically, everything.
10:49The system prompt, tools, skills, all of that is stuff that ends up in the context of the agent.
10:56And so, basically, people are trying to solve the integration problem by working on the context or the model.
11:06These are the two areas where we constantly see new advances.
11:10We also see, you know, new new things come out like skills and and MCP, um new technologies, new protocols.
11:17They are all coming out in the in the area of the context and the model.
11:25So, how does it actually work then?
11:28Well, basically, you work at a company, you occasionally need to do some business travel, so you've got a couple travel MCPs installed.
11:36You've also got, you know, Figma and Playwright installed on yours.
11:40And all of these are building up in that context layer.
11:43And then you've got some, you know, Gmail MCPs to go check your mail for you and some Google Sheets to go fill out some other uh some other uh expense reports or something like that.
11:53And then you've got skills.
11:54You're a developer, so you've got some React fixers and linters.
11:58This is actually, I think, like, the number one or the number two most popular MCP server that's out there.
12:03Um maybe you've got uh Matt's uh grill me skill or or maybe you've got the GitHub skill.
12:10And basically, what you're doing is you are inflating that context layer.
12:14And we have a term for this in engineering.
12:16It's called inheritance.
12:18The idea of inheritance is you take an object and then you add more attributes to it to allow that one object to have other properties.
12:29Right?
12:29And that's exactly what we are doing here with an agent.
12:33We're saying, "This agent is pretty good, but if we add all of these addish- additional extra layers, then the agent can do stuff that it previously couldn't do before."
12:44That is exactly what inheritance is.
12:46And the truth about inheritance is it works.
12:49It does work.
12:50That's why these things are out there and they are working.
12:55But, there's an old saying, "Composition over inheritance."
13:01And it turns out this this is as old as time.
13:04Eventually, inheritance starts to break down.
13:07Imagine, like, you know, okay, I've got five skills on uh ChatGPT or on or on Claude, excuse me.
13:13And uh that works pretty well.
13:15Now, what if I have 100 skills?
13:18What if I have 1,000 skills?
13:20There's some point at which I get diminishing returns from adding additional context.
13:25That's That's just obvious.
13:27We all kind of understand that implicitly.
13:30So, is there an alternative?
13:32Well, composition is the alternative to inheritance.
13:35It looks something like this.
13:35So, like imagine we have another little agent, and again, we're trying to provide Figma as an as a as a thing that can be done by our primary agent.
13:46Well, what we could do is have a tiny little agent where the actual system prompt of the agent is written specifically to be a Figma agent.
13:56It knows everything about Figma.
13:57It it knows all of its all of its context, all of its API, all of the right places to click and the things to do and mouse movements to make and everything like that.
14:06And then it has these precise tools that it needs to perform all of those actions, and nothing more.
14:13Just that.
14:14And then a very small message history, which just has to do with the Figma portion of this.
14:20And then you can have more of these.
14:22You can still have your Gmail and your travel and your Google Sheets and all that kind of stuff, but each of them is a separate isolated agent, a full agent.
14:31Not just a little server with tools on it.
14:33It's a full agent with its own message history, its own agentic loop.
14:38And then above these, you have a coordinator.
14:43And the communication mechanism for all of these small agents speaking to the larger agent above it is just English.
14:51They just talk to each other the way a human does.
14:55So, if the primary agent is saying, "Oh, I should I should check my mail to see if there's anything about going on a trip."
15:03Well, it knows to go ask Gmail for any new uh emails about a trip.
15:08Those funnel their way back up, says, "Oh, yeah, actually there's a trip coming up to Los Angeles this weekend."
15:15And then it can go to our travel agent and start to make bookings.
15:20That's kind of a a rough idea of how something like this could work.
15:27And the reality is it does work.
15:28And we know it works because this is actually how we got to the moon.
15:33There were teams of experts.
15:35Teams of experts with faces that looked like that and faces that looked like that.
15:41Each of them with different skills and capabilities.
15:44And faces that looked like that.
15:46This is the Apollo 11 launch day.
15:49And look right here, there's an agent.
15:50I just found an agent sitting right there.
15:53[laughter]
15:53That brain of his is that's his LLM.
15:56And here's his tools right there on the dashboard.
15:59Those are the tools.
16:00Now, he didn't have all the tools.
16:02He just had those tools.
16:03And he was really really really good at them.
16:06And then look at that mouth.
16:08That's the messages.
16:09Uh we are used to this.
16:11We can understand this.
16:12It implicitly works.
16:14It's almost a form of biomimicry for the agentic world.
16:18Um it works.
16:21And I call them domain-specific agents.
16:23Um I don't think I was the first person to utter the words domain-specific agents.
16:26Certainly not the first person to have this idea.
16:29Um but that is what I want to talk to you about.
16:32Agents that are just targeted to very specific domain.
16:37And we over here at Standard Agents have been building this ecosystem for quite some time.
16:43So we've gotten to have a really good inside look at how they actually work.
16:47And I'm not ready to come out here and announce a product or anything like that.
16:50Um but I can give you a little bit of a peek.
16:52First of all, they are far more token efficient.
16:56Far more token efficient.
16:57We regularly see over 80% token efficiency for any given task.
17:03Now, it's a little more complicated because you have to define those tasks a little bit more ahead of time.
17:10But if you can have an agent portability where I can take that Gmail agent, squeeze it up, and then send it to somebody else, we can create an ecosystem
17:19where we don't have to create every one of these skills and capabilities, but within that domain, you're going to get dramatic efficiency.
17:28Um and part of the reason is, if you think about the way that the context work works, I don't need to have the entire context of the conversation when I make a choice to do something.
17:39Instead, my primary coordinator level can just ask the Gmail uh hey, get that last email from Debbie.
17:47And that is the totality of the context.
17:49It literally just has the system message, its tools, and that message that came in.
17:55And so, it is then able to perform this very targeted, very specialized, tiny little thing without all of the surrounding context.
18:06It's also far more practical with small language models.
18:12If you look at the difference in two models like DeepSeek uh V4 Flash and uh Fable 5, the cost difference is mind-boggling.
18:25It is 137 times cheaper than Fable per task.
18:33137 and 37 times.
18:37Now, granted, if DeepSeek V4 Flash fails over and over and over again to do the job, then not only is it going to be, you know, not that much cheaper, it's also going to be much more annoying to use it.
18:51But that's why domain-specific agents are so great, because you don't need to have the V4 Flash do everything.
19:00Instead, it only needs to do the tasks that have been specifically picked for it to do.
19:06And with a very minimal context, it can execute those very faithfully.
19:11So, you get these dramatic cost reductions not only with the token efficiency, but also because you can use much smaller language models and even non-language models.
19:22You can use image generation and diffusion models.
19:25You can use all kinds of other models for smaller tasks.
19:29[snorts]
19:30You can also enforce really strict limits on the capabilities.
19:34And I think you know what I'm talking about.
19:36I'm talking about this.
19:37Uh we are all flying awfully close to the sun nowadays.
19:41We're everybody's just bypassing permissions left and right.
19:44And of course, you have to because a coding agent with a big model can do anything.
19:52And so, we use it to do everything.
19:56In a world that would be powered by smaller domain-specific agents, those agents can't do everything.
20:01They can only do the things that are already explicitly approved for them to do.
20:06It doesn't mean that you still can't have permissions and permission dialogues, but you are opting into a much more controlled ecosystem.
20:14And I promise you, when you explain that to Doug in IT, he puts his heart at ease understanding the difference between those two.
20:25[snorts]
20:26And uh fourth, they these have excellent scaling characteristics.
20:29Because each of these agents is its own small little execution environment, you can parallelize them.
20:35You can put them on the cloud very easily without needing like a giant VPC up there.
20:40You can run thousands of instances all at the same time um in in all kinds of regions of the world.
20:47They don't actually need to be uh you know, geographically co-located or anything like that.
20:52Um so, they have very, very good scaling characteristics.
20:56Unfortunately, they don't exist.
21:00That's the downside.
21:02[laughter]
21:02These domain-specific agents don't really exist.
21:06Um not in a big public way.
21:09Like I said, here at Standard Agents, we have them.
21:12We are working with them on a daily basis, but they are not out there in public very much yet.
21:19However, that's changing.
21:21That is going to change very quickly.
21:23We're about halfway through 2026, and and I'm here to make a public prediction that I think as we roll on from from this point to the end of 2026,
21:35we are going to see a dramatic uptick in people talking about building domain-specific agents, frameworks around them.
21:43All kinds of things are coming down the pipe.
21:46And it's not going to be a small trickle.
21:49It's it's going to accelerate rapidly, and this will become a one of the main players in the agentic ecosystem.
21:56And 2027, I would say, is basically the year of multi-agent orchestration.
22:02That's another word you'll start to hear a lot, I think.
22:05So, that's my big bold public prediction.
22:09I was really excited just a few days ago when Vercel released Eve.
22:14This is the first time I actually saw the term that I had been blasting out into the void come back and hit me in my own face.
22:22The framework for building agents build a company brain, personal assistant, or domain-specific agent.
22:29So, there we go.
22:30About halfway through the year, we're going to start picking up steam.
22:35That's my prediction.
22:37And there's a number of reasons.
22:38One of them is something that most people believe right now is that the cost of intelligence is going down.
22:45That trend reversed in 2026, actually.
22:49We track this on on a website.
22:52Tokens are not getting cheaper anymore.
22:54They are actually going up even when adjusted for IQ.
22:58They're up 29% when you adjust for IQ just this year, halfway through the year, we're already up 30% and that can be caused by lots of different things.
23:08Of course, um we've got this memory crunch and and you know, probably the long-term trend over a 10-year cycle or something is that intelligence will go down, but that does not mean that we need to be paying 137
23:21times the cost for something that can be done just as effectively.
23:25The problem is it's harder to break those things apart.
23:29Now, if you don't account for IQ, tokens are up 76% this year, almost 100% increase in tokens just this year.
23:41Um and we're we're not even halfway through it.
23:43So, we are really trending upwards on on token costs.
23:48So, anything we can do, especially with large businesses, to bring that down is going to be really important.
23:52Um the other the other use case to really consider is putting AI in front of customers.
23:58You can't put Fable in front of a customer, um unless that customer has a massive lifetime value.
24:05It's just too expensive.
24:07So, you need to find a way to create great efficacy while being efficient.
24:13And domain-specific agents are going to be the way to do that.
24:15So, I'm going to leave you here in momentarily, um but before I do, let's just dream a little bit.
24:22Let me dig in a little bit deeper to how an agent could be orchestrated and what an ideal agent would actually look like.
24:31And then, I promise to leave you alone.
24:35Here we go.
24:35So, remember we got that model and we got the system prompt.
24:39And then, at the tool layer, let's break that apart a little bit.
24:42On one hand, we have these like functions.
24:44This would be like an actual function that can get executed, like write a file to the file system.
24:50Then, we have prompts.
24:53Prompts are a lot like the system prompt, but they are smaller individual prompts that can get injected and and sub prompts that can you know, you can run a function
25:05that actually calls an LLM.
25:07So, let's say I have a main agent running, but I want to use Nano Banana just to generate an image when I'm using GLM you know, 5.2 as my primary.
25:18Well, you can just have a tool that's a prompt.
25:20That would be really cool if you could do that.
25:23And then another type of tool could be another full-blown agent, like a complete other domain specific agent could just be one of the tools.
25:33So, that's the tool layer.
25:35And then you have hooks.
25:37Uh what are hooks?
25:38Well, in this ideal world, a hook might be something that can kind of harness or change or mutate or perform side effects.
25:48So, let me give you an example.
25:49LLMs have no idea what time it is at any given point in time.
25:54Turns out a really great way to tell them what time it is is you inject an artificial message or an artificial tool call in the message history.
26:04So,
26:04[snorts]
26:04it looks like somebody just said, "Hey, what time it is?"
26:07and the other person replied, "Oh, it's 6:45 p.m. Pacific time."
26:13Pretty simple.
26:15Um you can do that with a hook.
26:17Or you could fire off some side effect using a hook.
26:20So, this is an important piece of an agent.
26:23And then finally, there's there's these agent rules.
26:26And agent rules are kind of complicated.
26:28It's like how many times should one side have a turn?
26:33Like can it go on for 10,000 turns uh or 10,000 steps before it's turn is up?
26:39You know, there's there's all kinds of interesting little rules.
26:41You know, when it calls uh a tool, you know, is it required to validate the whole thing or not, you know?
26:48All kinds of of very specific tools uh or or rules that that belong to a specific agent.
26:54And altogether, if we bundle all that up, we would call that an agent.
26:58But, it's kind of missing a couple things.
27:00One, every agent should really have a file system.
27:05If you've ever done this with ChatGPT or Claude or Codex, if you just ask it, you know, not inside of a project or anything like, "Hey, can you make me
27:15a PDF for my my son's birthday party?"
27:20Well, it'll do it.
27:21And it'll store it in its own little file system.
27:25So, the big labs have already realized that in order to create an effective chat interface, not to mention a big agent, it needs some sort of file system.
27:33So, every agent should have its own little sandbox file system.
27:37And also, every agent should have a sandboxed code execution location.
27:42So, it can write files, it can run those files, and it can do that safely without exfiltrating anything, without interacting with an OS at a higher level.
27:51That needs to be baked in as a primitive to every single domain-specific agent.
27:56Okay, so, let's say that that's our ideal agent.
28:00And now, let's talk about that little agent tool there.
28:02What is that?
28:03Well, those can be sub-agents, recursive sub-agents even.
28:08You could have an agent that calls a sub-agent that calls sub-agents that call sub-agents.
28:14Um, and there could be one or there could be many of these at different levels.
28:18So, for example, you could have this coordinator agent that's at the very top, and then you could have a Salesforce agent.
28:25And that agent knows Salesforce inside and out.
28:27It knows all of its APIs, it knows has all the credentials to communicate with your Salesforce instance in all the appropriate ways.
28:34And then, it needs to communicate with a Google Workspace agent.
28:38So, it can do all kinds of stuff in there.
28:39It can run spreadsheets.
28:41I can say, "Hey, what are all my tops uh my my top sales people this year?"
28:45Boom, it can look in Salesforce, it can coordinate with the sub agent, create a sheet for you, send that back.
28:51Perfect.
28:52But, maybe then you need to generate some assets.
28:55So, the Salesforce agent actually has another sub agent that it can talk to at any time it wants.
29:00And it's amazing at generating assets.
29:02Maybe that sub agent doesn't just have like, you know, Codex image generation, maybe it has Nano Banana, maybe it has an SVG generator, all kinds of stuff, so that way it is an amazing
29:14asset generator and performs some of its own reflection and QA.
29:19And then our primary agent might need a whole legal team agent just so it can check the work that's coming out of these other ones.
29:25And maybe the legal team agent really needs a GDPR compliance agent just for those European customers.
29:32You know, the main one doesn't have all, you know, you don't we don't want to have 45 megabytes of context just on GDPR.
29:38So, we make that a separate sub agent.
29:41Uh so, you know, may and then maybe the legal team also needs like an OSHA compliance agent, which is also very complicated.
29:47And so, it has a separate one for that.
29:49You kind of get the idea.
29:51You can end up with all kinds of highly efficient, small little agents that are all working together, but maintaining small minimal context windows all the way through.
30:04That's the idea behind domain-specific agents.
30:08So, thank you very much.
30:09I appreciate you listening to my talk.
30:11Again, uh Standard Agents is where we're working, standardagents.ai.
30:14You can actually sign up on there uh for an early access.
30:18Um we are slowly starting to roll this out to a few people.
30:21Um if your business is super ambitious and really wants to uh try out small domain-specific agents, then um you know, you can write me info@standardagents.
30:31And of course, I'd appreciate a follow.
30:32Thank you so much.
30:34Bye.