From coding to Knowledge work agents — Karan Vaidya, Composio
https://www.youtube.com/watch?v=xxfMT-bPEmU
AI Engineer · 20 min · 281 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:13] Uh hey folks.
[00:14] Uh I'm Karan Vedya, co-founder and CTO of Composio.
[00:19] Most agentic tool calls today are still happening in one field.
[00:23] No guesses, it's software engineering.
[00:26] Every other kind of work is trailing far behind.
[00:29] If models keep getting better, then why are we still limited to just agentic coding?
[00:35] That's the trillion-dollar question I'm here to answer.
[00:43] Three years ago, coding agents were just auto complete.
[00:47] Today, software engineering is fully autonomous.
[00:50] We went from pressing tab tab tab to let just Claude cook.
[00:56] That's just magic.
[01:00] And why did it happen so fast in coding?
[01:04] Most people would think it's models.
[01:07] Yeah, models got really better over time over the last two to three years.
[01:11] And so did the harnesses.
[01:14] Claude code, Codex, Cursor.
[01:16] But on their own, it wouldn't have been enough.
[01:20] It only worked because all the infrastructure and systems around coding were literally meant for agents.
[01:28] Code came with the support that agents needed.
[01:32] You have got the repo, the commit history, tests, CI/CD, review, linters, revert if anything goes wrong.
[01:40] The kind of stuff that makes you trust the agents, the systems around code.
[01:46] Now, we're pointing these same amazing agents at everything else.
[01:51] Support, finance, sales.
[01:54] But the agents that were doing phenomenally well in coding, are just working blind.
[01:59] Because the infrastructure around coding doesn't even exist in other fields.
[02:05] [snorts]
[02:05] So, how do we close the bridge between coding agents and knowledge work agents?
[02:11] We think it's core six primitives and coding had all six of them while knowledge work doesn't have any and that's what we need to build.
[02:23] First is centralization.
[02:26] Coding agents work so pretty well partly because they were very near the source of truth.
[02:34] They knew the what, the why and how.
[02:37] You give them the repo, the infrastructure as code and you close the loop and let the model cook.
[02:44] The agent starts at everything with everything they need all in a single place, that is the code base.
[02:51] This is exactly what knowledge work miss today.
[02:55] For example, a single deal is scattered across five different platforms.
[02:59] The records are in Salesforce, the docs in Notion, the emails in Gmail, conversations in Slack and the support history is in Zendesk.
[03:09] There's no single source of truth, single place to get all the information.
[03:13] Everything is separate and every app has its own login.
[03:17] Before a knowledge work agent can even start to do thing, it has to go and pull all the threads and kind of tie them together itself.
[03:27] And that's still the base point where coding agent had started.
[03:30] It already had it all.
[03:32] So, how can you expect knowledge work to do knowledge work agent to do the same level of work as coding agent?
[03:39] So, the first thing we build is the missing center.
[03:42] One place where all your apps, all your connections, all your logins exist.
[03:47] So, the agent doesn't need to do the hard work of stitching them all together.
[03:51] They find it all in a single place.
[03:53] And they get the baseline with the coding agent started with, which is the repo, the information across all the stacks in one single place.
[04:02] That's the foundation you start with and you can give right accesses to your agent.
[04:09] The next thing agent needs is a sense of history.
[04:12] The ability to look back in the past.
[04:16] In code, you get it for free.
[04:19] Git keeps a record of every single thing that went in, every single change that was made.
[04:24] So, the agent can always look back and see how a certain change was made, why something worked, why something didn't work.
[04:31] Think about the kind of thing you actually ask your agent to do.
[04:35] We had to revert a change in the past because of some failure, but that was pretty hard to pull off.
[04:41] Can you look at it and get it back again?
[04:43] It just reach to the history and get it back and cook it.
[04:47] The
[04:47] [snorts]
[04:47] history isn't just for agent.
[04:49] It's also for you to keep a record what the agent is doing.
[04:53] You can see what the agent is doing, where it is up, where it is doing successful things, and like instead of trusting what the agent is saying you, you can just go to those particular apps and look at what it has done.
[05:09] Now, ask those same questions about knowledge work.
[05:12] What led to the CRM being in a state where it is today?
[05:16] How did the How did my colleague craft that amazing mail that led to the closing of the deal?
[05:22] What's the actual process to escalate a support issue or even close one?
[05:27] The answers are smeared across hundreds of apps and none of them keep the history.
[05:31] So, the agent has no memory.
[05:33] It starts from blank state almost every time.
[05:36] No idea what was tried before, what worked, what didn't work.
[05:40] And you you have nothing to look at all as well.
[05:44] Once the agent runs, it tells you it has done successfully, you don't know if it has actually done successfully.
[05:49] There's no way to know if it is right or not.
[05:52] And that's what's missing, a record of work.
[05:57] Now, because everything finally runs through one single place, that centralization, we can build a layer on top of it, the record.
[06:06] Every single like action that agent takes can be logged across every other app.
[06:12] Whatever it touched, whatever it skipped, what worked, what didn't.
[06:17] Via this, firstly, the agent gets memory.
[06:20] It can look back at how similar tasks were done before, what was successful, and replicate it again.
[06:28] It doesn't start with a blank state all the time.
[06:31] Second, you get trust.
[06:33] You can finally see exactly what the agent is doing.
[06:36] So, instead of hoping it will do the right thing, you can just go back and check and catch it if it does something bad.
[06:44] And as you kind of see it more and more doing the right things, you'll develop the trust and offload more tasks to it.
[06:51] The next thing an agent needs is context.
[06:54] And there are really two kinds of context, if you think about it.
[06:57] The first the shape of shape of the platform, the architecture.
[07:01] How things flow into each other, how things are tied, the data flows.
[07:05] Like kind of like a map which a senior engineer carries in their head, and a junior engineer takes probably 3 months to develop.
[07:12] The second is style.
[07:13] This isn't what's objectively correct, but more like what good looks like in your company.
[07:19] So, how you do things, things like linter, type checks, etc. And maybe you use a TypeScript decorator which nobody else would.
[07:29] This is not exactly somewhere in a playbook, it's more in your code base.
[07:34] It's all available in your code base, so the agent can just go and look and figure out the specs, what you like, the like linters, the formatters, etc. Now, coming to knowledge work,
[07:47] the same thing.
[07:48] Say you're writing a doc to a customer.
[07:50] To even start, I would have to open the database to pull their usage, check PostHog of how they have been actually using things.
[07:58] And Salesforce to look at their deal details.
[08:01] Only then I can even start writing the first line of the doc.
[08:05] The answer wasn't isolated in just one of those tools.
[08:09] I'm able to write this is doc because I'm pulling the threads across all these tools into one single context in my head.
[08:15] So, putting history and context together, that's how you map how the organization works.
[08:20] And that part is not available to agent handily.
[08:26] So, as we did centralization and logging, the record we just built, the one that gives the agent memory and lets you check what it did, also does one more interesting thing.
[08:38] It If you log enough of what every agent is doing, you start to see patterns.
[08:42] You start to see how the organization works.
[08:44] And you start to form skills, which is some sort of distillation of how the organization has been working.
[08:51] Which approaches work, which don't, which what led to failures in the past, etc. The The record isn't just history of what happened anymore.
[09:00] It's a picture of how your company operates.
[09:03] And it actually works at three different levels.
[09:06] How a tool works in general, which is applicable to every person, how a company does things, and how you prefer to do things, what good like looks like to you.
[09:15] And that's the context that context that was missing for a knowledge work agent.
[09:19] How the work actually gets done, the real playbook of sorts, and the preference of a company of a personal user.
[09:27] And now the agent can query it and stop guessing how the company operates.
[09:34] The other reason coding agents work so well, they test themselves.
[09:38] The work checks itself.
[09:40] Verification.
[09:41] The moment the agent writes a code, a stack of checks follow.
[09:44] The unit tests can catch small mistakes.
[09:47] The integration tests catch the ones that only affect components three blocks away.
[09:52] The type system would not even work and run if anything is going wrong.
[09:57] The compiler will not even build.
[10:00] On top of it sits the softer checks, linters, formatters, bugboard.md review skills, etc. And these ensure that the code matches the way your time team likes to follow the standards of your team.
[10:12] None of it needs you.
[10:14] The agent completes the loop on its own and makes sure that it follows the standard and is able to make the code run.
[10:21] Now, think about like so, there's an a while back I pointed my open claw at a hiring outreach.
[10:28] Mass emails to candidates.
[10:30] It ran.
[10:31] It sent tons of emails.
[10:34] Uh some of you might have also gotten it from my open claw.
[10:37] It did exactly what I told it to do.
[10:40] It was also a disaster.
[10:41] The kind that ends up on Twitter with my name on top of it.
[10:46] Uh yeah, I think you can see a you Karan Vaidya.
[10:51] Uh I was not the happiest when it happened.
[10:54] And here's the thing, every check from the past slide would have passed.
[10:58] The emails were valid, their addresses were real.
[11:00] It actually got to real people who posted.
[11:04] Uh there was no best tool in the world to actually uh question what really mattered.
[11:10] Should this have gone at all?
[11:13] That's the gap in code.
[11:15] These tests tell you what's wrong and right.
[11:17] Here, the internet told me that I was wrong.
[11:21] So, we build the checks that are missing.
[11:24] The problem in the above thread was wasn't the outreach was wrong.
[11:28] It was that it went out before even I getting to know.
[11:31] So, the fix is simple.
[11:33] Catch before it's even real.
[11:35] So, we have two ways in which we do that.
[11:37] One, before the agent senses anything, it checks the draft emails that I've sent before.
[11:43] If it matches my style, if it matches the goodness that I like.
[11:47] The second, before doing anything destructive in the real world scenario, we provide the agents and boxes, which mock the real tools, and they can send they can do action on top of these sandboxes.
[11:59] So, instead of the blast radius hitting the real world, it will hit a sandbox, and then I can review it before the agent does the real thing.
[12:07] Put those two together, and you've got something knowledge work never had.
[12:11] A way for agent to check its own work before it's even real.
[12:16] It can finally close its own loop instead of stopping to wait for you.
[12:20] And with all that, you can trust the action it is taking without you getting bombarded with the tweets that I shoot.
[12:29] Next thing the agent needs is governance.
[12:32] Building trust is controlling what the agent can do.
[12:34] Putting up the right walls around the agents.
[12:39] In code, this is mostly solved and like has multiple layers.
[12:44] The agent can do whatever it wants on its own branch, but it can't merge to main.
[12:49] A human reviewer sits in between it merging to main.
[12:52] The critical files have code owners.
[12:54] So, whenever it touches one of them, the right people are getting involved.
[12:57] Uh we use agents to ship to preview deployments.
[13:00] Never let it touch the production deployments, so we control it there.
[13:04] The governance is not a single gate, but multiple of them, and each varying its sizes depending on the blast radius it exposes.
[13:12] None of it slows the agent down in safe paths.
[13:15] Just prevents it from up production.
[13:19] And the tighter those lines are, the more you can trust the agent and let it go berserk.
[13:26] You probably saw this one.
[13:27] The director of alignment at Meta Superintelligence Lab hooked up an agent to its email and it started destroying its email, deleting a lot of them.
[13:36] She told it to stop.
[13:37] It kept going.
[13:38] Finally, she had to run to a physical machine to stop it.
[13:41] But by then, 200 emails had actually vanished.
[13:45] She had told it beforehand in prompt to confirm before acting on such cases.
[13:50] But that was just a prompt which probably would have compacted away.
[13:54] And if someone whose sole job is AI alignment can't prompt it the agent correctly, then probably none of us can.
[14:03] And that's the real reason these agents are so hard to trust.
[14:08] Not because they're worse than the coding agents, but because there's no wall around them.
[14:12] In code, the wall was already built into the system while we were developing earlier.
[14:17] Knowledge work also has some bits and pieces here and there.
[14:19] For example, Gmail has scopes, Salesforce has permission levels.
[14:23] But it's so scattered all over the place that it's very hard to have real control and mostly people end up doing it via prompting.
[14:32] And prompting is fragile.
[14:34] The agent will find those loopholes.
[14:36] Things will get compacted away.
[14:38] And at scale, uh one of these fence will break and you'll also be in the same condition where 200 of your important emails are vanishing.
[14:47] So, what would actually stop it?
[14:49] Not like a better instruction, but wall that the agent can't cross even if it forgot that wall existed.
[15:00] So, we build these walls in two layers.
[15:03] The first layer is deterministic, control over what the agent can reach, what it has access to.
[15:09] Um a hiring agent can probably just read the emails.
[15:13] A support agent can create a draft email, but not actually send it.
[15:17] The boundary lives outside these agents.
[15:19] It can't be argued with by the agent or forgotten or compacted.
[15:24] Use instruction failed because it lived in agent's memory in the prompt.
[15:28] This doesn't.
[15:30] Uh but access alone wouldn't have saved her because she was actually building an email agent.
[15:36] So it definitely needed access to that email.
[15:39] Um the other thing that we do is provide policies, which is you can define natural language policies of what the agent can do even with those accesses.
[15:49] So things like never delete more than 10 emails without my permission.
[15:53] Never email outside a particular domain.
[15:56] Rules that with even those access control the behavior.
[16:00] So between those two things, one layer controls the what the agent can reach and the other layer can control the behavior with what it can do with that reach.
[16:09] Together, it's real governance for the agent.
[16:11] Not asking the agent to behave, but enforcing it what it can do.
[16:18] The last pillar, reversibility.
[16:20] And this is the one of the Uh this is where we reach when things go wrong.
[16:25] Can I undo it?
[16:27] In code, you almost always can.
[16:30] Every change is recorded.
[16:32] Things can be walked back.
[16:33] You can get revert the last commit or you can get bisect to the commit that broke your production and revert it.
[16:40] Um now Like I'm not saying it's good.
[16:44] I I won't pretend like that.
[16:46] If things go in production and break, it's always bad, but it's still not permanent.
[16:50] You can still walk back from it.
[16:51] And that's what gives you confidence to let your agents cook and let them do some magic because even if they they break the things, you can you have a pathway back.
[17:03] For knowledge work, there is no undo button.
[17:05] Things Like think about use inbox.
[17:07] Those 200 emails are gone.
[17:09] They have vanished.
[17:10] That's the normal case, by the way.
[17:12] The disaster case is a sent email, which you can't revert back.
[17:15] A wire that has already been made.
[17:17] So you can't get that money back.
[17:18] A deleted record, gone forever.
[17:20] Most actions actually in knowledge work don't have an undo button.
[17:25] And that changes the whole equation.
[17:27] Uh that changes the blast radius.
[17:29] Uh with code, you can trust the agent after the fact.
[17:31] Let it run, check the result, undo if it's wrong.
[17:34] Out here, there's no coming back.
[17:36] The only place left for you is to is to trust before the agent acts.
[17:40] That's what makes these agents feel dangerous in a way coding agents never did.
[17:44] It's not that they fail often.
[17:46] It's that out there failure is forever.
[17:50] So, either you completely go up front or never let it act.
[17:55] Let me be honest, reversibility is the hardest to replicate in knowledge work.
[17:59] Real undo, the way it exists for code, probably doesn't exist in all the scenarios in knowledge work.
[18:04] But we have some scenarios where undo exists and we call them.
[18:09] So, let's say you add a label.
[18:11] You can remove the label afterwards.
[18:14] But for actions that you can't undo at all, like hard deletes that disappear the emails from your inbox, we again provide a sandbox where the agent can do the thing first in the sandbox
[18:25] and you can review it and then actually goes into the production environment.
[18:30] None of it touches the real world.
[18:31] That's the whole flip.
[18:33] In code, you can undo the mistake after it happens.
[18:35] Here, you catch it before it does.
[18:37] Different timing, same result, a mistake that won't stick.
[18:41] Think about you again.
[18:42] The actions we could reverse, we would give it a reverse button.
[18:45] The ones we couldn't, the agent would hit the sandbox first and she would be notified, your 1,200 emails are going to get deleted.
[18:52] Do you want it?
[18:55] It's not done yet.
[18:57] Uh but across billions of actions that we're going through, we are learning on the way which ones can be walked back, which ones can't, and preparing the sandbox accordingly.
[19:09] If you take one thing away today, take this.
[19:11] For 2 years, the model was the bottleneck.
[19:14] So, everybody was racing towards better and better model.
[19:17] Now, the models have gotten good enough where software engineering is 100% autonomous.
[19:23] But, now everything else is the bottleneck.
[19:26] This the same model that writes your code can also do your hiring, sales, and other knowledge work.
[19:36] Um but, it right now it's working blind.
[19:39] No history, no context, no ways to verify, no guardrails, no undo.
[19:44] So, the bottleneck has moved.
[19:48] Now, it's infrastructure that nobody has yet built.
[19:51] And that's what we are building at Composio.
[19:55] Yeah.
[19:56] We are powering billion plus tool calls in total.
[19:59] Uh 300 million tool calls happening every month.
[20:02] And if you are building an agent, just point it to Composio and see the magic happen for knowledge work.
[20:08] And if you want to build the future of substrate of AI agents, then please come to me.
[20:13] We are definitely hiring, and there's a lot lot left to do.
[20:17] The models will keep getting better.
[20:19] The bottleneck won't be models.
[20:21] It will be the things around it.
[20:22] Thank you.