From coding to Knowledge work agents — Karan Vaidya, Composio

AI Engineer · 20 min · 281 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Uh hey folks.
00:14Uh I'm Karan Vedya, co-founder and CTO of Composio.
00:19Most agentic tool calls today are still happening in one field.
00:23No guesses, it's software engineering.
00:26Every other kind of work is trailing far behind.
00:29If models keep getting better, then why are we still limited to just agentic coding?
00:35That's the trillion-dollar question I'm here to answer.
00:43Three years ago, coding agents were just auto complete.
00:47Today, software engineering is fully autonomous.
00:50We went from pressing tab tab tab to let just Claude cook.
00:56That's just magic.
01:00And why did it happen so fast in coding?
01:04Most people would think it's models.
01:07Yeah, models got really better over time over the last two to three years.
01:11And so did the harnesses.
01:14Claude code, Codex, Cursor.
01:16But on their own, it wouldn't have been enough.
01:20It only worked because all the infrastructure and systems around coding were literally meant for agents.
01:28Code came with the support that agents needed.
01:32You have got the repo, the commit history, tests, CI/CD, review, linters, revert if anything goes wrong.
01:40The kind of stuff that makes you trust the agents, the systems around code.
01:46Now, we're pointing these same amazing agents at everything else.
01:51Support, finance, sales.
01:54But the agents that were doing phenomenally well in coding, are just working blind.
01:59Because the infrastructure around coding doesn't even exist in other fields.
02:05[snorts]
02:05So, how do we close the bridge between coding agents and knowledge work agents?
02:11We think it's core six primitives and coding had all six of them while knowledge work doesn't have any and that's what we need to build.
02:23First is centralization.
02:26Coding agents work so pretty well partly because they were very near the source of truth.
02:34They knew the what, the why and how.
02:37You give them the repo, the infrastructure as code and you close the loop and let the model cook.
02:44The agent starts at everything with everything they need all in a single place, that is the code base.
02:51This is exactly what knowledge work miss today.
02:55For example, a single deal is scattered across five different platforms.
02:59The records are in Salesforce, the docs in Notion, the emails in Gmail, conversations in Slack and the support history is in Zendesk.
03:09There's no single source of truth, single place to get all the information.
03:13Everything is separate and every app has its own login.
03:17Before a knowledge work agent can even start to do thing, it has to go and pull all the threads and kind of tie them together itself.
03:27And that's still the base point where coding agent had started.
03:30It already had it all.
03:32So, how can you expect knowledge work to do knowledge work agent to do the same level of work as coding agent?
03:39So, the first thing we build is the missing center.
03:42One place where all your apps, all your connections, all your logins exist.
03:47So, the agent doesn't need to do the hard work of stitching them all together.
03:51They find it all in a single place.
03:53And they get the baseline with the coding agent started with, which is the repo, the information across all the stacks in one single place.
04:02That's the foundation you start with and you can give right accesses to your agent.
04:09The next thing agent needs is a sense of history.
04:12The ability to look back in the past.
04:16In code, you get it for free.
04:19Git keeps a record of every single thing that went in, every single change that was made.
04:24So, the agent can always look back and see how a certain change was made, why something worked, why something didn't work.
04:31Think about the kind of thing you actually ask your agent to do.
04:35We had to revert a change in the past because of some failure, but that was pretty hard to pull off.
04:41Can you look at it and get it back again?
04:43It just reach to the history and get it back and cook it.
04:47The
04:47[snorts]
04:47history isn't just for agent.
04:49It's also for you to keep a record what the agent is doing.
04:53You can see what the agent is doing, where it is up, where it is doing successful things, and like instead of trusting what the agent is saying you, you can just go to those particular apps and look at what it has done.
05:09Now, ask those same questions about knowledge work.
05:12What led to the CRM being in a state where it is today?
05:16How did the How did my colleague craft that amazing mail that led to the closing of the deal?
05:22What's the actual process to escalate a support issue or even close one?
05:27The answers are smeared across hundreds of apps and none of them keep the history.
05:31So, the agent has no memory.
05:33It starts from blank state almost every time.
05:36No idea what was tried before, what worked, what didn't work.
05:40And you you have nothing to look at all as well.
05:44Once the agent runs, it tells you it has done successfully, you don't know if it has actually done successfully.
05:49There's no way to know if it is right or not.
05:52And that's what's missing, a record of work.
05:57Now, because everything finally runs through one single place, that centralization, we can build a layer on top of it, the record.
06:06Every single like action that agent takes can be logged across every other app.
06:12Whatever it touched, whatever it skipped, what worked, what didn't.
06:17Via this, firstly, the agent gets memory.
06:20It can look back at how similar tasks were done before, what was successful, and replicate it again.
06:28It doesn't start with a blank state all the time.
06:31Second, you get trust.
06:33You can finally see exactly what the agent is doing.
06:36So, instead of hoping it will do the right thing, you can just go back and check and catch it if it does something bad.
06:44And as you kind of see it more and more doing the right things, you'll develop the trust and offload more tasks to it.
06:51The next thing an agent needs is context.
06:54And there are really two kinds of context, if you think about it.
06:57The first the shape of shape of the platform, the architecture.
07:01How things flow into each other, how things are tied, the data flows.
07:05Like kind of like a map which a senior engineer carries in their head, and a junior engineer takes probably 3 months to develop.
07:12The second is style.
07:13This isn't what's objectively correct, but more like what good looks like in your company.
07:19So, how you do things, things like linter, type checks, etc. And maybe you use a TypeScript decorator which nobody else would.
07:29This is not exactly somewhere in a playbook, it's more in your code base.
07:34It's all available in your code base, so the agent can just go and look and figure out the specs, what you like, the like linters, the formatters, etc. Now, coming to knowledge work,
07:47the same thing.
07:48Say you're writing a doc to a customer.
07:50To even start, I would have to open the database to pull their usage, check PostHog of how they have been actually using things.
07:58And Salesforce to look at their deal details.
08:01Only then I can even start writing the first line of the doc.
08:05The answer wasn't isolated in just one of those tools.
08:09I'm able to write this is doc because I'm pulling the threads across all these tools into one single context in my head.
08:15So, putting history and context together, that's how you map how the organization works.
08:20And that part is not available to agent handily.
08:26So, as we did centralization and logging, the record we just built, the one that gives the agent memory and lets you check what it did, also does one more interesting thing.
08:38It If you log enough of what every agent is doing, you start to see patterns.
08:42You start to see how the organization works.
08:44And you start to form skills, which is some sort of distillation of how the organization has been working.
08:51Which approaches work, which don't, which what led to failures in the past, etc. The The record isn't just history of what happened anymore.
09:00It's a picture of how your company operates.
09:03And it actually works at three different levels.
09:06How a tool works in general, which is applicable to every person, how a company does things, and how you prefer to do things, what good like looks like to you.
09:15And that's the context that context that was missing for a knowledge work agent.
09:19How the work actually gets done, the real playbook of sorts, and the preference of a company of a personal user.
09:27And now the agent can query it and stop guessing how the company operates.
09:34The other reason coding agents work so well, they test themselves.
09:38The work checks itself.
09:40Verification.
09:41The moment the agent writes a code, a stack of checks follow.
09:44The unit tests can catch small mistakes.
09:47The integration tests catch the ones that only affect components three blocks away.
09:52The type system would not even work and run if anything is going wrong.
09:57The compiler will not even build.
10:00On top of it sits the softer checks, linters, formatters, bugboard.md review skills, etc. And these ensure that the code matches the way your time team likes to follow the standards of your team.
10:12None of it needs you.
10:14The agent completes the loop on its own and makes sure that it follows the standard and is able to make the code run.
10:21Now, think about like so, there's an a while back I pointed my open claw at a hiring outreach.
10:28Mass emails to candidates.
10:30It ran.
10:31It sent tons of emails.
10:34Uh some of you might have also gotten it from my open claw.
10:37It did exactly what I told it to do.
10:40It was also a disaster.
10:41The kind that ends up on Twitter with my name on top of it.
10:46Uh yeah, I think you can see a you Karan Vaidya.
10:51Uh I was not the happiest when it happened.
10:54And here's the thing, every check from the past slide would have passed.
10:58The emails were valid, their addresses were real.
11:00It actually got to real people who posted.
11:04Uh there was no best tool in the world to actually uh question what really mattered.
11:10Should this have gone at all?
11:13That's the gap in code.
11:15These tests tell you what's wrong and right.
11:17Here, the internet told me that I was wrong.
11:21So, we build the checks that are missing.
11:24The problem in the above thread was wasn't the outreach was wrong.
11:28It was that it went out before even I getting to know.
11:31So, the fix is simple.
11:33Catch before it's even real.
11:35So, we have two ways in which we do that.
11:37One, before the agent senses anything, it checks the draft emails that I've sent before.
11:43If it matches my style, if it matches the goodness that I like.
11:47The second, before doing anything destructive in the real world scenario, we provide the agents and boxes, which mock the real tools, and they can send they can do action on top of these sandboxes.
11:59So, instead of the blast radius hitting the real world, it will hit a sandbox, and then I can review it before the agent does the real thing.
12:07Put those two together, and you've got something knowledge work never had.
12:11A way for agent to check its own work before it's even real.
12:16It can finally close its own loop instead of stopping to wait for you.
12:20And with all that, you can trust the action it is taking without you getting bombarded with the tweets that I shoot.
12:29Next thing the agent needs is governance.
12:32Building trust is controlling what the agent can do.
12:34Putting up the right walls around the agents.
12:39In code, this is mostly solved and like has multiple layers.
12:44The agent can do whatever it wants on its own branch, but it can't merge to main.
12:49A human reviewer sits in between it merging to main.
12:52The critical files have code owners.
12:54So, whenever it touches one of them, the right people are getting involved.
12:57Uh we use agents to ship to preview deployments.
13:00Never let it touch the production deployments, so we control it there.
13:04The governance is not a single gate, but multiple of them, and each varying its sizes depending on the blast radius it exposes.
13:12None of it slows the agent down in safe paths.
13:15Just prevents it from up production.
13:19And the tighter those lines are, the more you can trust the agent and let it go berserk.
13:26You probably saw this one.
13:27The director of alignment at Meta Superintelligence Lab hooked up an agent to its email and it started destroying its email, deleting a lot of them.
13:36She told it to stop.
13:37It kept going.
13:38Finally, she had to run to a physical machine to stop it.
13:41But by then, 200 emails had actually vanished.
13:45She had told it beforehand in prompt to confirm before acting on such cases.
13:50But that was just a prompt which probably would have compacted away.
13:54And if someone whose sole job is AI alignment can't prompt it the agent correctly, then probably none of us can.
14:03And that's the real reason these agents are so hard to trust.
14:08Not because they're worse than the coding agents, but because there's no wall around them.
14:12In code, the wall was already built into the system while we were developing earlier.
14:17Knowledge work also has some bits and pieces here and there.
14:19For example, Gmail has scopes, Salesforce has permission levels.
14:23But it's so scattered all over the place that it's very hard to have real control and mostly people end up doing it via prompting.
14:32And prompting is fragile.
14:34The agent will find those loopholes.
14:36Things will get compacted away.
14:38And at scale, uh one of these fence will break and you'll also be in the same condition where 200 of your important emails are vanishing.
14:47So, what would actually stop it?
14:49Not like a better instruction, but wall that the agent can't cross even if it forgot that wall existed.
15:00So, we build these walls in two layers.
15:03The first layer is deterministic, control over what the agent can reach, what it has access to.
15:09Um a hiring agent can probably just read the emails.
15:13A support agent can create a draft email, but not actually send it.
15:17The boundary lives outside these agents.
15:19It can't be argued with by the agent or forgotten or compacted.
15:24Use instruction failed because it lived in agent's memory in the prompt.
15:28This doesn't.
15:30Uh but access alone wouldn't have saved her because she was actually building an email agent.
15:36So it definitely needed access to that email.
15:39Um the other thing that we do is provide policies, which is you can define natural language policies of what the agent can do even with those accesses.
15:49So things like never delete more than 10 emails without my permission.
15:53Never email outside a particular domain.
15:56Rules that with even those access control the behavior.
16:00So between those two things, one layer controls the what the agent can reach and the other layer can control the behavior with what it can do with that reach.
16:09Together, it's real governance for the agent.
16:11Not asking the agent to behave, but enforcing it what it can do.
16:18The last pillar, reversibility.
16:20And this is the one of the Uh this is where we reach when things go wrong.
16:25Can I undo it?
16:27In code, you almost always can.
16:30Every change is recorded.
16:32Things can be walked back.
16:33You can get revert the last commit or you can get bisect to the commit that broke your production and revert it.
16:40Um now Like I'm not saying it's good.
16:44I I won't pretend like that.
16:46If things go in production and break, it's always bad, but it's still not permanent.
16:50You can still walk back from it.
16:51And that's what gives you confidence to let your agents cook and let them do some magic because even if they they break the things, you can you have a pathway back.
17:03For knowledge work, there is no undo button.
17:05Things Like think about use inbox.
17:07Those 200 emails are gone.
17:09They have vanished.
17:10That's the normal case, by the way.
17:12The disaster case is a sent email, which you can't revert back.
17:15A wire that has already been made.
17:17So you can't get that money back.
17:18A deleted record, gone forever.
17:20Most actions actually in knowledge work don't have an undo button.
17:25And that changes the whole equation.
17:27Uh that changes the blast radius.
17:29Uh with code, you can trust the agent after the fact.
17:31Let it run, check the result, undo if it's wrong.
17:34Out here, there's no coming back.
17:36The only place left for you is to is to trust before the agent acts.
17:40That's what makes these agents feel dangerous in a way coding agents never did.
17:44It's not that they fail often.
17:46It's that out there failure is forever.
17:50So, either you completely go up front or never let it act.
17:55Let me be honest, reversibility is the hardest to replicate in knowledge work.
17:59Real undo, the way it exists for code, probably doesn't exist in all the scenarios in knowledge work.
18:04But we have some scenarios where undo exists and we call them.
18:09So, let's say you add a label.
18:11You can remove the label afterwards.
18:14But for actions that you can't undo at all, like hard deletes that disappear the emails from your inbox, we again provide a sandbox where the agent can do the thing first in the sandbox
18:25and you can review it and then actually goes into the production environment.
18:30None of it touches the real world.
18:31That's the whole flip.
18:33In code, you can undo the mistake after it happens.
18:35Here, you catch it before it does.
18:37Different timing, same result, a mistake that won't stick.
18:41Think about you again.
18:42The actions we could reverse, we would give it a reverse button.
18:45The ones we couldn't, the agent would hit the sandbox first and she would be notified, your 1,200 emails are going to get deleted.
18:52Do you want it?
18:55It's not done yet.
18:57Uh but across billions of actions that we're going through, we are learning on the way which ones can be walked back, which ones can't, and preparing the sandbox accordingly.
19:09If you take one thing away today, take this.
19:11For 2 years, the model was the bottleneck.
19:14So, everybody was racing towards better and better model.
19:17Now, the models have gotten good enough where software engineering is 100% autonomous.
19:23But, now everything else is the bottleneck.
19:26This the same model that writes your code can also do your hiring, sales, and other knowledge work.
19:36Um but, it right now it's working blind.
19:39No history, no context, no ways to verify, no guardrails, no undo.
19:44So, the bottleneck has moved.
19:48Now, it's infrastructure that nobody has yet built.
19:51And that's what we are building at Composio.
19:55Yeah.
19:56We are powering billion plus tool calls in total.
19:59Uh 300 million tool calls happening every month.
20:02And if you are building an agent, just point it to Composio and see the magic happen for knowledge work.
20:08And if you want to build the future of substrate of AI agents, then please come to me.
20:13We are definitely hiring, and there's a lot lot left to do.
20:17The models will keep getting better.
20:19The bottleneck won't be models.
20:21It will be the things around it.
20:22Thank you.