Building Agents Is Trivial Now, Context Is the Next Frontier — Jeff Ng, Unblocked

AI Engineer · 13 min · 143 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:13Hi all.
  3. 00:14Uh my name is Jeff.
  4. 00:15I'm a founding engineer at Unblock, and I'm here to talk to you about how building agents has actually gotten pretty easy.
  5. 00:22But unfortunately, they still get things confidently wrong.
  6. 00:26So, 6 months ago, it required a team's effort and basically a quarter to build out an agent.
  7. 00:33Um an agent is more than just models and tools.
  8. 00:37It's the models, the tools, and everything required to build out a production service.
  9. 00:43Here are some examples of the different systems that were necessary in order to build something out.
  10. 00:49Each one of these was basically its own company or at least a company function.
  11. 00:54Not going to go through each one of these, but you know, a few that stood out to me.
  12. 00:59First one, checkpoint and state persistence.
  13. 01:02Agent runs, they're typically long-lived and stateful.
  14. 01:06Um unfortunately, uh infrastructure itself though, those that's ephemeral.
  15. 01:12Crashing without durability can actually lead to a lot of state loss.
  16. 01:16And that state kind of includes things like message history, tool calls, as well as, you know, where you are in the loop.
  17. 01:26Without these things, you can't resume the session.
  18. 01:29Uh one option is, you know, maybe you want to restart the session.
  19. 01:32Unfortunately, that's actually quite expensive as well.
  20. 01:36Uh you lose out on all the tokens that you'd originally used, um as well as, you know, latency.
  21. 01:43Uh from a user experience standpoint, you've already triggered that session.
  22. 01:47Now you have to wait for the whole thing to go again.
  23. 01:50And lastly, side effects.
  24. 01:52Your agent might have performed some side effects, and now there's a chance of those doubling up.
  25. 01:58So, next thing, sandbox infrastructure, right?
  26. 02:01So, as we all know, we're running more and more agent-generated code as well as third-party code.
  27. 02:07This gets all run on your infrastructure, and due to that, there are some complexities.
  28. 02:13Uh because of that, we want to introduce isolated sandboxes, which help prevent uh unnecessary reads of environment secrets, unnecessary network access, you know, just in general, we don't want to take down the shared host.
  29. 02:29And then, observability.
  30. 02:31How do we answer the question, "Where did this fail?"
  31. 02:35Typically, this includes tracking logs and traces from across half a dozen systems.
  32. 02:43Everything I've mentioned here, none of this actually improves an agent's capabilities.
  33. 02:48They're all taxes one has to pay in order to get an agent out there to play the game.
  34. 02:56Thankfully, things have changed quite a bit.
  35. 02:59Um the whole ecosystem has matured quite a bit, and cloud infrastructure players such as Cloudflare, uh Vercel, AWS, they've gone and taken some of that complexity away and built primitives
  36. 03:13that these frameworks, Flu, Vercel E Maestra, with these together, you know, they've taken a lot of complexity away, and you can focus more on building the actual agent itself.
  37. 03:25The core logic that actually helps you and your team and your customers.
  38. 03:33So, here's an example of one.
  39. 03:36Uh I played around with Flu and Cloudflare, and as you can see on the left-hand side, you know, we basically handle everything as mentioned before.
  40. 03:45So, the primitives plus the framework lead to a situation where it's actually not that much code to define an agent.
  41. 03:53Uh one of the things I was shocked at when I first took a look at the documentation.
  42. 03:57To get in the details, all you really have to do when defining agent is A, deciding which model you want to use.
  43. 04:04B, the instructions or, you know, the system prompt.
  44. 04:07C, the tools that you want to ask the agent to have access to.
  45. 04:12Skills, the things I can do.
  46. 04:15As well as the sandbox location, where things are being run.
  47. 04:19So, uh to give you an example of this, I've actually gone and built out a issue enrichment system specifically for Linear.
  48. 04:31So, what this does is, given a Linear ticket and access to your code repository, it'll go out, you know, fetch the Linear ticket, determine whether or not it's a feature
  49. 04:41or a bug.
  50. 04:43From there, it'll do some code searching, provide all that context to the agent, and then come up with a plan of next steps.
  51. 04:51On the left-hand side here, this is a issue that one of my colleagues, uh smart engineer, had posted, I think, a month ago.
  52. 04:59Uh to summarize it, what had happened was, we had some pretty serious degradation in our agentic QA pipeline.
  53. 05:06Time to first character was taking 3 to 4 seconds when it should realistically be in the hundreds of milliseconds.
  54. 05:13So, let's see what happens when, you know, we put this through the system.
  55. 05:20So, as you'll see here, I've set up the agent to go fetch the agent.
  56. 05:24I've given it the skills and tools to actually go and fetch the code, search the code, and query against that.
  57. 05:31That's being passed back to the agent, which is doing some reasoning against that right now.
  58. 05:36And then, wait a little bit.
  59. 05:38At this point, we've updated the Linear issue ticket.
  60. 05:42The recommendation here is to re-enable our async dispatch, which makes sense.
  61. 05:47It allows us to run a lot more of our QE pipeline in parallel on a single machine.
  62. 05:52Sounds great, right?
  63. 05:54Unfortunately, uh this is wrong.
  64. 05:57This had actually caused an outage a few days ago in one of our uh support engineers had explicitly disabled this uh before this ticket was uh shown.
  65. 06:07So, where did things go wrong?
  66. 06:11Why was the uh you know, why did I get it wrong?
  67. 06:15The agent I had written, it didn't have a full picture.
  68. 06:18It was missing the context from the Slack discussion that happened after the issue where the engineers came together, uh went through the actual outage, what went wrong, what was the fix, and the next steps.
  69. 06:31It also was missing the postmortem uh linear ticket, which came as a result of that.
  70. 06:36In general, it had a narrow understanding of the problem.
  71. 06:41This concept of missing knowledge and intent that's stored across an organization and different systems is something that comes back and back again.
  72. 06:49And since this was deployed as a background agent, this is going to make that mistake silently in the background, misinforming both my teammates and potentially other agents.
  73. 07:00So, I guess the next question is, why don't we run into this locally?
  74. 07:03You know, we all use agents locally, we don't necessarily run into these issues.
  75. 07:08Well, you, the human, the engineers, we currently act as that context layer.
  76. 07:14When working with an agent, you know, you're there to ask questions, catch any errors, and supply the missing facts on every single turn.
  77. 07:23A person knew why the code is the way it is, what broke last time, and what we've decided to do about it.
  78. 07:30The agent, though, it only has what's on the right-hand side, right?
  79. 07:33It has instructions, the tools and skills we specifically gave it, the code, as well as the ticket in front of it.
  80. 07:40When an agent is in the loop, well, sorry, when a human is in the loop with the agent, we're there to catch the steer.
  81. 07:47Ultimately, we're there to babysit the agent.
  82. 07:50But as agents have gotten trivially easy to deploy as I showed earlier with Flu Cloudflare, the without the human in the loop, this issue becomes more and more prevalent.
  83. 08:02This missing context becomes a sign of failure.
  84. 08:05You know, all that intuition and knowledge that we've had as humans needs to be replaced.
  85. 08:10Something needs to carry the load.
  86. 08:14So, that thing, that's a context engine.
  87. 08:17A context engine is a system that provides task-relevant information based on who you are and what matters.
  88. 08:25It also resolves all the conflicts across multiple data sets.
  89. 08:29It understands your access rules or the agent's access rules and only uh respects that and only provides information that's relevant.
  90. 08:36And most importantly, it delivers a synthesized understanding that an agent can act on, not just a list of documents that I have to reason upon itself.
  91. 08:47So, how does this context engine work?
  92. 08:50Well, let's take a step back.
  93. 08:52What does an agent actually need?
  94. 08:55An agent needs Clearly, it needs context outside of just your source code.
  95. 09:00Think about everything that you need to work day-to-day.
  96. 09:03It's not just the code.
  97. 09:05It's, you know, the Slack discussions where decisions are made, the documentation where we show all the best practices.
  98. 09:13All that is important to your day-to-day process, and that's true for your agent as well.
  99. 09:17So, what we do here is we connect everything.
  100. 09:19The docs, code, tickets, conversations.
  101. 09:23We then build a model of your organization, of your system.
  102. 09:27And we piece how all these things work together and make it generally available to your agents.
  103. 09:32From that model, the agents are only provided a a of that data, which has been reconciled, ranked, and scoped to your permissions.
  104. 09:40Scattered context comes in, grounded context comes out.
  105. 09:45The obvious next question is, why can't we just do this with MCP, right?
  106. 09:49You could connect a Slack MCP, a Linear MCP, a GitHub MCP, and with that, all that data is accessible.
  107. 09:57MCP is great at access, but access isn't understanding.
  108. 10:02An MCP hands the agent the raw results, and you know, you're now dependent on that agent to actually decide what to believe in.
  109. 10:09You end up flooding the agent with irrelevant data, filling up the context window, and you know, overall context costs just go up.
  110. 10:17It also leaves the local agent to handle conflicts in data.
  111. 10:21You know, your Linear MCP and your Slack MCP may come back with different results.
  112. 10:25You're just leaving the agent to make that decision somewhat ad hoc at the moment.
  113. 10:32So, back to the original problem I had earlier.
  114. 10:35This is the same file, same engine, but now we've connected the context agent.
  115. 10:40Uh what we do here is is we're currently prompting Unblock to do some research on the ticket and provide that context to the agent.
  116. 10:49So, let's see that in action.
  117. 10:54Sorry about that.
  118. 11:01So, here we go.
  119. 11:02Uh we're doing the very similar thing.
  120. 11:04We're fetching the Linear ticket.
  121. 11:06But, you'll notice here that we're actually calling the Unblock context engine.
  122. 11:09And what's done here is actually it's found the relevant Linear postmortem, as well as a Slack conversation where we've had the entire discussion between the engineering teams.
  123. 11:18And as part of that, we've returned a understanding, and that's now been provided to the agent as a summary.
  124. 11:26So, the agent no longer has to actually reason from those documents.
  125. 11:31Uh at this point, you'll notice here the agent now has been updated.
  126. 11:41Uh the recommendation has gone from breaking and causing another issue to actually preventing a another outage.
  127. 11:53So, the example I've shown here is issue ticket management, but this context layer can actually go a lot further.
  128. 11:59Uh for example, coding.
  129. 12:02Everyone here does uh coding with uh cloud code or cortex.
  130. 12:05Using an Unblocked context engine to actually hydrate the agent plan goes a long way in terms of saving context and tokens.
  131. 12:13Uh code review.
  132. 12:15It makes the PRs look as if they've been reviewed by an expert on your team.
  133. 12:19Who doesn't like that?
  134. 12:21As well as surfacing the correct answers to your customer success team as well as sales.
  135. 12:27In general, there are many instances where you might want an agent to have institutional and tribal knowledge of your organization.
  136. 12:37Just want to leave you on this.
  137. 12:38I think this quote encapsulates what we're trying to solve at Unblocked.
  138. 12:42The gap isn't intelligence, it's context.
  139. 12:46So, thank you.
  140. 12:48Uh I'll be at booth P16 along with the rest of my team if you guys have any questions.
  141. 12:53There will be additional breakout sessions later tomorrow, I believe, that goes a lot more in depth about actually how the context engine works and you know, how you can benefit from that.
  142. 13:03Cheers.
  143. 13:04[applause]