WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=k2ClBT4aqAg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/k2ClBT4aqAg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.520 --> 00:00:20.240
Today I'm going to talk about how you can take your coding agents beyond code generation and how API context might be the answer for that.

s2
00:00:20.240 --> 00:00:21.080
I'm Kamal.

s3
00:00:21.080 --> 00:00:24.000
I'm a staff engineer at Postman.

s4
00:00:24.000 --> 00:00:27.840
And Postman, as you might know, is the end-to-end API platform.

s5
00:00:27.840 --> 00:00:37.160
But beyond the desktop app that allows you to test your REST APIs and build your APIs, uh Postman also has uh a lot of cloud features.

s6
00:00:37.160 --> 00:00:41.480
Empowering all of those cloud features is an extensive microservices architecture.

s7
00:00:41.480 --> 00:00:47.000
We have more than 150 microservices and thousands of uh REST API endpoints.

s8
00:00:47.000 --> 00:01:02.520
Uh so, I'm going to be using our real uh engineering use cases and workflows and on top of our coding agent experiences and what we learned from it and how we've been improving the experience and the effectiveness of our coding agents with API context graph.

s9
00:01:02.520 --> 00:01:08.360
So, our journey with the coding agents started similar to how many of you might have gone through the same journey as well.

s10
00:01:08.360 --> 00:01:13.600
Starting with line and function level auto completion, this is the Copilot uh days eras.

s11
00:01:13.600 --> 00:01:22.720
Uh moving on to full code base uh refactors and one-shot implementations and one-shot fixes, uh this is your composer eras and cloud code eras.

s12
00:01:22.720 --> 00:01:25.920
Now, moving on to fully autonomous coding agent workflows.

s13
00:01:25.920 --> 00:01:35.120
Uh we have agents that end-to-end start the workflow and then make the changes, verify the changes, send pull requests, and even verify pull requests as well.

s14
00:01:35.120 --> 00:01:45.200
What we see as the coding agents excelling really well at are greenfield projects or fixes and improvements within the context of a single project.

s15
00:01:45.200 --> 00:01:48.120
But production systems tend to be slightly more complex.

s16
00:01:48.120 --> 00:01:51.880
They tend to be distributed systems split across multiple microservices.

s17
00:01:51.880 --> 00:01:57.000
They're often deployed across multiple environments, even multiple different versions in each of these environments.

s18
00:01:57.000 --> 00:02:08.520
We have several applications, front end, CLI, API, all of them talking to our back end infrastructure, and all of this is spread across hundreds of repos where the code is distributed across all of them.

s19
00:02:08.520 --> 00:02:15.000
When we think about improving coding agent performance, the first area that we start looking into is documentation.

s20
00:02:15.000 --> 00:02:21.800
Documentation is a great place to start, uh but as we all know, developers, we hate writing and maintaining documentation.

s21
00:02:21.800 --> 00:02:23.200
It's an added overhead.

s22
00:02:23.200 --> 00:02:26.600
It costs time, and it becomes tricky to go and maintain over time.

s23
00:02:26.600 --> 00:02:36.840
But if you offload documentation purely to LLMs to author and maintain, research points out that purely LLM maintained and written documentation tends to degrade exponentially over time.

s24
00:02:36.840 --> 00:02:40.880
Some sort of human curation is definitely needed to make it more effective.

s25
00:02:40.880 --> 00:02:48.160
Skills and MCPs are a really good alternative to look at, but we all know skills and MCPs are only as good as the context that they're provided to.

s26
00:02:48.160 --> 00:02:52.120
Agent memory is actually a really good solution for uh this scenario.

s27
00:02:52.120 --> 00:03:02.720
Um in terms of real-world uh experiences, most of this knowledge is probably already tribal knowledge in people's heads, and now it's getting translated into agent memory within the coding agent are the harness.

s28
00:03:02.720 --> 00:03:17.239
The biggest limitation factor that we see with this is agent memory is still very limited to an individual user, and no standard patterns have now emerged yet in terms of sharing agent memories across your broader teams or across your entire engineering organization.

s29
00:03:17.400 --> 00:03:27.320
While we were trying to solve this problem for the coding agents on the engineering side, our go-to-market teams were also building business agents for go-to-market agents for their end-to-end

s30
00:03:27.320 --> 00:03:28.280
executive use cases.

s31
00:03:28.280 --> 00:03:43.200
So, they've been building uh custom agents that are purpose-built with customer data that is brought in from multiple data sources, and they've done several iterations to streamline all of this unstructured data coming in from all of these data sources into a structured data on top of which these agents operate.

s32
00:03:43.200 --> 00:03:55.120
What they have ended up with is a really optimized like rack plus graph rack pipeline that serves as a centralized context layer that makes all of these individual uh business agents work really well and solve the problem for all of them.

s33
00:03:55.120 --> 00:04:01.400
So, we started looking at how can we learn from this and what can we do to make the coding agent performance really better?

s34
00:04:01.400 --> 00:04:08.880
And when we looked at this, what we realized is in a distributed systems and microservices architecture, APIs are the context layer for you.

s35
00:04:08.880 --> 00:04:22.360
Complex engineering architecture is often broken down into domains and models, and each microservice is responsible for owning one single responsibility as per a model or a domain, and APIs are what defines what that responsibility

s36
00:04:22.360 --> 00:04:23.040
is.

s37
00:04:23.040 --> 00:04:30.600
And API calls between systems is usually how you identify workflows and user journeys that span across multiple systems.

s38
00:04:30.600 --> 00:04:34.040
So, agents are only as good as the context that they're given.

s39
00:04:34.040 --> 00:04:38.080
So, we set out on a journey to build the best context layer for the agents.

s40
00:04:38.080 --> 00:04:44.080
So, we went out to start building an API context graph for what Postman's engineering architecture looks like.

s41
00:04:44.080 --> 00:04:49.640
The journey started by identifying and cataloging every single microservice that was on production.

s42
00:04:49.640 --> 00:05:00.919
And then we identified every endpoint that is exposed by each of these REST API services, and then we captured how each and every one of them is implemented right down to the exact line of code where it was implemented.

s43
00:05:00.919 --> 00:05:08.760
And we also mapped how each and every service talks to each other, and how each endpoint connects to each each and every other endpoint.

s44
00:05:08.760 --> 00:05:17.520
We also tracked how the data goes all the way down across the stack, including databases and caches, and all the way down in the entire stack.

s45
00:05:17.520 --> 00:05:27.360
We also identified how applications like front ends, CLIs, APIs, how they interact with the whole back end architecture and back end infrastructure.

s46
00:05:27.360 --> 00:05:34.200
We get give all of this data and we index all of this data using LLM to produce a centralized and effective context graph layer.

s47
00:05:34.200 --> 00:05:48.440
But one key principle that we had in mind always was every single data point that landed on the context graph always had to be grounded down in a hard truth, and that hard truth is either a line of code that is implemented in your code base, or it could be a

s48
00:05:48.440 --> 00:05:52.120
an exact trace that is coming from a production telemetry.

s49
00:05:52.120 --> 00:06:00.600
With this in place, we set out on a journey to populate our context graph, and this is a snapshot of what our engineering architecture kind of looks like.

s50
00:06:00.600 --> 00:06:05.040
So, it is some of the very common patterns that you would expect start to emerge.

s51
00:06:05.040 --> 00:06:13.880
You start to see a central cluster of core services, of very tightly intertwined intertwined APIs and services that are tightly dependent on each other.

s52
00:06:13.880 --> 00:06:18.320
You find one or two heavily bloated services that are on the outer edges of these systems.

s53
00:06:18.320 --> 00:06:23.760
The ones at the center is typically what you would classify as tier zero or tier one services.

s54
00:06:24.560 --> 00:06:33.160
So, now that we have the context graph, when we start looking at how does how do we measure what does it add in terms of improvements to the coding agents?

s55
00:06:33.160 --> 00:06:39.840
So, we took real examples from the Postman Git organized from our organization's Git repos.

s56
00:06:39.840 --> 00:06:48.919
We took real PRs that were raised to the GitHub repos, and for each and every one of them, we identified what the developer was trying to do, and what the actual outcome was

s57
00:06:48.919 --> 00:06:50.080
at the end of the PR.

s58
00:06:50.080 --> 00:06:57.240
We also took like important projects, and what were the key design decisions and takeaways that were being discussed in those design discussions.

s59
00:06:57.240 --> 00:07:06.919
We mapped each and every one of them into an eval, and we mapped what the actual decision that was taken in retrospect as the ground truth that we wanted the evals to confirm into.

s60
00:07:06.919 --> 00:07:17.320
To to run the validations, we were able to get the Postman's AI agent connects to the API context graph, and we were able to validate all of these against the Postman AI agent as well.

s61
00:07:17.320 --> 00:07:27.240
But to keep the comparison fair, we also validated this against a generic coding agent, in this case Claude code, and we tried to verify these use cases simply by like

s62
00:07:27.240 --> 00:07:29.400
a straightforward GitHub code search.

s63
00:07:29.400 --> 00:07:38.919
We also took the same context graph, and we made it available as a skill, and also compared that against the generic coding agent, in this case Claude code.

s64
00:07:38.919 --> 00:07:42.080
So, you're going to see the results of of in a graph plotted like this.

s65
00:07:42.080 --> 00:07:48.440
So, we scored every eval on zero to five scale and we also measured the exact amount of tokens it took for each of these runs.

s66
00:07:48.440 --> 00:07:54.280
So, what you will see if something shows up on the top right, it means that the agent did a good job.

s67
00:07:54.280 --> 00:07:58.200
It was able to score high on the evaluation, but it spent a lot of tokens doing so.

s68
00:07:58.200 --> 00:08:03.160
If it's on the top left, it did a great job not only scoring high, but it also did it much more efficiently.

s69
00:08:03.160 --> 00:08:11.200
If something goes beyond the dash line, it just did not meet the bar of what a good score should be on that evaluation.

s70
00:08:11.200 --> 00:08:17.960
So, then we bucketed like come up some of the common use cases and scenarios that we see in day-to-day development and where do they fall under?

s71
00:08:17.960 --> 00:08:23.200
One key scenario that we always notice is API discovery.

s72
00:08:23.200 --> 00:08:30.600
Any developer who is making a change either to the front end or back end integrating with other systems, what is the right API to call?

s73
00:08:30.600 --> 00:08:38.960
And in real-world systems, you probably have hundreds of different of the hundreds of different APIs and most likely you have multiple APIs that probably do the same thing.

s74
00:08:38.960 --> 00:08:46.320
So, in this case, uh a simplified example that I've taken is what is the right API to to call to get the profile name for a particular user.

s75
00:08:46.320 --> 00:08:50.360
So, in this evaluation, this is a pure search use case and the results are what you would expect.

s76
00:08:50.360 --> 00:08:56.760
Like with the context graph, the agent scored really well as compared to like pure code search where the agents don't do really well.

s77
00:08:56.760 --> 00:09:03.520
And there's a slight deviation between like Postman and Claude, but the deviation is not that big enough uh for us to be uh diving deeper into.

s78
00:09:03.520 --> 00:09:07.160
The second second use case is where it gets really uh interesting.

s79
00:09:07.160 --> 00:09:11.160
So, this is what we call API design and redesign scenarios.

s80
00:09:11.160 --> 00:09:19.680
So, in this scenario, let's take a developer who's building a new feature and to build that feature out, they've designed a new API and they've designed the schema for the request input.

s81
00:09:19.680 --> 00:09:21.960
They have defined the uh implementation of it.

s82
00:09:21.960 --> 00:09:27.920
They have tested it out from their side and then they go integrate it with the product and they're happy with the product and they're ready to ship.

s83
00:09:27.920 --> 00:09:35.960
At the last minute, one of the dependencies comes to them and then say, "Hey, would it be nice if I just had this one additional property that I would like to pass to to API

s84
00:09:35.960 --> 00:09:38.320
that I would like to use at a later point in time.

s85
00:09:38.320 --> 00:09:44.440
So, the developer is like, "Maybe we don't have enough time to evaluate all the different ways in which it can be done.

s86
00:09:44.440 --> 00:09:46.360
What is the right schema to do this?"

s87
00:09:46.360 --> 00:09:55.240
So, they just add this one additional context, which is a free-form JSON object, and they add a to-do in the code saying, "Revisit this and add a schema 2 weeks later."

s88
00:09:55.240 --> 00:09:56.480
The feature ships to production.

s89
00:09:56.480 --> 00:09:57.680
Everybody's happy.

s90
00:09:57.680 --> 00:10:08.400
2 weeks later, the developer comes back and finds out that that one particular property which is supposed to send one attribute now has 15 attributes, and it's being passed by one service or it might be five services.

s91
00:10:08.400 --> 00:10:09.680
It might be 10 services.

s92
00:10:09.680 --> 00:10:21.040
Now, the developer has no idea what exactly is the shape in which the request is being passed down uh to the API, and what are all the different ways in which every single consumer has been passing data down to.

s93
00:10:21.040 --> 00:10:29.640
So, it can be uh So, this is what I call like analysis of a shape of input for each and every API, and why they are used like that.

s94
00:10:29.640 --> 00:10:31.760
So, this can be hard for a bunch of reasons.

s95
00:10:31.760 --> 00:10:36.840
First, tracking down every single consumer of your API can be impractical in real-world scenarios.

s96
00:10:36.840 --> 00:10:44.200
Even if you had service dependencies and service maps, finding out the exact shape of input used by each and every consumer can get really tricky.

s97
00:10:44.200 --> 00:10:52.839
Engineers often lack the detail or depth of information in production logs, and for privacy reasons, it may not even be available at a lot of times.

s98
00:10:52.839 --> 00:11:00.920
Even if the shapes are identified, finding out exactly why the consumer decided to pass it down like this is a very, very tricky problem.

s99
00:11:00.920 --> 00:11:08.040
But, in this scenario, when we took this evaluation and then ran it with the Postman agent, what was surprising was the Postman agent almost got it really well.

s100
00:11:08.040 --> 00:11:25.400
And the thing that helped in this case is, first of all, the context graph having an awareness of every single consumer of an API, but not just that, but having the drill down or the ground truth of where exactly each of these APIs were implemented in code, and the context surrounding that code helped the agent understand

s101
00:11:25.400 --> 00:11:26.720
why the change was done.

s102
00:11:26.720 --> 00:11:36.440
And very quickly, it was able to figure out exact unique shapes for every consumer that was calling this API, and you could be able to make the changes uh quite reliably and confidently.

s103
00:11:36.440 --> 00:11:50.440
And you can see that the pure code search does not do really well, but with Claude and the skill, it does slightly better, but it tends to give up like after it's done like enough number of turns and waiting for the user to kind of confirm what to do next.

s104
00:11:50.440 --> 00:11:55.760
So the other very common use cases that we start to see is impact assessment.

s105
00:11:55.760 --> 00:11:59.120
How do you figure out what is the impact of one particular change?

s106
00:11:59.120 --> 00:12:02.600
So if I change this API, what are the potential use cases?

s107
00:12:02.600 --> 00:12:08.200
Or maybe there's an existing API, there's a V2 API, and I'm interested in introducing a V3 API.

s108
00:12:08.200 --> 00:12:15.040
How do I understand how the V2 is API V2 API is used first so that I can design a better V3 API.

s109
00:12:15.040 --> 00:12:28.920
So this is one scenario where we see the context graph really shine well, and you can see that like Postman Agent not only does really well in terms of the quality of the output, but also it's able to do it with almost like half or even 1/3 when compared to pure code search.

s110
00:12:28.920 --> 00:12:30.840
But this is not smooth sailing all the time.

s111
00:12:30.840 --> 00:12:44.680
Sometimes we see failures, and in this case, one of the use cases that we threw at it was find out the impact of a change in one particular PR where it introduce a change to an endpoint and figure out all the all the downstream services that are affected by that endpoint,

s112
00:12:44.680 --> 00:12:48.640
and we started seeing a failure, and you can see the graph like almost inverted.

s113
00:12:48.640 --> 00:12:53.839
We see Postman not only scoring down, but also like consuming a lot of tokens while it did that.

s114
00:12:53.839 --> 00:13:05.680
And when we investigated in and into it and then tried to understand why it was failing, what we realized was that uh there were new consumers between the time where the eval was written and the time it the evaluation was run.

s115
00:13:05.680 --> 00:13:14.760
So the context graph is not only important to populate, but it's equally important that you keep it up to date, and any delays in syncing your context graph with the ground truth

s116
00:13:14.760 --> 00:13:16.920
uh creates gap in the result.

s117
00:13:16.920 --> 00:13:24.720
And if there's a gap in the result, it immediately leads to a loss of trust by the developer, and that translates into lack of usage.

s118
00:13:24.720 --> 00:13:34.320
Not only detecting new APIs and new dependencies, detecting removals of APIs and removal of dependencies is equally important and that can also lead to confusions and problems.

s119
00:13:34.320 --> 00:13:43.480
So, you might think that these use cases may or may not be common to you, but when we went back and retrospected on all the PRs that were raised in the top 10 repos, we noticed that almost 75%

s120
00:13:43.480 --> 00:13:46.640
of the PRs had some effect on the APIs.

s121
00:13:46.640 --> 00:13:57.840
So, either these changes are introducing a change to the external surface area of an API or they're introducing an integration of a new API and both of them have can have like really

s122
00:13:57.840 --> 00:14:00.680
cascading impact downstream.

s123
00:14:01.440 --> 00:14:09.360
So, these use cases were clear for us to do like a before and after comparison with the context graph and without the context graph, but as soon as we had this information,

s124
00:14:09.360 --> 00:14:13.520
we were able to ask questions that we were previously not able to ask.

s125
00:14:13.520 --> 00:14:23.440
And these questions asked allowed us to be able to take the Postman agent and say, "Hey, do a complete engineering analysis and an end-to-end architecture review of our entire

s126
00:14:23.440 --> 00:14:24.520
ecosystem."

s127
00:14:24.520 --> 00:14:27.760
So, this is where the light bulb moment happened for us.

s128
00:14:27.760 --> 00:14:36.240
When we asked the Postman agent to do this, it produced a 21-page engineering report going through every single weakness and points of vulnerability.

s129
00:14:36.240 --> 00:14:49.280
It found out interesting insights for us including cycles of dependencies, which can be really problematic in terms of an incident, risky long-running migrations, which are signals of technical debt, and then single points of failure, which can be really problematic

s130
00:14:49.280 --> 00:15:02.440
or issues that are waiting to happen, blind spots and telemetry, where stuff that is in the implementation does not reflect in telemetry, and then it came out with clear action items for hey, this is what your CEO, this is what your CTO, and this is what your staff engineers

s131
00:15:02.440 --> 00:15:03.839
might need to do.

s132
00:15:03.839 --> 00:15:17.720
On top of all of these things, when we look at what we can do for further improvements, one of the areas that stands out is bringing in the semantic layer of what does your business do and connecting it into the context layer of what your coding agents take as input

s133
00:15:17.720 --> 00:15:19.240
for their workflows.

s134
00:15:19.240 --> 00:15:26.160
Another layer of improvement that we're looking into is how can we improve and add additional reasoning of why does a data model exist?

s135
00:15:26.160 --> 00:15:27.920
Why does the domain map like this?

s136
00:15:27.920 --> 00:15:31.040
Why does this API exist versus other APIs?

s137
00:15:31.040 --> 00:15:33.640
Why Why should you use one API over the other API?

s138
00:15:33.640 --> 00:15:42.640
And all of these helps in subsequent coding decisions and all of that translates into really effective output when you're using them in your day-to-day development workflows.

s139
00:15:42.640 --> 00:15:44.480
So, takeaways and learnings.

s140
00:15:44.480 --> 00:15:50.480
We've realized that API context graph serves as the context layer for coding agents in a real-world distributed systems.

s141
00:15:50.480 --> 00:15:58.960
Gives a high-level view that helps you answer questions about your engineering architecture that you were not able to answer in the context of a single repo or a single microservice.

s142
00:15:58.960 --> 00:16:03.680
It shortens the time to find the right way to do a particular problem.

s143
00:16:03.680 --> 00:16:11.040
And all of this means that do the doing this either with the right amount of quality or being much more efficient in being able to do this.

s144
00:16:11.040 --> 00:16:23.600
So, efficiency translates either into your cost savings, but the way that we see it is if we can do one particular task a lot more efficiently, it means that we can do it more often and this allows us to do the same evaluations

s145
00:16:23.600 --> 00:16:32.839
at a lot more frequency including bringing it down at the PR level or even bringing it down right down to the development machine or in the development environment where the

s146
00:16:32.839 --> 00:16:34.480
where the engineer is working on.

s147
00:16:34.480 --> 00:16:45.280
So, the sooner we can move the decisions and catch problems earlier in the life cycle, we believe that the agents who are generating code can translate into peace of mind for the developer and help them ship

s148
00:16:45.280 --> 00:16:49.360
changes to production with a lot more confidence going forward.

s149
00:16:49.360 --> 00:16:51.280
So, that's everything from my side.

s150
00:16:51.280 --> 00:16:54.360
All of these capabilities are available within the Postman as a platform.

s151
00:16:54.360 --> 00:16:58.440
These are also available to use as early access.

s152
00:16:58.440 --> 00:17:02.640
I will be around outside right after the talk if you have any questions or follow-ups.

s153
00:17:02.640 --> 00:17:06.040
We also have a booth S32 right across the hall from here.

s154
00:17:06.040 --> 00:17:07.240
That's everything from my side.

s155
00:17:07.240 --> 00:17:09.520
Thank you so much.

s156
00:17:15.000 --> 00:17:17.000
Woo!

s157
00:17:24.081 --> 00:17:26.081
[music]