The Death of Developer Advocates — Stephanie Jarmak, Sourcegraph
https://www.youtube.com/watch?v=Lrw0jqBNaw0
AI Engineer · 18 min · 205 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Hi everyone.
[00:13] Sorry for the start with technical difficulties and all of that.
[00:17] Uh, we made it to the end of this track.
[00:19] Super exciting.
[00:21] Thank you everybody for sticking it out this long.
[00:24] Um, are there any developer advocates or devrel people in the audience?
[00:29] Raise your hand.
[00:30] Yeah, okay.
[00:31] So did you come to like throw tomatoes at me cuz I'm talking about the dead now.
[00:34] Okay, so it's not going to be all doom and gloom like that.
[00:38] Um, a bit of like backstory in this.
[00:41] Um, I'm a research scientist.
[00:43] So last year I was an astronomer.
[00:46] Um, and I just sort of like wound up.
[00:47] I didn't know what GTM was or any of that.
[00:49] I just sort of wound up in this.
[00:51] Um, and I submitted like a bunch of boring sciency eval talks that were unceremoniously I I assumed thrown into the trash uh, for this conference.
[01:00] But my manager, who is a developer advocate, he put in, you know, the death the death of developer advocates, which is, you know, appropriately buzzworthy and hypey.
[01:09] And so so that was great.
[01:11] But his title is developer advocate, so it didn't really necessarily make as much sense for him to be coming up here and giving his eulogy.
[01:18] So we brainstormed like maybe I would dress up as like a robot and like a maul him and attack him on the stage or something like that.
[01:25] Um, but then it just like logistically it was going to be hard to do that.
[01:29] Uh, so he just went on vacation.
[01:31] Uh, so I'm here uh, as the agent advocate uh, to talk about this sort of like new role and uh, try to advocate for it and uh, convince all of you that we should all be agent advocates to help
[01:46] uh, in this new era.
[01:48] So uh, zooming out a little bit and going back uh, in time a bit because uh, I was trying to talk about developer advocates to somebody at the conference yesterday and their eyes like glazed over they had no idea what I was talking about.
[01:59] So just to sort of talk about what what this thing is that I'm saying is dead.
[02:03] Uh so back in the '80s, right?
[02:05] It was called like software evangelism where one would go forth and speak the good word of the product and bring it out there.
[02:13] But then fast forward to the 2010s or so, that's when developer advocacy advocacy started to become a thing where now instead of having this single trajectory of the communication pathway, now it's a feedback loop and a two-way street where you have these people with very deep empathy for developers
[02:29] who understand them and speak their language and could understand um what their needs were um and then bring that back to the product.
[02:37] And then um these developers, right?
[02:39] Fast forward even more, they have so much influence within their company and basically become these like kingsmakers.
[02:46] Uh and so the developer experience became a very important aspect of the go-to-market sort of strategy.
[02:52] Um but now in 2026, uh developers are no longer working alone and what it means to be a developer is completely changing.
[03:00] Um and so our role, right, as developer advocates um developer in developer relations, we're relating to developers.
[03:07] And so as the role of developers fundamentally changing, so must then does the role of the developer advocate.
[03:15] Um so in this slide I'm just kind of talking about the other users, right?
[03:20] So what's happening uh with DevRel uh outside of the agent.
[03:24] So most of the talk is going to be talking about the agent as a user.
[03:27] But I also did did want to bring up, right, that engineers they're becoming like these orchestrators of these fleets of agents, um babysitters and whatnot of these things.
[03:38] Um and their job, like all of the job postings and whatnot, there's language is continuously changing, right?
[03:42] They're um expected to have this AI fluency.
[03:46] Um and at the same time, there's also, you know, people like me, like uh non-engineers, right?
[03:53] I was a research scientist.
[03:54] I had like zero commits on GitHub last year, and now I have 12,000, and I'm like an open source maintainer for multi-agent orchestration framework.
[04:01] Like, we have so much like capability now with all of these agents, and now anybody with these agents can use dev tools, essentially.
[04:09] So, you have this whole other persona and ICP uh to potentially be relating to and um having empathy with when you're there using your product.
[04:19] So, let's talk about now this whole new user that we have in the form of an agent.
[04:24] Um so, an agent is somewhat unique, right?
[04:28] In the sense that it is both the user of your tool in a very similar way to the developer.
[04:34] It's going out reading your docs, but it's just reading them differently cuz it's a machine.
[04:37] Um you know, it's calling the API.
[04:39] It's encount- it's ha- has its own frustrations with how it's encountering errors and recovering from them, right?
[04:44] But then it's also a recommender of your tools.
[04:47] Um but somewhat similar, right?
[04:48] To developers in the way that they are also recommenders of your tools in a more organic, bottom-up way.
[04:53] Um so, the whole, you know, basis for DevRel, right?
[04:56] Is to encourage that bottom-up adoption.
[04:58] But now the adoption and the recommendation system, a lot of it's being driven by the agent itself.
[05:04] That is either, you know, maybe servicing your product directly through like ChatGPT or Claude, like directly in a Q&A sort of environment, or it's, as we had heard like in some of the previous talks where the speaker asked folks like, "How many of you have just let your agent
[05:18] install a library for you?"
[05:19] And like, there were many hands went up, right?
[05:21] So, there's this like recommender of tools where basically it's just installing these like frameworks and things um directly and embedding them into the workflow um and sort of working with the developer
[05:32] um in that taste.
[05:36] So, I know it's late for numbers.
[05:37] You don't have to read them or anything like that.
[05:40] Um so, I have a couple different concrete examples for measuring these seats, right?
[05:44] Cuz I am a data science scientist nerd person.
[05:48] Um so one of my first projects when I was uh working on this um uh when I became an agent advocate was to build um a benchmark called CodeScaleBench.
[05:58] And so I developed hundreds of tasks that were reflective of the software development life cycle.
[06:02] And I basically unleashed these agents with and without um our product tooling.
[06:07] So I work at Sourcegraph and we have a code navigation MCP tool.
[06:10] Um and the point of that was to understand, okay, how is our tool helping the agent do the work that it's, you know, going to be doing.
[06:18] Um and when it isn't working well, why isn't it working well?
[06:21] So that we can then go in and actually fix that.
[06:24] Um so I have thousands and thousands of these traces.
[06:26] And I I as we have heard in like the previous talks, like now we have these amazing logs of data for like these really tight feedback loops where you can see exactly where it's breaking down and then go in and fix it.
[06:37] Uh so this one specific example here was um when I was looking at how it was like using a read tool.
[06:43] Um and the model had the these expectations based off of its like biases from how it from its training data of what it expected for a particular um command
[06:52] um that would be available within the tool.
[06:55] And there's nothing in our description uh that would have like led it to believe otherwise.
[06:59] So it tried to use like read line instead of start line or something like that.
[07:03] And then it ended up failing, but then at least the error told it why it failed.
[07:07] So it was like, okay, that that was a good part of it.
[07:09] So it was able to fix itself.
[07:11] But then it's burning right an entire turn just failing.
[07:15] And you could just go in and fix that um aspect of like how it's interacting with the tool.
[07:19] And this is really important, right, to gather that feedback um and understand the friction that like now your new agent user is having with your tool because it's the way that um different organizations are going to be evaluating your tool, right?
[07:30] In terms of not just is it working well, but like how many tokens is the agent dealing with to work with your tool?
[07:36] And how fast is it?
[07:37] Um so this is, you know, really an important aspect of the role is measure um, how these users are using it.
[07:44] The other side of it um, is like the recommendation layer, right?
[07:48] So, the uh, GEO instead of SEO.
[07:50] So, the generative engine optimization.
[07:54] Um, and I didn't mention it before, but in the previous slide um, I had a GitHub repo.
[07:59] Like, there's two different toy projects that I put together.
[08:01] At the end of the talk, there's like a QR code with a link that you can send your agent to to like have access to all this.
[08:07] So, don't worry about like taking screenshots All of all of the data will be released to you.
[08:13] Um, so anyway, back to this.
[08:15] Um, I set up a little experiment, right?
[08:18] To see how uh, these different chatbots and agents and whatnot were recommending our product or like mentioning it at all.
[08:26] Um, and so there's a, you know, process to that cuz you have you want to understand like, what is your ICP actually doing when you would want your product to be surfaced?
[08:36] So, there was a bit of a gap that I found.
[08:39] Um, if I had designed some of these prompts around somebody who like was actively shopping for this sort of code intelligence sort of tooling and doing a comparative sort of thing, then our product was ending up being recommended like 65%
[08:53] of the time.
[08:54] Um, but what I found was the arguably like the more typical use case and where we'd want to be showing up for people when they're encountering a specific pain or have a specific need where our product could serve them better,
[09:06] uh, zero mentions, right?
[09:09] So, in this particular instance, um, I put in a prompt that was like, we keep breaking downstream services when we change shared libraries because we can't see all the consumers.
[09:18] And you know, our one uh, part of our product is being able to have this observability layer to like see across all the repos.
[09:25] So, we'd want uh, some level of like attribution or recognition from um, an agent to say, "Hey, you could use something like this."
[09:32] But instead it said, uh, "You could just have your developers make a wiki page or something.
[09:38] Um but with this, you know, we wouldn't know that without running these sorts of experiments um and getting this sort of data.
[09:45] So, what this leads to is like then you can have a hypothesis of okay, maybe the messaging that we're putting out there isn't uh attributing some of these pains and use cases clearly enough
[09:55] for the agents to be picking it up.
[09:56] So, we have uh like a content campaign in the works to um make changes to our website and then we can directly measure whether that has like an actual lift and not necessarily in the form of like anything that was baked into the training data, but then how uh the agents that are using those like web search
[10:15] tool calls, how they are then interpreting um the information about your product.
[10:21] So, you know, there are just some um different ways that you could think about guiding the agents um to help support like the servicing, the discoverability of your product and this user finding it um at their moment of need, right?
[10:35] Um so, for example, um this whole field is moving so fast.
[10:41] Uh so, I mean, training data is always going to be stale.
[10:44] Actually, in the um GEO pilot study that I did, the data that I was showing there, that was using Claude Sonnet 4. It's very old um obviously and I just today, this afternoon, ran it with 4.6
[10:56] thinking that okay, surely it's going to it's going to be better.
[10:59] It's going to know like improved information about our product, but uh so, in the previous model, it kept pitching Cody, which was like one of our older products.
[11:08] Um but if I when I uh ran it again, it it pitched Cody even more, right?
[11:14] Cuz like now you have all of these like old models like uh outputting content that then is like compounding in the internet.
[11:21] So, you have to figure out like how to bury all of that uh noise with your true signal.
[11:27] Um and the way that some folks are working on that is as we've heard from other people like these LLMs at TXT uh sort of pages, right?
[11:35] So, you have more authoritative sources of truth that you're hoping to direct the agent to.
[11:40] But, they still need to be using the tools and using real-time information and provenance to be able to give accurate answers about your product.
[11:48] You also want to give like the agent something to quote, right?
[11:51] They they they want to bring something that they can really sell to the to the user, right?
[11:56] So, you want current examples and keep everything up-to-date.
[11:59] Like, even if your stuff hasn't changed in 2 years, which would be shocking.
[12:03] Even if it hasn't, like keep everything up-to-date and fresh because that, you know, part of that is how they have their relevance algorithm.
[12:10] And they also really really like charts and FAQs and things like that.
[12:14] And you also want to make sure your product is where the agents are, right?
[12:18] You're going to market.
[12:19] So, go go to agent market, right?
[12:22] So, make sure you're in the marketplace in the MCP registries, everywhere that you would expect an agent to be able to easily find you.
[12:28] And also make sure that you know, that whole you reduce as much friction as possible for an agent or and developer to go from finding out about your tool to embedding it in their workflow.
[12:40] Because if an agent realizes your tool requires like three different demos and emailing sales reps and stuff, they're never going to say, "Hey user, like here's what you should do, but FYI, you're going to have to do all this other stuff."
[12:52] It's like not going to happen.
[12:54] And then also make sure that you are covering that those pains, right?
[12:58] Because that's how a user is going to be most like in their time of need, right?
[13:03] That's going to be the best opportunity for your product and your service, right, to be surfaced to them.
[13:08] And so, you want to make sure that there's enough content out there on the internet for the agent to like be aware of that and make those connections for you.
[13:17] And so, right, there's this like ongoing question of what even the heck is DevRel and advocacy and now now this agent advocacy thing, right?
[13:28] So like where does it fit?
[13:29] Where does it go?
[13:30] Like is it engineering?
[13:31] Is it product?
[13:32] Is it marketing?
[13:32] It's like yeah, yes, yes.
[13:34] It's all of those things.
[13:36] And and with the rise of agents it hasn't gotten any clearer, right?
[13:41] Those seams haven't gotten any clearer.
[13:43] If anything though, everybody's role with across the organization has gotten fuzzier.
[13:47] So that actually helps in a lot of ways.
[13:50] Um and but you can sort of split it up and think about it in terms of like these different flavors, right?
[13:56] And you can mix and match depending on whatever skills and abilities various employees have within your organization and whatever the product needs at a given time.
[14:04] So you have like the engineering flavor, right?
[14:06] And those are folks that are partnering directly with the engineering team to make these interfaces for how the agent is talking to your product like through the MCP server and building out these evals and the instrumentation.
[14:18] Then you have the product flavor.
[14:19] So those are folks that are going to own the end-to-end agentic experience, right?
[14:22] And so translating these evals to bring it to the product team and like having the agent experience rubrics how they're encountering all of that content.
[14:31] And then you have the marketing flavor, right?
[14:32] And that should be the folks that are really owning that pipe gen and how the agents are like entering the funnel and finding out about your product and then bringing the developers along with them by surfacing those recommendations.
[14:48] So I know I you know said the death of developer advocates.
[14:53] But the core right of DevRel still holds.
[14:56] It's just you have a change in your audience.
[15:00] So it's still extremely important to do enablement, right?
[15:04] It's just the type of enablement is a bit different.
[15:07] You're educating developers now who are have a completely different type of job where they're orchestrating these fleets of agents.
[15:14] And you're also educating agents, right?
[15:17] So you're having to put out content that is machine readable, has like agent friendly APIs, all of these things to make it as easy as possible to use your product both for human developers
[15:26] and for the agents that they're using.
[15:28] And community is also more important than ever, right?
[15:31] Um having that human-to-human connection um where developers can come um and uh bring their agents also into the loop, right?
[15:41] So that's another component um that needs to be considered uh when you're building these different communities because there's all these questions, right, of privacy and like data concern as well.
[15:51] If people are like bringing their Claude's and whatnot like into the Discord and they're like uh recording all of the conversations and everything like this.
[15:57] It's just like a new thing they have to think of as a community builder.
[16:00] And then there's the feedback loop, so you're still uh responsible for bringing the voice of the developer who's using the agents back to the organization, but then you can also
[16:09] uh basically spin up like thousands of these agents to perform experiments on them and experiments that you can't really like do as easily with the developers who don't want to maybe talk to you that much.
[16:19] Um and then credibility, right?
[16:20] So you need to be earning credibility both from human developers.
[16:26] Um so like don't like not using Claude's slop at them, right?
[16:30] Then tell your AEs to stop that as well.
[16:33] Nobody Everybody knows what it is and nobody likes it.
[16:36] Um and but then credibility like actually Claude loves its own slop uh for whatever reason.
[16:42] So there's a bias, right, from agents of their own content.
[16:45] So whenever you're making like agent-facing content, as long as it's structured, you can have as many m dashes and whatever as as it wants.
[16:52] Um but it's just a completely different sort of uh credibility landscape, humans versus agents.
[16:57] So what I'm advocating for here, right, is like building out a curb cut.
[17:02] So curb cuts were built for wheelchairs, like built for a specific user to use them.
[17:07] Um but now everybody, you know, benefits from that, right?
[17:10] Anybody with wheels, right, strollers and um suitcases and all of those things.
[17:14] So my argument is that by serving the uh agents, uh the human path gets cleared, too.
[17:20] There's just, you know, there's just one more user in the room now, but they are still serving the human on the other end, and we're all working together on this.
[17:28] So, for, you know, DevRel, one quick thing that you could do like right away is point a coding agent at your docs, and then looking through that transcript and start developing your agent experience report.
[17:37] And then if you're more on the GTM side, start like developing some of these experiments with the GEO, putting together those prompts, and looking at the mentions versus recommendations.
[17:48] And I made this whole talk agent legible, right?
[17:52] So, there's a QR code there, as well as a couple different toy repos that have some templates for you to get started.
[17:58] And that's it.
[18:13] [music]