The Death of Developer Advocates — Stephanie Jarmak, Sourcegraph

AI Engineer · 18 min · 205 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13Sorry for the start with technical difficulties and all of that.
00:17Uh, we made it to the end of this track.
00:19Super exciting.
00:21Thank you everybody for sticking it out this long.
00:24Um, are there any developer advocates or devrel people in the audience?
00:29Raise your hand.
00:30Yeah, okay.
00:31So did you come to like throw tomatoes at me cuz I'm talking about the dead now.
00:34Okay, so it's not going to be all doom and gloom like that.
00:38Um, a bit of like backstory in this.
00:41Um, I'm a research scientist.
00:43So last year I was an astronomer.
00:46Um, and I just sort of like wound up.
00:47I didn't know what GTM was or any of that.
00:49I just sort of wound up in this.
00:51Um, and I submitted like a bunch of boring sciency eval talks that were unceremoniously I I assumed thrown into the trash uh, for this conference.
01:00But my manager, who is a developer advocate, he put in, you know, the death the death of developer advocates, which is, you know, appropriately buzzworthy and hypey.
01:09And so so that was great.
01:11But his title is developer advocate, so it didn't really necessarily make as much sense for him to be coming up here and giving his eulogy.
01:18So we brainstormed like maybe I would dress up as like a robot and like a maul him and attack him on the stage or something like that.
01:25Um, but then it just like logistically it was going to be hard to do that.
01:29Uh, so he just went on vacation.
01:31Uh, so I'm here uh, as the agent advocate uh, to talk about this sort of like new role and uh, try to advocate for it and uh, convince all of you that we should all be agent advocates to help
01:46uh, in this new era.
01:48So uh, zooming out a little bit and going back uh, in time a bit because uh, I was trying to talk about developer advocates to somebody at the conference yesterday and their eyes like glazed over they had no idea what I was talking about.
01:59So just to sort of talk about what what this thing is that I'm saying is dead.
02:03Uh so back in the '80s, right?
02:05It was called like software evangelism where one would go forth and speak the good word of the product and bring it out there.
02:13But then fast forward to the 2010s or so, that's when developer advocacy advocacy started to become a thing where now instead of having this single trajectory of the communication pathway, now it's a feedback loop and a two-way street where you have these people with very deep empathy for developers
02:29who understand them and speak their language and could understand um what their needs were um and then bring that back to the product.
02:37And then um these developers, right?
02:39Fast forward even more, they have so much influence within their company and basically become these like kingsmakers.
02:46Uh and so the developer experience became a very important aspect of the go-to-market sort of strategy.
02:52Um but now in 2026, uh developers are no longer working alone and what it means to be a developer is completely changing.
03:00Um and so our role, right, as developer advocates um developer in developer relations, we're relating to developers.
03:07And so as the role of developers fundamentally changing, so must then does the role of the developer advocate.
03:15Um so in this slide I'm just kind of talking about the other users, right?
03:20So what's happening uh with DevRel uh outside of the agent.
03:24So most of the talk is going to be talking about the agent as a user.
03:27But I also did did want to bring up, right, that engineers they're becoming like these orchestrators of these fleets of agents, um babysitters and whatnot of these things.
03:38Um and their job, like all of the job postings and whatnot, there's language is continuously changing, right?
03:42They're um expected to have this AI fluency.
03:46Um and at the same time, there's also, you know, people like me, like uh non-engineers, right?
03:53I was a research scientist.
03:54I had like zero commits on GitHub last year, and now I have 12,000, and I'm like an open source maintainer for multi-agent orchestration framework.
04:01Like, we have so much like capability now with all of these agents, and now anybody with these agents can use dev tools, essentially.
04:09So, you have this whole other persona and ICP uh to potentially be relating to and um having empathy with when you're there using your product.
04:19So, let's talk about now this whole new user that we have in the form of an agent.
04:24Um so, an agent is somewhat unique, right?
04:28In the sense that it is both the user of your tool in a very similar way to the developer.
04:34It's going out reading your docs, but it's just reading them differently cuz it's a machine.
04:37Um you know, it's calling the API.
04:39It's encount- it's ha- has its own frustrations with how it's encountering errors and recovering from them, right?
04:44But then it's also a recommender of your tools.
04:47Um but somewhat similar, right?
04:48To developers in the way that they are also recommenders of your tools in a more organic, bottom-up way.
04:53Um so, the whole, you know, basis for DevRel, right?
04:56Is to encourage that bottom-up adoption.
04:58But now the adoption and the recommendation system, a lot of it's being driven by the agent itself.
05:04That is either, you know, maybe servicing your product directly through like ChatGPT or Claude, like directly in a Q&A sort of environment, or it's, as we had heard like in some of the previous talks where the speaker asked folks like, "How many of you have just let your agent
05:18install a library for you?"
05:19And like, there were many hands went up, right?
05:21So, there's this like recommender of tools where basically it's just installing these like frameworks and things um directly and embedding them into the workflow um and sort of working with the developer
05:32um in that taste.
05:36So, I know it's late for numbers.
05:37You don't have to read them or anything like that.
05:40Um so, I have a couple different concrete examples for measuring these seats, right?
05:44Cuz I am a data science scientist nerd person.
05:48Um so one of my first projects when I was uh working on this um uh when I became an agent advocate was to build um a benchmark called CodeScaleBench.
05:58And so I developed hundreds of tasks that were reflective of the software development life cycle.
06:02And I basically unleashed these agents with and without um our product tooling.
06:07So I work at Sourcegraph and we have a code navigation MCP tool.
06:10Um and the point of that was to understand, okay, how is our tool helping the agent do the work that it's, you know, going to be doing.
06:18Um and when it isn't working well, why isn't it working well?
06:21So that we can then go in and actually fix that.
06:24Um so I have thousands and thousands of these traces.
06:26And I I as we have heard in like the previous talks, like now we have these amazing logs of data for like these really tight feedback loops where you can see exactly where it's breaking down and then go in and fix it.
06:37Uh so this one specific example here was um when I was looking at how it was like using a read tool.
06:43Um and the model had the these expectations based off of its like biases from how it from its training data of what it expected for a particular um command
06:52um that would be available within the tool.
06:55And there's nothing in our description uh that would have like led it to believe otherwise.
06:59So it tried to use like read line instead of start line or something like that.
07:03And then it ended up failing, but then at least the error told it why it failed.
07:07So it was like, okay, that that was a good part of it.
07:09So it was able to fix itself.
07:11But then it's burning right an entire turn just failing.
07:15And you could just go in and fix that um aspect of like how it's interacting with the tool.
07:19And this is really important, right, to gather that feedback um and understand the friction that like now your new agent user is having with your tool because it's the way that um different organizations are going to be evaluating your tool, right?
07:30In terms of not just is it working well, but like how many tokens is the agent dealing with to work with your tool?
07:36And how fast is it?
07:37Um so this is, you know, really an important aspect of the role is measure um, how these users are using it.
07:44The other side of it um, is like the recommendation layer, right?
07:48So, the uh, GEO instead of SEO.
07:50So, the generative engine optimization.
07:54Um, and I didn't mention it before, but in the previous slide um, I had a GitHub repo.
07:59Like, there's two different toy projects that I put together.
08:01At the end of the talk, there's like a QR code with a link that you can send your agent to to like have access to all this.
08:07So, don't worry about like taking screenshots All of all of the data will be released to you.
08:13Um, so anyway, back to this.
08:15Um, I set up a little experiment, right?
08:18To see how uh, these different chatbots and agents and whatnot were recommending our product or like mentioning it at all.
08:26Um, and so there's a, you know, process to that cuz you have you want to understand like, what is your ICP actually doing when you would want your product to be surfaced?
08:36So, there was a bit of a gap that I found.
08:39Um, if I had designed some of these prompts around somebody who like was actively shopping for this sort of code intelligence sort of tooling and doing a comparative sort of thing, then our product was ending up being recommended like 65%
08:53of the time.
08:54Um, but what I found was the arguably like the more typical use case and where we'd want to be showing up for people when they're encountering a specific pain or have a specific need where our product could serve them better,
09:06uh, zero mentions, right?
09:09So, in this particular instance, um, I put in a prompt that was like, we keep breaking downstream services when we change shared libraries because we can't see all the consumers.
09:18And you know, our one uh, part of our product is being able to have this observability layer to like see across all the repos.
09:25So, we'd want uh, some level of like attribution or recognition from um, an agent to say, "Hey, you could use something like this."
09:32But instead it said, uh, "You could just have your developers make a wiki page or something.
09:38Um but with this, you know, we wouldn't know that without running these sorts of experiments um and getting this sort of data.
09:45So, what this leads to is like then you can have a hypothesis of okay, maybe the messaging that we're putting out there isn't uh attributing some of these pains and use cases clearly enough
09:55for the agents to be picking it up.
09:56So, we have uh like a content campaign in the works to um make changes to our website and then we can directly measure whether that has like an actual lift and not necessarily in the form of like anything that was baked into the training data, but then how uh the agents that are using those like web search
10:15tool calls, how they are then interpreting um the information about your product.
10:21So, you know, there are just some um different ways that you could think about guiding the agents um to help support like the servicing, the discoverability of your product and this user finding it um at their moment of need, right?
10:35Um so, for example, um this whole field is moving so fast.
10:41Uh so, I mean, training data is always going to be stale.
10:44Actually, in the um GEO pilot study that I did, the data that I was showing there, that was using Claude Sonnet 4. It's very old um obviously and I just today, this afternoon, ran it with 4.6
10:56thinking that okay, surely it's going to it's going to be better.
10:59It's going to know like improved information about our product, but uh so, in the previous model, it kept pitching Cody, which was like one of our older products.
11:08Um but if I when I uh ran it again, it it pitched Cody even more, right?
11:14Cuz like now you have all of these like old models like uh outputting content that then is like compounding in the internet.
11:21So, you have to figure out like how to bury all of that uh noise with your true signal.
11:27Um and the way that some folks are working on that is as we've heard from other people like these LLMs at TXT uh sort of pages, right?
11:35So, you have more authoritative sources of truth that you're hoping to direct the agent to.
11:40But, they still need to be using the tools and using real-time information and provenance to be able to give accurate answers about your product.
11:48You also want to give like the agent something to quote, right?
11:51They they they want to bring something that they can really sell to the to the user, right?
11:56So, you want current examples and keep everything up-to-date.
11:59Like, even if your stuff hasn't changed in 2 years, which would be shocking.
12:03Even if it hasn't, like keep everything up-to-date and fresh because that, you know, part of that is how they have their relevance algorithm.
12:10And they also really really like charts and FAQs and things like that.
12:14And you also want to make sure your product is where the agents are, right?
12:18You're going to market.
12:19So, go go to agent market, right?
12:22So, make sure you're in the marketplace in the MCP registries, everywhere that you would expect an agent to be able to easily find you.
12:28And also make sure that you know, that whole you reduce as much friction as possible for an agent or and developer to go from finding out about your tool to embedding it in their workflow.
12:40Because if an agent realizes your tool requires like three different demos and emailing sales reps and stuff, they're never going to say, "Hey user, like here's what you should do, but FYI, you're going to have to do all this other stuff."
12:52It's like not going to happen.
12:54And then also make sure that you are covering that those pains, right?
12:58Because that's how a user is going to be most like in their time of need, right?
13:03That's going to be the best opportunity for your product and your service, right, to be surfaced to them.
13:08And so, you want to make sure that there's enough content out there on the internet for the agent to like be aware of that and make those connections for you.
13:17And so, right, there's this like ongoing question of what even the heck is DevRel and advocacy and now now this agent advocacy thing, right?
13:28So like where does it fit?
13:29Where does it go?
13:30Like is it engineering?
13:31Is it product?
13:32Is it marketing?
13:32It's like yeah, yes, yes.
13:34It's all of those things.
13:36And and with the rise of agents it hasn't gotten any clearer, right?
13:41Those seams haven't gotten any clearer.
13:43If anything though, everybody's role with across the organization has gotten fuzzier.
13:47So that actually helps in a lot of ways.
13:50Um and but you can sort of split it up and think about it in terms of like these different flavors, right?
13:56And you can mix and match depending on whatever skills and abilities various employees have within your organization and whatever the product needs at a given time.
14:04So you have like the engineering flavor, right?
14:06And those are folks that are partnering directly with the engineering team to make these interfaces for how the agent is talking to your product like through the MCP server and building out these evals and the instrumentation.
14:18Then you have the product flavor.
14:19So those are folks that are going to own the end-to-end agentic experience, right?
14:22And so translating these evals to bring it to the product team and like having the agent experience rubrics how they're encountering all of that content.
14:31And then you have the marketing flavor, right?
14:32And that should be the folks that are really owning that pipe gen and how the agents are like entering the funnel and finding out about your product and then bringing the developers along with them by surfacing those recommendations.
14:48So I know I you know said the death of developer advocates.
14:53But the core right of DevRel still holds.
14:56It's just you have a change in your audience.
15:00So it's still extremely important to do enablement, right?
15:04It's just the type of enablement is a bit different.
15:07You're educating developers now who are have a completely different type of job where they're orchestrating these fleets of agents.
15:14And you're also educating agents, right?
15:17So you're having to put out content that is machine readable, has like agent friendly APIs, all of these things to make it as easy as possible to use your product both for human developers
15:26and for the agents that they're using.
15:28And community is also more important than ever, right?
15:31Um having that human-to-human connection um where developers can come um and uh bring their agents also into the loop, right?
15:41So that's another component um that needs to be considered uh when you're building these different communities because there's all these questions, right, of privacy and like data concern as well.
15:51If people are like bringing their Claude's and whatnot like into the Discord and they're like uh recording all of the conversations and everything like this.
15:57It's just like a new thing they have to think of as a community builder.
16:00And then there's the feedback loop, so you're still uh responsible for bringing the voice of the developer who's using the agents back to the organization, but then you can also
16:09uh basically spin up like thousands of these agents to perform experiments on them and experiments that you can't really like do as easily with the developers who don't want to maybe talk to you that much.
16:19Um and then credibility, right?
16:20So you need to be earning credibility both from human developers.
16:26Um so like don't like not using Claude's slop at them, right?
16:30Then tell your AEs to stop that as well.
16:33Nobody Everybody knows what it is and nobody likes it.
16:36Um and but then credibility like actually Claude loves its own slop uh for whatever reason.
16:42So there's a bias, right, from agents of their own content.
16:45So whenever you're making like agent-facing content, as long as it's structured, you can have as many m dashes and whatever as as it wants.
16:52Um but it's just a completely different sort of uh credibility landscape, humans versus agents.
16:57So what I'm advocating for here, right, is like building out a curb cut.
17:02So curb cuts were built for wheelchairs, like built for a specific user to use them.
17:07Um but now everybody, you know, benefits from that, right?
17:10Anybody with wheels, right, strollers and um suitcases and all of those things.
17:14So my argument is that by serving the uh agents, uh the human path gets cleared, too.
17:20There's just, you know, there's just one more user in the room now, but they are still serving the human on the other end, and we're all working together on this.
17:28So, for, you know, DevRel, one quick thing that you could do like right away is point a coding agent at your docs, and then looking through that transcript and start developing your agent experience report.
17:37And then if you're more on the GTM side, start like developing some of these experiments with the GEO, putting together those prompts, and looking at the mentions versus recommendations.
17:48And I made this whole talk agent legible, right?
17:52So, there's a QR code there, as well as a couple different toy repos that have some templates for you to get started.
17:58And that's it.
18:13[music]