How We Got LLMs to Recommend Our Open Source Library — Christopher Burns, Inth
https://www.youtube.com/watch?v=V_5bn4q-vAI
AI Engineer · 16 min · 166 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] The talk title, we'll see if it lines up by the end of it, but when we put this talk title in just to be honest with you, so much changes in like
[00:22] 3 days at this point.
[00:24] We'll see how it goes.
[00:25] So, yeah, the whole point of it was that how I got LLMs to understand my open source library and what I did to do it well.
[00:37] Is it some kind of scientific background?
[00:39] Am I from a lab?
[00:40] No. That's My slidey clicky thing's not working.
[00:46] So, I just like to say again, I'm just like you.
[00:49] I'm just this side of the stage.
[00:52] I've just hacking hacking it together, figuring out what is useful, what is token efficient, these kind of things.
[00:59] And again, I am British.
[01:02] Please don't think my accent makes me an expert.
[01:07] So, for quick contacts, I'm Christopher Burns.
[01:10] I'm the founder of Inth.
[01:11] I created a open source cookie banner library called C15T.
[01:16] That really annoying thing on the internet.
[01:19] That is me.
[01:20] Uh I spoke at Next Conf uh after it started taking off and it had 1.2 thousand downloads at the time.
[01:27] Now it's closer to 2 million.
[01:29] In terms of like statistics, so we just checked that, you know, this is not theoretical.
[01:34] This is actual uh something that is succeeding.
[01:39] We have 3 million NPM downloads.
[01:41] 4.5 uh 4.5 45% month-on-month growth.
[01:46] 2.8 thousand websites using it in production from Minify to Z to Inphysical.
[01:54] And the whole concept of this talk was uh it goes back to we were doing all these things to make our library more efficient.
[02:04] You know, we were batting upwards compared to every other tool.
[02:08] Every other tool was built for marketers and lawyers.
[02:11] We were built for the developer.
[02:13] So, we had to make sure we had a very good developer experience.
[02:17] And we had an onboarding format said, "How did you hear about this?"
[02:22] And we started to get spikes that from April 13th, you know, now it is our number one source of inbound is Claude, ChatGPT, Codex, that is ChatGPT, Gemini recommending us.
[02:39] And I like to think of this as, you know, the iceberg.
[02:43] You know, we start with the top of C15T and there's many many tools that go into it from, you know, LLMs.txt to site maps to RSS feeds to robot.txts.
[02:56] So many micro optimizations that you can do from old methods of running the internet to new methods.
[03:03] And uh how many of you have, you know, made these kind of tools?
[03:10] How many of you have, really put simply, said, "Hey, agents, we need to this to be done."
[03:20] And yeah, it said, "We should install this library."
[03:24] And you've gone, "Okay."
[03:26] Raise your hands.
[03:26] How many people have done this?
[03:29] Pretty much most people.
[03:32] That's a lot of hands.
[03:34] So, what's really funny is that we went from wizards installing our software to agents installing them.
[03:41] And I just went through Y Combinator.
[03:45] And what's really interesting is it if you know who these two people are.
[03:50] These are the co-founders of Stripe, the Collison brothers, and they had a really classic saying of like a Collison brothers install.
[03:59] And they would hand you their laptop, and they would install Stripe.
[04:03] These days, it's kind of like just a prompt.
[04:06] Being in Y Combinator, we just give people a prompt.
[04:10] And really what that means is that our very good developer experience primitives are now hitting agent primitives.
[04:20] So, as we was pulling all these things together, there is no one tool that fixes everything.
[04:28] I like to think about these problems like, you know, Batman's utility belt.
[04:32] Loads of really small things targeted in different areas to get it done.
[04:38] And we built all of these things into C15T because we wanted C15T to be the best developer framework in this tool.
[04:47] Think of it like Stripe Docs.
[04:49] And as we was building more and more tools, more and more documentation websites, we actually started abstracting these tools into a side quest that we call Lead Type.
[05:01] So, all of the things that we're going to talk about now are things that we have already solved with this open source framework.
[05:10] We have our friends at other developer companies implementing it and seeing similar results about how to like optimize for the agent experience.
[05:19] So, again, this isn't a magic SEO tool.
[05:23] It's actually a very non-sexy title, but a framework-neutral docs pipeline.
[05:30] Complex.
[05:32] But really, all it basically does is take your .mdx files, you run Lead Type generate, and it will spit out everything for um optimized agent experience for your websites.
[05:45] And the rest of this talk is going to look a bit like a BuzzFeed list, to put simply, of these problems, because again, not everybody knows even how to put an LLM.txt
[05:57] on their website.
[05:58] So, you know, that comes to the first problem of if your docs have hundreds of pages, and how can it navigate them to find the right questions?
[06:10] The first solution is obviously an LLM.txt.
[06:14] What we found in our research is that it's much better not to just generate this.
[06:20] It is much better to write your LLM.txt from hand.
[06:24] Obviously, AutoRaptor, but write it as you are trying to get the answers across to the LLMs.
[06:33] For about 40 good lines beats 1,000 lines of noise from our testing.
[06:39] And that comes to the second issue of agents don't know how to browse.
[06:44] They know how to fetch.
[06:46] So, you then need the second part of the solution of the LLMs full.
[06:51] Again, think of this as a sitemap, where it takes the actual page and the links and a short description of what each page is for the LLMs to reference.
[07:03] Again, most people have heard these two solutions.
[07:07] But, where things are starting to get very complicated and we're seeing a lot of optimizations right now, is that HTML is expensive, and why can't we just ship markdown
[07:19] to the agents?
[07:20] And we can.
[07:22] And you've seen that everybody has started creating twin MDs.
[07:28] So, that's taking the normal website, such as Next.js quick start, and then having a dot MD on the end of it.
[07:37] And when you load that, it goes to the markdown But, what's really important here, and it's really worth noting, is this line at the bottom.
[07:48] If you look at all the best documentation websites, Minify, Vercel, C15T, pat myself on the back.
[07:56] Um they all have this in the header.
[07:59] This is saying to the agents whenever they visit the website that there is an alternative version of this in mark markdown.
[08:08] Again, who actually supports it?
[08:11] Don't ask me.
[08:12] Perplexity, some of the agents, it's all up in the air.
[08:17] And then, the second thing as well is that taking the .mds, you need to make sure that they're available through multiple methods.
[08:28] So, one of them is like the .md, so as you like copy it to an agent, you say .md.
[08:34] Another one is just taking the normal um link and then adding a uh redirect into your like your Next.js config, so that if it detects an agent has the header of accepting markdown,
[08:50] instead of returning the HTML, it will return the markdown.
[08:55] And then, the third one is that not all agents can append header tags.
[09:00] So, there's also a URL query of mode equals agent.
[09:06] So, they're the ones that pretty much everybody knows.
[09:10] Um and it's pretty basic internet knowledge at this point.
[09:14] Um but one of the really interesting ones is where we're going next.
[09:20] And our tooling is also helping this is that an agent can't ask your website anything.
[09:28] So, we need to think about the web MCP.
[09:31] And this is still very early, but our tool is already uh um exposing three different tools to WebMCP.
[09:41] Search docs, get pages, and ask docs.
[09:45] Again, um our library lead type is pulling all of that context together so an agent can easily ask it the right questions.
[09:54] I think we'll even see a future where communication happens over email and there's companies in San Francisco building that today.
[10:04] But this is actually the most interesting one and I think the most important one that anybody who has any type of developer module surface, NPM modules, cargo, Python, whatever.
[10:19] Is that the uncomfortable truth is that coding agents are actually never visiting the website if you have a library.
[10:27] They're actually visiting the node modules.
[10:31] They read the repo and they read the node modules.
[10:35] They They have previous stale training data and they're trying to work it out on what it can do from the the compiled source.
[10:46] So, again, following what people like Vercel are doing and people who are thought leaders in this industry is that we take the bundled markdown documents and then we also
[11:00] put them in the node modules with an agents.md file.
[11:07] And the agents.md file basically says, "If you've got a problem, if you've got a question, all the documents are here.
[11:14] Grab them."
[11:16] And we actually see that this has surprisingly real effects.
[11:22] We can see that between many different models almost 50% token saving on instead of trying to search the web, find the right tools, pulling the markdown files from your code base.
[11:35] So, if you have a library that's forever changing, then having the node modules built in is a very effective solution.
[11:44] This is also working without any skills, but if you want as well, you can add skills to it to say, "Look at the node modules and go from that."
[11:54] And again, just uh doubling down into this point, looking at like the agents.ai and DFile, you can say like when working with Z15 T Next.js library, read the bundles
[12:05] and verify that they match and go from there.
[12:11] So, that's really like how we've done it.
[12:16] I don't want to say this is like prescriptive, that I know the answers.
[12:21] If you have documentation websites, or if you have any type of markdown, if you're running your own blog, you know, I've been using our package as well on our marketing website.
[12:31] Every part of our marketing website also has a markdown file.
[12:35] It can be something that's used for many things.
[12:37] We're currently just um most people are just using it for documentation.
[12:43] But, you can literally run it and it will pull out all of these extra files.
[12:48] And one of the big things was when I put this talk together, you know, we were seeing the results that Claude was recommending, but there was not really any like test suites yet, or test harnesses,
[13:01] on like is your site agent ready?
[13:04] And Cloudflare brought one of them out.
[13:07] But, my favorite is actually one called Aura AI.
[13:12] Um this is brand new and it tests a lot.
[13:17] I'm happy to show off score of 59 because it's constantly changing.
[13:22] 3 weeks ago, it was a lot higher.
[13:24] And again, this is a forever changing area.
[13:27] So, aura.ai, put in your website, and it will start giving you recommendations.
[13:33] It's forever changing.
[13:34] Again, we can just stay on top of it.
[13:37] And yeah, this is like one of my final slides is that the slide the the market agents, LLMs, everything is forever changing.
[13:47] There is no such thing as perfection.
[13:50] When I started making these slides, I got so caught up of like, everyone expects me to be the expert here, but I've just been hacking on this problem a little more than you guys have so far.
[14:00] So, never get caught with being perfect.
[14:02] Every small little increase really does matter.
[14:06] Every small little thing you add really does matter.
[14:11] Thank you so much.
[14:12] You can find me on X, Burn Chris, and LinkedIn, and everywhere.
[14:16] [applause]
[14:17] WOO!
[14:19] I THINK I THINK WE HAVE TIME for one or two questions.
[14:24] Yeah, of course.
[14:32] So, if you were building um uh we're a website agency.
[14:36] We work with a lot of startups building like their own websites.
[14:39] Mhm.
[14:40] If you were just building a website, not necessarily like developer tool, but just a website to be found, which of these methods like would you concentrate on if you're starting from scratch?
[14:49] Yeah, I think the most important ones, and we're starting to see this more and more, is trying to provide a dot md file for every single page.
[15:00] A lot of CMSs are not built in this way.
[15:03] Um and we see this optimization happening more and more where I didn't put in the slide, but we're seeing more and more websites being visited by agents instead of real humans.
[15:16] So, in terms of even like trying to be proactive and token efficient, you should provide a markdown file if you can.
[15:25] Again, a lot of CMS's are not built this way.
[15:28] I actually built my own CMS.
[15:31] My name is Chris and I built ChrisCMS, short for Christmas.
[15:35] It's a whole It's a whole thing my team wishes I never built.
[15:38] But, it does work and it does bring this like token efficiency up.
[15:43] So, yeah, I would say llms.txt is your first shout.
[15:49] llms.txt full form .txt.
[15:52] Second, if you if you can, just do them manually.
[15:56] Say you're not even working on systems that have markdown, I still recommend them.
[16:03] But, you can always get creative with creating these files on the on on the go.