WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=9wZpvF3QleU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/9wZpvF3QleU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.400 --> 00:00:14.480 Hey everyone, good morning. s3 00:00:14.480 --> 00:00:17.920 Uh thanks for being here. s4 00:00:17.920 --> 00:00:20.000 Uh I see people are still coming. s5 00:00:20.000 --> 00:00:28.000 Uh but yeah, my name is uh AJ and uh I am a software engineer at uh LinkedIn. s6 00:00:28.000 --> 00:00:37.760 Today I'm going to be talking about how we are doing context engineering to improve the performance of coding agents at LinkedIn. s7 00:00:40.079 --> 00:00:40.800 Okay. s8 00:00:40.800 --> 00:00:57.520 Uh imagine you are a software engineer in a big tech company and your products are being used by millions of users on a daily basis and you are happen to be on a team which owns set of very critical services s9 00:00:57.520 --> 00:01:12.560 and you are on call right and you get an alert saying that there is an error spike in one of your services and uh by the time you're trying to figure out how to deal with this issue, s10 00:01:12.560 --> 00:01:19.920 you take the link to the alert, give it to a coding agent like Cloud Code or GitHub Copilot. s11 00:01:20.640 --> 00:01:31.920 While you're trying to figure out um how to deal with the issue, uh the coding agent is working in the background, it will fetch the instructions on how to debug such issues in your company s12 00:01:31.920 --> 00:01:38.960 and identifies that based on that instruction, it identifies that this alert is happening in uh specific service. s13 00:01:38.960 --> 00:01:51.040 Then it fetches instruction and context on how to debug that particular service and it will fetch the based on those instructions it will take actions like fetching logs metrics s14 00:01:51.040 --> 00:02:04.880 and then it uses those logs to identify the root cause of the issue right so it identifies based on the error logs where the issue is happening and it doesn't just find the root cause it also s15 00:02:04.880 --> 00:02:09.119 figures out the steps to mitigate the issue and once it finds all the det details. s16 00:02:09.119 --> 00:02:17.200 It summarizes and give it to you saying this is this is the error and this is the issue and this is the actions that you need to take to mitigate. s17 00:02:17.200 --> 00:02:25.200 And once you confirm, it also goes ahead and takes those actions on your behalf to mitigate the issue. s18 00:02:25.440 --> 00:02:27.680 And it doesn't just stop there. s19 00:02:27.680 --> 00:02:40.400 Uh it updates your incident management system with all the details, error metrics and dashboards etc. and also it checks out the code and creates a PR for you uh to fix the root cause of the issue. s20 00:02:40.400 --> 00:02:48.160 All of this happens in matter of few minutes which would have easily taken few hours if you were to do it manually. s21 00:02:48.160 --> 00:02:49.440 This is not fiction. s22 00:02:49.440 --> 00:03:03.680 So this is how teams at LinkedIn are using coding agents as effective co-workers with deep understanding of LinkedIn's internal systems and code to help the teams be really productive. s23 00:03:05.360 --> 00:03:13.440 And this is possible because of a system that we built called as contextual agent playbooks and tools at LinkedIn. s24 00:03:13.440 --> 00:03:22.239 And today I'm going to talk about why we built the system, how we built it and what are our learnings from the success. s25 00:03:23.840 --> 00:03:30.640 To understand why we built the this system, we have to go back to the early days of coding agents, right? s26 00:03:30.640 --> 00:03:42.560 So just like any other company even at LinkedIn we wanted to use the coding agents to be for our engineers and everyone to be really productive with the AI. s27 00:03:42.720 --> 00:03:55.599 So we started using started giving this coding agents to all of the engineers and the problem was the coding agents doesn't really or the w coding doesn't really work in a large enterprise s28 00:03:55.599 --> 00:03:59.439 at link um like LinkedIn. s29 00:03:59.760 --> 00:04:06.720 So the biggest problem is the coding agent or the LLMs are trained on open-source repos right. s30 00:04:06.720 --> 00:04:18.239 So they don't have the context of how we our uh mature code bases at LinkedIn or our internal frameworks or internal systems. s31 00:04:18.239 --> 00:04:29.759 So what used to happen was the engineers used to do wipe code or try the agentic coding but because it the agents lacked context they used to hallucinate and s32 00:04:29.759 --> 00:04:37.919 like get stuck in between or uh even more dangerous they used to make up things which is not correct. s33 00:04:37.919 --> 00:04:48.800 So the engineers had to prompt these agents manually um to do the right thing which used to take more time than the manual coding itself. s34 00:04:48.800 --> 00:04:52.160 So a lot of people a lot of engineers went back to manual coding. s35 00:04:52.160 --> 00:05:00.960 So coding agents was not effective to understand the problem uh to get more perspective. s36 00:05:00.960 --> 00:05:18.160 So if you look at the LinkedIn stack, we have over thousand repos which make up thousands of uh microservices and apps and we have a lot of all of these apps and services are built on lot of internal frameworks and libraries s37 00:05:18.160 --> 00:05:22.000 and we also have a lot of custombuilt infra. s38 00:05:22.000 --> 00:05:24.240 For example, we have our own databases. s39 00:05:24.240 --> 00:05:27.919 We have our own experimentation and tracking platform. s40 00:05:27.919 --> 00:05:43.120 We have our own configuration management system which is purely internal to LinkedIn and coding agents doesn't have any idea about them and engineers go through a week-long boot camp whenever a new engineer joins. s41 00:05:43.120 --> 00:05:47.039 So uh just to get familiar with these systems. s42 00:05:49.600 --> 00:06:12.560 So we looked at this problem and we asked ourselves the question how can we make any coding agent like cursor or cloud code or uh GitHub copilot understand our LinkedIn's internal system so well that they can ship the code that our engineers can trust. s43 00:06:12.560 --> 00:06:22.400 By trust I mean the code should be correct and also the quality of the code should be as good as uh it is written by an actual engineer. s44 00:06:22.400 --> 00:06:29.360 So that is the bar we set out and wanted to figure out how do we get there. s45 00:06:30.319 --> 00:06:43.840 So in early 2025 uh last year so uh Anthropic released MCB and it it quickly became the standard for industry standard for building tools to the agents. s46 00:06:43.840 --> 00:06:54.880 We leveraged that and pretty early on we built our own internal MCP and the first tool that we built was uh code search. s47 00:06:55.120 --> 00:07:02.400 So we have a pretty sophisticated code search system at LinkedIn where you um engineers can go and search for code. s48 00:07:02.400 --> 00:07:15.759 It will ingest all of uh search for any code across thousands of repos using keywords and custom filters and rejects etc. So we made that available to the coding agents via MCP. s49 00:07:15.759 --> 00:07:23.039 This was a really powerful unlock because now you don't have to manually figure out how to do better search the agent. s50 00:07:23.039 --> 00:07:24.240 You ask a question. s51 00:07:24.240 --> 00:07:27.759 Hey, how do I set up a particular thing? s52 00:07:27.759 --> 00:07:38.479 And the agent can use the code search tools to figure out the right examples of how we do things at LinkedIn and use that to give you answer and also s53 00:07:38.479 --> 00:07:41.919 uh implement it based on the its findings. s54 00:07:41.919 --> 00:07:44.000 This was really powerful. s55 00:07:44.000 --> 00:07:46.479 So we added more tools. s56 00:07:46.479 --> 00:07:56.879 We added docs, uh, Jiraa, Slack, um, even connected to all of our data platforms and even feature flags. s57 00:07:56.879 --> 00:08:10.080 So every core tool that we added to our internal MCP it created more value by comp it it's almost like a compounding effect because now a engineer can bring in the s58 00:08:10.080 --> 00:08:22.639 PRDS product requirement documents and design docs and also their Jira tasks which has mult uh different context and use all this to give to the coding agent to automate their s59 00:08:22.639 --> 00:08:25.680 um or help with their coding. s60 00:08:27.360 --> 00:08:29.120 But there was a problem. s61 00:08:29.120 --> 00:08:33.919 So you connect all these tools but it's not enough right? s62 00:08:33.919 --> 00:08:42.479 So even with a slightly complex workflow the agents used to not do really well. s63 00:08:42.479 --> 00:08:58.640 The problem for example if you give a context it is uh with the tools the agent was able to answer questions answer basic questions and find code examples but it cannot do a like a complete job reliably end to end. s64 00:08:58.640 --> 00:09:07.519 The main problem was to do a specific job end to end it needs to have a lot of tribal knowledge right. s65 00:09:07.519 --> 00:09:19.279 So all of for example how to fix a particular error or how to uh configure how do you debug a particular um error log right so all of this knowledge s66 00:09:19.279 --> 00:09:39.279 even though you have access to the tools it is scattered across lot of different surfaces for example docs wikis and slack conversations um etc and most of the times you may have experienced the docs and wikis might be outdated written and there might be like duplicate get docs, right? s67 00:09:39.279 --> 00:09:45.680 So, the problem is the agents even though they have access to the tools, they used to get lost. s68 00:09:45.680 --> 00:09:49.600 The second problem was context overload. s69 00:09:49.600 --> 00:10:04.399 As agents use more and more tools, their context gets overloaded which means every tool uh tool output it takes up space in the context which will eventually cause the agent to compact its s70 00:10:04.399 --> 00:10:11.120 um while it is working compacts it its context which causes it to lose some of the information. s71 00:10:11.120 --> 00:10:14.320 Then it has to do all over again. s72 00:10:14.320 --> 00:10:28.480 And the third problem was even though even if the agent was able to figure out all these uh details it can it doesn't have a way to retain this information it doesn't have a a durable memory. s73 00:10:28.480 --> 00:10:36.959 So every time a engineer asks the uh agent to do a certain task they have to start from scratch. s74 00:10:38.079 --> 00:10:40.320 So how do we solve this problem? s75 00:10:40.320 --> 00:10:43.120 So we give this instructions right away right? s76 00:10:43.120 --> 00:11:01.279 So we built a system, we invented a system in early 2025 called as uh playbooks where we not only provide the tools to the agents via MCP, we also allow the agents to access these instructions and prompts via MCP. s77 00:11:01.279 --> 00:11:03.200 We call it playbooks. s78 00:11:03.200 --> 00:11:10.640 And playbook it just appears just like any other regular tool. s79 00:11:10.640 --> 00:11:21.360 uh and they have uh names and description on what it does and the agent can decide to invoke that playbook just like any other regular tool and when the s80 00:11:21.360 --> 00:11:31.200 uh playbook is invoked the instructions and the context within that playbook are returned as the tool output to the coding agent. s81 00:11:31.200 --> 00:11:41.519 So that way the agents have both tools as well as instructions on how to use tools to set up a or uh uh perform a task. s82 00:11:41.519 --> 00:11:41.920 Right? s83 00:11:41.920 --> 00:11:48.880 For example, uh if the uh engineer goes and ask like how do I set up a airflow DAG uh at LinkedIn? s84 00:11:48.880 --> 00:12:01.440 The agent will first decide okay so I have a playbook for creating um that specific task and it will use that first fet uh uses that playbook to get the information s85 00:12:01.440 --> 00:12:10.000 and then it calls the necessary uh follows that instructions and calls the relevant tools to get the job done. s86 00:12:10.000 --> 00:12:26.079 This was really powerful um mainly because now anyone at LinkedIn can go ahead and create a set up a playbook and check it into a repository and make it available for everyone else at LinkedIn. s87 00:12:27.440 --> 00:12:41.839 So as people started creating more playbooks so we wanted so this is um one of two foundational principles we want everyone to follow when creating a playbook. s88 00:12:41.839 --> 00:12:53.279 The first one is a playbook should be self-contained which means it should do a very specific task only um for example um if it is for setting up a airflow s89 00:12:53.279 --> 00:12:59.680 DAG it should be about the instruction and the construct should be about uh one specific task. s90 00:12:59.680 --> 00:13:04.480 This helps the agents pick the right playbook for the right task. s91 00:13:04.480 --> 00:13:12.720 And the second uh most important one is to break a big playbook into multiple smaller playbooks. s92 00:13:12.720 --> 00:13:18.000 So this has um and reference those smaller playbooks from a bigger playbook. s93 00:13:18.000 --> 00:13:25.600 This is a really powerful um principle because just like um so it has two main advantages, right? s94 00:13:25.600 --> 00:13:28.880 So the first one is uh reusability. s95 00:13:28.880 --> 00:13:41.839 So if you have a small self-contained playbooks, it can be used from multiple uh reference from multiple playbooks and if you um the another big advantage is progressive discovery of um s96 00:13:41.839 --> 00:13:53.920 context which means the agent only when it needs to read a smaller playbook instead of reading the entire all of the playbooks at once it can uh progressively go and read the playbooks as it wants. s97 00:13:53.920 --> 00:13:57.040 So this is the same concept as skills as well. s98 00:13:57.040 --> 00:14:06.240 So playbooks are very similar to uh skills but we developed this entire system around playbooks even before skills was a thing. s99 00:14:06.240 --> 00:14:21.680 And uh playbooks are a little bit more nuanced because it helps us um it helps us seamlessly capture all of the organizational context and service via MCP without much of a setup. s100 00:14:23.199 --> 00:14:29.440 And another cool thing about this playbooks is this self-improving loop. s101 00:14:29.440 --> 00:14:35.680 So u you have uh engineers creating these playbooks and checking into the repository. s102 00:14:35.680 --> 00:14:40.800 And one of the main problem with any knowledge base is it gets outdated. s103 00:14:40.800 --> 00:14:46.000 How do you uh the biggest problem is how do you keep the context fresh right? s104 00:14:46.000 --> 00:14:49.839 So great thing about agents is they can improvise. s105 00:14:49.839 --> 00:15:00.240 So we have we encourage the agents to whenever they use a particular playbook at the end of the session to identify the learnings. s106 00:15:00.240 --> 00:15:13.760 So any outdated information or any discrepancy or any missing information and we also encourage the agents to figure out how to improve the playbook and use that context to s107 00:15:13.760 --> 00:15:26.320 check it uh to update the playbooks check out the uh repository and update the playbooks and create a PR and that once it gets upload it gets uh the playbooks gets updated s108 00:15:26.320 --> 00:15:32.480 right this creates a really seamless flywheel of a self-arning loop. s109 00:15:34.000 --> 00:15:39.519 So what does the architecture of a MCP server looks like? s110 00:15:39.519 --> 00:15:49.199 So this particular system we have one local MCP server and it is automatically installed on all of the LinkedIn laptops by default. s111 00:15:49.199 --> 00:16:04.000 So if you join LinkedIn and you get a laptop, it is pre-installed and any updates to the uh MCP server or the playbooks or the tools, it automatically gets updated every 1 hour on all the laptops. s112 00:16:04.240 --> 00:16:15.759 And we have a concept of two local playbooks and central playbooks which means so central playbooks are the playbooks which are crosscutting um in nature, right? s113 00:16:15.759 --> 00:16:30.639 So you have um these playbooks apply for multiple uh repositories not just one uh code repository and then you have local playbooks where it these are the playbooks which are very specific to your code repository s114 00:16:30.639 --> 00:16:43.680 and you can just have them checked in with your repo um uh with your repo and when only when the coding agents are working in your repo those playbooks will be automatically picked up. s115 00:16:43.680 --> 00:16:59.360 So this helps us scale the local playbooks which are very specific to repo without having to worry about um uh changing the uh central repository and also this is one MCP server which is serving all of the s116 00:16:59.360 --> 00:17:00.959 uh playbooks and tools. s117 00:17:00.959 --> 00:17:15.760 So this man helps us do a lot of uh central things like seamless authentication, telemetry and u that we can use for learning to make the whole uh ecosystem better. s118 00:17:16.400 --> 00:17:20.559 You may be wondering like how many tools and playbooks it can support, right? s119 00:17:20.559 --> 00:17:23.520 Uh so this is a common problem with MCP. s120 00:17:23.520 --> 00:17:34.720 we cannot scale it beyond 30 or 40 tools without degrading the uh context or degrading the performance of the system. s121 00:17:34.720 --> 00:17:44.559 So what we do is instead of uh surfacing all of these playbooks and tools through MCP we replace them with three meta tools. s122 00:17:44.559 --> 00:17:46.880 So the first one is search. s123 00:17:46.880 --> 00:17:55.440 The agent first uses this tool to search for the relevant tools and playbooks using keywords and tags. s124 00:17:55.440 --> 00:17:58.080 U so we also control the system instructions. s125 00:17:58.080 --> 00:18:07.039 Um so every coding agent is preconfigured with system instruction on how to use these tools and how to use the search really efficiently. s126 00:18:07.039 --> 00:18:19.760 And once it finds the uh right set of tool or playbook, it can then get the more details about that particular tool using get schema and then execute that tool or uh playbook. s127 00:18:19.760 --> 00:18:25.679 So this has allowed us to scale uh to thousands of tools in playbook. s128 00:18:27.039 --> 00:18:30.320 Um so this is u the growth chart. s129 00:18:30.320 --> 00:18:36.799 So now we have over 8,000 users daily um using the system daily. s130 00:18:36.799 --> 00:18:39.039 using uh tools and playbooks. s131 00:18:39.039 --> 00:18:48.480 So we have over,300 uh tools and over 600 uh playbooks and it's not not just engineering right. s132 00:18:48.480 --> 00:18:54.720 So it is uh not just engineers but also product managers, designers, uh TPMS. s133 00:18:54.720 --> 00:19:03.200 So across different functions they are using the tools and bringing their playbooks uh to automate their uh workflows. s134 00:19:05.280 --> 00:19:17.440 uh so I'll leave you with this takeaway uh key takeaways that can based on our learning the first one is the system was successful because we thought about quality and reliability s135 00:19:17.440 --> 00:19:28.080 uh from day one right so even uh before creating a MCP server uh we thought okay the our fundamental principle should be how do we ensure not just productivity s136 00:19:28.080 --> 00:19:37.360 but how do we ensure the quality and also reliability of the system so that it doesn't degrade um as we move fast. s137 00:19:37.360 --> 00:19:42.160 Uh and the second one was u the build the right infrastructure for agents. s138 00:19:42.160 --> 00:19:51.919 Uh in a large enterprise like LinkedIn, it's not enough just enough to give all of the engineers the all the latest and greatest tools and models. s139 00:19:51.919 --> 00:20:00.799 Uh these are they are not very effective if you don't build the right infrastructure for the agents to operate within your enterprise. s140 00:20:01.200 --> 00:20:03.360 Um yeah that's my time. s141 00:20:03.360 --> 00:20:06.523 Thank you for attending and feel free to connect with me on s142 00:20:06.523 --> 00:20:06.559 [applause] s143 00:20:06.559 --> 00:20:08.559 LinkedIn.