WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=9WiBJRO84yY NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/9WiBJRO84yY.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.480 --> 00:00:13.600 All right, everyone. s2 00:00:13.600 --> 00:00:14.680 Hi, everyone. s3 00:00:14.680 --> 00:00:16.640 Um, I want to start with a quick raise of hands. s4 00:00:16.640 --> 00:00:20.320 How many of you have built your own agent harness? s5 00:00:20.320 --> 00:00:20.880 All right. s6 00:00:20.880 --> 00:00:27.080 How many of you um, feel like it's way too intimidating to build an agent harness? s7 00:00:27.080 --> 00:00:27.920 Okay, cool. s8 00:00:27.920 --> 00:00:38.880 So, uh, over the next 18 minutes or so, I want to talk to you all about how you can actually build on top of the Codex harness and hopefully make it a bit easier and less daunting for you to s9 00:00:38.880 --> 00:00:42.840 um, actually build your own interfaces on an on an agent harness. s10 00:00:42.840 --> 00:00:44.680 And hopefully also a little bit fun. s11 00:00:44.680 --> 00:00:45.720 Uh, my name is Dom. s12 00:00:45.720 --> 00:00:50.920 I work uh, at OpenAI on developer experience specifically for Codex. s13 00:00:50.920 --> 00:00:53.080 Um, and I have one more raise of hands. s14 00:00:53.080 --> 00:00:55.000 I promise I'm going to give you all a break. s15 00:00:55.000 --> 00:00:58.880 Um, but how many of you have heard of the Codex app server? s16 00:00:58.880 --> 00:01:05.160 All right, and this was a bit of a trick question because if you attended the keynote yesterday, all of you should have raised your hand. s17 00:01:05.160 --> 00:01:07.560 Um, otherwise you probably didn't listen to Roman. s18 00:01:07.560 --> 00:01:13.200 Um, but uh, we talked briefly about it yesterday, but we didn't really explain what the app server is. s19 00:01:13.200 --> 00:01:18.480 Um, so the app server is actually the protocol that wraps the Codex harness. s20 00:01:18.480 --> 00:01:28.880 Um, if you've ever looked into something like agent client protocol, this is a very similar concept where basically we have a protocol that allows you to directly build on top of the uh, harness s21 00:01:28.880 --> 00:01:30.400 and deeply integrate into it. s22 00:01:30.400 --> 00:01:37.120 And we'll talk a bit in the talk about what the differences are between uh, app server and ACP. s23 00:01:37.120 --> 00:01:43.040 Uh, the big thing here to keep in mind is both the Codex harness as well as the app server itself are open source. s24 00:01:43.040 --> 00:01:46.480 So, we're going to cover a couple of different things on how you can engage with it. s25 00:01:46.480 --> 00:01:56.120 Um, but more importantly, we're going to both dive tomorrow in the afternoon into how a couple of these things work behind the scenes, but you can also always just ask Codex to read the repo s26 00:01:56.120 --> 00:01:57.760 and dive really deep into it. s27 00:01:57.760 --> 00:02:03.880 It's Apache 2 license, so feel free to fork it, do it make it your own and have have fun with it. s28 00:02:03.880 --> 00:02:10.920 You can even connect it to any other model provider as long as they have a responses API capable compatible API. s29 00:02:10.920 --> 00:02:17.520 We even have convenience settings to connect it easily to LM Studio or Ollama. s30 00:02:17.520 --> 00:02:31.360 Um, in terms of how it works, the app server protocol is a JSON RPC style protocol that we actually use to power the Codex app, the IDE extension, our VS Code extension, so all of our first-party interfaces, s31 00:02:31.360 --> 00:02:32.840 but also third-party ones. s32 00:02:32.840 --> 00:02:47.760 So, if you've ever used Codex through Xcode, JetBrains IDEs, or even some open-source projects like Theos T3 Code or Remote X, you can actually all of these are powered by the same harness, the same app server. s33 00:02:47.959 --> 00:02:53.400 We even used it earlier this year to put Codex into Cloud Code using that same app server. s34 00:02:53.400 --> 00:03:09.480 So, if you're still on Cloud Code and you want to use Codex from there either to have it review your code or you know, pass off tasks to Codex to handle it, you can actually use that plugin and it uses that same app server, that same harness behind the scenes. s35 00:03:09.480 --> 00:03:15.280 If you've looked into things like ACP, this should seem pretty familiar. s36 00:03:15.280 --> 00:03:24.720 At a high level, basically the app server, the way it works is we're going to send messages between the client, which is the application you're building, and the server, which is the Codex app server, s37 00:03:24.720 --> 00:03:28.040 to perform different actions and receive events. s38 00:03:28.040 --> 00:03:39.120 But, rather than showing you a bunch of hypothetical messages, I went a bit overboard and actually built a little inspector that intercepts all of the events between the real s39 00:03:39.120 --> 00:03:44.760 Codex app and the Codex app server, so we can look at some real events. s40 00:03:44.760 --> 00:03:48.720 Let me actually switch over here to the Codex app. s41 00:03:48.720 --> 00:03:55.080 So, I loaded up that same inspector in the um, that browser here of the Codex app. s42 00:03:55.080 --> 00:04:00.840 And so, if we send a message, we're going to see here a bunch of different events coming in. s43 00:04:00.840 --> 00:04:09.040 This might seem daunting at first, but as you're sort of going through some of these messages, a lot of this should feel relatively familiar. s44 00:04:09.040 --> 00:04:17.720 So, we can look through here um Some of these might be a Let me do this once more. s45 00:04:17.720 --> 00:04:25.720 Um I think I had some old ones buffered, so I'm just going to start a new one. s46 00:04:28.080 --> 00:04:30.080 There we go. s47 00:04:30.080 --> 00:04:33.720 Also, don't mind my Codex pad up here. s48 00:04:33.720 --> 00:04:44.720 Uh but you're going to see like different messages from like threads being started with sort of your different configurations of what is actually happening, what project are we working on, s49 00:04:44.720 --> 00:04:48.040 over to configuration reads. s50 00:04:48.040 --> 00:04:58.320 Um Where is new threads being started, and then here's like some of your delta messages of like data actually being streamed in. s51 00:04:58.320 --> 00:05:01.760 And so, there's a lot of additional information that you can find here. s52 00:05:01.760 --> 00:05:10.520 And one of the big things that makes this protocol so flexible is that we're really exposing anything that is really in the app, a key functionality. s53 00:05:10.520 --> 00:05:19.600 Meaning, if we're streaming in things, we're doing file search, all of that information is actually handed back and forth between the app server and the and the app. s54 00:05:19.600 --> 00:05:28.440 And so, if you're building your own harness harness integration, your own client, you can actually leverage the same capabilities. s55 00:05:29.720 --> 00:05:39.800 In fact, we have right now over 120 so different client client messages that we continuously add more to. s56 00:05:39.800 --> 00:05:55.400 Um and this goes from everything from initialization where you're able to configure some of the more like specific things to your client, over thread modifications and turns, which are sort of the things that you would assume are like the most classic things. s57 00:05:55.400 --> 00:06:09.640 But also stuff like setting a goal if you want to implement your own version of goal mode, and listing plugins, doing all of the different other aspects that you might see in the app, all the way to things like file search and modification modifying the file system. s58 00:06:09.640 --> 00:06:12.160 So you can really have that full access. s59 00:06:12.160 --> 00:06:16.400 We also try to make it easy so that you can configure the underlying agent. s60 00:06:16.400 --> 00:06:21.880 And that's a bit different to ACP where it's a more more generic protocol. s61 00:06:21.880 --> 00:06:26.720 We can actually give you more control over the individual parts of the harness as well through the protocol. s62 00:06:26.720 --> 00:06:28.360 And we'll look into that in a second. s63 00:06:28.360 --> 00:06:41.320 The other thing that is exciting for a lot of people is you can actually if you're using this protocol, you can have sign in with chat GPT as part of that, meaning that if someone is using your app, they can still bring their own chat GPT subscription s64 00:06:41.320 --> 00:06:44.640 and it goes off their Codex use usage limits. s65 00:06:44.640 --> 00:06:50.120 So if you use something like T3 code or similar, you're actually using that. s66 00:06:50.320 --> 00:06:59.800 In terms of customizing the agent, the most the biggest two things that you might want to look at if you're building on top of this app server is developer instructions s67 00:06:59.800 --> 00:07:07.600 as a way for you to augment the instructions the system instructions of the actual app, as well as bringing tools. s68 00:07:07.600 --> 00:07:12.919 So with developer instructions, we're actually appending on top of the system prompt of Codex. s69 00:07:12.919 --> 00:07:16.680 You can look at the actual system prompt of Codex as well as in the repo. s70 00:07:16.680 --> 00:07:21.240 Um but we generally recommend that you actually append to that using the developer instructions. s71 00:07:21.240 --> 00:07:31.560 That way you're still leveraging all of the work that we put into tweaking the prompts for the individual models, while still customizing it for your specific use case of the harness. s72 00:07:31.560 --> 00:07:43.920 This is for example what we do when you load up the Codex app, since there's some functionality that is more specific for the app over like the IDE extension or CLI example. s73 00:07:43.920 --> 00:07:52.400 And then the other part of this is dynamic tools, where you can expose additional functions that you want to that you want the agent to be able to use. s74 00:07:52.400 --> 00:08:02.200 You could use MCP or other things similar to what you would do in like the agent client protocol, but you can also just straight up define tools, and you can actually mark them as deferred loading, s75 00:08:02.200 --> 00:08:14.200 which means that rather than injecting them into the system prompt, it allows Codex to use a tool search tool, which we'll cover more tomorrow in the other session, but it allows it to s76 00:08:14.200 --> 00:08:19.640 find these tools through tool search rather than polluting the overall context window. s77 00:08:19.640 --> 00:08:26.960 And then on top of that, you can even on a per thread basis modify any other configuration that you might have for your agent. s78 00:08:26.960 --> 00:08:39.000 So, think about setting a different model provider, choosing a different model, or even doing fine-grained control on the sandbox, like what network connections it should have access to, or what files s79 00:08:39.000 --> 00:08:41.280 it can modify. s80 00:08:41.599 --> 00:08:50.160 In terms of how you can actually use the app server, the first thing to know is that the app server actually comes as part of the Codex CLI. s81 00:08:50.160 --> 00:08:51.560 It's the same binary. s82 00:08:51.560 --> 00:08:59.080 So, if you have the Codex CLI already on your system, you can just use that to spin up the app server right now. s83 00:08:59.080 --> 00:09:03.040 But you can also look at how we're, for example, shipping it with the Codex app. s84 00:09:03.040 --> 00:09:07.760 So, the Codex app has the binary as part of the actual bundle. s85 00:09:07.760 --> 00:09:16.360 So, if you're looking into your installed Codex app, you should find it in the contents of that bundle, and it changes with every version of the app that you're updating. s86 00:09:16.360 --> 00:09:19.760 We're shipping a new version of the CLI behind the scenes. s87 00:09:19.760 --> 00:09:29.760 On Windows, we actually ship with two versions of the binary, both the Linux compiled one and the Windows one, so that if you're loading the app in WSL mode, it uses s88 00:09:29.760 --> 00:09:35.120 the Linux one, and otherwise it starts up the app server in Windows native. s89 00:09:35.480 --> 00:09:41.160 Um from here, you actually have a lot of different um, additional convenience functionality. s90 00:09:41.160 --> 00:09:54.880 One, you can have Codex generate you for that particular version of the app server, the TypeScript bindings or the um, JSON schema, which is incredibly helpful since this uh, protocol regularly changes between versions, s91 00:09:54.880 --> 00:10:04.480 which is also why you actually want to bundle it with your application, which is what we're doing with the Codex app, so that you have full control over what are the actual capabilities s92 00:10:04.480 --> 00:10:07.960 of the version that you're expecting to build against. s93 00:10:07.960 --> 00:10:16.720 Um, in terms of protocols, you can spin up the app server both using standard input output, so STDIO, or you can use web sockets. s94 00:10:16.720 --> 00:10:23.720 It's going to depend on your use case and honestly the easiest way to figure out what works better for you is probably to just ask Codex. s95 00:10:23.720 --> 00:10:30.440 Um, from here there's two ways that you can integrate it um, most easily. s96 00:10:30.440 --> 00:10:34.160 The first one is the Python SDK that we recently re- uh, released. s97 00:10:34.160 --> 00:10:39.280 It handles a lot of the life cycle management for you, so you don't have to deal with any of that. s98 00:10:39.280 --> 00:10:46.839 And then on top of that, it makes it a bit easier for you to like integrate things like sign in with ChatGPT and gives you like a familiar interface. s99 00:10:46.839 --> 00:10:54.720 That being said, we are at an AI engineer conference, so um, probably most of you are going to choose option two, which is you just ask Codex. s100 00:10:54.720 --> 00:11:02.839 So unsurprisingly, Codex is incredibly good at implementing this protocol and you largely just have to point it at um, the documentation. s101 00:11:02.839 --> 00:11:07.560 In most cases, you don't even have to do that, but sometimes helps it to nudge it. s102 00:11:07.560 --> 00:11:15.960 Um, and from here you can combine it with any of our plugins like build macOS apps um, if you want to quickly spin up something or completely go wild. s103 00:11:15.960 --> 00:11:25.480 I used it for example to build a client using um, Visual Basic uh, to have a like truly Windows native app um, and it did it largely in one shot. s104 00:11:25.480 --> 00:11:36.200 So uh, you can build really impressive applications with this in a fairly short amount of time without you having to learn all the nitty-gritty parts of of building an agent. s105 00:11:36.560 --> 00:11:44.560 The most interesting thing for me with this with this app server though is that you're not limited to sort of the classic agent chat interface. s106 00:11:44.560 --> 00:11:58.320 You can really be creative whether it's um you how you actually like trigger it inside of your UI and your existing app, whether you're running in the background, whether you put it on weird hardware like Maddie does all the time, s107 00:11:58.320 --> 00:12:13.920 um or if you're like me, you think about how can I shove this into video games, um and recently um I think like a month ago or so I thought about how can I actually run Codex inside of Doom rather than running Doom inside of Codex. s108 00:12:13.920 --> 00:12:20.280 Um and I figured let's actually do that live and I'm going to dismiss a couple of messages here. s109 00:12:20.280 --> 00:12:28.840 Uh so I have this Codex app Doom version and I booted up in the wrong window, so let me pull that over. s110 00:12:29.440 --> 00:12:31.680 There we go. s111 00:12:36.400 --> 00:12:39.480 So this is uh it messed up the controls, my bad. s112 00:12:39.480 --> 00:12:40.600 One sec. s113 00:12:40.600 --> 00:12:53.480 Something with this like dual monitor setup sometimes messes with the control capture, so let's hope that this works better now. s114 00:12:53.560 --> 00:12:54.240 Yep. s115 00:12:54.240 --> 00:13:06.320 Okay, so this is um an Electron app that has uh uses Cloudflare's Doom Wasm render um and it's uh so the most of the game engine itself is implemented in s116 00:13:06.320 --> 00:13:19.000 uh C and then it loads the FreeDoom game except that I actually manipulate had Codex change the game file, so you can see here this is rendered natively in the video game using all of their that game engine. s117 00:13:19.000 --> 00:13:31.440 Um and if we actually interact with it here, we're booting up Codex and again this is rendered natively inside the um actual video game and then passes the passes it to the app server using s118 00:13:31.440 --> 00:13:34.280 Electron's um IPC protocol. s119 00:13:34.280 --> 00:13:39.120 So, if we say hello here, um we can see Codex actually responding. s120 00:13:39.120 --> 00:13:45.480 So, this is using 5.5. The fun part though is you can actually use dynamic tools to expose additional functionality. s121 00:13:45.480 --> 00:14:01.360 So, naturally, um we can get um we can get all of the um armor and health, and it's calling the tools gives me all of that, or even say like s122 00:14:01.360 --> 00:14:06.280 all the weapons with all ammo. s123 00:14:06.280 --> 00:14:09.080 Um and it's calling all of the different tools. s124 00:14:09.080 --> 00:14:14.200 And so, it's interacting with that, but again, remember this is still an app server actually running on my machine. s125 00:14:14.200 --> 00:14:20.680 So, if I say like um what's the directory? s126 00:14:20.680 --> 00:14:27.880 It's actually calling Z shell, getting the current working directory, and you can see here it's rendering is ha it has its own workspace. s127 00:14:27.880 --> 00:14:32.480 So, if you want, you can write code directly into code in inside Doom. s128 00:14:32.480 --> 00:14:34.200 Uh not sure if I would recommend it. s129 00:14:34.200 --> 00:14:37.440 Uh I think the app the Codex app is slightly better for this. s130 00:14:37.440 --> 00:14:42.360 But, you can really have fun here and and implement your own interface. s131 00:14:42.920 --> 00:14:43.960 Awesome. s132 00:14:43.960 --> 00:14:53.480 Uh I talked with a couple of you already during the show about uh the app server, and one question that came up a lot of times is what's the difference with Codex Exec. s133 00:14:53.480 --> 00:14:57.360 So, if you've used the Codex CLI before, you're might be familiar with this. s134 00:14:57.360 --> 00:15:07.080 There's another command called Codex Exec, which spins up a non-interactive interface for you to uh pass uh to kick off tasks to Codex. s135 00:15:07.080 --> 00:15:08.360 This is great for scripting. s136 00:15:08.360 --> 00:15:17.960 If you're building a CI/CD implementation, and you want to script certain acts like auto fixing a PR, um this is a great opportunity. s137 00:15:17.960 --> 00:15:29.040 But, if you're trying to build like a more comprehensive client uh where you're interfacing where you're showing the agent or more importantly if you're trying to do something where you're spinning up a bunch of different s138 00:15:29.040 --> 00:15:30.720 Codex threads at the same time. s139 00:15:30.720 --> 00:15:39.360 For example, you're running you're running your own like eval harness or you want to use like slash goal. s140 00:15:39.360 --> 00:15:48.800 That's where the app server really shines because you have that full control over the harness and you don't have to figure out your own way of parallelizing all of the Codex exact calls s141 00:15:48.800 --> 00:15:55.520 rather than just sending off a bunch of like new thread events and spinning off tasks that way. s142 00:15:56.080 --> 00:16:05.720 Before I let you all go to the keynote, I have a three more tips just to recap what we talked about, things that you should keep in mind if you're using the app server. s143 00:16:05.720 --> 00:16:09.520 The first one is you should actually ship your own version of the harness. s144 00:16:09.520 --> 00:16:25.760 This will eliminate so many headaches for you if you're if you're relying on the user's installed CLI, you're going to constantly run into things from like breaking changes even though we are trying to minimize them, but if you're relying on experimental features that might happen. s145 00:16:25.760 --> 00:16:35.280 But you also don't have to worry about whether a user has actually updated their version and has the latest feature that you want to use or have them nudge them updating it. s146 00:16:35.280 --> 00:16:37.240 So ship your own version of the harness. s147 00:16:37.240 --> 00:16:43.080 The other part is you should absolutely expose new functions as part of your interface. s148 00:16:43.080 --> 00:16:54.680 We do this with a Codex app, for example, for it to be able to control itself as part of the Codex thread, like spinning up automations, etc. But mark them as deferred. s149 00:16:54.680 --> 00:17:06.360 That way you're not polluting the context window unnecessarily with all of the different additional functions that you're adding and instead still have them available for the agent without necessarily confusing it. s150 00:17:06.360 --> 00:17:15.040 And if you are changing the instructions, try to start with over like adding developer instructions rather than changing the base instructions. s151 00:17:15.040 --> 00:17:24.839 It might feel like you want to immediately rush to I'm going to write my own system prompt, but there is a lot of different things that we constantly have to think about when we're building a harness to make sure that s152 00:17:24.839 --> 00:17:29.600 the actual prompt is in distribution and make sure that the agent performs well. s153 00:17:29.600 --> 00:17:34.280 And you're losing out on all of that if you're actually jumping in and like writing your own system prompt. s154 00:17:34.280 --> 00:17:37.320 So be careful there uh if you're doing that. s155 00:17:37.320 --> 00:17:39.000 With that, thank you so much. s156 00:17:39.000 --> 00:17:40.520 Uh thanks for taking your time. s157 00:17:40.520 --> 00:17:49.280 Feel free to scan that QR code if you want to have the slides and I'll be around for uh like the next 15 minutes at the Open AI booth if you want to ask me any questions there. s158 00:17:49.280 --> 00:17:51.640 Thank you so much. s159 00:18:05.347 --> 00:18:07.347 [music]