Cooking with Codex — Charlie Guo & Gabriel Chua, OpenAI

AI Engineer · 67 min · 880 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:18All right.
00:21Good morning, everybody.
00:24Uh, thanks for bearing with us.
00:25There was a long line outside, so we just wanted to make sure everybody could get in and get a seat.
00:30Um, thank you so much for being here.
00:32This is the cooking with Codeex workshop.
00:34We're so excited to have you.
00:36Uh, my name is Charlie.
00:37I'm on the developer experience team at OpenAI.
00:40And my name is Gabriel, also on the developer experience team, flying in from Singapore.
00:45Nice to be here.
00:48Uh so before we kind of dive into the nuts and bolts, you know, I wanted to do a quick review of what have folks been cooking with Codeex.
00:58It is a cooking show.
01:01Uh these you might have seen some of these tweets.
01:04Um but you know there's a real range of stuff that people have been making.
01:09Some of it pretty pretty incredible.
01:13Uh I think this one uh a session with nearly 300 sub aents.
01:18Has anybody beat that in a given session?
01:22No. I think I think Dom might have, but we'll see.
01:25Um Codeex working on a synth plugin uh for Ableton.
01:33Um really really cool stuff.
01:35I think on our team um we have a team member Brent who's doing at this point video editing with codeex driving premiere right which is not what you would normally expect to see from from a coding app.
01:51Uh playable you know games I think like gaming to me is is really a place where where the models and the app can shine.
01:59the ability to go and check your work um for the game as it's done or to hand off between different threads and you know manage audio versus art design versus game mechanics versus you know difficulty balancing.
02:13Um I think I think the different threads and sub aents are amazing here.
02:18And last but not least um you know building a self-driving golf cart with codeex uh which I think is just absolutely incredible.
02:28Um, it kind of makes me wish I was back at school so I could be hacking on uh stuff in my nights and weekends again.
02:38But where have people been cooking?
02:40I think that's pretty fun.
02:42So you we sort of see it everywhere.
02:44Everyone's codex mixing everywhere, anywhere, all the time.
02:48I mean, literally in between meetings, you've seen people like holding their laptops everywhere, keeping it open.
02:53It's become a meme at this point.
02:55Fortunately, we now have Codex on mobile.
02:58So, this uh gentleman in Japan was able to like bounce between ideas in bouncing between cities while on the bullet train or while walking a dog.
03:08And you don't just ha you don't necessarily have to use Codex on the Codex app or use it within like the chat GPD app.
03:16You can just put Codex anywhere with it being open source there being the app server which we'll talk about briefly at the end.
03:22You can put it here.
03:23So this Daniel for example put it into three e- in devices.
03:28So you really can codex max anywhere.
03:31So if it's your first time sort of using codeex uh do check it out.
03:35Do give it a download.
03:36What's really nice is that if you have existing configuration for other agents you can just bring it in and just get started really cook quick.
03:43And today we want to through the cooking show help you cook really fast and cook something really cool.
03:50So there's a quick agenda.
03:52We're going to talk about what's new and then the real pun intended meat of the session is the six ways to cook with codeex.
03:59Some closing thoughts and then the second hour of the session we'll have like a mini hackathon where we're going to have live cooking right in this room and we'll be like walking around helping to be your sue chef to your main chef.
04:11So what's new?
04:11What's fresh from the oven?
04:13A few a bunch of things I think fundamentally to just really help you give context to Codex really fast.
04:20There was record and replay.
04:21You spend time maybe describing, oh, you got to press that button, you got to press this button.
04:26Maybe just show codeexc by doing it.
04:28It builds on sort of our technology with computer use and so forth.
04:31Tread handoff really fun.
04:33You can sort of push things from the cloud to the remote to your local and vice versa.
04:38We'll demonstrate some of that.
04:39And the mobile app is now in generally available.
04:42So now there's like sort of onetoone device pairing, a much more secure way of working with codecs.
04:48So, it's it's been 2026.
04:51This slide barely summarizes everything we've shipped and we're only 50% literally 50% of the year.
04:58Some things have been really fun.
05:00The plugins, computer use, multimodality, being able to use Codex on the mobile, app shots, that's a personal favorite.
05:06These things really make Codex and the Codex app really fun to use.
05:09And we'll be talking about some of these.
05:11They are all sort of key ingredients.
05:14So, that's cooking a Codex.
05:18Uh we wanted to like start with this quote where it's from our favorite movie Ratatouille.
05:23Anyone can cook but only the fearless can be great.
05:25And today with the sort of ingredients we're going to share with you some of the recipes but with your ingenuity we think you're going to cook some really fun stuff.
05:34And these are the six steps to cooking with CEX.
05:37Number one you got to give Codex your context.
05:40There's a lot of context in like your documents, in your brain, in your conversations, in your email, in notion, linear, so forth.
05:48You got to give codeex that context.
05:50That's a really important ingredient.
05:52And then once you have it, you can literally just build things.
05:55And that's already just a step two.
05:57For the power users in the room, there's some interesting stuff where Codex can actually take action.
06:01You might notice right now like on I'm on Chrome and you just see this thing nice thing at the top saying Codex started debugging this browser.
06:08what's happening in background is actually codex is reading the slides as I'm speaking and then you can sort of collaborate with codex on longrunning tasks and you want to work over time where
06:18you want to put this make it repeatable sort of ensure consistency and really like build those loops and finally we want to bring codex everywhere if you want to embed it in your own product or in your own internal platforms that's also possible
06:30so let's just dive right in for the first two steps cooking a codeex where you give it your context and you can just build So, a few days ago, like on Friday,
06:42uh Charlie and Christine and myself, we were thinking, "Yikes, it's it's Friday.
06:47It's Friday night."
06:47We sent it at 4 4:44 p.m. We don't have a demo idea for this workshop.
06:52Uh we had any suggestions.
06:54Then Christine suggested, "Hey, let's build a real-time translator."
06:57Charlie uh suggested, "Hey, let's use GPT realtime translate.
07:00It's one of our latest models.
07:01It's really good."
07:02And then we can even have a plugin.
07:04I've never built a Mac OS app, but with the plug-in, all the best practices are there.
07:08So maybe you would like copy paste, you would like Google or search or what these things are.
07:13Then you go to codeex, but I'm going to do something fun.
07:15So I'm going to take my two thumbs over here, my my right thumb and my left thumb.
07:19I'm going to put it on the command key.
07:20I'm just going to press it.
07:22And bam, that's a app.
07:24All that context on Slack, all the conversation, that brainstorming is just right there.
07:28And let's just ask Codex.
07:32Hey, can you just build this?
07:38Yeah, cooking show is over.
07:39No, I'm kidding.
07:40And we're going to let Codex cook on that.
07:42And what happens with a appshot is that when you take a it's like a screenshot, but all the text is embedded as well.
07:48And it's very nice because that's where you have a lot of personal context, the conversations, there's brainstorming, you don't have to like repeat yourself.
07:55You just give codecs and you can like just figure it out.
07:57So, we're going to look let let that work on it for a bit and we're going to try something else as well.
08:02So, we have the agents SDK.
08:04It's open source.
08:05It's pretty nice.
08:06And uh you got like some things that have come up in the recent months.
08:10So, I was thinking, hey, maybe we could build like a website to sort of just briefly describe what's new and we're going to use this thing called sites.
08:16So, sites is something that we launched earlier in the month in June for in preview for enterprise and business plans.
08:22It lets you sort of create a simple site and it's really about just creating artifacts you can share.
08:26So here we sort I worked with Codex for a bit and let's just see what Codex cooked here.
08:30So we're going to open it up and it's all what's new with the agents SDK.
08:35You have the it's pretty nice.
08:36You have like six changes, sandbox agents, codex as a tool, tool search, realtimes agents.
08:40We'll be talking about that in a in the next few days as well.
08:43And then you can see like the code snippets are here.
08:45So yeah, it's pretty nice with sites.
08:46You can then create artifacts that you can share with your team.
08:49So that's another example of bringing context because you have a code base.
08:53It's a really large codebase.
08:54There's git commits, there's history to it.
08:56is a lot of things.
08:57That's that context and that's step one.
08:59You're just going to give it to Codex to build it and that's step two.
09:01You can just build things.
09:03So let's give in that last last example here.
09:05Maybe like now we're Charlie's out on the streets think uh taking a walk with his kids, his dog, and he's wondering, hey, maybe you wanted to build something.
09:12So I'll let Charlie uh take the stage here.
09:17Got it.
09:17Cool.
09:18Uh so what I want to show you is Codex remote.
09:21Gabe mentioned we might be out in the world, right?
09:23We might be, you know, on the beach.
09:25we might be um camping something but we still want to get work done.
09:28We still want to keep cooking.
09:29So Codex remote is a feature that's now available in chatbt a mobile app and allows us to connect to a you know laptop or remote server and issue commands that can be executed
09:39uh remotely.
09:40So in this case let's say we want to visualize something about this agents SDK right um for me personally you know last weekend I was building a bunch of IKEA furniture so that's kind of on the brain.
09:50Uh, why don't we try to visualize how this repo works in the style of an IKEA manual?
10:00Can you create a visualization of this repository in the style of an IKEA instruction manual?
10:05Uh, I want to understand how all of the primitives fit together.
10:12So, this is going to send for my phone.
10:15And here we go.
10:19So, you can use remote to keep working.
10:21Uh, you know, you leave your laptop on.
10:23Codex will keep the the laptop open while it's cooking in the background.
10:26Um, and it means that, you know, you can sort of free yourself from having to be at the computer all the time or having to walk around with your laptop open all the time, which is 100% a thing that I've been doing lately.
10:37It's almost like a regular occurrence.
10:40Every weekend, everyone's like leaving their laptop in the room at home and just like walking out touching grass, you know, still checking in.
10:46Hey Codex, is everything okay?
10:48Is there anything I can do to help unblock you?
10:49And in a sense, everyone's become a manager.
10:52So, let's see.
10:52Go back to the live translation.
10:54We'll see what's going on over here.
10:55All right, it's building.
10:56It's did some web search.
10:57It's finding out what's going on.
10:59So, I think we've sort of covered very briefly uh to another concept called plugins.
11:04So, just going to the slides.
11:05So, we go there.
11:14Here it is.
11:15Yeah.
11:17So plugins are a bundle that sort of bundle together skills which are best practices, external prompts, uh maybe practices that are custom to your team, apps which have the configuration
11:30and provide the connectivity to external services and MCP servers which provide the tools that are necessary and wow already sort of popped it up there.
11:39We'll just skip that in a bit.
11:41So that's plugins.
11:42And if you just take a look at some of the plugins we were using here.
11:46So if you just go over to the Codex app and you just scroll right up here, there's this button called plugins.
11:51And you'll see a bunch of plugins.
11:52So for example, what we were using was the built Mac OS plugin which has a bunch of best practices uh in terms of the skills for how to use AppKit, how to use liquid glass, how to sign and inspect the built application.
12:06And we were also actually using the OpenAI developers plugin.
12:09So it's 2026.
12:11Um we're not going to go to the developer platform and click uh create API key, save it somewhere, pray that you actually saved it somewhere.
12:18Now with Codex and the plugin, it can just directly create the API key for you as necessary and it really keeps things really smooth as you it doesn't really interrupt you in that flow.
12:28So those were the sort of two plugins we were using.
12:30But there are other plugins that may be relevant like say uh Gmail, Outlook, uh maybe use Microsoft Teams, use Slack and that provides all the relevant connectivity you need for your agent to work to bring context
12:43into the kitchen.
12:44I think I think I would add to that, you know, the Codex app um comes with the skills to build your own plugins.
12:50So in giving the app context, a lot of times I will work through a difficult problem or a workflow that you know I'm not sure how I wasn't sure before how to tell Codeex how to do it.
12:59And when I get to the end of that conversation, I will often say, "Hey, make this a skill or even better make this a plugin so that to next time I need to do this, you can just invoke the plugin and get the job done and we don't have to step through all of this work again."
13:10So this is what Codex built in say 4 minutes and 2 seconds.
13:15Let's see if it works.
13:16So it's a live translation.
13:18So let's let's maybe do like I I understand there's some German speakers in the room.
13:21So let's go let's do German and let's just make sure the audio is working as well.
13:26And let's see what Codex can do.
13:27So again, I've never used built a Mac OS app.
13:31I'm a data scientist by training.
13:33Not very not the best at like sort of working with like web RTC websockets.
13:38But with the Codex app and the plugins, I'm able to get that sort of best practices through the through that.
13:43So let's just do that.
13:44Make sure the input change to the phone, the output is here.
13:49And all right, fingers crossed.
13:50This is a live app.
13:50It's it's built.
13:51So this is like we don't know how this is going to taste.
13:53And fingers really fingers crossed.
13:57Oops.
14:10Hello there.
14:13Hello.
14:14Mic test.
14:14One, two, three.
14:19Okay, let's change the audio.
14:28You know it's a real demo, live demo when you got to do it on stage.
14:32It's real.
14:36Mic test.
14:37One, two, three.
14:37Are you there?
14:43Yeah.
14:46Thanks for the tip.
14:49Hello.
14:49Are you dead?
14:51Hello.
14:53So, right now I'm at the AI engineer conference
15:01and I'm like winging this live.
15:02I have really It's actually not the most set up.
15:10Yeah, cuz like actually right now the speakers my mic.
15:19How was that?
15:20All right.
15:31So, one thing is that context is not just about best practices or external information.
15:37There's a lot of context in our heads.
15:39Whenever you talk to Codex, there's a lot of a lot of things that Codex can learn about you.
15:43And that's where memory comes in.
15:45It's really useful.
15:46It's I think about it as bringing context over time.
15:49So it is off by default but if you go to Codex app you can turn it on and let's say like for example I use UV and then Codex would learn that over time.
15:56I prefer using UV for Python package management.
15:59But memories and context isn't just textural.
16:02It's also about what we're doing what's on screen and so forth.
16:04And we have a research preview called Chronicle for pro subscribers.
16:08Also worth checking out.
16:09And with that you could literally for example be on the GitLab GitHub looking at your pipeline it failed and you just ask Codex why is this failing and it sees ah you were looking at your GitLab your GitHub pipelines and then it'll sort of figure it out.
16:22So to sort of set that up it's quite straightforward you can just go to the Codex app you go into settings and then it's called personalization and you can enable it.
16:30So you can enable memories and also sort of given the um as a safety feature you can also sort of skip uh generating memories in chats that involve tools because sometimes the tools might return information that you might not want persisted over time.
16:45So that's also worth considering.
16:47Right now I'm using a business plan so that isn't chronicle but if you're on a pro plan you would see chronicle as option to enable here as well.
16:54And then there's this thing called custom instructions.
16:56So here you can sort of en it's like setting the agents.mmockdown in the home directory.
17:01It applies to all conversations.
17:03So for example, I said here always reply like a pirate.
17:06Then in the subsequent commentary, Codex would always reply like a pirate.
17:09A trivial example.
17:11So that's context.
17:12And to summarize, I'll let Charlie uh sort of summarize our first two dishes of the day.
17:21So to recap, uh part one, right, we want to give codecs context.
17:26Um you need to know which ingredients you're actually using for your recipe.
17:30Um that includes bringing in external content via plugins, right?
17:34Me personally, I am a pretty heavy user of Slack, Gmail, linear.
17:37Um and then it makes it really easy to just say like, hey, go look at my, you know, unread Slack messages for today and create a daily agenda for me to to think about when I get into work.
17:47Um, you want to be able to like keep the flow when you're adding your own individual context for the tasks that you're working on.
17:52Uh, I love dictation.
17:54I think just being able to, you know, as we've been demoing up here on stage, say, "Here's the context.
17:58Um, go do it."
17:59I think dictation to me is a really underutilized feature because we speak so much faster than we can type, right?
18:06And these days, you know, we're we're beyond the the era where you have to make sure every single word is correct when talking to a large language model.
18:14you can just give it reams and reams of your own rambling uh in transcription form and it'll generally figure out the right thing to do, right?
18:20And you're able to actually talk through the idea and and get to a better space by the time you send it to the model.
18:26Uh likewise, appshots I think are are incredible and also very heavily underused.
18:30Um the ability to just like from any app I think I I don't copy and paste, you know, anything from Slack anymore.
18:36I just appshot my my Slack window or I just appshot my browser and send it directly into codeex um and tell it go figure it out, right?
18:44And you know 90 plus% of the time it can figure out the right thing to do.
18:49Last but not least, we've got personalization.
18:51Um I think you know agents MD which hopefully folks know and love.
18:55Uh and then memories as well uh in the Codex app via Chronicle.
18:58Um, worth noting, I think memories, uh, and Chronicle will use up a little bit more of your token budget, but, uh, it can be a really magical experience to have the app just kind of understand what it should be doing based on conversations you've had it in the past.
19:11It really levels it up and makes it starts to feel like a true collaborator.
19:15Um, there's a ton more stuff, I think, both in like context and personalization in the Codex app.
19:20Strongly recommend you go check it out.
19:21Um, pets is like a universally beloved uh, personalization feature which we're not really going to showcase here today, but um, is would would yeah, definitely go make your pet.
19:32Cool.
19:34And once you have all the ingredients, it's you you're here to build and to really help you build and you want to distribute it.
19:41Sites is a great thing.
19:42At OpenAI, we've been using sites extensively.
19:45Uh I for example have been using sites to summarize like an event like an event after like like after an event like today's I would ask hey Codex can you summarize what happened today create a site then we can share it with the team the rest of the organization
19:57I create a site to sort of uh we we were planning offsite and then Cory Cory on the team created a site to help us plan choose different locations
20:05understand the trade-offs so you can create artifacts to help you make decisions specific for code you can even you can also it helps with code review so let's just give a quick example with a live translate what we had just built.
20:17You just do slashcode review and then you can review against uncommitted changes and then voila, you're firing off a code review task.
20:24It's a bit like having that uh you know sue chef like double check everything.
20:28You know, you've seen the movies right before the chef like sends out a dish, they always do a quick taste test.
20:32I think that's a bit like that.
20:34Image gen where you can create images.
20:36And let's see what the example Charlie kicked off created.
20:41Oh, okay.
20:41It used SVG but not too bad.
20:44Not too bad.
20:44So then what you could also do is use the image gen skill and ask codeex to sort of cook it.
20:49So we'll come back to that in a bit but let's try that.
20:52Let's use this and spreadsheets document and slides.
20:57We won't be talking much about it today but you can sort of build these artifacts as well.
21:01And in the subsequent talks in the rest of the days and the AI engineers uh Jason for example will be sharing more about how he's sort of building with codecs for sort of these uh
21:10deliverables.
21:12So those are the first two steps, context and building.
21:16But it doesn't just stop there.
21:18So if you're sort of new to Codex, it's your first time still using coding agents.
21:22I think that's really helpful.
21:23But I'm I'm pretty sure this AI engineers welfare.
21:25We're in a room full of power users.
21:27So for the next four steps, we hope we're going to share something that's practically useful for you.
21:31And the first one is computer use.
21:33So Codex can sort of take actions like as you see right now in the browser, Codex is debugging something.
21:38I won't tell you what's happening.
21:40I will we'll reveal the surprise at the end but sort of working in the background thinking about something.
21:45So let's give an example.
21:46So I I used to work in a large organization before open AAI and not everything had an API.
21:52You had a lot of enterprise software that had like they were legacy.
21:54There were different dashboards.
21:56You wanted to download data.
21:57So let's say for example we have this dashboard that somehow pulls data from the OpenAI Python GitHub repo.
22:03There's this nice date range picker.
22:05But imagine I had to get data for every month from the start of the year the year to today I'll press Monday to Sunday, Monday to Sunday, Monday to Sunday, Monday to Sunday.
22:17It's a lot of buttons you have to press.
22:18Then there are different data sets you could choose.
22:20So what if Codex could help me with that?
22:22And the answer is yes.
22:23So again, I'm going to take a shot my two thumbs and then just take it.
22:28Okay, let's just make sure we're in a new conversation.
22:32I'll just ask,
22:34hey, um, can you help me download the, uh, number of commits for the last seven days as a CSV and the number of pull requests, the pull request data as a JSON
22:46from the start of the month to today.
22:48Uh, yeah, use like computer use and go for it.
22:52So again, it's like two different dates.
22:54I'm asking it for the like Monday to Sunday or the last seven dates days but also from like the 1st of June to today which is the 29th of June and you would see okay Codex has correctly invoked the computer use skill and very soon it's going to like start looking at the application
23:10um you see here looked at application and very soon in the chain of thought summaries you would see more of it and very very soon again my hands are not on the keyboard and you can see this is my cursor I'm just going to like park it over here right next to enterprise is uh you would see
23:25another cursor pop up.
23:26There it goes.
23:27And yeah, there it is.
23:28It's moving.
23:29It's downloading it again.
23:30My hands are here.
23:30Charlie's hands are here.
23:31My phone is like unlock.
23:32It's locked.
23:33And Codex is downloading all this data by itself.
23:36It's a bit like a mag.
23:36It's like pretty magical the first time you see it.
23:39And that's we're going to let Codex work in the background.
23:41And that's the beauty of computer use of Codeex.
23:43You don't have to wrestle with the agent for control of your screen.
23:46You can let it work in the background.
23:48I I think it's also worth pointing out that on Mac OS, you know, it doesn't uh steal your cursor, right?
23:53Like Gabe still opening the app.
23:54He can still use a different tab.
23:56I can go scroll on Twitter and something.
23:58Yeah.
23:58Yeah.
23:58And so you can still drive your computer while things are just happening, you know, in the apps that Codex needs access to.
24:04So we're going to let that work.
24:06But sometimes not everything is like a native application.
24:09And there's some sometimes you're like maybe using Google Drive, you're creating a feedback form like for today's workshop, you know, I mean feedback forms are I personally find it very cumbersome to create a feedback form.
24:21you need to like uh create a question, you need to like fill in the different options and I always forget to mark it as mandatory even though I want it to be mandatory and it's you press so many buttons with creating these feedback forms.
24:32So I already have the questions drafted here and let's see if Codex can help me with that.
24:36So I just create a new thread.
24:38So again, appshot, it's quite addictive, I must say, the user interface.
24:45And we just say, hey, um, can you create these uh this feedback form for me using the questions in the Google docs?
24:52Um, remember to make mark each question as mandatory.
24:57And just to give codeex a bit of a I mean just to make things for exposition purposes, what we're using here is the Chrome extension, which is slightly similar to computer use.
25:08So that's this is computer use, but here it's directly controlling a specific application which is Chrome.
25:13So we're going to fire that off and we're going to see very soon another cursor come up.
25:17So I'm going to put this two side by side over here and you would see codeex thinking about the task.
25:23It realizes, hey, I have the question form questions already.
25:26Then it still needs to drive the live tab to actually create it.
25:30And again, like what we saw earlier, it's going to sort of start pressing buttons.
25:33So let's just wait for a bit to see the first button get pressed and then we'll swap back to the slides in a bit.
25:39So ah there it is.
25:40You see it again.
25:41My cursor is here and then you see the blue ones here.
25:43So you can continue to go do different tasks.
25:45Let's say like maybe I'm interested to learn something about the codeex repo.
25:49We were talking about memories.
25:51So I can just like ask hey um can I configure the model that's used for memories in codecs and what's the config uh term I need to change?
26:01So you see while you're continuing to do other tasks, codeex in the background is sort of doing the button pressing for you and we'll fire that off.
26:08So that's computer use and there are like different flavors of it like you know how maybe you know your favorite dishes come in different flavors and I'll let Charlie just really give a nice summary of what where the different flavors are.
26:24Yeah, at this point there's there's like mentioned a few different ways that Codex can start controlling the applications that you're using.
26:29Um, and it's kind of important to understand, you know, when you should reach for each one.
26:33The three that we currently have today are computer use, which you saw, uh, the Chrome extension, which you also saw, uh, and the inapp browser, which I think we did not demo.
26:41Um, the inapp browser, why don't I start there, is a feature where in the Codex app, you know, it has it the ability to open, uh, a browser tab on the side just like you would kind of a normal um, you know, desktop coding editor these days.
26:53Uh but in addition to just opening and rendering web pages, uh Codex has really deep hooks into that browser engine uh deeper than it has at the Chrome extension.
27:02Um and so as a result, uh I think the inapp browser is really great for when you're building local web applications and you want to do a lot of deep testing
27:10um on the interaction design on how it's rendering because Codex can see much further into the uh into the the layout at that point.
27:18But going back to the two that we talked about today, um we have computer use, right?
27:22Which is great for um I mean anything on your on your laptop or your computer.
27:26Um but it really shines when you have applications that otherwise don't have a programmatic interface to them, right?
27:32Um it's great if it's a web app and you can reverse engineer the API calls, but you know there's a lot of desktop software out there.
27:38There's a lot of legacy software out there which doesn't easily expose that kind of interface to an agent.
27:44And so if you just need C codeex to drive the app in the same way that you a human being would computer use is really incredible and kind of unlike anything else that that I've tried it it to this day it's still just so magical to watch those little cursors fly around.
27:57Um then there's the Chrome extension uh which is your sign-in browser and I think that is the most useful for when you need to hand off um state or authorization
28:06to the agent.
28:07So, I think we've all, you know, probably run into the issue where we want to build an agent to go and, you know, use the worldwide web the exact same way that we do, and the first thing that it immediately runs into is like, "Oops, I can't log into your Gmail account."
28:18Um, and worse than that, like Google has detected I'm on a headless Chromium, and it's just sending me this infinite capture loop, right?
28:24Uh, and so the Chrome extension allows codeex with your permission to go and read from certain websites as you uh and borrow your credentials so that you don't have to worry about the authentication
28:34step as you're building.
28:38Cool.
28:38Um, so that's Codex take action and then next we're going to move on to collaborating on complex tasks.
28:45So this one's a really fun one.
28:47So I think people have really talked about how Codex is really good at like complex refactors, longunning tasks, and in the sort of next 15 to 20 minutes, I want to share some tips on how you can sort of work best with codecs on these complex tasks.
29:03So, it's going to be a bit hard to like kick off a 20our longrunning task and we're all just going to sit here, enjoy lunch, dinner, and breakfast.
29:10So, I started off some tasks the night before, and they're like four different tasks, each covering a slightly different flavor.
29:17So, this one, it's still running for like 15 hours.
29:20It's running on my remote uh uh instance.
29:24You can see at the bottom here, I'm running a goal.
29:27It's been running for 15 hours, 30 minutes, and 55 seconds.
29:31So, I got to scroll, scroll, scroll, and here I am right at the top.
29:33So, it's a bit of like a green field project.
29:36I want to build a live Q&A site.
29:39Uh, it's going to be in a room for like 400 people.
29:42There's like a admin view.
29:43There's a participant view.
29:45There is a onstage view.
29:47And then like I describe, oh, maybe the admin can do like AI generated tags.
29:51There's a moderation features on the stage view.
29:54There's a 16x9 uh run every application through our moderation API.
29:58make the experience feel like something OpenAI would launch so on and so forth.
30:02Uh here I'm being particularly prosaic in the prompt for exposition purposes.
30:07You could just ramble to codeex as well and it'll get it'll get the job done.
30:10But here just like exposition purposes, I know people are excited to take photos.
30:14This is like an example of how you're being clear.
30:16I think one misconception these days is like people always ask does prompt engineering matter?
30:22In so far as the instructions are clear, it still matters.
30:26Like I can't read your mind.
30:27Codex can't read your mind.
30:28So you just need to tell Codex very clearly what you want.
30:31But it's prompt engineering where you see as oh I got to use this magic phrase.
30:35I don't think we're in that state anymore.
30:36Reasoning models are really intelligent and they sort of the real thing is about context.
30:40So actually goes back to our first step giving context clearly.
30:43So how I wrote this prompt was I was just talking to Codex.
30:45I was I was yapping it on yapping to it on the phone.
30:48I just hey Codex can like polish this prompt so I can show it on stage.
30:52So yeah it's building this thing and I'm using like a few different things like convex.
30:56There's a a plugin for convex that helps to do uh real-time interactions going to deploy in a cell and of course the open AAI uh plugin and all right the feedback form is done.
31:06So that very nicely came up and yeah that's one example so it's like a green field project but you know not everything is like you're going to start from scratch.
31:13So here's another example that ran for uh 5 hours and 57 seconds.
31:19So here's an open source repo for a form builder and I wondered, hey, could I like repurpose this and build it internally in sites?
31:26So maybe you don't want to buy SAS.
31:28It's a small team of like seven of you.
31:29You want to just have a internal form builder.
31:31You want to use the chat GPT authentication that comes with these sites to sort of ensure that only people within your team use it and then you got to let Codex sort of build it.
31:40And I I will show you an example what that looks like later.
31:43Then the last example of a long-running task that was running.
31:46This one ran for uh let's see I think 7 hours 1 minute and 40 seconds.
31:51So it's like some uh gradu uh research papers from my graduate school days.
31:56Uh it was like written in like Julia then I think like an R and I was like hey can I like write a Python package with a Russ backend and then I told Codex again I gave some constraints I don't want to use dynamic programming you got to use like some heruristics and then codex read the papers
32:10and did it.
32:11So those are the three longrunning tasks and I want to like give some examples and some of the best practices that come with it.
32:17So in one of these for example what I told Codex was this over here you can define a goal.
32:25So there is actually a skill in the openi skills repo called define goal and I told codex hey don't create one goal create a series of goals and put it in a goals.mmuckdown
32:35file.
32:35Again, it's just very arbitrary, but it's just a collection of files that you sort of see and you can can inspect it and then create a dashboard, progress dashboard.html,
32:43so you can go back to it and see what's going on.
32:45Because in this seven hours, you don't really know what's going on.
32:48You want to have some some way to like see what's happening.
32:51And rather than having this one thread do all the work, it's actually spawning new threads.
32:56So, if we go to the see, I've pinned the project, but if we go to the the thread, but if we go to the project, you'll see it's it's spun many other threads.
33:04So for example at different milestones it ran code review and then it also ran a goal audit.
33:10So it's seeing like hey are you on track there's this initial thing you talked about like goals mount down are you deviating from it?
33:17If it is then it will nudge back the main thread.
33:19Hey remember this or do we need to rethink the plan and so forth.
33:23So it ran at different times.
33:24That's why there's like different milestones M2 M3A and so forth.
33:28Uh there's another example here with like the live Q&A site.
33:32Uh this is still running.
33:33So this is running on a remote session.
33:35So it's like my gaps remote and I want to like do testing.
33:38So what I told Codex is that hey you can spawn a local thread on my machine and use the Chrome extension to then inspect it.
33:45So you would see for example this is a thread that came and you would see it says here send by codeex from another thread.
33:51So the remote thread can communicate with the local thread and says hey you got to do click ops you got to do browser you got to do QA testing.
33:58Then the local thread did all the QA testing and it send the results back to the remote thread.
34:03So that's another thing that's uh worth uh thinking about for these longunning tasks.
34:07And let's see what else we can call out here in the long running tasks.
34:12Ah communication.
34:13So one other way of thinking about this is that maybe you could just ask codeex to send you Slack messages.
34:20So here I have a few demos.
34:22So, I was building a live Q&A site and without having to set up additional infrastructure, you can just use the Slack plugin and just say, "Hey, um, reply to this thread and give me like a summary of what's done, what's not yet done, and are there any blockers?"
34:35Because sometimes there are blockers.
34:36Maybe there's like a authentication issue, there's an API key that you need to give to Codeex and so forth.
34:41So, I asked Codex to speak to me.
34:42And over the weekend, like on Sunday while I was out with my friends, I was just regularly checking, is there anything I can help unblock Codex for these different tasks?
34:50Um let's just give some screenshots of what that process looked like.
34:55So just take a look at it.
34:58So another nice feature with codeex is that while you have the main thread.
35:01So right here in you see in the screenshot on the right left is the main thread on the right you can start a site thread.
35:09So you literally see here and say let's focus on signing in with chat GPD first.
35:13I want to sleep soon.
35:14I assume you need me to do that because I know like okay I think Codex needs probably some help with that because I'm running a building a new instance on the app server a new user interface so it needs to get some credentials so it needs to know where my credentials are and it says okay actually the
35:27credentials are already here so okay update the main thread so you can use the site thread to update the main thread so let's do an example of that live as well so here
35:37this one is still running you could just do slash site and it creates a site thread and you can just ask codeex like what's going on.
35:44So maybe we just ask it takes a while this I think due to the network but let's say uh hey um what's going on can you list like uh what you've done so far maybe in the last hour what do you plan to do next is there anything I can do to unblock you
35:57and then from this when you get the agent tells what's going to do next you can also get a sense of like what maybe it's like on the wrong wrong track then you can sort of tell codex hey maybe you don't want to do that we should do this instead then the site thread can communicate with the
36:09main thread so that's a pretty nice uh feature
36:12it's worth noting with side threads.
36:13Those are ephemeral.
36:14So if there is any context that you want to keep longterm in your project, make sure you're communicating that back to the main thread because they will get stale after a while.
36:23Uh here's another example where what we were doing earlier was a site thread talking to the main thread.
36:29This example is where two different main threads are talking to each other.
36:33Specifically on the right uh the left side um we have uh codecs saying test spawning a local thread in the local form builder project.
36:43If it succeeds make that the default location for all local future local threats especially for Chrome or end to end testing.
36:50So again on the what's happening here is that it's running on the remote service but so it doesn't have access to my chrome chrome browser.
36:57Then it send a message to the local host and then it can then from there uh see uh chrome appears to be installed and then from there it can do the testing with chrome.
37:06So there you can get the remote session to work with the local session and this was the test that it did.
37:12So while it's working in the cloud locally on my laptop like me at 3:00 a.m. while sleeping Codex opened Chrome it went through the form building flow and it created a form end to end.
37:23Sub aents are also a particularly helpful thing in these longunning tasks because you want to make sure context is well separated.
37:29Uh agents sort of focus on different tasks and here's one nice example.
37:34So it was building a application and I just like took a peek at it and I just not necessarily best practice.
37:41I just sent it to the main thread saying not very good.
37:45Yeah.
37:45And then Codex said yeah agreed that's technically cleaner but it's still a bad chat interface.
37:50And you notice here what it did was that it messaged the sub agent.
37:54So the sub agent was already previously spawned but then the main agent messaged the sub aent and you would see this the message over here.
38:00So for example it says this.
38:02So it sent it and says update your verification task with this new user screenshot and directive.
38:07So there you have the sort of inter agent communication.
38:11Uh so this was that what it did thereafter.
38:14So it did its own tests because I sent a screenshot with the messages flushed to the bottom.
38:18I didn't want that.
38:19So then Codex did its own test and it says uh send retroex visible okay in one short sentence and it sent it.
38:25So CEX itself did its own QA testing to sort of verify my feedback.
38:31Um and this is an example for the live QA where the main thread was spawning different other site threads for example the audit of the goal the code review but also doing that communication back to Slack.
38:42So this thread over here where it says provide slack thread URL that's over here where I was providing all these updates.
38:48So a separate sort of session was managing that.
38:52We earlier talked about goal and here is a dashboard that the codex created for the goal.
38:57So you would see it has all these different milestones, what's been completed, what's active, what's yet to be started.
39:04And then the dashboard also gave me a sense of like what the different threads are doing like oh this thread is the main orchestrator.
39:10There's a communication thread.
39:11There's one thread doing like visual UX.
39:14There's one doing like the managing the app server protocol and so forth.
39:18And even like creates it into like more conceptual work streams like one workstream on off one workstream on testing one workstream on UI polish.
39:28But throughout the night sometimes codeex does get blocked.
39:31So then with the example of convex in versel, it said oh um I I marked it as blocked because the critical path it didn't have the authentication didn't have like the identity to access these external services
39:42but still I did everything I could up till that point.
39:44So it said it was blocked so it will come up in the UI and then I just selected it and I say add to site chat and it says can you please explain what's going on because I just woke up like I I didn't have context of what's going on but Codex is telling me he was blocked.
39:57So I'm trying to get Codex to to catch me up to speed.
40:01So here it's like saying explain step by step what I can do to unblock you here.
40:05So it gave me different options how I can give the code VLE details the convex details and then I sort of did it.
40:12So here it I sort of I selected it and say do that.
40:16So then the site thread told the main thread to sort of create the CLI login flow and I'll just click it over here and I'll be able to then authenticate and give vers uh codeex which again is running on a remote instance
40:27access and the right permissions to manage my versel project.
40:31Then in the case of convex there's a deployment key.
40:34So I was like hey I'm not going to paste that into the chat.
40:36So I asked Codex give me a secure way to do it.
40:39Then it gave me this like bash code that I could just paste it in and I could then set the key into the remote environment.
40:45So that was like a way where you could collaborate with Codex in a remote instance, do different testing, managing permissions and so forth.
40:52Um, so let's see what they actually built from this entire process.
40:56So let's look at a site builder site one.
40:58So this is it, the internal form builder.
41:00So to recap again, there's an existing repo where of a form builder, but I want to like customize it for like maybe my internal team.
41:08We're like maybe five of us.
41:10I don't want to spend money on SAS.
41:12So we can build our own form builder.
41:14So it creates a new draft.
41:15You could call it like AI engineers uh workshop.
41:19Hello.
41:20And then just say like yeah I should I should ask the Chrome extension to do this but you know for exposition purposes.
41:27Okay.
41:28And you can like preview the form and then you can sort of share it thereafter.
41:32You can like copy publish the link.
41:35Then there'll be a sharable link.
41:36You open the link and then someone else in your team can fill it in.
41:39And again because this is hosted on sites only people that have access to your chat GPT workspace can then access this form.
41:47So that's really nice.
41:47Yeah.
41:48Then you can see all the responses and so forth.
41:49So this was again what we did.
41:51So to summarize on the complex task, we had three different projects.
41:56One was a machine learning one where I wanted to reimplement a package with a Rust back end.
42:02one was to take a existing uh to build from scratch like a green field project a live Q&A site and the third was to take an existing open source project and customize it for my sort of needs and codex was able to sort of get it done in those cases.
42:18So underlying this are a few sort of uh concepts and primitives here.
42:22Let's go back to the slides.
42:28So the first thing is compaction.
42:30You saw how codex worked for seven hours.
42:32It worked for 5 hours.
42:33It worked for like right now I think the other one is like 12 hours.
42:36And many times it's compacting its context window.
42:39This is not a new technology.
42:41In November 19 when we released GPT 5.1 Codex Max, yes the name was GPT 5.1 Codex Max, we natively train the model on multiple uh context windows through a process called compaction.
42:55And that's what lets Codex be really good at it.
42:57Um some examples of what people have said about compaction to quote did open AI basically solve compaction I pretty much never had issue with 5.5 and codecs across ultra long threads spanning
43:09many compactions.
43:11Uh this individual from Japan said like in fact codeex has strong compaction.
43:16So even if you make a lot a lot in the same thread it's less likely to forget.
43:20And in that sense because it only remembers the good stuff.
43:22There's some token efficiency to that I suppose.
43:25And as always, people think it's pretty quoted.
43:29I'll let uh Charlie share a bit more about like specific uh primitives and concepts that enable this such as goal.
43:35Yeah, thanks Gabe.
43:38So, as you saw, we we have a goal that's already been running for 15 hours.
43:41And uh I think you know goal is is one of these primitives that um is incredibly powerful if you can use it in the right way.
43:48Uh, one of the biggest things that I tell people who are just trying it out for the first time is be really specific with the criteria that you're giving the model to to know when it's done.
43:57Um, and specifically, you want that criteria to be as verifiable as possible.
44:02Uh, sometimes I do feel a little bit like goal is is kind of like a genie in the sense that when it works well, it is insanely magical.
44:09uh and when it doesn't sometimes I feel like I've entered into a monkeykey's paw type situation where you know it has done the thing technically that I asked it to do but like very much not in the way that I was expecting.
44:18Um so it when you give it a task it'll run off it'll keep evaluating itself against the success criteria against the verifiable criteria that you've given it.
44:27Um and at each turn it'll check hey did this actually complete um and if not should I keep going right um there are quite a lot of use cases
44:35uh for very large projects in this way.
44:37Um, you know, Tama, as you can see, used a SLGO that was running for 40 hours, uh, that did a reimplementation of Doom in native Swift code.
44:46Um, if that's the type of thing that that you want to give a shot.
44:49Um, but also things like code migrations, large refactors, uh, retrying loops, uh, experiments, and, you know, like games or or full oneshotting apps with fairly detailed specs.
45:00Um, I think goal is is pretty incredible.
45:04Uh we've also got sub aents which we uh saw a little bit of a preview of here right sub aents in codeex um you know by default will get used out of the box.
45:13You can tell tell the model hey uh you know think about delegating to sub agents here um but you can also customize them.
45:19So if you want if you know you have a sub agent that should be your um you know code reviewer right you can have a specific prompt set of developer instructions here.
45:29Um, if you know you want that code reviewer to just be focused on um, like just be a lightweight model so that it runs quickly.
45:36Or if you want it to be a much more intelligent model so that it's very thorough, you can customize that in its own uh, sub aents.totml file.
45:43Um, and you can give these names.
45:44You can say, oh, go, you know, check with the code reviewer, go check with the docs researcher, um, in order to to implement, right?
45:51Sub aents uh, can work in tandem with uh, thread-to-thread handoff in codec.
45:56So, um, thread to thread is is quite new.
45:58I think we just released it in the last week.
46:01Yeah.
46:01Um and so uh that's just an even higher level of abstraction.
46:06Um and the way that I think about balancing them is, um when the the delegation of work needs to be visible to you, the user versus when it just needs to be visible to the model.
46:17So sub aents we started with because I think that is when the work, you know, primarily needs to be visible to the model.
46:22um if you want to delegate, you know, some uh like a subcontractor, you know, I don't really care what they're talking about.
46:28I just want to make sure that the work comes back good.
46:31Sub aents are a great fit and the model is great at figuring out when it should be um handing off into sub aents.
46:36Uh it also keeps the the context separate.
46:38I think unlike you know a lot of other tools that um we've we've brought to codeex to help keep context windows efficient.
46:45Um, sub agents are one where uh, you know, you're keeping the context separate and the mo the the models are communicating with each other.
46:52Thread to thread handoff is useful for when you still want to be in the loop and you still want to be able to like look at everything that's happening.
46:57Um, and perhaps even larger than that, you are mentally thinking about separating these two uh, these two threads, right?
47:03So, um, one example is over the weekend I was working on a game um, and I had a creative director thread, right?
47:10I could have used a sub agent and that would have worked fine, but I wanted to see all of the different things that were happening and I I didn't want it to just be delegated into the ether via sub agents.
47:19So, with the creative director thread, I opened up um different threads, one to work on the art direction, one to work on the music, one to polish the animations, one to figure out the game mechanics,
47:29and I was using those um and I told them in the agents MD file, when you're done, go and check with the creative director thread, which I'd given a bunch of extra context on, you know, the look and feel I wanted for the game.
47:41Go check with the creative director and get sign off before you consider this task done, right?
47:46Uh and they all did so.
47:47that, you know, it was actually really interesting to watch the threads just constantly sending messages to each other back and forth as they worked on the game.
47:56Uh, we didn't quite show it here, but we also have hooks in the Codex app.
48:00Um, if you want to set up your own hooks, you can do so in your config.toml file uh in your codeex home directory.
48:07Um, these are really great for when you want to start introducing uh deterministic behavior at specific checkpoints in your software development life cycle or in your just you know general
48:17uh development life cycle.
48:19So I think in these cases you might use hooks to do things like um you know send your conversation to a logging engine right um you might want to scan inputs or outputs.
48:28I think there's quite a lot of security use cases for hooks, right?
48:31Before Codeex takes a specific action, maybe you want to look at the prompts and figure out, hey, have we accidentally pasted an API key that's that's going to go somewhere
48:39um or before it takes an action to call a tool?
48:42Hey, is this tool, you know, on like not on our approved white list or something that we need to be extra sensitive about when calling the tool to evaluate if it's if it's safe to use.
48:51Um, and we can also do custom validation checks as well when the turn stops.
48:55these days, you know, you can tell the agency MD, hey, go and use the the llinter, right, before you consider the work done to check your work.
49:02Um, and it's really good about doing that, but if there are things that are a little bit more sophisticated or a little bit more complex that you want to do to validate the output,
49:09uh, you can use hooks to to take care of that as well.
49:11Um, you know, I find that when building more and more complex projects, some of the most important things are figuring out, like I've said, the verification criteria or the success criteria.
49:20You want to give the model as many boundaries as it can as you can and let it fill in the lines um in accordance with like the boundaries that you've given it.
49:31Cool.
49:32Um and that brings us to working over time.
49:36Thanks Charlie.
49:37So just to quickly share on hooks uh slipped to my mind as well.
49:43So we have the OpenAI agents SDK and there's actually a quick example of hooks right over there.
49:49So if you go to settings, all the hooks are sort of there.
49:51And you would see the hook for the agents SDK repo.
49:54And here it's sort of doing at the every end of its turn a Python script to just tidy the repo.
50:00So again, there like some of these housekeeping that maybe you want the language model to do.
50:04You can run it deterministically with a hook.
50:06And hooks become particularly useful with longunning tasks because there is that trade-off.
50:11You're giving the agent more autonomy.
50:12It's going to work off work longer and longer without your supervision or abstracted supervision.
50:18And to that to that end hooks become useful in getting the guardrail.
50:22So on working over time what we're trying to show here is that now we sort of collaborated codecs we have some of the power tool power user tools like computer use.
50:31How can we like sitch this all together and let you know codeex work on its own?
50:35I mean for exposition purpose here we are like prompting and like being more methodical but in reality you can just like let codeex work on it like with thread handoffs.
50:45In practice, I could have all my demos in one thread and just ask that thread to segregate to separate it and send it and delegate it to the various threads.
50:53So here's one quick example of a automation.
50:57So right now you notice like on my uh screen over here that these like blue orbs.
51:02So this is just showing that it's a remote host and maybe you want to create one.
51:06So we have a nice plugin here with the digital ocean plugin which we just uh released last week and then you saw do it.
51:13You can just try it in chat.
51:14I'm just going to like fire it off as well.
51:17And what it's going to do is going to like provision this infrastructure.
51:20It's going to take some time.
51:21And I'm not going to like stand here and look for for it to provision this infrastructure.
51:25Uh and Codex is not going to like just keep um like waiting.
51:29What it's going to do is create a heartbeat automation.
51:31So it's going to like monitor and so forth.
51:33So it says yes.
51:34And let's just show in the spirit of this being a cooking show an example that was already done.
51:40So here provision a digital ocean droplet.
51:42Okay.
51:43Yes.
51:43And then it created the automation here.
51:46I set a codeex heartbeat to resume this track in about 5 minutes.
51:50So it's going to keep tracking every 5 minutes.
51:52Is it ready?
51:53Is it ready?
51:54Is it ready?
51:54Is it ready?
51:55And once it's ready, it will take the next step.
51:57So in this case, it did it.
51:58So okay, the droplet's ready.
52:00And it even gave a nice link.
52:02So if you click this link, it's a deep link.
52:04It's a link like a deep link to the settings page within the Codex app which automatically sets up the SSH instance for you.
52:11So you just need to press it and then it'll preconfigure everything.
52:14It knows where it will create the SSH key for you if necessary and so forth.
52:18So that's nice and you can extend this to other parts of like say the software development life cycle.
52:23Uh maybe you're building something you need to deploy.
52:25You have a pipeline that triggers event based and it takes time.
52:28So you can get codeex to like run every 5 minutes, every 10 minutes check is it ready, is it ready, is it ready and maybe depending on the event then it sort of spawns different tasks and so forth.
52:37So that's a heartbeat automation.
52:39It runs within the thread.
52:40But we can also do automations that sort of spawn new threads.
52:44Uh so one example of this so right now I have actually the Slack channel.
52:51So see if it's up.
52:54So remember the Mac OS translation app.
52:57So in the last say 45 minutes we had a bunch of users all called Gabriela and they've been using it.
53:06They've been giving a lot of feedback on the translation app.
53:09for example, oh the screen feels focused.
53:12I can tell where to start, so it's positive.
53:14Is there a way to do a quick sound check before the audience sees caption?
53:17So, so there's a lot of feedback.
53:18People are giving feedback.
53:20And I want to like, you know, give this feedback the attention it deserves.
53:24So, I want to create an automation to do that.
53:27So, just make sure we are in I'm just going to close this since we don't need it already.
53:32Over here, we create a new thread.
53:34And then I'm just going to like do a appshot.
53:40Um, every half an hour, can you just like look at the comments and uh suggestions in this channel and reply to them?
53:47Uh, I think there'll be like a few categories.
53:49There's one category where it's like compliments.
53:52There's one category where it's a bug report.
53:54There's one category where it's a feature request.
53:57Uh, so forth.
53:58Let me just like fire that off.
54:00So, it's going to create an automation.
54:01And I I just sent it, but I realized, oh, wait.
54:04Um, if it's a feature request or a bug fix, I want to do something different.
54:07So, I'm going to like steer this.
54:11Oh.
54:11Um, if it's a uh bug fix or feature request, can you also then like spin a new thread in a work tree and actually address that and then open a PR and then like request Dom to review it and then so forth and ask Dom to like approve it within like 24 hours.
54:26If not, just keep reminding him.
54:28So yeah, it's it's it's a comical example, but here it's where I'm try I what I was first doing was steering.
54:34So I'm going to I don't have to wait for codeex to finish.
54:37I can sort of steer it midway.
54:39And what I'm trying to do is in addition to all this feedback and classifying it if it's a it's a bug report, if it's something that uh we can fix.
54:47Codex would then sort of spawn a thread.
54:48So we're putting all these primitives together, the automation, the plug-in which can can read slack.
54:53The fact that one codex thread can communicate with other codex threads and even create work tree threads and so forth.
54:59So imagine then you wake up and this is a bit like that software factory where all the PRs are ready then you can even add additional comp layers to it where it's all this review you
55:08build a test version you send it to someone they test it and then they give the thumbs up then it's good to go that feedback gets fed back to the PR then the PR reviewer sees all this holistically.
55:17So yeah, there's a lot of it you can chain together to build that automation.
55:21So then every half an hour it would do this and it will find DOM as well in the uh channel.
55:29So you we were talking earlier about how we've actually been running a automation in the background and let's find that.
55:35So let's see here it is.
55:39So earlier while everyone was coming into the room I was asking Codex can you run an automation every five minutes taking a screenshot of what's happening on screen the Google Chrome
55:49uh Google drive presentation we only have 16 minutes and I'm asking Codex to estimate to what extent are we going to overrun so right now it's at checkpoint 13 it's seeing if the demo is there it's flashing 12 uh 10 12 and so forth so Codex is doing its own assessment of like to what extent are we
56:05going to finish on time so it's a bit of a meta example of how you can use automations and chain together other tools like in this case the Chrome extension.
56:12It looks like it's telling us we might be a little bit behind the ball here.
56:15Yeah, it's it's a you know it is what it is.
56:17So automations have been really useful in my personal life to extent really like getting a lot of things done.
56:23Uh some automations that may be useful too.
56:26Uh one automation you can run maybe every Friday you ask Codex to look at all your conversation threads and see are there skills that you could improve?
56:33Are there like things you could add to the agents markdown?
56:35Or are there skills you could just remove?
56:38maybe skills that you haven't been using.
56:40A second automation or approach that may be useful.
56:42Maybe you're using automations to draft email replies.
56:45You can have one automation that drafts the email reply, but you can run a second automation that sort of cross validates against the eventual reply you sent because that is the ground truth.
56:55Then it updates like a markdown file with the best practices, tips and so forth.
56:59And in that regard, the email drafting automation can get better over time.
57:04So, um I'll let uh Charlie just recap what automations are and share a bit more about the app server as well.
57:10Our final step.
57:14Cool.
57:15Uh so I think yeah, automations uh there's two types two types of automations, right?
57:19Um I think the more powerful one these days tends to be the inapp or inthread heartbeat automations.
57:24And so those just live in a single thread.
57:26They run on a timer.
57:27Um they you know keep going over and over until they're they're done.
57:30Um, these I find like have just almost unlimited use cases.
57:34Uh, me person, Gabe shared some of his.
57:36I think me personally, um, I feel like I'm drowning in Slack and email and linear notifications every day.
57:41Uh, so I have an automation that just sort of, uh, goes through and checks the latest updates from, you know, all the sources that might need my attention.
57:48I have a local Obsidian uh, vault and so it's just constantly updating my daily notes.
57:53It's constantly updating my uh, notes on different work streams and it's keeping a lot of context there.
57:58And that way I can just pull from that later if I say, "Hey, you know, what's going on with the latest uh AI engineer world's fair talk?"
58:03Like, you know, how far are we on the demos, how far are we on the notes?
58:06And then I can just Codex can just quickly tell me, here's what's happened since the last time you checked in.
58:11Uh, but automations can also live in a new thread.
58:13If you set them up in the Codex app, uh, under scheduled, the scheduled page, um, you can have them spin out in their own thread as well.
58:20Um, and then that's useful if you just want to say, "Look, every week I want you to do a recap of what all of my direct reports have been doing, or I want you to look at all of the analytics from this uh system, and then we're just going to do a one-time look together, and then I'm
58:32going to archive the thread and not worry about it again."
58:37Uh, I think last but not least, we've got embedding codecs anywhere.
58:41Um, and so I think we're for this one, uh, I want to talk a little bit about the app server protocol, right?
58:47Which is, um, the core protocol that underpins the Codex app, the Codex CLI, uh, the Codex VS code extension.
58:54Um, the app server is basically, uh, what allows all of these services to talk with the main codeex harness, right?
59:02Um, and it's integrating all of the tool calls and the plugins and the context compaction that you know and love.
59:07Um we have you know built a lot of the components of the Codex ecosystem to be open source because uh we really want to encourage developers to build wherever you are and for whatever use case that that you have.
59:19And so to that extent um you know if you're not aware you can embed codecs into your product using app server.
59:25Uh I think this brings a lot of the uh power of codecs directly to you.
59:30Um, and it brings the ability for your users to uh use their chat GBT subscription and perhaps more importantly use their chat GBT token budget uh to power the products that you're building.
59:41Um, and like I said, it has all the same underlying features that we've been demoing.
59:44You know, compaction, steering, the ones that need to live on your computer like computer use are are, you know, not going to be there.
59:49But for everything that exists in the core, uh, you know, agent harness, we've got that in the app server.
59:58So as a quick demo of that, so there was a fourth longrunning task that I kicked off last night.
60:03It was actually to build my own user interface for codeex with the app server called Retroex.
60:09So it's been working for a while.
60:11I think there are different tasks I worked with it.
60:13And this is sort of the work in progress retroex.
60:16It's using the app server.
60:17Uh let's just okay zoom in then optimize it for the stage.
60:20But yeah, you can like choose the model 5.5.4 mini the reasoning effort.
60:24So I told like oh make it a bit funky when you change the reasoning effort.
60:27So at low there's like a small glitter at medium the color changes you think high and extra high it's like woo a bit more.
60:35So you could just like simply say like uh hello and let's see this is in retroex going to send hello and then allow access.
60:44It's going to run and it's going to reply back hello.
60:47And what's nice and to just prove that it's all running on the same app server is that if we go to the Codex app in the same project uh retroex
60:55you should see that over here let's see yeah it should we can change the threads I believe but yeah that's an example of how you can build it's it's a trivial example of building a funky user interface but let's say you want to build agents on the cloud you want to manage different agent providers
61:17uh you're considering in a large organization your different sort of providers you're using and you want to build a unified layer the app server is a useful way for you to integrate it into your own platforms
61:27and because it's open source if you go to you get clone openai/codex you can just ask codeex about codeex you could just say like uh hey can you tell me more about the codeex app server
61:38and then you can fire it off and you can start building your own applications or integrations from there immediately so that's the app server And we have a nice blog post talking about it and there have been more engineering blog posts about like our Windows sandbox
61:53first of its kind how we built the agent loop amazing stuff and it's a good reference point also like if you're AI engineer seeing the codeex repo it's a great source and we'll have uh further talks in the subsequent days about this.
62:05So I'll let Charlie share some closing thoughts after seeing this entire cooking show.
62:13Uh, thanks Gabe and thanks everybody again for being here.
62:16I think um hopefully you've learned a bit today about how you can use Codex to kind of level up your workflows.
62:20Some of the co closing thoughts that I want to end on here are, you know, questions that I repeatedly ask myself as I'm trying to get to the frontier of what Codex can do, right?
62:29Um they start pretty simple just by asking, have I tried asking Codex to do it?
62:34Right?
62:34Um I think like one of the things around the office that we say uh more frequently than ever now is have you asked Codex?
62:40Um, but I think as you evolve that workflow, right, you want to go from just asking in individual turns to building loops, right?
62:47We've all heard about loop maxing.
62:48That is the new thing.
62:49Um, and I think we want to think about how does that how does that work, right?
62:52Like what is it that the loop needs to be checking at each turn, right?
62:56Um, as you go down this path, there's going to be additional questions like, okay, you know, as it's progressing, how how like what blocked Codeex from actually getting to the answer, right?
63:06Um, was there context that it needed?
63:08Were there permissions that it needed?
63:10Are there ways to build that context into repeatable tools like skills or plugins that Codex can use the next time it runs into this problem?
63:18Um, how do I start scaling kind of from one agent to many?
63:21Right?
63:22Like as you think about the work, is there ways that you could be prompting Codex to implicitly delegate the structure of the things that you are building?
63:28Um, or to start orchestrating across multiple threads to have different Codex instances communicating with each other.
63:34And then ultimately uh you know if you take a step back and ask the question okay why does this process need a human at all right I think like
63:41as we've shown you know one thing that you might assume uh when building with codeex or when building with the open AI API is that a human needs to be involved to like go to the platform page and go get an API key and then paste it into the environment
63:54right I think like uh as we've shown with the developers plugin that is not the case anymore the plug-in itself can just go fetch the API key and set it up locally for you um and ultimately you know we do still want humans in the loop like where they need to be but uh for everything that's not
64:08that like how do we figure out how to take the human out of the process to unblock ourselves and how do we figure out how to build the machine that builds the machine right I think like it is one thing to
64:18sit with a coding agent and build a single piece of software is another thing to start zooming out and say actually I want to build a software factory right I want to build a system that um that then builds individual pieces of software on an assembly line as we Um, and so a lot of these things are
64:35kind of how I start to get from, um, just an individual ask to something significantly bigger.
64:43Cool.
64:44And I think with that, um, we are
64:46a recap of
64:47Yeah.
64:47Yeah.
64:49Good.
64:49So, um, thanks so much for joining us today on a 9:00 a.m. on a Monday.
64:54It's the I hope it's a great way to kick off your, uh, AI engineer welfare.
64:59And that's not it.
65:00So, tomorrow we have our opening keynote.
65:02It's going to be fun.
65:03Do check it out.
65:05We also have Jason on the developer experience team giving a talk about how he gets the most out of Codeex.
65:10It's a nice foil to today's presentation about how he's using on a day-to-day basis.
65:16And then he has a workshop after that where he goes through step by step like setting up the plugins, doing app shots and so forth.
65:22So today's session is more uh like a whirlwind tour.
65:26This workshop will go through the step by steps of how to get to set yourself up for success with codecs.
65:32We then have a session on uh from the harness about the harness and then sessions about voice agents.
65:39Charlie will be talking about that as well.
65:41Uh Dom will be doing a session on building on the Codex harness and then another session explaining behind the Codex harness.
65:49So the first one is you build on top of it.
65:51Then the second one is explain the harness.
65:54Then lastly a session about LLM inference in production and so how we sort of do that at OpenAI.
66:01So thank you for joining us and that's the end of the first segment of today's workshop where we sort of give that whirlwind tour of everything codex and hopefully you have something fun to cope with.
66:12So it's time to build and to help you with that for today's session.
66:15Uh first things first, if you've not downloaded downloaded the Codex app, do give it a download.
66:20And for credits, so we are giving 100 USD in Codex credits and 100 USD in API credits.
66:27So we'll leave leave this QR code up there's their phones up and take a photo.
66:31Yeah, I could like appshot this and say, "All right, send this via email to everyone who attended, but we don't have your emails."
66:37Yeah.
66:38Cool.
66:39And I would also mention uh please stick around uh to the end of the build session um if you can.
66:45We brought uh swag for everybody.
66:48So at the end of the 2our mark um on your way out you can pick up a sweatshirt I believe.
66:52We love to see what you're building as well.
66:54Yeah.
66:55Yeah.
66:55Well, we will be here.
66:56We will be here in the audience.
66:57We've got uh opening eye staff here as well um to help unblock you or to answer any questions about uh using codecs.
67:03Thank you.