WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=cJ0EOzey--o NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/cJ0EOzey--o.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.440 --> 00:00:13.120 Excellent. s2 00:00:13.120 --> 00:00:17.760 I will say that um I might speed run through this. s3 00:00:17.760 --> 00:00:21.200 Feel free if you don't disag- agree with something to yell out. s4 00:00:21.200 --> 00:00:24.480 It's way more fun for me if things get interactive. s5 00:00:24.480 --> 00:00:27.120 Um otherwise, I will go through this. s6 00:00:27.120 --> 00:00:32.640 Uh first, can I have like a vague show of hands of who knows what RLHF is? s7 00:00:32.640 --> 00:00:33.680 Oh, excellent. s8 00:00:33.680 --> 00:00:37.560 I might be able to skip through that part quickly and get into the interactive stuff. s9 00:00:37.560 --> 00:00:39.320 So, my name's Tiago Almeida. s10 00:00:39.320 --> 00:00:41.400 I'm talking about what's next after RLHF. s11 00:00:41.400 --> 00:00:48.120 More accurately, I think this should be called what's next after the chat GPT era that I think we're all in. s12 00:00:48.120 --> 00:00:52.160 And my hint for you guys is it is not the Claude code era. s13 00:00:52.160 --> 00:00:56.960 I will justify this later on, but I actually believe them to be part of the same era. s14 00:00:56.960 --> 00:00:58.360 Why should you listen to me? s15 00:00:58.360 --> 00:01:04.199 I was co-authored to what what is basically OpenAI's greatest hits, at least published hits. s16 00:01:04.199 --> 00:01:09.520 Co-authored to GPT-4, chat GPT, RLHF {slash} instruct GPT. s17 00:01:09.520 --> 00:01:13.880 Um the team I was part of basically invented post-training as a concept. s18 00:01:13.880 --> 00:01:17.480 So, um very qualified on a lot of this stuff. s19 00:01:17.480 --> 00:01:25.960 But what makes me somewhat unique here is that I'm one of the few people at OpenAI who actually hates on chat GPT. s20 00:01:25.960 --> 00:01:27.120 Uh thank you. s21 00:01:27.120 --> 00:01:30.320 Uh I don't hate chat GPT as a product, to be clear. s22 00:01:30.320 --> 00:01:36.480 I think chat GPT is a world-changing product that will probably stay with us for the rest of time unless something better comes up. s23 00:01:36.480 --> 00:01:49.040 But I also acknowledge its limitations and I I I think a lot of what's happened in the state of the field can be traced back to minor decisions we made in making the algorithms behind chat GPT. s24 00:01:49.040 --> 00:01:56.200 Um I feel like the question that's relevant to everyone in AI right now is what's actually going on. s25 00:01:56.200 --> 00:02:06.360 Um there's a lot of like differing opinions, and I think it's really useful to like map out the spectrum and figure out how can smart people have like such different opinions. s26 00:02:06.360 --> 00:02:08.479 There's cult one. s27 00:02:08.479 --> 00:02:12.400 Um, AI is not just going well, it's going insanely well. s28 00:02:12.400 --> 00:02:20.440 Every single benchmark we surpass human level, and as far as we can measure, we are continuously surpassing human performance. s29 00:02:20.440 --> 00:02:25.040 You know, like basically every new benchmark, and it's only getting faster and accelerating. s30 00:02:25.040 --> 00:02:28.240 You have uh, you know, every Can I see my mouse? s31 00:02:28.240 --> 00:02:28.959 Excellent. s32 00:02:28.959 --> 00:02:39.320 Basically every like NLP benchmark is getting crushed, and not only that, allegedly the time that LLMs can operate autonomously is growing exponentially. s33 00:02:39.320 --> 00:02:45.800 On the other hand, you have AI is not just going poorly, it's going like insanely poorly. s34 00:02:45.800 --> 00:02:56.840 AI is a bubble, it's basically generating no value, it's just circular financing deals, etc., etc. And, you know, if AI is so great, why is why is everything just like a chat app right now? s35 00:02:56.840 --> 00:02:58.560 Or like a cloud go thing? s36 00:02:58.560 --> 00:03:06.800 Um, and a a lot of the people have actually kind of given up on what was the old guard's terminology of a transformative AI revolution. s37 00:03:06.800 --> 00:03:08.959 People aren't really talking about that anymore. s38 00:03:08.959 --> 00:03:12.440 They're talking about it being like massively valuable like B2B SaaS. s39 00:03:12.440 --> 00:03:18.680 So, the only thing that everyone agrees on is like there's just a these extreme points of view and like nothing in between. s40 00:03:18.680 --> 00:03:22.800 And everyone basically thinks AI is insane, but like for different reasons. s41 00:03:22.800 --> 00:03:26.800 And what I would want to talk about is what is the sane view of AI? s42 00:03:26.800 --> 00:03:30.800 Let's take all the evidence of like cult one, it's going super well. s43 00:03:30.800 --> 00:03:33.560 Take all the evidence of cult two, it's going super poorly. s44 00:03:33.560 --> 00:03:39.720 Like, uh, you know, map them out and try to explain what what what explains that divide. s45 00:03:39.720 --> 00:03:51.800 Like, what is the simplest possible explanation of why some things are too good to be true, and some things are not just bad, they are so bad that we would still employ human workers to do like, you know, like kind of like dumb tasks. s46 00:03:51.800 --> 00:03:54.120 Um, no offense to any of them. s47 00:03:54.120 --> 00:03:59.120 A lot of these tasks on the right seem way, way, way easier than the stuff on the left. s48 00:03:59.120 --> 00:04:07.959 Like, how can we be solving like, you know, unsolved math problems, but still customer service requires like humans in the loop in order to actually like make decisions? s49 00:04:07.959 --> 00:04:10.880 This I think is like a kind of like a wild state of affairs. s50 00:04:10.880 --> 00:04:20.000 And in my opinion, anyone who works adjacent to AI should have an answer to this because this is like the evidence in the field right now. s51 00:04:20.000 --> 00:04:29.720 Um, I would normally pause and ask people if they want to like yell out their thoughts in this, but uh, that I don't think we have time for that and I've been told to not take Q&A until after. s52 00:04:29.720 --> 00:04:34.680 Um, but I'll just give you my answer to this, which is, in my opinion, the simplest explanation. s53 00:04:34.680 --> 00:04:40.680 All the stuff on the left is not just a task that happens to have a human in the loop. s54 00:04:40.680 --> 00:04:44.840 In the left, the task The goal of it is to please the human in the loop. s55 00:04:44.840 --> 00:04:47.880 These tasks are intrinsically human in the loop tasks. s56 00:04:47.880 --> 00:04:51.400 The like Claude code's job is not to just make code work. s57 00:04:51.400 --> 00:04:54.840 Um, the the the the way it converses would be totally different. s58 00:04:54.840 --> 00:04:57.040 The goal is to please the human in it. s59 00:04:57.040 --> 00:05:03.560 And on the other side, all of these tasks that seem way more basic, the goal is to not have remove the human loop. s60 00:05:03.560 --> 00:05:11.760 Ideally, it would be running in the background in a server that you never even look at and ideally it eventually becomes like legacy software that you don't really worry about. s61 00:05:11.760 --> 00:05:15.560 So, and this is the divide between assistance and automation. s62 00:05:15.560 --> 00:05:25.919 Um, lesson one for my talk is that today's AI, everything inherited from our LHF, is incredible at the human in the loop stuff, but not for automation tasks. s63 00:05:25.919 --> 00:05:26.720 Tasks. s64 00:05:26.720 --> 00:05:36.280 This is a longer side, but the lesson basically every business has learned is do not use AI for decisions with stakes to your business. s65 00:05:36.280 --> 00:05:41.600 Um, a common pattern is make sure that all of the costs are to the user and not to your business. s66 00:05:41.600 --> 00:05:49.200 So, um, it's oh, totally okay to throw the user at infinite docs in customer service, but it is not okay to make it make expensive decisions. s67 00:05:49.200 --> 00:05:52.520 Horrible pattern, but that is the state of AI right now. s68 00:05:52.520 --> 00:05:57.680 Uh I can I can blitz through the what is RLHF part cuz you all seem to know what it what it is. s69 00:05:57.680 --> 00:06:02.440 Um it's the algorithm behind not just ChatGPT, but basically every LLM today. s70 00:06:02.440 --> 00:06:08.960 As far as I can tell by usage, 100% roughly of LLMs are trained with RLHF. s71 00:06:08.960 --> 00:06:15.160 And we have this we as in we the OpenAI team had this great blog post on how it worked. s72 00:06:15.160 --> 00:06:19.120 Um I will not get into that because you all know it, and this is super boring. s73 00:06:19.120 --> 00:06:25.600 Um the summary of this is it is just collect human preferences, optimize for human preferences. s74 00:06:25.600 --> 00:06:32.800 Um and if you want to see like an annotated version of this, you can see which parts are collecting human preferences, which ones are optimizing for them. s75 00:06:32.800 --> 00:06:33.004 And s76 00:06:33.004 --> 00:06:33.480 [snorts] s77 00:06:33.480 --> 00:06:42.440 this, I think, provides a really clear answer to everyone in the field asking, "Why do all LLMs require a human in the loop?" s78 00:06:42.440 --> 00:06:45.640 The And the simple answer is we literally put them in the loop. s79 00:06:45.640 --> 00:06:48.200 The goal of the loop is to optimize for human preference. s80 00:06:48.200 --> 00:06:50.720 It is not to run software autonomously. s81 00:06:50.720 --> 00:06:53.680 It's kind of super obvious. s82 00:06:53.680 --> 00:06:56.440 Thank you, my man at the back. s83 00:06:56.440 --> 00:06:57.400 The Yeah. s84 00:06:57.400 --> 00:06:59.800 I I I love that you're laughing at this. s85 00:06:59.800 --> 00:07:04.919 Um and because of that, overpromising is a feature. s86 00:07:04.919 --> 00:07:06.200 This is by design. s87 00:07:06.200 --> 00:07:08.440 This is an old meta study. s88 00:07:08.440 --> 00:07:24.520 Um and the the numbers probably have changed, but by construction, every RLHF model will always have a big difference between human preference and results, even if the results are good, because the main objective you're optimizing for is for human preference. s89 00:07:24.520 --> 00:07:27.600 This is just like natural to how LLMs work. s90 00:07:27.600 --> 00:07:38.440 Um I love this tweet of um uh sending ChatGPT an audio file of fart sound effects and asking like what What do you think of the music I made? s91 00:07:38.440 --> 00:07:40.640 Here's a straight honest reaction. s92 00:07:40.640 --> 00:07:44.480 It's a very eerie vibe atmosphere piece. s93 00:07:44.480 --> 00:07:45.840 And this is just how RLHF works. s94 00:07:45.840 --> 00:07:51.760 If it doesn't know, it will err on the side of doing what it thinks is best for human preference. s95 00:07:51.760 --> 00:07:58.880 And this makes total sense if you are a user in the loop because like the end game for all RLHF models is optimizing for engagement. s96 00:07:58.880 --> 00:08:09.840 But what you really want if you want automation is for it to just like not give a about the humans and just do the task correctly in a calibrated way. s97 00:08:09.840 --> 00:08:17.760 Um Lesson number two is that today's AI was designed for assistance through optimizing for human preference. s98 00:08:17.760 --> 00:08:19.640 This is like it's like in the name. s99 00:08:19.640 --> 00:08:22.480 This is not like a controversial take. s100 00:08:22.480 --> 00:08:37.400 And the consequences are maybe more controversial, but it's like very obvious if you think about what we really are optimizing for, which is no matter how wrong the models are, they will look right because of the asymmetry within the reward model in RLHF. s101 00:08:37.400 --> 00:08:46.960 Um and this is where a lot of like the dilemma in the field stems from because people really want automation to happen. s102 00:08:47.280 --> 00:08:47.680 Cool. s103 00:08:47.680 --> 00:08:49.400 So, back to the original question. s104 00:08:49.400 --> 00:08:52.360 I'm over halfway done with the talk and I haven't even answered it. s105 00:08:52.360 --> 00:08:54.320 I was just talking about what's RLHF. s106 00:08:54.320 --> 00:09:01.240 But this was a framing to talk about what RLHF is to talk about what's next. s107 00:09:01.240 --> 00:09:09.520 And I would say the real question is what's next after AI's assistance era, which I think that we are like very firmly in right now. s108 00:09:09.520 --> 00:09:12.680 And back to the original clue of why it's not Claude code. s109 00:09:12.680 --> 00:09:18.560 It's actually a super fun nuanced discussion, but it's not Claude code because Claude code is still part of that assistance era. s110 00:09:18.560 --> 00:09:27.760 Claude code is still RLHF and it'll it would look very very different if it was purely This is a little advanced, but if it was purely RLVR, it would look very very different. s111 00:09:27.760 --> 00:09:34.800 And this is why you get like this dilemma with models where sometimes it gets really good at agentic stuff, but it stops following what you actually want. s112 00:09:34.800 --> 00:09:43.400 This This like the trade-off in optimization space that keeps dancing, but both of these trade-offs in optimization space do not add to the automation component. s113 00:09:43.400 --> 00:09:51.000 And like that leads to what I think the the logical answer of what's next after assistance is real automation. s114 00:09:51.000 --> 00:09:57.840 Um to talk about a little bit about the automation and how that would work, I want to talk about software. s115 00:09:57.840 --> 00:10:04.080 Um maybe this is a little bit philosophical for you guys, but I think it's when it clicks and hopefully it clicks if I do a good job. s116 00:10:04.080 --> 00:10:08.880 It it I I hopefully it'll be like really clear, which is I'm a lover of software. s117 00:10:08.880 --> 00:10:11.080 I assume everyone here loves software. s118 00:10:11.080 --> 00:10:13.280 Software is like super valuable. s119 00:10:13.280 --> 00:10:14.800 See all the SaaS. s120 00:10:14.800 --> 00:10:23.720 And kind of like the craziest part of software in my opinion is that all of the SaaS basically has not changed since 2019. s121 00:10:23.720 --> 00:10:35.960 Like SaaS is not really changed in the LLM era, except sometimes a chatbot is like latched on, which is like kind of insane if you think about like the progress made in AI, but is actually very predictable s122 00:10:35.960 --> 00:10:38.960 when you think that AI is assistance native, right? s123 00:10:38.960 --> 00:10:40.600 Like AI is made for assistance. s124 00:10:40.600 --> 00:10:42.040 What can you do in SaaS? s125 00:10:42.040 --> 00:10:44.320 Just provide an assistant on the side. s126 00:10:44.320 --> 00:10:49.600 And this is not what early AI pioneers used to think would happen. s127 00:10:49.600 --> 00:10:58.600 Like when you see like the early wording in opening eyes uh charter, it's about like doing like tons of work, not about like making profit or anything like that. s128 00:10:58.600 --> 00:11:06.040 And we used to think that software would get a lot smarter, not just cheaper to write, which is kind of the direction we're going down right now. s129 00:11:06.040 --> 00:11:09.880 And I actually really like this phrasing from Garry Tan. s130 00:11:09.880 --> 00:11:14.839 Um I think he means this as a compliment to what's going on right now. s131 00:11:14.839 --> 00:11:21.160 We're entering the golden age of just-in-time software, but I actually think that this is like a like a double-edged sword. s132 00:11:21.160 --> 00:11:24.960 Like I don't just want just-in-time software, which is cool. s133 00:11:24.960 --> 00:11:27.920 I I love cloud code, to be clear, just like I love ChatGPT. s134 00:11:27.920 --> 00:11:28.960 I would keep using it. s135 00:11:28.960 --> 00:11:31.760 But like what I want is smarter software. s136 00:11:31.760 --> 00:11:36.240 Why can't like B2B Why can't software just be more expressive? s137 00:11:36.240 --> 00:11:41.160 Like why are the like the building blocks of software actually still the same? s138 00:11:41.160 --> 00:11:46.640 And um I think this is a question that the whole AI industry should ask itself. s139 00:11:46.640 --> 00:11:54.720 And basically every time you're thinking about we want to do automation, it is not about like, you know, an amalgamation of like automating a person's work. s140 00:11:54.720 --> 00:11:57.400 It's about like, "Hey, there's this extremely rote work. s141 00:11:57.400 --> 00:12:01.880 It's so simple that we can like communicate to someone else that this thing should be done." s142 00:12:01.880 --> 00:12:06.720 And ideally it's like it it's so basic that it could be done repeatedly for basically free. s143 00:12:06.720 --> 00:12:08.080 Um or it could be done by computers. s144 00:12:08.080 --> 00:12:10.440 And that's really not happening right now. s145 00:12:10.440 --> 00:12:13.280 What we're doing is we're just automating the writing of the software. s146 00:12:13.280 --> 00:12:15.240 But then it its expressibility is the same. s147 00:12:15.240 --> 00:12:19.200 And that's That to me is like tragic in the state of the world. s148 00:12:19.200 --> 00:12:22.680 Um Cool. s149 00:12:23.200 --> 00:12:25.960 Oh, lesson three. s150 00:12:26.080 --> 00:12:29.680 This is something that I believe strongly in. s151 00:12:29.680 --> 00:12:38.480 I believe that like eventually the field will write the I wouldn't say Arlatech is a wrong, but it was like a weird detour and one that we didn't expect. s152 00:12:38.480 --> 00:12:41.520 Tomorrow's AI, I believe, will be for automation. s153 00:12:41.520 --> 00:12:45.360 And we will eventually have a world with smarter software. s154 00:12:45.360 --> 00:12:52.760 Like there will start to be actual work that is automated, which I, you know, right now it's a rounding error despite LLM's intelligence. s155 00:12:52.760 --> 00:12:57.120 And that is what we are working on at TypeSafe. s156 00:12:57.120 --> 00:13:00.360 We are still kind of stealthy. s157 00:13:00.360 --> 00:13:03.760 Like I'm willing to give these talks, but these are like some of the early ones. s158 00:13:03.760 --> 00:13:11.360 Um Our core question is what if the AI stack was redesigned for reliability and automation? s159 00:13:11.360 --> 00:13:14.000 Like how would that all change? s160 00:13:14.000 --> 00:13:15.400 What what would you do? s161 00:13:15.400 --> 00:13:23.800 And actually there's a lot It's a It's a very interesting fork in the road for what's go you know, like from basically every LLM that's built today. s162 00:13:23.800 --> 00:13:28.040 And I think it's one of the most satisfying things I've worked on, and I've worked on some pretty cool stuff. s163 00:13:28.040 --> 00:13:29.520 We are releasing soon. s164 00:13:29.520 --> 00:13:39.480 So, um if you want to work with us or you want to like you know, be the first one of the first to build smart software, please sign up on either our mating mailing list or careers page. s165 00:13:39.480 --> 00:13:41.160 And I am trying to start a Twitter. s166 00:13:41.160 --> 00:13:44.440 So, follow me and I will post really spicy things. s167 00:13:44.440 --> 00:13:48.640 I actually will post something later today that I guarantee will be very spicy. s168 00:13:48.640 --> 00:13:53.360 Uh the hint is that the original scaling laws were incorrect. s169 00:13:53.600 --> 00:13:54.160 Cool. s170 00:13:54.160 --> 00:13:57.080 Um that uh that's it for my prepared stuff. s171 00:13:57.080 --> 00:14:00.640 I would love Do I have time for for people yelling out questions? s172 00:14:00.640 --> 00:14:03.960 I would love questions, feedback, disagreements, strong stuff. s173 00:14:03.960 --> 00:14:04.800 I can repeat the question. s174 00:14:04.800 --> 00:14:06.600 You don't have to worry about the mic. s175 00:14:06.600 --> 00:14:08.520 Hell yeah. s176 00:14:08.520 --> 00:14:09.320 Uh cool. s177 00:14:09.320 --> 00:14:14.600 Uh the question was roughly what if you trained like a classifier head with pre-training as well? s178 00:14:14.600 --> 00:14:17.960 Uh roughly uh like Yoshua Bengio is suggesting. s179 00:14:17.960 --> 00:14:22.440 Um I will say that that's complicated. s180 00:14:22.440 --> 00:14:26.000 And I actually think I don't have the time to answer that particular question. s181 00:14:26.000 --> 00:14:28.240 I will give like my simplified view on this. s182 00:14:28.240 --> 00:14:33.240 And it the answer is I actually don't think that pre-training is the problem. s183 00:14:33.240 --> 00:14:36.680 I think pre-training is uh phenomenal. s184 00:14:36.680 --> 00:14:43.560 Like the fact that we compress the knowledge of the internet into like this core of intelligence that then can be utilized is incredible. s185 00:14:43.560 --> 00:14:47.560 And the pre-trained models are incredibly intelligent. s186 00:14:47.560 --> 00:14:50.920 Uh and I believe that the problem is like how we unearth it. s187 00:14:50.920 --> 00:14:56.880 And hallucination [clears throat] to me is intrinsic to um optimizing for human preference. s188 00:14:56.880 --> 00:15:00.640 Like there's an asymmetry in the reward model kind of like a GANs have. s189 00:15:00.640 --> 00:15:02.839 Oh, I really should not get This is a very advanced topic. s190 00:15:02.839 --> 00:15:18.200 But there's an asymmetry in the reward model like what GANs have that allow for um that encourage the models to drop modes and be confident because it's very easy to see when the model is not confident and to punish that from a reward model perspective. s191 00:15:18.200 --> 00:15:24.120 It's very complicated, but uh I'm happy to chat afterwards if you want to jam. s192 00:15:25.960 --> 00:15:27.080 Cool. s193 00:15:27.080 --> 00:15:29.080 Oops. s194 00:15:29.520 --> 00:15:34.880 Um I have other slides from other talks as well that I could go into more about that. s195 00:15:34.880 --> 00:15:36.000 I have a minute left. s196 00:15:36.000 --> 00:15:37.880 Hell yeah. s197 00:15:37.880 --> 00:15:40.000 Say it again. s198 00:15:43.080 --> 00:15:44.920 It is definitely not RLVR. s199 00:15:44.920 --> 00:15:46.480 So, it is a new thing. s200 00:15:46.480 --> 00:15:53.680 Every single optimization stack I will actually go into an old presentation that I have because I think this is super important. s201 00:15:53.680 --> 00:16:00.240 Um in terms of like to me what the like Sutton's bitter lesson is that algorithms matter more than compute. s202 00:16:00.240 --> 00:16:03.240 This is true in games, but not true in reality. s203 00:16:03.240 --> 00:16:11.440 I actually think that the full stack is that data matters more than compute and doing the right task matters way more than data. s204 00:16:11.440 --> 00:16:19.800 And basically every single branch of LLM post-training if you want to call it has its own North Star of what it's optimizing for. s205 00:16:19.800 --> 00:16:22.400 So, RLHF is optimizing for human preference. s206 00:16:22.400 --> 00:16:36.040 RLVR is optimizing for like log error rates of pure correctness, but we are doing a third thing that is optimized for calibrated decision-making and like basically mainlining the intelligence of pre-trained models s207 00:16:36.040 --> 00:16:41.240 into like being actually useful for software, which I think is like quite different. s208 00:16:46.240 --> 00:16:48.800 Uh could you say that again? s209 00:16:52.880 --> 00:16:56.280 Uh they're asking if the the reward is injected through the whole process. s210 00:16:56.280 --> 00:17:05.520 I will actually say that even the shape of the API is different because the shape of the API for RLHF is different from RLVR, which is different from what we are doing. s211 00:17:05.520 --> 00:17:12.319 So, we are like thinking about it from scratch just like no one thought about instruction following before we made instruction following happen. s212 00:17:12.319 --> 00:17:21.400 Um usually when there's a big branch in new ways to post-train, like it it it just looks like totally alien, and then in hindsight becomes super obvious. s213 00:17:22.199 --> 00:17:23.319 Cool. s214 00:17:23.319 --> 00:17:28.160 I believe I'm overtime cuz this red thing is is beeping, but please find me afterwards. s215 00:17:28.160 --> 00:17:29.200 I love questions. s216 00:17:29.200 --> 00:17:31.040 I love the interactivity. s217 00:17:31.040 --> 00:17:35.480 Um and uh follow me on Twitter for spicy stuff. s218 00:17:35.480 --> 00:17:37.360 Heck, yeah. s219 00:17:37.360 --> 00:17:39.080 Oh, oh yeah, it's over here. s220 00:17:39.080 --> 00:17:40.680 Complete skeptic. s221 00:17:40.680 --> 00:17:44.200 Um it's it's on brand for me. s222 00:17:44.200 --> 00:17:44.960 Cool. s223 00:17:44.960 --> 00:17:45.360 Heck, yeah. s224 00:17:45.360 --> 00:17:47.600 Thank you. s225 00:18:01.417 --> 00:18:03.417 [music]