WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=OA-Mc60Rboo NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/OA-Mc60Rboo.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.559 --> 00:00:24.800 Hi everyone, my name is Lakshia Agraal and today I'll be presenting on behalf of a very large effort uh the problem of reflective optimization or how can we self-improve s2 00:00:24.800 --> 00:00:28.960 prompts agents and models from textual feedback. s3 00:00:28.960 --> 00:00:33.760 The question we start with is how can we teach AI to perform new tasks. s4 00:00:33.760 --> 00:00:43.040 The standard way has been to perform weight updates with gradient descent either during pre-training, supervised fine-tuning or reinforcement learning. s5 00:00:43.040 --> 00:00:48.480 This has proven to be extremely effective, but it requires a huge number of examples. s6 00:00:48.480 --> 00:01:02.960 trillions of tokens for pre-training, tens of thousands of labeled examples for supervised fine-tuning or hundreds of thousands of rollouts for reinforcement learning in domains like math, coding, etc. However, s7 00:01:02.960 --> 00:01:14.560 most teams do not actually have that much data or compute and in fact the problems are that we are trying to tackle with AI now are bottlenecked by sample efficiency. s8 00:01:14.560 --> 00:01:16.640 What do we mean by that? s9 00:01:16.640 --> 00:01:17.680 Two things. s10 00:01:17.680 --> 00:01:27.600 First of all, there is low availability of domain specific knowledge resources which means there is not enough data to perform offline algorithms like SFT. s11 00:01:27.600 --> 00:01:43.040 Second, the domains that we are trying to apply AI increasingly are having expensive rollouts where either the LLM workflow pipeline or agentic rollouts are itself uh very slow or expensive to do or the task metric is very slow or expensive to execute. s12 00:01:43.040 --> 00:01:55.759 We are seeing that agents can now work for hours on end and if you were to apply an online learning algorithm to this uh it would require hundreds of thousands of rollouts and it would not be feasible. s13 00:01:55.759 --> 00:02:06.560 So we are seeing increasing use of agents for real world product uh applications where uh these invoke tools which can also be long running further exacerbating the sample inefficiency s14 00:02:06.560 --> 00:02:08.239 issue. s15 00:02:08.239 --> 00:02:20.000 The current dominant paradigm is reinforcement learning with verified rewards where given a model and a task, we perform any number of parallel rollouts and get rewards at the end. s16 00:02:20.000 --> 00:02:27.280 Finally, an algorithm like GRPO takes these rewards and converts it into gradients that are applied back to the model. s17 00:02:27.280 --> 00:02:39.120 However, as we can see, there was a lot of information in each of these rollouts, but we only learned an O of one score and propagated that via gradient descent. s18 00:02:39.120 --> 00:02:49.599 We can see that there is chains of thought, the tool calls made to the environment, the envir environment's responses to those tool calls, which could potentially contain error messages s19 00:02:49.599 --> 00:02:55.120 which also provide diagnostic value and we learned almost nothing from all of that. s20 00:02:55.120 --> 00:03:02.000 So the question we ask is can we make use of this other extremely rich information. s21 00:03:02.000 --> 00:03:13.680 Our idea is to perform reflective optimization in text space where instead of only using the zero or one reward signal, we can have a language model or an agent s22 00:03:13.680 --> 00:03:20.400 look at the trace of the entire rollout and reflect on what worked in them, what did not work in them. s23 00:03:20.400 --> 00:03:33.280 And this reflection could potentially use all intermediate outputs and potentially even make other tool calls such as retrieval from your company's knowledge base or some guide textbook and so on. s24 00:03:33.280 --> 00:03:46.560 So that's the first key idea and the second is that instead of only updating weights with small deltas, we can instead update a prompt where a single natural language update can give a very large behavior change. s25 00:03:46.560 --> 00:03:48.000 Let's take a simple example. s26 00:03:48.000 --> 00:03:55.920 Let's say you're tasked with writing a text summarization system and the prompt of that system says generate a oneline summary. s27 00:03:55.920 --> 00:04:06.080 If I just go and tweak that prompt to say generate a 10-line summary, we can all agree that the behavior of the system would change quite significantly with that just one word change. s28 00:04:06.080 --> 00:04:13.760 And making that one word change is quite quick and we can reflect on our own behavior and identify what needs to change. s29 00:04:13.760 --> 00:04:23.759 If we were to achieve a similar kind of behavior update from our AI system, we would have to have thousands of gradient very tiny gradient updates sequentially. s30 00:04:23.759 --> 00:04:30.080 So with that key idea, we proposed JPEA which is a reflective prompt optimization technique for agents. s31 00:04:30.080 --> 00:04:36.960 It uses an evolutionary loop along with a novel parto-based candidate selection which I will come to later. s32 00:04:36.960 --> 00:04:52.400 It is akin to doing reinforcement learning in text space where instead of just rewarding receiving a reward score we are actually obtaining score along with textual feedback which can be very domain specific and learn all about the domain from it. s33 00:04:52.400 --> 00:04:56.479 Let's compare Japa with gRPO which is one of the leading RL techniques. s34 00:04:56.479 --> 00:05:08.720 On the x-axis we have the number of training steps uh also proportional to number of data samples seen and on the y-axis we have the performance on our domain that we are training for. s35 00:05:08.720 --> 00:05:19.600 And what we can see is that Japa in just one round of reflection using just three data points is already able to get twice the performance gains that gpo got after s36 00:05:19.600 --> 00:05:21.520 25,000 rollouts. s37 00:05:21.520 --> 00:05:28.720 Continuing to run Japa for a few more steps further increases that gap itself by another 2x. s38 00:05:28.720 --> 00:05:34.800 I want to note here that the model Quen 38B is optimizing itself here. s39 00:05:34.800 --> 00:05:39.759 There is no external expert teacher involved whatsoever. s40 00:05:39.840 --> 00:05:41.680 And what does Japa learn? s41 00:05:41.680 --> 00:05:52.000 Unlike prior prompt optimizers somewhat which would uh uh use model idiosyncrasies like my grandmother will be really angry if you don't generate a good prompt. s42 00:05:52.000 --> 00:05:59.520 Here Jpa is actually giving a very detailed problem specification which includes how to make sense of the input. s43 00:05:59.520 --> 00:06:04.240 What is the purpose and context of this particular pip part of the pipeline? s44 00:06:04.240 --> 00:06:07.199 What are some key observations and lessons from the data? s45 00:06:07.199 --> 00:06:17.680 So the prompt we are seeing here is for the second hop of a multihop question answering system where given a question we need to retrieve some documents that could potentially answer that question. s46 00:06:17.680 --> 00:06:20.800 Look at those documents summarize it and then finally answer the question. s47 00:06:20.800 --> 00:06:33.280 And here what we see is Japa has found out that first hop documents that often cover one entity or aspect and the second hop should actually be uh recovering documents that are related to it. s48 00:06:33.280 --> 00:06:45.440 We have seen that human engineering teams whenever a new model comes out spend weeks of their time manually tweaking one word here and there trying to discover the problem specification. s49 00:06:45.440 --> 00:06:55.680 This entire process is fully automated now with Japa which takes about half an hour to 1 hour to run depending on your uh pipelines. s50 00:06:56.560 --> 00:07:00.240 We can also apply Japa to leading proprietary models. s51 00:07:00.240 --> 00:07:16.240 Just for an example here we were able to optimize GPT 4.1 minis performance to outperform GPT 4.1 on a math task and we can see the kind of information distillation JPA has done in the prompt space itself. s52 00:07:16.240 --> 00:07:28.639 Coming back to the problem of sample efficiency, AMD developed a new hardware accelerator called NPU XDNA2 which had used a completely new API to program which had almost zero available s53 00:07:28.639 --> 00:07:38.400 information over on the internet and because of this uh the leading models at the time which was GPT4 was failing miserably to perform this task. s54 00:07:38.400 --> 00:07:50.639 We are able to take an existing agent which was getting 4.25% 25% on this task and apply Japa without any other change to the agent itself and we got this prompt and pushed this performance s55 00:07:50.639 --> 00:07:53.120 7x to 30.52%. s56 00:07:53.120 --> 00:08:06.639 So what this is uh what this goes to say is there can be lots of domain specific information which if you include in your AI systems prompts the models could actually perform much better and JPA can help you fully automatically s57 00:08:06.639 --> 00:08:07.520 discover that. s58 00:08:07.520 --> 00:08:19.280 I want to highlight the sentence saying avoid including ADF.h H. Now the interesting thing is AMD actually ships a library called ADF.h for programming NPUs but that did not work with this latest s59 00:08:19.280 --> 00:08:25.520 uh generation of hardware that we were working with and Jeppo was able to discover that in just one step. s60 00:08:25.520 --> 00:08:27.039 So how does it work? s61 00:08:27.039 --> 00:08:35.680 It's an extremely simple algorithm which simply takes your AI pipeline written in any agentic framework or even raw LLM calls that you may have. s62 00:08:35.680 --> 00:08:40.479 It simply runs your systems on a few examples and collects domain specific feedback. s63 00:08:40.479 --> 00:08:44.000 whatever information your environment contains is observed. s64 00:08:44.000 --> 00:08:50.880 Second, it runs reflection with an LLM or agent that reads the feedback and proposes a better prompt. s65 00:08:50.880 --> 00:09:01.360 Finally, and most importantly, it keeps a parto pool where it keeps every single candidate that wins on even one training example and not just the top scorer. s66 00:09:01.360 --> 00:09:04.480 The question is, but why keep a parto pool? s67 00:09:04.480 --> 00:09:10.959 And we kept getting asked this question a lot that is Jepper really better than running the model in a loop. s68 00:09:10.959 --> 00:09:17.519 So we went and tested it out and what happens is a loop keeps only the best and gets stuck in a local optima. s69 00:09:17.519 --> 00:09:26.320 So on the left hand side you see a search tree that was generated by using an LLM in a loop starting from a seed prompt at the top left s70 00:09:26.320 --> 00:09:29.680 where um we asked the LLM to improve the prompt. s71 00:09:29.680 --> 00:09:33.920 It improved the prompt and it generated a prompt that gave us the middle node. s72 00:09:33.920 --> 00:09:42.320 However, this prompt got stuck in a local optima and once again when we asked the LLM to try and improve it, it proposed something but that was not actually better. s73 00:09:42.320 --> 00:09:48.240 So, it went back and it again tried to improve it and it kept doing this and it exhausted all of the search budget. s74 00:09:48.240 --> 00:09:58.720 On the other hand, with Japa's parto based candidate selection strategy on the right, we can see that it maintains a much more balanced search process eventually converging to a much higher score. s75 00:09:58.720 --> 00:10:11.760 Across four benchmarks, we saw that more than half of the gains seen with Japa actually account for this and it gets almost twice the performance gains that you would get with just applying the model in a loop. s76 00:10:11.760 --> 00:10:14.640 Japa can perform really well across diverse benchmarks. s77 00:10:14.640 --> 00:10:30.560 Here we see results on question answering, instruction following, claim verification as well as math which all the leading frontier model companies are already optimizing their models a lot for and we are still able to get plus 10% just by optimizing the prompt on it. s78 00:10:30.560 --> 00:10:36.480 So we have so far seen Japa only optimizing the prompts but Japa goes far beyond prompts. s79 00:10:36.480 --> 00:10:47.440 And because prompts are just text artifacts that determine AI system behavior, the same algorithm can improve anything that you can express as a piece of text and you can score. s80 00:10:47.440 --> 00:10:59.440 For example, your entire agent harness is eventually just a Python or a JavaScript file and we can apply the same kind of reflective optimization process to that entire file and we can work with it. s81 00:10:59.440 --> 00:11:04.000 So if you can write it as text and score it, JPA can optimize it. s82 00:11:04.000 --> 00:11:17.680 So with that insight in mind, we propose optimize anything which is a universal API for optimizing any text parameter given any domain like code optimization where let's say you want to optimize the CUDA kernel code. s83 00:11:17.680 --> 00:11:29.920 The input is just that CUDA kernel code where an evaluator looks at this piece of code, maybe compiles it, profiles it, generates a bunch of related information that we call as actionable side information s84 00:11:29.920 --> 00:11:40.320 which is then provided to an LLM which proposes an better candidate maintaining this parto and it keeps the uh repeating this process um till we get convergence. s85 00:11:40.320 --> 00:11:59.040 The same thing can be applied to numeric optimization where your numbers can actually be serialized as text or harness optimization where an entire harness can be serialized as text or even cloud scheduling policy optimization where the scheduling policy or heristic algorithm can be expressed as a piece of text and the evaluator can be something like the negative s86 00:11:59.040 --> 00:12:09.120 of cost or some function measuring accuracy uh efficiency and the actionable side information can be something like job traces SLA violations and so on. s87 00:12:09.120 --> 00:12:11.680 The API is dead simple to use. s88 00:12:11.680 --> 00:12:24.399 All it requires is you give us the set of problems that you care to be solved along with an evaluator function or a fitness function that returns a score along with any available domain specific side information. s89 00:12:24.399 --> 00:12:27.200 If your domain produces expert feedback, return that. s90 00:12:27.200 --> 00:12:33.120 If your domain produces compiler error messages, profiler messages, tool call error messages, return that. s91 00:12:33.120 --> 00:12:36.959 If you have maybe a written up documentation, return that. s92 00:12:36.959 --> 00:12:39.360 any kind of it's a very open-ended dictionary. s93 00:12:39.360 --> 00:12:52.160 You can return literally anything and all you do is you call optimize anything with this fitness function and the set of problems that you have and optimize anything will sort of take care of it um and give you a optimized solution. s94 00:12:52.160 --> 00:12:53.680 Let's see some applications. s95 00:12:53.680 --> 00:12:58.320 Let's say you were tasked with generating a 3D unicorn. s96 00:12:58.320 --> 00:13:08.000 This is all the code that you would write or your agent can now write it because we have seen that optimize anything is a very easy to use API for leading agents like plot code. s97 00:13:08.000 --> 00:13:13.760 So all you do is write this code which says optimize a Python program to generate a 3D unicorn. s98 00:13:13.760 --> 00:13:20.160 Um and the candidate is a Python script that produces a PNG rendering whatever and here is the result. s99 00:13:20.160 --> 00:13:26.320 On the left hand side we can see claude opus 4.6. If you gave it this task, this is what it generated. s100 00:13:26.320 --> 00:13:31.360 And on the right hand side, what what we the unicorn that we get with optimize anything. s101 00:13:31.360 --> 00:13:32.720 This just for fun. s102 00:13:32.720 --> 00:13:39.120 But let's say you were tasked with writing an agent to solve a specific task. s103 00:13:39.120 --> 00:13:48.639 Typically teams spend lots and lots of time tweaking their agents, building tools for it, writing tool descriptions, uh carefully orchestrating the control flow and so on. s104 00:13:48.639 --> 00:13:57.680 Here we started with a simple four-line Python program that was simply calling a model's uh chain of thought to solve an RKGI problem. s105 00:13:57.680 --> 00:14:14.720 Within just 16 rounds of reflection, JEPA within optimize anything was able to find this sophisticated sixstep agent that took RKGI accuracy on RKGI uh that took RKGI accuracy of Gemini flash from 32.5% s106 00:14:14.720 --> 00:14:16.480 to 89.5%. s107 00:14:16.480 --> 00:14:22.800 And we can see that this agent is automatic like by itself doing rule hypothesis induction code synthesis. s108 00:14:22.800 --> 00:14:28.880 It executes and traces the code automatically debugs this code goes back and proposes new versions of that code. s109 00:14:28.880 --> 00:14:33.440 And finally it runs it on the actual test inputs and returns the output. s110 00:14:33.440 --> 00:14:34.959 This is a runnable example. s111 00:14:34.959 --> 00:14:39.839 You can go to this QR code and you can run this example right now. s112 00:14:40.079 --> 00:15:00.000 So um applying the same uh uh like approach of discovering agent harnesses to math 500 we are able to push its accuracy of GPT 4.1 nano by 20% by simply creating a two-step agent and again I want to emphasize that all we did is we asked optimize anything to optimize s113 00:15:00.000 --> 00:15:09.760 an agent file and it was automatically discovering the sophisticated agent architecture and we did not have to do anything other than specifying the objective and the task. s114 00:15:10.160 --> 00:15:24.720 Finally, every single one of us is using uh some coding agent like cloud code or codex or maybe your favorite agent and agent skills has become a very leading part of the ecosystem where almost all coding agents understand skills. s115 00:15:24.720 --> 00:15:29.279 Let's say you want to optimize skills for your specific repository. s116 00:15:29.279 --> 00:15:32.880 This is the code that you write which says learn a skill from the trajectory. s117 00:15:32.880 --> 00:15:36.720 When the coding agent is presented with similar problem, the skill should be helpful. s118 00:15:36.720 --> 00:15:39.360 We just gave it this natural language behavior. s119 00:15:39.360 --> 00:15:51.519 And what we see is we started with minisu agent with GPT5 mini because we were very budget constrainted and we were able to take its performance from 24% to 93%. s120 00:15:51.519 --> 00:16:04.399 An almost 3x jump on go repository issue resolution but more importantly the skills that were optimized very cheaply on a GPT5 mini agent we are able to take that and apply to the latest claude sonnet. s121 00:16:04.399 --> 00:16:19.120 This was done of uh about a few months back but we applied it to clots onet 4.5 pushing its accuracy to 100% issue resolution while more importantly cutting down the execution time or issue resolution time by almost 50%. s122 00:16:19.120 --> 00:16:31.920 We cut it down into half which also means it spent less tokens because skills contain information about how the repository is organized, how to invoke the test cases, where a particular feature is implemented, s123 00:16:31.920 --> 00:16:36.000 um what are the build system used by this repository and so on. s124 00:16:36.000 --> 00:16:38.639 This is a a feature called GSkill. s125 00:16:38.639 --> 00:16:42.320 You can find it in the JPA repository and it's fully open source as well. s126 00:16:42.320 --> 00:16:46.800 So, optimize anything is a single uh interface that provides three optimization modes. s127 00:16:46.800 --> 00:16:53.440 If you have just a single problem like there is a single matrix multiplication kernel that you want to optimize you can use it that way. s128 00:16:53.440 --> 00:17:08.720 If you have any number of related problems like you want to optimize a matrix multiplication kernel along with a dot product kernel and you know there might be some information transfer between these two you can use what we call as the multitask search mode and finally build a skill which is if you want to optimize on a s129 00:17:08.720 --> 00:17:14.000 set number of problems but your uh deployment can actually come up with many new problems. s130 00:17:14.000 --> 00:17:23.919 So like uh in case of math op like in case of math prompt optimization we are training on some examples but when we deploy it we can receive a completely new kind of query. s131 00:17:23.919 --> 00:17:25.439 So we care about generalization mode. s132 00:17:25.439 --> 00:17:30.320 So there you can do prompt optimization agent architecture optimization and so on. s133 00:17:30.320 --> 00:17:43.679 So optimize anything is can be used for a broad set of domains including cloud scheduling policy optimization where we were able to cut costs by almost 40% compared to expert huristics s134 00:17:43.679 --> 00:17:53.039 write custom solvers to match and exceed Optina even in blackbox mathematical optimization create agent skills prompt optimization and so on. s135 00:17:53.039 --> 00:18:02.240 It is so easy to use that within just 20 hours of releasing it, people at Snorkel had already improved some of their internal benchmarks with it and were tweeting about it. s136 00:18:02.240 --> 00:18:06.320 So, and Japa also improves multimodel VLM models performance. s137 00:18:06.320 --> 00:18:11.760 Here we are able to cut OCR error rates for leading models by almost 35%. s138 00:18:11.760 --> 00:18:13.919 And this is an externally validated report. s139 00:18:13.919 --> 00:18:22.080 Um, similar similarly, data bricks actually achieved 90x cost reduction in their deployed agents performance. s140 00:18:22.080 --> 00:18:31.039 uh uh performance and here they were able to tune GPT OSS 120B to outperform Claude Opus while being 90x cheaper. s141 00:18:31.039 --> 00:18:38.960 More importantly, the performance delta improvement that you see on top of Claude Opus is actually bigger than the one you see on open source models. s142 00:18:38.960 --> 00:18:44.480 Some people have asked me that oh as models get better the importance of prompt optimization will go down. s143 00:18:44.480 --> 00:19:02.320 I argue the opposite which is as models get better they will get better at instruction following and the more precise instruction about your task that you have to give to a very smart model the better that model will be at a uh solving your task and this is exactly what we see happening here the better the instruction s144 00:19:02.320 --> 00:19:16.960 was claopus actually jumped much uh higher some people have this question of uh what if we have subjective tasks which are very hard to evaluate jpa can actually learn evals for your task from production traces. s145 00:19:16.960 --> 00:19:21.120 The way to do that is you collect a bunch of production traces from your agent. s146 00:19:21.120 --> 00:19:26.400 Get a human to annotate just about 50 of those trajectories giving very detailed feedback. s147 00:19:26.400 --> 00:19:27.520 This is a long response. s148 00:19:27.520 --> 00:19:28.880 This is a short response. s149 00:19:28.880 --> 00:19:29.919 This is a good response. s150 00:19:29.919 --> 00:19:32.480 This uses this terminology, whatever. s151 00:19:32.480 --> 00:19:37.280 And once you get those human annotations, you can use JPA to optimize an LLM as a judge prompt. s152 00:19:37.280 --> 00:19:43.360 And you can use that LLM as a judge prompt then to go back and optimize your agent and deploy that agent. s153 00:19:43.360 --> 00:19:46.720 And this becomes a data flywheel where you can keep improving it. s154 00:19:46.720 --> 00:19:51.760 And this is a successful paradigm that uh some leading teams in production are already using. s155 00:19:51.760 --> 00:20:05.200 Then the question we get asked is like can we actually use this uh reflective optimization to train models and we recently had this paper called learning fast and slow where we propose fast slow learning where we can co-optimize s156 00:20:05.200 --> 00:20:12.160 model weights and prompt harnesses and this shows some very strong properties that one would want in a continual learning algorithm. s157 00:20:12.160 --> 00:20:27.440 Um I don't have much time to go over details but please uh look at the uh papers and uh since uh since release Japa has been used in production by these companies as well as the main methodology in these papers s158 00:20:27.440 --> 00:20:38.000 and here the CEO of Dropbox and Shopify are talking about their use of Japa and OpenAI also wrote a blog post about how you can build self-improving AI systems with Japa. s159 00:20:38.000 --> 00:20:40.480 Um so it's very simple to get started. s160 00:20:40.480 --> 00:20:46.159 It can plug into any framework, any model and it has absolutely zero hard dependencies. s161 00:20:46.159 --> 00:20:49.600 So you can deploy it any in any kind of setting. s162 00:20:49.600 --> 00:20:55.840 So um don't be afraid to optimize in the tech space and many problems can be framed as optimization. s163 00:20:55.840 --> 00:21:06.000 So bring actionable side information and surface as much domain specific information as you can to optimizers and the optimizers of future will be able to work with them. s164 00:21:06.000 --> 00:21:07.679 So please go and check it out. s165 00:21:07.679 --> 00:21:10.240 Thank you very much.