WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=TyPpSRXGhbc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/TyPpSRXGhbc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.400 --> 00:00:15.240 Everyone, thanks for joining. s3 00:00:15.240 --> 00:00:19.480 I'm Charles and I'm a research scientist at Snorkel AI. s4 00:00:19.480 --> 00:00:29.360 And today I'm going to be talking about how we got a 4 billion parameter model to outsmart its 235 billion parameter variant. s5 00:00:29.360 --> 00:00:38.560 And this is a specific result in a simulated financial environment, but I think there are general takeaways that I want to make sure taken home. s6 00:00:38.560 --> 00:00:49.240 And the first is for enterprise AI workloads, reliability and specialization uh can actually outweigh raw parameter scale. s7 00:00:49.240 --> 00:00:56.880 And this is possible with high-quality data that actually represents the domain and task distribution that you're targeting. s8 00:00:56.880 --> 00:01:04.400 And uh this can result in smaller models that can achieve parity with frontier models at just a fraction of the cost. s9 00:01:04.400 --> 00:01:10.560 So, before I get to it, I'm going to take this opportunity to introduce ourselves a little bit. s10 00:01:10.560 --> 00:01:11.200 This is our team. s11 00:01:11.200 --> 00:01:26.920 We're a frontier AI data lab and our team, we have founders from labs at University of Washington, Wisconsin, Stanford, and an internal team of really talented FTEs, researchers, and we're in a fortunate position to play in, by that I mean build s12 00:01:26.920 --> 00:01:31.320 and deploy data sets and environments uh for frontier AI. s13 00:01:31.320 --> 00:01:39.520 And that's at the intersection of academia and industry research, but also grounded in enterprise deployments. s14 00:01:39.520 --> 00:01:49.040 Um and this is all to say that we have a team that's built to focus on building the best data possible to define in advanced AI. s15 00:01:49.040 --> 00:01:52.280 And the way we do this is threefold. s16 00:01:52.280 --> 00:02:03.760 First is open research and benchmarks, so we're constantly contributing to open-source benchmarking, and the project I'll be talking about today actually falls under this pillar. s17 00:02:03.760 --> 00:02:12.240 Um but I also want to acknowledge frontier live data and environments, as well as enterprise uh deployments are the things we work on. s18 00:02:12.240 --> 00:02:24.160 And under open research and benchmarks, we support research through grants, and I'll talk about opportunities here at the end of this talk, so um get ready for that. s19 00:02:24.160 --> 00:02:33.240 And then co-authoring with academic uh or industry partners um and leading independent research and publishing open artifacts. s20 00:02:33.240 --> 00:02:45.400 So, what I'm talking about today is uh a partnership with UC Berkeley Sky Computing Lab, and we built, like I said, a 4 billion parameter Qwen model to outperform its 235 s21 00:02:45.400 --> 00:02:49.000 billion model variant on financial tasks. s22 00:02:49.000 --> 00:02:59.760 We saw the 4 billion parameter model, spoiler alert, uh achieve almost a 60% pass at one, while the 235 billion model uh achieved 51%. s23 00:02:59.760 --> 00:03:08.720 So, exciting results, and this is all uh in collaboration with I got to give credit where credit's due, Manon Rupta, Shujian Tang from the Sky Computing Lab, as well as Bhavishya s24 00:03:08.720 --> 00:03:11.720 and Chris Glaze and myself uh from Snorkel. s25 00:03:11.720 --> 00:03:23.600 And you can find all the assets, the uh training scripts, synthetic data uh on our GitHub, so check those out, and this is all open-source and available to you. s26 00:03:23.600 --> 00:03:36.280 So, I'll cover today what uh we see financial institutions need um from AI, and this is from our enterprise deployments uh learnings, as well as the FinQA open-source benchmark. s27 00:03:36.280 --> 00:03:42.640 And this is expert-validated financial question-answering data that we trained our uh and evaluated our model on. s28 00:03:42.640 --> 00:03:49.080 And finally, uh the training process for the RLLM FinQA 4B model. s29 00:03:49.080 --> 00:03:51.440 So, let's get started. s30 00:03:51.519 --> 00:03:57.680 This is uh the one and only slide with the Charles Dickens reference, I promise, but this is a tale of two models, I'd say. s31 00:03:57.680 --> 00:04:01.840 First, the large generalist versus small specialist. s32 00:04:01.840 --> 00:04:13.560 And the point is here, when financial institutions ask us how well does the model or the agent reason, what they typically mean is can it operate in my stack of complex schemas, legacy APIs, s33 00:04:13.560 --> 00:04:14.560 and tech debt? s34 00:04:14.560 --> 00:04:19.400 And can we trust it in realistic uh and long workflows without compounding errors? s35 00:04:19.400 --> 00:04:21.000 And can we reliably audit the agent? s36 00:04:21.000 --> 00:04:24.160 That's important in financial and legal domains, for example. s37 00:04:24.160 --> 00:04:33.040 Which brings us to the question is uh how can we evaluate and improve agents to make them reliable in these uh real-world environments? s38 00:04:33.040 --> 00:04:38.120 So, um the reflex may be to scale. s39 00:04:38.120 --> 00:04:44.200 We've all seen scaling law results, uh but today I'm going to argue that specialized workflows don't actually need generalists. s40 00:04:44.200 --> 00:04:45.640 We don't need polymaths. s41 00:04:45.640 --> 00:04:51.160 You know, you wouldn't call Terence Tao for a tax audit, um if you're lucky enough to have him in your contacts. s42 00:04:51.160 --> 00:04:55.360 You'd call the specialist that knows the forms, the tools, and the rules. s43 00:04:55.520 --> 00:04:57.000 So, that's why we built FinQA. s44 00:04:57.000 --> 00:05:00.520 This is expert-validated financial question-answering data. s45 00:05:00.520 --> 00:05:06.480 And I'll walk through the multi-stage pipeline of how we built this benchmark and training data. s46 00:05:06.480 --> 00:05:10.360 So, first is schema and data extraction. s47 00:05:10.360 --> 00:05:13.200 This is coming from 10-K reports. s48 00:05:13.200 --> 00:05:24.240 Uh these are annual reports required by the SEC for all public companies, and they're used uh by analysts for identifying risks and informing investment decisions. s49 00:05:24.240 --> 00:05:35.280 Uh we pull these from the Edgar system and use the Qwen 3 30 billion model to produce tables, SQL tables, approximately 6,900 of those. s50 00:05:35.280 --> 00:05:41.360 And we use each table to generate a single question-answer pair. s51 00:05:41.360 --> 00:05:50.040 And this is uh produced along with the question taxonomy that was developed with financial experts that work with these documents. s52 00:05:50.040 --> 00:05:59.560 And these are used as context uh for the model to generate a question-answer pair along with metadata that we'll see uh what that's used for later. s53 00:05:59.560 --> 00:06:01.960 And finally, the third step is verification. s54 00:06:01.960 --> 00:06:05.200 And this is a three-layer verification process. s55 00:06:05.200 --> 00:06:08.000 First is programmatic consistency checks. s56 00:06:08.000 --> 00:06:15.840 That's where we use um the metadata, uh which includes lineage and table names and columns to make sure we're not hallucinating those. s57 00:06:15.840 --> 00:06:19.560 The second is automated reviews from independent agents. s58 00:06:19.560 --> 00:06:21.760 And finally, expert manual review. s59 00:06:21.760 --> 00:06:28.480 So, humans actually going through these and making sure the questions and answers are realistic and verified. s60 00:06:29.800 --> 00:06:32.800 And the outcome is Snorkel's FinQA data. s61 00:06:32.800 --> 00:06:35.160 And this is our first pass at this. s62 00:06:35.160 --> 00:06:41.280 Uh realistic queries requiring planning, tool use, uh tool calling, and reasoning. s63 00:06:41.280 --> 00:06:46.840 And um answers are single, verifiable, final answers. s64 00:06:46.840 --> 00:07:01.200 And from this, um we get from the uh original set of roughly 7,000 tables, 4,000 in train, 500 in val, and 290 in the held-out benchmark. s65 00:07:01.200 --> 00:07:08.040 And the data is split so that no company in the benchmark overlaps um in any split. s66 00:07:08.520 --> 00:07:19.400 And on this data, we identified um even with frontier models, uh reoccurring discipline gaps, uh failure modes. s67 00:07:19.400 --> 00:07:21.600 And the first is schema hallucination. s68 00:07:21.600 --> 00:07:32.600 So, a model will actually assume that tables exist, column names exist, and um this can be an artifact of pre-training data, something it's seen um in its past. s69 00:07:32.600 --> 00:07:33.800 The next is context flooding. s70 00:07:33.800 --> 00:07:37.760 So, actually flooding its own context um with poorly playing queries. s71 00:07:37.760 --> 00:07:45.360 For example, just calling select star, um and this could overwhelm its context limits if successful. s72 00:07:45.360 --> 00:07:49.040 And finally, poor recovery um from errors. s73 00:07:49.040 --> 00:07:57.200 So, when 3235 would actually repeat the same failed strategy um instead of considering the error messages and adapting. s74 00:07:57.200 --> 00:08:05.960 And these failure modes are not actually just seen in the financial environment that I'm talking about today, but also what we saw in insurance underwriting. s75 00:08:05.960 --> 00:08:15.480 And this is a work that I'm calling out that was presented at the CAIS conference um few weeks ago. s76 00:08:15.480 --> 00:08:22.360 So, now I'll get into actually training the R LLM FinQA 4 billion model. s77 00:08:22.360 --> 00:08:31.280 Uh for this we used the R LLM uh framework and this was developed by the R Sky Computing Lab collaborators. s78 00:08:31.280 --> 00:08:41.520 And if you haven't used this yet, um this is a really user-friendly tool and a few core features um I want to call out is it really works with any agent framework, s79 00:08:41.520 --> 00:08:48.160 requires near zero code changes just using uh a decorator pattern, CLI first workflow, battle-tested results. s80 00:08:48.160 --> 00:08:57.360 So, um we've seen R LLM actually improve not just on this finance um case study, but also in math reasoning for example. s81 00:08:57.360 --> 00:09:10.000 Um multiple RL algorithms are built in, GRPO, reinforce RL, OOO for example, and you can customize those and then supports multiple uh training backends. s82 00:09:10.720 --> 00:09:18.920 All right, now for our training environment, um this is the uh training environment we used for our model. s83 00:09:18.920 --> 00:09:25.160 The agent runs in a react loop and has access to tools uh to interact with the tables we generated. s84 00:09:25.160 --> 00:09:34.240 The environment includes the roughly 7,000 tables we created and finally reward is binary correctness uh determined by a language model as judge. s85 00:09:34.240 --> 00:09:40.600 In this case, GPT 5 nano using reference-based evaluation. s86 00:09:41.560 --> 00:09:43.320 Training details are listed here. s87 00:09:43.320 --> 00:09:47.200 This is uh I guess would be interesting to folks. s88 00:09:47.200 --> 00:09:58.720 We have the Qwen 3 4 billion model is our base model, GRPO and binary reward uh for optimization and the RL and framework. s89 00:09:58.720 --> 00:10:09.680 Um we ran roughly 1,000 concurrent environments to generate trajectories and this all was a cost under $500 to get the results that we saw. s90 00:10:09.680 --> 00:10:17.480 Um broken down between compute roughly 420 and $40 for a judge uh the judge API. s91 00:10:17.480 --> 00:10:20.680 And this was on eight H100s. s92 00:10:21.240 --> 00:10:33.720 The first result um is the central question, can a 4 billion model uh armed with the specialized tools and training actually compete with its larger variant? s93 00:10:33.720 --> 00:10:34.760 And the answer is yes. s94 00:10:34.760 --> 00:10:44.920 So, what we saw was actually the base model's accuracy um more than doubled and it outperforms its larger variant, the 235 parameter model. s95 00:10:44.920 --> 00:10:47.920 And this is uh despite being a fraction of the size. s96 00:10:47.920 --> 00:10:56.040 And we evaluate all models shown here um on the expert curated held out 290 samples. s97 00:10:56.839 --> 00:11:01.120 So, a natural concern is whether this actually transfers to harder problems. s98 00:11:01.120 --> 00:11:05.200 So, this is just single question answer pairs on a single table. s99 00:11:05.200 --> 00:11:10.920 Um and to investigate this, we uh looked at our FinQA reasoning data. s100 00:11:10.920 --> 00:11:16.040 So, this is a development on the FinQA um data I presented earlier. s101 00:11:16.040 --> 00:11:26.640 And the difference here is that now we're having a dependencies on multiple tables, two to five tables, and requires sequential um decision-making and planning. s102 00:11:26.640 --> 00:11:29.160 And we evaluated on this data set. s103 00:11:29.160 --> 00:11:31.520 You can see the breakdown um as well. s104 00:11:31.520 --> 00:11:37.520 We have roughly 1,000 in train, 120 in validation, 80 in benchmarks. s105 00:11:38.720 --> 00:11:46.680 And what we see is the tool use discipline actually generalized directly from training on the simple data. s106 00:11:46.680 --> 00:11:56.480 And no explicit training on multi-table examples was needed actually to see lift in this multi-table variant of the data set. s107 00:11:56.480 --> 00:11:59.800 And we'll investigate why with the ablation studies. s108 00:11:59.800 --> 00:12:07.120 But the next concern we had was whether this result would generalize to more general use cases. s109 00:12:07.120 --> 00:12:13.280 So for this we use the BFCL benchmark and this measures general tool calling capability. s110 00:12:13.280 --> 00:12:24.880 And overall accuracy was actually slightly improved and then um we had also minor gains on multi-turn and memory. s111 00:12:24.880 --> 00:12:34.440 So specialization didn't actually erode the model's broader tool use competence when we did RL fine-tuning for this model. s112 00:12:34.960 --> 00:12:42.760 So to isolate what drove this performance, we ran ablation study on data mixtures, so different ways we can combine the training data we had. s113 00:12:42.760 --> 00:12:51.200 The first is just training on FinQA data and the second single and multi-table and finally a curriculum where we start with single and then move to multi. s114 00:12:51.200 --> 00:12:56.960 And surprisingly, just training on the simpler set resulted in the most lift. s115 00:12:56.960 --> 00:13:01.280 And this partially explains why the generalization result we saw earlier. s116 00:13:01.280 --> 00:13:09.200 The bottleneck was never reasoning depth, it was just tool use, meaning the model where it was actually failing and improving the reliability of that. s117 00:13:09.200 --> 00:13:13.960 And once you master the fundamentals, it could compose those skills. s118 00:13:18.200 --> 00:13:21.800 Yeah, so oops, let me go back. s119 00:13:21.800 --> 00:13:34.000 The next natural question is whether more informative reward signals, Um, for example, rewarding intermediate steps, uh, table access, query completeness, things like that could actually accelerate learning. s120 00:13:34.000 --> 00:13:36.880 Uh, so this is different from curriculum, for example. s121 00:13:36.880 --> 00:13:41.600 And the most sophisticated variant we came up with was, uh, a rubric. s122 00:13:41.600 --> 00:13:45.560 And this uses fine-grained scoring with, uh, multiple weighted components. s123 00:13:45.560 --> 00:13:48.800 And this is actually coming came up with with experts. s124 00:13:48.800 --> 00:13:54.640 Um, and despite this investment in reward engineering, we actually again saw, um, simpler was better. s125 00:13:54.640 --> 00:14:02.160 A single binary reward signal, um, was what got us the most lift on this data. s126 00:14:02.160 --> 00:14:07.120 Which led us to a blueprint, um, that we've been developing for enterprise agents. s127 00:14:07.120 --> 00:14:19.440 Um, and generally, I'd say this study showed us that small models uh, can successfully be trained uh, with RL um, when you have quality data to uh, represent the target s128 00:14:19.440 --> 00:14:21.680 um, outcomes and tasks. s129 00:14:21.680 --> 00:14:27.880 And this can result in better deployment economics uh, for specialized tasks and domains. s130 00:14:27.880 --> 00:14:36.200 And um, we've seen this and repeated this result in other areas including health care, law, and insurance, um, which I pointed to earlier. s131 00:14:36.200 --> 00:14:37.960 So, check that out. s132 00:14:37.960 --> 00:14:51.960 Um, and our point of view as far as building um, axes for evaluating agents is this is environment complexity is um, one of the most important axes. s133 00:14:51.960 --> 00:14:57.480 How complex, how realistic and dynamic are the environments uh, the agents operating in. s134 00:14:57.480 --> 00:14:59.720 The next is autonomy horizon. s135 00:14:59.720 --> 00:15:06.680 Um, are agents being tested and evaluated at the lengths and the horizon lengths in the way people actually want to use agents? s136 00:15:06.680 --> 00:15:10.080 Um, are they making safe and good decisions by themselves? s137 00:15:10.080 --> 00:15:16.040 And can they actually improve um, their judgment as a co-pilot over time? s138 00:15:16.040 --> 00:15:18.320 And finally, output complexity. s139 00:15:18.320 --> 00:15:22.240 We want to capture the wide range of outputs in our day-to-day work. s140 00:15:22.240 --> 00:15:26.160 Um, and I would say that this is currently under explored. s141 00:15:26.160 --> 00:15:42.600 New benchmarks are coming up in this direction, but uh still I see a big primary focus on um text artifacts and creating more rigorous evaluations and these settings is a big opportunity. s142 00:15:42.600 --> 00:15:45.280 Progress is being made here. s143 00:15:45.280 --> 00:15:52.240 So, on that note, um I'm going to advertise our open benchmarks grants, which funded this project. s144 00:15:52.240 --> 00:15:58.800 And this is a multi-million dollar commitment to producing open benchmarks like this. s145 00:15:58.800 --> 00:16:14.320 We've had a lot of success agent, last exam, judgment bench, um others are coming out and this is um really accelerating our ability to measure the frontier and shape the frontier. s146 00:16:14.320 --> 00:16:18.200 So, to apply uh check out our website. s147 00:16:18.200 --> 00:16:27.600 Um you'll go through a submission selection review process, develop, and then finally publication, launch, um and promotion. s148 00:16:27.600 --> 00:16:40.760 And I also want to call out that we're hiring in professional research, um you name it, engineers, and um yeah, check that out. s149 00:16:40.760 --> 00:16:47.080 You can check out our website and finally, thank you so much for attending. s150 00:16:47.080 --> 00:16:56.600 Um you can use this QR code to see the blog as well as pointers to all the artifacts that released on GitHub, Hugging Face. s151 00:16:56.600 --> 00:16:59.520 Yeah, thank you all.