WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Fa3hQVIKShw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Fa3hQVIKShw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.654 --> 00:00:02.654 [music] s2 00:00:04.920 --> 00:00:15.600 So, I really love this article from Hannes at Duct Tape B, where he asks if we lost a decade chasing distributed architectures that presumed compute and storage were slow and expensive. s3 00:00:15.600 --> 00:00:22.160 Capability in every era though defines architecture and tooling, and the tooling likes behind consistently. s4 00:00:22.160 --> 00:00:28.200 We first made compute cheaper than storage, and then we gave it to everybody, and then we made it faster. s5 00:00:28.200 --> 00:00:32.200 And every single time architectures undergo this massive change. s6 00:00:32.200 --> 00:00:38.400 All of our data infrastructure today is built on the idea that transformation is expensive. s7 00:00:38.400 --> 00:00:41.360 What if that wasn't true anymore? s8 00:00:41.400 --> 00:00:52.400 I believe we've solved the final bottleneck in transformation, general intelligence or judgment, and we've made it accessible on tap all along the Pareto frontier. s9 00:00:52.400 --> 00:00:55.200 Which puts us right on the verge of a paradigm shift. s10 00:00:55.200 --> 00:01:06.160 So, in this talk, we're going to cover why single ontologies always fail, the three roles models play in pipelines, intersecting model priors for better results, and Pareto serving. s11 00:01:06.160 --> 00:01:12.760 How today's models can tune a workload for cost, speed, and accuracy all at the same time. s12 00:01:12.760 --> 00:01:14.320 Here's a quick example. s13 00:01:14.320 --> 00:01:25.360 When Jeff launched, we already had an entity resolution pipeline running, working out Ohio's campaign finance records and whether they belong to the same person, company, or committee. s14 00:01:25.360 --> 00:01:28.360 It was projected to cost about $10,000. s15 00:01:28.360 --> 00:01:35.400 Our systems took that new model, worked out where it fit, tested over a dozen approaches, and integrated it. s16 00:01:35.400 --> 00:01:42.960 10 days after that first call, we had final ground truth for every donor case in Ohio under $800. s17 00:01:42.960 --> 00:01:47.640 That is the power of Pareto serving and just-in-time transformations. s18 00:01:47.640 --> 00:01:51.000 So, let's get into it. s19 00:01:52.080 --> 00:01:56.480 Ontologies are the enterprise version of mind maps. s20 00:01:56.480 --> 00:02:00.480 It's this carpet that sucks us all in, including me. s21 00:02:00.480 --> 00:02:06.120 I talked about a single universal schema when I founded Southbridge 2 years ago. s22 00:02:06.120 --> 00:02:14.120 The popular conception, and it's not wrong, it's just small part of the picture, is that migrations and ontologies fail because of a lack of buying. s23 00:02:14.120 --> 00:02:18.600 And they're abandoned halfway through, that organizations and people never switch over. s24 00:02:18.600 --> 00:02:20.560 All of that is true. s25 00:02:20.560 --> 00:02:21.600 100%. s26 00:02:21.600 --> 00:02:24.160 But buying isn't the primary reason. s27 00:02:24.160 --> 00:02:35.400 Say, if you could integrate and migrate all of your information into a single super schema in the blink of an eye, you still have to contend with the truth that there is no super schema. s28 00:02:35.400 --> 00:02:37.720 There actually are four schemas. s29 00:02:37.720 --> 00:02:42.280 The first one belongs to the source, wherever the information comes from. s30 00:02:42.280 --> 00:02:43.840 I take temperature as an example. s31 00:02:43.840 --> 00:02:46.600 Say, I ask you to write down the temperature. s32 00:02:46.600 --> 00:02:49.320 The source schema is whatever you have access to. s33 00:02:49.320 --> 00:02:56.080 Your phone gives you your region's temperature, sensor on your desk, which is less accurate, but it's more local, or your watch, say. s34 00:02:56.080 --> 00:03:01.360 But even your patience in following through on my request is part of that source schema. s35 00:03:01.360 --> 00:03:04.760 The second one is the storage schema. s36 00:03:04.760 --> 00:03:07.360 There's only so much you can write down. s37 00:03:07.360 --> 00:03:10.519 You can't write down the firmware of the device you're using. s38 00:03:10.519 --> 00:03:16.280 You can't even write the time down to the second because you couldn't finish writing all of that inside of a second. s39 00:03:16.280 --> 00:03:19.280 This is the schema imposed on you by storage. s40 00:03:19.280 --> 00:03:22.200 The third one belongs to the decision. s41 00:03:22.200 --> 00:03:24.680 Why do I need you to write the temperature down? s42 00:03:24.680 --> 00:03:33.239 Maybe you're figuring out what makes you comfortable, in which case I need temperatures from you spread throughout the day, and I need to make sure that it's you that's doing the measuring. s43 00:03:33.239 --> 00:03:44.160 Maybe I'm building weather migration maps, in which case I don't care who does the measuring, but I care a lot about your exact location, the time, the device you used, and whether that's changed. s44 00:03:44.160 --> 00:03:53.160 Or maybe I'm deciding whether to install sensitive equipment and where, and you yourself become this variable that I need to cancel out. s45 00:03:53.160 --> 00:03:56.200 There's a fourth one though, which we'll get to when we talk about priors. s46 00:03:56.200 --> 00:03:58.320 It's this recent addition. s47 00:03:58.320 --> 00:04:06.600 The true problem is that you will never know all four schemas at the same time except at the moment that the decision is being made. s48 00:04:06.600 --> 00:04:13.040 In almost every enterprise and every home, most of the data being recorded has little idea of what it's going to be used for. s49 00:04:13.040 --> 00:04:17.079 Take a customer, the first box in every ontology. s50 00:04:17.079 --> 00:04:20.120 To building, a customer is whoever pays. s51 00:04:20.120 --> 00:04:22.000 To supported, whoever calls. s52 00:04:22.000 --> 00:04:25.000 To compliance, it's whoever we're on the hook for. s53 00:04:25.000 --> 00:04:27.840 And the only time you know is when the decision is known. s54 00:04:27.840 --> 00:04:29.600 Should we refund this order? s55 00:04:29.600 --> 00:04:31.880 Can we extend this account more credit? s56 00:04:31.880 --> 00:04:38.120 Suddenly, what you need to know, how and to what degree, becomes extremely clear. s57 00:04:38.120 --> 00:04:40.400 This is why I was wrong. s58 00:04:40.400 --> 00:04:42.560 There is no universal schema. s59 00:04:42.560 --> 00:04:47.280 There is no concept of a customer that's divorced from the question that you want to answer. s60 00:04:47.280 --> 00:04:53.919 And pulling these schemas together is the process of transformation, of making information actually useful. s61 00:04:53.919 --> 00:04:59.120 Taking the context that it was recorded in and merging it with the context of the decision. s62 00:04:59.120 --> 00:05:01.680 And that takes discernment. s63 00:05:01.680 --> 00:05:08.960 Doing that merge when the decision space is known instead of ahead of time is what I mean by a just-in-time ontology. s64 00:05:08.960 --> 00:05:18.120 So, we'll come back to discernment with a story from my last job where like most of my job was to make sure that certain numbers just did not exist ever. s65 00:05:19.040 --> 00:05:21.919 Before that, a little bit about me and us. s66 00:05:21.919 --> 00:05:25.360 We are best described as obsessives. s67 00:05:25.360 --> 00:05:28.160 Obsessives who embed intelligence into pipelines. s68 00:05:28.160 --> 00:05:38.880 Pipelines that can scale, that can run over and over again to take work truly off our hands, and pipelines that can mature in cost, performance, and speed all on their own. s69 00:05:38.880 --> 00:05:42.960 And most of the work we do in data breaks into three regimes. s70 00:05:42.960 --> 00:05:46.000 The first is write once, run once. s71 00:05:46.000 --> 00:05:51.840 If you do any kind of development, think of these as monster scale immediately invoked functions. s72 00:05:51.840 --> 00:06:00.040 They're built for something single use, like joining a data set, mining information, or answering one specific question, and then they get torn down. s73 00:06:00.040 --> 00:06:08.680 The second is write many, run once, where lots of smaller and larger workloads are built for one large common goal. s74 00:06:08.680 --> 00:06:11.760 The closest thing you might know is like large migrations. s75 00:06:11.760 --> 00:06:15.480 Years of records moved out of systems nobody fully remembers. s76 00:06:15.480 --> 00:06:23.120 Lots of workloads built for one rule and verified individually at source, checked against the others because all of it only happens once. s77 00:06:23.120 --> 00:06:28.680 The third, though, and this is bread and butter, is the most common, is write many, run many. s78 00:06:28.680 --> 00:06:34.680 These are repeatable pipelines that are built and maintained over time as we add more information in. s79 00:06:34.680 --> 00:06:45.880 And at Southridge, we've moved so much work off the main thread and into these repeatable systems, from benchmark design, tracking, coding, to making subtitles for movies that we want to watch. s80 00:06:45.880 --> 00:06:52.880 Almost everything we do in the modern world is a transformation of data, or it's bottlenecked by how hard it is to transform data. s81 00:06:52.880 --> 00:06:55.440 And in all of these cases, our job is simple. s82 00:06:55.440 --> 00:06:58.040 Make sure people can rely on the results. s83 00:06:58.040 --> 00:07:15.440 In just 2 years, we've gone from figuring out how to make early 3.5 turbo level models reliable on things like health care, aging, mining research, to building Hank Wave, our open-source harness for data work, and then using that to get models to work together at terabyte scale. s84 00:07:15.440 --> 00:07:18.000 Today, we have self-optimizing systems. s85 00:07:18.000 --> 00:07:24.200 You give them the problem, a way to fail safely, a place to work, and they find the shape of the solution. s86 00:07:24.200 --> 00:07:32.640 They pick the right models, they run the pilots, they scale up, and they Pareto serve their way to the best option they can find. s87 00:07:34.360 --> 00:07:42.840 So, story at my last job, the question we heard most often was, "Can we make this decision data led?" s88 00:07:42.840 --> 00:07:47.240 And sometimes that was the right question when we had thousands of vessels on the platform. s89 00:07:47.240 --> 00:07:51.560 The problem is that we also asked it when we only had 50 users. s90 00:07:51.560 --> 00:07:57.680 And if you know statistics, you can explain really well why AB testing 50 users is a bad idea. s91 00:07:57.680 --> 00:08:01.640 Not because you're not going to get a result, but because you will. s92 00:08:01.640 --> 00:08:10.000 So the only real option was to do not produce a number at all because the moment a number exists, it argues for a result. s93 00:08:10.000 --> 00:08:12.200 Today though, it is worse. s94 00:08:12.200 --> 00:08:14.880 Interactive agents will always give you an answer. s95 00:08:14.880 --> 00:08:16.320 They'll give it to you quickly. s96 00:08:16.320 --> 00:08:20.560 You ask for a transformation, whatever, and you get a contra and yes, sir. s97 00:08:20.560 --> 00:08:23.400 What this means is mistakes add up quickly. s98 00:08:23.400 --> 00:08:28.520 Moment in chat can be a broad nightmare forever just waiting to be discovered. s99 00:08:28.520 --> 00:08:39.960 The primary reason that this happens is that intelligence, as it relates to data work today, occupies three separate roles that coding agents kind of mash into one context window. s100 00:08:39.960 --> 00:08:46.360 Let's pull those apart because it'll help a lot when we get to decision models because they fit into exactly one of these three. s101 00:08:46.360 --> 00:08:54.400 The first one is building and designing the pipeline, which is breaking down the problem, writing the early specs, authorizing spend. s102 00:08:54.400 --> 00:09:04.480 The second one, and this is usually confused with the first and given to the same models, is working inside the pipeline, actually doing the work, reading the PDF, writing to databases, s103 00:09:04.480 --> 00:09:05.880 checking results. s104 00:09:05.880 --> 00:09:11.200 Before we get to the third one, I want to make it clear that these two roles pull in very different directions. s105 00:09:11.200 --> 00:09:26.920 The first one wants slower models with heavier, thicker tokens, models that can think more, walk around a problem and see the angles, digest documentation, understand how these pipelines get built, and predict how something will do before it even starts, right? s106 00:09:26.920 --> 00:09:28.839 Opus or up. s107 00:09:28.839 --> 00:09:32.640 The models you want inside pipelines though are the opposite. s108 00:09:32.640 --> 00:09:39.360 You want them cheaper, faster, specialized, good at following instructions, repeatable workers with noble defects. s109 00:09:39.360 --> 00:09:40.640 You see the same thing in nature. s110 00:09:40.640 --> 00:09:45.200 A leafcutter ant colony, for example, runs on four different sizes of worker, right? s111 00:09:45.200 --> 00:09:47.720 Small, medium, high, extra high. s112 00:09:47.720 --> 00:09:54.880 As you watch any system that you design, review, or hear about, listen for what slots in where in this layer cake. s113 00:09:54.880 --> 00:09:56.520 Makes a big difference. s114 00:09:56.520 --> 00:10:04.000 Because we had this distinction, bringing in Jev in this newly exploding class of decision models was very easy cuz they're workers. s115 00:10:04.000 --> 00:10:10.480 They're built for quick contact with information and they're deployed at the bottom end of large workloads. s116 00:10:10.480 --> 00:10:12.680 So, third category, though. s117 00:10:12.680 --> 00:10:15.760 This is something we call the hill climbers or servers. s118 00:10:15.760 --> 00:10:19.320 Models that can take a first one and self-optimize. s119 00:10:19.320 --> 00:10:34.240 Given the right knowledge, instructions, and an incredibly stable surface to stand on, they can try new things, vary budgets to limit cost exposure, debug quickly, take notes, surf their way not just to the right solution, but the optimal one. s120 00:10:34.240 --> 00:10:38.280 We call that burrito surfing and we'll cover it at the end. s121 00:10:38.280 --> 00:10:41.400 But, first, let's talk about our experience with Jev. s122 00:10:41.400 --> 00:10:47.400 Jev launched on a Tuesday when we were deep into an entity resolution pipeline. s123 00:10:47.400 --> 00:10:50.160 You were working on donors in Ohio's campaign finance. s124 00:10:50.160 --> 00:10:53.040 And at this point, a lot of people have seen this chart. s125 00:10:53.040 --> 00:10:57.520 We talked about the results before and adapted some of the ongoing work to the data set. s126 00:10:57.520 --> 00:11:00.720 So, you can download it for yourself, prompts and everything. s127 00:11:00.720 --> 00:11:08.120 What we didn't talk about is everything behind that one arrow and what happened when you ran the same idea on the whole state. s128 00:11:08.120 --> 00:11:10.320 Here all of the architectures we tried. s129 00:11:10.320 --> 00:11:13.839 We started with three designs on a small development set. s130 00:11:13.839 --> 00:11:23.120 A cheap reasoning model on its own, Jev with a reasoning model for anything it wasn't sure about, and a reasoning model proposing answers with Jev checking them. s131 00:11:23.120 --> 00:11:31.440 On about 200 health out families, the cheap reasoning model on its own did as well as Fable for about 2% of the cost. s132 00:11:31.440 --> 00:11:39.640 Jev first got the core answers right, but it missed nine household links because it had confidently said no to every one of them. s133 00:11:39.640 --> 00:11:41.600 So, we rewrote the questions. s134 00:11:41.600 --> 00:11:50.440 Without touching the models of the thresholds, the household question now said what counts as enough evidence, and Jeff went from no on all 13 development questions to yes. s135 00:11:50.440 --> 00:11:55.520 And as we scaled it statewide, we ran over a dozen rounds of such qualification. s136 00:11:55.520 --> 00:12:02.640 Each round fixed a definition, what a committee is, what a filer means, when two people live at one address, what does that mean? s137 00:12:02.640 --> 00:12:12.480 Every one of Ohio's 203,317 cases now has a label that says what the answer rests on and how far it was checked. s138 00:12:12.480 --> 00:12:16.320 This is the hierarchy we discovered for this specific problem. s139 00:12:16.320 --> 00:12:19.920 And here's everything we learned about putting decision models into pipelines. s140 00:12:19.920 --> 00:12:22.880 I'll leave this up so you can screenshot it. s141 00:12:22.880 --> 00:12:24.440 But here's where the money went. s142 00:12:24.440 --> 00:12:31.040 The whole statewide job came to 795 on an $800 budget that was set ahead of time. s143 00:12:31.040 --> 00:12:41.280 Checking cost about four times as much as doing, 545 on second opinions, settling disagreements, and the audit against 128 for first answers. s144 00:12:41.280 --> 00:12:44.920 Before Jeff, we expected this work to cost about $10,000. s145 00:12:44.920 --> 00:12:50.920 And now that the pipeline is built and verified, we expect the audit budget for the next state to be close to zero. s146 00:12:50.920 --> 00:13:01.160 The most amazing thing is that our agent 4A, running on Astra and then later on Opus, found most of these approaches incrementally on its own. s147 00:13:01.160 --> 00:13:06.080 By building the system, running it, debugging it, and trying new things. s148 00:13:06.080 --> 00:13:09.440 It ran for 10 days, tried over a dozen architectures. s149 00:13:09.440 --> 00:13:22.360 Its hangs ran for more than 4,000 times alongside 90,000 calls to Jeff, tried practically every model on the Pareto frontier, and the models inside those runs went through roughly 6 billion tokens. s150 00:13:22.360 --> 00:13:27.880 And once all of that was done, we wrote what we learned back into our library of agentic parts. s151 00:13:27.880 --> 00:13:34.360 This work leaned on these existing ones and we're adding new tested ones like the checking letter. s152 00:13:34.360 --> 00:13:44.440 This is to me the value of taking work off the main thread for humans and for agents because every single one of these iterations was a version controlled Hank. s153 00:13:44.440 --> 00:13:48.720 They were easy to revert, understand, debug, and critique. s154 00:13:48.720 --> 00:13:52.120 So, can models do this natively now? s155 00:13:52.120 --> 00:13:53.080 Not quite. s156 00:13:53.080 --> 00:14:02.120 They're missing four things which we'll we'll go over and there's something that doesn't go away though as models get bigger and more capable, the prior. s157 00:14:02.120 --> 00:14:08.560 Just like humans, models bring their own priors, their beliefs about what should be in the world to every task. s158 00:14:08.560 --> 00:14:14.400 This is the fourth schema that I mentioned before, the one the model brings with it. s159 00:14:14.840 --> 00:14:18.840 Bigger models in a strange way are worse at this sometimes. s160 00:14:18.840 --> 00:14:22.880 They hold onto their beliefs and argue for them even more vehemently. s161 00:14:22.880 --> 00:14:24.920 Here's a few that we ran into. s162 00:14:24.920 --> 00:14:29.000 One pipeline looked for in kind spelled with an underscore. s163 00:14:29.000 --> 00:14:32.280 The file spelled it with a capital I and a hyphen. s164 00:14:32.280 --> 00:14:39.240 So, the model counted 15,658 in kind contributions as ordinary money, didn't double-check. s165 00:14:39.240 --> 00:14:42.560 The loan payment repayments also disappeared the same way. s166 00:14:42.560 --> 00:14:47.080 Another one tried four date formats when the file used the fifth. s167 00:14:47.080 --> 00:14:55.760 99.4% of transaction dates then came out empty and every row count and dollar so matched so the agent moved on. s168 00:14:55.760 --> 00:15:03.320 Several runs noticed that loan amounts were negative, decided that the sign was redundant, and stored the absolute value. s169 00:15:03.320 --> 00:15:12.160 A net of 784,000 became 5.3 million dollars of debt in the end, validated by multiple models. s170 00:15:12.160 --> 00:15:17.640 And this is quickly becoming the largest class of error I see across companies at data work as models take over. s171 00:15:17.640 --> 00:15:23.080 Sometimes you can solve this just by not vibing, by checking results and applying human intuition. s172 00:15:23.080 --> 00:15:26.960 And what you're actually doing there when you do this is interesting. s173 00:15:26.960 --> 00:15:30.640 You're intersecting your priors with the models. s174 00:15:30.640 --> 00:15:37.640 In that first example, any human would catch missing in-kind contributions if they knew campaign finance or thing to check. s175 00:15:37.640 --> 00:15:38.920 So, we found it. s176 00:15:38.920 --> 00:15:42.800 But humans, speaking as a human on this particular task, are expensive. s177 00:15:42.800 --> 00:15:50.000 They are slow, they pause timelines, they're even more non-deterministic than models, and they hate interruptions. s178 00:15:50.000 --> 00:15:54.880 And what it turns out you can also get really far with cross-model prior intersection. s179 00:15:54.880 --> 00:16:00.760 On our 200 held out families, Jev and Luna agreed on 194. s180 00:16:00.760 --> 00:16:03.040 And none of those shared answers was wrong. s181 00:16:03.040 --> 00:16:06.640 Every error sat in the six families where they disagree. s182 00:16:06.640 --> 00:16:13.680 Models from different families, different pedigrees, tend to disagree more and have different priors than models in the same group, for example. s183 00:16:13.680 --> 00:16:20.400 The exception, I'm not going to say who, are models that are distilled outside the same company. s184 00:16:20.400 --> 00:16:31.760 So, with that kind of work, you can shrink the surface quite a bit, and the errors along with it down to a much smaller but persistent class of problems that apply to LLMs as a species. s185 00:16:31.760 --> 00:16:39.960 So, in one benchmark, four different model families produced the same total down to this end because they all shared a sign convention that the answer key didn't. s186 00:16:39.960 --> 00:16:47.320 And once you get that far, the only way forward is to list them, look out for them, and apply specific solutions. s187 00:16:47.600 --> 00:16:53.080 So, going back, what are the four things we're missing to be able to self-optimize? s188 00:16:53.080 --> 00:16:56.080 Number one is a stable deployment surface. s189 00:16:56.080 --> 00:17:10.280 We use Hank Weave for this, a harness that takes the fast-moving surface of agent runtimes like cloud code and pie, and then exposes a consistent, well-tested log and API interface with checkpointing, rollback, budget enforcement, everything built in. s190 00:17:10.280 --> 00:17:12.160 Something you can build on. s191 00:17:12.160 --> 00:17:14.680 But other things fit this description. s192 00:17:14.680 --> 00:17:20.439 To be fair, for headless data work, Hank Weave, I would say, still the oldest, most stable, and the most open. s193 00:17:20.439 --> 00:17:23.240 I'll leave the skill up if you'd like to try it. s194 00:17:23.240 --> 00:17:26.400 Number two is some way to bound your experiments. s195 00:17:26.400 --> 00:17:31.880 You can't test unless you can control cost, time, and access as resources. s196 00:17:31.880 --> 00:17:35.440 The third one is know-how and methodology. s197 00:17:35.440 --> 00:17:44.920 We found that this new class of models can take everything we've learned from two years of deployment, agentic patterns, debugging techniques, prompting, and apply to complex problems. s198 00:17:44.920 --> 00:17:50.880 With the right abstraction layer, you can express it very token efficiently, and the models can implement it. s199 00:17:50.880 --> 00:17:54.600 The fourth and the most important one is a failure surface. s200 00:17:54.600 --> 00:17:58.240 Archimedes said, "Give me a place to stand, and I will move the Earth." s201 00:17:58.240 --> 00:18:08.680 And in our case, we haven't met a problem or a constraint that we couldn't iterate ourselves to solving given a stable stack to build on and a failure surface to iterate against. s202 00:18:08.680 --> 00:18:15.240 That last one has been the hardest to put into words and into tooling and instructions that helps models get there. s203 00:18:15.240 --> 00:18:21.560 The current best models are still only barely good at this, sparks of intelligence, so to speak. s204 00:18:21.560 --> 00:18:24.400 The whole process though involves going through six stages. s205 00:18:24.400 --> 00:18:26.560 First, we inspect the job and we split it. s206 00:18:26.560 --> 00:18:35.080 For Jeff, that means splitting some 800,000 rows into 200,000 smaller, specialized cases, and each of those cases into typed questions. s207 00:18:35.080 --> 00:18:40.440 What kind of donor, one donor, you know, different different questions that needed to get answered. s208 00:18:40.440 --> 00:18:42.360 Then we decide the failure surface. s209 00:18:42.360 --> 00:18:50.480 In this case, we ran good models on a small slice of the data with some of it being manually checked to set a baseline that we can fail against. s210 00:18:50.480 --> 00:19:04.680 In other cases, we've set automated failure systems, we've set schema checks, we've also covered behavior as a thing that we can measure against, but it's very important to have to define a failure surface before you ever start. s211 00:19:04.680 --> 00:19:06.000 But now comes the fun part. s212 00:19:06.000 --> 00:19:10.240 We want minimal bounded pilots to test architecture. s213 00:19:10.240 --> 00:19:13.680 And once something starts working, we move those tags into rigs. s214 00:19:13.680 --> 00:19:17.840 Code that runs between bounded work segments that we call codons. s215 00:19:17.840 --> 00:19:24.480 These codons are then reused, looped independently, validated, and placed in sequence. s216 00:19:24.920 --> 00:19:29.880 Finally, working pipelines get deployed to production, where they can slowly mature. s217 00:19:29.880 --> 00:19:33.280 And that middle part is where the surfing happens. s218 00:19:33.280 --> 00:19:42.720 We find an architecture that gives a solid result, we use it as a baseline, and we swap models and some other parts for faster and cheaper options, compare against known intermediates, s219 00:19:42.720 --> 00:19:49.720 tighten the requirement for success, each time moving closer and closer to the ideal, and you just watch the frontier move. s220 00:19:49.720 --> 00:19:58.600 Being able to break a complex problem into pieces means each piece can move its own to its own best position, as Adam Smith would say. s221 00:19:58.600 --> 00:20:06.880 In a system like this, ontologies exist as intermediates between intelligence and result, which makes them easier to reuse. s222 00:20:06.880 --> 00:20:10.320 That's the other half of what I mean by just-in-time ontologies. s223 00:20:10.320 --> 00:20:17.120 It's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing. s224 00:20:17.120 --> 00:20:20.120 And once you have those, you can make these things cheaper. s225 00:20:20.120 --> 00:20:24.360 You can you can run them in parallel across data sets to scale up. s226 00:20:24.360 --> 00:20:26.320 I close with an apology, right? s227 00:20:26.320 --> 00:20:29.360 That the answer is there is no ontology. s228 00:20:29.360 --> 00:20:31.080 Not a universal one, at least. s229 00:20:31.080 --> 00:20:32.520 There never was one. s230 00:20:32.520 --> 00:20:39.760 The good news is that this makes every ontology and schema valid, as long as you can label and preserve the context around it. s231 00:20:39.760 --> 00:20:51.080 And if we can harness this cheap intelligence is starting to percolate through the world around us, we're rapidly approaching a future where any information, in any format, is made as useful as it can be. s232 00:20:51.080 --> 00:20:55.480 And at Outpace, we're working as hard as we can to make that future a reality. s233 00:20:55.480 --> 00:20:57.680 Thank you.