WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=3ZMUiFaQ3qg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/3ZMUiFaQ3qg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.560 --> 00:00:14.360 Thank you to the organizers for having me. s2 00:00:14.360 --> 00:00:17.200 I'm one of the co-founders and CTO at Latch. s3 00:00:17.200 --> 00:00:23.680 We are basically a vertical AI lab for benchmark and agent engineering. s4 00:00:23.680 --> 00:00:28.480 I'm hoping to motivate and explain exactly what that means today. s5 00:00:28.480 --> 00:00:33.680 Starting directly with motivation for agents in in bio generally. s6 00:00:33.680 --> 00:00:42.600 Um Many people in my domain are familiar with this curve, but this is basically the log linear curve of data generated over the years in in biology. s7 00:00:42.600 --> 00:00:46.720 And the reason I'm bringing it up it will become directly important to the kinds of things we want to do in engineering. s8 00:00:46.720 --> 00:00:52.440 Um This curve is driven by a very small handful of experimental classes. s9 00:00:52.440 --> 00:00:54.400 One is called single cell biology. s10 00:00:54.400 --> 00:00:58.480 This is where we split up cells, break them apart, and measure their RNA. s11 00:00:58.480 --> 00:01:02.440 The second is spatial biology, which will become the focus of the next segment of the talk. s12 00:01:02.440 --> 00:01:04.839 Same thing as single cell, but you get spatial resolution. s13 00:01:04.839 --> 00:01:08.560 You can look at how RNA is spread out geometrically over a tissue. s14 00:01:08.560 --> 00:01:10.160 And the third thing is proteomics. s15 00:01:10.160 --> 00:01:11.640 It's a broad category of different techniques. s16 00:01:11.640 --> 00:01:13.000 They measure proteins. s17 00:01:13.000 --> 00:01:15.840 Um Less less abundant in ordering. s18 00:01:15.840 --> 00:01:21.160 Like less less data volume generated relative to the other two, but still important. s19 00:01:21.160 --> 00:01:33.080 Um you guys are technical and I always think it's good to ground things somewhat quantitatively, but um these are really big numbers and the experimental data from these techniques is growing quite rapidly. s20 00:01:33.080 --> 00:01:39.480 Um Almost greater than any other domain of science other than particle collider um machines. s21 00:01:39.480 --> 00:01:43.400 Single cell experiments can yield two to six terabytes per run. s22 00:01:43.400 --> 00:01:45.920 Spatial runs can yield seven terabytes of run. s23 00:01:45.920 --> 00:01:48.000 Proteomics a few hundred gigs. s24 00:01:48.000 --> 00:01:58.080 Um and the only reason I bring this up is to say hey like the output of a single experiment can exceed what a scientist can safely store on a consumer laptop in many cases. s25 00:01:58.080 --> 00:02:05.200 And the laws driving how the the molecular capture works point to rapid gains in this throughput over the coming years. s26 00:02:05.200 --> 00:02:13.080 Um one thing I like to do uh is when I read a new paper is decompose it and align it to this framework because it will be important in a second. s27 00:02:13.080 --> 00:02:17.040 Uh modern biology research is centered around those experiments. s28 00:02:17.040 --> 00:02:21.920 You basically choose a model biological model, not the kind of models you guys are used to. s29 00:02:21.920 --> 00:02:30.640 You generate data from that model, you process the data, you creatively think about the results in the context of prior literature, and you make a claim. s30 00:02:30.640 --> 00:02:36.120 Almost all modern experiments papers that you see published follow this loose structure with a lot of nuance. s31 00:02:36.120 --> 00:02:40.560 Um all all all that to say is they be they become something of a a panning experiment. s32 00:02:40.560 --> 00:02:44.120 You're looking for a signal using measurement in a sea of noise. s33 00:02:44.120 --> 00:02:50.160 Um and so this is building up to the claim that like code and sweet data analysis scaffold to genetic biology. s34 00:02:50.160 --> 00:02:54.760 It becomes this executable substrate that we can use to train things. s35 00:02:54.760 --> 00:02:58.240 It introduces a natural way to benchmark and climb capability. s36 00:02:58.240 --> 00:03:00.600 Um I've written about this a lot at this blog. s37 00:03:00.600 --> 00:03:02.080 Uh link here. s38 00:03:02.080 --> 00:03:06.320 There's a lot more depth to this claim, so I wouldn't take it at face value, but it's something to look into. s39 00:03:06.320 --> 00:03:17.160 All you can take away from this is like just like code provided a verifiable substrate for complex software tasks that are not inherently verifiable, uh data analysis might do the same thing in bio. s40 00:03:17.160 --> 00:03:18.440 So how do we get started? s41 00:03:18.440 --> 00:03:24.360 We were originally a data tool vendor for biotech and pharma. s42 00:03:24.360 --> 00:03:28.400 Um where we started 5 years ago out of Berkeley on 25. s43 00:03:28.400 --> 00:03:29.720 We started when I was 20. s44 00:03:29.720 --> 00:03:33.680 We stored, transformed, filed data from large experiments as a service. s45 00:03:33.680 --> 00:03:36.840 Uh we'll try to build products, explored lots of things. s46 00:03:36.840 --> 00:03:45.000 Over the last 2 years, um we started moving away from biotech and pharma and more towards the people who build those kits I was talking about. s47 00:03:45.000 --> 00:03:51.160 Package the software into kind of white-labeled things that they s- provided to the scientists themselves. s48 00:03:51.160 --> 00:03:53.840 Um Help them analyze their data. s49 00:03:53.840 --> 00:04:03.840 And then over time, there became this strong interaction uh with the agents uh using the infrastructure components as tools and the loop context you guys are familiar with. s50 00:04:03.840 --> 00:04:05.920 Except in our domain, the tools can take days or weeks. s51 00:04:05.920 --> 00:04:06.959 I'm serious. s52 00:04:06.959 --> 00:04:13.640 Um What started to happen around last summer is agent prototypes uh started to work. s53 00:04:13.640 --> 00:04:21.239 So, we took uh coding models uh to our knowledge at the time they were not seriously post-trained on any tasks in biology to this point. s54 00:04:21.239 --> 00:04:25.280 And we started to build products that look a lot like all the other agent products. s55 00:04:25.280 --> 00:04:32.840 They have a chat interface for you to ask questions to and they build dashboards and dispatch operations to external compute. s56 00:04:32.840 --> 00:04:39.600 Um the kinds of things these that this this agent would do is take large file data from the types of experiments I was talking about earlier. s57 00:04:39.600 --> 00:04:45.600 Say like uh tissue bisect biopsy from cancer with a spatial measurement. s58 00:04:45.600 --> 00:04:51.680 And then the the scientist is just iterating with it to get at some question they have. s59 00:04:51.680 --> 00:04:58.040 You know, maybe between a malignant and non-malignant part of the tissue, what kind of genes are being overexpressed? s60 00:04:58.040 --> 00:05:04.320 Um but what was fascinating is even though it was pretty bad, it showed the early signs of working. s61 00:05:04.320 --> 00:05:08.840 And uh It became clear to us at this time that agentic biology might look a lot like code. s62 00:05:08.840 --> 00:05:21.840 Actually lifted this slide from Anthropic's um Claude science announcement yesterday, but we uh just like, you know, you had this like kind of faulty silly engineer that became better and then as it improved in capability, s63 00:05:21.840 --> 00:05:24.800 you could dispatch work to teams of them that work together. s64 00:05:24.800 --> 00:05:27.720 The same pattern will probably emerge in science. s65 00:05:27.720 --> 00:05:36.360 And products and harnesses will emerge to orchestrate uh work and abstract it so that teams of teams of agentic scientists can take on capability. s66 00:05:36.360 --> 00:05:38.800 But we needed focused post-training. s67 00:05:38.800 --> 00:05:42.560 Uh cuz at the time and still now, um frontier models cannot be trusted to do real work. s68 00:05:42.560 --> 00:05:45.800 They're missing some capability between knowing biology and writing code. s69 00:05:45.800 --> 00:05:49.760 And this is exactly extracting scientific insight from real-world data. s70 00:05:49.760 --> 00:05:57.000 Unlike code, it which is one constituent component of this work, uh it also involves data analysis and domain reasoning, scientific reasoning. s71 00:05:57.000 --> 00:06:00.280 We thought spatial biology is a good place to start, so we started building agents. s72 00:06:00.280 --> 00:06:01.800 This is a technical green field. s73 00:06:01.800 --> 00:06:03.840 We had many existing customers. s74 00:06:03.840 --> 00:06:07.080 Uh it's also just a beautiful example of measurement drives progress. s75 00:06:07.080 --> 00:06:10.480 You can actually see biological phenomena play out. s76 00:06:10.480 --> 00:06:13.400 Um you can look at a developing mouse embryo. s77 00:06:13.400 --> 00:06:19.960 Um And we had to get in the guts of how the data was captured and analyzed to build good agents. s78 00:06:19.960 --> 00:06:28.720 Um I'm not going to get into this in detail, but I put up this tree of uh different capture technologies in spatial biology to highlight the diversity of things that exist. s79 00:06:28.720 --> 00:06:34.200 They really span advances in chemistry, optics, semiconductors, physics. s80 00:06:34.200 --> 00:06:41.600 Each branch uh is induced from decades of cumulative work to figure out how to measure a a type of molecule. s81 00:06:41.600 --> 00:06:55.520 As a specific example, one technique we work with is called sequencing base spatial, and it's where you take a slide of little beads with clumps of DNA attached to them that fuse to the RNA inside of a tissue section, so s82 00:06:55.520 --> 00:07:04.800 biologists can like lay like a chunk of a tumor over it, and then it'll capture all the RNA in it, and then let you know with precise uh geometric resolution where the RNA was. s83 00:07:04.800 --> 00:07:05.880 It's cool stuff. s84 00:07:05.880 --> 00:07:11.960 The data, when it comes out, ends up looking like a big matrix of numbers in a large high-content image. s85 00:07:11.960 --> 00:07:16.720 Um you have to take it through a sequence of steps uh to get to the end thing that you want. s86 00:07:16.720 --> 00:07:22.120 These steps are highly variable, especially across technology types, tissue, disease contexts. s87 00:07:22.120 --> 00:07:30.480 Um there isn't a lot of consensus in the field uh for each step, so we really needed a measuring stick to understand if the agents we're building were doing scientific work. s88 00:07:30.480 --> 00:07:37.040 Uh the existing benchmarks we saw at the time did not measure the tasks relevant to this category of work. s89 00:07:37.040 --> 00:07:43.800 Um they mostly measured things in a Q&A settings, like what what do in a kind of academic way, or they weren't sufficiently focused on the experiment type. s90 00:07:43.800 --> 00:07:49.920 This is an actual screenshot of Anthropic's model card at the time that we built this benchmark. s91 00:07:49.920 --> 00:07:51.320 So, we built one. s92 00:07:51.320 --> 00:07:52.840 It's called spatial bench. s93 00:07:52.840 --> 00:07:55.680 Um last December, there's 146 problems. s94 00:07:55.680 --> 00:07:59.520 They spanned the different kits I talked about or attempted to. s95 00:07:59.520 --> 00:08:03.160 And then they spanned all those different tasks that I talked about as well. s96 00:08:03.160 --> 00:08:12.760 So, the thing that we found at this time, and still to an extent is true today, is the the grading of these end outcomes in biology uh is too sparse because the models are pretty bad. s97 00:08:12.760 --> 00:08:19.360 So, you have to break things up into manageable chunks to get some semblance of verifiability. s98 00:08:19.360 --> 00:08:25.960 And that's kind of induced by um sticking to these little components of like that DAG, that analysis DAG. s99 00:08:25.960 --> 00:08:34.560 Um getting data to a state where it would exist right before a scientist or theoretician could do work on it, and then um figuring out what the ground truth would be in that context. s100 00:08:34.560 --> 00:08:41.200 So, a single evaluation kind of looks like one or more data nodes, again like a matrix of numbers, high content image, something like this. s101 00:08:41.200 --> 00:08:48.520 A task prompt carefully describing some scientific goal, configuration for a grader, and then a deterministic grader, so a Python function. s102 00:08:48.520 --> 00:08:50.320 If you guys notice, this looks a lot like SweetBench. s103 00:08:50.320 --> 00:08:53.560 We borrowed a lot of the early ideas and tried to extend them as much as possible. s104 00:08:53.560 --> 00:08:56.440 Evaluation ends up looking like this. s105 00:08:56.440 --> 00:08:58.040 It's a lot of JSON. s106 00:08:58.040 --> 00:09:02.480 And we ended up identifying properties of like what we thought good biological tests were. s107 00:09:02.480 --> 00:09:06.720 Um little little different from code, and we built on these over time, but they still hold up. s108 00:09:06.720 --> 00:09:08.120 They got to be verifiable. s109 00:09:08.120 --> 00:09:11.160 You have to be able to check the success condition with a function. s110 00:09:11.160 --> 00:09:12.680 Um nothing's changed there. s111 00:09:12.680 --> 00:09:15.560 We'll get into some rubric stuff later, but still holds. s112 00:09:15.560 --> 00:09:17.320 Durability is particularly important. s113 00:09:17.320 --> 00:09:19.760 Science does not admit clear ground truth. s114 00:09:19.760 --> 00:09:28.880 Um if you are lazy with your ground truth construction of the task, a possible valid analysis path can come with the correct answer, um and you'll fail it uh incorrectly. s115 00:09:28.880 --> 00:09:33.000 So, you got to make sure you're reasoning about something that's somehow invariant across analysis paths. s116 00:09:33.000 --> 00:09:35.760 And then obviously, we're we're working with the genetics stuff here. s117 00:09:35.760 --> 00:09:38.880 You don't want the model to answer the question in one turn. s118 00:09:38.880 --> 00:09:42.720 You You want the conclusion to require interaction with the data, not some memorized knowledge. s119 00:09:42.720 --> 00:09:44.200 In practice, that's pretty difficult. s120 00:09:44.200 --> 00:09:52.520 We learned a lot about what models could do and which ones to use in specific context for this category of work for our customers, and we thought, "Hey, this is pretty cool. s121 00:09:52.520 --> 00:09:56.120 Let's Let's start to improve and learn more about this benchmarking problem." s122 00:09:56.120 --> 00:09:59.320 So, we jumped to human verification long horizon extension. s123 00:09:59.320 --> 00:10:00.520 I'm going to quickly breeze through these. s124 00:10:00.520 --> 00:10:03.320 So, human verification is incredibly important in science. s125 00:10:03.320 --> 00:10:05.720 Science is not in the clear ground truth. s126 00:10:05.720 --> 00:10:16.760 Uh after watching trajectory data from multiple rounds of model releases, circa like January to March of this year, um we really realized a lot of our assumptions were pretty bad. s127 00:10:16.760 --> 00:10:25.200 Um and in the absence of like a canonical answer, uh having a bunch of scientists grade each other's work ended up being like the best proxy. s128 00:10:25.200 --> 00:10:29.720 So, I'm going to look at one one issue to highlight exactly what I'm talking about. s129 00:10:29.720 --> 00:10:31.880 Um this problem ambiguity. s130 00:10:31.880 --> 00:10:41.440 A task might ask an agent to split a gene list into two groups of activity, microglial activation, oligodendrocyte inflammation, just like biological categories of things. s131 00:10:41.440 --> 00:10:50.560 Score the cells, find neighboring oligodendrocytes around some region using an appropriate radius, compute a Spearman correlation at two time points. s132 00:10:50.560 --> 00:10:55.920 Uh as you can probably clearly deduce, the original problem statement creates a host of open choices. s133 00:10:55.920 --> 00:10:57.720 How do you split the gene list? s134 00:10:57.720 --> 00:10:59.760 How do you count what inflammatory genes are? s135 00:10:59.760 --> 00:11:01.280 It's like somewhat ambiguous word. s136 00:11:01.280 --> 00:11:02.880 How do you normalize the data? s137 00:11:02.880 --> 00:11:05.320 Um what what what the hell is an appropriate radius? s138 00:11:05.320 --> 00:11:07.880 How do you pull the counts within the selected radius? s139 00:11:07.880 --> 00:11:16.200 Um These are all problems that pointed to tasks that were bad, that only became revealed with human verification. s140 00:11:16.200 --> 00:11:22.720 Another issue is just like a lot of people in bioinformatics canonically have used like numerical thresholds to QC stuff. s141 00:11:22.720 --> 00:11:24.600 Just like completely arbitrary stuff. s142 00:11:24.600 --> 00:11:31.080 Um cool thing about evaluation like coding is it forces you to reason about things more rigorously than you would when you're doing the thing yourself. s143 00:11:31.080 --> 00:11:38.960 If you have to teach a machine to do it, uh you you might be picking out some structure that's more important or more durable than what you were doing if if you're just doing it on your own. s144 00:11:38.960 --> 00:11:42.280 So, we just found a lot of these numerical thresholds to be like that. s145 00:11:42.280 --> 00:11:44.120 Uh I'm not going to get into this. s146 00:11:44.120 --> 00:11:48.240 Um after two rounds of human attempts, we produce a verified subset of the benchmark. s147 00:11:48.240 --> 00:11:49.800 Uh we we publish it. s148 00:11:49.800 --> 00:11:51.120 That was fun. s149 00:11:51.120 --> 00:11:53.240 Uh and then we also tried to increase the time horizon. s150 00:11:53.240 --> 00:11:59.400 So, I want to be clear the frontier of knowledge is still not quite there with biology. s151 00:11:59.400 --> 00:12:04.920 Like the labs are starting to catch up with the post training, but we kind of want to stay ahead. s152 00:12:04.920 --> 00:12:11.520 Um so, we built a benchmark that we thought would recapitulate like really difficult true work. s153 00:12:11.520 --> 00:12:13.880 Um so, we built a space much longer. s154 00:12:13.880 --> 00:12:15.760 Um real biological tasks are messy. s155 00:12:15.760 --> 00:12:18.120 They use lots of different experiment types. s156 00:12:18.120 --> 00:12:21.120 They are use the whole workflow, don't use little chunks. s157 00:12:21.120 --> 00:12:30.400 They are uh tasks every step is interpreted against the experimental design or contextualized with some prior literature and the original goal of what you're doing in the first place. s158 00:12:30.400 --> 00:12:41.000 Um so, we built a bank of these tasks that are really trying to simulate the result sections of entire papers or the kinds of decisions you make in practice in industry to make a go no go decision on a drug program. s159 00:12:41.000 --> 00:12:46.040 Um these got these tasks took like a week for a group of three people to make each. s160 00:12:46.040 --> 00:12:47.600 Taught us a bunch of stuff. s161 00:12:47.600 --> 00:12:54.440 Um example is can an agent reconstruct a like metastatic niche in a tumor? s162 00:12:54.440 --> 00:13:09.440 Um if you have like a tumor biopsy and a bunch of metastatic biopsies from like where it metastasized and spread across the body, can it use like both the genetics and mRNA of the metastatic lesions and the tumor to like find the part of the tumor that initially seeded the metastatic growth and let it spread? s163 00:13:09.440 --> 00:13:12.920 From that you can figure out like, "Hey, what parts of the tumor are more like genetically fit? s164 00:13:12.920 --> 00:13:14.760 Which ones actually cause problems?" s165 00:13:14.760 --> 00:13:18.000 And construct targeted medicines to nip them in the bud. s166 00:13:18.000 --> 00:13:22.400 Like for example, this is one of the benchmark uh evals in the long horizon set. s167 00:13:22.400 --> 00:13:23.560 None of the models get this right. s168 00:13:23.560 --> 00:13:24.920 Um but they're getting there. s169 00:13:24.920 --> 00:13:31.080 As As imagine with these long horizon extensions, uh verifiable reward at the end are like somewhat uninformative. s170 00:13:31.080 --> 00:13:35.040 So, we we're starting to play with rubrics, uh constructing these choke points. s171 00:13:35.040 --> 00:13:38.960 If you can imagine like the set of analysis paths is inducing some sort of tree. s172 00:13:38.960 --> 00:13:49.040 Um there are nodes that are invariant with respect to yeah, different paths, and you can use these to build rubrics um using knowledge of how the task work. s173 00:13:49.040 --> 00:13:51.720 Uh we we we're playing with these. s174 00:13:51.720 --> 00:14:06.240 We noticed that um they're associated with the verifiable outcomes, uh which is exciting, but they're loosely correlated numerically, um making us not fully have confidence in them for things like RL or benchmarking. s175 00:14:06.240 --> 00:14:09.400 Um a lot of lot more work to do here still. s176 00:14:09.400 --> 00:14:19.120 Uh we we still strongly believe the verifiability structure is what's going to carry in- intelligence uh a bit a bit longer. s177 00:14:19.120 --> 00:14:25.400 And so, these days uh excitingly, we've been expanding um from this initial spatial focus. s178 00:14:25.400 --> 00:14:30.080 Uh really cool to see the frontier labs and community adopt these benchmarks organically. s179 00:14:30.080 --> 00:14:36.800 Um we had this interesting position by like, you know, building and shipping products uh early and kind of playing with the coding agents. s180 00:14:36.800 --> 00:14:45.760 So, I think we just had uh early advantage, but the benchmarks are now in like the recent Anthropic model cards, and uh this is a picture from yesterday. s181 00:14:45.760 --> 00:14:49.400 Eric just showing the benchmarks um at their cloud science launch. s182 00:14:49.400 --> 00:14:50.480 They don't tell us this happens. s183 00:14:50.480 --> 00:14:54.440 They just like do it, um and you like read about it, and it's cool. s184 00:14:54.440 --> 00:15:07.040 Uh we published a bunch more papers um beyond spatial to other omics classes, so other experiment types, single-cell, epigenomics, so RNA, and then the bit above the the DNA, s185 00:15:07.040 --> 00:15:09.520 and then long horizon extensions of these things. s186 00:15:09.520 --> 00:15:16.720 Um and then we're starting to index and measure the very gnarly complex landscape that is drug discovery. s187 00:15:16.720 --> 00:15:27.960 We just put out our first benchmark on um preclinical pharmacology for small molecules, and then systematically biting off pieces of the program landscape from discovery to development to translation, s188 00:15:27.960 --> 00:15:33.760 strati- stratifying it by um therapeutic types and experiment types. s189 00:15:33.760 --> 00:15:39.560 We just acquired a uh company building in biosecurity to form a biosecurity team. s190 00:15:39.560 --> 00:15:44.760 And then we just put out a collaboration with American Wetware and a surveillance company called Aquid. s191 00:15:44.760 --> 00:15:48.360 Um some new work that was first released this morning. s192 00:15:48.360 --> 00:15:53.000 I don't know if you guys have been hearing uh fuzzles kind of suck in biology right now. s193 00:15:53.000 --> 00:15:58.640 If you ask Fable basic questions about like mitochondria, it'll won't answer. s194 00:15:58.640 --> 00:15:59.920 It's kind of stupid. s195 00:15:59.920 --> 00:16:02.920 So, I mean this is just like an evaluation problem. s196 00:16:02.920 --> 00:16:07.480 There's a lot There's a lot more nuance to this, but I just use that cuz people tend to recognize it. s197 00:16:07.480 --> 00:16:18.160 Uh where we build routine tasks that simulate the kinds of things a scientist would ask for, and then more sinister red team tasks, which are supposed to look innocuous but have some structure s198 00:16:18.160 --> 00:16:23.600 that is bad, like, "Hey, I want to clone a gene into a bacteria, and I'm telling you it's GFP." s199 00:16:23.600 --> 00:16:30.400 It's like a glowing protein, but in reality it's like a toxin um or it could be used to bootstrap a virus. s200 00:16:30.400 --> 00:16:39.880 We found that the routine tasks like drastic get drastic used drastically more frequently than the red team tasks, which is uh not great. s201 00:16:39.880 --> 00:16:49.440 Um we're aggregating a lot of these results, uh essential resource along with all the preprints and a lot of evals and trajectories for you guys to check out. s202 00:16:49.440 --> 00:16:51.400 Um and I actually did okay on time. s203 00:16:51.400 --> 00:16:52.480 Let's go. s204 00:16:52.480 --> 00:16:53.200 Uh and that's it. s205 00:16:53.200 --> 00:17:02.640 So, we we are kind of like I don't even I hate the word lab, but we're kind of like a research lab for bio, and we do research and deployment of these agents. s206 00:17:02.640 --> 00:17:04.520 So, we we still have like a lot of customers. s207 00:17:04.520 --> 00:17:10.319 Um we work with the kit manufacturers, and we use that to inform what kinds of things we make benchmarks for. s208 00:17:10.319 --> 00:17:15.360 We try to get the labs to compete um on the benchmarks cuz then it makes the models better at our products. s209 00:17:15.360 --> 00:17:18.160 And it's a It's been a pretty rewarding flywheel. s210 00:17:18.160 --> 00:17:21.280 A lot of growth, and we're hiring aggressively across engineering and science. s211 00:17:21.280 --> 00:17:23.800 So, if you're interested in this work, please find me afterwards. s212 00:17:23.800 --> 00:17:25.920 Thank you. s213 00:17:39.802 --> 00:17:41.802 [music]