WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=OXMMN-XbxwA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/OXMMN-XbxwA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.040 --> 00:00:06.640 Hey, I'm Vitis and I'm a senior research engineer at Hyark on our labs team. s2 00:00:06.640 --> 00:00:22.800 So, at Hyark our labs team is basically our research and development arm where we have machine learning researchers who kind of explore the frontier of AI/ML uh and figure out ways to apply it to home building. s3 00:00:26.920 --> 00:00:34.520 Now, because we're in home building and we're in spatial reasoning, uh we pretty much end up needing to use a lot of what's out there in AI. s4 00:00:34.520 --> 00:00:41.920 So, computer vision to scan hand-sketched floor plans and parse them into our internal data model, s5 00:00:41.920 --> 00:00:42.000 [music] s6 00:00:42.000 --> 00:00:58.800 um reasoning agents to kind of carry the user through these agentic experiences, we have custom transformers, uh we use diffusion models for image gen, uh and basically a lot of kind of what's out there in AI/ML s7 00:00:58.800 --> 00:01:06.320 uh we end up needing to use because of the kind of multidisciplinary nature of our product. s8 00:01:06.320 --> 00:01:13.400 So, hopefully that video gives you some idea of the research areas that we focus on that I've mapped out here. s9 00:01:13.400 --> 00:01:24.760 And that kind of leads us to uh the problem, which is in this in this challenge of getting frontier research into production, uh we need to start working with s10 00:01:24.760 --> 00:01:41.320 software engineers, let's say platform engineers, infrastructure engineers, back-end engineers who are very familiar with building robust and production-grade code, but are likely not familiar with the methodology and research s11 00:01:41.320 --> 00:01:49.680 in computer vision, in training your own LLMs, uh and even some top secret uh topics that I can't reveal here. s12 00:01:49.680 --> 00:02:02.760 Now, you kind of have the flip side problem with our uh ML researchers who are very up-to-date with the latest papers and can uh pull together these concepts in novel and creative ways to develop new features, s13 00:02:02.760 --> 00:02:11.640 but they have not really worked as software engineers typically where they've been responsible for production grade APIs. s14 00:02:11.640 --> 00:02:22.680 So, that's kind of what I want to get into is this this handoff, this this baton pass how do we facilitate that and how do we do it productively? s15 00:02:22.680 --> 00:02:35.440 And we look at this basically as a systems and process problem and I want to kind of zero in on three main focus areas that you can use to s16 00:02:35.440 --> 00:02:42.640 improve this the velocity of teams that that are bringing research into production. s17 00:02:42.640 --> 00:02:45.840 So, the first thing we'll look at is research legibility. s18 00:02:45.840 --> 00:02:52.080 So, let's say you have a ML researcher has produced a prototype concept. s19 00:02:52.080 --> 00:03:02.560 How can they map that out and make it digestible and understandable for these different software engineers and product managers who are going to be jumping into the project. s20 00:03:02.560 --> 00:03:05.720 The second is how to structure your code base. s21 00:03:05.720 --> 00:03:07.800 So, we use a mono repo. s22 00:03:07.800 --> 00:03:17.880 How do you arrange your code and modularize it so that it's ready to receive these new prototype concepts and turn them around and stand them up quickly. s23 00:03:17.880 --> 00:03:28.720 And then the third thing we'll look at is basically, you know, making the jump from a mapped out proven out research prototype that's going to be going into this s24 00:03:28.720 --> 00:03:29.680 repo. s25 00:03:29.680 --> 00:03:37.680 How do you decompose that prototype and stand it up on best practices in software engineering. s26 00:03:37.680 --> 00:03:47.519 So, taking a look at the first step, there's a really good article that I that I want to link a blog from the the pragmatic engineer where he talks about s27 00:03:47.519 --> 00:04:00.280 software engineering teams as they grow and start to scale how important it is to write out technical design documents, request for comment, whatever you want to call it, specifications s28 00:04:00.280 --> 00:04:04.560 before building software as a way to align teams. s29 00:04:04.560 --> 00:04:16.160 So, we have a very analogous document um that we require from all research prototypes that we call the research prototype taxonomy document. s30 00:04:16.160 --> 00:04:30.040 So, it's really just a technical design document from software engineering uh with some specific twists uh that um make it more digestible given its its nature in machine learning. s31 00:04:30.040 --> 00:04:41.160 So, the first thing in that document, we use notion for this, but obviously any written document will work, is that we start with the kind of domain context and like what are the domain specific s32 00:04:41.160 --> 00:04:44.160 you know, we're in the architectural domain and home building. s33 00:04:44.160 --> 00:04:47.400 So, what are these kind of novel ways to represent data? s34 00:04:47.400 --> 00:05:00.760 Maybe it's a party diagram, maybe it's a graph to to represent the kind of circulation graph through a home, maybe it's embedding models or latent space representations. s35 00:05:00.760 --> 00:05:06.640 I like to say kind of picture a software engineer who just we just hired from JP Morgan. s36 00:05:06.640 --> 00:05:14.240 What are the kind of uh specific lingos and and data representations that they might need to know before jumping into this project. s37 00:05:14.240 --> 00:05:17.400 The second is kind of mapping out the business goal, right? s38 00:05:17.400 --> 00:05:23.560 What's the why does solving this problem matter and what's the value in this ML tool? s39 00:05:23.560 --> 00:05:32.640 And then the four uh remaining parts of this document are pretty kind of conventional software engineering principles you might see in a TDD. s40 00:05:32.640 --> 00:05:46.840 So, the type safety, so we'll see later what our machine learning repo looks like, but what is the type contract between our core product repository and this machine learning repo? s41 00:05:46.840 --> 00:05:50.480 How are those types shared and how do they stay in sync? s42 00:05:50.480 --> 00:05:53.880 Um then kind of mapping out the the persistence layer. s43 00:05:53.880 --> 00:05:55.440 Is there a database? s44 00:05:55.440 --> 00:06:13.040 Uh this is an area where uh we think it's probably best not to have our researcher spend too much time in the persistence layer and just map out how far they got and this is a great first entry point once we start bringing in software engineering s45 00:06:13.040 --> 00:06:14.760 help on the project. s46 00:06:14.760 --> 00:06:18.320 Uh then kind of mapping out the the overall system architecture. s47 00:06:18.320 --> 00:06:20.120 Um is this a workflow? s48 00:06:20.120 --> 00:06:22.040 Is this a chaining of workflows? s49 00:06:22.040 --> 00:06:24.320 Are there external LLM calls? s50 00:06:24.320 --> 00:06:26.160 Like what is the anatomy? s51 00:06:26.160 --> 00:06:30.240 What is the kind of taxonomy of this research prototype? s52 00:06:30.240 --> 00:06:32.720 And lastly, how are we going to merge this? s53 00:06:32.720 --> 00:06:34.560 How are we going to decompose it? s54 00:06:34.560 --> 00:06:38.840 Um and we'll get into that more uh later as well. s55 00:06:38.880 --> 00:06:47.240 So, going back to these kind of three levers we can pull, the first is kind of that research project taxonomy document. s56 00:06:47.240 --> 00:06:57.800 Uh the second I want to get into are how we structured our code and how we serve our existing uh features. s57 00:06:57.800 --> 00:07:07.919 So, we basically have a a separate um repository from our core product repo and this is uh all Python based, right? s58 00:07:07.919 --> 00:07:18.080 It's all AI ML stuff and it's basically a mono repo of uh cleanly isolated and fully decoupled microservices. s59 00:07:18.480 --> 00:07:24.120 And so, let's say data-driven entity prediction is our custom transformer model. s60 00:07:24.120 --> 00:07:36.600 So, that is able to up to um kind of grow and be iterated on and be fully decoupled from a different research initiative and it's pretty much a one-to-one s61 00:07:36.600 --> 00:07:41.440 researcher to microservice ratio. s62 00:07:41.440 --> 00:07:43.360 So, we find that works really well. s63 00:07:43.360 --> 00:07:48.320 We have a kind of gateway that that guards requests um and it's all in one Docker Bridge network. s64 00:07:48.320 --> 00:07:58.320 So, the core um consumers our clients in our in our web application make kind of API calls to this gateway which then routes them to the appropriate microservice. s65 00:07:58.920 --> 00:08:08.640 Now, uh looking a bit closer at these uh individual microservices, uh we tend to build them in a pretty simple layered architecture, right? s66 00:08:08.640 --> 00:08:12.440 There's an API layer, the business logic and the data layer. s67 00:08:12.440 --> 00:08:25.360 And then we tend to have like some really um really cleanly documented specs uh so that agents can navigate these repositories and help accelerate our ML researchers as much as possible. s68 00:08:25.360 --> 00:08:33.080 So, yeah, kind of taking a another way to look at this layered architecture, we have kind of the core business logic at the services layer. s69 00:08:33.080 --> 00:08:45.000 Uh that might, you know, make external uh LLM calls to to foundation models or we might need to pull in our own uh machine learning model weights in CICD. s70 00:08:45.000 --> 00:08:48.080 Then we wrap that business logic with controllers. s71 00:08:48.080 --> 00:08:53.839 Um then we put API routers around those and expose them in FastAPI applications. s72 00:08:53.839 --> 00:08:57.040 And each of those microservices is a standalone application. s73 00:08:57.040 --> 00:09:01.720 And like I said in the last slide, it's not really the client that makes a direct call to this microservice. s74 00:09:01.720 --> 00:09:08.280 It kind of goes to that uh gateway first which routes it to the appropriate microservice. s75 00:09:08.280 --> 00:09:19.280 And then within each of those microservices um in the root in the root directory, we have essentially metadata, build instructions, a Dockerfile on how to how to build this application, s76 00:09:19.280 --> 00:09:27.040 uh the the kind of project dependencies, and then poetry or or UV uh lock files um as needed. s77 00:09:27.040 --> 00:09:35.560 And then so I'd like to kind of map out this um anatomical arrangement of our kind of three microservices that we have in production. s78 00:09:35.560 --> 00:09:41.520 And we can kind of see these these trends and and consistent skeletal backbones of these projects. s79 00:09:41.520 --> 00:09:49.920 Uh and it's very easy to map them out and make sure that they're growing um along best practices in software engineering. s80 00:09:49.920 --> 00:09:52.160 So, that's basically it for our for our mono repo. s81 00:09:52.160 --> 00:10:03.640 We have these kind of microservices and then uh sharing basically like underneath all of that in the repository, we have some GitHub actions to to build and deploy our automated s82 00:10:03.640 --> 00:10:09.520 test suites and kind of linting, formatting, and type checks, all the stuff you'd you'd kind of expect. s83 00:10:09.520 --> 00:10:21.000 Uh we have some Jupyter notebooks that run on on Modal for GPU compute, um as well as some additional ML studies, but really the way we look at it is we have this kind of tooling layer and even this kind of fun s84 00:10:21.000 --> 00:10:32.840 CLI, but these all just support our ML engineers in bundling up these microservices that we uh serve in production. s85 00:10:32.840 --> 00:10:42.320 So, that's kind of the the RPT is the is the handoff document between an ML researcher and additional software engineering uh talent. s86 00:10:42.320 --> 00:10:48.839 The mono repo is is where that um research project is ultimately going. s87 00:10:48.839 --> 00:10:52.120 And then the only remaining step is, well, how do we get there, right? s88 00:10:52.120 --> 00:10:54.320 How do we jump from one to two? s89 00:10:54.320 --> 00:11:01.720 And the third lever I want to talk about is the kind of decomposition uh and PR review plan. s90 00:11:01.720 --> 00:11:12.520 Uh and we look at that really as a design problem where we need to kind of um figure out how to slice and dice a a large research uh monolithic prototype. s91 00:11:12.520 --> 00:11:26.480 So, here are just some images of of um an uh platform-wide agent feature where you know, we really studied what axes to slice uh and dice these projects up on, what that dependency graph would look like. s92 00:11:26.480 --> 00:11:40.760 And then we use Graphite um for kind of stack diffs to then uh decompose these large monolithic prototypes that have been proven out um and then get the right eyes on on review to make sure that these are ready for production. s93 00:11:40.760 --> 00:11:45.080 Uh we really like Graphite because it allows for asynchronous review, right? s94 00:11:45.080 --> 00:11:51.480 I could be working on a PR all the way up here while a domain specialist is still reviewing a different PR. s95 00:11:51.480 --> 00:12:06.920 And once we've kind of decomposed these PRs uh thoughtfully, we can start tapping uh subject matter experts throughout the organization on the specific slices or tightly uh decomposed PRs s96 00:12:06.920 --> 00:12:10.040 uh that need to look at. s97 00:12:10.080 --> 00:12:23.720 And so again, there's there's a lot of overlap between this initial research project uh taxonomy document where once we've mapped out these layers, these the architecture, what kind of persistence there is in the types, s98 00:12:23.720 --> 00:12:31.240 that tends to inform uh your decomposition strategy on how to bring it into the into the mono repo. s99 00:12:31.240 --> 00:12:43.520 So, to wrap things up, I want to revisit these kind of three focus areas that you can use to evaluate how well your team is bringing research into production. s100 00:12:43.520 --> 00:12:53.320 And I want to talk about kind of some some diagnostic frameworks to understand if you need to spend a bit more time and attention in any of these three. s101 00:12:53.320 --> 00:13:08.000 So, in the first step with your research legibility, as your team starts to staff research initiatives with product people, with software engineers, with AI engineers, is it obvious where they should concentrate s102 00:13:08.000 --> 00:13:09.160 their efforts? s103 00:13:09.160 --> 00:13:17.160 And is it clear uh what kind of tasks they need to pluck off to bring this prototype into production? s104 00:13:17.160 --> 00:13:22.880 If there's ambiguity in there, you may need to spend some time revisiting this process. s105 00:13:22.880 --> 00:13:26.160 In step two, your code repository. s106 00:13:26.160 --> 00:13:37.320 As you start getting ready to bring prototypes into your production grade code base, is it clear where these buckets lie to put in new code? s107 00:13:37.320 --> 00:13:42.960 Are there templates and existing frameworks and patterns that you can mimic? s108 00:13:42.960 --> 00:13:54.480 Or is it possible that maybe you've started to outgrow that code base and that system architecture, and every time you bring in new research concepts, you're fighting these old abstractions s109 00:13:54.480 --> 00:13:58.960 and having headaches from limitations of your repository. s110 00:13:58.960 --> 00:14:12.680 And then lastly, in the decomposition phase, are you able to consistently estimate the timelines and delivery dates for moving research concepts into your repository? s111 00:14:12.680 --> 00:14:20.760 Is it clear which subject matter experts you should tap for review and for productionizing this research? s112 00:14:20.760 --> 00:14:35.680 Uh and if you're having issues here, it probably points to the stream issues, either in uh how this research is being coordinated and handing off, or perhaps the code base that's hosting it. s113 00:14:35.680 --> 00:14:37.800 Uh anyway, that's about all I've got. s114 00:14:37.800 --> 00:14:39.400 I hope this is helpful. s115 00:14:39.400 --> 00:14:47.760 I love uh thinking about ways how our team can speed up how quickly we can bring research concepts into production. s116 00:14:47.760 --> 00:14:51.280 And if you made it this whole way, you're probably interested in that stuff, too. s117 00:14:51.280 --> 00:14:56.720 And I would love to compare notes uh and trade ideas anytime.