WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ra8X7_bnujw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ra8X7_bnujw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.200 --> 00:00:13.360 I'm Scott Worley. s3 00:00:13.360 --> 00:00:21.160 I'm the CTO of Norm AI and I'm privileged enough to be sharing the stage here with John Stecher who's the CTO of Blackstone. s4 00:00:21.160 --> 00:00:24.640 Um From what I understand, John has basically three full-time jobs. s5 00:00:24.640 --> 00:00:36.640 It sounds like you've responsible for all of the technology at Blackstone, also advises on their investments and is as well as serves as a resource for the portfolio companies for all things technology. s6 00:00:36.640 --> 00:00:37.360 Did I miss anything? s7 00:00:37.360 --> 00:00:39.000 Nope, all perfectly mapped. s8 00:00:39.000 --> 00:00:40.400 And so cool job. s9 00:00:40.400 --> 00:00:45.200 Yeah, and if you're not familiar with what Blackstone does, then you probably have wandered into the wrong conference. s10 00:00:45.200 --> 00:00:48.240 Um so I'm not going to go into that, but you may not know what Norm does. s11 00:00:48.240 --> 00:00:53.240 So really briefly, uh Norm's an AI company and we operate in the legal and regulatory space. s12 00:00:53.240 --> 00:01:05.000 We have an enterprise AI platform that's deployed in some of the largest financial uh institutions in the world and then we also have an affiliated law firm, Norm Law, and our AI platform also supports that. s13 00:01:05.000 --> 00:01:13.520 Um so since the end of last year and certainly all of this year, everyone's been racing to figure out how to deploy AI agents. s14 00:01:13.520 --> 00:01:17.280 Now I'd love to hear how you're thinking about this at Blackstone and how you're trying to make sure you get this right. s15 00:01:17.280 --> 00:01:24.680 Yeah, I mean I think the the race has definitely been on as like the technology has improved mightily uh since you know really like February, March of last year. s16 00:01:24.680 --> 00:01:32.160 Like just in terms of context windows being able to run stuff for significantly longer amounts of time have like resulted in people being able to produce just more value. s17 00:01:32.160 --> 00:01:45.560 And one of the things that like we've been focused on a lot and I think every company in our portfolio advise on and work with all the time that they need to have top of mind for like being successful with AI agents in general is you have to get your data story right. s18 00:01:45.560 --> 00:01:56.320 And so like one of the biggest things, impediments, and tripping points that I see for companies out there when they want they're all in on AI, they want to build as many agents as possible, they really want to kind of extract value. s19 00:01:56.320 --> 00:02:06.572 If they don't have a cohesive data strategy that's moving stuff out of transactional systems into something that can be accessed and easily managed and recalled from like an agent perspective, s20 00:02:06.572 --> 00:02:07.972 it is going to fail. s21 00:02:07.972 --> 00:02:16.612 And so that's been one of the biggest things that we've been focused on you know as a big asset manager we have a ton of data proprietary data public data all woven together. s22 00:02:16.612 --> 00:02:21.812 We have to move that into out of systems into something that we can really leverage from an AI perspective. s23 00:02:21.812 --> 00:02:31.332 I think the second thing that you know has nothing to do with AI but you can leverage AI to actually figure out how to do the best possible way is straight through processing still matters. s24 00:02:31.332 --> 00:02:36.452 I'm always shocked at how everybody just talks about like agents as being the biggest differentiator in the world. s25 00:02:36.452 --> 00:02:53.452 The thing that I've leveraged myself using factory and a couple other ways to go off and build code inside of the firm is like taking time having agents help to craft something for me understand the workflows out there that people are doing and then make sure that I'm like actually writing code to optimize for straight through processing s26 00:02:53.452 --> 00:03:00.412 and then using agents to inject an intelligence where necessary where humans were making decisions in the past for a variety of wrote work things. s27 00:03:00.412 --> 00:03:05.052 So that's really like the two key elements that I I spend a ton of time thinking about every single day. s28 00:03:05.052 --> 00:03:16.292 And then the biggest thing from an agent perspective is really stepping back and looking at like what is work that we want to build agents to do better to offload like intellectual s29 00:03:16.292 --> 00:03:22.332 like cycles from our investment professionals our finance professionals our engineers such that they can do higher value work. s30 00:03:22.332 --> 00:03:32.332 And there's a ton of stuff when you think about in the software development life cycle writing test cases functional test cases performance test cases version versioning forward libraries underneath s31 00:03:32.332 --> 00:03:33.252 applications. s32 00:03:33.252 --> 00:03:35.132 It's the bane of every engineer's existence. s33 00:03:35.132 --> 00:03:37.452 I've hated it for 27 years that I've been doing this. s34 00:03:37.452 --> 00:03:44.572 Having an agent be able to go off and do these things for me and renovate my code and my source code has made all the difference in the world for the velocity that we deliver at. s35 00:03:44.572 --> 00:03:46.412 So how are you thinking about it at Norm? s36 00:03:46.412 --> 00:03:56.732 Yeah so Norm was founded in 2023 so you know that's 3 year I guess 4 years after GPT-1 but still a year before there was like you know chain of thought. s37 00:03:56.732 --> 00:03:59.172 So it was like an interesting time in terms of development. s38 00:03:59.172 --> 00:04:07.934 I remember when we had our first team get together we were at Stanford's campus for a week and I mean, I'm probably misremembering the exact timeline, but it felt like we went from 4K s39 00:04:07.934 --> 00:04:14.574 to 8K to 128K context windows and then we had multimodal and everything just felt like it was kind of coming all at once. s40 00:04:14.574 --> 00:04:18.774 And then there was a bit of a time where things did slow down when it was coming from the frontier. s41 00:04:18.774 --> 00:04:26.654 And so I think like a lot of companies in like early 2024, you know, we were trying to deploy these models as part of like our workflow and as part of like our production software. s42 00:04:26.654 --> 00:04:33.654 And so we ended up adding a lot of structure to the system and and and this comes from you know, all different parts of the stack. s43 00:04:33.654 --> 00:04:46.174 So whether that was from like how we went and took the content that we knew needed to be like eventually added to context to the models and we had like very specific ways in which we did that or if there was like you know, just the sequencing of the models, s44 00:04:46.174 --> 00:04:53.574 if there was like steps we needed to do in terms of looking for consistency and then ultimately like distilling everything down into some useful output. s45 00:04:53.574 --> 00:04:56.774 We ended up with like something looked like a large like map reduce problem effectively, right? s46 00:04:56.774 --> 00:05:03.854 Whereas like we had a lot of scaffolding for the agents and we were very particular about like how these well, I would say it's really just how these models ran. s47 00:05:03.854 --> 00:05:05.414 It was much more like a workflow. s48 00:05:05.414 --> 00:05:16.214 And what this allowed us to do was get like really, you know, high quality and also with accurate very accurate, but also just very consistent, which is very important for us in the regulatory space outputs. s49 00:05:16.214 --> 00:05:22.654 Um, but then one fortuitous thing as part of doing this cuz we wanted to have a some way to regression test this big system. s50 00:05:22.654 --> 00:05:25.294 We ended up with a really large set of evaluation samples. s51 00:05:25.294 --> 00:05:29.254 We had tens of thousands actually of like hand labeled examples by experts. s52 00:05:29.254 --> 00:05:38.054 And so as soon as we moved to a place where it was like clear that agents were actually a thing and by agents I simply mean you know, given like an LLM a system s53 00:05:38.054 --> 00:05:47.414 or given it a set of tools that it can use, give it some goal and give it some way to verify itself against that goal, allowing it just to go figure out how to do the thing in the best way possible. s54 00:05:47.414 --> 00:06:00.854 We were able to just take this benchmark set and just immediately run on top of it with some of these more agentic approaches and we saw, you know, I think a lot of people had this moment where you're staring at hundreds of thousands of lines of code your team had really thoughtfully written and realizing most of it s55 00:06:00.854 --> 00:06:09.095 was pretty obsolete and was just kind of actually in in And so to update quickly on this, what we did was we um we took I got the engineering leadership team. s56 00:06:09.095 --> 00:06:12.735 We actually rented a a Sprinter van, and we drove to Upstate New York. s57 00:06:12.735 --> 00:06:14.335 It was actually after a big snowstorm. s58 00:06:14.335 --> 00:06:17.255 We had an Airbnb, and it was like, "Forget norm for a minute. s59 00:06:17.255 --> 00:06:23.575 Let's just say what would we build today as far as a platform, knowing what we know right now um if we were going to be starting over?" s60 00:06:23.575 --> 00:06:29.655 And then from that, we left with basically a set of specs about like what this platform needed to be able to do to enable two things. s61 00:06:29.655 --> 00:06:32.935 One, deliver agentic experiences to our clients. s62 00:06:32.935 --> 00:06:39.295 And then also um it needed to be amenable to having AIs like co-developers alongside of our own development team. s63 00:06:39.295 --> 00:06:41.855 And that's what we've been executing against uh ever since then. s64 00:06:41.855 --> 00:06:44.135 So, I think this Do you have a question? s65 00:06:44.135 --> 00:06:49.535 No, no. I was going to say, I think you hit on like a key point, Scott, which is like the eval side of the fence that you were talking about before. s66 00:06:49.535 --> 00:06:56.895 If you're working in financial services, like the the quality bar for what agents have to deliver on is very, very high. s67 00:06:56.895 --> 00:07:08.815 And so, I think really thinking through like what is your eval set for how you basically test your agents, what they're actually, you know, whatever work they're performing, that the output makes sense, and it's consistent and repeatable. s68 00:07:08.815 --> 00:07:14.015 And I've seen across the board as model versions change underneath the covers, you get different answers. s69 00:07:14.015 --> 00:07:17.375 You get skew in terms of the accuracy of information that comes back. s70 00:07:17.375 --> 00:07:23.455 So, having good evals so that you can run those consistently and really measure the health of a system is paramount to success. s71 00:07:23.455 --> 00:07:27.775 Same thing happens if harnesses change that you're running stuff on top of, you'll get different answers. s72 00:07:27.775 --> 00:07:29.055 Data changes, different answers. s73 00:07:29.055 --> 00:07:35.885 So, it's it's something where I think, you know, if you're trafficking in financial services, like the duty to be accurate is like s74 00:07:35.885 --> 00:07:35.975 [snorts] s75 00:07:35.975 --> 00:07:40.295 up here, and hitting that is like something like we think about all the time. s76 00:07:40.295 --> 00:07:40.535 Definitely. s77 00:07:40.535 --> 00:07:44.415 I think this leads nicely into this concept that we've been talking about just like with the bitter lesson. s78 00:07:44.415 --> 00:07:55.855 So, like everyone's familiar with it, and you know, this is this idea that these hand-crafted um systems built by, you know, clever people thinking hard about the problem will often times eventually be dominated s79 00:07:55.855 --> 00:07:58.175 by systems that scale with compute. s80 00:07:58.175 --> 00:07:58.335 Right? s81 00:07:58.335 --> 00:08:03.815 And that this is actually I think Rich Sutton originally talked about this in 2019, but this has been since the beginning of neural networks. s82 00:08:03.815 --> 00:08:13.377 If you go back to like the '50s and '60s like with Rosenblatt's little perceptron, I think the New York Times ran an article or somebody did saying it was the first first machine that could ever have its own original thoughts, you know? s83 00:08:13.377 --> 00:08:25.817 And so and there was actually it's there's actually quite a fascinating history here of how the you know, expert systems and the connectionists and what ended up happening a lot of it actually was highly litigated in in England and a lot of AI research was dismantled, s84 00:08:25.817 --> 00:08:36.097 but one funny thing if you go back to the 80s even, there was a lead at IBM's translation lab and he made a comment, which is very much the bitter lesson, every time I fire a linguist my models improve, right? s85 00:08:36.097 --> 00:08:42.377 No, the point of having these like linguists in there and trying to think about all the rules of language versus like training a statistical model based on just probabilities. s86 00:08:42.377 --> 00:08:44.737 And so we speed up to today and like we see this all the time. s87 00:08:44.737 --> 00:08:54.657 We've been touching on lots of points about this and I'm curious when you think about your road maps, when people on your team are bringing you project ideas, like how do you keep this in mind as you're thinking about this for Blackstone? s88 00:08:54.657 --> 00:09:05.257 Yeah, I mean a lot of what we do is like, you know, simple equation is is giving enough compute every problem is solvable by like hardware and software at some point in time in the future. s89 00:09:05.257 --> 00:09:09.857 It's Elon talks about this energy-wise, etc. So it's like it's a big map, it's a whole bunch of stuff. s90 00:09:09.857 --> 00:09:11.377 How do you bring it back to reality? s91 00:09:11.377 --> 00:09:19.857 Like what I spend a lot of my time doing from like my seat working with engineers in my team, with product people, with our SRE function, with like stakeholders s92 00:09:19.857 --> 00:09:25.097 is really trying to define like, you know, what is the ROI of what I'm being asked to work on. s93 00:09:25.097 --> 00:09:32.297 And so like when I think of building agents, I think about you know, getting data into the right spot, think about building dashboards, trading platforms, whatever it is. s94 00:09:32.297 --> 00:09:44.377 It's like if I look at like what's the ROI that's going to be created from the actual ask that's coming in to me from a team perspective and then like what's the what's the true cost of actually building something today? s95 00:09:44.377 --> 00:09:51.337 Like how can I balance that equation and then I make a decision of like, you know, is this something where the models are going to get good enough? s96 00:09:51.337 --> 00:09:56.097 I'm going to have more than enough compute in 6 months that it's not worth trying to tackle the problem today. s97 00:09:56.097 --> 00:09:59.897 It's better to delay it and actually address it with one of the new models. s98 00:09:59.897 --> 00:10:12.938 We have made that decision before is like let's actually wait for the model to become slightly more sophisticated so that we can solve these problems with it because it will cost less at that time and the need for that application to be built today s99 00:10:12.938 --> 00:10:13.738 isn't pressing. s100 00:10:13.738 --> 00:10:20.218 There's no like real business value increase that can be like will be created because it's produced today versus 6 months from now. s101 00:10:20.218 --> 00:10:25.178 So, that's a trade-off that we actually, you know, make regularly and I think about regularly. s102 00:10:25.178 --> 00:10:29.338 Um the other thing that we do is like you don't want to wait too long to basically build something. s103 00:10:29.338 --> 00:10:33.138 And so, you can always be waiting for the new model, waiting for the new technology, or whatever else. s104 00:10:33.138 --> 00:10:42.418 Like the beautiful thing with like all of the various different software development harnesses today is if you lean into them, it's amazing how quick you can go from like concept, s105 00:10:42.418 --> 00:10:50.218 mapping out your planning and concept with like Fable for like, you know, just hey, work with me and let's define this with the product person or with the stakeholder s106 00:10:50.218 --> 00:10:54.218 to actually writing code to having it like a workable proof of concept. s107 00:10:54.218 --> 00:10:58.498 And when I say workable proof of concept, it's something people could click through, interact with, whatever else it is. s108 00:10:58.498 --> 00:11:07.258 Doesn't have cybersecurity review, doesn't have all the other things that you need to do in our industry to make sure you like put something into production that scales, but you can get to a POC or a POV s109 00:11:07.258 --> 00:11:08.098 very quickly. s110 00:11:08.098 --> 00:11:13.978 And so, like that's one big thing I try to encourage all my engineers to do frequently is like just get to POC. s111 00:11:13.978 --> 00:11:16.458 Don't worry about PowerPoints, don't worry about anything else. s112 00:11:16.458 --> 00:11:20.978 Leverage tokens, which like I think it's a fallacy to be like counting tokens in some way, shape, or form. s113 00:11:20.978 --> 00:11:23.178 It's better to think about producing value. s114 00:11:23.178 --> 00:11:29.378 Go off, produce value, produce POVs, put it in front of people, iterate on it quickly, and throw it away if you don't need it. s115 00:11:29.378 --> 00:11:32.138 You will spend far less time and money doing those types of things. s116 00:11:32.138 --> 00:11:35.058 And so, that's like a big way that we think about trying to solve it. s117 00:11:35.058 --> 00:11:44.698 It doesn't maps into the parallel like the you know, the how do you deal with increasing compute, making it easier to do these things, but like I just I think in a much shorter time frame. s118 00:11:44.698 --> 00:11:51.098 Yeah, it's so interesting cuz it's like, you know, I mean, you touched on so many things that I think everybody's probably been thinking about, too. s119 00:11:51.098 --> 00:11:57.818 And if you go back to 2023 when we were originally like sitting around as a small group talking about what we're building, we were aware of the better lesson then. s120 00:11:57.818 --> 00:11:58.378 We talked about it. s121 00:11:58.378 --> 00:12:00.018 We said, "How do we not get in this trap?" s122 00:12:00.018 --> 00:12:03.778 And I think at this point, it's just really evident to me that it's like it's definitely a gradient, right? s123 00:12:03.778 --> 00:12:13.219 It's not this binary thing about you can somehow build the perfect technology to index into all the up you know, everything coming in from the frontier in a a that you don't have raft to throw things away. s124 00:12:13.219 --> 00:12:28.499 really it gets down to like where did we want to invest heavily and where are we willing to invest conservatively because like you said, we need something today, but at the same time we're just aware that like some of these investments we make particularly around things like we've built, you know, fairly complicated systems around all of our s125 00:12:28.499 --> 00:12:29.379 agent primitives. s126 00:12:29.379 --> 00:12:39.979 So, the harness, the sandbox, how egress works, you know, all of these things and we we also want to make sure that as like things are coming out either from like a frontier lab or from like a a third party that we're able to index into that. s127 00:12:39.979 --> 00:12:43.219 And so, I think a lot of it is just kind of when do you update? s128 00:12:43.219 --> 00:12:49.859 And I that's one of the hardest things to do and I like for me a good example was I really updated really hard on cloud code. s129 00:12:49.859 --> 00:12:54.979 Not so much that it was suddenly tools that like in my profession that we could use that were really incredible. s130 00:12:54.979 --> 00:13:01.379 What it really was for me was just this idea cuz I was a bit skeptical up to that point of how we would get something that was like truly generalizable. s131 00:13:01.379 --> 00:13:03.059 It's like, well, what does cloud code give you? s132 00:13:03.059 --> 00:13:08.499 Well, you can write code and then it can execute the code and like we're basically running a computer in some verifiable way. s133 00:13:08.499 --> 00:13:17.299 At that moment, I was like, well, we have all the ingredients for like a truly generalizable intelligence because if you can go do this at scale, then I mean, you know, of course like you can basically do anything s134 00:13:17.299 --> 00:13:20.859 in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today. s135 00:13:20.859 --> 00:13:26.739 And you fast forward to now and like all these tools that people are using these like AI assistants and things, this is why they're so amazing. s136 00:13:26.739 --> 00:13:30.379 It's cuz you can ask a question that's never seen before fully out of distribution. s137 00:13:30.379 --> 00:13:32.499 It can spin up a computer and go do this thing in the background. s138 00:13:32.499 --> 00:13:36.459 And for consumer, it's amazing and then we're also seeing this now within the enterprise, too. s139 00:13:36.459 --> 00:13:40.499 And so, for me that moment was like, well, where are we going to invest heavily? s140 00:13:40.499 --> 00:13:47.899 Of course, we still need to invest in what we're building on the agent side, but going back to what uh John mentioned, how we do our data capture and distillation s141 00:13:47.899 --> 00:13:50.579 and how we make this available as like state to these models. s142 00:13:50.579 --> 00:13:53.419 And then also just how we do our benchmarks and evaluations. s143 00:13:53.419 --> 00:14:00.299 It's And so, with like the law firm for example, we have this really unique situation where we started this law firm, um you know, from nothing. s144 00:14:00.299 --> 00:14:03.219 So, we we've kind of grown together, they've grown with Anom together. s145 00:14:03.219 --> 00:14:07.379 And so, we work directly with lawyers to develop to stand up these benchmarks and evaluations. s146 00:14:07.379 --> 00:14:13.419 And effectively, rather than think of a traditional road map, like when when are you going to build an agent that can go do these three things? s147 00:14:13.419 --> 00:14:22.181 It's let's have a scorecard that determines when an agent or when a human even is has success is successful at completing these tasks across the various dimensions. s148 00:14:22.181 --> 00:14:24.341 And then we want to be able to see this system improve over time. s149 00:14:24.341 --> 00:14:33.981 So, as we make general improvements to the system, we should see an uplift to these, you know, these things like an attorney would expect to improve their workflow, but at the same time not not getting s150 00:14:33.981 --> 00:14:43.141 kind of going back to the state that we were originally where we had added a lot of structure to it in a way that ultimately we found that we were actually we could get even better performance if we were to remove that up front s151 00:14:43.141 --> 00:14:43.621 structure. s152 00:14:43.621 --> 00:14:44.021 Yeah. s153 00:14:44.021 --> 00:14:52.781 I think like another thing that's important to think through is like you can produce, like I was saying, a lot more proof of concepts, proof of value very quickly. s154 00:14:52.781 --> 00:14:57.221 I still think like the hard yards for a lot of development out there is getting it to production. s155 00:14:57.221 --> 00:15:07.021 And what I mean by that is like infrastructure buildout that actually runs inside of the infrastructure that's available inside of any financial services or enterprise company for that matter out there. s156 00:15:07.021 --> 00:15:10.021 Like that is a challenge that a lot of folks like underestimate. s157 00:15:10.021 --> 00:15:15.981 If you're selling AI tools, it's that is hard on the other side, just so you know, I deal with it quite a bit. s158 00:15:15.981 --> 00:15:21.821 The other thing is is just like dealing with like the cybersecurity side of the fence and the supportability side of the fence. s159 00:15:21.821 --> 00:15:30.261 And so like one thing I've noticed, you know, the agents are getting better at this now, but it's making sure that the actual software that's being written is truly traceable s160 00:15:30.261 --> 00:15:34.541 and it feeds into your SRE platform so that it can be monitorable in production. s161 00:15:34.541 --> 00:15:37.221 Those are things still that like I like to have people think about. s162 00:15:37.221 --> 00:15:39.861 It's like they haven't started to scale yet with agents. s163 00:15:39.861 --> 00:15:44.861 They will, but it's something that from my perspective I think about regularly. s164 00:15:44.861 --> 00:15:45.221 Makes sense. s165 00:15:45.221 --> 00:15:53.181 And we our approach to this, I'm sure a lot of other companies and maybe even you guys have taken this approach as well, is there's parts of our code base that we s166 00:15:53.181 --> 00:16:02.101 like still require like complete human oversight from when it comes to designing, building, code ownership, even approving any kind of changes to the code base. s167 00:16:02.101 --> 00:16:03.461 And these are like our core primitives. s168 00:16:03.461 --> 00:16:11.181 And these are things where, you know, basically you can build things on top of these, but the things you build on top of these are not allowed to like change these systems without having like human review. s169 00:16:11.181 --> 00:16:18.542 And it's because of things you mentioned is like, you know, data security for us is, you know, is is is like as paramount as delivering quick. s170 00:16:18.542 --> 00:16:19.382 It's actually more important. s171 00:16:19.382 --> 00:16:32.902 So, and for depends on where you are and if whether it's a consumer product or for us with obviously working with highly sensitive data, like we have to build in so much infrastructure around this and ensure that we're not ever getting out ahead of like, you know, these parts of the stack, which s172 00:16:32.902 --> 00:16:39.142 it can be hard cuz maybe somebody brings you and I'm sure you've seen this like a really cool demo and it's really exciting and you're looking at it terrified s173 00:16:39.142 --> 00:16:45.222 cuz we're like, "This is the This is very cool, but to get to this point in terms of capabilities, like it would expose so much that we're not able to do yet." s174 00:16:45.222 --> 00:16:46.942 So, I think it's just always that trade-off. s175 00:16:46.942 --> 00:16:57.342 So, we just, you know, basically we invest heavily in these things and then invest conservatively in places where we realize as the frontier continues to improve, um we want to make sure we do we have like conduits. s176 00:16:57.342 --> 00:17:04.022 The way I described it actually this morning with the team was like, "Basically throughout our platform we want to have like conduits for what's coming from the frontier. s177 00:17:04.022 --> 00:17:09.782 You don't want to have dams where like the intelligence is coming in and it can't quite get into your stack because you built things in such a way." s178 00:17:09.782 --> 00:17:10.502 Again, it's a gradient. s179 00:17:10.502 --> 00:17:12.062 We're going to get some of it wrong, some of it right. s180 00:17:12.062 --> 00:17:20.942 Just being very cognizant of it, talking about it a lot, and making sure that our core pieces are are locked down, but then also, you know, being very open about um just ripping and replacing. s181 00:17:20.942 --> 00:17:24.142 I think that's like a thing that I think everybody's gotten really good at. s182 00:17:24.142 --> 00:17:28.262 I I think that's like one of the biggest I mean I like I I've been doing this for 27 years. s183 00:17:28.262 --> 00:17:30.262 Sometimes I feel like it's been like 100 years. s184 00:17:30.262 --> 00:17:31.822 Other times I feel like it's been five. s185 00:17:31.822 --> 00:17:40.502 I think the the thing that's like really made a material difference like when I I look at the ability to write software today versus what I was doing 10 years ago s186 00:17:40.502 --> 00:17:48.582 was 10 years ago it took so much time, effort, and money, and people's like intellectual capital to produce a piece of software. s187 00:17:48.582 --> 00:17:50.422 It was almost like the ship had sailed. s188 00:17:50.422 --> 00:17:57.782 Like by the time you designed it and started writing code, it was out to sea and you were going to ride that baby for like as long as you could until it got to the other side of the ocean. s189 00:17:57.782 --> 00:18:05.622 And like what I've become very comfortable with today is like for the most part a lot of like internal applications can be just thrown away. s190 00:18:05.622 --> 00:18:15.902 It's build them, have them work for a purpose in a point in time, but understand that they're going to disappear at some point in time a lot now much more quickly than ever before, s191 00:18:15.902 --> 00:18:18.062 and then just think about like what comes next. s192 00:18:18.062 --> 00:18:30.983 And so I I think one of the challenges with like sitting in my seat or any like good engineer seat should be constantly questioning like, you know, should I rewrite this and does it cost me less to rewrite it than it cost me to keep versioning the current version forward? s193 00:18:30.983 --> 00:18:35.103 And that is like a question that a decade ago never would have came up in anyone's head. s194 00:18:35.103 --> 00:18:40.143 If you went to say I want to rewrite an internal application, like your CFO would have looked at you like you were insane. s195 00:18:40.143 --> 00:18:47.783 And I think now it's a completely relevant thing that all of us should be doing cuz net new applications might produce way more business value at the end of the day. s196 00:18:47.783 --> 00:18:49.383 So, it's just something that I encourage folks to do. s197 00:18:49.383 --> 00:18:51.903 It sounds like you guys are doing quite a bit at Norm. s198 00:18:51.903 --> 00:18:52.423 Yeah, exactly. s199 00:18:52.423 --> 00:18:55.983 And I think it's just that concept of what used to be expensive that's now cheap. s200 00:18:55.983 --> 00:19:03.503 And if you don't update quickly on that, then it's, you know, it like you it's incredible you said it feels like 27 or 5 years or 100 years or 5 years. s201 00:19:03.503 --> 00:19:11.623 It's like that if you over if you over index too hard on something that was in June and now it's October, like, you know, it it feels like in some ways there's been like 10 years of technological advancement. s202 00:19:11.623 --> 00:19:13.223 So, um Well, great. s203 00:19:13.223 --> 00:19:14.783 We only have a minute or so left. s204 00:19:14.783 --> 00:19:16.063 Are there any closing remarks you s205 00:19:16.063 --> 00:19:24.743 I mean, I think like the the one thing I'd like leave the audience on if you if you want like, you know, like good takeaways from this, it's like we talked about a lot of like very interesting things. s206 00:19:24.743 --> 00:19:30.063 I think like what makes me like excited to go to work is like it's cool to get back to engineering. s207 00:19:30.063 --> 00:19:39.663 One of the things that like I value is like, you know, I was a hardcore low-level engineer for 10 years at IBM and like wrote like, you know, this is way back in the day of using notepad and VI s208 00:19:39.663 --> 00:19:42.543 to write like instruction level code for RISC machines. s209 00:19:42.543 --> 00:19:43.743 It was like awesome. s210 00:19:43.743 --> 00:19:44.703 It's nerdy as all heck. s211 00:19:44.703 --> 00:19:46.863 No one does this stuff anymore, but it was like fun for me. s212 00:19:46.863 --> 00:19:48.463 I still enjoy those days. s213 00:19:48.463 --> 00:20:00.983 As you get more senior in your career, it like the barrier to actually writing code on a daily basis disappears or it becomes more like hard to get through like dealing with a stream like processing stuff off a stream. s214 00:20:00.983 --> 00:20:04.903 It's just all these things are like, oh my god, I can't even get started cuz the scaffolding is so hard. s215 00:20:04.903 --> 00:20:12.863 With like AI, it's been like reinvigorating to get back and actually build stuff cuz it's like I never lost my knowledge on how to architect things. s216 00:20:12.863 --> 00:20:14.983 I see this with a lot of engineers out there. s217 00:20:14.983 --> 00:20:18.703 They understand because they've been in the market 10, 20, 30 years. s218 00:20:18.703 --> 00:20:25.585 They understand the complexity of system design which hasn't changed and now they can actually express that knowledge with these new cool agents. s219 00:20:25.585 --> 00:20:34.145 And so I think it's a super exciting time to be like starting a company, working at a company and like building software cuz there's like a whole new world that we can create out there. s220 00:20:34.145 --> 00:20:35.385 So that's what like excites me. s221 00:20:35.385 --> 00:20:37.145 So well thank you so much. s222 00:20:37.145 --> 00:20:37.705 It's been an honor. s223 00:20:37.705 --> 00:20:38.905 Yep, thank you guys for having us here today. s224 00:20:38.905 --> 00:20:41.345 I appreciate it.