WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=HLTa7Vcs4X0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/HLTa7Vcs4X0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.360
Hello everyone.

s3
00:00:13.360 --> 00:00:14.160
Hi.

s4
00:00:14.160 --> 00:00:15.759
Um, my name is Gabe.

s5
00:00:15.759 --> 00:00:19.439
I'm one of the co-founders and CEO of Meticulus.

s6
00:00:19.439 --> 00:00:24.480
Um, I'll just wait one more minute for people to come in and then we can and then we can get started.

s7
00:00:24.480 --> 00:00:31.039
Um so in today's demo or talk I'll talk about verification of code.

s8
00:00:31.039 --> 00:00:41.840
Um I'll spend one minute introducing meticulous and then one minute giving some background context on the company and then sort of eight minutes giving a demo and technical technical

s9
00:00:41.840 --> 00:00:43.120
overview.

s10
00:00:43.120 --> 00:00:53.680
Um so if you have exhaustive verification then you can ship code at the speed that your agents write it.

s11
00:00:53.680 --> 00:01:07.360
If you don't, then someone somewhere at your organization is spending time verifying code and that is now your new bottleneck and you're forced into a position where you're trading off velocity

s12
00:01:07.360 --> 00:01:10.720
against bugs and user experience.

s13
00:01:12.640 --> 00:01:15.920
So I'll give um super quick context on Meticulus.

s14
00:01:15.920 --> 00:01:24.479
So, Meticulus gives you exhaustive or near exhaustive verification of your front-end codebase with zero developer effort.

s15
00:01:24.479 --> 00:01:37.439
Um, it's deployed across the entire engine organization at all of these companies like Discord, Whiz, Dropbox, Notion, 11 Labs, Launch Darkly, and many, many other great companies.

s16
00:01:37.439 --> 00:01:43.759
So, if an engineer touches front-end code at one of these companies, they use Meticulus every every day.

s17
00:01:43.759 --> 00:01:51.680
However, before we get into the product and talk more about verification, um, let's just cover the current sort of state of the world.

s18
00:01:51.680 --> 00:01:57.920
So, AI writes code faster than humans can review it.

s19
00:01:57.920 --> 00:02:03.600
And review and verification is now the new bottleneck.

s20
00:02:04.399 --> 00:02:09.840
The other state of the world is that assertionbased testing alone isn't enough.

s21
00:02:09.840 --> 00:02:26.239
So no matter how hard a human or an agent tries to define the correct behavior of software a priority upfront in assertions the space of possible regressions is too vast to exhaustively

s22
00:02:26.239 --> 00:02:28.239
cover.

s23
00:02:28.800 --> 00:02:41.680
And so a good clarifying question is if an engineer in your organization uh AI generated this pull request, would you feel comfortable hitting the the merge merge and ship button?

s24
00:02:41.680 --> 00:02:54.560
And in most organizations, the answer is no. You would want to go and do the hard work of checking out all the different feature flags, all the different roles, permissions, settings, configurations,

s25
00:02:54.560 --> 00:03:04.000
and the various different edge cases in order to understand the exhaustive impact of this of this change and see if you can spot any issues.

s26
00:03:04.400 --> 00:03:08.720
So there are three sort of like downstream consequences of that current state.

s27
00:03:08.720 --> 00:03:14.720
The first one is bugs or regressions uh which have a a business impact.

s28
00:03:14.720 --> 00:03:25.599
The second one is engine organizations will spend doubledigit percentages of their time maintaining end to-end test suites.

s29
00:03:25.599 --> 00:03:38.640
So across manual validation, review, uh debugging flakes or updating or maintaining the test suite, um all spend a lot of time maintaining maintaining maintaining tests.

s30
00:03:38.640 --> 00:03:53.519
And then the third like problem or downstream downstream consequence of of the state of the world is that um if you if you can get exhaustive verification then you can program in a new and different way.

s31
00:03:53.519 --> 00:04:04.400
So all of a sudden you can bump all your dependencies, you can do sweeping refactors, you can make any AI generated change with complete complete confidence.

s32
00:04:04.400 --> 00:04:07.599
So then the question becomes um that sounds great.

s33
00:04:07.599 --> 00:04:10.480
How do you how do you get exhaustive verification?

s34
00:04:10.480 --> 00:04:13.760
What does it look like and how do you how do you get there?

s35
00:04:13.760 --> 00:04:19.359
Um so I'll now give you a demo of how how meticulous meticulous works.

s36
00:04:19.359 --> 00:04:27.360
So, we give you uh give me one second to set this up.

s37
00:04:34.160 --> 00:04:35.840
Great.

s38
00:04:35.840 --> 00:04:42.400
Um, so we give you one line of JavaScript to inject onto nonproduction environments.

s39
00:04:42.400 --> 00:04:45.440
So that's localhost, QA, dev, staging.

s40
00:04:45.440 --> 00:04:52.479
That JavaScript instruments the web browser and records thousands or tens of thousands of flows or workflows.

s41
00:04:52.479 --> 00:04:57.440
So someone clicking the login button, clicking the settings panel, clicking the analytics panel.

s42
00:04:57.440 --> 00:05:09.120
And then when you open a pull request on your CI runner, you spin up your web application on localhost 3000 and Meticulus takes a subset of those recorded workflows

s43
00:05:09.120 --> 00:05:10.800
and it replays them.

s44
00:05:10.800 --> 00:05:14.000
So it dispatches each event one by one.

s45
00:05:14.000 --> 00:05:23.759
And as it dispatches those events, so click the login button, click the senses panel, click the analytics panel, it takes a screenshot at each atomic moment throughout time.

s46
00:05:23.759 --> 00:05:30.800
And so it generates a sequence of screenshots for every workflow, one before your code change and one after your code change.

s47
00:05:30.800 --> 00:05:41.360
It then diffs those two sequences together to show you what would be different about your application if you were to merge that code in before before you do.

s48
00:05:41.360 --> 00:05:46.000
So in this example, there's two changes here.

s49
00:05:46.000 --> 00:05:52.720
One is someone's changed the name and the second is they've introduced an error to a to a drop down.

s50
00:05:52.720 --> 00:05:58.560
And so when you open a pull request, Meticulus posts a comment typically within a few minutes.

s51
00:05:58.560 --> 00:06:03.360
And when the developer clicks into this comment, the Wi-Fi is slow, so I'm just going to switch tabs.

s52
00:06:03.360 --> 00:06:09.919
But when a developer clicks into this comment, Meticulus will not tell you whether or not there's a bug.

s53
00:06:09.919 --> 00:06:17.759
What it will show you is a set of diffs that show you what's different about the application if you were to merge that code in.

s54
00:06:17.759 --> 00:06:22.720
And so each diff has a before state and an after state.

s55
00:06:22.720 --> 00:06:28.319
And so by reviewing the diff, you can determine whether that change is expected or unexpected.

s56
00:06:28.319 --> 00:06:30.639
So here's the name change on light mode.

s57
00:06:30.639 --> 00:06:32.720
Here's the name change on dark mode.

s58
00:06:32.720 --> 00:06:36.800
And then this is a manifestation of that logical logical error.

s59
00:06:36.800 --> 00:06:52.960
So typically when you write an assertionbased test, you um use your business context and judgment to encapsulate the correct behavior of software up front and and encapsulate that into your assertion based test.

s60
00:06:52.960 --> 00:07:01.599
In meticulous, we just show you what's different and you exercise your judgment at review time or or or an agent agent does.

s61
00:07:01.599 --> 00:07:07.280
Um give me one second to switch this back.

s62
00:07:12.479 --> 00:07:13.280
Cool.

s63
00:07:13.280 --> 00:07:22.800
Um, so there are sort of three technologies that are really important for helping build a mental model of the tool and how it works, how it works and why it works.

s64
00:07:22.800 --> 00:07:26.160
So the first one is that everything is mocked out.

s65
00:07:26.160 --> 00:07:37.120
So um at record time we record all the network requests and responses and then at replay time we stub those in and we perform that mocking for two reasons.

s66
00:07:37.120 --> 00:07:39.120
One is to make the test item potent.

s67
00:07:39.120 --> 00:07:44.400
So you can run them a thousand times in a row and get the same results each and every time.

s68
00:07:44.400 --> 00:07:53.840
And the second reason is it means that every test is isolated from every other test which eliminates the risk of race conditions and allows you to paralyze the test horizontally

s69
00:07:53.840 --> 00:07:56.960
and get get the results in it in a few minutes.

s70
00:07:56.960 --> 00:08:07.520
The second key technology is that you get radically less or orders of magnitude less flakes with meticulous than any other tool including cypus or playright.

s71
00:08:07.520 --> 00:08:14.240
And the reason why is we augment the browser from the scheduling engine layer up to be fully deterministic.

s72
00:08:14.240 --> 00:08:18.560
So when browsers were first designed determinism was not a design goal.

s73
00:08:18.560 --> 00:08:21.120
And so there's many different sources of randomness.

s74
00:08:21.120 --> 00:08:30.879
For instance, an animation spinner depends upon the clock speed of the CPU of the machine rendering the browser or the interle between set timeout and set interval.

s75
00:08:30.879 --> 00:08:40.399
And so we handle all of that randomness and uh make the browser deterministic or near deterministic so that you have radically radically less flakes.

s76
00:08:40.399 --> 00:08:51.200
And then the third technology which is the most um important one for grocking like why this works is if you want this to be exhaustive it has to cover

s77
00:08:51.200 --> 00:08:58.640
every feature plan combination every permission every role every config every possible branching through the application.

s78
00:08:58.640 --> 00:09:00.640
So how do you how do you do that?

s79
00:09:00.640 --> 00:09:11.839
So one technique that we use is that for every session or flow that we record, we replay it once against main branch or master branch and we monitor what lines of code get executed

s80
00:09:11.839 --> 00:09:22.959
and then we build a map or an index that maps each individual workflow to lines lines of code executed and choose a subset of workflows that maximizes code coverage across your application.

s81
00:09:22.959 --> 00:09:26.320
And typically uh give me one second.

s82
00:09:26.320 --> 00:09:40.320
Typically I would say that code coverage in our opinionated view is a terrible metric for every tool in the world apart from meticulous and the reason why is you could write a cypress test or playright test that covers 100%

s83
00:09:40.320 --> 00:09:59.920
of your codebase but only makes a single assertion and so code covered is not the same as code tested but with meticulous it actually is approximately the same because we do this really unusual and radical thing which is we're taking a screenshot at every possible moment throughout a flow.

s84
00:09:59.920 --> 00:10:07.200
So if there's a 412 set flow and there's a single pixel diff, meticulous meticulous will will will flag it.

s85
00:10:07.200 --> 00:10:10.880
And so you can start to see how all these technologies layer together.

s86
00:10:10.880 --> 00:10:16.640
Um you only get exhausted verification because of this code coverage algorithm.

s87
00:10:16.640 --> 00:10:21.360
That algorithm only really works because you take a screenshot at every possible moment.

s88
00:10:21.360 --> 00:10:26.480
If you take a screenshot every possible moment, you're taking on the order of tens, hundreds of millions of screenshots.

s89
00:10:26.480 --> 00:10:30.240
And so, traditionally, you would just drown in noise or drown in flakiness.

s90
00:10:30.240 --> 00:10:33.120
And so, you have to solve flakiness at the root level.

s91
00:10:33.120 --> 00:10:40.079
Um, which is which is what we what we what we do by by augmenting augmenting the browser.

s92
00:10:40.240 --> 00:10:45.040
Um, these are some nice things that our customers customers have said about us.

s93
00:10:45.040 --> 00:10:48.560
Um, and that concludes today's talk.

s94
00:10:48.560 --> 00:10:57.279
If you're interested in finding out more then come to booth LG6 and thank you so much for listening.

s95
00:10:57.627 --> 00:10:59.627
[applause]

s96
00:11:12.692 --> 00:11:14.692
[music]