Why AI Didn't Actually Make You Ship Faster — Gabriel Spencer-Harper, Meticulous

AI Engineer · 11 min · 96 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:12Hello everyone.
  3. 00:13Hi.
  4. 00:14Um, my name is Gabe.
  5. 00:15I'm one of the co-founders and CEO of Meticulus.
  6. 00:19Um, I'll just wait one more minute for people to come in and then we can and then we can get started.
  7. 00:24Um so in today's demo or talk I'll talk about verification of code.
  8. 00:31Um I'll spend one minute introducing meticulous and then one minute giving some background context on the company and then sort of eight minutes giving a demo and technical technical
  9. 00:41overview.
  10. 00:43Um so if you have exhaustive verification then you can ship code at the speed that your agents write it.
  11. 00:53If you don't, then someone somewhere at your organization is spending time verifying code and that is now your new bottleneck and you're forced into a position where you're trading off velocity
  12. 01:07against bugs and user experience.
  13. 01:12So I'll give um super quick context on Meticulus.
  14. 01:15So, Meticulus gives you exhaustive or near exhaustive verification of your front-end codebase with zero developer effort.
  15. 01:24Um, it's deployed across the entire engine organization at all of these companies like Discord, Whiz, Dropbox, Notion, 11 Labs, Launch Darkly, and many, many other great companies.
  16. 01:37So, if an engineer touches front-end code at one of these companies, they use Meticulus every every day.
  17. 01:43However, before we get into the product and talk more about verification, um, let's just cover the current sort of state of the world.
  18. 01:51So, AI writes code faster than humans can review it.
  19. 01:57And review and verification is now the new bottleneck.
  20. 02:04The other state of the world is that assertionbased testing alone isn't enough.
  21. 02:09So no matter how hard a human or an agent tries to define the correct behavior of software a priority upfront in assertions the space of possible regressions is too vast to exhaustively
  22. 02:26cover.
  23. 02:28And so a good clarifying question is if an engineer in your organization uh AI generated this pull request, would you feel comfortable hitting the the merge merge and ship button?
  24. 02:41And in most organizations, the answer is no. You would want to go and do the hard work of checking out all the different feature flags, all the different roles, permissions, settings, configurations,
  25. 02:54and the various different edge cases in order to understand the exhaustive impact of this of this change and see if you can spot any issues.
  26. 03:04So there are three sort of like downstream consequences of that current state.
  27. 03:08The first one is bugs or regressions uh which have a a business impact.
  28. 03:14The second one is engine organizations will spend doubledigit percentages of their time maintaining end to-end test suites.
  29. 03:25So across manual validation, review, uh debugging flakes or updating or maintaining the test suite, um all spend a lot of time maintaining maintaining maintaining tests.
  30. 03:38And then the third like problem or downstream downstream consequence of of the state of the world is that um if you if you can get exhaustive verification then you can program in a new and different way.
  31. 03:53So all of a sudden you can bump all your dependencies, you can do sweeping refactors, you can make any AI generated change with complete complete confidence.
  32. 04:04So then the question becomes um that sounds great.
  33. 04:07How do you how do you get exhaustive verification?
  34. 04:10What does it look like and how do you how do you get there?
  35. 04:13Um so I'll now give you a demo of how how meticulous meticulous works.
  36. 04:19So, we give you uh give me one second to set this up.
  37. 04:34Great.
  38. 04:35Um, so we give you one line of JavaScript to inject onto nonproduction environments.
  39. 04:42So that's localhost, QA, dev, staging.
  40. 04:45That JavaScript instruments the web browser and records thousands or tens of thousands of flows or workflows.
  41. 04:52So someone clicking the login button, clicking the settings panel, clicking the analytics panel.
  42. 04:57And then when you open a pull request on your CI runner, you spin up your web application on localhost 3000 and Meticulus takes a subset of those recorded workflows
  43. 05:09and it replays them.
  44. 05:10So it dispatches each event one by one.
  45. 05:14And as it dispatches those events, so click the login button, click the senses panel, click the analytics panel, it takes a screenshot at each atomic moment throughout time.
  46. 05:23And so it generates a sequence of screenshots for every workflow, one before your code change and one after your code change.
  47. 05:30It then diffs those two sequences together to show you what would be different about your application if you were to merge that code in before before you do.
  48. 05:41So in this example, there's two changes here.
  49. 05:46One is someone's changed the name and the second is they've introduced an error to a to a drop down.
  50. 05:52And so when you open a pull request, Meticulus posts a comment typically within a few minutes.
  51. 05:58And when the developer clicks into this comment, the Wi-Fi is slow, so I'm just going to switch tabs.
  52. 06:03But when a developer clicks into this comment, Meticulus will not tell you whether or not there's a bug.
  53. 06:09What it will show you is a set of diffs that show you what's different about the application if you were to merge that code in.
  54. 06:17And so each diff has a before state and an after state.
  55. 06:22And so by reviewing the diff, you can determine whether that change is expected or unexpected.
  56. 06:28So here's the name change on light mode.
  57. 06:30Here's the name change on dark mode.
  58. 06:32And then this is a manifestation of that logical logical error.
  59. 06:36So typically when you write an assertionbased test, you um use your business context and judgment to encapsulate the correct behavior of software up front and and encapsulate that into your assertion based test.
  60. 06:52In meticulous, we just show you what's different and you exercise your judgment at review time or or or an agent agent does.
  61. 07:01Um give me one second to switch this back.
  62. 07:12Cool.
  63. 07:13Um, so there are sort of three technologies that are really important for helping build a mental model of the tool and how it works, how it works and why it works.
  64. 07:22So the first one is that everything is mocked out.
  65. 07:26So um at record time we record all the network requests and responses and then at replay time we stub those in and we perform that mocking for two reasons.
  66. 07:37One is to make the test item potent.
  67. 07:39So you can run them a thousand times in a row and get the same results each and every time.
  68. 07:44And the second reason is it means that every test is isolated from every other test which eliminates the risk of race conditions and allows you to paralyze the test horizontally
  69. 07:53and get get the results in it in a few minutes.
  70. 07:56The second key technology is that you get radically less or orders of magnitude less flakes with meticulous than any other tool including cypus or playright.
  71. 08:07And the reason why is we augment the browser from the scheduling engine layer up to be fully deterministic.
  72. 08:14So when browsers were first designed determinism was not a design goal.
  73. 08:18And so there's many different sources of randomness.
  74. 08:21For instance, an animation spinner depends upon the clock speed of the CPU of the machine rendering the browser or the interle between set timeout and set interval.
  75. 08:30And so we handle all of that randomness and uh make the browser deterministic or near deterministic so that you have radically radically less flakes.
  76. 08:40And then the third technology which is the most um important one for grocking like why this works is if you want this to be exhaustive it has to cover
  77. 08:51every feature plan combination every permission every role every config every possible branching through the application.
  78. 08:58So how do you how do you do that?
  79. 09:00So one technique that we use is that for every session or flow that we record, we replay it once against main branch or master branch and we monitor what lines of code get executed
  80. 09:11and then we build a map or an index that maps each individual workflow to lines lines of code executed and choose a subset of workflows that maximizes code coverage across your application.
  81. 09:22And typically uh give me one second.
  82. 09:26Typically I would say that code coverage in our opinionated view is a terrible metric for every tool in the world apart from meticulous and the reason why is you could write a cypress test or playright test that covers 100%
  83. 09:40of your codebase but only makes a single assertion and so code covered is not the same as code tested but with meticulous it actually is approximately the same because we do this really unusual and radical thing which is we're taking a screenshot at every possible moment throughout a flow.
  84. 09:59So if there's a 412 set flow and there's a single pixel diff, meticulous meticulous will will will flag it.
  85. 10:07And so you can start to see how all these technologies layer together.
  86. 10:10Um you only get exhausted verification because of this code coverage algorithm.
  87. 10:16That algorithm only really works because you take a screenshot at every possible moment.
  88. 10:21If you take a screenshot every possible moment, you're taking on the order of tens, hundreds of millions of screenshots.
  89. 10:26And so, traditionally, you would just drown in noise or drown in flakiness.
  90. 10:30And so, you have to solve flakiness at the root level.
  91. 10:33Um, which is which is what we what we what we do by by augmenting augmenting the browser.
  92. 10:40Um, these are some nice things that our customers customers have said about us.
  93. 10:45Um, and that concludes today's talk.
  94. 10:48If you're interested in finding out more then come to booth LG6 and thank you so much for listening.
  95. 10:57[applause]
  96. 11:12[music]