HTML is All You Need (for Agents to Make Graphics) - Amol Kapoor, Nori

AI Engineer · 7 min · 115 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:07[music]
  2. 00:10Hi, I'm Amol, CEO of Nori Atentic.
  3. 00:14[music]
  4. 00:14We deploy an AI employee that understands your company, your code, docs, Slack,
  5. 00:20[music]
  6. 00:21and other kinds of data.
  7. 00:23We spend a lot of time thinking about how coding agents really work.
  8. 00:28Most people think coding agents only write code.
  9. 00:31But if you ask me, that's just bad marketing.
  10. 00:33[music]
  11. 00:33Forget the name for a second.
  12. 00:35Coding agents can do almost anything.
  13. 00:38There's just one trick.
  14. 00:39You have to be able to think like an agent to get it to do what you want it to do.
  15. 00:43[music]
  16. 00:44Today, we're going to talk about how we use coding agents to do something most people think agents are terrible at, make visual artifacts, like slides, docs,
  17. 00:53[music]
  18. 00:53and yeah, even video.
  19. 00:58[music]
  20. 00:59Every day, the world pours something like 34,000 human years into making slide decks.
  21. 01:05Most of that time isn't the thinking, it's the fiddling.
  22. 01:08[music]
  23. 01:09A deck that takes 10 hours should really take about 25 minutes once you remove all the formatting and the branding and the moving things around.
  24. 01:18Say you need to make a slide.
  25. 01:20What do you do?
  26. 01:21You open a tool, PowerPoint, Slides, Figma, Canva, and then you start manipulating a canvas.
  27. 01:29Every one of these tools is built for human hands and human eyes.
  28. 01:32Click, drag, drop, resize, snap to grid, all motions and patterns that make sense for our geospatial view of the world.
  29. 01:41There is a data structure underneath, but it's in a format that only the application can read.
  30. 01:47What happens when you hand these tools to an agent?
  31. 01:49Well, the output comes out all wrong.
  32. 01:52Things overlap in weird ways, you can't see the text, there's no alignment, It's just garbage.
  33. 01:58AI skeptics say that it's not just the tools.
  34. 02:01Agents fundamentally can't reason about space.
  35. 02:04And there are whole benchmarks like Arc AGI that are built exactly around that premise.
  36. 02:10There's a famous little test for this from developer Simon Willison.
  37. 02:14He asks every new model the same thing.
  38. 02:17Can you draw a pelican riding a bicycle?
  39. 02:20But there's a trick.
  40. 02:21The agent is only allowed to use SVG.
  41. 02:25It's a quick gut check for whether a model can reason about space at all.
  42. 02:29Here's some examples of what the models actually give you on this test.
  43. 02:32And
  44. 02:32[music]
  45. 02:33yeah, these are pretty bad.
  46. 02:35Like genuinely deeply really bad.
  47. 02:38[music]
  48. 02:39So, does that mean it's hopeless?
  49. 02:41Agents are just doomed to be bad at graphics?
  50. 02:43No, I don't think so.
  51. 02:45If you ask me, it's not the model, it's the medium.
  52. 02:49If I asked you, someone who is presumably human, to handwrite an SVG of a pelican, you wouldn't be able to do that either.
  53. 02:57[music]
  54. 02:57SVGs are just a wall of numbers.
  55. 03:00You can't go from a wall of numbers to a pelican, you just can't see that way.
  56. 03:05That's just not how people think.
  57. 03:07We think graphically.
  58. 03:09So, we build tools that let us draw on a canvas.
  59. 03:12Figma MCPs, PowerPoint CLIs, screenshot and replace loops, what do all of these agent tools have in common?
  60. 03:19They all approach the problem like a human.
  61. 03:23But an AI is not a human.
  62. 03:25Asking an AI to use a canvas is like asking a human to write SVG by hand.
  63. 03:29It doesn't really make sense.
  64. 03:31You need to give the AI tools based on how it thinks, not in pixels, in language.
  65. 03:37Words, tokens, structure, that is its native medium.
  66. 03:42Imagine a language that's incredible at describing layout.
  67. 03:46That models have seen and trained on billions of examples of.
  68. 03:50That they understand intuitively.
  69. 03:52that renders to pixels and can run everywhere.
  70. 03:57Oh, right.
  71. 03:59HTML lets the model think in structure.
  72. 04:02HTML tags have meanings built into the language.
  73. 04:05A heading, a chart, a grid, and the browser turns it all into pixels.
  74. 04:09So, the model never actually places a coordinate, and you can get all sorts of visual effects, charts and layouts, fonts and motion, all of it for free.
  75. 04:19Remember that pelican from earlier?
  76. 04:21Now, ask it to do the same exact task, but in HTML.
  77. 04:25Same bird, but now it's in a structure that the model can reason about.
  78. 04:29And you can read and theme and edit every single line of it.
  79. 04:34I spent my whole life building slide decks with PowerPoint.
  80. 04:37So, I always thought that those two things, slide decks and PowerPoint, were synonyms.
  81. 04:41[music]
  82. 04:42But that's just not really true, is it?
  83. 04:44PowerPoint is a tool that you use to make slide decks.
  84. 04:49The deck itself, that's just the presentation mode.
  85. 04:52And as it turns out, no one in your audience is going to care how you got to the presentation mode.
  86. 04:57The editing format is totally arbitrary.
  87. 05:00[music]
  88. 05:00So, you can just pick the editing format that the agents are already good at, HTML, and if you need to render to a different format like PDF later on.
  89. 05:11We use this HTML trick to build all of our slide decks, our board decks, and our sales decks.
  90. 05:16[music]
  91. 05:16These are real things that we actually present and send out constantly.
  92. 05:20We use it for our docs, too.
  93. 05:22It gives our docs color and vibrancy, all while following our brand.
  94. 05:26And of course, we also use it to make videos like this one.
  95. 05:31What you're watching is just HTML and CSS.
  96. 05:34It's literally just divs all the way down.
  97. 05:40Almost everything is better with a little structure and a little bit of color.
  98. 05:44Plain text is a choice, generally a choice of convenience, but it's usually the wrong one if you're actually trying to create something of use.
  99. 05:53Now, I do want to take a quick beat here and point out that a beautiful deck on its own is generally not worth anything.
  100. 06:00You still have to go and get all of that content, all of the things that actually populate that deck, right?
  101. 06:05Well, again, we can think like the model.
  102. 06:09If you just give the model access to your data, say your call transcripts or your emails, you can have the model build the deck end to end.
  103. 06:17Let your agents do all the grunt work while you focus on vision and story.
  104. 06:22That's what Nori Sessions lets you do.
  105. 06:24I've built entire board decks for my phone on the subway during my commute.
  106. 06:29Why?
  107. 06:30Because our Nori bot lives in the fabric of our company.
  108. 06:33Of course, Nori ships with everything you need to make this all work, so don't bother reinventing the wheel.
  109. 06:39That's my little spiel.
  110. 06:40Thanks for listening.
  111. 06:42If you have just one takeaway, it's this.
  112. 06:45Stop thinking like a user.
  113. 06:47Think like the model.
  114. 06:49Give it the right language, and for graphics, all you need is HTML.
  115. 06:54[music]