Why We Rebuilt Microsoft Office From Scratch for AI Agents - Yijia Liang, Compound AI

AI Engineer · 15 min · 172 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12I was previously an analyst in a previous life.
3. 00:14I was a vice president at Coatue for about 3 years specializing in late stage private software investments.
4. 00:20Spent way too much time in Excel, PowerPoint, and late nights doing that type of work and so decided to split and start building tools to replace my old job.
5. 00:30Quick show of hands, how many people in here have used Muser Instinct in the last day?
6. 00:36All right, I expected more hands, but uh maybe people are too shy to raise them.
7. 00:40And how many people know an analyst in finance that fully delegates their work to AI, like fully hands-off running autonomously?
8. 00:48All right, one.
9. 00:49But a lot fewer.
10. 00:52I think if you think about the question, why did personal agents take off within the last 3 months?
11. 00:57Internally, we came up with three main answers.
12. 01:00The first, reliable background agents.
13. 01:02You can delegate a task, book a restaurant, book a flight, and it'll do it reliably now.
14. 01:08Computer use agents have gotten sophisticated.
15. 01:10The models themselves have gotten better.
16. 01:14The second is proactivity.
17. 01:15It'll do things without you asking.
18. 01:17It's part of that magical experience when Instinct wakes you up saying it's your wife's anniversary, go buy some flowers, or better yet, I already bought the flowers for you.
19. 01:25And finally, integrations.
20. 01:27It's connected to the systems of context that matter most for you.
21. 01:30Email, drive, calendar.
22. 01:34But all the factors that make personal agents work so well doesn't really translate to institutional finance today.
23. 01:43If we think about background tasks, something as simple as booking a restaurant reservation, the analog equivalent in buy-side finance might be how do you go from a data room to a fully prepared set of IC models, slides, and memos,
24. 01:57taking a process that would normally take 4 weeks, multiple rounds of iteration and review, and compressing that to a few days.
25. 02:05If we think about proactivity, your managing director doesn't really want agent rogue running around doing a bunch of things and burning tokens without them providing and soliciting input.
26. 02:16And then finally, if you think about the context that matters, it's not just email, it's not just calendar, it could be a data room with 15,000 files, it could be SharePoint box with petabytes of unstructured data a firm has collected in its 40-year history.
27. 02:29So, how do you deal with these problems?
28. 02:32So, the purpose of today's talk is to discuss a bit more about how we at Compound have approached the problem of solving long-running agents in finance.
29. 02:41On one hand, it's creating a real teammate that collaborates with you and the deal team.
30. 02:46So, that's multiplayer embedding that into the app.
31. 02:49On the other hand, it's solving for how you can you run long tasks in parallel in the background, reliably execute them.
32. 02:57And so, for that, we actually rebuilt Office from scratch.
33. 03:01And then finally, it's how do you manage the context of everything, everything we just discussed, petabytes of data in SharePoint and Box.
34. 03:07And we'll talk a little bit about how we've designed a solution for that.
35. 03:12So, the first question we have to ask, I think, is why does multiplayer actually matter for finance?
36. 03:17If you look at the anatomy on a deal team, right, the type of tasks that matter for long-running type of workflows, for example, going from a data room into all the sorts of materials that make its way in front of an investment committee,
37. 03:30it doesn't work for the single-player siloed workflows.
38. 03:33Today, for people who have deployed Claude or ChatGPT in their institutions, it's all single player.
39. 03:39People working in a plugin, people working in the web app, producing outputs, but invariably, those people need to collaborate with their team.
40. 03:47And what does that look like?
41. 03:48It means downloading the file, uploading it into email, uploading it to Teams, asking for feedback.
42. 03:55I think the right way to approach this, and this is the way we've thought about at Compound, is how do you get every person who matters on a deal onto the same thread?
43. 04:02How can they all talk to the same agent so that one agent can share the context of everyone who matters for any particular work stream?
44. 04:11When we thought about how do we actually bring multiplayer into a familiar context, we looked again towards personal agents.
45. 04:17Why did they work so well?
46. 04:19Well, you can easily text one in iMessage or you can text one in WhatsApp.
47. 04:23So for agents at Compound, every single one you create comes with a built-in inbox.
48. 04:27Because ultimately at the end of the day for institutional finance, most workflows run through Outlook.
49. 04:32You'll have chains of emails 20 or more running.
50. 04:35You want an agent to be part of every single one of those conversations.
51. 04:40Designing email for agents is actually non-trivial.
52. 04:43It's not just as simple as hooking it up with an email inbox.
53. 04:47It requires A, authenticating, making sure everyone part of that thread has read and write access to that agent.
54. 04:54Ultimately at these financial firms, privacy is incredibly important.
55. 04:57You're always under NDA no matter what deal you look at.
56. 05:00And so you want to make sure everyone on that thread can actually talk to the agent.
57. 05:03But then beyond authentication, you also have to think about how do you design a real human-like experience?
58. 05:08Because after all, we're trying to create a real teammate.
59. 05:10If multiple people email in the same thread in quick succession, you don't want multiple acknowledgements spamming that group.
60. 05:17You actually want to bake in some more latency to make it feel more human.
61. 05:21And so there are other design decisions when thinking about how to create real email agents for enterprise finance.
62. 05:28But multiplayer aside, we realized even after you solicit the input of everyone who matters, if you wanted to scale up the number of tasks and run them all in parallel,
63. 05:37existing off-the-shelf solutions simply don't cut it.
64. 05:40We evaluated a lot.
65. 05:41The desktop plugins of Excel, PowerPoint, and Word.
66. 05:45We evaluated computer use using virtual machines to try to scale up the number of agents that could work in Excel, PowerPoint, and Word natively.
67. 05:52And we also evaluated a ton of off-the-shelf engines, whether those are open-source packages like Open PyExcel or commercial engines that are available for purchase.
68. 06:01None of them really cut it for what we were trying to look at.
69. 06:04And with coding agents dramatically improving and bring the cost of software down, we decided to just build our own office from scratch.
70. 06:13And like a great AI engineer would, we ran agents in many loops, spend 1.5 trillion input-output tokens and over 8 million lines of code to rewrite office from scratch.
71. 06:23And I'll walk through a little bit of how we approached it.
72. 06:25But it's as simple as pointing agents at an objective function, which is recreate office in full fidelity, and then letting it run.
73. 06:34So, take Excel as an example.
74. 06:37We started with an open-source solution called Iron Calc.
75. 06:39We decided, "Okay, how can you recreate this perfectly for Excel, use office as an oracle, and just run loops?"
76. 06:46Benchmarking it on real files from both real-world corpus that we grabbed online, and also ones that internally we created.
77. 06:53And 800,000 lines of Rust later, we had a performant much more performant compared to Excel, one-for-one replica of the product.
78. 07:03With this, we're actually able to do that for not only Excel, but also PowerPoint and Word.
79. 07:08Right?
80. 07:08So, we ran loops across every single one of these products, and we have faithful recreations.
81. 07:13You can try it for yourself today at the compound.ai.
82. 07:16And it's the same method, loop agents against office as a judge.
83. 07:20Using Rust renders to make sure that the rendering of these products was pixel perfect.
84. 07:26So, what you see in Compound is what you see when you download to office.
85. 07:30We needed one-for-one parity because at the end of the day, these firms only trust what they can download and they what they can send.
86. 07:38Now, let's talk about performance.
87. 07:40Why is a headless engine so much better compared to using a VM or using desktop Excel?
88. 07:45Well, for one, desktop Excel or PowerPoint or Word is bound to that CPU.
89. 07:50With headless engines, you can do tasks in parallel much, much faster.
90. 07:54You can build out a complex model in a much shorter amount of time.
91. 07:58You can recalculate 20 MB Excels 4x faster.
92. 08:02So, every single piece of work of an analyst would expect an agent to do cuz again, their expectation is quality of the human at much lower speed and much lower cost.
93. 08:12You have to have a headless engine to do this.
94. 08:14If you try to scale this up with Excel and PowerPoint, I think you'd be going down the wrong path.
95. 08:21But, it's not just about performance.
96. 08:23Owning the editor actually allows you to create a much richer product experience that allows you to work and collaborate with agents asynchronously much better than you can with other products.
97. 08:32So, this entire slide deck you're seeing today, I built in Compound just by dropping in comments.
98. 08:39I had it create a first draft, and then as you can see here on the UI, I dropped in comments for every single piece of formatting.
99. 08:45From my consulting days, from my Coatue days, all the learnings I had on what makes a great slide, I imparted with the agent.
100. 08:52This can also work for multiplayer.
101. 08:53You can have multiple people part of the same team using the same multiplayer infrastructure we set up, dropping in comments.
102. 08:59So, for a team of five, you can rip through a 100-page deck in an afternoon rather than sitting in a room revising it for multiple weeks.
103. 09:08Finally, with respect to building your own engine, it also allows us to run iterative checking loops much more as well.
104. 09:14So, if you have a task that, for example, is writing a complex model, it might take a few hours.
105. 09:21But, how do you How can we make that faster?
106. 09:22How can we make it more accurate?
107. 09:24If you're using, for example, an OpenPyXL code library, there's no render you can use to check formatting.
108. 09:29You're just guessing and hoping that the model writes the right thing, and maybe you do some parsing to see a cell has a particular color, has a particular font.
109. 09:37If you try to use Microsoft's own interface, you're going to be limited by what Microsoft provides you, which is admittedly very little.
110. 09:44So, with an own engine, you actually own the write, render, check, and fix loop.
111. 09:49So, you can rapidly iterate on the output for the user.
112. 09:55So, the last thing I sort of want to discuss today beyond multiplayer and building our own office is how to deal with the context that these firms have.
113. 10:05For firms on the buy side, they deal with situations where cross is both breadth and depth of data.
114. 10:11You may have situations where you have a data room of over 15,000 files, and then files in that data room might be well over 20 megabytes.
115. 10:18People in finance complain all the time, "Hey, I have an Excel model that's 30 megabytes.
116. 10:22It takes 5 minutes to recalculate because we have 2 million formulas and five sensitivity tables that need to recalculate every single time."
117. 10:29So, how do you deal with this?
118. 10:31Well, for breadth, I think again we take a look at a page out of the coding book and apply agent swarms to the large number of files.
119. 10:39No longer do you have to index everything in a vector DB unless it's for it's for a very specific use case.
120. 10:44It's actually much preferable to use coding agents to read files either through grep or through some other methodology, and then have a lead agent slice the contents of the file directory,
121. 10:55and then have swarms of agents come back to you with insights.
122. 10:58And then you iterate that in a loop until you're satisfied with the output required.
123. 11:04Furthermore, through this process you're actually able to ground every single finding inside citations in the source document, so that you know exactly what you show on the screen is linked to some number
124. 11:13that the analyst can trust.
125. 11:16If we think about the depth of data, right?
126. 11:19Take that 30 megabyte workbook.
127. 11:21How can you actually process this more accurately?
128. 11:25In today's world, if you were to just use ChatGPT or Claude off the shelf to process that 30 megabyte model, it would write code, it would use open-source packages like openpyxl.
129. 11:34It would be expensive and slow because it's printing out every single piece of context about that model in code for the model to read.
130. 11:42But because we've developed our own engine for Excel, we're actually not only more accurate, but a lot faster and cheaper because the engine processes it and then lays it out in a very
131. 11:51easy-to-read and easy-to-process format for the model.
132. 11:56This means we can expose a much richer set of APIs and a much richer set of tools for the agent to work over.
133. 12:02And that allows us to process these models in a much faster manner.
134. 12:08So, if we sort of take a step back and think about, "Okay, what is all of this context built for?"
135. 12:13At the end of the day, what we want to do is prevent people from spending time finding files, from reminding the model this is how to do things.
136. 12:21Instead, they should be focused on importing their judgment on the deal process.
137. 12:24They should be focused on tailoring, training agents, and deploying them on every part of the deal flow.
138. 12:31And so, it goes from the bottom level, connected sources, off-the-shelf, you know, connectors to SharePoint or Outlook, to the cited findings grounded in the sources that matter.
139. 12:41And then finally, embedding in the preferences, company knowledge that matters.
140. 12:45How do you do it, model?
141. 12:46How should it look?
142. 12:47Which differs by every single firm.
143. 12:48So, that for humans, they spend most of their time imparting judgment.
144. 12:53They spend most of their time thinking about is this actually good deal?
145. 12:56Not did all the numbers tie?
146. 12:58Does the model look right?
147. 12:59Will my VP yell at me because it looks different way?
148. 13:03So, Oh.
149. 13:06Cursor needs to update.
150. 13:10If we take a step back and think about the type of tasks that currently are on autopilot um versus what are on copilot, I would say there's already several finance tasks you can do pretty well with off-the-shelf tools.
151. 13:23Spreading comps, as the ChatGPT head of finance mentioned earlier, they have now their own products for this.
152. 13:29But I think there are a lot of longer running tasks that are not capable with existing solutions.
153. 13:33How do you draft a 50-page IC presentation off the cuff?
154. 13:37How do you get agents to build out a model exactly the way you want it for a private equity professional or a hedge fund professional?
155. 13:46I think those are tasks that with off-the-shelf tools still exist as co- as sort of within the copilot realm.
156. 13:52People want to be embedded deeply in that process without letting go.
157. 13:57But with some of the tools we've built today, I think we're able to expand the aperture for which autopilot can take over in finance firms.
158. 14:05So, rather than just using tools embedded in Excel as plugins, hand-holding along the way, you can start delegating.
159. 14:12You can start becoming more ambitious about what AI is capable of.
160. 14:16If I think about where we are today in the world of finance, it very much reminds me of when Opus 4.5 launched last December, where people stopped using code
161. 14:27as a simple co-pilot to generate the next file or the next line of code, but they started delegating far more ambitious tasks.
162. 14:34They started realizing, maybe I can just truly vibe it.
163. 14:37If I have sufficient checks, if I have sufficient tests, maybe I can just trust all the code it writes.
164. 14:44I think with some of the tooling we've built today, along with what we're planning in the future, these are the type of problems you have to solve before you get to that point for finance.
165. 14:53How can analysts, how can the entire deal team become more ambitious with AI?
166. 14:58How can they just drop in a data room and decide, okay, I'm going to trust every single one of the outputs, and then let's do that tenfold so we can see more deals, so we can expand the aperture of what matters.
167. 15:09So, to conclude, enterprise agents are certainly coming to finance.
168. 15:13I strongly believe we're at an inflection point in the adoption curve, and that people are slowly becoming more ambitious, and they're moving away from co-pilots to auto-pilots.
169. 15:21And so, this was sort of how we've approached it here at Compound.
170. 15:24If you work at a firm and you're interested in deploying our agents to drive more of your deal work, please reach out.
171. 15:30Or if you're engineer and you're curious about talking about our approach and sharing ideas, please either reach out online or I'll be around if you want to find me.
172. 15:37Thanks so much.