Agents in Production: How OpenGov Built and Scaled OG Assist - Gabe De Mesa, OpenGov

AI Engineer · 18 min · 172 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Hi everyone.
2. 00:01My name is Gabe de Mesa.
3. 00:03I'm an engineer here at OpenGov and today we're going to be talking about agents in production, specifically how OpenGov built and scaled OG Assist.
4. 00:13Uh so um this presentation is going to be jam-packed with just so much good stuff.
5. 00:21Uh we're going to talk about uh AI agents.
6. 00:23We're going to talk about our harness.
7. 00:25We're going to talk about um evals, observability, traces.
8. 00:30We're going to talk about um tools and skills.
9. 00:34Um It's There's going to be a lot of good stuff in here.
10. 00:38We're going to talk to you guys about uh what we do at OpenGov and how we operate at the scale that uh we operate at um in production, so you'll be able to see a real use case and workload
11. 00:50uh with AI agents.
12. 00:51Um so without further ado, let's get started.
13. 00:56Okay, agenda.
14. 00:57So just really quickly going to go through uh high-level what we're going to talk about today.
15. 01:03Uh I'm going to tell you guys a little bit about OG Assist and what uh OpenGov is.
16. 01:07I'm going to tell you guys the origin story of how this all kind of came to be.
17. 01:12Uh we're going to talk about OG Assist's uh big bet on effect.
18. 01:17Uh a little bit into our core agent loop.
19. 01:20Uh we're going to talk about the A2A protocol, evals, and sandboxing.
20. 01:24We're going to talk about how we manage long context.
21. 01:27We're going to talk about um monitoring, observability, how we collect feedback uh and how we iterate on that feedback.
22. 01:35We're going to lastly uh also talk about tools and skills and how at OpenGov uh we use um AI not only externally uh that we uh serve to customers, but also internally to improve our development workflows.
23. 01:51Just a little bit about me before we go any further.
24. 01:54My name is Gabe.
25. 01:55I'm a software engineer here at OpenGov.
26. 01:57I work on the AI agents team and I'm one of the folks that helped build OG Assist and some of the systems that you guys will be seeing today.
27. 02:07So, a little bit about OpenGov.
28. 02:09OpenGov is a software company on a mission to power more effective and accountable government.
29. 02:16So, OpenGov sells ERP software.
30. 02:18That's things like budgeting, procurement, asset management, and permitting.
31. 02:21And we were founded about 14 years ago and what's cool is um we have this thing called OG Assist and OG Assist is this little button on the top of all of our products in the in the navigation bar.
32. 02:38And what's cool is all of our product suites and product teams have built tools and skills in order to power this button.
33. 02:48So, for example, if I open up this this if I click this button and I open up OG Assist, it says, "Hey, I'm going to ask about rate codes, which is very specific to utility billing, the current product that I'm in."
34. 03:03And you can see that inside of this kind of chat interface, I'm able to speak to an agent and the agent is able to make tool calls in order to
35. 03:12look up information against data inside of that suite.
36. 03:15So, it's really cool to be able to kind of first-party create these experiences through the capability that we've built called OG Assist.
37. 03:27Okay, so just a quick story about how this all came to be.
38. 03:31So, a little while back we we we saw that AI was really starting to take off and a principal uh spun up this new team called the AI agents team and asked me to join.
39. 03:42And um instantly I said yes and OG Assist started to to grow and we started to integrate OG Assist into all our products.
40. 03:51And uh not only our back-end capabilities, but also our front-end capabilities as well.
41. 03:56So, you'll see that one of the capabilities that we give the agent is it's able to um see what's on the screen and and see and and and take action
42. 04:06on what's on the page.
43. 04:08So, you could see that uh I'm asking the agent here, "Hey, hey, what's on the screen?
44. 04:12Can you maybe highlight uh some of the next steps that I could take?"
45. 04:16So, you can see that the agent here is thinking.
46. 04:19It's saying, "Okay, what tools do I have available to use?"
47. 04:21And "Hey, let me go and highlight something that you could actually click on and and tell you more about it."
48. 04:26So, just another capability of OG Assistant.
49. 04:29Just a little short story about how this all came to be.
50. 04:35So, the big bet on Effect.
51. 04:37Um so, I really wanted to include this slide because um here on the Agents team, we made a huge bet to um to to bet on Effect.
52. 04:47And suffice to say it's paid off in dividends.
53. 04:52Um we write Effect.
54. 04:54So, Effect is this library for TypeScript.
55. 04:56It's open source and it helps you write better um TypeScript code.
56. 05:01Uh you know, it's got a lot of uh stuff baked in it like a a schema similar to like Zod, if you've ever used that.
57. 05:08It's also got um things for error handling, uh for logging, for traces, for uh it's just got so much in there.
58. 05:17It really helps write better code and structure your code better and uh helps with architecture, spinning up new services for uh and and for us on the Agents team, really
59. 05:28helping uh design and build the the core agent loop.
60. 05:31So, you'll see throughout this presentation sprinkled in um how Effect on our team uh has paid off in dividends.
61. 05:40So, we we really love Effect here at OpenGov and we encourage other folks to try it out and um yeah, let's keep going.
62. 05:50The Effect Native Loop.
63. 05:52So, originally we were on LangGraph and that was fine until the team really started to scale uh and our use cases started to evolve.
64. 06:02So, we decided to move over to our own kind of Effect Native Agent Loop to have full regency over this uh Agent Loop such that if we have complex use cases or features that we need to build,
65. 06:17we could kind of get in we we had full control of the of the Agent Loop.
66. 06:22And not only that, but now we're fully on Effect.
67. 06:25So, all the cool things you get with Effect is now propagated throughout the entire Agent Loop, like the tracing, structured concurrency, the logging, everything is more fine-grained control, and it it really allows us to really unlock the full potential
68. 06:41uh having our own Agent Loop from the ground up.
69. 06:45Um so, another thing I wanted to mention is on the left side, you'll see a code example.
70. 06:51This is really the basics of the Effect Loop that we're using.
71. 06:56Uh we're using this thing called the Effect AI package, and in that package, there's this thing called um there's a chat and a language model.
72. 07:05So, with the chat, you can instantiate like an a chat, for example, and then you could stream text using um that that kind of stream text function, you could pass in a prompt.
73. 07:16And what's cool is uh with a language model under the hood uh of since we're kind of doing dependency injection, we could pass in a different language model if we were to
74. 07:28uh hot swap to another one, for example.
75. 07:31So, really just having full control of our own Agent Loop just kind of gives us all the levers, and it really just unlocks the full capabilities of the model, and uh for the team as well, to have full
76. 07:43agency over this this loop.
77. 07:46Another thing I wanted to mention is the agent-to-agent protocol.
78. 07:50So, here on the agents team we've had a lot of success with this protocol.
79. 07:54So, this protocol being the protocol that Google created.
80. 07:58Um kind of an open protocol for agents to intercommunicate, but um we found this very useful for uh defining our agent routes.
81. 08:08So, for example, in the back end and our model and our schema to follow this kind of agent protocol.
82. 08:15So, we modeled So, for example, there's this thing called an agent card which you see here and it's got the name of the agent, a description, etc., right?
83. 08:24And having this kind of rigorous protocol, this rigorous spec really helped drive our development and drive alignment because, you know, all we had to do was um align with this spec and follow this spec and
84. 08:40we knew that this was kind of the contract that our front end and back end would both consume and and produce.
85. 08:48So, um this uh I would say also has been uh very helpful for us.
86. 08:54And And what's really cool is A2A has a lot of extensions, right?
87. 08:57So, you could extend the protocol uh add in like metadata.
88. 09:01Uh there's also A2UI.
89. 09:04Um so, lots of fun stuff uh with A2A protocol, but uh this is kind of what's worked for us.
90. 09:12So, just sharing that with with you folks.
91. 09:16Feedback and evals.
92. 09:18So, here the quote is, "Shipping is the start, not the finish."
93. 09:22So, what we do here uh on the agents team is we have kind of multiple ways we do evals and collect feedback.
94. 09:29Um Obviously, you know, we'll have folks uh call in or or email us or just let us know and tell us, but the main way is we have this thumbs up and thumbs down mechanism and here
95. 09:41someone is able to tell us, "Hey, this this worked really well.
96. 09:44This was a great response."
97. 09:45Or, "That wasn't a great response."
98. 09:46And that signal we take and we're able to iterate on.
99. 09:50Uh, and we can take it back and help improve, uh, you know, the response in the future.
100. 09:55Um, we also have automated evals, so in in the in our CI we we have evals that run against real completions, so we could test the prompt against, "Hey, did it hit some tools?
101. 10:07Did it do what it's supposed to do?"
102. 10:09And that also helps with our accuracy.
103. 10:11So, uh, those automated evals in conjunction with collecting feedback really help us, um, improve our uh, our our tools, our skills, um, our harness and and that's really how
104. 10:27how we're able to iterate so fast and so quickly.
105. 10:31Humans in the loop.
106. 10:32So, this is a really cool feature we built where we deterministically interrupt the agent loop if there is a tool call approval required.
107. 10:41So, if an agent tries to make a tool call that it needs human approval for, it'll show this UI and the human uh, can click accept or reject.
108. 10:50So, explicitly rejecting or explicitly accepting, uh, the action that the agent is trying to make.
109. 10:55And this ensures that, uh, you know, we're building trust and also ensuring that, uh, you know, we're being safe, especially when the agent's trying to do a mutating operation
110. 11:06and always always always making sure that, um, humans are in the driver's seat.
111. 11:13Sandboxing.
112. 11:14So, another thing that we, uh, worked on, um, kind of similar to the safety slide we just saw was, um, whenever an agent tries to execute code or tries to create files,
113. 11:27it does so in a sandbox.
114. 11:29So, we gave our agents sandboxes such that it could spin up these sandboxes on demand and it could use those sandboxes to honestly write code, execute code, create files and it's kind of this safe ephemeral
115. 11:44isolated space such that the agent can can can take action in there and not and we don't have to worry about uh any risk to you know our our production systems.
116. 11:55Um and it's it's really cool cuz they also get uh tied uh tear down at the end so um in this example I said, "Hey, create a PDF uh for the folks of the AI Engineer Conference 2026."
117. 12:09And um allow me to download it so I could share it with them.
118. 12:12And you could see that the agent created this really cool PDF inside of that sandbox.
119. 12:17So just really wanted to cover the sandbox feature and just give you a brief kind of overview of of sandboxing.
120. 12:27Long context.
121. 12:28So um inside of Ogie Assist, we have hit many hurdles like with especially with legacy models now like with uh token limits or just way too much just completely
122. 12:43overloaded with context especially as conversations get longer.
123. 12:49So we found that having some sort of um rolling summarization was more effective than you know always stuffing in the latest and most recent uh messages uh rather just
124. 13:06you know give like a running summary after n number of messages and uh maybe you only want the like n minus five most recent messages or n minus 10 most recent messages, right?
125. 13:18Um and it may be that you're only talking about a specific topic now, but you may want to refer to uh context earlier like a hundred messages above, then
126. 13:29um it uh that's where kind of the the memory component comes in cuz when you have this rolling summary of a a really long conversation, then you could do recall over that uh summarization.
127. 13:41And um you know, if you ask the agent, "Hey, remember that thing that we talked about?"
128. 13:46then the agent within the thread will be like, "Yeah, I do know what you were talking about.
129. 13:49I have kind of this short little tidbit."
130. 13:51And it can, you know, follow up and and and do more kind of with that uh summary rolling summary in mind.
131. 13:58So, that's kind of how we handled long context and memory, and it's worked pretty well for us.
132. 14:02So, uh just wanted to share a little bit about that and how we've uh solved the long context problem.
133. 14:11UI on the fly.
134. 14:12So, um in this example, I said to the agent, "Hey, generate me a long essay, but give me some examples about what the essay could be about."
135. 14:20So, what's really cool is the agent had this primitive registered uh uh this form, and it was able to build out this form for me at runtime and give me some options of what I could choose from.
136. 14:31So, it feels very personal and it feels very kind of in the moment that it's able to to give me these options just at runtime.
137. 14:38So, um this is kind of just a short little thing I wanted to include here about generative UI and how we uh are able to render UIs on the fly.
138. 14:53You can't scale what you can't see.
139. 14:56So, um this uh this kind of section is about tracing uh and observability.
140. 15:04Really, what's cool about Effect is you kind of get tracing out of the box.
141. 15:10Um you know, when you use these effect functions, they all get kind of tagged automatically with like these spans, and kind of the span gets picked up and feeds into these traces so that you can kind of get these
142. 15:23kind of drill downs of these function calls.
143. 15:26So, here's an example of a trace from the effect team.
144. 15:31I have it linked.
145. 15:33You could see that like, "Hey, when you hit this API, go to this endpoint, to this handler."
146. 15:38And you know, etc. And it takes And what what's really cool is you could profile all your traces, right?
147. 15:44So, this takes a total of this many seconds and you can see where the bottleneck is here.
148. 15:49If there's a failure, you can cross-reference data across services.
149. 15:53So, really, really important, especially working in agentic systems where we're we're integrating with other teams and other APIs and other um other platform capabilities.
150. 16:06So, what's cool with effect is you get all this tracing out of the box and it really makes building this agentic experience, debugging it, and maintaining it just a breeze.
151. 16:19Tools and skills.
152. 16:20So, uh not only did we make a big bet on effect, but we also made a big bet on tools and skills.
153. 16:27So, we believe that tools and skills are really all you need.
154. 16:31And in this case, you could see on the left, we have this tool called get dad joke.
155. 16:37And this is kind of the effect way and kind of the building blocks of how we do things here at OpenGov, but you know, this is pulled from the effect website, but you could see, "Hey, this is how you make a tool."
156. 16:50And then you add it to a tool kit, which is a collection of tools, and then you can register this this tool kit with the with the language model.
157. 16:58So, for example, if you had a prompt that said, "Hey, generate some dad jokes about pirates."
158. 17:04Well, guess what?
159. 17:05The agent has a tool uh that can help get a dad joke.
160. 17:11So, um really, this is the the building blocks of how we did tools and eventually skills and really just is has paid off wonderfully for for our organization.
161. 17:25So, um really recommend trying out this effect AI package from uh effect and um just trying out building out your own tools and skills.
162. 17:38Developer velocity.
163. 17:39So, not only do we build agents for our customers, but we also use agents um here internally uh in OpenGov.
164. 17:48So, we uh use a lot of Claude and Cursor uh and it's just really been a game-changer for our team.
165. 17:55Um it's funny because we're building tools and skills for customer-facing uh agents and and that has been great, but we're also building them internally as well to help accelerate our development workflows.
166. 18:05So, things like Claude, Cursor, Claude agents, they really help accelerate how we read, write, review code and and ship.
167. 18:14Um so, it's it's just been such an accelerate.
168. 18:16So, definitely just wanted to mention that uh before we wrap up.
169. 18:22That's it.
170. 18:23Thanks so much for watching.
171. 18:24You've made it to the end.
172. 18:25Let's build agents that ship to production.