WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=RGiXcVxSD3s NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/RGiXcVxSD3s.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.480 --> 00:00:05.480 Chat and citations won't save your vertical AI. s2 00:00:06.000 --> 00:00:09.720 I'm sure many of us are building products in what in a vertical industry. s3 00:00:09.720 --> 00:00:12.560 For it be healthcare, legal, or taxes. s4 00:00:12.560 --> 00:00:15.760 And usually doing so, we sell one promise to the customers. s5 00:00:15.760 --> 00:00:18.600 Either we're going to save you money or save you cost. s6 00:00:18.600 --> 00:00:22.200 And usually the pitch goes, AI agents are here. s7 00:00:22.200 --> 00:00:25.440 Uh they will do the work for you while you sleep. s8 00:00:25.440 --> 00:00:29.280 Now, primary interface that we use for this are chat and citations. s9 00:00:29.280 --> 00:00:31.120 To interact with any AI agent, right? s10 00:00:31.120 --> 00:00:32.759 Chats are usually used for inputs. s11 00:00:32.759 --> 00:00:35.120 Uh they allow you to be flexible. s12 00:00:35.120 --> 00:00:37.280 You can talk to the agents as you please. s13 00:00:37.280 --> 00:00:39.720 While citations are used usually used for outputs. s14 00:00:39.720 --> 00:00:44.120 They allow you to see the results, verify the results, and so on. s15 00:00:44.120 --> 00:00:50.800 Now, I'm here to tell you that citations and chat alone will not keep allow you to keep the promise that you made to the customers. s16 00:00:50.800 --> 00:00:54.120 Promise of, you know, saving time and money for them. s17 00:00:54.120 --> 00:00:54.920 So, who am I? s18 00:00:54.920 --> 00:00:55.680 I'm Atul. s19 00:00:55.680 --> 00:00:57.840 I'm the CTO and co-founder at Filed. s20 00:00:57.840 --> 00:01:01.280 Uh I've been fortunate enough to build products for more than a decade now. s21 00:01:01.280 --> 00:01:07.120 Um at Filed, uh we build products for tax professionals in the US. s22 00:01:07.120 --> 00:01:09.840 We have raised more than $17 million to do so. s23 00:01:09.840 --> 00:01:14.760 And about 2 years now doing this, we have seen massive growth in the space. s24 00:01:14.760 --> 00:01:21.360 Uh just to give you an idea, last just last month, we closed more revenue than what we have done in the one year alone before that. s25 00:01:21.360 --> 00:01:25.320 So, the growth has been tremendous and we're very excited for this. s26 00:01:25.320 --> 00:01:29.680 Doing this entire journey for the last 2 years, we've learned quite a bit, you know. s27 00:01:29.680 --> 00:01:37.840 Uh And I think most of the learnings of building these AI agents for the taxes industry is essentially transferable to any other AI product. s28 00:01:37.840 --> 00:01:39.320 Now, chat is great. s29 00:01:39.320 --> 00:01:41.000 Like it allows us to build fast. s30 00:01:41.000 --> 00:01:45.360 It allows customers to interact with products in ways that was previously possible. s31 00:01:45.360 --> 00:01:46.960 It's a communication platform, right? s32 00:01:46.960 --> 00:01:53.560 Uh you can interact with agents to do tasks, flexible tasks that were not constrained to the UIs that you have built. s33 00:01:53.560 --> 00:01:55.200 Uh citations are also amazing. s34 00:01:55.200 --> 00:01:56.960 They ground answers in truth. s35 00:01:56.960 --> 00:02:01.200 Other than allowing the users to verify the outputs, they also serve another purpose. s36 00:02:01.200 --> 00:02:10.200 They allow the agents to essentially be more accurate because now they are forced to present the references to their answers. s37 00:02:10.200 --> 00:02:16.000 Agents are Agents perform much better with So, they reduce hallucinations essentially. s38 00:02:16.560 --> 00:02:19.440 But this is There's a key problem here, right? s39 00:02:19.440 --> 00:02:20.520 Chat is synchronous. s40 00:02:20.520 --> 00:02:28.400 If you think about it, even when you're coding, you're Let's say you're using product code as as as long as you're typing the request. s41 00:02:28.400 --> 00:02:31.440 Once the request is typed, you're waiting for the agent to respond. s42 00:02:31.440 --> 00:02:37.640 This synchronous medium does not allow the customers to leave the platform and go and do their work. s43 00:02:37.640 --> 00:02:42.480 Similarly, citations also puts the verification burden back into the customer. s44 00:02:42.480 --> 00:02:48.959 So, think about going and reviewing the agent's work now one by one to ensure everything is correct. s45 00:02:48.959 --> 00:03:01.320 Especially in vertical industry space like health care, legal, and taxes, this becomes very crucial because now this this adds an extra work, and customers usually complain that no the promise of, you know, s46 00:03:01.320 --> 00:03:05.160 agents doing the work for me while I sleep is not really kept here. s47 00:03:05.160 --> 00:03:09.239 Now, before we dive in, I want to take you through a brief history of how products have evolved. s48 00:03:09.239 --> 00:03:13.400 Specifically, I think there are three levels of abstractions, I would call it. s49 00:03:13.400 --> 00:03:17.840 So, let's imagine a bank to make it an make it easier. s50 00:03:17.840 --> 00:03:19.080 Let's imagine a bank. s51 00:03:19.080 --> 00:03:21.080 The banks used to have a physical presence before. s52 00:03:21.080 --> 00:03:25.959 You would go to a bank branch, you would take the money out by talking to a employee of the bank, right? s53 00:03:25.959 --> 00:03:32.200 And the point here was the the person who's doing the task is the employee of a particular company. s54 00:03:32.200 --> 00:03:39.600 So, you as a user would go in, you would delegate the task to an employee, they would do the task for you, and then, you know, come back with the result. s55 00:03:39.600 --> 00:03:43.357 Be it withdrawing money or seeing how much balance you have and so on. s56 00:03:43.357 --> 00:03:43.480 [snorts] s57 00:03:43.480 --> 00:03:50.040 Now, this essentially meant the bottleneck was the number of users number of employees that a company had. s58 00:03:50.040 --> 00:03:53.040 That's the bottleneck for creating value, right? s59 00:03:53.040 --> 00:03:56.560 Now, as time progressed, digital transformation era came. s60 00:03:56.560 --> 00:03:58.440 Basically, banks became online. s61 00:03:58.440 --> 00:04:04.480 You could Now, the users can essentially open up the mobile app or or the online portal and can make the transaction themselves. s62 00:04:04.480 --> 00:04:06.800 Can actually go and see the balance himself. s63 00:04:06.800 --> 00:04:08.080 This was great. s64 00:04:08.080 --> 00:04:14.400 Because now the bottleneck moved from the number of employees a company have to number of users that the company have. s65 00:04:14.400 --> 00:04:18.239 The more users meant more value you can generate. s66 00:04:18.239 --> 00:04:23.640 Now, I think we have reached a point of another transformation layer called agentic delegation. s67 00:04:23.640 --> 00:04:28.520 So, no longer the users are coming in your product to use the product. s68 00:04:28.520 --> 00:04:33.040 I think they're coming to delegate more and more work to AI agents to do. s69 00:04:33.040 --> 00:04:44.960 So, the point here is um if a user [clears throat] comes to your product and starts delegating work, long-running work, what would happen here is there's the bottleneck of number of users also goes away. s70 00:04:44.960 --> 00:04:54.040 So, it's no longer than amount of value that you generate is the amount of the number of times the user have visited your platform because agents can do the work while s71 00:04:54.040 --> 00:04:57.960 the users have gone to sleep, you know, have delegated the task and went off. s72 00:04:57.960 --> 00:05:04.760 So, the bottleneck has shifted, meaning you can generate more value than ever before for your customers. s73 00:05:04.760 --> 00:05:14.919 Okay, so to make it easier, I You can think of a product You have agentic product as a conveyor belt, you know, and the users as the supervisors of the conveyor belt. s74 00:05:14.919 --> 00:05:16.080 So, think of it this way. s75 00:05:16.080 --> 00:05:20.320 The Previously, in a conveyor belt, usually there's a number of tasks that are happening. s76 00:05:20.320 --> 00:05:26.600 There are workers who are doing the work for you, and there's a supervisor who is delegating the task, you know, and monitoring how the things are going. s77 00:05:26.600 --> 00:05:36.240 Similarly, now AI agents are your workers in that conveyor belt, while, you know, the conveyor belt and the entire infrastructure around it is your product. s78 00:05:36.240 --> 00:05:45.080 So, users can come in, uh which who is the supervisor, can come into the product, can start delegating tasks to the agents. s79 00:05:45.080 --> 00:05:52.320 And this would mean it give it would give an idea of what are the tools that you need to build in your product to make this happen. s80 00:05:52.320 --> 00:05:59.680 So that the agents can do the work and the supervisor or the user is, you know, pretty confident that the work is happening. s81 00:06:00.040 --> 00:06:00.640 All right. s82 00:06:00.640 --> 00:06:02.760 So, how do we build this conveyor belt? s83 00:06:02.760 --> 00:06:05.120 So, think of this this way. s84 00:06:05.120 --> 00:06:14.160 Like, when you're trying to build a product feature, think if a user wants to delegate a task instead of doing it themselves in your platform, what does what would that interface look like? s85 00:06:14.160 --> 00:06:18.360 Essentially, design for delegation, not participation. s86 00:06:18.360 --> 00:06:26.080 Now, to keep it more concrete, I think there are four key pieces when building a agentic product. s87 00:06:26.080 --> 00:06:32.840 And there are four key features or components that you need to have so that this conveyor belt that we're imagining can work. s88 00:06:32.840 --> 00:06:35.040 So, first thing is the easiest one. s89 00:06:35.040 --> 00:06:37.880 You need to find the task to delegate. s90 00:06:37.880 --> 00:06:41.440 The the tasks that are coming in the conveyor belt are delegatable, right? s91 00:06:41.440 --> 00:06:43.600 And that can generate value for users. s92 00:06:43.600 --> 00:06:53.280 Second is, you need ability for users to teach the agents of, you know, uh the supervisor should be able to come in teach how the work needs to be done. s93 00:06:53.280 --> 00:06:59.680 And lastly, uh there's no point of a conveyor belt where you cannot monitor the work yourself, right? s94 00:06:59.680 --> 00:07:04.240 And and of course, intervene when something goes wrong. s95 00:07:04.240 --> 00:07:06.720 Now, let's take delegation. s96 00:07:06.720 --> 00:07:11.360 Um so, delegation means, you know, coming and handing over the task to another person, right? s97 00:07:11.360 --> 00:07:18.200 So, think of your users coming to a platform and delegating a task or handing off task to an AI agent. s98 00:07:18.200 --> 00:07:29.480 So, when building for this particular piece, you as a developer or a product engineer have to find tasks that your users do that take more than a couple of hours. s99 00:07:29.480 --> 00:07:38.200 So, in case of taxes, um there are like three different tasks that we have identified in the tax workflow, which take more than a an hour for our users to perform. s100 00:07:38.200 --> 00:07:42.600 Similarly, in your industry, you can figure out, you know, which are those tasks. s101 00:07:42.600 --> 00:07:51.000 It's important that these tasks are repeatable or sort of repeatable with definitely applicable to each of their, you know, use cases. s102 00:07:51.000 --> 00:07:56.240 And these tasks are what you use to build what you call a long-running background agents. s103 00:07:56.240 --> 00:08:03.520 This is where the core value is, you know, because you're taking off hours of work from your users' hands. s104 00:08:03.520 --> 00:08:04.520 Awesome. s105 00:08:04.520 --> 00:08:15.960 Um so, next is, you know, uh in any industry, in any professional industry, uh like in case of coding, for example, everyone every single user have their own preferences s106 00:08:15.960 --> 00:08:17.880 and have have their own way of doing a task. s107 00:08:17.880 --> 00:08:29.960 An example is, you know, coding, for example, we every single company or every single group of developers have their own ways of dealing with best practices and their own conventions and practices that they follow. s108 00:08:29.960 --> 00:08:38.840 So, if you create a background agent, for example, and it does this end-to-end task of creating a product in case of coding, for example, it will produce output, sure, right? s109 00:08:38.840 --> 00:08:48.800 It produce It will get you most of like 80 to 90% there, but think of it this way, like that will not yet solve the problems of of the user, right? s110 00:08:48.800 --> 00:08:52.120 As a user, you would want it to do it the way that you do the work. s111 00:08:52.120 --> 00:08:54.640 So, this is where skills come into play. s112 00:08:54.640 --> 00:08:58.360 Skills already exist in the today's world of agent development. s113 00:08:58.360 --> 00:09:06.720 So, we need to ensure that your product also has skills in place to capture where you can teach your agents how your users do the work. s114 00:09:06.720 --> 00:09:11.600 This is the last 20% of the of the work that you need to take care of. s115 00:09:11.600 --> 00:09:16.960 This is where the real value is, the quirks of the work, you know, that you're capturing. s116 00:09:17.120 --> 00:09:21.640 In our case, in Taxes, we capture all the all of these skills automatically. s117 00:09:21.640 --> 00:09:26.480 You do not need to have like a complete separate interface where users you go and create skills, that won't work. s118 00:09:26.480 --> 00:09:33.280 In many cases, you need to like look at the product usage and figure out whether you can create a skill for that particular use case or not. s119 00:09:33.280 --> 00:09:38.960 So, we automatically do in our case, and a prime example you would have used a product called this before. s120 00:09:38.960 --> 00:09:41.320 They also have like automatic skills. s121 00:09:41.320 --> 00:09:45.640 So you use and it keeps on learning as you use the product. s122 00:09:46.400 --> 00:09:47.240 Now monitor. s123 00:09:47.240 --> 00:09:49.920 Monitor is monitoring is a key key part right. s124 00:09:49.920 --> 00:10:00.920 So since agents are long running now the whole point is they'll be like multiple tasks that are running and the users need to keep track of what is happening in this world. s125 00:10:00.920 --> 00:10:08.680 So so one simple example here could be a task list that you can do to keep track of where the processing is for each of those tasks. s126 00:10:08.680 --> 00:10:12.840 And second example here is like building traces in your product, you know. s127 00:10:12.840 --> 00:10:15.680 Trace how the agent did the work that it did. s128 00:10:15.680 --> 00:10:19.280 They're long running tasks so there will be multiple pieces that are moving. s129 00:10:19.280 --> 00:10:20.240 So you need to trace back. s130 00:10:20.240 --> 00:10:28.400 So in our case we trace back each and every value that the AI agent produced in a particular, you know, in the format that the users can easily see. s131 00:10:28.400 --> 00:10:29.920 This is where the trust is built. s132 00:10:29.920 --> 00:10:35.840 So it's and this is where the visibility is built so it's it's paramount that you know, you build this correctly. s133 00:10:35.840 --> 00:10:41.280 This is where most of the complaints can be get getting addressed if you build this right. s134 00:10:41.280 --> 00:10:43.960 And lastly, think of it this way, right? s135 00:10:43.960 --> 00:10:44.680 Control. s136 00:10:44.680 --> 00:10:53.720 So when something requires a judgment or when something goes wrong your platform should inspire confidence that the users can take back control. s137 00:10:53.720 --> 00:11:04.080 This is critical because level three product or the conveyable product that you're building will allow you the your customers to, you know, schedule a lot of tasks. s138 00:11:04.080 --> 00:11:13.720 But if they don't have the confidence that, you know, if something goes wrong they can take them back control then then the users will completely lose trust. s139 00:11:13.720 --> 00:11:21.520 So it should feel like, you know, they're taking the users are taking the wheel, not abandoning the car and, you know, creating a new car, for example. s140 00:11:21.520 --> 00:11:26.760 So ideally it should go like you pause the belt, fix the problem, you start it back up again. s141 00:11:26.760 --> 00:11:34.200 In our case we could do it very simply by, you know, pausing whenever the agent was trying to make an assumption, we pause. s142 00:11:34.200 --> 00:11:38.640 And then the users would come in just like in Slack or any other chat platform. s143 00:11:38.640 --> 00:11:44.760 They can come and tag the agent and respond with how to deal with that particular conflict. s144 00:11:46.360 --> 00:11:48.320 Okay, some bonus points I said. s145 00:11:48.320 --> 00:12:03.080 Um so, just as physical bank branches didn't disappear when the mobile banking arrived or you know, when the level two arrived, um I don't think when the agent take layer up to just it's an abstraction over the last two layers s146 00:12:03.080 --> 00:12:07.600 that you know, all other mobile apps and things like that will disappear. s147 00:12:07.600 --> 00:12:15.120 Uh the point here is a user can user will only come and delegate work they believe that you know, they can always take back control. s148 00:12:15.120 --> 00:12:23.360 So, so first step is you know, and having giving that the confidence that they can take back control, you know, when they when something goes wrong. s149 00:12:23.360 --> 00:12:27.280 So, you need to build level two features into your product as well. s150 00:12:27.280 --> 00:12:29.960 That's the key point here to build trust. s151 00:12:29.960 --> 00:12:35.839 And second thing is some actions are usually irreversible and are dangerous actions. s152 00:12:35.839 --> 00:12:38.360 And in those cases, you need to present a plan. s153 00:12:38.360 --> 00:12:47.960 So, think of it like in case of bank [clears throat] transactions, for example, before making a bank transaction, the plan should be there, you know, if the certain amount of value is going somewhere else. s154 00:12:47.960 --> 00:12:58.920 Same way in our cases, when we before we do data entry into your tax software, which can erase their existing data, we create a plan for them so that they can approve the plan before they move on. s155 00:12:58.920 --> 00:13:11.400 These are the key pillars that allows customers to feel you know, control and allows customers to feel as if you know, if something goes wrong, they can take back control at any point in time. s156 00:13:11.400 --> 00:13:21.760 And lastly, since we're building products now completely differently, like we're not building for the users to come in and use the product, we're building for users to come and delegate work. s157 00:13:21.760 --> 00:13:25.480 It's important that we change the way we measure this value. s158 00:13:25.480 --> 00:13:32.280 So, the most popular metric that customers that the the products use today are weekly active users. s159 00:13:32.280 --> 00:13:38.760 Now, this is great in the in the era where, you know, people used to come and do the work in the platform themselves. s160 00:13:38.760 --> 00:13:42.160 But, it doesn't really translate well with agent and delegation. s161 00:13:42.160 --> 00:13:45.360 So, the point is we we have to change the way we measure. s162 00:13:45.360 --> 00:13:50.280 And the I think the the appropriate measure here is weekly active sessions. s163 00:13:50.280 --> 00:13:56.240 It's I treat it as, you know, a task that is completed by a human or an agent even when the user is not in your platform. s164 00:13:56.240 --> 00:14:03.440 So, uh your actual aim should be the weekly active users go down while weekly active sessions go up, right? s165 00:14:03.440 --> 00:14:09.400 Because you want your users to have enough trust in your platform that they can come and delegate your task as much as possible. s166 00:14:09.400 --> 00:14:13.280 And those tasks are executed without a human help as much as possible. s167 00:14:13.280 --> 00:14:15.320 So, the number of sessions should go up. s168 00:14:15.320 --> 00:14:20.520 Weekly active sessions should go up while number of weekly active users in your platform should ideally go down. s169 00:14:20.520 --> 00:14:24.080 It should not be zero, of course, but should go down. s170 00:14:25.480 --> 00:14:26.880 So, what are the key takeaways? s171 00:14:26.880 --> 00:14:31.160 The key takeaways are first of all, design for delegation, not participation. s172 00:14:31.160 --> 00:14:37.960 Design your products in a way that users are coming in to your product to delegate work, not do it themselves. s173 00:14:37.960 --> 00:14:42.120 Secondly, to do to make that possible, you need to think of your product as a conveyor belt. s174 00:14:42.120 --> 00:14:45.200 So, users are now the supervisors, they're not operators. s175 00:14:45.200 --> 00:14:51.360 They're coming here to delegate work and monitor, you know, figure out if there are problems and then take actions. s176 00:14:51.360 --> 00:14:56.920 And finally, you need to change the way you measure how the work is done, uh not the time on the platform. s177 00:14:56.920 --> 00:15:00.600 Uh so, weekly active sessions instead of weekly active users. s178 00:15:00.600 --> 00:15:07.600 If you keep these three in mind, you will be able to build a very successful uh vertically air product, I think. s179 00:15:07.600 --> 00:15:08.000 All right. s180 00:15:08.000 --> 00:15:08.920 Uh thank you for your time. s181 00:15:08.920 --> 00:15:12.320 Um and good luck out there.