Building GTM AI Agents: Lessons from Deploying to 6,000 Users — Sait Izmit, Snowflake
https://www.youtube.com/watch?v=DrTdD-ttjCY
AI Engineer · 20 min · 327 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Hi everyone.
[00:13] So, I think I'm one of the last speakers that is standing between you and the long weekend.
[00:18] So, I hope I can get your energy levels up.
[00:21] Um so, I'm responsible for our internal AI tools for our sales team.
[00:25] And the reason I'm here today is indeed like we launched our internal go-to-market assistant uh in September last year.
[00:32] It answered more than 1 million questions so far.
[00:35] We have roughly answered 40,000 questions a week.
[00:38] Um and we are the customer zero for a lot of Snowflake products.
[00:42] So, this is built on Snowflake co-work.
[00:45] Um and I meet a lot of customers every week.
[00:48] Okay?
[00:48] So, I meet a lot of enterprises, Fortune 500 companies, and then they're all trying to build similar things, and they all struggle, right?
[00:55] So, and then I end up like having this discussion with them all the time.
[00:58] Like they ask like how did you guys do it?
[01:00] And then we share our best practices.
[01:02] So, I will try to share some of those things with you.
[01:04] Uh I'm told that I need to have some code in my presentation.
[01:07] I don't, but I will try to show you at least some architectural diagrams just to make it more interesting for the engineering audience.
[01:13] Uh but let's jump into it.
[01:15] Um I think before we start like I think I already I was watching the other presentations like I think everyone tries to give their interpretation of like you know, why are we even building things for go-to-market.
[01:25] Okay?
[01:25] So, this is how I explain it to family and friends.
[01:28] So, let's take Snowflake.
[01:30] Okay?
[01:30] So, we are a company of like, you know, close to 10,000 people.
[01:34] So, if you look at that or our organization, almost half of our basically workforce is sales, right?
[01:39] And what are they responsible for?
[01:41] They're responsible for revenue generation.
[01:43] What do they struggle with?
[01:45] And I into I talked to a lot of customers.
[01:47] It's very common.
[01:48] You know, everyone's data is siloed.
[01:50] We work with a lot of first-party data, a lot of third-party data.
[01:53] It's all locked down in these like SaaS tools and things like that.
[01:56] And literally we have for example like reps who are using 15 different tools, not because they love the UI of those tools, because every tool has a different data point, and then they end up stitching all of that together in spreadsheets and running it there, right?
[02:08] And the data is endless.
[02:10] Like we have reps who have 1,000 accounts assigned to them, 1,000 customers.
[02:15] They have to stay on top of their recent news, what's happening with their consumption, did they get in support tickets recently, what was their latest earning results, everything.
[02:23] There's no It's not a single human on this planet that can stay on top of that much data.
[02:27] And then they need to do that 30 times, 40 times a day, right?
[02:31] So, what does AI offer for them?
[02:33] It offers that data democratization.
[02:35] No more like 1,000 dashboards, right?
[02:37] No more access to analysts.
[02:38] Like you know, um it offers automation possibilities for them, right?
[02:42] It frees up their inbox.
[02:44] It offers tool consolidation.
[02:46] No longer 15 different tools that I need to work for.
[02:50] And that brings productivity savings.
[02:52] It frees up your time, right?
[02:53] You can use that time on other things.
[02:55] It helps you become a better seller.
[02:56] You're more effective with your customers.
[02:58] And that translates to business results.
[03:00] You can cover more of your book, you know, you can have better win rates, uh you can have shorter deal cycles, and ultimately what everyone cares about, you can get incremental revenue.
[03:09] Okay, so that's the reason why I'm I'm working for, you know, making the go-to-market organizations more effective.
[03:16] But, there's a catch.
[03:17] These are non-deterministic systems, right?
[03:21] And I run into this problem every time with users.
[03:25] I see many, many, many AI projects failed, and then it fails on this principle.
[03:31] User trust is earned extremely hard and is lost overnight, right?
[03:38] So, at the end what you're doing is you're putting a free-form chatbot there, right?
[03:43] And people will come in and they will ask any question they can think of.
[03:47] If they like what they see in the first five questions, they come back.
[03:52] If they don't like what they see, it's 10 times more effort for you to win them back, if you can ever win them back.
[03:57] Right?
[03:58] So, we have a saying in our team, we say quality is P minus one.
[04:04] And that's basically we take that very, very seriously.
[04:08] So, one of the things that we really cared about is when I first joined the team, you know, the team had all these like data sources connected from our top dashboards.
[04:16] We they had a knowledge assistant built into it and so on.
[04:19] We had three lines of agent instructions.
[04:22] And then before I even tried the agent, I opened a spreadsheet, I took the sales process, I wrote down 150 questions.
[04:28] And then the sales our engineering team was like, "What are you doing?
[04:31] We don't have that data in the agent."
[04:33] I was like, "It doesn't matter.
[04:34] These are the questions your sellers are going to ask."
[04:36] Right?
[04:37] And then we run our test, 50% accuracy, you know, like everyone's depressed and so on.
[04:42] So, we said, "Okay, let's make sure that we don't go for coverage, but we go for quality."
[04:46] Right?
[04:47] We don't want to try to answer 100 questions and get them 70% right.
[04:50] We want to answer 50 questions, but get them 95% right.
[04:54] Right?
[04:54] Because with that you get a first impression, good first impression, you build a trust with them.
[04:58] And then rather than being in that boat of, "Oh, this thing doesn't work."
[05:02] people are like, "Oh, this thing is awesome.
[05:03] Can I get more of that?"
[05:05] Right?
[05:06] So, we started small and 60% of the data we actually added after the launch, after the 6-7 months post launch.
[05:14] Today, if you look into our agent, I mean, it's not a small agent.
[05:17] We have 15 semantic views, 85 tables, 3,000 columns of data.
[05:22] We have like five to six different MCP connections on it.
[05:25] You know, close to 20 skills connected to that and so on and so on.
[05:28] Right?
[05:29] So, it's a huge system that we are managing in here.
[05:33] And then you cannot just launch these things to everyone, right?
[05:35] So, we said that, "Look, we need to do this in a controlled way because we want to make sure that we earn that first five questions.
[05:41] We don't want to burn our bridges in that first five questions."
[05:44] Right?
[05:45] So, that's why with every product we do, we do a face launch.
[05:49] The first one is a pilot.
[05:50] The goal of the pilot is to prove the accuracy, prove the quality, right?
[05:54] You get your top, you know, AI native folks in the organization who are eager to work with you, give you feedback, improve the product, make sure that you got the rough edges through that, right?
[06:07] And then after a couple of weeks, you come to a point where it looks like, okay, those rough edges are more smoother now.
[06:12] Okay?
[06:13] Then you go into your better launch.
[06:14] We do 10% better, right?
[06:16] With 600 people.
[06:17] There you are looking at do I truly have an basically a minimum viable product?
[06:22] Is the MVP really there, right?
[06:24] And what will happen is that you will start getting tons of requests.
[06:27] Can you connect this data?
[06:28] Can you connect that data and everything?
[06:30] And then you are looking at like where are the actually the concentrations happening?
[06:34] Because that means that if you don't get those things in, you don't truly have an MVP, right?
[06:38] Then it's not going to work for their daily workflows.
[06:41] And then at this stage, you're also trying to prove are they coming back?
[06:45] Right?
[06:46] So, the things that we really track there is basically like, okay, how many questions they're asking and everything, but what is the retention rate?
[06:53] So, we exited for example that at like more than 70% retention rate that the weekly active users were coming back.
[06:59] Okay, now we're in a good place, right?
[07:01] We have confidence on the accuracy, we have on the confidence of the basically the coverage of the product we have, and people are coming back.
[07:07] Okay, now let's go to GA, and then you launch through the GA.
[07:11] And then you have your next problem.
[07:13] So, I know that this is a technical conference, but this is also where a lot of these products fail.
[07:18] It's basically how do you drive change management?
[07:20] So, you launch your product, you are 2 weeks into the launch, and then you are here.
[07:25] And all your management is like disappointed or frustrated.
[07:29] Why aren't people using this?
[07:30] Why are numbers are real low?
[07:32] Right?
[07:33] And I show them this graph.
[07:35] I say that only 20% of your basically organization actually tried the product.
[07:40] I cannot do anything.
[07:41] This is not the product's fault if people are not even taking 5 minutes to try try the product.
[07:46] Right?
[07:47] If they try it and if they don't come back, okay, that's my problem.
[07:50] Right?
[07:51] But if they don't try it, then we have another problem.
[07:53] So, the first and I've been, you know, I've seen this with many many sales organization in my past life as well and so on.
[07:59] Usually this is a couple of month process.
[08:01] And then you significantly invest in basically change management, in activation.
[08:06] I will spend 60 70% of my time in sales meetings, giving demos, building dashboards, which teams adopted, you know, shaming the like the managers whose team is actually doing good, getting sponsorship from sales leaders to basically like make sure that they you know, they push their people to try these things and so on.
[08:22] And then ultimately that gets your blue line up and then your questions are start coming up and then your focus can shift into, okay, how do I drive more depth?
[08:31] Right?
[08:32] And I want to really really emphasize this because if you hadn't done this, we would probably be doing, you know, half of where we are today.
[08:39] So, this is a very very important part.
[08:41] And then as engineers, if you spend all your effort, you want to have a good product, make sure that the activation and the change management is like lined up, like post launch of the product as well.
[08:52] Now, you run into another issue.
[08:54] Okay, you are let's say that four to six months down the road.
[08:58] Right?
[08:59] What happens is you successfully launched the product.
[09:01] You are first like rockstars in the company.
[09:04] Right?
[09:04] People literally show you on the corridor like, "Hey, your product is awesome.
[09:07] We can talk to our data now.
[09:09] We don't need to wait on the queue to like, you know, get access to like analysts to answer our questions in every 2 weeks and so on.
[09:15] Right?"
[09:16] And after a couple of months, they start coming back to you with frustrations.
[09:18] Say, "I cannot do this in the product anymore.
[09:21] Right?
[09:21] I I would like to I mean, I saw this other AI product that does this and so on."
[09:25] This is what I call the collapsing of the wow factor.
[09:28] Okay?
[09:29] So, initially you are cool, but then and that becomes a habit, right?
[09:33] You basically change their habit and it becomes standard for them.
[09:36] Now, you need to raise the bar again.
[09:38] So, the journey that we usually see with the sales teams is like you start with talk to your data.
[09:43] How do we get you out of those like, you know, hundreds of dashboards situation, dependency to the analyst, and then first we'll basically like democratize the data for you so that you can basically talk to your data.
[09:54] Then the next wave comes with all the MCP connections, right?
[09:57] All the integrations that you are building.
[09:59] Now it becomes like automate my workflows.
[10:02] We literally have now sellers who are going to use our agent basically to monitor their inbox, they monitor their Slack channels, you know, keep track of all the customer questions coming about like product questions,
[10:12] uh use the agent to draft responses that save that in Gmail, review them afterwards like send those things out, right?
[10:18] Or they automate their like outreach workflows and so on.
[10:21] Okay, that's great.
[10:22] Now I became an orchestrator, right?
[10:24] I'm basically automating my workflow workflows.
[10:27] Then the next thing you see start happening is teams, they get these like, you know, tool democratization, this empowerment coming to them, right?
[10:35] Because historically a lot of these go-to-market teams, they have been always in the backlog of someone, backlog of of an IT team or like trying to get a SaaS budget to learn and get a vendor on board to actually like enable something.
[10:47] And now all of a sudden they're able to build team skills.
[10:50] They're able to build like, you know, the custom dashboards that are basically like fully, you know, optimized for what their team needs.
[10:56] Are able to like deploy applications, automations, alerts, and things like that, right?
[11:02] And then the comes the phase of hyper-personalization, right?
[11:05] Everyone is able to now like get everything personalized for them, not only for themselves, but also for their customers with living context of customers, contacts, and things like that.
[11:14] I think the main message I want to give here is if you just do the first stage, and if you just wait there, you will get disrupted in a month or two,
[11:24] right?
[11:25] Because now you already raised their expectations, that already became a baseline, and then they will find another product that does better than you, and right now the switch is very easy.
[11:33] They're going to just switch over night.
[11:34] Okay?
[11:35] So, you need to keep iterating.
[11:37] You need to keep that wow factor, and I cannot just rely on the fact that, you know, what I built so far is going to stay cool forever.
[11:45] And the next thing is, how do you deal with basically the changing technology?
[11:49] So, I talked to a lot of customers.
[11:52] And then, you know, it sometimes you run into these customers, big enterprises, very big brands.
[11:56] And then they are still trying to purchase that perfect architecture.
[12:00] They're trying to like test different frameworks.
[12:02] They're trying to see how the, you know, the technology is maturing and everything and so on.
[12:06] But, the thing that they don't do is they don't build, and then they don't launch, and they don't learn.
[12:11] Right?
[12:12] All these blue boxes that you see here, those are all the things we added after the launch.
[12:17] Right?
[12:17] When we literally first launched the agent, it was a nine-page long agent instructions.
[12:23] It was couple of Cortex analyst tools, semantic views.
[12:25] It was a Cortex search service for our unstructured data.
[12:28] And we were managing the agent instructions versions out of a Google Doc.
[12:32] That's how we launched it.
[12:33] To 6,000 people.
[12:35] Right?
[12:35] Now we realized, okay, it's not going to work out.
[12:37] Let's figure out CICD.
[12:38] It's not going to work out.
[12:39] Let's figure out our basically eval infrastructure with all the like the unit test, routing test, and everything.
[12:44] Right?
[12:45] Then we start basically like coming to a point where, for example, we were creating all these like business processes and workflows.
[12:51] We couldn't fit them into the agent instructions anymore.
[12:53] And then the skills came, and we were like, "Oh, perfect.
[12:55] Let's build a skill library."
[12:57] You know, then the MCPs came.
[12:59] Perfect.
[13:00] But now, like we have to put bunch of other instructions to basically orchestrate that, we hit the limits on the agent instructions.
[13:05] What do we do?
[13:06] Okay, let's do the progressive disclosures.
[13:08] Right?
[13:09] And then user memory comes, task scheduling comes.
[13:12] We want to go beyond the chat screen and then, you know, chat interface and start doing the Slack interface and things like that.
[13:17] If I look at the PRD and the architectural diagram we wrote in the beginning of the project, if I compare to this architecture we have now, 80% of it It match.
[13:26] Okay?
[13:27] So, like if you look at our sprints, like maybe 60-70% of the work we are doing is adding new features, improving quality, and all kind of things.
[13:36] But 30-40% of the work is that we are constantly re-architecting with the new technology.
[13:41] So, this is a time where like you need to get your hands dirty, you need to run with the new technology, and then you shouldn't be like, you know, too much tied to your architecture.
[13:48] You should be okay to like pivot very easily, so that you can basically double on down on these like new capabilities and things like that.
[13:55] And then the longer you wait, the more, you know, you lose towards your competition, because if your competition is doing these kind of things like 3-4 months ahead of you,
[14:03] right?
[14:04] That means that they're also getting more customers.
[14:08] Um last thing is I would really, really recommend investing in your logs.
[14:13] Okay?
[14:14] Because they create the basically the feedback loop.
[14:17] So, first of all, technically it's very fun.
[14:19] Okay?
[14:20] So, you basically use LLMs to like classify your logs and things like that.
[14:24] As I said, like we have 1.2 million questions, we get 40,000 questions every week.
[14:28] It's technically very fun, you know, how you do that at scale without breaking the bank and so on.
[14:32] You know, our data scientists love working on those things, and then they really experiment with new things.
[14:37] But as a result of that, what we get is we get a very extremely detailed breakdown of topics and, you know, things that we are having.
[14:43] I'm just to showing you the top category categorization level there, but then basically we are able to track like, you know, what kind of questions they are asking.
[14:51] We are able to break down each of those categories to subcategories, you know, they are able to get like detailed example questions, this and that, and so on.
[14:59] All good, but how do we use that?
[15:01] Then we start creating the basically the feedback loops.
[15:03] Right?
[15:04] I know, I mean, I still interview, of course, users, but now I see in real time what my feature gaps are.
[15:09] I'm clearly seeing what people are asking and we are not able to answer or where we have a like a quality issue, where they are swearing at the agent or like at repeating their question,
[15:17] so that we see where to improve.
[15:20] For sales enablement is a goldmine.
[15:22] Let's say that we launch a new product.
[15:25] Usually, you know, they would need to interview maybe 100 sellers a week to be able to understand like how basically the you know, the topics are changing where there's gaps in terms of like knowledge documents, battle cards.
[15:35] I see that in real time in a minute or two by just asking an element question.
[15:39] And then we can then, you know, connect to Confluence, we can connect to Jira, we can connect to Slack channels, we can ingest the PRDs, and in couple of minutes we can actually like, you know, generate battle cards, sales enablement document and then feed it back into the agent.
[15:51] Right?
[15:51] I mean, you cannot do that kind of a like a feedback loop with humans, right?
[15:55] So then you we can basically automate these kind of things.
[15:58] Um within the sales organization, there will be different teams that are good trying to connect each other.
[16:02] They are trying to maybe target similar accounts from different angles.
[16:06] They don't know about each other.
[16:07] We do.
[16:08] We are now able to ping them.
[16:09] And then we are able to basically do matchmaking.
[16:11] Right?
[16:12] I'm just giving you couple of examples, but this is also one of those areas where like you start building your AI platform, you start building your architecture.
[16:20] The first features are difficult to get out.
[16:23] The next ones are easy.
[16:24] And then once you start tapping into your logs, this this like hockey stick exponential thing actually starts happening and it's magical.
[16:34] So, if you were to take a couple of things from this talk, like quality over coverage.
[16:40] I'm very, very like religious about this.
[16:43] If you go for the coverage, you are going to shoot yourself in the foot.
[16:47] Okay?
[16:49] Change management.
[16:50] A lot of engineers doesn't think about this, right?
[16:53] A lot of these AI initiatives, they don't fail because there's an issue with the technology, there's an issue with that as assuming you did the first one, right?
[17:00] Right?
[17:01] So, they fail actually in activation.
[17:04] So, make sure that you have a plan for that, especially in larger organizations where we are dealing with like 6,000 go-to-market users, right?
[17:12] Um again, don't forget this concept of collapsing law factor.
[17:16] You cannot stay where you are.
[17:18] You cannot just say that, "Hey, I did an innovation.
[17:20] I'm going to surf that for a year."
[17:23] You know, every time people are happy, you should be paranoid.
[17:25] You should be like, "Okay, what am I going to show them in a month or two now?"
[17:29] How do I basically keep that excitement going on?
[17:32] Right?
[17:33] Build fast with today's stack.
[17:34] Like, don't try to invest in these like high, you know, super like plat, you know, architectures and have these like 6 9 months of long projects and things like that.
[17:43] How do you turn around these things in weeks, days, and so on?
[17:47] And just be comfortable with the fact that you are constantly going to be re-architecting.
[17:50] That's fine.
[17:51] Right?
[17:53] Just don't over-invest in the current architecture.
[17:55] Just make sure that you keep your like flexibility out there.
[17:58] Um and then the feedback loops.
[18:00] I think that's what kind of like gives you really that like, you know, the incremental part of like that hockey stick exponential part of the thing.
[18:06] Um we constantly publish like blog posts, I mean, where we kind of like try to have our, you know, learnings shared with uh our customers and so on.
[18:14] Like, we have blog posts on like how we do agent instructions, how we do our structured data with semantic views, you know, how we basically build our rack-based like knowledge assistants,
[18:22] the non-technical side of the story, like how do you derive change management, and so on.
[18:26] So, feel free to check those.
[18:28] Um and yeah, I think that's the end of my talk.
[18:33] Okay.
[18:36] We have time for one question.
[18:39] Okay, there you go.
[18:47] Thanks for the talk.
[18:48] Um I don't know if you already said this, but I saw in the the titles of the articles Snowflake Intelligence.
[18:54] Is that an underlying context or layer that the tool or system you built was on top of, or was that the tool itself or something else?
[19:04] Yeah, Snowflake Intelligence, we renamed that to Snowflake Co-work a couple of weeks ago in our summit.
[19:09] That's basically our no-code agent platform that we basically build have available for our business users.
[19:16] I mean, the advantage of that is that all of these tools are on like, you know, you know, Cortex analyst or Cortex search or Cortex sense, a lot of those things are basically comes out of the box.
[19:26] We made a strategic choice for our internal thing where we said that, "Look, it is important that we bring all our data together."
[19:32] And we do that in Snowflake.
[19:33] We bring all the first-party, the third-party data, all the Salesforce data, everything, the call transcripts, and so on, all together.
[19:39] And then these agents then can basically basically inherit a lot of the role-based access controls and so on.
[19:45] And I literally can deploy these agents without writing a single line of code, right?
[19:49] And then, you know, you don't need to worry about the UI, the chat UI comes out of the box, and so on.
[19:54] And then we have been the customer zero of that like internally to build this ourselves.
[19:58] And then, you know, our customers are able to go and then build similar things basically on Snowflake over platform as well.
[20:04] And it comes with the guardrails and things where you don't really need to worry about them going very, you know, crazy on, you know, what data sources to do things, and so on.
[20:14] So we are able to do a lot of curation.
[20:16] We are able to do a lot of security guardrails in there as well.
[20:21] Thank you.