Building GTM AI Agents: Lessons from Deploying to 6,000 Users — Sait Izmit, Snowflake

AI Engineer · 20 min · 327 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13So, I think I'm one of the last speakers that is standing between you and the long weekend.
00:18So, I hope I can get your energy levels up.
00:21Um so, I'm responsible for our internal AI tools for our sales team.
00:25And the reason I'm here today is indeed like we launched our internal go-to-market assistant uh in September last year.
00:32It answered more than 1 million questions so far.
00:35We have roughly answered 40,000 questions a week.
00:38Um and we are the customer zero for a lot of Snowflake products.
00:42So, this is built on Snowflake co-work.
00:45Um and I meet a lot of customers every week.
00:48Okay?
00:48So, I meet a lot of enterprises, Fortune 500 companies, and then they're all trying to build similar things, and they all struggle, right?
00:55So, and then I end up like having this discussion with them all the time.
00:58Like they ask like how did you guys do it?
01:00And then we share our best practices.
01:02So, I will try to share some of those things with you.
01:04Uh I'm told that I need to have some code in my presentation.
01:07I don't, but I will try to show you at least some architectural diagrams just to make it more interesting for the engineering audience.
01:13Uh but let's jump into it.
01:15Um I think before we start like I think I already I was watching the other presentations like I think everyone tries to give their interpretation of like you know, why are we even building things for go-to-market.
01:25Okay?
01:25So, this is how I explain it to family and friends.
01:28So, let's take Snowflake.
01:30Okay?
01:30So, we are a company of like, you know, close to 10,000 people.
01:34So, if you look at that or our organization, almost half of our basically workforce is sales, right?
01:39And what are they responsible for?
01:41They're responsible for revenue generation.
01:43What do they struggle with?
01:45And I into I talked to a lot of customers.
01:47It's very common.
01:48You know, everyone's data is siloed.
01:50We work with a lot of first-party data, a lot of third-party data.
01:53It's all locked down in these like SaaS tools and things like that.
01:56And literally we have for example like reps who are using 15 different tools, not because they love the UI of those tools, because every tool has a different data point, and then they end up stitching all of that together in spreadsheets and running it there, right?
02:08And the data is endless.
02:10Like we have reps who have 1,000 accounts assigned to them, 1,000 customers.
02:15They have to stay on top of their recent news, what's happening with their consumption, did they get in support tickets recently, what was their latest earning results, everything.
02:23There's no It's not a single human on this planet that can stay on top of that much data.
02:27And then they need to do that 30 times, 40 times a day, right?
02:31So, what does AI offer for them?
02:33It offers that data democratization.
02:35No more like 1,000 dashboards, right?
02:37No more access to analysts.
02:38Like you know, um it offers automation possibilities for them, right?
02:42It frees up their inbox.
02:44It offers tool consolidation.
02:46No longer 15 different tools that I need to work for.
02:50And that brings productivity savings.
02:52It frees up your time, right?
02:53You can use that time on other things.
02:55It helps you become a better seller.
02:56You're more effective with your customers.
02:58And that translates to business results.
03:00You can cover more of your book, you know, you can have better win rates, uh you can have shorter deal cycles, and ultimately what everyone cares about, you can get incremental revenue.
03:09Okay, so that's the reason why I'm I'm working for, you know, making the go-to-market organizations more effective.
03:16But, there's a catch.
03:17These are non-deterministic systems, right?
03:21And I run into this problem every time with users.
03:25I see many, many, many AI projects failed, and then it fails on this principle.
03:31User trust is earned extremely hard and is lost overnight, right?
03:38So, at the end what you're doing is you're putting a free-form chatbot there, right?
03:43And people will come in and they will ask any question they can think of.
03:47If they like what they see in the first five questions, they come back.
03:52If they don't like what they see, it's 10 times more effort for you to win them back, if you can ever win them back.
03:57Right?
03:58So, we have a saying in our team, we say quality is P minus one.
04:04And that's basically we take that very, very seriously.
04:08So, one of the things that we really cared about is when I first joined the team, you know, the team had all these like data sources connected from our top dashboards.
04:16We they had a knowledge assistant built into it and so on.
04:19We had three lines of agent instructions.
04:22And then before I even tried the agent, I opened a spreadsheet, I took the sales process, I wrote down 150 questions.
04:28And then the sales our engineering team was like, "What are you doing?
04:31We don't have that data in the agent."
04:33I was like, "It doesn't matter.
04:34These are the questions your sellers are going to ask."
04:36Right?
04:37And then we run our test, 50% accuracy, you know, like everyone's depressed and so on.
04:42So, we said, "Okay, let's make sure that we don't go for coverage, but we go for quality."
04:46Right?
04:47We don't want to try to answer 100 questions and get them 70% right.
04:50We want to answer 50 questions, but get them 95% right.
04:54Right?
04:54Because with that you get a first impression, good first impression, you build a trust with them.
04:58And then rather than being in that boat of, "Oh, this thing doesn't work."
05:02people are like, "Oh, this thing is awesome.
05:03Can I get more of that?"
05:05Right?
05:06So, we started small and 60% of the data we actually added after the launch, after the 6-7 months post launch.
05:14Today, if you look into our agent, I mean, it's not a small agent.
05:17We have 15 semantic views, 85 tables, 3,000 columns of data.
05:22We have like five to six different MCP connections on it.
05:25You know, close to 20 skills connected to that and so on and so on.
05:28Right?
05:29So, it's a huge system that we are managing in here.
05:33And then you cannot just launch these things to everyone, right?
05:35So, we said that, "Look, we need to do this in a controlled way because we want to make sure that we earn that first five questions.
05:41We don't want to burn our bridges in that first five questions."
05:44Right?
05:45So, that's why with every product we do, we do a face launch.
05:49The first one is a pilot.
05:50The goal of the pilot is to prove the accuracy, prove the quality, right?
05:54You get your top, you know, AI native folks in the organization who are eager to work with you, give you feedback, improve the product, make sure that you got the rough edges through that, right?
06:07And then after a couple of weeks, you come to a point where it looks like, okay, those rough edges are more smoother now.
06:12Okay?
06:13Then you go into your better launch.
06:14We do 10% better, right?
06:16With 600 people.
06:17There you are looking at do I truly have an basically a minimum viable product?
06:22Is the MVP really there, right?
06:24And what will happen is that you will start getting tons of requests.
06:27Can you connect this data?
06:28Can you connect that data and everything?
06:30And then you are looking at like where are the actually the concentrations happening?
06:34Because that means that if you don't get those things in, you don't truly have an MVP, right?
06:38Then it's not going to work for their daily workflows.
06:41And then at this stage, you're also trying to prove are they coming back?
06:45Right?
06:46So, the things that we really track there is basically like, okay, how many questions they're asking and everything, but what is the retention rate?
06:53So, we exited for example that at like more than 70% retention rate that the weekly active users were coming back.
06:59Okay, now we're in a good place, right?
07:01We have confidence on the accuracy, we have on the confidence of the basically the coverage of the product we have, and people are coming back.
07:07Okay, now let's go to GA, and then you launch through the GA.
07:11And then you have your next problem.
07:13So, I know that this is a technical conference, but this is also where a lot of these products fail.
07:18It's basically how do you drive change management?
07:20So, you launch your product, you are 2 weeks into the launch, and then you are here.
07:25And all your management is like disappointed or frustrated.
07:29Why aren't people using this?
07:30Why are numbers are real low?
07:32Right?
07:33And I show them this graph.
07:35I say that only 20% of your basically organization actually tried the product.
07:40I cannot do anything.
07:41This is not the product's fault if people are not even taking 5 minutes to try try the product.
07:46Right?
07:47If they try it and if they don't come back, okay, that's my problem.
07:50Right?
07:51But if they don't try it, then we have another problem.
07:53So, the first and I've been, you know, I've seen this with many many sales organization in my past life as well and so on.
07:59Usually this is a couple of month process.
08:01And then you significantly invest in basically change management, in activation.
08:06I will spend 60 70% of my time in sales meetings, giving demos, building dashboards, which teams adopted, you know, shaming the like the managers whose team is actually doing good, getting sponsorship from sales leaders to basically like make sure that they you know, they push their people to try these things and so on.
08:22And then ultimately that gets your blue line up and then your questions are start coming up and then your focus can shift into, okay, how do I drive more depth?
08:31Right?
08:32And I want to really really emphasize this because if you hadn't done this, we would probably be doing, you know, half of where we are today.
08:39So, this is a very very important part.
08:41And then as engineers, if you spend all your effort, you want to have a good product, make sure that the activation and the change management is like lined up, like post launch of the product as well.
08:52Now, you run into another issue.
08:54Okay, you are let's say that four to six months down the road.
08:58Right?
08:59What happens is you successfully launched the product.
09:01You are first like rockstars in the company.
09:04Right?
09:04People literally show you on the corridor like, "Hey, your product is awesome.
09:07We can talk to our data now.
09:09We don't need to wait on the queue to like, you know, get access to like analysts to answer our questions in every 2 weeks and so on.
09:15Right?"
09:16And after a couple of months, they start coming back to you with frustrations.
09:18Say, "I cannot do this in the product anymore.
09:21Right?
09:21I I would like to I mean, I saw this other AI product that does this and so on."
09:25This is what I call the collapsing of the wow factor.
09:28Okay?
09:29So, initially you are cool, but then and that becomes a habit, right?
09:33You basically change their habit and it becomes standard for them.
09:36Now, you need to raise the bar again.
09:38So, the journey that we usually see with the sales teams is like you start with talk to your data.
09:43How do we get you out of those like, you know, hundreds of dashboards situation, dependency to the analyst, and then first we'll basically like democratize the data for you so that you can basically talk to your data.
09:54Then the next wave comes with all the MCP connections, right?
09:57All the integrations that you are building.
09:59Now it becomes like automate my workflows.
10:02We literally have now sellers who are going to use our agent basically to monitor their inbox, they monitor their Slack channels, you know, keep track of all the customer questions coming about like product questions,
10:12uh use the agent to draft responses that save that in Gmail, review them afterwards like send those things out, right?
10:18Or they automate their like outreach workflows and so on.
10:21Okay, that's great.
10:22Now I became an orchestrator, right?
10:24I'm basically automating my workflow workflows.
10:27Then the next thing you see start happening is teams, they get these like, you know, tool democratization, this empowerment coming to them, right?
10:35Because historically a lot of these go-to-market teams, they have been always in the backlog of someone, backlog of of an IT team or like trying to get a SaaS budget to learn and get a vendor on board to actually like enable something.
10:47And now all of a sudden they're able to build team skills.
10:50They're able to build like, you know, the custom dashboards that are basically like fully, you know, optimized for what their team needs.
10:56Are able to like deploy applications, automations, alerts, and things like that, right?
11:02And then the comes the phase of hyper-personalization, right?
11:05Everyone is able to now like get everything personalized for them, not only for themselves, but also for their customers with living context of customers, contacts, and things like that.
11:14I think the main message I want to give here is if you just do the first stage, and if you just wait there, you will get disrupted in a month or two,
11:24right?
11:25Because now you already raised their expectations, that already became a baseline, and then they will find another product that does better than you, and right now the switch is very easy.
11:33They're going to just switch over night.
11:34Okay?
11:35So, you need to keep iterating.
11:37You need to keep that wow factor, and I cannot just rely on the fact that, you know, what I built so far is going to stay cool forever.
11:45And the next thing is, how do you deal with basically the changing technology?
11:49So, I talked to a lot of customers.
11:52And then, you know, it sometimes you run into these customers, big enterprises, very big brands.
11:56And then they are still trying to purchase that perfect architecture.
12:00They're trying to like test different frameworks.
12:02They're trying to see how the, you know, the technology is maturing and everything and so on.
12:06But, the thing that they don't do is they don't build, and then they don't launch, and they don't learn.
12:11Right?
12:12All these blue boxes that you see here, those are all the things we added after the launch.
12:17Right?
12:17When we literally first launched the agent, it was a nine-page long agent instructions.
12:23It was couple of Cortex analyst tools, semantic views.
12:25It was a Cortex search service for our unstructured data.
12:28And we were managing the agent instructions versions out of a Google Doc.
12:32That's how we launched it.
12:33To 6,000 people.
12:35Right?
12:35Now we realized, okay, it's not going to work out.
12:37Let's figure out CICD.
12:38It's not going to work out.
12:39Let's figure out our basically eval infrastructure with all the like the unit test, routing test, and everything.
12:44Right?
12:45Then we start basically like coming to a point where, for example, we were creating all these like business processes and workflows.
12:51We couldn't fit them into the agent instructions anymore.
12:53And then the skills came, and we were like, "Oh, perfect.
12:55Let's build a skill library."
12:57You know, then the MCPs came.
12:59Perfect.
13:00But now, like we have to put bunch of other instructions to basically orchestrate that, we hit the limits on the agent instructions.
13:05What do we do?
13:06Okay, let's do the progressive disclosures.
13:08Right?
13:09And then user memory comes, task scheduling comes.
13:12We want to go beyond the chat screen and then, you know, chat interface and start doing the Slack interface and things like that.
13:17If I look at the PRD and the architectural diagram we wrote in the beginning of the project, if I compare to this architecture we have now, 80% of it It match.
13:26Okay?
13:27So, like if you look at our sprints, like maybe 60-70% of the work we are doing is adding new features, improving quality, and all kind of things.
13:36But 30-40% of the work is that we are constantly re-architecting with the new technology.
13:41So, this is a time where like you need to get your hands dirty, you need to run with the new technology, and then you shouldn't be like, you know, too much tied to your architecture.
13:48You should be okay to like pivot very easily, so that you can basically double on down on these like new capabilities and things like that.
13:55And then the longer you wait, the more, you know, you lose towards your competition, because if your competition is doing these kind of things like 3-4 months ahead of you,
14:03right?
14:04That means that they're also getting more customers.
14:08Um last thing is I would really, really recommend investing in your logs.
14:13Okay?
14:14Because they create the basically the feedback loop.
14:17So, first of all, technically it's very fun.
14:19Okay?
14:20So, you basically use LLMs to like classify your logs and things like that.
14:24As I said, like we have 1.2 million questions, we get 40,000 questions every week.
14:28It's technically very fun, you know, how you do that at scale without breaking the bank and so on.
14:32You know, our data scientists love working on those things, and then they really experiment with new things.
14:37But as a result of that, what we get is we get a very extremely detailed breakdown of topics and, you know, things that we are having.
14:43I'm just to showing you the top category categorization level there, but then basically we are able to track like, you know, what kind of questions they are asking.
14:51We are able to break down each of those categories to subcategories, you know, they are able to get like detailed example questions, this and that, and so on.
14:59All good, but how do we use that?
15:01Then we start creating the basically the feedback loops.
15:03Right?
15:04I know, I mean, I still interview, of course, users, but now I see in real time what my feature gaps are.
15:09I'm clearly seeing what people are asking and we are not able to answer or where we have a like a quality issue, where they are swearing at the agent or like at repeating their question,
15:17so that we see where to improve.
15:20For sales enablement is a goldmine.
15:22Let's say that we launch a new product.
15:25Usually, you know, they would need to interview maybe 100 sellers a week to be able to understand like how basically the you know, the topics are changing where there's gaps in terms of like knowledge documents, battle cards.
15:35I see that in real time in a minute or two by just asking an element question.
15:39And then we can then, you know, connect to Confluence, we can connect to Jira, we can connect to Slack channels, we can ingest the PRDs, and in couple of minutes we can actually like, you know, generate battle cards, sales enablement document and then feed it back into the agent.
15:51Right?
15:51I mean, you cannot do that kind of a like a feedback loop with humans, right?
15:55So then you we can basically automate these kind of things.
15:58Um within the sales organization, there will be different teams that are good trying to connect each other.
16:02They are trying to maybe target similar accounts from different angles.
16:06They don't know about each other.
16:07We do.
16:08We are now able to ping them.
16:09And then we are able to basically do matchmaking.
16:11Right?
16:12I'm just giving you couple of examples, but this is also one of those areas where like you start building your AI platform, you start building your architecture.
16:20The first features are difficult to get out.
16:23The next ones are easy.
16:24And then once you start tapping into your logs, this this like hockey stick exponential thing actually starts happening and it's magical.
16:34So, if you were to take a couple of things from this talk, like quality over coverage.
16:40I'm very, very like religious about this.
16:43If you go for the coverage, you are going to shoot yourself in the foot.
16:47Okay?
16:49Change management.
16:50A lot of engineers doesn't think about this, right?
16:53A lot of these AI initiatives, they don't fail because there's an issue with the technology, there's an issue with that as assuming you did the first one, right?
17:00Right?
17:01So, they fail actually in activation.
17:04So, make sure that you have a plan for that, especially in larger organizations where we are dealing with like 6,000 go-to-market users, right?
17:12Um again, don't forget this concept of collapsing law factor.
17:16You cannot stay where you are.
17:18You cannot just say that, "Hey, I did an innovation.
17:20I'm going to surf that for a year."
17:23You know, every time people are happy, you should be paranoid.
17:25You should be like, "Okay, what am I going to show them in a month or two now?"
17:29How do I basically keep that excitement going on?
17:32Right?
17:33Build fast with today's stack.
17:34Like, don't try to invest in these like high, you know, super like plat, you know, architectures and have these like 6 9 months of long projects and things like that.
17:43How do you turn around these things in weeks, days, and so on?
17:47And just be comfortable with the fact that you are constantly going to be re-architecting.
17:50That's fine.
17:51Right?
17:53Just don't over-invest in the current architecture.
17:55Just make sure that you keep your like flexibility out there.
17:58Um and then the feedback loops.
18:00I think that's what kind of like gives you really that like, you know, the incremental part of like that hockey stick exponential part of the thing.
18:06Um we constantly publish like blog posts, I mean, where we kind of like try to have our, you know, learnings shared with uh our customers and so on.
18:14Like, we have blog posts on like how we do agent instructions, how we do our structured data with semantic views, you know, how we basically build our rack-based like knowledge assistants,
18:22the non-technical side of the story, like how do you derive change management, and so on.
18:26So, feel free to check those.
18:28Um and yeah, I think that's the end of my talk.
18:33Okay.
18:36We have time for one question.
18:39Okay, there you go.
18:47Thanks for the talk.
18:48Um I don't know if you already said this, but I saw in the the titles of the articles Snowflake Intelligence.
18:54Is that an underlying context or layer that the tool or system you built was on top of, or was that the tool itself or something else?
19:04Yeah, Snowflake Intelligence, we renamed that to Snowflake Co-work a couple of weeks ago in our summit.
19:09That's basically our no-code agent platform that we basically build have available for our business users.
19:16I mean, the advantage of that is that all of these tools are on like, you know, you know, Cortex analyst or Cortex search or Cortex sense, a lot of those things are basically comes out of the box.
19:26We made a strategic choice for our internal thing where we said that, "Look, it is important that we bring all our data together."
19:32And we do that in Snowflake.
19:33We bring all the first-party, the third-party data, all the Salesforce data, everything, the call transcripts, and so on, all together.
19:39And then these agents then can basically basically inherit a lot of the role-based access controls and so on.
19:45And I literally can deploy these agents without writing a single line of code, right?
19:49And then, you know, you don't need to worry about the UI, the chat UI comes out of the box, and so on.
19:54And then we have been the customer zero of that like internally to build this ourselves.
19:58And then, you know, our customers are able to go and then build similar things basically on Snowflake over platform as well.
20:04And it comes with the guardrails and things where you don't really need to worry about them going very, you know, crazy on, you know, what data sources to do things, and so on.
20:14So we are able to do a lot of curation.
20:16We are able to do a lot of security guardrails in there as well.
20:21Thank you.