WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=DrTdD-ttjCY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/DrTdD-ttjCY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.440
Hi everyone.

s3
00:00:13.440 --> 00:00:18.080
So, I think I'm one of the last speakers that is standing between you and the long weekend.

s4
00:00:18.080 --> 00:00:21.000
So, I hope I can get your energy levels up.

s5
00:00:21.000 --> 00:00:25.440
Um so, I'm responsible for our internal AI tools for our sales team.

s6
00:00:25.440 --> 00:00:32.560
And the reason I'm here today is indeed like we launched our internal go-to-market assistant uh in September last year.

s7
00:00:32.560 --> 00:00:35.400
It answered more than 1 million questions so far.

s8
00:00:35.400 --> 00:00:38.480
We have roughly answered 40,000 questions a week.

s9
00:00:38.480 --> 00:00:42.040
Um and we are the customer zero for a lot of Snowflake products.

s10
00:00:42.040 --> 00:00:45.080
So, this is built on Snowflake co-work.

s11
00:00:45.080 --> 00:00:48.200
Um and I meet a lot of customers every week.

s12
00:00:48.200 --> 00:00:48.480
Okay?

s13
00:00:48.480 --> 00:00:55.040
So, I meet a lot of enterprises, Fortune 500 companies, and then they're all trying to build similar things, and they all struggle, right?

s14
00:00:55.040 --> 00:00:58.160
So, and then I end up like having this discussion with them all the time.

s15
00:00:58.160 --> 00:01:00.080
Like they ask like how did you guys do it?

s16
00:01:00.080 --> 00:01:02.040
And then we share our best practices.

s17
00:01:02.040 --> 00:01:04.600
So, I will try to share some of those things with you.

s18
00:01:04.600 --> 00:01:07.400
Uh I'm told that I need to have some code in my presentation.

s19
00:01:07.400 --> 00:01:13.240
I don't, but I will try to show you at least some architectural diagrams just to make it more interesting for the engineering audience.

s20
00:01:13.240 --> 00:01:15.040
Uh but let's jump into it.

s21
00:01:15.040 --> 00:01:25.480
Um I think before we start like I think I already I was watching the other presentations like I think everyone tries to give their interpretation of like you know, why are we even building things for go-to-market.

s22
00:01:25.480 --> 00:01:25.720
Okay?

s23
00:01:25.720 --> 00:01:28.960
So, this is how I explain it to family and friends.

s24
00:01:28.960 --> 00:01:30.360
So, let's take Snowflake.

s25
00:01:30.360 --> 00:01:30.640
Okay?

s26
00:01:30.640 --> 00:01:34.000
So, we are a company of like, you know, close to 10,000 people.

s27
00:01:34.000 --> 00:01:39.280
So, if you look at that or our organization, almost half of our basically workforce is sales, right?

s28
00:01:39.280 --> 00:01:41.080
And what are they responsible for?

s29
00:01:41.080 --> 00:01:43.800
They're responsible for revenue generation.

s30
00:01:43.800 --> 00:01:45.160
What do they struggle with?

s31
00:01:45.160 --> 00:01:47.400
And I into I talked to a lot of customers.

s32
00:01:47.400 --> 00:01:48.840
It's very common.

s33
00:01:48.840 --> 00:01:50.320
You know, everyone's data is siloed.

s34
00:01:50.320 --> 00:01:53.520
We work with a lot of first-party data, a lot of third-party data.

s35
00:01:53.520 --> 00:01:56.680
It's all locked down in these like SaaS tools and things like that.

s36
00:01:56.680 --> 00:02:08.800
And literally we have for example like reps who are using 15 different tools, not because they love the UI of those tools, because every tool has a different data point, and then they end up stitching all of that together in spreadsheets and running it there, right?

s37
00:02:08.800 --> 00:02:10.280
And the data is endless.

s38
00:02:10.280 --> 00:02:15.160
Like we have reps who have 1,000 accounts assigned to them, 1,000 customers.

s39
00:02:15.160 --> 00:02:23.120
They have to stay on top of their recent news, what's happening with their consumption, did they get in support tickets recently, what was their latest earning results, everything.

s40
00:02:23.120 --> 00:02:27.080
There's no It's not a single human on this planet that can stay on top of that much data.

s41
00:02:27.080 --> 00:02:31.320
And then they need to do that 30 times, 40 times a day, right?

s42
00:02:31.320 --> 00:02:33.920
So, what does AI offer for them?

s43
00:02:33.920 --> 00:02:35.800
It offers that data democratization.

s44
00:02:35.800 --> 00:02:37.560
No more like 1,000 dashboards, right?

s45
00:02:37.560 --> 00:02:38.800
No more access to analysts.

s46
00:02:38.800 --> 00:02:42.920
Like you know, um it offers automation possibilities for them, right?

s47
00:02:42.920 --> 00:02:44.720
It frees up their inbox.

s48
00:02:44.720 --> 00:02:46.280
It offers tool consolidation.

s49
00:02:46.280 --> 00:02:50.080
No longer 15 different tools that I need to work for.

s50
00:02:50.080 --> 00:02:52.080
And that brings productivity savings.

s51
00:02:52.080 --> 00:02:53.520
It frees up your time, right?

s52
00:02:53.520 --> 00:02:55.160
You can use that time on other things.

s53
00:02:55.160 --> 00:02:56.440
It helps you become a better seller.

s54
00:02:56.440 --> 00:02:58.440
You're more effective with your customers.

s55
00:02:58.440 --> 00:03:00.560
And that translates to business results.

s56
00:03:00.560 --> 00:03:09.720
You can cover more of your book, you know, you can have better win rates, uh you can have shorter deal cycles, and ultimately what everyone cares about, you can get incremental revenue.

s57
00:03:09.720 --> 00:03:16.000
Okay, so that's the reason why I'm I'm working for, you know, making the go-to-market organizations more effective.

s58
00:03:16.440 --> 00:03:17.880
But, there's a catch.

s59
00:03:17.880 --> 00:03:21.239
These are non-deterministic systems, right?

s60
00:03:21.239 --> 00:03:25.600
And I run into this problem every time with users.

s61
00:03:25.600 --> 00:03:31.600
I see many, many, many AI projects failed, and then it fails on this principle.

s62
00:03:31.600 --> 00:03:38.080
User trust is earned extremely hard and is lost overnight, right?

s63
00:03:38.080 --> 00:03:43.680
So, at the end what you're doing is you're putting a free-form chatbot there, right?

s64
00:03:43.680 --> 00:03:47.519
And people will come in and they will ask any question they can think of.

s65
00:03:47.519 --> 00:03:52.200
If they like what they see in the first five questions, they come back.

s66
00:03:52.200 --> 00:03:57.880
If they don't like what they see, it's 10 times more effort for you to win them back, if you can ever win them back.

s67
00:03:57.880 --> 00:03:58.960
Right?

s68
00:03:58.960 --> 00:04:04.360
So, we have a saying in our team, we say quality is P minus one.

s69
00:04:04.360 --> 00:04:08.120
And that's basically we take that very, very seriously.

s70
00:04:08.120 --> 00:04:16.920
So, one of the things that we really cared about is when I first joined the team, you know, the team had all these like data sources connected from our top dashboards.

s71
00:04:16.920 --> 00:04:19.799
We they had a knowledge assistant built into it and so on.

s72
00:04:19.799 --> 00:04:22.560
We had three lines of agent instructions.

s73
00:04:22.560 --> 00:04:28.600
And then before I even tried the agent, I opened a spreadsheet, I took the sales process, I wrote down 150 questions.

s74
00:04:28.600 --> 00:04:31.080
And then the sales our engineering team was like, "What are you doing?

s75
00:04:31.080 --> 00:04:33.280
We don't have that data in the agent."

s76
00:04:33.280 --> 00:04:34.280
I was like, "It doesn't matter.

s77
00:04:34.280 --> 00:04:36.760
These are the questions your sellers are going to ask."

s78
00:04:36.760 --> 00:04:37.600
Right?

s79
00:04:37.600 --> 00:04:42.040
And then we run our test, 50% accuracy, you know, like everyone's depressed and so on.

s80
00:04:42.040 --> 00:04:46.520
So, we said, "Okay, let's make sure that we don't go for coverage, but we go for quality."

s81
00:04:46.520 --> 00:04:47.040
Right?

s82
00:04:47.040 --> 00:04:50.800
We don't want to try to answer 100 questions and get them 70% right.

s83
00:04:50.800 --> 00:04:54.040
We want to answer 50 questions, but get them 95% right.

s84
00:04:54.040 --> 00:04:54.640
Right?

s85
00:04:54.640 --> 00:04:58.960
Because with that you get a first impression, good first impression, you build a trust with them.

s86
00:04:58.960 --> 00:05:02.040
And then rather than being in that boat of, "Oh, this thing doesn't work."

s87
00:05:02.040 --> 00:05:03.520
people are like, "Oh, this thing is awesome.

s88
00:05:03.520 --> 00:05:05.160
Can I get more of that?"

s89
00:05:05.160 --> 00:05:06.160
Right?

s90
00:05:06.160 --> 00:05:14.040
So, we started small and 60% of the data we actually added after the launch, after the 6-7 months post launch.

s91
00:05:14.040 --> 00:05:17.600
Today, if you look into our agent, I mean, it's not a small agent.

s92
00:05:17.600 --> 00:05:22.040
We have 15 semantic views, 85 tables, 3,000 columns of data.

s93
00:05:22.040 --> 00:05:25.120
We have like five to six different MCP connections on it.

s94
00:05:25.120 --> 00:05:28.560
You know, close to 20 skills connected to that and so on and so on.

s95
00:05:28.560 --> 00:05:29.160
Right?

s96
00:05:29.160 --> 00:05:33.200
So, it's a huge system that we are managing in here.

s97
00:05:33.240 --> 00:05:35.840
And then you cannot just launch these things to everyone, right?

s98
00:05:35.840 --> 00:05:41.320
So, we said that, "Look, we need to do this in a controlled way because we want to make sure that we earn that first five questions.

s99
00:05:41.320 --> 00:05:44.400
We don't want to burn our bridges in that first five questions."

s100
00:05:44.400 --> 00:05:45.240
Right?

s101
00:05:45.240 --> 00:05:49.040
So, that's why with every product we do, we do a face launch.

s102
00:05:49.040 --> 00:05:50.480
The first one is a pilot.

s103
00:05:50.480 --> 00:05:54.760
The goal of the pilot is to prove the accuracy, prove the quality, right?

s104
00:05:54.760 --> 00:06:07.080
You get your top, you know, AI native folks in the organization who are eager to work with you, give you feedback, improve the product, make sure that you got the rough edges through that, right?

s105
00:06:07.080 --> 00:06:12.760
And then after a couple of weeks, you come to a point where it looks like, okay, those rough edges are more smoother now.

s106
00:06:12.760 --> 00:06:13.200
Okay?

s107
00:06:13.200 --> 00:06:14.720
Then you go into your better launch.

s108
00:06:14.720 --> 00:06:16.320
We do 10% better, right?

s109
00:06:16.320 --> 00:06:17.960
With 600 people.

s110
00:06:17.960 --> 00:06:22.560
There you are looking at do I truly have an basically a minimum viable product?

s111
00:06:22.560 --> 00:06:24.840
Is the MVP really there, right?

s112
00:06:24.840 --> 00:06:27.280
And what will happen is that you will start getting tons of requests.

s113
00:06:27.280 --> 00:06:28.280
Can you connect this data?

s114
00:06:28.280 --> 00:06:30.200
Can you connect that data and everything?

s115
00:06:30.200 --> 00:06:34.160
And then you are looking at like where are the actually the concentrations happening?

s116
00:06:34.160 --> 00:06:38.320
Because that means that if you don't get those things in, you don't truly have an MVP, right?

s117
00:06:38.320 --> 00:06:41.320
Then it's not going to work for their daily workflows.

s118
00:06:41.320 --> 00:06:45.680
And then at this stage, you're also trying to prove are they coming back?

s119
00:06:45.680 --> 00:06:46.280
Right?

s120
00:06:46.280 --> 00:06:53.640
So, the things that we really track there is basically like, okay, how many questions they're asking and everything, but what is the retention rate?

s121
00:06:53.640 --> 00:06:59.440
So, we exited for example that at like more than 70% retention rate that the weekly active users were coming back.

s122
00:06:59.440 --> 00:07:01.160
Okay, now we're in a good place, right?

s123
00:07:01.160 --> 00:07:07.720
We have confidence on the accuracy, we have on the confidence of the basically the coverage of the product we have, and people are coming back.

s124
00:07:07.720 --> 00:07:11.120
Okay, now let's go to GA, and then you launch through the GA.

s125
00:07:11.120 --> 00:07:13.400
And then you have your next problem.

s126
00:07:13.400 --> 00:07:18.480
So, I know that this is a technical conference, but this is also where a lot of these products fail.

s127
00:07:18.480 --> 00:07:20.919
It's basically how do you drive change management?

s128
00:07:20.919 --> 00:07:25.600
So, you launch your product, you are 2 weeks into the launch, and then you are here.

s129
00:07:25.600 --> 00:07:29.520
And all your management is like disappointed or frustrated.

s130
00:07:29.520 --> 00:07:30.720
Why aren't people using this?

s131
00:07:30.720 --> 00:07:32.680
Why are numbers are real low?

s132
00:07:32.680 --> 00:07:33.400
Right?

s133
00:07:33.400 --> 00:07:35.120
And I show them this graph.

s134
00:07:35.120 --> 00:07:40.720
I say that only 20% of your basically organization actually tried the product.

s135
00:07:40.720 --> 00:07:41.919
I cannot do anything.

s136
00:07:41.919 --> 00:07:46.680
This is not the product's fault if people are not even taking 5 minutes to try try the product.

s137
00:07:46.680 --> 00:07:47.320
Right?

s138
00:07:47.320 --> 00:07:50.760
If they try it and if they don't come back, okay, that's my problem.

s139
00:07:50.760 --> 00:07:51.000
Right?

s140
00:07:51.000 --> 00:07:53.800
But if they don't try it, then we have another problem.

s141
00:07:53.800 --> 00:07:59.640
So, the first and I've been, you know, I've seen this with many many sales organization in my past life as well and so on.

s142
00:07:59.640 --> 00:08:01.840
Usually this is a couple of month process.

s143
00:08:01.840 --> 00:08:06.280
And then you significantly invest in basically change management, in activation.

s144
00:08:06.280 --> 00:08:22.880
I will spend 60 70% of my time in sales meetings, giving demos, building dashboards, which teams adopted, you know, shaming the like the managers whose team is actually doing good, getting sponsorship from sales leaders to basically like make sure that they you know, they push their people to try these things and so on.

s145
00:08:22.880 --> 00:08:31.000
And then ultimately that gets your blue line up and then your questions are start coming up and then your focus can shift into, okay, how do I drive more depth?

s146
00:08:31.000 --> 00:08:32.159
Right?

s147
00:08:32.159 --> 00:08:39.760
And I want to really really emphasize this because if you hadn't done this, we would probably be doing, you know, half of where we are today.

s148
00:08:39.760 --> 00:08:41.599
So, this is a very very important part.

s149
00:08:41.599 --> 00:08:52.160
And then as engineers, if you spend all your effort, you want to have a good product, make sure that the activation and the change management is like lined up, like post launch of the product as well.

s150
00:08:52.839 --> 00:08:54.240
Now, you run into another issue.

s151
00:08:54.240 --> 00:08:58.040
Okay, you are let's say that four to six months down the road.

s152
00:08:58.040 --> 00:08:59.080
Right?

s153
00:08:59.080 --> 00:09:01.800
What happens is you successfully launched the product.

s154
00:09:01.800 --> 00:09:04.120
You are first like rockstars in the company.

s155
00:09:04.120 --> 00:09:04.280
Right?

s156
00:09:04.280 --> 00:09:07.320
People literally show you on the corridor like, "Hey, your product is awesome.

s157
00:09:07.320 --> 00:09:09.000
We can talk to our data now.

s158
00:09:09.000 --> 00:09:15.520
We don't need to wait on the queue to like, you know, get access to like analysts to answer our questions in every 2 weeks and so on.

s159
00:09:15.520 --> 00:09:16.280
Right?"

s160
00:09:16.280 --> 00:09:18.960
And after a couple of months, they start coming back to you with frustrations.

s161
00:09:18.960 --> 00:09:21.200
Say, "I cannot do this in the product anymore.

s162
00:09:21.200 --> 00:09:21.520
Right?

s163
00:09:21.520 --> 00:09:25.240
I I would like to I mean, I saw this other AI product that does this and so on."

s164
00:09:25.240 --> 00:09:28.360
This is what I call the collapsing of the wow factor.

s165
00:09:28.360 --> 00:09:29.120
Okay?

s166
00:09:29.120 --> 00:09:33.480
So, initially you are cool, but then and that becomes a habit, right?

s167
00:09:33.480 --> 00:09:36.680
You basically change their habit and it becomes standard for them.

s168
00:09:36.680 --> 00:09:38.880
Now, you need to raise the bar again.

s169
00:09:38.880 --> 00:09:43.400
So, the journey that we usually see with the sales teams is like you start with talk to your data.

s170
00:09:43.400 --> 00:09:54.160
How do we get you out of those like, you know, hundreds of dashboards situation, dependency to the analyst, and then first we'll basically like democratize the data for you so that you can basically talk to your data.

s171
00:09:54.160 --> 00:09:57.560
Then the next wave comes with all the MCP connections, right?

s172
00:09:57.560 --> 00:09:59.680
All the integrations that you are building.

s173
00:09:59.680 --> 00:10:02.240
Now it becomes like automate my workflows.

s174
00:10:02.240 --> 00:10:12.880
We literally have now sellers who are going to use our agent basically to monitor their inbox, they monitor their Slack channels, you know, keep track of all the customer questions coming about like product questions,

s175
00:10:12.880 --> 00:10:18.840
uh use the agent to draft responses that save that in Gmail, review them afterwards like send those things out, right?

s176
00:10:18.840 --> 00:10:21.240
Or they automate their like outreach workflows and so on.

s177
00:10:21.240 --> 00:10:22.120
Okay, that's great.

s178
00:10:22.120 --> 00:10:24.320
Now I became an orchestrator, right?

s179
00:10:24.320 --> 00:10:27.280
I'm basically automating my workflow workflows.

s180
00:10:27.280 --> 00:10:35.240
Then the next thing you see start happening is teams, they get these like, you know, tool democratization, this empowerment coming to them, right?

s181
00:10:35.240 --> 00:10:47.120
Because historically a lot of these go-to-market teams, they have been always in the backlog of someone, backlog of of an IT team or like trying to get a SaaS budget to learn and get a vendor on board to actually like enable something.

s182
00:10:47.120 --> 00:10:50.640
And now all of a sudden they're able to build team skills.

s183
00:10:50.640 --> 00:10:56.800
They're able to build like, you know, the custom dashboards that are basically like fully, you know, optimized for what their team needs.

s184
00:10:56.800 --> 00:11:02.000
Are able to like deploy applications, automations, alerts, and things like that, right?

s185
00:11:02.000 --> 00:11:05.680
And then the comes the phase of hyper-personalization, right?

s186
00:11:05.680 --> 00:11:14.440
Everyone is able to now like get everything personalized for them, not only for themselves, but also for their customers with living context of customers, contacts, and things like that.

s187
00:11:14.440 --> 00:11:24.440
I think the main message I want to give here is if you just do the first stage, and if you just wait there, you will get disrupted in a month or two,

s188
00:11:24.440 --> 00:11:25.160
right?

s189
00:11:25.160 --> 00:11:33.440
Because now you already raised their expectations, that already became a baseline, and then they will find another product that does better than you, and right now the switch is very easy.

s190
00:11:33.440 --> 00:11:34.960
They're going to just switch over night.

s191
00:11:34.960 --> 00:11:35.520
Okay?

s192
00:11:35.520 --> 00:11:37.040
So, you need to keep iterating.

s193
00:11:37.040 --> 00:11:44.480
You need to keep that wow factor, and I cannot just rely on the fact that, you know, what I built so far is going to stay cool forever.

s194
00:11:45.280 --> 00:11:49.920
And the next thing is, how do you deal with basically the changing technology?

s195
00:11:49.920 --> 00:11:52.160
So, I talked to a lot of customers.

s196
00:11:52.160 --> 00:11:56.800
And then, you know, it sometimes you run into these customers, big enterprises, very big brands.

s197
00:11:56.800 --> 00:12:00.320
And then they are still trying to purchase that perfect architecture.

s198
00:12:00.320 --> 00:12:02.440
They're trying to like test different frameworks.

s199
00:12:02.440 --> 00:12:06.760
They're trying to see how the, you know, the technology is maturing and everything and so on.

s200
00:12:06.760 --> 00:12:11.600
But, the thing that they don't do is they don't build, and then they don't launch, and they don't learn.

s201
00:12:11.600 --> 00:12:12.120
Right?

s202
00:12:12.120 --> 00:12:17.120
All these blue boxes that you see here, those are all the things we added after the launch.

s203
00:12:17.120 --> 00:12:17.680
Right?

s204
00:12:17.680 --> 00:12:23.160
When we literally first launched the agent, it was a nine-page long agent instructions.

s205
00:12:23.160 --> 00:12:25.760
It was couple of Cortex analyst tools, semantic views.

s206
00:12:25.760 --> 00:12:28.920
It was a Cortex search service for our unstructured data.

s207
00:12:28.920 --> 00:12:32.600
And we were managing the agent instructions versions out of a Google Doc.

s208
00:12:32.600 --> 00:12:33.960
That's how we launched it.

s209
00:12:33.960 --> 00:12:35.000
To 6,000 people.

s210
00:12:35.000 --> 00:12:35.960
Right?

s211
00:12:35.960 --> 00:12:37.560
Now we realized, okay, it's not going to work out.

s212
00:12:37.560 --> 00:12:38.720
Let's figure out CICD.

s213
00:12:38.720 --> 00:12:39.440
It's not going to work out.

s214
00:12:39.440 --> 00:12:44.320
Let's figure out our basically eval infrastructure with all the like the unit test, routing test, and everything.

s215
00:12:44.320 --> 00:12:45.160
Right?

s216
00:12:45.160 --> 00:12:51.760
Then we start basically like coming to a point where, for example, we were creating all these like business processes and workflows.

s217
00:12:51.760 --> 00:12:53.880
We couldn't fit them into the agent instructions anymore.

s218
00:12:53.880 --> 00:12:55.839
And then the skills came, and we were like, "Oh, perfect.

s219
00:12:55.839 --> 00:12:57.920
Let's build a skill library."

s220
00:12:57.920 --> 00:12:59.520
You know, then the MCPs came.

s221
00:12:59.520 --> 00:13:00.200
Perfect.

s222
00:13:00.200 --> 00:13:05.440
But now, like we have to put bunch of other instructions to basically orchestrate that, we hit the limits on the agent instructions.

s223
00:13:05.440 --> 00:13:06.360
What do we do?

s224
00:13:06.360 --> 00:13:08.800
Okay, let's do the progressive disclosures.

s225
00:13:08.800 --> 00:13:09.640
Right?

s226
00:13:09.640 --> 00:13:12.200
And then user memory comes, task scheduling comes.

s227
00:13:12.200 --> 00:13:17.520
We want to go beyond the chat screen and then, you know, chat interface and start doing the Slack interface and things like that.

s228
00:13:17.520 --> 00:13:26.839
If I look at the PRD and the architectural diagram we wrote in the beginning of the project, if I compare to this architecture we have now, 80% of it It match.

s229
00:13:26.839 --> 00:13:27.640
Okay?

s230
00:13:27.640 --> 00:13:36.080
So, like if you look at our sprints, like maybe 60-70% of the work we are doing is adding new features, improving quality, and all kind of things.

s231
00:13:36.080 --> 00:13:41.040
But 30-40% of the work is that we are constantly re-architecting with the new technology.

s232
00:13:41.040 --> 00:13:48.960
So, this is a time where like you need to get your hands dirty, you need to run with the new technology, and then you shouldn't be like, you know, too much tied to your architecture.

s233
00:13:48.960 --> 00:13:55.520
You should be okay to like pivot very easily, so that you can basically double on down on these like new capabilities and things like that.

s234
00:13:55.520 --> 00:14:03.880
And then the longer you wait, the more, you know, you lose towards your competition, because if your competition is doing these kind of things like 3-4 months ahead of you,

s235
00:14:03.880 --> 00:14:04.240
right?

s236
00:14:04.240 --> 00:14:07.400
That means that they're also getting more customers.

s237
00:14:08.200 --> 00:14:13.800
Um last thing is I would really, really recommend investing in your logs.

s238
00:14:13.800 --> 00:14:14.240
Okay?

s239
00:14:14.240 --> 00:14:17.480
Because they create the basically the feedback loop.

s240
00:14:17.480 --> 00:14:19.720
So, first of all, technically it's very fun.

s241
00:14:19.720 --> 00:14:20.040
Okay?

s242
00:14:20.040 --> 00:14:24.120
So, you basically use LLMs to like classify your logs and things like that.

s243
00:14:24.120 --> 00:14:28.240
As I said, like we have 1.2 million questions, we get 40,000 questions every week.

s244
00:14:28.240 --> 00:14:32.720
It's technically very fun, you know, how you do that at scale without breaking the bank and so on.

s245
00:14:32.720 --> 00:14:37.080
You know, our data scientists love working on those things, and then they really experiment with new things.

s246
00:14:37.080 --> 00:14:43.839
But as a result of that, what we get is we get a very extremely detailed breakdown of topics and, you know, things that we are having.

s247
00:14:43.839 --> 00:14:51.280
I'm just to showing you the top category categorization level there, but then basically we are able to track like, you know, what kind of questions they are asking.

s248
00:14:51.280 --> 00:14:59.080
We are able to break down each of those categories to subcategories, you know, they are able to get like detailed example questions, this and that, and so on.

s249
00:14:59.080 --> 00:15:01.160
All good, but how do we use that?

s250
00:15:01.160 --> 00:15:03.760
Then we start creating the basically the feedback loops.

s251
00:15:03.760 --> 00:15:04.440
Right?

s252
00:15:04.440 --> 00:15:09.960
I know, I mean, I still interview, of course, users, but now I see in real time what my feature gaps are.

s253
00:15:09.960 --> 00:15:17.839
I'm clearly seeing what people are asking and we are not able to answer or where we have a like a quality issue, where they are swearing at the agent or like at repeating their question,

s254
00:15:17.839 --> 00:15:20.040
so that we see where to improve.

s255
00:15:20.040 --> 00:15:22.840
For sales enablement is a goldmine.

s256
00:15:22.840 --> 00:15:25.120
Let's say that we launch a new product.

s257
00:15:25.120 --> 00:15:35.360
Usually, you know, they would need to interview maybe 100 sellers a week to be able to understand like how basically the you know, the topics are changing where there's gaps in terms of like knowledge documents, battle cards.

s258
00:15:35.360 --> 00:15:39.320
I see that in real time in a minute or two by just asking an element question.

s259
00:15:39.320 --> 00:15:51.400
And then we can then, you know, connect to Confluence, we can connect to Jira, we can connect to Slack channels, we can ingest the PRDs, and in couple of minutes we can actually like, you know, generate battle cards, sales enablement document and then feed it back into the agent.

s260
00:15:51.400 --> 00:15:51.680
Right?

s261
00:15:51.680 --> 00:15:55.440
I mean, you cannot do that kind of a like a feedback loop with humans, right?

s262
00:15:55.440 --> 00:15:58.600
So then you we can basically automate these kind of things.

s263
00:15:58.600 --> 00:16:02.920
Um within the sales organization, there will be different teams that are good trying to connect each other.

s264
00:16:02.920 --> 00:16:06.000
They are trying to maybe target similar accounts from different angles.

s265
00:16:06.000 --> 00:16:07.480
They don't know about each other.

s266
00:16:07.480 --> 00:16:08.080
We do.

s267
00:16:08.080 --> 00:16:09.520
We are now able to ping them.

s268
00:16:09.520 --> 00:16:11.880
And then we are able to basically do matchmaking.

s269
00:16:11.880 --> 00:16:12.160
Right?

s270
00:16:12.160 --> 00:16:20.520
I'm just giving you couple of examples, but this is also one of those areas where like you start building your AI platform, you start building your architecture.

s271
00:16:20.520 --> 00:16:23.160
The first features are difficult to get out.

s272
00:16:23.160 --> 00:16:24.720
The next ones are easy.

s273
00:16:24.720 --> 00:16:32.720
And then once you start tapping into your logs, this this like hockey stick exponential thing actually starts happening and it's magical.

s274
00:16:34.520 --> 00:16:40.320
So, if you were to take a couple of things from this talk, like quality over coverage.

s275
00:16:40.320 --> 00:16:43.640
I'm very, very like religious about this.

s276
00:16:43.640 --> 00:16:47.839
If you go for the coverage, you are going to shoot yourself in the foot.

s277
00:16:47.839 --> 00:16:49.480
Okay?

s278
00:16:49.480 --> 00:16:50.720
Change management.

s279
00:16:50.720 --> 00:16:53.520
A lot of engineers doesn't think about this, right?

s280
00:16:53.520 --> 00:17:00.400
A lot of these AI initiatives, they don't fail because there's an issue with the technology, there's an issue with that as assuming you did the first one, right?

s281
00:17:00.400 --> 00:17:01.240
Right?

s282
00:17:01.240 --> 00:17:04.319
So, they fail actually in activation.

s283
00:17:04.319 --> 00:17:12.120
So, make sure that you have a plan for that, especially in larger organizations where we are dealing with like 6,000 go-to-market users, right?

s284
00:17:12.120 --> 00:17:16.600
Um again, don't forget this concept of collapsing law factor.

s285
00:17:16.600 --> 00:17:18.360
You cannot stay where you are.

s286
00:17:18.360 --> 00:17:20.839
You cannot just say that, "Hey, I did an innovation.

s287
00:17:20.839 --> 00:17:23.240
I'm going to surf that for a year."

s288
00:17:23.240 --> 00:17:25.520
You know, every time people are happy, you should be paranoid.

s289
00:17:25.520 --> 00:17:29.320
You should be like, "Okay, what am I going to show them in a month or two now?"

s290
00:17:29.320 --> 00:17:32.080
How do I basically keep that excitement going on?

s291
00:17:32.080 --> 00:17:33.160
Right?

s292
00:17:33.160 --> 00:17:34.800
Build fast with today's stack.

s293
00:17:34.800 --> 00:17:43.560
Like, don't try to invest in these like high, you know, super like plat, you know, architectures and have these like 6 9 months of long projects and things like that.

s294
00:17:43.560 --> 00:17:47.000
How do you turn around these things in weeks, days, and so on?

s295
00:17:47.000 --> 00:17:50.280
And just be comfortable with the fact that you are constantly going to be re-architecting.

s296
00:17:50.280 --> 00:17:51.960
That's fine.

s297
00:17:51.960 --> 00:17:53.120
Right?

s298
00:17:53.120 --> 00:17:55.480
Just don't over-invest in the current architecture.

s299
00:17:55.480 --> 00:17:58.840
Just make sure that you keep your like flexibility out there.

s300
00:17:58.840 --> 00:18:00.440
Um and then the feedback loops.

s301
00:18:00.440 --> 00:18:06.960
I think that's what kind of like gives you really that like, you know, the incremental part of like that hockey stick exponential part of the thing.

s302
00:18:06.960 --> 00:18:14.360
Um we constantly publish like blog posts, I mean, where we kind of like try to have our, you know, learnings shared with uh our customers and so on.

s303
00:18:14.360 --> 00:18:22.840
Like, we have blog posts on like how we do agent instructions, how we do our structured data with semantic views, you know, how we basically build our rack-based like knowledge assistants,

s304
00:18:22.840 --> 00:18:26.640
the non-technical side of the story, like how do you derive change management, and so on.

s305
00:18:26.640 --> 00:18:28.560
So, feel free to check those.

s306
00:18:28.560 --> 00:18:33.480
Um and yeah, I think that's the end of my talk.

s307
00:18:33.480 --> 00:18:35.480
Okay.

s308
00:18:36.160 --> 00:18:39.120
We have time for one question.

s309
00:18:39.400 --> 00:18:42.160
Okay, there you go.

s310
00:18:47.680 --> 00:18:48.840
Thanks for the talk.

s311
00:18:48.840 --> 00:18:54.400
Um I don't know if you already said this, but I saw in the the titles of the articles Snowflake Intelligence.

s312
00:18:54.400 --> 00:19:04.680
Is that an underlying context or layer that the tool or system you built was on top of, or was that the tool itself or something else?

s313
00:19:04.680 --> 00:19:09.720
Yeah, Snowflake Intelligence, we renamed that to Snowflake Co-work a couple of weeks ago in our summit.

s314
00:19:09.720 --> 00:19:16.040
That's basically our no-code agent platform that we basically build have available for our business users.

s315
00:19:16.040 --> 00:19:26.320
I mean, the advantage of that is that all of these tools are on like, you know, you know, Cortex analyst or Cortex search or Cortex sense, a lot of those things are basically comes out of the box.

s316
00:19:26.320 --> 00:19:32.000
We made a strategic choice for our internal thing where we said that, "Look, it is important that we bring all our data together."

s317
00:19:32.000 --> 00:19:33.160
And we do that in Snowflake.

s318
00:19:33.160 --> 00:19:39.240
We bring all the first-party, the third-party data, all the Salesforce data, everything, the call transcripts, and so on, all together.

s319
00:19:39.240 --> 00:19:45.400
And then these agents then can basically basically inherit a lot of the role-based access controls and so on.

s320
00:19:45.400 --> 00:19:49.880
And I literally can deploy these agents without writing a single line of code, right?

s321
00:19:49.880 --> 00:19:54.960
And then, you know, you don't need to worry about the UI, the chat UI comes out of the box, and so on.

s322
00:19:54.960 --> 00:19:58.400
And then we have been the customer zero of that like internally to build this ourselves.

s323
00:19:58.400 --> 00:20:04.440
And then, you know, our customers are able to go and then build similar things basically on Snowflake over platform as well.

s324
00:20:04.440 --> 00:20:14.160
And it comes with the guardrails and things where you don't really need to worry about them going very, you know, crazy on, you know, what data sources to do things, and so on.

s325
00:20:14.160 --> 00:20:16.040
So we are able to do a lot of curation.

s326
00:20:16.040 --> 00:20:20.400
We are able to do a lot of security guardrails in there as well.

s327
00:20:21.560 --> 00:20:23.760
Thank you.