WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ksjwtQop4os

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ksjwtQop4os.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:13.920
Thank you for coming.

s2
00:00:13.920 --> 00:00:15.200
Uh my name is Jeff Estle.

s3
00:00:15.200 --> 00:00:16.160
I work at Elastic.

s4
00:00:16.160 --> 00:00:18.960
I am a AI architect.

s5
00:00:18.960 --> 00:00:22.400
We picked a new title this year so make it relevant.

s6
00:00:22.400 --> 00:00:23.439
This is James.

s7
00:00:23.439 --> 00:00:24.480
You want to introd yourself?

s8
00:00:24.480 --> 00:00:25.840
Yeah, I'm James Williams.

s9
00:00:25.840 --> 00:00:29.359
I'm a uh solutions architect focused on search.

s10
00:00:29.359 --> 00:00:34.640
So, I help customers with all the technical stuff you're about to walk through today.

s11
00:00:34.640 --> 00:00:39.360
So, this is a very hands-on, as in I do not intend to talk much.

s12
00:00:39.360 --> 00:00:43.120
I've been talking a lot lately on the road, so we're going to have me not talk a lot.

s13
00:00:43.120 --> 00:00:45.200
Um, you're going to go through hands-on stuff.

s14
00:00:45.200 --> 00:00:49.360
I will say in the upper right corner, you'll see that short URL.

s15
00:00:49.360 --> 00:00:51.760
Um, it's a ELA elastic.

s16
00:00:51.760 --> 00:00:53.840
This is the workshop environment.

s17
00:00:53.840 --> 00:00:56.719
Um, quick note.

s18
00:00:56.719 --> 00:00:58.480
So, they never told me how many people were coming.

s19
00:00:58.480 --> 00:01:00.320
So, I took a random guess of 75.

s20
00:01:00.320 --> 00:01:02.160
Looks like I undersshot a bit.

s21
00:01:02.160 --> 00:01:06.320
So, there um 75 of you will start up pretty quickly.

s22
00:01:06.320 --> 00:01:10.080
The other handful that are a little slower on the on the trigger here.

s23
00:01:10.080 --> 00:01:12.159
It's just a few extra minutes to start up.

s24
00:01:12.159 --> 00:01:17.040
Um but basically, this is going to take you to an instruct environment.

s25
00:01:17.040 --> 00:01:18.159
And then I'll give a little intro.

s26
00:01:18.159 --> 00:01:20.240
I'm just giving this spiel while it goes.

s27
00:01:20.240 --> 00:01:25.200
Um we'll go to this.

s28
00:01:25.200 --> 00:01:29.360
You're going to go here and then you'll hit start and that's going to spin up an environment for you.

s29
00:01:29.360 --> 00:01:37.600
So while that's doing it, um, so how many people are familiar with Elastic, the company, Elastic, Elastic Search.

s30
00:01:37.600 --> 00:01:38.560
All right, pretty good.

s31
00:01:38.560 --> 00:01:39.759
It's good turnout.

s32
00:01:39.759 --> 00:01:48.880
Um, and how many of you have used Elastic to build a like a a ragbacked application, retrieval, augmented generation?

s33
00:01:48.880 --> 00:01:49.680
Great.

s34
00:01:49.680 --> 00:01:55.840
How many of you are sick of hearing the term rag from the past two three years?

s35
00:01:55.840 --> 00:01:56.720
Fantastic.

s36
00:01:56.720 --> 00:02:06.560
I know I said it about a thousand times at many conferences and um my boss who no longer works here actually used to get really mad when we keep bringing up rag.

s37
00:02:06.560 --> 00:02:10.160
Now the thing is um return a meant generation didn't actually go away.

s38
00:02:10.160 --> 00:02:11.760
Luckily it just kind of fell to the background.

s39
00:02:11.760 --> 00:02:22.160
But as agents got more and more uh capable and you know effective they just kind of took over the management of being able to you know retrieve data and and analyze it.

s40
00:02:22.160 --> 00:02:24.959
So the models are getting certainly smarter and smarter.

s41
00:02:24.959 --> 00:02:28.560
Um you know Fable came out for a minute and teased us and then ran away.

s42
00:02:28.560 --> 00:02:41.360
But um there's still a a need for pulling in information into these models be it you know things that are time based that change proprietary information um price data

s43
00:02:41.360 --> 00:02:46.560
anything out there that the model hopefully hasn't been trained on whether they tell us or not.

s44
00:02:46.560 --> 00:02:50.400
Um but there's a need for that and so that's where things like elastic search come in.

s45
00:02:50.400 --> 00:02:54.879
It's the retrieval layer that helps make these these agents very capable.

s46
00:02:54.879 --> 00:03:06.640
And the the trick or the catch, I guess, is if you feed these models, no matter how smart they are, garbage data and you confidently tell them this is the answer,

s47
00:03:06.640 --> 00:03:09.360
pretty much they're going to believe you because that's what you're telling them.

s48
00:03:09.360 --> 00:03:13.200
So, we want to be able to give accurate information and and give it.

s49
00:03:13.200 --> 00:03:15.440
So, this is kind of what this workshop is about.

s50
00:03:15.440 --> 00:03:25.040
I'm sure there's 20 other workshops that explain how to build agents and I we actually have some too if you want to go to our website but this one's going to talk to you kind of remind about

s51
00:03:25.040 --> 00:03:39.840
um vector search semantic search why it's important where it's useful but also where it breaks down and then we'll go into um the kind of lexical search BM25 so that's that keyword search we've been doing for for decades

s52
00:03:39.840 --> 00:03:46.480
um where it's still relevant today and useful and then also again where that kind of breaks down and then putting bring them together into hybrid search.

s53
00:03:46.480 --> 00:03:49.920
So, being able to kind of get the best of both.

s54
00:03:49.920 --> 00:03:54.720
We'll do combinations with uh RF, which is an out of the box combination.

s55
00:03:54.720 --> 00:04:01.519
And then we'll do some a little bit more advanced um linear combination where you can tweak those balances a little bit.

s56
00:04:01.519 --> 00:04:16.160
And then should you have time, we'll actually put it into um a little bit of Elastic's agent builder platform where you can see what happens if you retrieve bad information and tell the agent it's it's good, it'll

s57
00:04:16.160 --> 00:04:16.720
doesn't work.

s58
00:04:16.720 --> 00:04:18.799
I mean, it works, but it gives you a bad answer.

s59
00:04:18.799 --> 00:04:25.520
And if you give it the correct answers and things like that, um should you be very quick and get all the way to the end, there is a bonus

s60
00:04:25.520 --> 00:04:29.919
um one about um sorry, rerankers.

s61
00:04:29.919 --> 00:04:37.919
So reranking where it comes in useful that um the workshop is kind of basically four plus a bonus um chapters challenges.

s62
00:04:37.919 --> 00:04:40.080
Each one has two parts.

s63
00:04:40.080 --> 00:04:46.160
There's the part in the dev tools console where you just run API commands within Elastic Search.

s64
00:04:46.160 --> 00:04:52.400
Um each one of these environments if you're in it and hit start and everything, what's happening is a serverless instance is being spun up for you.

s65
00:04:52.400 --> 00:04:54.160
Um and then you'll load some data.

s66
00:04:54.160 --> 00:04:57.919
It's a small data set, you know, it's a workshop, so nothing crazy.

s67
00:04:57.919 --> 00:05:08.400
And um you'll run the kind of the the short version of each one of these in the dev tools console and then you'll hop over to a Python notebook tab and you'll run the more in-depth one.

s68
00:05:08.400 --> 00:05:17.520
What I would say is I'm going to stop talking in a minute and let you work on the workshop and then every 20 25 minutes we'll kind of check in and see where people are at.

s69
00:05:17.520 --> 00:05:21.840
I'll tell a little summary about the the cha the chapter challenge you're at.

s70
00:05:21.840 --> 00:05:34.960
Um, to get the most out of this workshop, I would definitely spend time reading through what's in the the Jupyter notebook, especially it kind of talks about why the documents come up in the orders they are and how to tweak it and how to tune and why it's important.

s71
00:05:34.960 --> 00:05:39.759
Um, if you just blindly hit run all in the notebook and fire through, you'll be you'll be done 10 minutes.

s72
00:05:39.759 --> 00:05:44.320
But if you actually spend through it, it's going to probably take longer than this workshop session.

s73
00:05:44.320 --> 00:05:47.280
So, um, with that, I'll let you get started.

s74
00:05:47.280 --> 00:05:49.199
So, some of you should start up right away.

s75
00:05:49.199 --> 00:05:54.639
Some of you will take a few more minutes in the meantime.

s76
00:05:54.639 --> 00:05:55.840
Is anyone actually started?

s77
00:05:55.840 --> 00:05:56.960
Sometimes you take a few minutes.

s78
00:05:56.960 --> 00:05:57.759
That's the thing.

s79
00:05:57.759 --> 00:05:58.240
Yeah,

s80
00:05:58.240 --> 00:05:59.440
we got it.

s81
00:05:59.440 --> 00:05:59.840
Okay.

s82
00:05:59.840 --> 00:06:01.440
And if you get stuck, raise your hand.

s83
00:06:01.440 --> 00:06:02.880
I'm basically the runner.

s84
00:06:02.880 --> 00:06:12.000
So, I'll check out what you've got going on on your on your system and and and potentially help unstick you if if it's something trivial.

s85
00:06:12.000 --> 00:06:14.800
And if you have any questions, feel free to raise your hand.

s86
00:06:14.800 --> 00:06:25.120
happy to provide a little bit more detail about what's going on in the exercise if um you're looking at something and you're you're a little bit confused and want want won't want won't want won't want won't want won't want won't want to ask a question there

s87
00:06:28.639 --> 00:06:42.319
and this is all also available um in a repo that should be public yeah I made it public um so you can run the notebooks um outside of instruct either after or or whenever if you want

s88
00:06:42.319 --> 00:06:52.479
kind of glossed over this But the one big thing that I think has changed with rag over the last year year and a half is some of the data providers

s89
00:06:52.479 --> 00:07:00.720
for vector database and we're in that in that bucket have brought inference into the vector database.

s90
00:07:00.720 --> 00:07:06.960
So you will see embedding generation happen within elastic search serverless.

s91
00:07:06.960 --> 00:07:10.479
We have an inference service called Elastic Inference Service.

s92
00:07:10.479 --> 00:07:17.680
Very um creative name that powers the models that are being used for these workshops.

s93
00:07:17.680 --> 00:07:20.560
So we've got GINA models from from an acquisition that we made.

s94
00:07:20.560 --> 00:07:24.000
So we actually do our own model development now.

s95
00:07:24.000 --> 00:07:28.319
And um and it's focused mostly on dense vector models and re-ranking models.

s96
00:07:28.319 --> 00:07:36.880
So you're going to get insight into some new tech that wasn't part of Elastic Search a year ago.

s97
00:07:37.120 --> 00:07:39.440
That's that's all I have to add there.

s98
00:07:39.440 --> 00:07:40.720
We've heard of chunking.

s99
00:07:40.720 --> 00:07:42.720
We do we do we do help with chunking.

s100
00:07:42.720 --> 00:07:46.560
So yeah, we can answer this and feel free to jump jump in.

s101
00:07:46.560 --> 00:07:48.639
He's the manager of the search specialist team.

s102
00:07:48.639 --> 00:07:49.840
I just wander around.

s103
00:07:49.840 --> 00:07:57.680
Um so as James mentioned the we have you know the text field semantic text in elastic search.

s104
00:07:57.680 --> 00:07:58.800
So there's a couple ways to do this.

s105
00:07:58.800 --> 00:08:02.720
Semantic text is kind of the automatic easy way on road.

s106
00:08:02.720 --> 00:08:08.400
Um, so by default we will chunk on sentence boundaries and break into a certain amount.

s107
00:08:08.400 --> 00:08:09.840
I can't remember off hand what the number is

s108
00:08:09.840 --> 00:08:10.720
250.

s109
00:08:10.720 --> 00:08:11.360
There you go.

s110
00:08:11.360 --> 00:08:13.840
And so that's you know configurable.

s111
00:08:13.840 --> 00:08:16.639
So by default breaks on sentence you can break on word boundaries.

s112
00:08:16.639 --> 00:08:17.680
You can change it.

s113
00:08:17.680 --> 00:08:24.240
Um the the Gina model that James mentioned we you know part of Gina now or Gina is part of us whatever.

s114
00:08:24.240 --> 00:08:29.360
Um it has a very very large context window but you're not always going to want to fill that for various reasons.

s115
00:08:29.360 --> 00:08:36.719
Um, and so you can configure chunking down to the word to paragraph, page boundary, things like that.

s116
00:08:37.120 --> 00:08:44.640
We have a semantic mapping field in elastic which basically takes a piece of text and chunks it.

s117
00:08:44.640 --> 00:08:57.360
So if you send us five pages of text, we're going to chunk that into 250 token chunks with one sentence overlap and store that in the in the system for retrieval.

s118
00:08:57.360 --> 00:09:07.680
There are frameworks that talk about semantic chunking and there are other labels for what I will call um large context window chunking.

s119
00:09:07.680 --> 00:09:11.440
We actually and and this is talked about in the lab in the later sections.

s120
00:09:11.440 --> 00:09:24.720
We do have a technique that we use called late chunking that can work on large context windows that is not exposed via elastic inference service yet but it is something that the GINA models do support.

s121
00:09:24.720 --> 00:09:29.600
That's the closest thing that I would say we would have to semantic chunking.

s122
00:09:29.600 --> 00:09:44.399
But we don't really have a need for that construct because the semantic text field itself uh allows us to deliver the same thing you would get with semantic chunking just by chunking the data and then and then running searches across it

s123
00:09:44.399 --> 00:09:48.240
um in a you know in a semantic flavor and you you'll get exposed to that in a lab.

s124
00:09:48.240 --> 00:09:56.080
So we would we would solve the problem with traditional chunking and hybrid search, not any exotic semantic chunking scheme.

s125
00:09:56.080 --> 00:09:58.320
You're welcome.

s126
00:10:01.519 --> 00:10:02.080
All right.

s127
00:10:02.080 --> 00:10:04.080
Yeah.

s128
00:10:05.600 --> 00:10:06.320
Yes.

s129
00:10:06.320 --> 00:10:10.160
So I made it available for a week because I know how things go.

s130
00:10:10.160 --> 00:10:13.040
Um so it is available for a week there.

s131
00:10:13.040 --> 00:10:18.720
Um, and this isn't a this isn't anywhere.

s132
00:10:18.720 --> 00:10:24.560
Um, I the workshop is in my repo for better or worse.

s133
00:10:24.560 --> 00:10:30.320
Um, and so the the notebooks are all in here.

s134
00:10:30.320 --> 00:10:41.760
So if you if you want to run just the notebooks, um, you know, you can hook it up to your serverless cluster or any kind of cluster should should work that has semantic text

s135
00:10:41.760 --> 00:10:42.240
in there.

s136
00:10:42.240 --> 00:10:46.160
But yeah, the the workshop itself is good for a week.

s137
00:10:49.839 --> 00:10:52.800
And where was I?

s138
00:10:54.160 --> 00:11:00.480
And yeah, so like I mentioned, the the first part the first half of each lab is this dev tools where you run a mapping.

s139
00:11:00.480 --> 00:11:02.480
The left side is the input for your API.

s140
00:11:02.480 --> 00:11:04.399
The right side is the output.

s141
00:11:04.399 --> 00:11:19.279
And then you'll get down to part two where you pop into the Python notebook and it'll automatically connect to the um the Elastic Search environment that spun up for you.

s142
00:11:22.079 --> 00:11:28.959
I see they're letting folks in as as folks kind of go out to make sure the room is still full.

s143
00:11:28.959 --> 00:11:35.040
Um the top right hand corner is really the URL that you want to hit.

s144
00:11:35.040 --> 00:11:37.440
That's where all the workshop content is.

s145
00:11:37.440 --> 00:11:44.000
And then there's a second URL that is the source code repo for all the notebooks for the workshop today.

s146
00:11:44.000 --> 00:11:46.079
Those are the two most useful resources.

s147
00:11:46.079 --> 00:11:58.800
So if you're coming in and you haven't fired up the workshop yet, top right hand corner and then the bottom right hand corner if you're interested in the Python notebooks that power the content that you're walking through today.

s148
00:12:00.240 --> 00:12:04.959
I'll probably repeat that message every time I see someone walk in.

s149
00:12:08.639 --> 00:12:19.600
That is a really good question and I'll repeat it so the other folks can hear it but is is a major point of this um lab sequence to let you know that

s150
00:12:19.600 --> 00:12:37.519
VM25based search which is traditional lexical search we've been doing for a long time and dense vector semantic search do those two things need to be put together to form hybrid search that is the core thesis right now that we have as a company

s151
00:12:37.519 --> 00:12:57.279
and the only thing that we add to that well obviously that that's of value is the option of adding rerankers on top of that but yes that is a core thesis we're firm believers in that approach and that's in line with everything that I've seen since before even joining elastic even if you're using a

s152
00:12:57.279 --> 00:13:06.160
dedicated just pure vector database you often have to pair that with some other retrieval system and do the equivalent of hybrid search.

s153
00:13:06.160 --> 00:13:09.279
The advantage to elastic is and you're seeing this in the labs.

s154
00:13:09.279 --> 00:13:10.800
It's one API call.

s155
00:13:10.800 --> 00:13:21.200
We handle all the inference that happens from ingest chunking to real-time search where the query gets translated into a query vector and we do all the work for you under the hood.

s156
00:13:21.200 --> 00:13:32.880
And that's what we we consider that of high value especially when you're talking about largecale use cases where latency, accuracy, tunability matter.

s157
00:13:40.000 --> 00:13:41.040
They are configurable.

s158
00:13:41.040 --> 00:13:42.720
The chunking settings, are they configurable?

s159
00:13:42.720 --> 00:13:43.200
They are.

s160
00:13:43.200 --> 00:13:48.160
The documentation has insight into how you can change them and what chunk settings can be changed.

s161
00:13:48.160 --> 00:13:53.120
So, we've got very detailed um API docs that walk through that.

s162
00:13:53.120 --> 00:14:00.800
Uh they're not described in detail in the workshop, but if you go to the elastic docs, you'll see what the what the configuration settings are for chunking.

s163
00:14:00.800 --> 00:14:06.000
And some of our customers do bring their own vectors and chunk on their own.

s164
00:14:06.000 --> 00:14:08.800
We support that through standard um SDKs.

s165
00:14:08.800 --> 00:14:20.240
So if you're using Python, Java, Net Core, like pretty much any TypeScript, any modern programming language, um you could still use Elastic in the format for these workshops.

s166
00:14:20.240 --> 00:14:23.519
You're just doing more work as a developer.

s167
00:14:24.000 --> 00:14:27.600
And so I'm sure people are moving along and you can continue on as at your own pace.

s168
00:14:27.600 --> 00:14:33.120
Um, like I said, I'm going to check in every once in a while and kind of talk about the the labs a little bit.

s169
00:14:33.120 --> 00:14:43.120
So, the the first one, right, the the whole point is essentially this little chart right here talking about how vector search works and and semantic search and what you're getting it.

s170
00:14:43.120 --> 00:14:52.160
And so, when when you use an embedding model that's, you know, hooked up to Elastic and you know, we talk about Gina as the out of the box one, but you can certainly bring your own.

s171
00:14:52.160 --> 00:14:54.560
You can hook up to other inference services.

s172
00:14:54.560 --> 00:15:01.680
um we just make it easy for you to get started with Gina, but certainly bring your own and use semantic searches or semantic text as well.

s173
00:15:01.680 --> 00:15:18.000
But the the nice thing about this particular way it sets up is you know when you create an an embedding at query you know at inest time you create the embedding from your your document but then at query time you need to use the same embedding model to do the query time otherwise it's you know like me

s174
00:15:18.000 --> 00:15:21.199
trying to speak Portuguese it would not go well.

s175
00:15:21.199 --> 00:15:36.000
Um so by when when I started when we started doing this a few years ago um in the early days early um you know you had to keep track of the inference ID you had to keep track of the model and then the model you know then you have developers all and you have to like make sure

s176
00:15:36.000 --> 00:15:37.760
everyone's using the right one and there's all this things.

s177
00:15:37.760 --> 00:15:43.279
So by embedding it in the ID in the uh field itself it helps on two ways.

s178
00:15:43.279 --> 00:15:53.519
one, you only have to set it once and as the person who's setting up the index, um, you know, there might be other people using that same index to ingest their own data or query it, right?

s179
00:15:53.519 --> 00:15:58.560
So, you can set that up once again, whether it's the Gina model or or some other model.

s180
00:15:58.560 --> 00:16:01.839
Um, anytime you use that field, it just becomes like another field.

s181
00:16:01.839 --> 00:16:05.360
You just either send data in or you query it.

s182
00:16:05.360 --> 00:16:10.399
So then at query time, what you're seeing up here is um the text comes in.

s183
00:16:10.399 --> 00:16:14.560
It may or may not get chunked up depending on the the size and settings.

s184
00:16:14.560 --> 00:16:19.279
It goes to the um maybe the elastic inference service which is default out of the box.

s185
00:16:19.279 --> 00:16:20.800
Maybe goes to another inference service.

s186
00:16:20.800 --> 00:16:21.600
That's fine.

s187
00:16:21.600 --> 00:16:28.480
And then we generate the embeddings and we then pass it back to elastic to do the um HSW&N.

s188
00:16:28.480 --> 00:16:29.440
We do have other discount.

s189
00:16:29.440 --> 00:16:35.680
We have disk ANN but by default we're using HSW the hierarchical navigable small worlds graph.

s190
00:16:35.680 --> 00:16:50.240
um basically just allows quick traversal to find nearest neighbors of of relevant documents and then you get back your top k five 10 whatever you want documents and that allows you to ideally if everything's working well find documents

s191
00:16:50.240 --> 00:17:05.120
or you know chunks that are semantically similar to what you're asking and as you learned in the lab it's not the exact keywords it's the idea of what you're asking so if if all of us in this room ask you know how to set up security

s192
00:17:05.120 --> 00:17:07.839
on, you know, TLS on elastic search or something.

s193
00:17:07.839 --> 00:17:10.720
We're all going to phrase a little differently, but the idea is all the same.

s194
00:17:10.720 --> 00:17:13.839
And so hopefully you get back the same documents.

s195
00:17:13.839 --> 00:17:18.559
And you kind of saw where sometimes that breaks down.

s196
00:17:18.559 --> 00:17:32.160
Um, one of the key places is if you have a very specific keyword, um, certainly if you have a keyword that is not a word like a product ID that has makes no sense, it's very specific,

s197
00:17:32.160 --> 00:17:35.600
um, semantic search then kind of falls apart because it doesn't really need that.

s198
00:17:35.600 --> 00:17:47.919
And that's what the second lab is where bringing in um, BM25, lexical, keyword filtering, and all the other traditional so-called search uh, methodologies we've been doing for a long time help can augment that.

s199
00:17:47.919 --> 00:17:54.160
And so I'll give you I'm sure some of you are progressed past that one, but I'll check in in a few minutes and review lab two.

s200
00:17:54.160 --> 00:17:58.559
And again, if there's any questions, feel free to ask,

s201
00:18:06.640 --> 00:18:07.039
right?

s202
00:18:07.039 --> 00:18:07.440
Page.

s203
00:18:07.440 --> 00:18:07.679
Yeah.

s204
00:18:07.679 --> 00:18:08.080
Paging.

s205
00:18:08.080 --> 00:18:11.120
So if you have like a hundred results, you want to get back 10 at a time.

s206
00:18:11.120 --> 00:18:23.600
Is that is that what you're asking for or Oh yeah.

s207
00:18:23.600 --> 00:18:25.600
So you can do that.

s208
00:18:25.600 --> 00:18:30.880
There's um I forgot the the exact thing, but there's a basically essentially a checkpoint ID.

s209
00:18:30.880 --> 00:18:32.320
And so then it it continues on.

s210
00:18:32.320 --> 00:18:34.000
And so yeah, you're not researching again.

s211
00:18:34.000 --> 00:18:35.919
You're not rerunning the search for that part.

s212
00:18:35.919 --> 00:18:36.559
It goes on.

s213
00:18:36.559 --> 00:18:40.799
So you can that's what our pageionation API part does.

s214
00:18:40.799 --> 00:18:47.520
So if you know that's a possibility, you use that and then we send you that key and it's basically like this is where you start and continue on.

s215
00:18:47.520 --> 00:18:49.919
I think the official label for it is point in time.

s216
00:18:49.919 --> 00:18:50.400
There you go.

s217
00:18:50.400 --> 00:18:57.600
If you're looking in Elastic Docs, uh just you search for point in time and it shows you how to use point in time to do to do just that.

s218
00:18:57.600 --> 00:18:58.640
Oh, good question.

s219
00:18:58.640 --> 00:19:06.000
Um so the the summary field is generated just for this workshop and it was generated by an LM.

s220
00:19:06.000 --> 00:19:08.000
It it's not exactly part of it.

s221
00:19:08.000 --> 00:19:15.200
The real reason is when I first ran through this and you print out the whole body field, it was way too long and it was cluttering everything up.

s222
00:19:15.200 --> 00:19:16.640
So just having this little summary field.

s223
00:19:16.640 --> 00:19:30.640
Now that said, you can use like for example an ingest pipeline in elastic search and make an inference call and say I do want to generate a summary of of this you know document or I do want to pull out the possible queries that could do that.

s224
00:19:30.640 --> 00:19:32.559
So you can do that on uh ingest.

s225
00:19:32.559 --> 00:19:35.280
It's just for this particular thing it was just to make it easier.

s226
00:19:35.280 --> 00:19:38.000
But so the the vector field you're searching is is the title.

s227
00:19:38.000 --> 00:19:40.000
Yeah.

s228
00:19:40.160 --> 00:19:49.679
And as we get deeper into the labs, we don't use that that construct, but there is LLM completion API calls in in the later labs that it would be the exact same thing.

s229
00:19:49.679 --> 00:19:52.080
You just wired in your ingest pipeline

s230
00:19:52.080 --> 00:19:54.080
question.

s231
00:19:54.559 --> 00:19:55.919
It's a great question.

s232
00:19:55.919 --> 00:20:08.559
So when it comes to work that we've done to make HNSW more efficient, um that started two years ago actually two plus years ago where we introduced int 8 quantization

s233
00:20:08.559 --> 00:20:11.360
which I still I think is still the default.

s234
00:20:11.360 --> 00:20:16.240
So by default we're going to take float 32 vectors and quantize them before we store them in memory.

s235
00:20:16.240 --> 00:20:22.559
We still we still keep the full vector um but we're only storing the quantized vector in memory.

s236
00:20:22.559 --> 00:20:29.760
We also have uh BBQ which is yeah and we got a blog that talks about BBQ.

s237
00:20:29.760 --> 00:20:41.039
This is a little over a year old maybe even a little bit longer than a year old where we've compressed the HNSW um graph vector that we use even more all the way down to

s238
00:20:41.039 --> 00:20:48.480
uh basically a bit and and there's sophisticated stuff under the hood that supports that and still keeps the recall high.

s239
00:20:48.480 --> 00:20:57.280
And then we just introduced disk BBQ which pretty much eliminates the memory overhead for um for vector search.

s240
00:20:57.280 --> 00:21:09.760
The only uh difference I would say between disbel as the underlying mechanism for search.

s241
00:21:09.760 --> 00:21:14.799
Um and then and the newest form of that is disbrietary to elastic.

s242
00:21:14.799 --> 00:21:19.679
No one else implements that [clears throat] graph-based search.

s243
00:21:19.679 --> 00:21:21.849
You want to take that one?

s244
00:21:21.849 --> 00:21:23.360
[laughter]

s245
00:21:23.360 --> 00:21:24.480
So, graph based search.

s246
00:21:24.480 --> 00:21:28.640
Uh, so to be fully clear, we are not a graph database.

s247
00:21:28.640 --> 00:21:31.039
There are some very specific graph databases.

s248
00:21:31.039 --> 00:21:33.679
Um, graph has its place.

s249
00:21:33.679 --> 00:21:38.799
It usually has a lot of overhead to keep and maintained.

s250
00:21:38.799 --> 00:21:40.000
If you do that, great.

s251
00:21:40.000 --> 00:21:49.520
Um for the vast majority of the people though it's we find that it's not really needed where the ability to especially with quantization where you can search across large amount of documents quickly.

s252
00:21:49.520 --> 00:21:54.400
Um now especially with the agent world too where you can do multiple queries in.

s253
00:21:54.400 --> 00:22:07.200
So you can one of the labs later on it does a couple round trips I think just two as an example but you can go in get your kind of first phase get a sample of documents and then see oh okay so they're all related to this tag and then go searching.

s254
00:22:07.200 --> 00:22:12.480
So that's where things like filtering and um some of our other search methodologies help.

s255
00:22:12.480 --> 00:22:21.600
Um and so yeah, I mean I just it comes up, but we haven't seen it, you know, too often as an actual blocker where you have to set it up.

s256
00:22:21.600 --> 00:22:33.280
And and the last thing I'll say about that is what we see is far more common is um machine learning models that do classification on the fly, which is another way to solve that graph challenge.

s257
00:22:33.280 --> 00:22:39.840
works in some situations, not not in all, but Neo Forj and other graph databases have been around for a long time.

s258
00:22:39.840 --> 00:22:54.159
I don't know of any that scale really well, and we're trying to target multi-billion vector data sets, which means that um we're very careful not to not to intro introduce a construct that keeps us from being able to scale at that level.

s259
00:22:54.159 --> 00:22:59.200
And we we we haven't found the need to go there yet, although it comes up a lot.

s260
00:22:59.200 --> 00:23:01.520
It was a big theme last year at this conference.

s261
00:23:01.520 --> 00:23:07.679
I remember being here and seeing graph rag was was heavily talked about, probably will continue to be.

s262
00:23:07.679 --> 00:23:14.240
Um, I haven't seen a technology nail that yet.

s263
00:23:14.480 --> 00:23:20.720
And maybe there's a Neo Forj person in here that would contest me on that and say, "No, no, no. We have it figured out."

s264
00:23:20.720 --> 00:23:25.200
I think they do, but not at not at the scale that we operate at.

s265
00:23:25.440 --> 00:23:26.080
Yeah.

s266
00:23:26.080 --> 00:23:27.280
Oh, so you came from their session.

s267
00:23:27.280 --> 00:23:27.679
Nice.

s268
00:23:27.679 --> 00:23:29.679
Okay.

s269
00:23:32.480 --> 00:23:34.880
in the back.

s270
00:23:35.600 --> 00:23:45.120
I think the question was we can use classification models to uh um to achieve some of the same results you might get with with a with a graph traversal.

s271
00:23:45.120 --> 00:23:54.640
Um so we have the concept in elastic of running uh machine learning models in in elastic on what we call ML nodes.

s272
00:23:54.640 --> 00:24:02.400
um you could basically upload whatever classification model that you've identified that works and wire it into elastic that way.

s273
00:24:02.400 --> 00:24:12.400
The other model the other option would be to create your own model that you have your own external inference to and and potentially use that um in place of uh of our ML technology

s274
00:24:12.400 --> 00:24:17.679
and I'm hoping that we'll introduce more sophistication down the road in that area.

s275
00:24:17.679 --> 00:24:19.760
But we don't have anything noteworthy to announce right now.

s276
00:24:19.760 --> 00:24:28.880
We're focused on taking the genome models that we have and pushing the limit to what we can do with handling unstructured data and and dense vector embeddings and rerankers.

s277
00:24:28.880 --> 00:24:39.520
We'll probably end up pivoting off of off of that once we nail the um the reranking piece and and the and the dense vector models that are that are in development and

s278
00:24:39.520 --> 00:24:42.720
you know doing quite well in the market.

s279
00:24:46.400 --> 00:24:58.080
I'll add in too that along with that so there's classification models and you can also run things like NE models name entity recognition so you know if you pull out the entities if you're trying to identify

s280
00:24:58.080 --> 00:25:07.200
person location um business names things like that or product names you can run those on the fly as part of a you know part of the query where you as the the

s281
00:25:07.200 --> 00:25:13.600
query comes in you run it through the ne model and ne models are very fast because they're they're small.

s282
00:25:13.600 --> 00:25:25.279
They're really small models and you pull out these entities and then you know again now with agents and things like this you don't have to wire up crazy um painless scripts the agent can take those and then create a query that's more tuned for that.

s283
00:25:25.279 --> 00:25:32.960
So there's a lot of ML capabilities that we don't talk about as much as we used to because again like Jim said we kind of pivoted to focus on

s284
00:25:32.960 --> 00:25:39.840
other things but they're still there in the stack and very capable filtering in that's great.

s285
00:25:40.880 --> 00:25:41.840
What's up man?

s286
00:25:41.840 --> 00:25:46.960
Um one of our distinguished essays over in the corner can ask him all our questions.

s287
00:25:46.960 --> 00:25:49.039
I'm gonna now redirect.

s288
00:25:49.039 --> 00:25:57.120
Um for the people who came in aside from Sneil, the upper right corner is the short URL for the workshop and you can spin that up.

s289
00:25:57.120 --> 00:26:07.919
It opens up a kind of lab for you lab environment and also spins up a um elastic search serverless cluster all set up for you.

s290
00:26:07.919 --> 00:26:11.039
So feel free to run through there.

s291
00:26:11.760 --> 00:26:23.200
So, there's um there's one main knob for reciprocal uh uh RF and it is and I'm I'm spacing out here off the top of my head.

s292
00:26:23.200 --> 00:26:23.919
I do not know.

s293
00:26:23.919 --> 00:26:26.640
You're you're pulling it up.

s294
00:26:29.679 --> 00:26:38.159
I don't know if we expose it in the in the RF um attributes for the retriever.

s295
00:26:43.200 --> 00:26:45.520
What's the default constant

s296
00:26:45.520 --> 00:26:46.000
constant?

s297
00:26:46.000 --> 00:26:46.960
The rank constant.

s298
00:26:46.960 --> 00:26:49.120
I It is something that you can configure.

s299
00:26:49.120 --> 00:26:52.480
Um I don't know what the range is.

s300
00:26:53.279 --> 00:26:55.440
That's what lab five is the optional one.

s301
00:26:55.440 --> 00:26:59.440
It's all about well lab five is rebranking.

s302
00:26:59.440 --> 00:27:00.880
RF is in there somewhere.

s303
00:27:00.880 --> 00:27:03.840
Um sorry brain melted for a sec.

s304
00:27:03.840 --> 00:27:10.480
Um it's in there somewhere about talks about what what Sil's talking about a little bit more example and you can change things like that.

s305
00:27:10.480 --> 00:27:12.480
So

s306
00:27:14.000 --> 00:27:16.799
So I don't know did that answer your question?

s307
00:27:16.799 --> 00:27:19.200
Okay cool

s308
00:27:23.440 --> 00:27:25.760
fire away.

s309
00:27:38.400 --> 00:27:39.440
That's a great question.

s310
00:27:39.440 --> 00:27:45.760
There's actually a table that kind of explains some of those decision trees in the later workshop.

s311
00:27:45.760 --> 00:27:50.320
I want to say workshop number four.

s312
00:27:50.320 --> 00:28:02.880
But the short answer is for a small document corpus and and and my rule of thumb is always see what you can accomplish with a combination of BM25 and semantic search.

s313
00:28:02.880 --> 00:28:09.360
If that gets you the recall or precision uh target that you're looking for, whatever relevance targets you have, stop.

s314
00:28:09.360 --> 00:28:14.480
Anytime you add anything on top of that, you're going to be taking on an additional latency hit.

s315
00:28:14.480 --> 00:28:15.919
Only do that if you need to.

s316
00:28:15.919 --> 00:28:30.640
And the next uh you know extension point would be to um to use a rebanker model and um and then it's just a matter of picking one that that you think is going to work best for your corpus and doing some testing and validating that it moves the needle enough.

s317
00:28:30.640 --> 00:28:34.320
Um that would be kind of the standard happy path approach.

s318
00:28:34.320 --> 00:28:46.640
There are some folks that would change that uh recommendation slightly and and say look just do BM25 just regular search um and and then and then add semantic search if it's not

s319
00:28:46.640 --> 00:28:58.799
um relevant enough but I'm a big fan of dense vectors so I would just go ahead and wire them into the architecture right out of the gate and uh that way I don't have to worry about things like

s320
00:28:58.799 --> 00:29:13.919
you know you you you get into these weird situations with lexical where you're starting to add synonyms or get into advanced um lexical tuning techniques that you might not need if the semantic search is able to to pair well with a lexical search.

s321
00:29:13.919 --> 00:29:15.919
Welcome.

s322
00:29:16.320 --> 00:29:18.320
All right, so I know people are moving along which is great.

s323
00:29:18.320 --> 00:29:25.039
Um just give a little wrap up on lab two which again was about your favorite be lexical search.

s324
00:29:25.039 --> 00:29:30.960
Uh but hopefully you did learn uh you know get across the point that it does it does have its its points.

s325
00:29:30.960 --> 00:29:43.600
Um, we just barely touched the surface on Lexical because, you know, for 15 years or so, you know, a dozen years at Elastic, that's what we had.

s326
00:29:43.600 --> 00:29:55.600
So, if you've ever been on the Elastic Search doc sites and you go to the search part and you click on the search API, there's an insane amount of ways to change and tune and tweak lexical search.

s327
00:29:55.600 --> 00:30:02.320
Because to James's point, before we could do semantic search, we had to do all these uh crazy things to get relevant search.

s328
00:30:02.320 --> 00:30:07.200
Um luckily now we don't have to do quite as much, at least usually once in a while still.

s329
00:30:07.200 --> 00:30:11.200
Um but with lexical search, you do it does have a it's its point.

s330
00:30:11.200 --> 00:30:20.159
Um but it does tend to break down um anytime you break out of looking for very specific keywords or a very specific phrase.

s331
00:30:20.159 --> 00:30:25.919
Um the upside of it is it's I mean I don't even want to say I was going to say it's generally faster.

s332
00:30:25.919 --> 00:30:31.200
Although now with quantization things like that that's not always the case but um by default it's kind of faster.

s333
00:30:31.200 --> 00:30:39.120
So um in lab three most people are probably along there but you move on to combining them.

s334
00:30:39.120 --> 00:30:50.799
And I will say that even though I'm a big fan of dense vector search that lexical search usually does have lower latency profiles in almost all use cases and that's because there is no ML inference happening.

s335
00:30:50.799 --> 00:31:06.080
So anytime you go semantic search with with dense vectors you're introducing um a call to um you know an inference endpoint and more often than not that inference endpoint is going to be more of a bottleneck than the rest of the system.

s336
00:31:06.080 --> 00:31:16.720
And that's another reason why we wired um the inference into the into the data platform because when you're doing endtoend testing, it doesn't matter if your vector database returns vectors,

s337
00:31:16.720 --> 00:31:17.600
you know, really fast.

s338
00:31:17.600 --> 00:31:22.320
If the inference that you're depending on is slow, then your search is going to be slow.

s339
00:31:22.320 --> 00:31:35.279
I think that's a question that is shapeshifting as the data platforms wire more capability into the systems and these models improve the um and every vendor is going to have their own specific take.

s340
00:31:35.279 --> 00:31:50.080
Our take is that everything starts with traditional search which makes a lot of sense given our heritage and then you add semantic capabilities on top of that and and then you layer in re-ranking

s341
00:31:50.080 --> 00:31:53.519
if you if you need to in terms of where that stuff happens.

s342
00:31:53.519 --> 00:31:55.200
That's where the devil's in the details.

s343
00:31:55.200 --> 00:32:01.279
As as a developer trying to build an architecture around this stuff, your options are are quite vast.

s344
00:32:01.279 --> 00:32:14.399
And depending on the stack you start with, you're going to have perceptions of what you should be doing in the data platform versus what you might have some sort of middleware around it that that is that is wired up.

s345
00:32:14.399 --> 00:32:24.799
And and I'm using the word middleware kind of loosely, but there's a whole application runtime construct that's went away ever since this this AI boom started.

s346
00:32:24.799 --> 00:32:36.880
that is going to have to come back and it's either going to land in the data platform itself where we're doing inference and some of these other things you would normally have to coordinate with these external services or

s347
00:32:36.880 --> 00:32:49.279
um there's going to be you know platforms to try to try to accomplish um that and then use the data platform as just kind of this thing that you plug in and I'm thinking about like old school stuff like back in the

s348
00:32:49.279 --> 00:32:57.120
old days when I started my my tech career we were doing object relational mapping and and the database was, you know, supposedly pluggable.

s349
00:32:57.120 --> 00:33:01.679
You swap out your credentials and it doesn't matter because you're not using SQL anymore.

s350
00:33:01.679 --> 00:33:14.399
Um, I think that we're in the very early days of AI where for retrieval purposes as soon as you um soon as you start writing code, you're locked into whatever stack you've picked.

s351
00:33:14.399 --> 00:33:21.120
and and that dictates a lot of different design decisions that that you have to make especially as you start to scale.

s352
00:33:21.120 --> 00:33:31.279
And where we kick in and have an advantage against let's call it a simpler stack would be when the the data volumes get larger and the latency requirements become more

s353
00:33:31.279 --> 00:33:37.279
um you know more I don't want to say severe but the SLAs's are really tight.

s354
00:33:37.279 --> 00:33:49.120
That's when we're going to shine because we can do things that the other platforms can't do and we've got a level of tunability especially on the relevance side that that can't be accomplished with other data platforms.

s355
00:33:49.760 --> 00:34:00.480
That's a very good question that we do talk about in the lab but I don't think we provide that authoritative answer key of here's how you construct your weights based on

s356
00:34:00.480 --> 00:34:06.880
you know things that you can tell from the shape of your data or the shape of the searches that you know users are going to execute.

s357
00:34:06.880 --> 00:34:12.159
The um the short answer to that is today that's actually hard to do.

s358
00:34:12.159 --> 00:34:25.839
Um the longer answer is and and this is something that I think you'll see come from us is um there's a missing element to relevance tuning that hasn't been

s359
00:34:25.839 --> 00:34:29.359
um addressed in a way that makes this turnkey yet.

s360
00:34:29.359 --> 00:34:33.200
And um I've actually got a reference architecture that I put out a year ago.

s361
00:34:33.200 --> 00:34:34.159
It needs to be refined.

s362
00:34:34.159 --> 00:34:38.879
It's based on Mashra about auto that covers autotuning and that concept.

s363
00:34:38.879 --> 00:34:49.200
Um, but in a perfect world, you would actually be able to take the searches that are coming through the system and the telemetry from those user experiences and things like click-through

s364
00:34:49.200 --> 00:34:53.440
traffic and and and thumbs up, thumbs down, basically telemetry from the users.

s365
00:34:53.440 --> 00:35:04.160
And then also um any other business metrics that you have wired into your use case to uh to actually tune the relevance which might introduce different weights for linear.

s366
00:35:04.160 --> 00:35:15.119
Um it might um toggle between RF and linear with specific weights depending on the type of query coming in or um you have all sorts of other types of techniques.

s367
00:35:15.119 --> 00:35:24.720
The one that I think is more common than even those two is just coming up with different sets of of filters um based on the the query that's coming in.

s368
00:35:24.720 --> 00:35:27.920
So um Sunnil built out the original architecture for this.

s369
00:35:27.920 --> 00:35:30.240
We called it an intelligent search layer.

s370
00:35:30.240 --> 00:35:35.599
Um and it's something that Azure search has tried to kind of wire into their product.

s371
00:35:35.599 --> 00:35:42.640
It is something that you can actually do today with elastic using the agent builder construct which is covered in the last lab.

s372
00:35:42.640 --> 00:35:48.880
Um, but that's that's a little bit different technique than just tweaking the weights on the fly based on what we're seeing.

s373
00:35:48.880 --> 00:36:00.720
You you would actually have um the um the LLM interrogate the query and then and then execute searches with different filters based on that interrogation, which I actually love that technique.

s374
00:36:00.720 --> 00:36:05.119
I think it's fantastic and I'm glad that we have a construct for it with with Elastic.

s375
00:36:05.119 --> 00:36:10.720
And that's that's all, you know, stuff that agent builder designed to solve for.

s376
00:36:11.599 --> 00:36:13.599
Yep.

s377
00:36:16.400 --> 00:36:18.400
Exactly.

s378
00:36:19.280 --> 00:36:26.560
And that's where I'm hoping and and this isn't something that that we've announced that we're going to do yet, but I'm hoping that at some point you're going to see

s379
00:36:26.560 --> 00:36:42.960
uh the concept of autotuned relevance um be something you can enable and and have attributes or settings that you specify to to turn that on and maybe even turn it up depending on the uh the use case and and what you're seeing with actual usage of the system.

s380
00:36:42.960 --> 00:36:54.320
And and just to be clear with everybody in the room, we have not released any technology that does that yet, but it comes up often enough in the field when where I'm working with customers that

s381
00:36:54.320 --> 00:36:59.680
it's something that if we had it, it would move the needle big time and make developers lives a lot easier.

s382
00:36:59.680 --> 00:37:01.920
Would you agree?

s383
00:37:01.920 --> 00:37:07.920
Okay, I'm gonna I'm gonna bring it up again to our CTO and make him make him say no again.

s384
00:37:09.040 --> 00:37:09.359
Right.

s385
00:37:09.359 --> 00:37:11.280
So, we're gonna just walk through this one.

s386
00:37:11.280 --> 00:37:14.400
Most of you are probably along with that past it.

s387
00:37:14.400 --> 00:37:15.440
If not, it's all right.

s388
00:37:15.440 --> 00:37:28.640
Um so right so we're trying to combine the two the two different methodologies here and the the crux of the problem is and it's in here somewhere right like

s389
00:37:28.640 --> 00:37:44.960
BM25 has a scale it scores documents from zero to kind of whatever whereas vector search normally it's zero to one and so when you have these scales that do not that aren't the same obviously

s390
00:37:44.960 --> 00:37:57.280
A poor result in a lexical BM25 search is almost always going to beat out from a pure number perspective a good semantic search.

s391
00:37:57.280 --> 00:38:01.520
So then it's the fun game of how do you combine them?

s392
00:38:01.520 --> 00:38:04.160
And so there's a couple of different ways.

s393
00:38:04.160 --> 00:38:11.440
Um we don't get into the kind of choosing one on the fly here but there are you know different ways to get into that.

s394
00:38:11.440 --> 00:38:13.040
Kind of getting the basics here.

s395
00:38:13.040 --> 00:38:24.400
So RF is kind of designed to be the out of the box works pretty well um until it doesn't.

s396
00:38:24.400 --> 00:38:25.920
But it's the it's a good first start.

s397
00:38:25.920 --> 00:38:27.760
You don't have to know how to tune.

s398
00:38:27.760 --> 00:38:32.160
You don't have to really know um the details of your data.

s399
00:38:32.160 --> 00:38:37.359
But the idea then is that essentially forot there's a table in here.

s400
00:38:37.359 --> 00:38:39.440
You you throw out the scores.

s401
00:38:39.440 --> 00:38:42.079
So you you run the two searches in parallel.

s402
00:38:42.079 --> 00:38:43.440
you take the order.

s403
00:38:43.440 --> 00:38:53.920
There's a little bit of that rank constant in there, but essentially if you're high, if your result document is high in both, it's probably a good m it's probably a good match.

s404
00:38:53.920 --> 00:38:57.359
When they're not, that's where the kind of combination gets into play.

s405
00:38:57.359 --> 00:39:00.720
And you have this pretty simple, where is that?

s406
00:39:00.720 --> 00:39:02.000
I scroll past it.

s407
00:39:02.000 --> 00:39:04.240
Um, scoring in here.

s408
00:39:04.240 --> 00:39:05.200
And then you have a table.

s409
00:39:05.200 --> 00:39:14.000
And so essentially what you're returning is not a a uh list of uh score documents scored order.

s410
00:39:14.000 --> 00:39:15.200
It's just the rank order.

s411
00:39:15.200 --> 00:39:17.680
And so that's why there's a rank constant in there.

s412
00:39:17.680 --> 00:39:20.400
Um because you're just getting the rank of the documents.

s413
00:39:20.400 --> 00:39:25.359
And you can see in here, where is it working?

s414
00:39:27.200 --> 00:39:29.040
Yeah.

s415
00:39:29.040 --> 00:39:43.680
Um it talks about it in this table where the the documents come in in different um different orders, but by ranking them then you hopefully get the best answer for you.

s416
00:39:43.680 --> 00:39:44.880
But that doesn't always work.

s417
00:39:44.880 --> 00:39:51.760
And actually James mentioned this a second ago, filtering is a really powerful and effective way to help along.

s418
00:39:51.760 --> 00:40:07.119
So if you can filter your data by keyword, geoloccation, date, anything that gets that data set down to a smaller candidate pool, you're going to have better results usually.

s419
00:40:07.119 --> 00:40:10.400
And because you have a smaller thing, you're not just casting a net over everything.

s420
00:40:10.400 --> 00:40:12.720
You're filtering it down and you're narrowing it down, right?

s421
00:40:12.720 --> 00:40:27.520
It's like if if I want pizza tonight, you know, I'm not just going to search for food maybe, but I can if I did search for food, I can then filter by pizza and at least I'll just get pizza results or you know, things like that.

s422
00:40:27.520 --> 00:40:36.960
So, um that's where filtering comes into play and it's it's something elastic search does very very well and and very quickly and we have a lot of different ways to do it.

s423
00:40:36.960 --> 00:40:40.079
But then it gets into that doesn't always work.

s424
00:40:40.079 --> 00:40:45.839
And so you can get into linear combination and there's we talk about minmax in here.

s425
00:40:45.839 --> 00:40:47.119
There's different things you can do.

s426
00:40:47.119 --> 00:40:50.079
Um but this is where you're normalizing the scores.

s427
00:40:50.079 --> 00:40:56.160
So instead of taking the order, you're going to then instead just normalize the scores.

s428
00:40:56.160 --> 00:41:00.880
So everything has the same scoring range from zero to one and you can put them together.

s429
00:41:00.880 --> 00:41:04.720
And there's a little bit of the the tuning examples in here.

s430
00:41:04.720 --> 00:41:15.680
It's a again it's kind of the um intro level to tuning for sure, but you can tune the weights and you can see um which one works well.

s431
00:41:15.680 --> 00:41:17.520
And this is where judgment sets comes in.

s432
00:41:17.520 --> 00:41:28.800
And this is where things like um you know we have a lot of different workshops and things like that and the the intelligent query layer and things like this do this more automatically

s433
00:41:28.800 --> 00:41:38.400
not in and sophisticated way but essentially you have your judgments that that a human these days you can get an LLM to help you but um is based on this query.

s434
00:41:38.400 --> 00:41:40.160
these are the exact documents I want.

s435
00:41:40.160 --> 00:41:52.160
And then you run your experiments and you see which weights work for you to give those results as as accurately as possible through here.

s436
00:41:52.480 --> 00:41:54.720
Do you have something you want to add?

s437
00:41:54.720 --> 00:41:55.920
Sorry.

s438
00:41:55.920 --> 00:42:07.680
And I will add one more item because this is actually a technique that we used in the intelligent um intelligent query layer architecture that um that that we were talking about quite a bit.

s439
00:42:07.680 --> 00:42:10.319
uh last year and the year before.

s440
00:42:10.319 --> 00:42:14.240
Um, Elastic supports the concept of search templates.

s441
00:42:14.240 --> 00:42:18.720
So, you could technically have three different search strategies.

s442
00:42:18.720 --> 00:42:28.560
One that uses RF, one that uses linear and maybe another one that uses um a combination of linear or RF with a reranker.

s443
00:42:28.560 --> 00:42:39.119
And um if you have a layer on top, let's call it an LLM that's got a simple prompt that says based on this type of query, I want you to route to this search template.

s444
00:42:39.119 --> 00:42:42.160
Based on this other type of query, I want you to route to this search template.

s445
00:42:42.160 --> 00:42:54.319
You can literally have the equivalent of a what I'll call poor man's um autotuning where you're literally toggling between different search strategies based on what the search query that's coming in looks like.

s446
00:42:54.319 --> 00:43:02.079
and you have something and it doesn't have to be an LLM, but you have some way of interrogating that search to decide which search template to invoke.

s447
00:43:02.079 --> 00:43:10.800
Most developers that are building out an advanced use case where they don't have a single strategy that they can use for all queries are going to end up with something like that under the hood.

s448
00:43:10.800 --> 00:43:13.520
It's just a matter of how hard it is to actually code that up.

s449
00:43:13.520 --> 00:43:25.200
And in the case of elastic um a lot of folks don't know about search templates but that is a powerful mechanism that can be used uh to reduce the amount of boilerplate code you have to write for that type of architecture.

s450
00:43:25.760 --> 00:43:32.480
Not covered in the labs but noteworthy to at least have an understanding that that that concept exists in elastic.

s451
00:43:33.839 --> 00:43:48.079
So the table at the end again is kind of the take it with a grain of salt but it's a good way to get started right like if you don't know your your data too well or it's all you know kind of an unknown query mix you can start with RF it does work pretty well it's not

s452
00:43:48.079 --> 00:44:04.720
something we didn't invent RF but we do use it other places uses it um and then beyond that you can start getting into um other ways to com to combine the data um but you don't always have to do hybrid search sometimes semantic search is enough sometimes times BM25.

s453
00:44:04.720 --> 00:44:16.000
Um, but usually we we find that there's some combination of of semantic and lexical search that helps the um the retrieval work most effectively.

s454
00:44:20.400 --> 00:44:36.560
And then lab four is where you um despite the title you do put it into a model but we're going to show you how again as I said at the top of this LLMs are extremely

s455
00:44:36.560 --> 00:44:50.560
you know smart's not their best word but you know they are effective at giving you answers um but you anytime you have to feed them data of that they're outside their training knowledge or anything, you can pretty confidently

s456
00:44:50.560 --> 00:44:59.599
get them to give you the wrong answer by providing them the the set of data and say the sky is orange and then usually it'll tell you the sky is orange.

s457
00:44:59.599 --> 00:45:08.960
Um there's ways around that, but this is where we kind of show you that retrieval is still important to agentic um answers.

s458
00:45:19.680 --> 00:45:21.200
Quick show of hands.

s459
00:45:21.200 --> 00:45:24.176
Are there any folks that are not

s460
00:45:24.176 --> 00:45:24.240
[snorts]

s461
00:45:24.240 --> 00:45:28.960
actually past lab three yet?

s462
00:45:29.520 --> 00:45:30.480
Okay, good.

s463
00:45:30.480 --> 00:45:32.960
People are cruising along.

s464
00:45:32.960 --> 00:45:34.960
Awesome.

s465
00:45:42.960 --> 00:45:55.280
And so it's worth mentioning too, this is where we actually get into agent builder um elastic's agentic platform which has um the ability to obviously to run agents to run tools and skills.

s466
00:45:55.280 --> 00:45:58.079
Um you can hook into lots of LLMs.

s467
00:45:58.079 --> 00:46:03.200
The elastic infrance service has you know all your favorite LLMs available to it.

s468
00:46:03.200 --> 00:46:18.400
Um but you can again bring your own um your LLM and and it's able to then run these tools directly against uh the data within elastic and actually now it's not covered in the work in the the workshop but we have

s469
00:46:18.400 --> 00:46:26.400
uh workflows and workflows essentially are anytime you have a an ordered set of tasks you want completed can have decision trees and things like that.

s470
00:46:26.400 --> 00:46:29.280
Um the agents can call these workflows and they can also reach out.

s471
00:46:29.280 --> 00:46:35.599
So you can call out to other systems, you can call MCP tools, you can call APIs, call anything to your heart's content.

s472
00:46:35.599 --> 00:46:46.480
Um, but again outside the scope of this workshop, but since we have a little time here, this is where the agent has the ability then to start doing things.

s473
00:46:46.480 --> 00:46:58.720
So not just retrieving your answers for you, but there's a whole other workshop that you could go to where because of you know an incident for example the the observability

s474
00:46:58.720 --> 00:47:06.000
the hotel trace data you bring in it can decide yeah the system is not is running out of of resources it needs to scale.

s475
00:47:06.000 --> 00:47:09.920
So you can call a workflow for example to autoscale your Kubernetes environment.

s476
00:47:09.920 --> 00:47:20.800
It's just one example, but this is where we actually get into agent builder and it walks you through a little bit of the um the capabilities.

s477
00:47:31.599 --> 00:47:45.040
and one of the more interesting things about agent builder for folks that are software developers is the fact that everything that you see in a cabana user interface in agent builder is is programmable

s478
00:47:45.040 --> 00:47:46.960
programmable via API.

s479
00:47:46.960 --> 00:47:58.880
So if you decided that you wanted to do a quick prototype and you're using our agent builder mechanism within Elastic and you think, "Oh, I have to rewrite all this code from scratch now and build out a real app."

s480
00:47:58.880 --> 00:47:59.920
You don't have to do that.

s481
00:47:59.920 --> 00:48:08.800
You can reuse those same endpoints in your application and make the same calls in a custom user experience that um that you control.

s482
00:48:08.800 --> 00:48:19.839
And then you have the added benefit if you decide that you wanted to pull those agent builder components out and wire in another framework or another, you know, another set of APIs that you either build or use a different framework for,

s483
00:48:19.839 --> 00:48:21.760
uh, we would be compatible with that as well.

s484
00:48:21.760 --> 00:48:30.960
So the goal for us is to provide as much turnkey experience as possible, but not lock you in so that you're forced to build applications the way that we want you to build them.

s485
00:48:30.960 --> 00:48:33.599
We're very flexible, almost to a fault.

s486
00:48:33.599 --> 00:48:39.119
So when you ask us a question, oftentimes the answer starts with it, it depends.

s487
00:48:39.119 --> 00:48:43.359
Did anybody come in recently that doesn't have the URL?

s488
00:48:43.359 --> 00:48:44.480
Sorry.

s489
00:48:44.480 --> 00:48:46.640
Thought so, but start up there.

s490
00:48:46.640 --> 00:48:48.640
Um, so it's in the upper right.

s491
00:48:48.640 --> 00:48:52.400
It's that short ELD one.

s492
00:48:52.400 --> 00:49:00.800
And in case anyone um came in later and didn't catch me say it, this will be available for a week, the actual workshop itself.

s493
00:49:00.800 --> 00:49:08.960
And then the blue by James um link is the the repo that has it has all the the notebooks in there.

s494
00:49:08.960 --> 00:49:12.960
It has all the other code in there, but the relevant stuff you'll use without instruct is the notebooks.

s495
00:49:12.960 --> 00:49:17.359
You can hook it up to any elastic search cluster you have free one.

s496
00:49:17.359 --> 00:49:20.160
Well, sort of if you hook up a different service.

s497
00:49:20.160 --> 00:49:22.559
Um but it'll load the data and you can run through it all.

s498
00:49:22.559 --> 00:49:27.839
So that that'll be available until someone deletes it.

s499
00:49:27.839 --> 00:49:30.319
delete me.

s500
00:49:35.440 --> 00:49:46.240
By the way, thanks to our distinguished architect over here, Sunnil, we uh wanted to make sure that we give you bonus content, bonus workshop materials.

s501
00:49:46.240 --> 00:49:54.800
There are additional workshops available online that have the same format that we used here today where there basically is an instruct environment.

s502
00:49:54.800 --> 00:50:01.839
It's pre pre-built so that you can walk through the concepts and learn more about how Elastic works under the hood.

s503
00:50:01.839 --> 00:50:05.920
We highly recommend that you hit that URL and and use the workshops.

s504
00:50:05.920 --> 00:50:13.599
It's essentially free training and once you complete those workshops, you'll probably increase your market value by I don't know$100 $200,000 a year.

s505
00:50:13.599 --> 00:50:16.720
It's a guaranteed pay raise.

s506
00:50:16.800 --> 00:50:17.040
Okay.

s507
00:50:17.040 --> 00:50:20.960
So, while people are working on that, I'm going to do two things.

s508
00:50:20.960 --> 00:50:36.160
So, one of the So, sorry [clears throat] I kept talking a couple times about um you know providing new information and up-to-date and um fresh and whatever proprietary information to the these LLMs

s509
00:50:36.160 --> 00:50:42.559
as a way to augment it's where the augment the a and rag comes in augment their knowledge.

s510
00:50:42.559 --> 00:50:52.480
The other thing that comes up a lot and this is coming up more and more in the in well everywhere especially as agents start running around and and doing things even faster

s511
00:50:52.480 --> 00:50:54.800
is being able to site the sources.

s512
00:50:54.800 --> 00:51:00.880
So models are really good at coming up with information and oftentimes it is correct.

s513
00:51:00.880 --> 00:51:07.200
But what they're also great at is lying to you because not because they're specifically trained for it.

s514
00:51:07.200 --> 00:51:17.119
But what happens is if there's a very thin line line between say two other lines that you ask a question about and it doesn't know that specific answer but it knows enough other information

s515
00:51:17.119 --> 00:51:29.440
around the answer and about other topics and related it can come up with a extremely convincing um you know response that unless you're an expert or really know the actual answer can can trick people.

s516
00:51:29.440 --> 00:51:38.800
So, all that [clears throat] to say, one of the things that's in this lab, it's kind of in the middle, and it's up on the screen right now, is the ability to have an LLM site sources.

s517
00:51:39.119 --> 00:51:46.000
Most models do not have the ability to site their sources because it's not a database.

s518
00:51:46.000 --> 00:51:50.240
It's it's these models are not doing a table lookup and pulling information.

s519
00:51:50.240 --> 00:51:52.880
It's generating tokens based on his training.

s520
00:51:52.880 --> 00:52:04.960
Anyways, so by providing the actual answers to the LM and having it parse it out and put together a human-friendly answer or whatever format you want it in, it can also then cite its sources.

s521
00:52:04.960 --> 00:52:12.880
This is extremely important in things like legal, which is you've probably seen in the news come up a lot of times where lawyers are getting in trouble because they're

s522
00:52:12.880 --> 00:52:19.119
having cases and then they're making up answers, citing cases that don't exist or twisting the answers.

s523
00:52:19.119 --> 00:52:28.559
So if you can have the model process you know large amounts of data complex data but still site the answer um it makes them a lot more useful and

s524
00:52:28.559 --> 00:52:32.880
trustworthy to a point.

s525
00:52:32.960 --> 00:52:48.480
So when you go into this as James mentioned um one of the nice things about the agent builder platform is that it is all also API driven.

s526
00:52:48.480 --> 00:52:56.480
So that's kind of why the the lab in this one was a little bit reversed to show that where you can build all this programmatically.

s527
00:52:56.480 --> 00:53:13.359
So you can build skills and tools and agents and you know through I built this all programmatically and that way you can create it, tear it down, scale it, change it as you need um but still use all the capabilities built into the stack and things like that.

s528
00:53:13.359 --> 00:53:22.319
Um so we'll go in here and then you can um ask a question you know we go into overview we have a lot of skills out of the box.

s529
00:53:22.319 --> 00:53:33.520
So these are the skills and I'm sure everyone here knows this because you're an AI conference right but a skill has a lot of different things but essentially it's explaining to

s530
00:53:33.520 --> 00:53:38.480
um you know a model or an agent how to perform a task in a certain way.

s531
00:53:38.480 --> 00:53:46.079
um when when tools should be used, how to create presentations, how to create other services.

s532
00:53:46.079 --> 00:53:48.240
Um but it's anything you can write down to kind of help it out.

s533
00:53:48.240 --> 00:53:51.920
So it doesn't have to just figure it out by itself.

s534
00:53:51.920 --> 00:53:54.240
Um some things is just ways to make it faster.

s535
00:53:54.240 --> 00:53:57.040
Some things is when you run very specific outcomes.

s536
00:53:57.040 --> 00:54:02.000
Um if you want to give it core instructions as as shown in here.

s537
00:54:02.000 --> 00:54:12.480
Um and so we provide a lot of out of the box things um skills for you like if you want to create graphs, visualizations, searching um but we also create tools.

s538
00:54:12.480 --> 00:54:16.000
So the difference then is a tool is performing an action.

s539
00:54:16.000 --> 00:54:23.359
So a tool that we pro you know provide here out of the box are things like running searches, listing workflows.

s540
00:54:23.359 --> 00:54:24.400
I talked about workflows.

s541
00:54:24.400 --> 00:54:34.640
There's the way to actually reach out and you know query external services or run things in a very specific um deterministic manner but you can create your own tools.

s542
00:54:34.640 --> 00:54:43.760
So if we go into the the agent that you created for example and you look at the overview um it's a very you know simple agent again for the

s543
00:54:43.760 --> 00:54:46.160
purposes of today it has one skill.

s544
00:54:46.160 --> 00:54:52.880
It talks about how to diagnose and and um an issue that you're looking for.

s545
00:54:52.880 --> 00:55:01.680
So, it's pretty pretty um pretty short here compared to the out of the box ones, but this is where you can provide as much information to to help it out.

s546
00:55:01.680 --> 00:55:07.520
So, it helps to find the clause, find the fix, talks about um looking at the error codes.

s547
00:55:07.520 --> 00:55:10.880
The other things that it can do is I think it has one tool.

s548
00:55:10.880 --> 00:55:11.599
Yeah.

s549
00:55:11.599 --> 00:55:20.720
So here we're um we have a search tool but if I add other tools just to show create a tool create a new tool.

s550
00:55:20.720 --> 00:55:25.200
Um [clears throat] this is always a fun question I like to ask.

s551
00:55:25.200 --> 00:55:30.640
Who here knows or has heard of ESQL within Elastic?

s552
00:55:32.640 --> 00:55:33.920
Okay,

s553
00:55:33.920 --> 00:55:34.640
we got one.

s554
00:55:34.640 --> 00:55:35.760
Nice.

s555
00:55:35.760 --> 00:55:39.119
It's about what I expect.

s556
00:55:39.119 --> 00:55:42.880
I was happy to hear though that a a lot of you are familiar with elastic in elastic search.

s557
00:55:42.880 --> 00:55:43.839
So that that's good.

s558
00:55:43.839 --> 00:55:49.520
Um ESQL is a newer u query language within elastic.

s559
00:55:49.520 --> 00:55:52.079
It's the ES pipeQL.

s560
00:55:52.079 --> 00:55:56.720
So it's a uh SQL near SQL like um language.

s561
00:55:56.720 --> 00:56:01.440
So you have a SQL like syntax and then essentially the pipe is in there to you pipe commands together.

s562
00:56:01.440 --> 00:56:05.119
You you run the thing, it passes it to the next pipe and so on and so on.

s563
00:56:05.119 --> 00:56:05.599
Um do that.

s564
00:56:05.599 --> 00:56:11.280
So you can have VSQL tools to query um you know documents in elastic search.

s565
00:56:11.280 --> 00:56:16.400
Index search is [clears throat] kind of an interesting tool or yeah tool.

s566
00:56:16.400 --> 00:56:18.640
So ESQL is you're running a specific query.

s567
00:56:18.640 --> 00:56:22.400
You're you're querying you write down the query you get back specific fields.

s568
00:56:22.400 --> 00:56:27.440
Index search is kind of this I almost think of it as a documentation tool.

s569
00:56:27.440 --> 00:56:33.520
So you're you're telling it for a specific index or set of indices this is what the index is for.

s570
00:56:33.520 --> 00:56:35.200
This is what the field mapping is.

s571
00:56:35.200 --> 00:56:36.480
These are what the fields are.

s572
00:56:36.480 --> 00:56:44.400
If you have say a product index and a pricing inventory index, it tells how to combine them.

s573
00:56:44.400 --> 00:56:47.920
Like with ESQL, you can join uh indices together.

s574
00:56:47.920 --> 00:56:53.440
So it talks about that and agents because they can do round trips and do exploration.

s575
00:56:53.440 --> 00:57:00.079
They will generally be able to figure out um you know the indexes in your in your index in your cluster.

s576
00:57:00.079 --> 00:57:07.680
So if you just ask a question and it doesn't know anything, it's going to do this query search tool and it's going to look at the mappings or it's a mapping tool.

s577
00:57:07.680 --> 00:57:09.680
It's going to find all the indices and then the mappings.

s578
00:57:09.680 --> 00:57:13.839
It's going to try to guess like what sounds like a product thing and then it's going to try to put them together.

s579
00:57:13.839 --> 00:57:21.200
Anyways, it'll roundroin for a while and that costs tokens and we all love token spending.

s580
00:57:21.200 --> 00:57:27.280
So this is a tool that helps speed up that dramatically and kind of help point the agent in the right direction.

s581
00:57:27.280 --> 00:57:28.160
Workflows.

s582
00:57:28.160 --> 00:57:34.400
I think I talked about this a few times, but we can run things in a uh orderly fashion and reach out to other systems.

s583
00:57:34.400 --> 00:57:39.200
And then MCP, if you want to call out to an external MCP server, you can [clears throat] do that.

s584
00:57:39.200 --> 00:57:48.720
I will mention at this point also that the tools in agent builder, both the out of the boxes one and custom tools are exposed as an MCP.

s585
00:57:48.720 --> 00:57:51.839
So, Kabana exposes an MCP uh server.

s586
00:57:51.839 --> 00:58:02.720
So any of the tools in there you can use you know if your agent has um wants to connect via MCP and the last one.

s587
00:58:02.720 --> 00:58:12.640
So you do this in the lab but this shows basically how the agent can then do multiple um multiple loops.

s588
00:58:12.640 --> 00:58:23.920
So it has to do first it loads the skill to see how to answer the question about um keeps dying with a 137 why does it happen and how so this is a two-part question right

s589
00:58:23.920 --> 00:58:38.799
why does this happen why is my cluster dying with you know the 137 uh exit code so it has to figure out the answer to that and then how do I change my settings to to prevent it so it's a two-part question

s590
00:58:38.799 --> 00:58:44.079
and again so runs that skill and then it starts to find the answer.

s591
00:58:44.079 --> 00:58:52.000
So goes a couple loops and it diagnoses your problem of you know tells you why this 137 is happening.

s592
00:58:52.000 --> 00:58:58.000
It's actually it's the o the fun out of memory manager running and killing you.

s593
00:58:58.000 --> 00:59:04.240
Um then it talks about how to uh tune your JVM options to to fix it but talks about preventing.

s594
00:59:04.240 --> 00:59:14.720
So because the answer within was within elastic and it had the ability to you know query it and then run multiple tools and things like this it's able to kind of get you that answer

s595
00:59:14.720 --> 00:59:18.640
and uh get you get you on your way.

s596
00:59:19.599 --> 00:59:19.839
Yeah.

s597
00:59:19.839 --> 00:59:32.160
And the one thing I'll add is that if you're if you're in active prototype mode and you're trying to to get to um search logic or query language um logic that that actually solves for your use case.

s598
00:59:32.160 --> 00:59:42.640
This is actually a a pretty nice developer tool and debugging tool to find out what um what searches will work to retrieve uh the data that you need.

s599
00:59:42.640 --> 00:59:47.280
And the last thing I'll say about uh I'm a big fan of ESQL.

s600
00:59:47.280 --> 00:59:53.599
I started out a little bit skeptical because there were some features in Elastic Query language that weren't in ESQL.

s601
00:59:53.599 --> 01:00:04.000
I think that it's feature complete enough uh for you to to at least take a look at it, especially if you're building a gentic search use cases because there's some really cool concepts in ESQL

s602
01:00:04.000 --> 01:00:15.920
uh that lend themselves well to um to solving for for certain search um tasks better than elastic query language, which you should consider a superset of what ESQL is today.

s603
01:00:15.920 --> 01:00:27.920
Um, and the other thing about ESQL that makes it fantastic is uh the ability to have parameterization wired into the MCP tool calling chain and and wired up with agent builder, which is

s604
01:00:27.920 --> 01:00:41.839
um a fantastic ad that technically is doable outside of ESQL, but the way that we have architected ESQL and our agent builder and our MCP endpoint, it is the happy path to

s605
01:00:41.839 --> 01:00:45.280
a lot of the um let's call them agentic search use cases.

s606
01:00:45.280 --> 01:00:46.079
that you might have.

s607
01:00:46.079 --> 01:00:50.240
So if you have not looked at ESQL yet, highly recommend you do that.

s608
01:00:50.240 --> 01:01:00.720
And if you were trying to pay attention to some of the innovative stuff that we're doing around search, first place to to look is actually going through the ESQL docs and seeing what new novel

s609
01:01:00.720 --> 01:01:11.920
um retrieval mechanisms we're coming up with because we're doing some very interesting things with federated search and potentially calling out the different systems within an elastic query language um

s610
01:01:11.920 --> 01:01:15.920
you know query that that is not considered traditional elastic.

s611
01:01:15.920 --> 01:01:21.520
Um, and that's stuff that's in development now, but you're going to see that hit hit the product.

s612
01:01:21.520 --> 01:01:26.559
And when it does, ESQL is going to be your gateway into that.

s613
01:01:26.880 --> 01:01:40.799
And I'm talking about things like being able to do like parquet queries on things that are external to Elastic with with an SQL call, which could be incredibly powerful if you're also joining that with data that comes from Elastic.

s614
01:01:42.880 --> 01:01:48.000
You can actually see that the the agent used ESQL for this uh this call.

s615
01:01:48.000 --> 01:01:49.520
So this is what it looks like.

s616
01:01:49.520 --> 01:01:58.960
Um so it can do things like fork off and then so this is where it's combining uh two of the two of the search threads.

s617
01:01:58.960 --> 01:02:01.040
That's what the fuse is down there.

s618
01:02:01.040 --> 01:02:03.839
And um so it's able to ask these questions and then return the answer.

s619
01:02:03.839 --> 01:02:15.680
So you can see it's you know there's the pipes that can um again you run a query you run a phase of it essentially get the answer and it moves on to the next and next pipe

s620
01:02:15.680 --> 01:02:16.400
language like that.

s621
01:02:16.400 --> 01:02:22.079
So it's um yeah it's very parameterizable is

s622
01:02:22.079 --> 01:02:31.359
parameterizable and it's it's trying to accomplish all the things that the Splunk query language tried to you know to do with with Splunk in its heyday.

s623
01:02:31.359 --> 01:02:46.319
uh and then um and then also marry that with SQL concepts which are obviously tried and true and I think that the uh the last thing that you'll see with ESQL is the queries are are more tur they don't have as much markup in them

s624
01:02:46.319 --> 01:02:57.040
uh to actually get to get the retrieval right which is one one added benefit to that piped uh construct and there are other other companies that that leverage the same technique in order to build out

s625
01:02:57.040 --> 01:03:07.280
um you know search usually it's for unstructured data sets logs Um, and you know, and I'd say that that was the primary reason we started building ESQL was to build,

s626
01:03:07.280 --> 01:03:13.359
you know, the best best-in-class search experience so people could get rid of all their Splunk stuff.

s627
01:03:13.359 --> 01:03:22.319
And and now we've pivoted to where this construct is so useful that it should be available in all the pillars of elastic, search, security, and observability.

s628
01:03:22.319 --> 01:03:31.920
It's a first class citizen construct for for for all three pillars going forward with a primary it being the primary mechanism for where we're going to innovate.

s629
01:03:33.359 --> 01:03:38.640
And so you can see when it's doing its one of the the hybrid search query, it [clears throat] responds.

s630
01:03:38.640 --> 01:03:43.119
It shows you so you have the you have the audit trail essentially of how it got to these answers.

s631
01:03:43.119 --> 01:03:53.280
So, not only citing the answers all the way at the bottom and you can click on them and if you want to deep dive into the fun world of JVM settings, you can do that.

s632
01:03:53.280 --> 01:03:55.440
Um, never doing that again.

s633
01:03:55.440 --> 01:03:57.760
But, um, just kidding.

s634
01:03:57.760 --> 01:03:59.520
One of the nice things too is then it'll show you.

s635
01:03:59.520 --> 01:04:02.640
So, we're limiting five and it it does five.

s636
01:04:02.640 --> 01:04:05.039
It tells you it found five results.

s637
01:04:05.039 --> 01:04:08.240
um doesn't clutter it all up in in the UI here for you.

s638
01:04:08.240 --> 01:04:13.039
But you can click on that and you can hop over to this is discover.

s639
01:04:13.039 --> 01:04:15.760
It's kind of where we discover documents.

s640
01:04:15.760 --> 01:04:19.359
It's the kind of normal searching around interface of Cabana.

s641
01:04:19.359 --> 01:04:24.160
It's changed a little bit over the years if you haven't used it in a little while, but now you can do ESQL in there.

s642
01:04:24.160 --> 01:04:28.400
And you can see here then it has the five documents that it came back with.

s643
01:04:28.400 --> 01:04:34.960
And if I close like some of these panels, builds a nice little chart for you.

s644
01:04:34.960 --> 01:04:42.079
I don't know if that's the most helpful chart, but you can see then with the query up at the top, you have the actual documents.

s645
01:04:42.079 --> 01:04:47.200
And go in and see the full thing.

s646
01:04:47.200 --> 01:04:49.359
And so here's the body of it.

s647
01:04:49.359 --> 01:04:54.079
And so we were searching part of this.

s648
01:04:54.079 --> 01:04:55.200
Oh, here it is.

s649
01:04:55.200 --> 01:04:56.000
Looking for the URL.

s650
01:04:56.000 --> 01:05:07.680
So the URL again, this is because we provide the URL as part of the document, we're able to, you know, provide essentially a link to more information without having to just guess.

s651
01:05:09.680 --> 01:05:12.480
I think this room closes out at the 420 mark.

s652
01:05:12.480 --> 01:05:13.920
Is that correct?

s653
01:05:13.920 --> 01:05:14.319
Yeah.

s654
01:05:14.319 --> 01:05:16.720
Timer says 18 minutes till

s655
01:05:16.720 --> 01:05:18.160
18 minutes, which means you have

s656
01:05:18.160 --> 01:05:19.680
Let the hammer come and hit us.

s657
01:05:19.680 --> 01:05:19.920
Yeah.

s658
01:05:19.920 --> 01:05:23.280
Which means you you do have a little bit of time to ask questions.

s659
01:05:23.280 --> 01:05:23.440
Yeah.

s660
01:05:23.440 --> 01:05:25.359
And there is the Yeah, feel free to ask questions.

s661
01:05:25.359 --> 01:05:25.839
Sorry.

s662
01:05:25.839 --> 01:05:34.400
Um there is the lab five about um rerankers.

s663
01:05:34.880 --> 01:05:37.359
Lights are like burning my brain.

s664
01:05:37.359 --> 01:05:47.599
Um so reranking in there is is an interesting um thing that I think a lot of people don't kind of fully understand why you want to use it.

s665
01:05:47.599 --> 01:05:54.079
And so that's why we added this on um to kind of explain why you want to use it.

s666
01:05:54.079 --> 01:06:04.000
And if this spins, see if that comes up.

s667
01:06:04.000 --> 01:06:05.680
Maybe it's my thing.

s668
01:06:05.680 --> 01:06:10.000
Reranking is taking the So if if

s669
01:06:10.000 --> 01:06:10.079
[snorts]

s670
01:06:10.079 --> 01:06:16.240
the queries that we've been doing here and everything is the first phase, call it of of retrieval.

s671
01:06:16.240 --> 01:06:19.359
The second phase you can do retrie uh reranking.

s672
01:06:19.359 --> 01:06:31.599
And so if you're running vector search, semantic search, hybrid search across a first phase and get back a large set of data, then you can take that data and you can

s673
01:06:31.599 --> 01:06:32.880
rerank it.

s674
01:06:32.880 --> 01:06:37.280
And reranking has I wish this thing was working.

s675
01:06:37.280 --> 01:06:39.760
I think it's me.

s676
01:06:42.880 --> 01:06:51.920
re-ranking combines the the query and the the document as scoring time together.

s677
01:06:51.920 --> 01:07:03.039
So normally with with vectors you you know you ingest all your data whenever that is and then at query time you query it and it just does um HSW

s678
01:07:03.039 --> 01:07:11.359
or or one of the disk algorithms and returns a set of documents but the the query and the the documents aren't really scored together.

s679
01:07:11.359 --> 01:07:20.880
Reranking combines those together, but because they're running together at at that query that phase second phase time, you can't pre-mbed.

s680
01:07:20.880 --> 01:07:28.960
So you pay the penalty of having to run inference at, you know, at query time.

s681
01:07:28.960 --> 01:07:36.640
Now the way you do this then to kind of help this along is by not reranking your entire corpus of data.

s682
01:07:36.640 --> 01:07:46.960
You run the first phase to get a smaller set of data and then rerank with a model to then you know rerank that that small set of data.

s683
01:07:47.680 --> 01:07:50.480
Sorry, there we go.

s684
01:07:53.039 --> 01:07:58.720
Trying to show an example instead of just talk.

s685
01:08:03.920 --> 01:08:06.240
Here we go.

s686
01:08:08.000 --> 01:08:10.720
My lab was dying.

s687
01:08:18.719 --> 01:08:30.560
So if you get to this in here, you can see toward at the end um again kind of the points on here about when you can add a ranker if you have that that the latency budget.

s688
01:08:30.560 --> 01:08:45.440
If you don't if you can spend a a little bit of extra time to let this second phase run and generating embeddings because that precision of the you know if it's answer one versus answer two if that is a critical step that's where paying

s689
01:08:45.440 --> 01:08:51.440
you know the the slight um latency penalty comes into play.

s690
01:08:51.440 --> 01:08:58.719
If you just get your documents good enough in the first phase and let the agent go after them, then you probably don't need a reranker.

s691
01:08:58.719 --> 01:09:10.159
But it's when you need to have that or that specific order come into play um come in and um and we see that a lot with agentic search use cases.

s692
01:09:10.159 --> 01:09:21.199
We're seeing a lot of situations where folks are testing the evaluation based on the first, second, and third document chunk coming back, which gives you a very small window of opportunity to get the answer right.

s693
01:09:21.199 --> 01:09:31.839
If you're just handing a small set of candidates to the to the large language model to answer the question versus a larger candidate set and then and then asking the model to get it right.

s694
01:09:31.839 --> 01:09:34.560
And and that's where a lot of relevancy issues come in.

s695
01:09:34.560 --> 01:09:36.880
You're passing in too much context.

s696
01:09:36.880 --> 01:09:47.120
It's a multiple choice where two or three answers look equally right to the LLM and it just picks one and and if it picks the wrong one then your your users

s697
01:09:47.120 --> 01:09:52.000
and their beta testers for your new agentic search system complain about the search not working.

s698
01:09:52.000 --> 01:09:54.320
Very common, very very common.

s699
01:09:54.320 --> 01:10:01.840
And um you know oftentimes a reranker like this will will actually be very very useful.

s700
01:10:01.840 --> 01:10:08.000
And so there's two points or two types of um reankers at least you know that we support.

s701
01:10:08.000 --> 01:10:14.239
So there's point-wise these are what most reankers are today and this is what that the V2 model is.

s702
01:10:14.239 --> 01:10:24.239
And so the point-wise it it still combines the it takes the query and the document together but it just scores them independently.

s703
01:10:24.239 --> 01:10:29.360
It's only looking at the the query and the document, nothing else.

s704
01:10:29.360 --> 01:10:41.040
Listwise, which is what is in the Gina V3 model, takes your candidate set and I think it right now, for example, up to 64 documents or 64 candidates and it

s705
01:10:41.040 --> 01:10:42.960
looks at them all together.

s706
01:10:42.960 --> 01:10:53.760
And so that way way if the if the answers essentially or the the potential candidates are all very similar like they're all pretty much talking about the same thing.

s707
01:10:53.760 --> 01:11:06.480
That's where having the listwise can edge out pointwise because it can see the nuance see it can see the nuances um within that list because it sees the multiple documents together.

s708
01:11:06.480 --> 01:11:19.280
Um it does take a little bit of extra u memory or or time because again it has to generate embeddings for all those candidates all at once and hold them together in memory and run

s709
01:11:19.280 --> 01:11:26.400
um run its evaluation across them as opposed to just firing one at a time and do that.

s710
01:11:28.000 --> 01:11:39.600
So if your if your data is pretty diverse and where the answers come back and it's you know it's a pretty distinct set of candidates where you know one two three four five are all pretty

s711
01:11:39.600 --> 01:11:44.080
different enough that the pointwise is hitting it.

s712
01:11:44.960 --> 01:11:46.719
Sorry.

s713
01:11:46.719 --> 01:11:48.400
There we go.

s714
01:11:48.400 --> 01:11:52.640
Um then point wise usually pretty much usually works out.

s715
01:11:52.640 --> 01:12:02.080
But again, if you have that set where the the potential candidates are all extremely similar, listwise is probably going to help you out for the best.

s716
01:12:08.560 --> 01:12:13.920
and somewhere in here because we want to be helpful.

s717
01:12:13.920 --> 01:12:26.480
You don't have to run um while it is a two step a two-phase um you know phase one is query phase two is your ranking you don't have to manually

s718
01:12:26.480 --> 01:12:35.520
um have those as separate separate API calls we can do it all in one shot um you do it in the lab it's in the retriever you can you can do it

s719
01:12:35.520 --> 01:12:37.440
can you rerank any SQL

s720
01:12:37.440 --> 01:12:37.840
yes

s721
01:12:37.840 --> 01:12:38.420
yeah okay

s722
01:12:38.420 --> 01:12:39.760
[laughter]

s723
01:12:39.760 --> 01:12:51.040
um so you can do it in SQL too but one dot basically you have the reranker you tell what reranker you want to use you know the model and the query and it runs both phases for you so that way again

s724
01:12:51.040 --> 01:13:02.960
you're not having to manage the inference setting separately you're not having to call the inference API then query then rerank you can just fire it all in one shot and we'll take care of it for you

s725
01:13:06.880 --> 01:13:12.960
so I'm going to put up that slide again with few of the links again.

s726
01:13:12.960 --> 01:13:16.320
So the upper right is what we've been doing today.

s727
01:13:16.320 --> 01:13:17.920
That's available for a week.

s728
01:13:17.920 --> 01:13:24.880
The bottom blue reference repo is where the the code for this lives, the notebooks and things like that.

s729
01:13:24.880 --> 01:13:27.440
And then there's the two red ones in the middle.

s730
01:13:27.440 --> 01:13:33.040
Um Sunnil's secret elastic environment.

s731
01:13:33.600 --> 01:13:35.600
Oh man, typo it too.

s732
01:13:35.600 --> 01:13:37.840
Um everyone's gonna take a picture of it.

s733
01:13:37.840 --> 01:13:41.199
So, and then the other god

s734
01:13:41.199 --> 01:13:42.800
just proves that we're in a live session.

s735
01:13:42.800 --> 01:13:43.679
I'm gonna blame the lights

s736
01:13:43.679 --> 01:13:48.000
and that we're just, you know, we're doing stuff on the fly.

s737
01:13:49.199 --> 01:13:51.679
Still got to run.

s738
01:13:51.679 --> 01:13:53.040
Okay.

s739
01:13:53.040 --> 01:13:55.679
Um, and then the other workshops online.

s740
01:13:55.679 --> 01:14:00.960
So, we do have a bunch of other workshops that online that are completely free.

s741
01:14:00.960 --> 01:14:12.239
And so things like working with streams, if you have streams of data coming in, it's extremely um efficient way to handle that data.

s742
01:14:12.239 --> 01:14:17.360
Another rag one, lexical search, vector search tutorials.

s743
01:14:17.520 --> 01:14:27.920
If you have anyone's doing uh observability work, so things with open telemetry data or looking for root cause analysis, things like that, we have a labs on that.

s744
01:14:27.920 --> 01:14:31.040
So, a couple observability labs.

s745
01:14:31.040 --> 01:14:33.280
Should be some security ones in here.

s746
01:14:33.280 --> 01:14:37.440
So, security is a big platform, our big part of our platform.

s747
01:14:37.440 --> 01:14:41.360
Um, doing anything from like attack discovery and detection rules.

s748
01:14:41.360 --> 01:14:44.080
I know there's new ones coming out too.

s749
01:14:44.080 --> 01:14:52.560
Um, as we add in more and more agent capability into the stack.

s750
01:14:56.400 --> 01:14:58.719
Any questions?

s751
01:14:59.280 --> 01:15:00.560
All right.

s752
01:15:00.560 --> 01:15:04.159
So, let me ask a question real quick.

s753
01:15:04.159 --> 01:15:09.520
If you hang out in here, do you automatically get a seat in the next workshop?

s754
01:15:10.080 --> 01:15:12.000
I'm just curious if that's why people hang out.

s755
01:15:12.000 --> 01:15:12.400
Okay.

s756
01:15:12.400 --> 01:15:17.199
I'm trying to judge who's still working and who's just hanging out, which is totally fine.

s757
01:15:17.199 --> 01:15:21.760
I Yeah, I get to leave either way.

s758
01:15:21.760 --> 01:15:27.360
We're going to assume that most of you all are hanging out and that you're not, you know, waiting for help on anything.

s759
01:15:27.360 --> 01:15:34.080
But if if you are, you've got another 7 minutes and 57 seconds.

s760
01:15:34.960 --> 01:15:38.640
Sil, you want to demo anything for seven minutes?

s761
01:15:40.719 --> 01:15:44.320
Hopefully this was a good use of your time for the folks that are still in the room.

s762
01:15:44.320 --> 01:15:46.159
And we'd love to get your feedback.

s763
01:15:46.159 --> 01:15:51.120
We didn't tell you this, but there is a feedback um data entry point at the end of the workshop.

s764
01:15:51.120 --> 01:15:56.320
So, when you finish it, you do have the ability to tell us how awesome the workshop is or or how much you didn't like it.

s765
01:15:56.320 --> 01:15:59.040
And if and if you say bad things, then it will hurt our feelings.

s766
01:15:59.040 --> 01:16:03.440
But but we will make the workshop better for others after the fact.