WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=aV_vgUSatqs

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/aV_vgUSatqs.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.800 --> 00:00:16.160
Hi everyone, welcome in.

s2
00:00:16.160 --> 00:00:17.520
My name is Mark Lumus.

s3
00:00:17.520 --> 00:00:20.000
I'm head of developer tools at PayPal.

s4
00:00:20.000 --> 00:00:22.480
With me I have Naveen Kumar.

s5
00:00:22.480 --> 00:00:26.240
He's our tech lead for the developer tools team.

s6
00:00:26.240 --> 00:00:30.400
And today we're going to talk to you a little bit about CLI.

s7
00:00:30.800 --> 00:00:40.559
The title of our presentation is meant to be a little provocative and you'll understand why at the end of the presentation.

s8
00:00:43.120 --> 00:01:00.800
So today we're going to talk a little bit about a specific kind of CLI, interactive CLI, and we're going to set up an exercise and we're going to give you for those of you who wish to

s9
00:01:00.800 --> 00:01:04.559
participate, this is a workshop after all.

s10
00:01:04.559 --> 00:01:10.799
We'll give you a CLI that you can make work better in an Agentic workflow.

s11
00:01:10.799 --> 00:01:14.320
So that's the purpose of this workshop today.

s12
00:01:14.320 --> 00:01:15.680
So we're going to give you plenty of time.

s13
00:01:15.680 --> 00:01:16.880
We'll talk for a few minutes.

s14
00:01:16.880 --> 00:01:21.840
Try to keep the talking to a minimum and then we'll get you started.

s15
00:01:24.479 --> 00:01:26.080
Now it's kind of interesting.

s16
00:01:26.080 --> 00:01:29.600
I think someone said what's old is new.

s17
00:01:29.600 --> 00:01:34.320
When I started my career long time ago, I have gray hair.

s18
00:01:34.320 --> 00:01:36.560
Everything was a CLI.

s19
00:01:36.560 --> 00:01:39.119
Everything was a command line interface.

s20
00:01:39.119 --> 00:01:45.119
Any of you remember working with MS DOSs before we had Microsoft Windows?

s21
00:01:45.119 --> 00:01:46.640
There you go.

s22
00:01:46.640 --> 00:01:51.280
My first job out of college, I was designing embedded systems.

s23
00:01:51.280 --> 00:02:02.719
And so you would connect to a computer over a serial port and you would pull up a VT100 compatible terminal and you would start typing.

s24
00:02:02.719 --> 00:02:04.399
And that was your CLI.

s25
00:02:04.399 --> 00:02:08.640
That was the the the way you interfaced with the computer.

s26
00:02:08.640 --> 00:02:20.319
So with today CLIs have kind of had a resurgence I guess you know with the maybe it was clawed code that really made them much more popular and now we have codecs

s27
00:02:20.319 --> 00:02:32.800
um and so a lot of people are are adding CLI uh tools so that it's easier to access the capabilities that they want to expose to an agent.

s28
00:02:32.800 --> 00:02:40.720
Now, with interactive CLI, this is why we're here today with what we want to talk about.

s29
00:02:40.720 --> 00:02:47.599
Um, agents shell out to Bash constantly, and that's good.

s30
00:02:47.599 --> 00:02:57.200
But if they run an interactive CLI, that may be bad because not all harnesses handle interaction very well.

s31
00:02:57.200 --> 00:03:00.560
And if they don't handle it very well, they'll fail quietly.

s32
00:03:00.560 --> 00:03:01.360
You won't know.

s33
00:03:01.360 --> 00:03:04.239
No errors, no nothing.

s34
00:03:05.280 --> 00:03:07.840
But why do we care?

s35
00:03:07.840 --> 00:03:19.440
Interactive CLIs are great because the next question can emerge based on what way you answer a previous question.

s36
00:03:19.440 --> 00:03:27.200
Flags multiply if you try to create flags that predict all the possible flows.

s37
00:03:27.280 --> 00:03:34.239
The choices are contextual and they can also be responsive to what's happening in an environment.

s38
00:03:34.239 --> 00:03:44.879
So if your CLI is calling an API and getting a response and then the question comes up based on what the API returns, that's really really challenging to handle

s39
00:03:44.879 --> 00:03:52.959
with flags and the guard rails can live in the interactive flow.

s40
00:03:55.920 --> 00:04:01.040
So here's an example over here on the on the left.

s41
00:04:01.040 --> 00:04:03.680
We see a a sample CLI.

s42
00:04:03.680 --> 00:04:05.840
It's interactive.

s43
00:04:05.840 --> 00:04:12.159
The CLI's opened an interactive prompt, an error key prompt.

s44
00:04:12.159 --> 00:04:13.920
The human sees it.

s45
00:04:13.920 --> 00:04:19.680
The human can use the arrow keys, but the agent doesn't see it.

s46
00:04:19.680 --> 00:04:23.759
The agent sees a redraw loop, not a question.

s47
00:04:23.759 --> 00:04:28.400
and the harness loses the UI state.

s48
00:04:28.400 --> 00:04:29.759
So here's the failure mode.

s49
00:04:29.759 --> 00:04:33.440
It's silent and invisible.

s50
00:04:35.280 --> 00:04:39.680
So the common fix will just add flags.

s51
00:04:39.680 --> 00:04:44.400
And flags are fine for simple use cases.

s52
00:04:44.400 --> 00:04:51.199
You have uh you know what the you know what the flows are up front.

s53
00:04:51.199 --> 00:04:53.520
It's a single pass.

s54
00:04:53.520 --> 00:04:55.759
All the options are independent.

s55
00:04:55.759 --> 00:05:02.000
Great great use case for a a a simple flag.

s56
00:05:02.400 --> 00:05:13.600
But when the answers change based on the context or they're changing based on reactions to real time events, that's when it becomes a problem.

s57
00:05:13.600 --> 00:05:17.759
You really can't use flags for those situations.

s58
00:05:19.039 --> 00:05:24.880
Now, an agent is not like an automation script.

s59
00:05:24.880 --> 00:05:28.160
An automation script knows nothing.

s60
00:05:28.160 --> 00:05:32.000
All of the decisions are pre-baked up front.

s61
00:05:32.000 --> 00:05:34.560
The agent can reason.

s62
00:05:34.560 --> 00:05:36.479
It can react.

s63
00:05:36.479 --> 00:05:37.680
It can ask.

s64
00:05:37.680 --> 00:05:38.720
It can retry.

s65
00:05:38.720 --> 00:05:40.639
It can back up.

s66
00:05:40.639 --> 00:05:48.560
And most importantly, an agent can walk a decision tree if you provide provide one.

s67
00:05:48.560 --> 00:05:51.520
You can't do that with flags.

s68
00:05:53.919 --> 00:05:58.720
So we already have a model today for what's happening here.

s69
00:05:58.720 --> 00:06:01.199
It's called accessibility.

s70
00:06:01.280 --> 00:06:12.160
And so if you map what we do for humans to what we need to do for agents, then the solutions become apparent.

s71
00:06:12.160 --> 00:06:16.720
It's the same discipline with a new reader.

s72
00:06:19.360 --> 00:06:23.039
So here's the here's the challenge.

s73
00:06:23.039 --> 00:06:33.440
How might we keep the guard rails and the guidance and stop pretending that agents and humans have the same abilities?

s74
00:06:33.440 --> 00:06:34.880
Now, I'm not going to answer this question.

s75
00:06:34.880 --> 00:06:38.479
That's the whole purpose of this workshop.

s76
00:06:40.240 --> 00:06:42.319
So, here's the challenge.

s77
00:06:42.319 --> 00:06:44.240
We've got a CLI.

s78
00:06:44.240 --> 00:06:47.199
We call it Bris Barista 9000.

s79
00:06:47.199 --> 00:07:00.400
It's going to take coffee orders and it's going to fail when you try to run it from an agent because it's interactive.

s80
00:07:03.280 --> 00:07:11.919
And if you choose to participate in the challenge, the winner will win a Mac Mini.

s81
00:07:12.000 --> 00:07:15.280
Walk walk away with it today.

s82
00:07:15.919 --> 00:07:17.544
It's in that bag right there.

s83
00:07:17.544 --> 00:07:19.544
[laughter]

s84
00:07:20.639 --> 00:07:23.680
That was a challenge in and of itself getting a Mac Mini.

s85
00:07:23.680 --> 00:07:25.520
I'll be straight up.

s86
00:07:25.520 --> 00:07:26.720
Okay.

s87
00:07:26.720 --> 00:07:30.720
Now, [clears throat] simple rules.

s88
00:07:30.720 --> 00:07:34.479
We're going to give you a CLI.

s89
00:07:35.440 --> 00:07:44.720
You make changes to that CLI, run it through an agent, and it's going to send some information up to our server.

s90
00:07:44.720 --> 00:07:47.599
And then it's going to update a leaderboard.

s91
00:07:47.599 --> 00:07:50.080
And we'll put put the leaderboard up in a moment.

s92
00:07:50.080 --> 00:07:57.759
We don't have two inputs, so we're going to change the slide deck out for the leaderboard so you can see it.

s93
00:07:57.759 --> 00:08:00.240
Now, here's Naveen.

s94
00:08:00.400 --> 00:08:01.919
He's going to talk you through this.

s95
00:08:01.919 --> 00:08:02.560
Thank you, Mark.

s96
00:08:02.560 --> 00:08:04.639
All right.

s97
00:08:05.680 --> 00:08:07.680
Cool.

s98
00:08:07.759 --> 00:08:08.160
Yeah.

s99
00:08:08.160 --> 00:08:21.735
Uh I think it's an exciting assignment for everybody and this topic is very near and dear to me but uh keeping things aside Mark are we getting any coffee after we build this

s100
00:08:21.735 --> 00:08:22.639
[laughter]

s101
00:08:22.639 --> 00:08:25.199
I hope so if we're successful

s102
00:08:25.199 --> 00:08:40.080
okay cool so yeah let me uh before we start into um the assignment I want to take to a journey right uh when I started my engineering career I started to play with ids.

s103
00:08:40.080 --> 00:08:45.600
I started to use Eclipse because I was primarily using Java.

s104
00:08:45.600 --> 00:08:51.760
And as time went by, moved into Intelligj uh Jet Brains.

s105
00:08:51.760 --> 00:08:54.000
It was amazing.

s106
00:08:54.000 --> 00:08:58.720
Looking back, five or six years behind uh things started to evolve.

s107
00:08:58.720 --> 00:09:09.680
there were new ids in the market especially Microsoft VS code which gave birth to a lot of u other ids as well.

s108
00:09:11.120 --> 00:09:25.200
So if you look right now Microsoft had its own VS code editor but later on new AI id started to pop out like cursor windsurf to name a few.

s109
00:09:25.200 --> 00:09:30.080
In all these journeys, CLIs were constant, right?

s110
00:09:30.080 --> 00:09:35.600
At some point, I thought, okay, you know, CLIs are having a slow death of their own.

s111
00:09:35.600 --> 00:09:38.560
Maybe they are not going to come back.

s112
00:09:38.560 --> 00:09:50.640
But, you know, uh I I I actually like to see this hangover movie dialogue that um you know, all all the four friends are in the car and they say, "Hey, we are back."

s113
00:09:50.640 --> 00:09:53.519
Imagine CLI seeing that to each other.

s114
00:09:53.519 --> 00:09:57.519
And in the world of AI, they are actually VC, right?

s115
00:09:57.519 --> 00:10:00.080
Whoever bought it, yeah, I don't know.

s116
00:10:00.080 --> 00:10:04.880
But the whole story is now the CLI have changed.

s117
00:10:04.880 --> 00:10:07.040
And who is responsible for that?

s118
00:10:07.040 --> 00:10:08.959
Agents, right?

s119
00:10:08.959 --> 00:10:16.320
So this workshop is primarily focusing on agents, your CLIs.

s120
00:10:16.320 --> 00:10:19.920
And this is going to be a hands-on exercise.

s121
00:10:19.920 --> 00:10:22.000
There are rules to it.

s122
00:10:22.000 --> 00:10:24.240
There are no rules to it.

s123
00:10:24.240 --> 00:10:30.480
So feel free to hack as much as possible and you know the prize is here.

s124
00:10:30.480 --> 00:10:36.640
Um so yeah we have maybe uh ven if I can get the laptop.

s125
00:10:36.640 --> 00:10:39.360
So we have couple of things.

s126
00:10:39.360 --> 00:10:45.440
So we have yeah um so we have the repo which is PayPal cafe.

s127
00:10:45.440 --> 00:10:49.200
It has the CLI that you can actually download.

s128
00:10:49.200 --> 00:10:51.200
That's the barebone CLI.

s129
00:10:51.200 --> 00:10:54.399
Now, it's not going to be that easy uh for a Mac Mini.

s130
00:10:54.399 --> 00:10:55.279
Yeah.

s131
00:10:55.279 --> 00:10:59.279
Uh so, you will get some initial commands with it.

s132
00:10:59.279 --> 00:11:00.640
Uh play around with it.

s133
00:11:00.640 --> 00:11:02.240
Register yourself.

s134
00:11:02.240 --> 00:11:04.880
Uh provide your username, whatever you like.

s135
00:11:04.880 --> 00:11:11.279
Um your nicknames, anything, and try to brew a coffee out of it.

s136
00:11:11.279 --> 00:11:13.440
Okay, that's that's the goal.

s137
00:11:13.440 --> 00:11:25.760
And in this whole exercise, we want you to also be innovative in your approaches and going forward, it's going to get interesting with agents using the CLI.

s138
00:11:25.760 --> 00:11:29.200
So think it from that perspective.

s139
00:11:29.600 --> 00:11:31.360
I change this.

s140
00:11:31.360 --> 00:11:35.600
I just change my laptops here.

s141
00:11:36.399 --> 00:11:43.279
Now during the during the session, we're we're not starting you out with any hints.

s142
00:11:43.279 --> 00:12:01.519
Once we feel like enough people have gotten the the repo installed, can run the the CLI as is, then maybe every 15 or 20 minutes, we'll share a good idea.

s143
00:12:01.519 --> 00:12:04.720
Now, I'm not going to call it a best practice.

s144
00:12:04.720 --> 00:12:14.160
I'm just going to call it a good idea because I feel like we're early enough in the journey here where we don't know what the best practices are.

s145
00:12:14.160 --> 00:12:17.279
We're kind of feeling them out now.

s146
00:12:17.279 --> 00:12:19.040
They're good ideas.

s147
00:12:19.040 --> 00:12:28.160
I'm also hoping that at the end we'll hear some great ideas from you in the audience so we can share.

s148
00:12:28.160 --> 00:12:30.480
So just keep that in mind.

s149
00:12:30.480 --> 00:12:32.320
We're we're all learning.

s150
00:12:32.320 --> 00:12:34.800
This is a journey.

s151
00:12:34.880 --> 00:12:38.639
We'll come back to that towards the end.

s152
00:12:46.720 --> 00:12:49.360
Is it not working?

s153
00:12:52.959 --> 00:12:54.160
Give us a couple of minutes.

s154
00:12:54.160 --> 00:12:58.399
I think we'll get the technical issues resolved.

s155
00:13:13.279 --> 00:13:16.480
seems like people are actually using it.

s156
00:13:16.480 --> 00:13:28.000
So on the right side you will actually see get CLI and um if you see PayPal cafe/ rules that's where you will have all the rules outlined to it.

s157
00:13:28.000 --> 00:13:33.600
Feel free to start working on the CLI till we figure it out.

s158
00:13:34.880 --> 00:13:36.880
Okay.

s159
00:13:53.519 --> 00:13:56.240
Do we have the Europe?

s160
00:13:57.680 --> 00:14:00.560
Oh, is it that one?

s161
00:14:13.199 --> 00:14:13.519
Yeah.

s162
00:14:13.519 --> 00:14:14.720
Hey, there you go.

s163
00:14:14.720 --> 00:14:16.720
Nice.

s164
00:14:20.480 --> 00:14:20.880
Awesome.

s165
00:14:20.880 --> 00:14:21.360
Thank you.

s166
00:14:21.360 --> 00:14:22.480
Thank you so much.

s167
00:14:22.480 --> 00:14:28.399
Hey Cool.

s168
00:14:28.399 --> 00:14:34.160
So you can see this is Oh, we do see some signings.

s169
00:14:34.160 --> 00:14:34.399
Okay.

s170
00:14:34.399 --> 00:14:36.880
So on the right side you will see get CLI.

s171
00:14:36.880 --> 00:14:40.240
You can download that and I hope you guys can see it.

s172
00:14:40.240 --> 00:14:42.320
It's at the bottom right.

s173
00:14:42.320 --> 00:14:44.160
And there are some rules here.

s174
00:14:44.160 --> 00:14:49.519
You can click on these rules and it will pop out some rules.

s175
00:15:00.000 --> 00:15:04.399
So, we'll probably take around about 25 to 30 minutes.

s176
00:15:04.399 --> 00:15:09.805
Um, and in between we'll interrupt with more slides and make it interesting.

s177
00:15:09.805 --> 00:15:11.805
[laughter]

s178
00:15:12.000 --> 00:15:14.880
Where's the laptop for the other slides?

s179
00:15:14.880 --> 00:15:15.760
Venode's got it.

s180
00:15:15.760 --> 00:15:18.399
Okay, that's fine.

s181
00:15:20.959 --> 00:15:22.880
and we're here to help.

s182
00:15:22.880 --> 00:15:25.120
If you're having trouble, please raise your hand.

s183
00:15:25.120 --> 00:15:29.120
We'll try to assist you as best we can.

s184
00:15:52.160 --> 00:15:55.040
a little bit about the models.

s185
00:15:56.880 --> 00:15:59.759
Don't use models.

s186
00:16:10.480 --> 00:16:13.759
Now, we had a little bit of a debate internally.

s187
00:16:13.759 --> 00:16:16.959
There are some great models out there.

s188
00:16:19.040 --> 00:16:33.519
The interesting thing about this challenge, if you use the highest thinking mode on the bleeding edge LLMs, you can probably brute way brute force your way through this.

s189
00:16:33.519 --> 00:16:40.800
But I feel like in the current climate, teams are starting to think about their token budget.

s190
00:16:40.800 --> 00:16:51.040
And so you don't necessarily want to waste your highinking tokens on running a CLI tool.

s191
00:16:51.040 --> 00:17:03.040
So the real challenge is can you make Haik coup or sonnet or something at that level run your CLI?

s192
00:17:03.040 --> 00:17:05.199
We know you can do it with the highest.

s193
00:17:05.199 --> 00:17:10.880
Can you do it with something that's somewhere in the middle that's faster and cheaper?

s194
00:17:10.880 --> 00:17:16.160
That's the That's the interesting challenge if you if you want to take it on.

s195
00:17:16.160 --> 00:17:16.880
Sounds fun.

s196
00:17:16.880 --> 00:17:18.880
Yeah.

s197
00:17:20.959 --> 00:17:22.959
Awesome.

s198
00:17:56.880 --> 00:18:08.400
Since we are on the coffee topic, uh Mark, were you aware that San Francisco City has the best bakeries in the entire United States?

s199
00:18:08.400 --> 00:18:09.760
Were you aware about that fact?

s200
00:18:09.760 --> 00:18:10.960
I was not aware of that.

s201
00:18:10.960 --> 00:18:11.840
Yeah, you should definitely.

s202
00:18:11.840 --> 00:18:12.320
Go on.

s203
00:18:12.320 --> 00:18:12.559
Yeah.

s204
00:18:12.559 --> 00:18:14.000
Go on, tell me more.

s205
00:18:14.000 --> 00:18:20.160
Um, I have I've done some bakery crawls uh around in the city and they are so fun.

s206
00:18:20.160 --> 00:18:25.039
Like you should definitely check out all the amazing bakeries out here.

s207
00:18:25.039 --> 00:18:25.679
Okay.

s208
00:18:25.679 --> 00:18:26.960
Yeah.

s209
00:18:26.960 --> 00:18:28.480
Any favorites?

s210
00:18:28.480 --> 00:18:30.720
Butter and crumble.

s211
00:18:30.720 --> 00:18:31.760
My favorite one.

s212
00:18:31.760 --> 00:18:33.512
Although I couldn't go there.

s213
00:18:33.512 --> 00:18:35.512
[laughter]

s214
00:18:38.320 --> 00:18:40.320
Awesome.

s215
00:18:41.120 --> 00:18:47.919
So as of now we have J Gray Codex at the top.

s216
00:18:47.919 --> 00:21:08.000
So guys please catch up with him or her So with the init command, you could actually register your names.

s217
00:21:08.000 --> 00:21:12.080
I hope some of you have figured that out.

s218
00:22:06.400 --> 00:24:51.039
Do you want to talk about the first You might have to think about Once I plug it in, I

s219
00:25:44.507 --> 00:25:46.507
[sighs]

s220
00:25:51.919 --> 00:25:52.559
Go ahead.

s221
00:25:52.559 --> 00:25:53.600
Okay,

s222
00:25:53.600 --> 00:25:54.400
cool.

s223
00:25:54.400 --> 00:25:57.440
Um, let's move on to some presentations.

s224
00:25:57.440 --> 00:25:58.880
Meanwhile, you can start working.

s225
00:25:58.880 --> 00:26:03.600
I mean, you'll continue working with the CLI that you have.

s226
00:26:03.600 --> 00:26:09.440
But as as a first hint, you know, it's it's a good idea to start using T-Max.

s227
00:26:09.440 --> 00:26:12.159
How many of you know about T-Max here?

s228
00:26:12.159 --> 00:26:15.440
Okay, quite a fair amount of audience out here.

s229
00:26:15.440 --> 00:26:25.679
Um, so yeah u in short start with running T-Max and um start by putting help command.

s230
00:26:25.679 --> 00:26:38.218
Now, help is really important because every AI agent in today's world relies on the help command before it even starts using that command.

s231
00:26:38.218 --> 00:26:38.880
[laughter]

s232
00:26:38.880 --> 00:26:49.840
So, yeah, send keys and capture pain gets you far.

s233
00:26:49.840 --> 00:27:04.240
Now, T-Max actually um for those who actually don't know about it is like a multiple you could actually use multiple terminal sessions uh with the T-Max and it's like a split pane

s234
00:27:04.240 --> 00:27:07.760
feature that you could leverage with T-Max.

s235
00:27:07.840 --> 00:27:12.720
So, with this example, you can see here um I mean the screen is very short for me.

s236
00:27:12.720 --> 00:27:15.120
So, let me just zoom in a little bit.

s237
00:27:15.120 --> 00:27:28.653
Yeah, T-Max D. Now, what this flag does is it detaches the entire uh CLI and puts it under a different emox instance all together for

s238
00:27:28.653 --> 00:27:28.720
[laughter]

s239
00:27:28.720 --> 00:27:30.720
you.

s240
00:27:32.720 --> 00:27:44.559
So you could see size uh T-max capture pane like I said you could actually have split panes in T-maxes.

s241
00:27:44.559 --> 00:27:57.279
So you could actually take care of u working around with um in in the CLI and uh that's one of those cool features that can help your agents perform lot of activities.

s242
00:27:57.279 --> 00:28:02.960
Um, think of it when you want to run multiple agents.

s243
00:28:02.960 --> 00:28:18.000
You could actually spin up different sessions and T-Max actually takes a log of those things where it knows what are you working on and resumes those sessions whenever it's needed.

s244
00:28:18.000 --> 00:28:25.840
So, it's actually helpful for agents to communicate and start communicating when they have multiple sessions open.

s245
00:28:25.840 --> 00:28:31.440
And these are some long running tasks that you could actually run inside T-Max.

s246
00:28:32.189 --> 00:28:34.189
[snorts]

s247
00:28:34.240 --> 00:28:38.320
Second good idea is IPC site channels.

s248
00:28:38.320 --> 00:28:43.360
So like I mentioned before, T-Max was a good upgrade.

s249
00:28:43.360 --> 00:28:50.529
So instead of doing screen scraping, which T-Max actually does um for visual in inference,

s250
00:28:50.529 --> 00:28:51.520
[laughter]

s251
00:28:51.520 --> 00:28:55.120
you could start using side channels.

s252
00:28:55.120 --> 00:28:55.769
Now

s253
00:28:55.769 --> 00:28:56.480
[snorts]

s254
00:28:56.480 --> 00:29:08.159
site channels would help you define your arrow keys that you have and help you understand and I'll go in the next slide uh how it works but the agent

s255
00:29:08.159 --> 00:29:24.000
should know what is the output JSON now this becomes really critical in the entire talk is output JSONs and Asians want that since we have multiple personalities now earlier we used to only cater to human beings

s256
00:29:24.000 --> 00:29:25.679
now we have agents.

s257
00:29:25.679 --> 00:29:34.640
So think of both these personas so that your CLI can actually take advantage of both these use cases.

s258
00:29:34.640 --> 00:29:46.799
So site channel for those who don't know about it let me explain to you in in simple words think of it like adding a REST API to your web application

s259
00:29:46.799 --> 00:29:52.880
and the website gives humans a visual interfere uh interface.

s260
00:29:52.880 --> 00:30:09.785
The API gives you a machine structured interface right so think of side channel as exactly the same that's one of those analogies that I can refer read state send input as a data

s261
00:30:09.785 --> 00:30:24.399
[clears throat] so we have the bista brew hyphen agent now this this is a command that will help you run your CLI with background agents, right?

s262
00:30:24.399 --> 00:30:36.559
Um, you want your agents to be running in the background and uh you put all of these commands, they are going to help you try using it in your current CLI.

s263
00:30:36.559 --> 00:30:46.640
See if you are able to um make some progress in the barista application that is out there for in your laptops.

s264
00:30:46.720 --> 00:30:52.000
Now, there are a couple of things that you could actually provide as an example out here.

s265
00:30:52.000 --> 00:30:57.440
You can see barista [clears throat] brew ask and then there's a bunch of information out there.

s266
00:30:57.440 --> 00:30:58.240
Oh, what is it?

s267
00:30:58.240 --> 00:31:06.320
Okay, fields then small, large, medium, maybe the sizes of the coffee and yeah, default as large.

s268
00:31:06.320 --> 00:31:07.679
What does that even mean?

s269
00:31:07.679 --> 00:31:10.080
That's input data, right?

s270
00:31:10.080 --> 00:31:20.559
So all you have to do is feed in this input data so that agent actually knows that and you got to do it in machine readable formats.

s271
00:31:20.559 --> 00:31:25.120
So this is where um that thing comes in play.

s272
00:31:25.679 --> 00:31:30.559
So don't just tell the agent what the human sees, okay?

s273
00:31:30.559 --> 00:31:32.320
Think it out of the box.

s274
00:31:32.320 --> 00:31:35.039
What are the agents going to think?

s275
00:31:35.039 --> 00:31:42.559
They have different skill sets and you have to start leveraging those skill sets.

s276
00:31:43.519 --> 00:31:46.000
Same prompt with a different brain.

s277
00:31:46.000 --> 00:31:51.919
Now, [clears throat] so on the left side you see I was talking about different personas.

s278
00:31:51.919 --> 00:31:56.240
One is the human beings and the other one is the agents.

s279
00:31:57.360 --> 00:32:09.039
The CLI is the same but when you look at the way it is structured human beings want set of options but on the other hand you have agents wherein

s280
00:32:09.039 --> 00:32:12.927
you could provide all this information in one single input file.

s281
00:32:12.927 --> 00:32:14.559
[snorts]

s282
00:32:14.559 --> 00:32:15.679
Okay.

s283
00:32:15.679 --> 00:32:22.559
So this is where we call this as context injection comes in play.

s284
00:32:22.559 --> 00:32:28.240
You provide context in such a way that let me take an example.

s285
00:32:28.240 --> 00:32:31.760
You you are actually low on oat milk.

s286
00:32:31.760 --> 00:32:32.799
Okay.

s287
00:32:32.799 --> 00:32:39.200
And uh you have only two gallons of uh oat milk left.

s288
00:32:39.200 --> 00:32:45.600
Tell your agent what kind of information it needs to ask the user.

s289
00:32:45.600 --> 00:32:48.559
confirm it if if two ounces are good enough.

s290
00:32:48.559 --> 00:32:51.919
Do you want me to pick a backup for you?

s291
00:32:51.919 --> 00:33:03.120
This information should be provided using context in injection as a technique and you can leverage this in the CLI that you are actually building.

s292
00:33:03.120 --> 00:33:04.159
Try using it.

s293
00:33:04.159 --> 00:33:08.159
Maybe you'll be on the highest on the leaderboard.

s294
00:33:12.080 --> 00:33:23.440
So we started a journey from T-Max which is basically screen scraping to injecting skills.

s295
00:33:23.919 --> 00:33:25.360
So T-Max is low effort.

s296
00:33:25.360 --> 00:33:27.919
I think a lot of you might have tried it.

s297
00:33:27.919 --> 00:33:29.563
But this is where site channels

s298
00:33:29.563 --> 00:33:29.840
[snorts]

s299
00:33:29.840 --> 00:33:42.000
also will help you in providing [clears throat] good um sort of feedback to your agents and context injection will provide you those different scenarios that you can think about

s300
00:33:42.000 --> 00:33:46.000
which doesn't happen in the human case.

s301
00:33:47.440 --> 00:33:50.880
So yeah, keep pushing your scores.

s302
00:33:54.399 --> 00:33:55.519
You want to show this?

s303
00:33:55.519 --> 00:33:57.519
Yeah.

s304
00:34:14.159 --> 00:34:17.919
It does take some time.

s305
00:34:17.919 --> 00:34:18.480
There you go.

s306
00:34:18.480 --> 00:34:19.343
Okay,

s307
00:34:19.343 --> 00:34:19.440
[snorts]

s308
00:34:19.440 --> 00:34:19.760
cool.

s309
00:34:19.760 --> 00:34:21.963
We do see a lot of traffic out here.

s310
00:34:21.963 --> 00:34:23.928
[clears throat]

s311
00:34:23.928 --> 00:34:25.928
[laughter]

s312
00:34:30.079 --> 00:34:33.919
Oh, somebody is really close to 5,000.

s313
00:34:34.399 --> 00:34:36.879
Is it bank?

s314
00:34:37.119 --> 00:34:38.993
Okay.

s315
00:34:38.993 --> 00:34:55.839
[clears throat] Any questions so far?

s316
00:34:55.839 --> 00:34:58.800
Anybody wants to get it answered?

s317
00:34:58.800 --> 00:35:01.520
Any queries?

s318
00:35:04.800 --> 00:35:06.800
Sorry.

s319
00:35:09.124 --> 00:35:10.079
[laughter]

s320
00:35:10.079 --> 00:35:12.399
Okay, let me repeat that question to you.

s321
00:35:12.399 --> 00:35:17.280
[clears throat] What happens if multiple people reach a score of 5,000?

s322
00:35:17.280 --> 00:35:19.200
Mark, you want to take that?

s323
00:35:19.200 --> 00:35:30.880
So, the the idea was we would look at score and if there's a tie, we would look at which one took the least amount of time.

s324
00:35:30.880 --> 00:35:40.240
Now, based on the resolution of this, I see several that are that have completed three of three in zero seconds.

s325
00:35:40.240 --> 00:35:40.564
I see

s326
00:35:40.564 --> 00:35:41.119
[laughter]

s327
00:35:41.119 --> 00:35:45.119
a bunch of these that have completed it in zero seconds.

s328
00:35:45.119 --> 00:35:52.240
So, I need to consult with someone because that's not supposed to be possible.

s329
00:35:53.040 --> 00:36:04.079
But the idea is that we would look at the high score the the score first and then the time it takes to respond second.

s330
00:36:04.079 --> 00:36:14.880
So that an accurate response returned quickly would be biased over an accurate response returned slowly.

s331
00:36:26.240 --> 00:36:29.440
So we have only one for now.

s332
00:36:29.440 --> 00:36:30.800
Okay, sorry.

s333
00:36:30.800 --> 00:36:33.440
We have only one.

s334
00:36:33.440 --> 00:36:50.160
eventually comes to this is the bottom of the leaderboard.

s335
00:36:50.320 --> 00:36:51.305
Nicholas

s336
00:36:51.305 --> 00:36:51.440
[snorts]

s337
00:36:51.440 --> 00:36:52.615
game up.

s338
00:36:52.615 --> 00:37:11.760
[clears throat] 0 second.

s339
00:37:37.040 --> 00:37:44.160
Do you want to wait two more minutes?

s340
00:37:44.560 --> 00:37:52.800
We can open it up ideas how engaged they are.

s341
00:38:07.040 --> 00:38:09.359
Did we have

s342
00:38:23.006 --> 00:38:25.006
[clears throat]

s343
00:38:26.880 --> 00:38:32.320
In just a few minutes, we'll open up the floor for some idea sharing.

s344
00:38:32.320 --> 00:38:41.920
So, I'm I'm hoping that we'll hear some great ideas that go above and beyond our good ideas because I know that there's some out there.

s345
00:38:58.480 --> 00:39:03.200
Mark, were you expecting so many people to be in the 4,000?

s346
00:39:03.200 --> 00:39:05.359
Yes, absolutely.

s347
00:39:05.359 --> 00:39:06.800
At this conference?

s348
00:39:06.800 --> 00:39:07.440
Absolutely.

s349
00:39:07.440 --> 00:39:09.440
Awesome.

s350
00:39:14.000 --> 00:39:14.960
Were you not?

s351
00:39:14.960 --> 00:39:16.062
No, I was.

s352
00:39:16.062 --> 00:39:18.062
[laughter]

s353
00:39:52.800 --> 00:39:53.440
What do you think?

s354
00:39:53.440 --> 00:39:56.560
Anybody can make it to 5,000?

s355
00:39:58.720 --> 00:39:59.440
I don't know.

s356
00:39:59.440 --> 00:40:01.257
We'll hold it open.

s357
00:40:01.257 --> 00:40:02.480
[laughter]

s358
00:40:02.480 --> 00:40:03.920
Why don't we open the floor up now?

s359
00:40:03.920 --> 00:40:04.960
Yeah, sure.

s360
00:40:04.960 --> 00:40:06.720
We'd love to hear ideas.

s361
00:40:06.720 --> 00:40:15.599
So, did anybody in the room come up with other ideas that were able to get them to such a high score so quickly?

s362
00:40:21.440 --> 00:40:24.182
Nobody wants to leave the desk.

s363
00:40:24.182 --> 00:40:26.182
[laughter]

s364
00:40:35.040 --> 00:40:38.160
Oh, we have Yes.

s365
00:40:53.119 --> 00:40:59.119
Let me let me come over to you [clears throat] because I can barely hear you.

s366
00:41:05.920 --> 00:41:07.920
Okay.

s367
00:41:09.359 --> 00:41:18.960
it it's a very easy to game benchmark because the only thing that matters is you fulfill one order within like small like a very small latency.

s368
00:41:18.960 --> 00:41:32.960
So if you do a deterministic parser um it which defeats the spirit of this challenge entirely but you deterministic parser and you have something like really close to the server you can get the latency down uh really quick if you just brute force it.

s369
00:41:32.960 --> 00:41:35.200
Okay, which is that what you did?

s370
00:41:35.200 --> 00:41:36.319
That's what I'm doing right now.

s371
00:41:36.319 --> 00:41:37.119
Okay.

s372
00:41:37.119 --> 00:41:38.000
All right.

s373
00:41:38.000 --> 00:41:47.359
Now you said something interesting the the deterministic parser that's running in your CLI.

s374
00:41:47.359 --> 00:41:48.640
Okay.

s375
00:41:48.640 --> 00:41:55.760
I actually like the idea of the determinism in the CLI.

s376
00:41:55.760 --> 00:42:09.119
And [clears throat] the reason is that if the CLI is a tool to be used by an LLM, then you absolutely want it to be deterministic.

s377
00:42:09.119 --> 00:42:13.599
The LLM is the non-deterministic player here.

s378
00:42:13.599 --> 00:42:21.040
We want the CLI to be as someone described it a small sharp instrument.

s379
00:42:21.040 --> 00:42:24.400
Should be very very focused on what it does.

s380
00:42:24.400 --> 00:42:30.640
And every time you give it a command, you want it to come back with the same response.

s381
00:42:30.640 --> 00:42:41.119
So you don't want to build an agent into the CLI and then hand that CLI off to another agent because you don't know what you're going to get.

s382
00:42:46.400 --> 00:42:49.280
Any other interesting ideas?

s383
00:42:49.280 --> 00:42:53.524
Obviously we a lot of people have figured out how to game the system.

s384
00:42:53.524 --> 00:42:54.960
[laughter]

s385
00:42:54.960 --> 00:42:58.000
Where there's a Mac Mini, there's a way, I guess.

s386
00:42:58.000 --> 00:43:00.000
Yeah.

s387
00:43:00.729 --> 00:43:02.729
[laughter]

s388
00:43:16.800 --> 00:43:27.680
Wait five minutes.

s389
00:43:27.680 --> 00:43:29.599
Five minutes.

s390
00:43:29.599 --> 00:43:37.599
The score hasn't changed in a while before the site crashes.

s391
00:43:42.720 --> 00:43:45.280
Did you do refresh?

s392
00:43:57.060 --> 00:43:57.715
[cough]

s393
00:43:57.715 --> 00:43:59.715
[clears throat]

s394
00:44:41.599 --> 00:44:43.359
Why does it say reconnecting?

s395
00:44:43.359 --> 00:44:44.400
Oh, there we go.

s396
00:44:44.400 --> 00:44:46.400
Okay.

s397
00:44:49.520 --> 00:44:52.240
See, the network is a little laggy.

s398
00:44:52.240 --> 00:44:56.000
The network is a little laggy out here.

s399
00:44:58.160 --> 00:45:01.040
A lot of lot of computers.

s400
00:45:01.040 --> 00:45:03.119
Sounds good.

s401
00:45:03.119 --> 00:45:05.119
Crash.

s402
00:45:35.200 --> 00:45:37.276
And we're frozen right now.

s403
00:45:37.276 --> 00:45:46.960
[clears throat] Here we go.

s404
00:45:47.599 --> 00:45:49.119
No change in the results.

s405
00:45:49.119 --> 00:45:51.119
Okay.

s406
00:46:11.440 --> 00:46:13.680
Do we want to wait a few more minutes or call it?

s407
00:46:13.680 --> 00:46:14.960
Yeah, five minutes.

s408
00:46:14.960 --> 00:46:15.520
Five minutes.

s409
00:46:15.520 --> 00:46:17.200
Okay.

s410
00:46:17.200 --> 00:46:23.920
You want to bring the PPD to PPD for the closing arguments.

s411
00:46:28.000 --> 00:46:29.920
We're going to give it about five more minutes.

s412
00:46:29.920 --> 00:46:30.319
Yeah.

s413
00:46:30.319 --> 00:46:32.960
And then we'll call it.

s414
00:46:33.040 --> 00:46:33.599
Where's Bon?

s415
00:46:33.599 --> 00:46:34.240
Where's Bonk?

s416
00:46:34.240 --> 00:46:36.560
Raise your hand.

s417
00:46:36.640 --> 00:46:37.520
Who is Bon?

s418
00:46:37.520 --> 00:46:38.800
Okay, we have something.

s419
00:46:38.800 --> 00:46:39.200
All right.

s420
00:46:39.200 --> 00:46:39.839
Big smile.

s421
00:46:39.839 --> 00:46:40.812
Big smile.

s422
00:46:40.812 --> 00:46:42.812
[laughter]

s423
00:46:44.000 --> 00:46:44.160
Now,

s424
00:46:44.160 --> 00:46:46.880
we do want to hear how you did it.

s425
00:46:46.880 --> 00:46:52.000
So once once we call it, then we'll we'll we'll we'll let you explain what you did.

s426
00:46:52.000 --> 00:46:54.480
I'd love to hear.

s427
00:46:56.532 --> 00:46:58.532
[laughter]

s428
00:47:15.440 --> 00:47:16.480
I need to go over here.

s429
00:47:16.480 --> 00:47:18.079
Yeah,

s430
00:47:18.079 --> 00:47:20.319
hold on.

s431
00:47:20.480 --> 00:47:20.720
Sorry.

s432
00:47:20.720 --> 00:47:27.119
I should have asked for another I should have asked for another microphone for the room for questions.

s433
00:47:30.800 --> 00:47:31.839
Um, yeah.

s434
00:47:31.839 --> 00:47:38.000
So, Bonk is just a an agent uh thing built on top of cloud agent SDK.

s435
00:47:38.000 --> 00:47:39.839
Um, it built the whole thing.

s436
00:47:39.839 --> 00:47:49.680
Prior to this, I already had a sort of eval bench skill that will create sort of an environment and um optimize.

s437
00:47:49.680 --> 00:47:59.920
So, it basically ended up building a uh deterministic CLI solver and then now it's just running on a T-m slamming your APIs.

s438
00:47:59.920 --> 00:48:00.319
Okay.

s439
00:48:00.319 --> 00:48:00.720
Yeah.

s440
00:48:00.720 --> 00:48:01.200
All right.

s441
00:48:01.200 --> 00:48:01.520
aso.

s442
00:48:01.520 --> 00:48:07.359
I think I got lucky because I hit it at one point at like 29 milliseconds.

s443
00:48:07.359 --> 00:48:09.920
Uh probably because we're in the same region.

s444
00:48:09.920 --> 00:48:10.240
Okay.

s445
00:48:10.240 --> 00:48:11.359
It's just a network latency,

s446
00:48:11.359 --> 00:48:11.520
right?

s447
00:48:11.520 --> 00:48:13.280
And this is all done on claude

s448
00:48:13.280 --> 00:48:14.720
uh claude agent SDK.

s449
00:48:14.720 --> 00:48:15.119
Okay.

s450
00:48:15.119 --> 00:48:16.720
Just driving it through Slack.

s451
00:48:16.720 --> 00:48:18.079
No, which model?

s452
00:48:18.079 --> 00:48:19.280
Opus 4.8.

s453
00:48:19.280 --> 00:48:19.599
Okay.

s454
00:48:19.599 --> 00:48:20.240
Yeah.

s455
00:48:20.240 --> 00:48:22.319
Did you try it with any other models

s456
00:48:22.319 --> 00:48:25.359
for the build uh for building the CLI or

s457
00:48:25.359 --> 00:48:28.319
for for your a for your agent?

s458
00:48:28.319 --> 00:48:29.359
How would we do that?

s459
00:48:29.359 --> 00:48:31.280
The CLI is completely deterministic.

s460
00:48:31.280 --> 00:48:31.440
Sure.

s461
00:48:31.440 --> 00:48:31.839
Sure.

s462
00:48:31.839 --> 00:48:36.480
So, so the agent built the CLI and is just running it on T-Max now.

s463
00:48:36.480 --> 00:48:36.800
Okay.

s464
00:48:36.800 --> 00:48:37.040
Yeah.

s465
00:48:37.040 --> 00:48:38.079
Awesome.

s466
00:48:38.079 --> 00:48:40.079
Congrats.

s467
00:48:46.720 --> 00:48:47.839
Going to go to the next one.

s468
00:48:47.839 --> 00:48:48.880
Danny.

s469
00:48:48.880 --> 00:48:51.520
Who's Danny here?

s470
00:48:58.720 --> 00:49:05.760
You see last one.

s471
00:49:11.463 --> 00:49:13.760
[clears throat] What do we want to know?

s472
00:49:13.760 --> 00:49:14.000
Yeah.

s473
00:49:14.000 --> 00:49:16.079
How'd you build it?

s474
00:49:16.079 --> 00:49:16.400
Yeah.

s475
00:49:16.400 --> 00:49:20.400
So, I don't know anything about uh coding at all.

s476
00:49:20.400 --> 00:49:22.960
I'm I'm using uh I'm basically a Vibe coder.

s477
00:49:22.960 --> 00:49:26.240
I have a couple agents and I just said, "Hey, here's the problem.

s478
00:49:26.240 --> 00:49:28.960
Read it and then fix it and fix it."

s479
00:49:28.960 --> 00:49:32.079
So, I just have Yeah.

s480
00:49:32.079 --> 00:49:36.000
Cloud code up on uh VSC.

s481
00:49:36.000 --> 00:49:38.160
Have you ever built a CLI before?

s482
00:49:38.160 --> 00:49:43.200
Uh no. I I saw a couple hours ago that obviously there was a prize to be won.

s483
00:49:43.200 --> 00:49:44.869
So, I got in and I said, "Hey,

s484
00:49:44.869 --> 00:49:46.079
[laughter]

s485
00:49:46.079 --> 00:49:47.280
uh figure this out.

s486
00:49:47.280 --> 00:49:49.839
Explain it to me like I'm, you know, like I don't know anything."

s487
00:49:49.839 --> 00:49:53.680
And then explain like I'm five.

s488
00:49:53.680 --> 00:49:55.280
Yeah, I I I don't know.

s489
00:49:55.280 --> 00:50:04.400
So, and then we came up with uh yeah, we came up with some good results and I said, "Hey, we need to make it faster and give me some suggestions

s490
00:50:04.400 --> 00:50:06.000
and yeah, seriously."

s491
00:50:06.000 --> 00:50:07.140
So, I I guess

s492
00:50:07.140 --> 00:50:07.280
[snorts]

s493
00:50:07.280 --> 00:50:17.359
we have uh two scripts running and the first one got the uh 49.98 and then I said, "Hey, make a copy and give me some suggestions for how we can get the 49.99."

s494
00:50:17.359 --> 00:50:18.880
Right.

s495
00:50:18.880 --> 00:50:20.800
Did you get any suggestions?

s496
00:50:20.800 --> 00:50:23.760
Uh it's this agent's pretty automated.

s497
00:50:23.760 --> 00:50:32.400
So I just say like hey here's the start here's the end give me a couple you know questions at the beginning if you need it but then fix the problems

s498
00:50:32.400 --> 00:50:34.880
so I don't have to deal with it kind of thing.

s499
00:50:34.880 --> 00:50:37.200
So sorry that's not a very good

s500
00:50:37.200 --> 00:50:39.040
No it's good.

s501
00:50:39.040 --> 00:50:52.240
I'm I'm sure a lot of people are wondering like writing a CLI is that something that you would ever do or would you ask your agent to do it for you?

s502
00:50:52.240 --> 00:51:04.880
So clearly we have many people in the room and just going to point Claude or Codeex at this and say build a CLI, make it fast, make it deterministic.

s503
00:51:04.880 --> 00:51:08.240
And those are the three ideas, right?

s504
00:51:09.280 --> 00:51:10.559
Okay.

s505
00:51:10.559 --> 00:51:14.640
You want to change the slide?

s506
00:51:36.559 --> 00:51:38.880
There it is.

s507
00:51:51.940 --> 00:51:53.680
[clears throat]

s508
00:51:53.680 --> 00:51:58.880
So, we talked about all of our suggestions.

s509
00:51:59.040 --> 00:52:01.440
We heard some ideas.

s510
00:52:01.440 --> 00:52:05.200
The predominant idea was let your agent figure it out for you.

s511
00:52:05.200 --> 00:52:06.960
I love that idea.

s512
00:52:06.960 --> 00:52:11.839
Um, anything else before we close it out?

s513
00:52:14.640 --> 00:52:17.680
All right, there you go.

s514
00:52:18.400 --> 00:52:24.400
Takeaway for us based on our instructions, our recommendations.

s515
00:52:24.400 --> 00:52:30.800
Give it good context, give it a side channel, and tell it to use T-Mox.

s516
00:52:30.800 --> 00:52:35.359
Those are three good ideas to start with.

s517
00:52:35.760 --> 00:52:37.839
Thanks everyone.

s518
00:52:37.839 --> 00:52:39.756
Bonk, you want to come on up?

s519
00:52:39.756 --> 00:52:40.079
[applause]

s520
00:52:40.079 --> 00:52:41.839
We still have Monk as the winner, right?

s521
00:52:41.839 --> 00:52:42.480
Yeah.

s522
00:52:42.480 --> 00:52:42.800
Okay.

s523
00:52:42.800 --> 00:52:43.680
No, no, we called it.

s524
00:52:43.680 --> 00:52:44.880
We called it five minutes ago.

s525
00:52:44.880 --> 00:52:45.520
Okay.

s526
00:52:45.520 --> 00:52:45.760
Yeah.

s527
00:52:45.760 --> 00:52:46.480
Yeah.

s528
00:52:46.480 --> 00:52:48.160
I'm not gonna change [clears throat] I'm not going to change it.

s529
00:52:48.160 --> 00:52:48.480
Yeah.

s530
00:52:48.480 --> 00:52:51.119
We We called it.

s531
00:52:55.040 --> 00:52:55.476
Awesome.

s532
00:52:55.476 --> 00:52:57.476
[applause]

s533
00:52:59.760 --> 00:53:04.800
Let us [clears throat] We'll get We'll just get a picture,

s534
00:53:04.960 --> 00:53:06.079
you know, social media.

s535
00:53:06.079 --> 00:53:08.240
That's right.

s536
00:53:14.160 --> 00:53:16.000
It's a real Mac Mini.

s537
00:53:16.000 --> 00:53:17.040
It's a real Mac.

s538
00:53:17.040 --> 00:53:19.056
Oh my god, these exist.

s539
00:53:19.056 --> 00:53:20.160
[laughter]

s540
00:53:20.160 --> 00:53:22.559
All right, awesome.

s541
00:53:25.040 --> 00:53:27.359
Where'd it come?

s542
00:53:40.016 --> 00:53:42.016
[music]