Burn Your Flags: Interactive CLIs for Agents — Mark Lummus & Navinkumar Patil, PayPal

AI Engineer · 53 min · 542 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hi everyone, welcome in.
00:16My name is Mark Lumus.
00:17I'm head of developer tools at PayPal.
00:20With me I have Naveen Kumar.
00:22He's our tech lead for the developer tools team.
00:26And today we're going to talk to you a little bit about CLI.
00:30The title of our presentation is meant to be a little provocative and you'll understand why at the end of the presentation.
00:43So today we're going to talk a little bit about a specific kind of CLI, interactive CLI, and we're going to set up an exercise and we're going to give you for those of you who wish to
01:00participate, this is a workshop after all.
01:04We'll give you a CLI that you can make work better in an Agentic workflow.
01:10So that's the purpose of this workshop today.
01:14So we're going to give you plenty of time.
01:15We'll talk for a few minutes.
01:16Try to keep the talking to a minimum and then we'll get you started.
01:24Now it's kind of interesting.
01:26I think someone said what's old is new.
01:29When I started my career long time ago, I have gray hair.
01:34Everything was a CLI.
01:36Everything was a command line interface.
01:39Any of you remember working with MS DOSs before we had Microsoft Windows?
01:45There you go.
01:46My first job out of college, I was designing embedded systems.
01:51And so you would connect to a computer over a serial port and you would pull up a VT100 compatible terminal and you would start typing.
02:02And that was your CLI.
02:04That was the the the way you interfaced with the computer.
02:08So with today CLIs have kind of had a resurgence I guess you know with the maybe it was clawed code that really made them much more popular and now we have codecs
02:20um and so a lot of people are are adding CLI uh tools so that it's easier to access the capabilities that they want to expose to an agent.
02:32Now, with interactive CLI, this is why we're here today with what we want to talk about.
02:40Um, agents shell out to Bash constantly, and that's good.
02:47But if they run an interactive CLI, that may be bad because not all harnesses handle interaction very well.
02:57And if they don't handle it very well, they'll fail quietly.
03:00You won't know.
03:01No errors, no nothing.
03:05But why do we care?
03:07Interactive CLIs are great because the next question can emerge based on what way you answer a previous question.
03:19Flags multiply if you try to create flags that predict all the possible flows.
03:27The choices are contextual and they can also be responsive to what's happening in an environment.
03:34So if your CLI is calling an API and getting a response and then the question comes up based on what the API returns, that's really really challenging to handle
03:44with flags and the guard rails can live in the interactive flow.
03:55So here's an example over here on the on the left.
04:01We see a a sample CLI.
04:03It's interactive.
04:05The CLI's opened an interactive prompt, an error key prompt.
04:12The human sees it.
04:13The human can use the arrow keys, but the agent doesn't see it.
04:19The agent sees a redraw loop, not a question.
04:23and the harness loses the UI state.
04:28So here's the failure mode.
04:29It's silent and invisible.
04:35So the common fix will just add flags.
04:39And flags are fine for simple use cases.
04:44You have uh you know what the you know what the flows are up front.
04:51It's a single pass.
04:53All the options are independent.
04:55Great great use case for a a a simple flag.
05:02But when the answers change based on the context or they're changing based on reactions to real time events, that's when it becomes a problem.
05:13You really can't use flags for those situations.
05:19Now, an agent is not like an automation script.
05:24An automation script knows nothing.
05:28All of the decisions are pre-baked up front.
05:32The agent can reason.
05:34It can react.
05:36It can ask.
05:37It can retry.
05:38It can back up.
05:40And most importantly, an agent can walk a decision tree if you provide provide one.
05:48You can't do that with flags.
05:53So we already have a model today for what's happening here.
05:58It's called accessibility.
06:01And so if you map what we do for humans to what we need to do for agents, then the solutions become apparent.
06:12It's the same discipline with a new reader.
06:19So here's the here's the challenge.
06:23How might we keep the guard rails and the guidance and stop pretending that agents and humans have the same abilities?
06:33Now, I'm not going to answer this question.
06:34That's the whole purpose of this workshop.
06:40So, here's the challenge.
06:42We've got a CLI.
06:44We call it Bris Barista 9000.
06:47It's going to take coffee orders and it's going to fail when you try to run it from an agent because it's interactive.
07:03And if you choose to participate in the challenge, the winner will win a Mac Mini.
07:12Walk walk away with it today.
07:15It's in that bag right there.
07:17[laughter]
07:20That was a challenge in and of itself getting a Mac Mini.
07:23I'll be straight up.
07:25Okay.
07:26Now, [clears throat] simple rules.
07:30We're going to give you a CLI.
07:35You make changes to that CLI, run it through an agent, and it's going to send some information up to our server.
07:44And then it's going to update a leaderboard.
07:47And we'll put put the leaderboard up in a moment.
07:50We don't have two inputs, so we're going to change the slide deck out for the leaderboard so you can see it.
07:57Now, here's Naveen.
08:00He's going to talk you through this.
08:01Thank you, Mark.
08:02All right.
08:05Cool.
08:07Yeah.
08:08Uh I think it's an exciting assignment for everybody and this topic is very near and dear to me but uh keeping things aside Mark are we getting any coffee after we build this
08:21[laughter]
08:22I hope so if we're successful
08:25okay cool so yeah let me uh before we start into um the assignment I want to take to a journey right uh when I started my engineering career I started to play with ids.
08:40I started to use Eclipse because I was primarily using Java.
08:45And as time went by, moved into Intelligj uh Jet Brains.
08:51It was amazing.
08:54Looking back, five or six years behind uh things started to evolve.
08:58there were new ids in the market especially Microsoft VS code which gave birth to a lot of u other ids as well.
09:11So if you look right now Microsoft had its own VS code editor but later on new AI id started to pop out like cursor windsurf to name a few.
09:25In all these journeys, CLIs were constant, right?
09:30At some point, I thought, okay, you know, CLIs are having a slow death of their own.
09:35Maybe they are not going to come back.
09:38But, you know, uh I I I actually like to see this hangover movie dialogue that um you know, all all the four friends are in the car and they say, "Hey, we are back."
09:50Imagine CLI seeing that to each other.
09:53And in the world of AI, they are actually VC, right?
09:57Whoever bought it, yeah, I don't know.
10:00But the whole story is now the CLI have changed.
10:04And who is responsible for that?
10:07Agents, right?
10:08So this workshop is primarily focusing on agents, your CLIs.
10:16And this is going to be a hands-on exercise.
10:19There are rules to it.
10:22There are no rules to it.
10:24So feel free to hack as much as possible and you know the prize is here.
10:30Um so yeah we have maybe uh ven if I can get the laptop.
10:36So we have couple of things.
10:39So we have yeah um so we have the repo which is PayPal cafe.
10:45It has the CLI that you can actually download.
10:49That's the barebone CLI.
10:51Now, it's not going to be that easy uh for a Mac Mini.
10:54Yeah.
10:55Uh so, you will get some initial commands with it.
10:59Uh play around with it.
11:00Register yourself.
11:02Uh provide your username, whatever you like.
11:04Um your nicknames, anything, and try to brew a coffee out of it.
11:11Okay, that's that's the goal.
11:13And in this whole exercise, we want you to also be innovative in your approaches and going forward, it's going to get interesting with agents using the CLI.
11:25So think it from that perspective.
11:29I change this.
11:31I just change my laptops here.
11:36Now during the during the session, we're we're not starting you out with any hints.
11:43Once we feel like enough people have gotten the the repo installed, can run the the CLI as is, then maybe every 15 or 20 minutes, we'll share a good idea.
12:01Now, I'm not going to call it a best practice.
12:04I'm just going to call it a good idea because I feel like we're early enough in the journey here where we don't know what the best practices are.
12:14We're kind of feeling them out now.
12:17They're good ideas.
12:19I'm also hoping that at the end we'll hear some great ideas from you in the audience so we can share.
12:28So just keep that in mind.
12:30We're we're all learning.
12:32This is a journey.
12:34We'll come back to that towards the end.
12:46Is it not working?
12:52Give us a couple of minutes.
12:54I think we'll get the technical issues resolved.
13:13seems like people are actually using it.
13:16So on the right side you will actually see get CLI and um if you see PayPal cafe/ rules that's where you will have all the rules outlined to it.
13:28Feel free to start working on the CLI till we figure it out.
13:34Okay.
13:53Do we have the Europe?
13:57Oh, is it that one?
14:13Yeah.
14:13Hey, there you go.
14:14Nice.
14:20Awesome.
14:20Thank you.
14:21Thank you so much.
14:22Hey Cool.
14:28So you can see this is Oh, we do see some signings.
14:34Okay.
14:34So on the right side you will see get CLI.
14:36You can download that and I hope you guys can see it.
14:40It's at the bottom right.
14:42And there are some rules here.
14:44You can click on these rules and it will pop out some rules.
15:00So, we'll probably take around about 25 to 30 minutes.
15:04Um, and in between we'll interrupt with more slides and make it interesting.
15:09[laughter]
15:12Where's the laptop for the other slides?
15:14Venode's got it.
15:15Okay, that's fine.
15:20and we're here to help.
15:22If you're having trouble, please raise your hand.
15:25We'll try to assist you as best we can.
15:52a little bit about the models.
15:56Don't use models.
16:10Now, we had a little bit of a debate internally.
16:13There are some great models out there.
16:19The interesting thing about this challenge, if you use the highest thinking mode on the bleeding edge LLMs, you can probably brute way brute force your way through this.
16:33But I feel like in the current climate, teams are starting to think about their token budget.
16:40And so you don't necessarily want to waste your highinking tokens on running a CLI tool.
16:51So the real challenge is can you make Haik coup or sonnet or something at that level run your CLI?
17:03We know you can do it with the highest.
17:05Can you do it with something that's somewhere in the middle that's faster and cheaper?
17:10That's the That's the interesting challenge if you if you want to take it on.
17:16Sounds fun.
17:16Yeah.
17:20Awesome.
17:56Since we are on the coffee topic, uh Mark, were you aware that San Francisco City has the best bakeries in the entire United States?
18:08Were you aware about that fact?
18:09I was not aware of that.
18:10Yeah, you should definitely.
18:11Go on.
18:12Yeah.
18:12Go on, tell me more.
18:14Um, I have I've done some bakery crawls uh around in the city and they are so fun.
18:20Like you should definitely check out all the amazing bakeries out here.
18:25Okay.
18:25Yeah.
18:26Any favorites?
18:28Butter and crumble.
18:30My favorite one.
18:31Although I couldn't go there.
18:33[laughter]
18:38Awesome.
18:41So as of now we have J Gray Codex at the top.
18:47So guys please catch up with him or her So with the init command, you could actually register your names.
21:08I hope some of you have figured that out.
22:06Do you want to talk about the first You might have to think about Once I plug it in, I
25:44[sighs]
25:51Go ahead.
25:52Okay,
25:53cool.
25:54Um, let's move on to some presentations.
25:57Meanwhile, you can start working.
25:58I mean, you'll continue working with the CLI that you have.
26:03But as as a first hint, you know, it's it's a good idea to start using T-Max.
26:09How many of you know about T-Max here?
26:12Okay, quite a fair amount of audience out here.
26:15Um, so yeah u in short start with running T-Max and um start by putting help command.
26:25Now, help is really important because every AI agent in today's world relies on the help command before it even starts using that command.
26:38[laughter]
26:38So, yeah, send keys and capture pain gets you far.
26:49Now, T-Max actually um for those who actually don't know about it is like a multiple you could actually use multiple terminal sessions uh with the T-Max and it's like a split pane
27:04feature that you could leverage with T-Max.
27:07So, with this example, you can see here um I mean the screen is very short for me.
27:12So, let me just zoom in a little bit.
27:15Yeah, T-Max D. Now, what this flag does is it detaches the entire uh CLI and puts it under a different emox instance all together for
27:28[laughter]
27:28you.
27:32So you could see size uh T-max capture pane like I said you could actually have split panes in T-maxes.
27:44So you could actually take care of u working around with um in in the CLI and uh that's one of those cool features that can help your agents perform lot of activities.
27:57Um, think of it when you want to run multiple agents.
28:02You could actually spin up different sessions and T-Max actually takes a log of those things where it knows what are you working on and resumes those sessions whenever it's needed.
28:18So, it's actually helpful for agents to communicate and start communicating when they have multiple sessions open.
28:25And these are some long running tasks that you could actually run inside T-Max.
28:32[snorts]
28:34Second good idea is IPC site channels.
28:38So like I mentioned before, T-Max was a good upgrade.
28:43So instead of doing screen scraping, which T-Max actually does um for visual in inference,
28:50[laughter]
28:51you could start using side channels.
28:55Now
28:55[snorts]
28:56site channels would help you define your arrow keys that you have and help you understand and I'll go in the next slide uh how it works but the agent
29:08should know what is the output JSON now this becomes really critical in the entire talk is output JSONs and Asians want that since we have multiple personalities now earlier we used to only cater to human beings
29:24now we have agents.
29:25So think of both these personas so that your CLI can actually take advantage of both these use cases.
29:34So site channel for those who don't know about it let me explain to you in in simple words think of it like adding a REST API to your web application
29:46and the website gives humans a visual interfere uh interface.
29:52The API gives you a machine structured interface right so think of side channel as exactly the same that's one of those analogies that I can refer read state send input as a data
30:09[clears throat] so we have the bista brew hyphen agent now this this is a command that will help you run your CLI with background agents, right?
30:24Um, you want your agents to be running in the background and uh you put all of these commands, they are going to help you try using it in your current CLI.
30:36See if you are able to um make some progress in the barista application that is out there for in your laptops.
30:46Now, there are a couple of things that you could actually provide as an example out here.
30:52You can see barista [clears throat] brew ask and then there's a bunch of information out there.
30:57Oh, what is it?
30:58Okay, fields then small, large, medium, maybe the sizes of the coffee and yeah, default as large.
31:06What does that even mean?
31:07That's input data, right?
31:10So all you have to do is feed in this input data so that agent actually knows that and you got to do it in machine readable formats.
31:20So this is where um that thing comes in play.
31:25So don't just tell the agent what the human sees, okay?
31:30Think it out of the box.
31:32What are the agents going to think?
31:35They have different skill sets and you have to start leveraging those skill sets.
31:43Same prompt with a different brain.
31:46Now, [clears throat] so on the left side you see I was talking about different personas.
31:51One is the human beings and the other one is the agents.
31:57The CLI is the same but when you look at the way it is structured human beings want set of options but on the other hand you have agents wherein
32:09you could provide all this information in one single input file.
32:12[snorts]
32:14Okay.
32:15So this is where we call this as context injection comes in play.
32:22You provide context in such a way that let me take an example.
32:28You you are actually low on oat milk.
32:31Okay.
32:32And uh you have only two gallons of uh oat milk left.
32:39Tell your agent what kind of information it needs to ask the user.
32:45confirm it if if two ounces are good enough.
32:48Do you want me to pick a backup for you?
32:51This information should be provided using context in injection as a technique and you can leverage this in the CLI that you are actually building.
33:03Try using it.
33:04Maybe you'll be on the highest on the leaderboard.
33:12So we started a journey from T-Max which is basically screen scraping to injecting skills.
33:23So T-Max is low effort.
33:25I think a lot of you might have tried it.
33:27But this is where site channels
33:29[snorts]
33:29also will help you in providing [clears throat] good um sort of feedback to your agents and context injection will provide you those different scenarios that you can think about
33:42which doesn't happen in the human case.
33:47So yeah, keep pushing your scores.
33:54You want to show this?
33:55Yeah.
34:14It does take some time.
34:17There you go.
34:18Okay,
34:19[snorts]
34:19cool.
34:19We do see a lot of traffic out here.
34:21[clears throat]
34:23[laughter]
34:30Oh, somebody is really close to 5,000.
34:34Is it bank?
34:37Okay.
34:38[clears throat] Any questions so far?
34:55Anybody wants to get it answered?
34:58Any queries?
35:04Sorry.
35:09[laughter]
35:10Okay, let me repeat that question to you.
35:12[clears throat] What happens if multiple people reach a score of 5,000?
35:17Mark, you want to take that?
35:19So, the the idea was we would look at score and if there's a tie, we would look at which one took the least amount of time.
35:30Now, based on the resolution of this, I see several that are that have completed three of three in zero seconds.
35:40I see
35:40[laughter]
35:41a bunch of these that have completed it in zero seconds.
35:45So, I need to consult with someone because that's not supposed to be possible.
35:53But the idea is that we would look at the high score the the score first and then the time it takes to respond second.
36:04So that an accurate response returned quickly would be biased over an accurate response returned slowly.
36:26So we have only one for now.
36:29Okay, sorry.
36:30We have only one.
36:33eventually comes to this is the bottom of the leaderboard.
36:50Nicholas
36:51[snorts]
36:51game up.
36:52[clears throat] 0 second.
37:37Do you want to wait two more minutes?
37:44We can open it up ideas how engaged they are.
38:07Did we have
38:23[clears throat]
38:26In just a few minutes, we'll open up the floor for some idea sharing.
38:32So, I'm I'm hoping that we'll hear some great ideas that go above and beyond our good ideas because I know that there's some out there.
38:58Mark, were you expecting so many people to be in the 4,000?
39:03Yes, absolutely.
39:05At this conference?
39:06Absolutely.
39:07Awesome.
39:14Were you not?
39:14No, I was.
39:16[laughter]
39:52What do you think?
39:53Anybody can make it to 5,000?
39:58I don't know.
39:59We'll hold it open.
40:01[laughter]
40:02Why don't we open the floor up now?
40:03Yeah, sure.
40:04We'd love to hear ideas.
40:06So, did anybody in the room come up with other ideas that were able to get them to such a high score so quickly?
40:21Nobody wants to leave the desk.
40:24[laughter]
40:35Oh, we have Yes.
40:53Let me let me come over to you [clears throat] because I can barely hear you.
41:05Okay.
41:09it it's a very easy to game benchmark because the only thing that matters is you fulfill one order within like small like a very small latency.
41:18So if you do a deterministic parser um it which defeats the spirit of this challenge entirely but you deterministic parser and you have something like really close to the server you can get the latency down uh really quick if you just brute force it.
41:32Okay, which is that what you did?
41:35That's what I'm doing right now.
41:36Okay.
41:37All right.
41:38Now you said something interesting the the deterministic parser that's running in your CLI.
41:47Okay.
41:48I actually like the idea of the determinism in the CLI.
41:55And [clears throat] the reason is that if the CLI is a tool to be used by an LLM, then you absolutely want it to be deterministic.
42:09The LLM is the non-deterministic player here.
42:13We want the CLI to be as someone described it a small sharp instrument.
42:21Should be very very focused on what it does.
42:24And every time you give it a command, you want it to come back with the same response.
42:30So you don't want to build an agent into the CLI and then hand that CLI off to another agent because you don't know what you're going to get.
42:46Any other interesting ideas?
42:49Obviously we a lot of people have figured out how to game the system.
42:53[laughter]
42:54Where there's a Mac Mini, there's a way, I guess.
42:58Yeah.
43:00[laughter]
43:16Wait five minutes.
43:27Five minutes.
43:29The score hasn't changed in a while before the site crashes.
43:42Did you do refresh?
43:57[cough]
43:57[clears throat]
44:41Why does it say reconnecting?
44:43Oh, there we go.
44:44Okay.
44:49See, the network is a little laggy.
44:52The network is a little laggy out here.
44:58A lot of lot of computers.
45:01Sounds good.
45:03Crash.
45:35And we're frozen right now.
45:37[clears throat] Here we go.
45:47No change in the results.
45:49Okay.
46:11Do we want to wait a few more minutes or call it?
46:13Yeah, five minutes.
46:14Five minutes.
46:15Okay.
46:17You want to bring the PPD to PPD for the closing arguments.
46:28We're going to give it about five more minutes.
46:29Yeah.
46:30And then we'll call it.
46:33Where's Bon?
46:33Where's Bonk?
46:34Raise your hand.
46:36Who is Bon?
46:37Okay, we have something.
46:38All right.
46:39Big smile.
46:39Big smile.
46:40[laughter]
46:44Now,
46:44we do want to hear how you did it.
46:46So once once we call it, then we'll we'll we'll we'll let you explain what you did.
46:52I'd love to hear.
46:56[laughter]
47:15I need to go over here.
47:16Yeah,
47:18hold on.
47:20Sorry.
47:20I should have asked for another I should have asked for another microphone for the room for questions.
47:30Um, yeah.
47:31So, Bonk is just a an agent uh thing built on top of cloud agent SDK.
47:38Um, it built the whole thing.
47:39Prior to this, I already had a sort of eval bench skill that will create sort of an environment and um optimize.
47:49So, it basically ended up building a uh deterministic CLI solver and then now it's just running on a T-m slamming your APIs.
47:59Okay.
48:00Yeah.
48:00All right.
48:01aso.
48:01I think I got lucky because I hit it at one point at like 29 milliseconds.
48:07Uh probably because we're in the same region.
48:09Okay.
48:10It's just a network latency,
48:11right?
48:11And this is all done on claude
48:13uh claude agent SDK.
48:14Okay.
48:15Just driving it through Slack.
48:16No, which model?
48:18Opus 4.8.
48:19Okay.
48:19Yeah.
48:20Did you try it with any other models
48:22for the build uh for building the CLI or
48:25for for your a for your agent?
48:28How would we do that?
48:29The CLI is completely deterministic.
48:31Sure.
48:31Sure.
48:31So, so the agent built the CLI and is just running it on T-Max now.
48:36Okay.
48:36Yeah.
48:37Awesome.
48:38Congrats.
48:46Going to go to the next one.
48:47Danny.
48:48Who's Danny here?
48:58You see last one.
49:11[clears throat] What do we want to know?
49:13Yeah.
49:14How'd you build it?
49:16Yeah.
49:16So, I don't know anything about uh coding at all.
49:20I'm I'm using uh I'm basically a Vibe coder.
49:22I have a couple agents and I just said, "Hey, here's the problem.
49:26Read it and then fix it and fix it."
49:28So, I just have Yeah.
49:32Cloud code up on uh VSC.
49:36Have you ever built a CLI before?
49:38Uh no. I I saw a couple hours ago that obviously there was a prize to be won.
49:43So, I got in and I said, "Hey,
49:44[laughter]
49:46uh figure this out.
49:47Explain it to me like I'm, you know, like I don't know anything."
49:49And then explain like I'm five.
49:53Yeah, I I I don't know.
49:55So, and then we came up with uh yeah, we came up with some good results and I said, "Hey, we need to make it faster and give me some suggestions
50:04and yeah, seriously."
50:06So, I I guess
50:07[snorts]
50:07we have uh two scripts running and the first one got the uh 49.98 and then I said, "Hey, make a copy and give me some suggestions for how we can get the 49.99."
50:17Right.
50:18Did you get any suggestions?
50:20Uh it's this agent's pretty automated.
50:23So I just say like hey here's the start here's the end give me a couple you know questions at the beginning if you need it but then fix the problems
50:32so I don't have to deal with it kind of thing.
50:34So sorry that's not a very good
50:37No it's good.
50:39I'm I'm sure a lot of people are wondering like writing a CLI is that something that you would ever do or would you ask your agent to do it for you?
50:52So clearly we have many people in the room and just going to point Claude or Codeex at this and say build a CLI, make it fast, make it deterministic.
51:04And those are the three ideas, right?
51:09Okay.
51:10You want to change the slide?
51:36There it is.
51:51[clears throat]
51:53So, we talked about all of our suggestions.
51:59We heard some ideas.
52:01The predominant idea was let your agent figure it out for you.
52:05I love that idea.
52:06Um, anything else before we close it out?
52:14All right, there you go.
52:18Takeaway for us based on our instructions, our recommendations.
52:24Give it good context, give it a side channel, and tell it to use T-Mox.
52:30Those are three good ideas to start with.
52:35Thanks everyone.
52:37Bonk, you want to come on up?
52:39[applause]
52:40We still have Monk as the winner, right?
52:41Yeah.
52:42Okay.
52:42No, no, we called it.
52:43We called it five minutes ago.
52:44Okay.
52:45Yeah.
52:45Yeah.
52:46I'm not gonna change [clears throat] I'm not going to change it.
52:48Yeah.
52:48We We called it.
52:55Awesome.
52:55[applause]
52:59Let us [clears throat] We'll get We'll just get a picture,
53:04you know, social media.
53:06That's right.
53:14It's a real Mac Mini.
53:16It's a real Mac.
53:17Oh my god, these exist.
53:19[laughter]
53:20All right, awesome.
53:25Where'd it come?
53:40[music]