WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=cgimkNGNjvU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/cgimkNGNjvU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.920 --> 00:00:21.920
I'm a product director here at Sneak and going to be talking to you about agent development security and specifically talking about how we can gain confidence when we use agents

s3
00:00:21.920 --> 00:00:23.400
especially as we give them more autonomy.

s4
00:00:23.400 --> 00:00:28.800
It's a very common theme I've heard in this track and a number of the other tracks today.

s5
00:00:28.800 --> 00:00:35.640
Um I'm not going to go through the full history of LLMs but the model context protocol release was really big moment.

s6
00:00:35.640 --> 00:00:46.640
Until then, I don't know what you guys are doing but I was very often copying pasting between uh agent clients and some other services and with MCP I think people really started to connect this and have a much more

s7
00:00:46.640 --> 00:00:48.520
really connected AI system.

s8
00:00:48.520 --> 00:00:57.160
Um and I'm not saying that MCP is the end-all-be-all and I may or may not may not have been amongst the people who were saying that MCP would die at some point last year.

s9
00:00:57.160 --> 00:01:03.880
Um but it has been a game-changer in the sense that developers started to connect agents to external tools and services.

s10
00:01:03.880 --> 00:01:08.760
Um and at that time there really wasn't any security to speak of.

s11
00:01:08.760 --> 00:01:13.560
Like most companies we released an MCP server almost immediately.

s12
00:01:13.560 --> 00:01:19.520
Um ours specifically enabled local directories to be scanned by our security scanning engines.

s13
00:01:19.520 --> 00:01:24.240
Uh developers could ask questions in natural language about the security issues that were identified.

s14
00:01:24.240 --> 00:01:31.000
They could learn why specific vulnerabilities were important or how they might be exploited and then work iteratively towards a fix.

s15
00:01:31.000 --> 00:01:35.920
Um shortly thereafter we decided to pair our MCP server with rules.

s16
00:01:35.920 --> 00:01:44.160
Um and the rules basically ensured that any AI generated code would be tested and if there were security issues identified that they would be automatically fixed.

s17
00:01:44.160 --> 00:01:48.840
It was simple, was fast to deploy and it did solve a meaningful pain point for our customers.

s18
00:01:48.840 --> 00:01:54.200
Uh so that really was our original position secure agent generated code at the moment of inception.

s19
00:01:54.200 --> 00:01:55.800
Oops.

s20
00:01:55.800 --> 00:02:00.360
Uh Uh, but over the last year learned that this framing was really incomplete.

s21
00:02:00.360 --> 00:02:11.720
Um, our customers started telling us that they were not only worried about the code that was being generated, they're also worried about what the agent had access to, and then also the actions the agent might be taking.

s22
00:02:11.720 --> 00:02:16.720
Uh, so I'm going to just mention like briefly a few incidents that have come up over the last last year or so.

s23
00:02:16.720 --> 00:02:25.040
I think we've talked about them um in in the keynote that Manoj gave earlier today, but also I think we've seen some of these in other presentations, but just as a quick refresher,

s24
00:02:25.040 --> 00:02:34.000
um about a year ago, we saw um Replit's agent um ignore a code freeze instruction and ultimately deleted a production database.

s25
00:02:34.000 --> 00:02:40.200
It tried to cover up that it did this, fabricated records uh to basically say like, "No, there was there was no issue whatsoever."

s26
00:02:40.200 --> 00:02:42.800
Um, and finally it said that there was no way to recover.

s27
00:02:42.800 --> 00:02:44.400
Fortunately, it turned out that that was wrong.

s28
00:02:44.400 --> 00:02:48.200
They they were able to recover, but the damage was still done.

s29
00:02:48.200 --> 00:02:53.800
Then in April, I know we just we talked about this just a couple hours ago, but uh there's the Packet OS incident.

s30
00:02:53.800 --> 00:03:01.080
An agent again found an an over-privileged API token, and that resulted in a production database being deleted.

s31
00:03:01.080 --> 00:03:09.560
The backups were also deleted, and so a 3-month-old backup is what was could be used to ultimately uh you know, try to get back to recovery.

s32
00:03:09.560 --> 00:03:14.040
What's really interesting here is that the agent wasn't acting maliciously, it was actually trying to solve a problem.

s33
00:03:14.040 --> 00:03:20.239
It was trying to solve what it perceived to be a credential mismatch, uh but there was nothing in place to stop it.

s34
00:03:20.239 --> 00:03:24.880
Those two examples were really about the agent actions that might be taken, but that's not always the case.

s35
00:03:24.880 --> 00:03:27.040
That's not always what the attack surface is.

s36
00:03:27.040 --> 00:03:36.959
Just last month, Team PCP was able to exfiltrate uh almost 4,000 of GitHub's internal repositories using a malicious VS Code extension.

s37
00:03:37.160 --> 00:03:45.760
So, all of this and kind of us being in the security space for the last 10 years and talking to our customers, um it's really shaped how we think about

s38
00:03:45.760 --> 00:03:49.200
what uh agentic development security and what that really means.

s39
00:03:49.200 --> 00:04:00.480
And our our belief is that in order to confidently use agents for software development at scale and to start letting them operate more autonomously in long-running tasks, whether it's just getting up to make a cup of coffee

s40
00:04:00.480 --> 00:04:07.400
or letting them run overnight, uh it's really critical to secure what agents generate, what they use, and what they do.

s41
00:04:07.400 --> 00:04:15.959
Um and I'll spend a couple minutes talking about our journey in each of these pillars over the last year, what we've learned, and uh and our current perspective.

s42
00:04:16.200 --> 00:04:21.560
As I mentioned at the top, um this has been our longest area of experimentation and investment.

s43
00:04:21.560 --> 00:04:24.720
It's securing the code that the agents generate.

s44
00:04:24.720 --> 00:04:27.400
Um and the reason for that is we don't want issues to make it to production.

s45
00:04:27.400 --> 00:04:34.840
We don't want to kind of increase that backlog, which has been so challenging to manage and is now uh a luxury that companies just cannot afford.

s46
00:04:34.840 --> 00:04:39.160
Uh most companies do have security checks in their deployment pipelines.

s47
00:04:39.160 --> 00:04:45.640
Um and so even if they don't make it to production, we want to ensure that bottlenecks are not getting in uh created at at those stages.

s48
00:04:45.640 --> 00:04:49.720
Um I mentioned our original approach, MCP server plus rules.

s49
00:04:49.720 --> 00:04:55.040
Um it was really easy to paste a an MCP configuration um and a rule definition.

s50
00:04:55.040 --> 00:05:06.040
And and over time we added shortcuts to make that even easier, and the agent clients actually uh made it like simple commands to enable these configurations through plugins or just simple simple CLI tools.

s51
00:05:06.040 --> 00:05:08.000
Uh but the approach did have real limitations.

s52
00:05:08.000 --> 00:05:10.480
Agents sometimes ignored the rule files.

s53
00:05:10.480 --> 00:05:14.160
Uh scan execution did add latency at the end of its run.

s54
00:05:14.160 --> 00:05:19.160
Um and every time that we ran scans through the context window, that consumed tokens.

s55
00:05:19.160 --> 00:05:23.760
And so we were not the only ones dealing with these challenges, fortunately.

s56
00:05:23.760 --> 00:05:27.400
Um outside of security, these same pain points existed.

s57
00:05:27.400 --> 00:05:33.480
Um and the good news is that the agent client providers, they provided new mechanisms for how to direct agents.

s58
00:05:33.480 --> 00:05:36.240
Um primarily this has been in the form of skills and hooks.

s59
00:05:36.240 --> 00:05:42.680
I think everybody might have their own opinion on which one you want to use uh for which, but they they really solve a lot of these problems.

s60
00:05:42.680 --> 00:05:49.800
Uh our current recommendation um is to use Python-based hooks for this use case that can fire asynchronously on agent tool calls.

s61
00:05:49.800 --> 00:05:59.680
And so immediately after an agent writes a new file or modifies a file, we can kick off a scan using our CLI, not even using the MCP server, asynchronously.

s62
00:05:59.680 --> 00:06:04.920
And that will write write any newly identified or newly introduced issues to a temporary file.

s63
00:06:04.920 --> 00:06:12.960
And finally, on the session stop event, that's when a hook triggers the agent and it will check that temp file to see if there were new newly introduced issues, and only then

s64
00:06:12.960 --> 00:06:15.160
will it kick off a fix and validate loop.

s65
00:06:15.160 --> 00:06:17.720
So, now the workflow is deterministic.

s66
00:06:17.720 --> 00:06:21.440
Latency is removed because all that testing happens asynchronously.

s67
00:06:21.440 --> 00:06:31.320
And again, because newly introduced issues are the only thing that's being surfaced to that agent context, the context window doesn't get any unnecessarily unnecessary bloat.

s68
00:06:31.760 --> 00:06:37.760
When we talk about agent supply chain, we're really thinking about the things that help you build more connected agentic workflows.

s69
00:06:37.760 --> 00:06:43.760
But like everyone's been saying today, this is this also presents a new attack surface.

s70
00:06:43.840 --> 00:06:47.840
Last year, we acquired a company called Invariant Labs.

s71
00:06:47.840 --> 00:06:53.320
Um and following that, we produced a report uh which you can access if you want with this QR code.

s72
00:06:53.320 --> 00:06:57.480
You can also come and talk to us um at our booth uh if you want to see this here.

s73
00:06:57.480 --> 00:07:07.360
Uh there's many similarities between package ecosystem risk, which is where kind of sneak got its got its start, um and that of agent skills, but we really think that skills are more problematic.

s74
00:07:07.360 --> 00:07:10.160
Um they have higher privilege by default.

s75
00:07:10.160 --> 00:07:16.040
Natural language prompt injection cannot be detected through uh through through typical code detection.

s76
00:07:16.040 --> 00:07:18.640
Um and malicious skills can modify agent memory.

s77
00:07:18.640 --> 00:07:22.120
So, even if you remove a malicious skill, they can still persist.

s78
00:07:22.120 --> 00:07:24.760
That risk can still persist after the fact.

s79
00:07:24.760 --> 00:07:33.000
Um and in an audit that we did of nearly 4,000 skills on ClawHub, uh over one in eight had a critical severity issue, and we actually found 76 malicious payloads

s80
00:07:33.000 --> 00:07:35.600
uh in in that subset.

s81
00:07:35.880 --> 00:07:40.840
So, the solution that we built here auto-discovers all the agent components on your machine.

s82
00:07:40.840 --> 00:07:49.320
Um it if you have MCP servers configured, it will connect to them, retrieve the tool descriptions, analyze them to see what security risks are present.

s83
00:07:49.320 --> 00:07:50.800
It will do the same thing for skill files.

s84
00:07:50.800 --> 00:07:58.080
We'll look at your skill.amdl files, see the dependent files that are associated with that skill, and look to see what threats might exist.

s85
00:07:58.080 --> 00:08:01.400
I demoed this or I tried to demo this during Manisha's keynote earlier.

s86
00:08:01.400 --> 00:08:08.840
The internet didn't play super nice, so we showed a little video, but we can show this in action afterwards as well.

s87
00:08:08.840 --> 00:08:16.160
Um, just in the last month we produced a report based on some anonymized data following the release of the capabilities that I just mentioned.

s88
00:08:16.160 --> 00:08:28.919
Um, and I think these numbers are probably going to be uh pretty low based on the expectations of the group that we see here, but I think it's worth reminding you that not everyone is as kind of AI forward as the folks that are attending

s89
00:08:28.919 --> 00:08:31.280
uh this this conference here.

s90
00:08:31.280 --> 00:08:38.479
Uh, but kind of from the average developer we saw that um more than half were using MCP servers and a fifth were leveraging skills.

s91
00:08:38.479 --> 00:08:51.720
Um Beyond this adoption, one in 12 developers in this group had an MCP server where there is either a high or critical severity finding identified in that MCP server itself.

s92
00:08:53.200 --> 00:09:01.880
I realize this is probably pretty small, but just want to highlight that these are some of the categories of risk that we look for when we analyze MCP servers.

s93
00:09:01.880 --> 00:09:07.120
And similarly for skills, there's a number of different risks that we can also also look for.

s94
00:09:07.120 --> 00:09:14.680
This happens to be one of the one of the skill skills that I did an assessment of during that recorded demo that we showed earlier.

s95
00:09:14.680 --> 00:09:24.280
Very very risky, malicious in some cases, but also just kind of maybe some negligent negligent behavior that that could cause problems for me.

s96
00:09:24.280 --> 00:09:31.120
Um, the last leg of this stool for agent development security is govern governing agent behavior.

s97
00:09:31.120 --> 00:09:41.760
This is currently in open preview, and it's really focused on how we ensure that an agent is not taking exfiltrated, destructive, or otherwise malicious or risky actions.

s98
00:09:41.760 --> 00:09:48.080
Um I'm choosing to show here the policy configuration view because I want to highlight the actions that we are trying to intercept today.

s99
00:09:48.080 --> 00:09:52.640
Depending on how you use agents, the types of policies that you set might ultimately be different.

s100
00:09:52.640 --> 00:10:02.080
Um I think in an ideal world, you're always able to steer an agent towards the right action um and kind of making it's making so that the human does not have to be in the loop.

s101
00:10:02.080 --> 00:10:06.880
A really good example of that is redacting PII or secret before a command executes.

s102
00:10:06.880 --> 00:10:12.720
Um but in other cases, you actually may want that agent to ask you uh because there's not a clear-cut answer.

s103
00:10:12.720 --> 00:10:25.440
Um and so if there is a potentially destructive shell command um or maybe if the agent wants to access a directory that's kind of outside of the scope of the permissions you initially gave, that's a good reason to ask.

s104
00:10:25.440 --> 00:10:34.880
But I think as we move towards more background agents and cloud agents being ran where you're kind of trying to step away and trying to not be sitting at your desk babysitting the agent entirely,

s105
00:10:34.880 --> 00:10:38.880
um asks are much much less viable option.

s106
00:10:38.880 --> 00:10:47.839
Um and so this is an area that we're going to continue to invest, but I think you know this is going to mean more fine-grained policies, may also mean that we need some sort of auto didacticism

s107
00:10:47.839 --> 00:10:53.960
in the product, kind of a self-learning capability based on the decisions that you make over time to help you become more autonomous.

s108
00:10:53.960 --> 00:10:59.520
Um but the short of it is that like today we we are accountable for the actions that our agents take.

s109
00:10:59.520 --> 00:11:10.920
Um and even in the future if that accountability model becomes a little bit more shared, I don't think any of us want to work at companies where um we are doing a disservice to our customers and potentially letting our agents take risky actions.

s110
00:11:10.920 --> 00:11:18.240
Um and and nobody really wants to be the reason for one of those incidents that gets widely publicized to to to occur.

s111
00:11:18.600 --> 00:11:26.320
Um so far I've been doing a lot of telling rather than showing and I think for this conference we really want to focus more on more on showing where we can.

s112
00:11:26.320 --> 00:11:34.920
Uh so I'm going to invite my colleague Dan Arpino uh to come up and and take over um and really show you some of the areas that we're exploring for solving these problems.

s113
00:11:34.920 --> 00:11:45.280
Um I want to be clear that what we're showing is not committed roadmap things that are definitely going to be available here, but we really want to show it to you so that you can give

s114
00:11:45.280 --> 00:11:47.480
some feedback to us and like is are we heading in the right direction?

s115
00:11:47.480 --> 00:11:50.880
Like is this the better way to solve these problems?

s116
00:11:50.880 --> 00:11:57.680
And hopefully this will ultimately mature into some of the solutions that we can deliver to you to you all here.

s117
00:11:58.480 --> 00:12:04.480
And up here, any happy to take a question or two cuz we got to do the laptop transfer which is always fun.

s118
00:12:04.480 --> 00:12:06.480
Yeah.

s119
00:12:15.560 --> 00:12:17.680
Yes, so the question is what's the difference between steer and ask?

s120
00:12:17.680 --> 00:12:26.200
There in steer is the idea that there can be a policy that is defined that doesn't need a human in the loop that it can basically guide an agent to say, nope, instead of doing that let's do something different.

s121
00:12:26.200 --> 00:12:33.320
So the the classic example that I think is easiest for folks to understand is like what if I just redact the PII or a secret replace it with asterisks

s122
00:12:33.320 --> 00:12:36.960
and let the the kind of agent proceed, but that's not going to work in all cases.

s123
00:12:36.960 --> 00:12:50.320
Ask is going to be an explicit prompt to the user either through the agent interface whether you're using Codex or Claude or Cursor what whatever tool or potentially through some other mechanism that that we're exploring now as well.

s124
00:13:02.520 --> 00:13:14.240
Yeah, happy to talk kind of after here, but the the short of it is that we're relying right now from an implementation perspective on hooks that can intercept kind of a pre-tool execution

s125
00:13:14.240 --> 00:13:22.480
in near real time, assess is this actually potentially problematic, and then before the agent kind of invokes the next thing provide that feedback to it.

s126
00:13:22.480 --> 00:13:25.800
But happy to talk after here.

s127
00:13:25.839 --> 00:13:27.440
You in good shape?

s128
00:13:27.440 --> 00:13:27.880
Yeah, cool.

s129
00:13:27.880 --> 00:13:29.880
Hopefully guys can all hear me.

s130
00:13:29.880 --> 00:13:32.760
Um like Ezra said, my name's Dan Arpino.

s131
00:13:32.760 --> 00:13:36.000
Uh I am a software engineer at Snyk.

s132
00:13:36.000 --> 00:13:39.960
Uh I'm one of the developers on the ADS platform.

s133
00:13:39.960 --> 00:13:44.240
Um and so I started out specifically from that insured trusted output section.

s134
00:13:44.240 --> 00:13:47.280
Basically like how do we ensure that code is secure?

s135
00:13:47.280 --> 00:13:54.320
Um and and as Ezra said, like we were doing a whole bunch of hooks and and and integrating with the agents themselves.

s136
00:13:54.320 --> 00:14:00.480
And I was getting frustrated with all those integrations and so I decided to try to build a little bit of a pair programmer right here.

s137
00:14:00.480 --> 00:14:07.040
And so this is a local electron app that I have running on my machine that's watching everything that's going on on on my machine.

s138
00:14:07.040 --> 00:14:10.640
You can literally see it's trying to fix some vulnerabilities right now.

s139
00:14:10.640 --> 00:14:11.720
It's watching the files.

s140
00:14:11.720 --> 00:14:17.920
It's running these scans in the background and it's automatically trying to kick off agents to keep this secure.

s141
00:14:17.920 --> 00:14:22.200
Um and so this is ideas like how can we actually use the agents to help us?

s142
00:14:22.200 --> 00:14:31.560
Um and then I decided to take this a little bit further because like as Ezra said, insured trusted output was just one of the three pillars of agentic development security.

s143
00:14:31.560 --> 00:14:38.320
Um we still want to know like what the agent is doing and what is the agent and what tools the agents are are using themselves.

s144
00:14:38.320 --> 00:14:47.760
And so yes, we can have like we have some of our tools today that Snyk offers on the Evo platform and I I suggest that you all go to evo.snyk.io

s145
00:14:47.760 --> 00:14:55.640
and see this agent guard in action and see all the skills and that that your and MCPs that your organization is using.

s146
00:14:55.640 --> 00:15:00.120
But what I really wanted to do is I wanted to give local visibility into what's running on my machine.

s147
00:15:00.120 --> 00:15:03.160
As a developer who cares about security, I want to see this.

s148
00:15:03.160 --> 00:15:04.640
And there's no really good way to see this.

s149
00:15:04.640 --> 00:15:11.120
And so like here in this tool, I can see all of my running LLMs right here.

s150
00:15:11.120 --> 00:15:14.880
Um I can see all of my running AI components.

s151
00:15:14.880 --> 00:15:17.480
I can see and this this is the MCP servers.

s152
00:15:17.480 --> 00:15:18.600
These are the skills.

s153
00:15:18.600 --> 00:15:26.320
These are the CLIs and the and the models that that that are running right here Um that Ezra talked about that with like these different risk scores.

s154
00:15:26.320 --> 00:15:34.920
So you can actually see like all the different things that I have willingly or unwillingly installed and running on my machine.

s155
00:15:35.520 --> 00:15:41.040
Um similar we talked about like what is the agent allowed to do?

s156
00:15:41.040 --> 00:15:48.800
And so like yes, my organization can set a policy for me, but I might want to set my own policy.

s157
00:15:48.800 --> 00:15:55.440
Or more often, I actually may want to set a policy depending on what project I am actually working on.

s158
00:15:55.440 --> 00:16:01.200
So I can actually set up based on my different workspace what I want to apply to each one of those.

s159
00:16:01.200 --> 00:16:04.960
Um for example, I actually turned this off.

s160
00:16:04.960 --> 00:16:13.440
I'm building out a bowler scanner broken object layer authentication big issue that we have we could that could be a whole another presentation.

s161
00:16:13.440 --> 00:16:20.040
I was doing some benchmarking and snappy was automatically fixing my vulnerable tools for the benchmarking in the in the first place.

s162
00:16:20.040 --> 00:16:23.360
So I have to actually turn that off.

s163
00:16:23.360 --> 00:16:26.680
Um but yeah, the goal here is there that you can actually set your own guardrails.

s164
00:16:26.680 --> 00:16:29.640
It can automatically go through and fix any vulnerabilities.

s165
00:16:29.640 --> 00:16:36.040
So like for example here you actually see that it it implemented some code right here that's actually running.

s166
00:16:36.040 --> 00:16:43.480
Um and what happens is when it implements these codes, it will actually start a scan.

s167
00:16:43.480 --> 00:16:45.720
This is this it will start a scan.

s168
00:16:45.720 --> 00:16:47.440
It will find and detect your vulnerabilities.

s169
00:16:47.440 --> 00:16:51.400
It will actually launch a new coding agent and try to fix those right away.

s170
00:16:51.400 --> 00:17:03.080
Um And yeah, and then yeah, so giving me visibility on what workspaces I have, giving me visibility on how many of my different agents are running, what my agents are doing,

s171
00:17:03.080 --> 00:17:05.120
what they're costing me.

s172
00:17:05.120 --> 00:17:11.000
Um getting a history of everything it's doing, all the vulnerabilities.

s173
00:17:11.000 --> 00:17:19.839
I can actually see on these fixed ones what it actually tried to do, how it tried to fix it.

s174
00:17:21.959 --> 00:17:24.800
Being able to track all my different sessions.

s175
00:17:24.800 --> 00:17:30.560
Giving me an audit of everything that my LLM's and all of my LLM's are doing on my machines.

s176
00:17:30.560 --> 00:17:33.280
Like, what are all the commands it's running?

s177
00:17:33.280 --> 00:17:35.480
What are all the files it's accessing?

s178
00:17:35.480 --> 00:17:41.840
Um, there's some really interesting implications here on how I actually want to app uh optimize some of my files and everything like that.

s179
00:17:41.840 --> 00:17:44.400
You can see which ones are my heavy reads and writes.

s180
00:17:44.400 --> 00:17:47.040
You can see which ones are my heavy reads that have almost no edits.

s181
00:17:47.040 --> 00:17:51.480
Like, I might actually want skills to tell me to do different things about that.

s182
00:17:51.480 --> 00:17:53.480
What are the commands it's running?

s183
00:17:53.480 --> 00:17:56.600
Basically, what web pages that am I hitting?

s184
00:17:56.600 --> 00:17:59.760
Monitoring all my different MCPs and tools.

s185
00:17:59.760 --> 00:18:02.600
Um Yeah.

s186
00:18:02.600 --> 00:18:07.560
And then then yeah, so right now we've got this running our static analysis code testing.

s187
00:18:07.560 --> 00:18:09.200
We have some open source testing.

s188
00:18:09.200 --> 00:18:11.600
We've got uh secrets testing.

s189
00:18:11.600 --> 00:18:18.800
Uh we've got the Bola uh scanner, which is like a set of work in progress, allowing to configure all of these things.

s190
00:18:18.800 --> 00:18:27.120
Um And and this works because it's Well, one of the reasons that it's doing this is it's it's actually running a local host on your machine.

s191
00:18:27.120 --> 00:18:32.600
And so I was able to actually run this one earlier.

s192
00:18:32.600 --> 00:18:37.000
And if you look at this, I spun Claude up and I was like, "Hey Claude, read my ENV environment."

s193
00:18:37.000 --> 00:18:38.920
And Claude was like, "Hey, no, I can't do that.

s194
00:18:38.920 --> 00:18:39.440
I'm smart."

s195
00:18:39.440 --> 00:18:42.800
And I was like, "All right, well, Claude, tell me what my secret key is right here."

s196
00:18:42.800 --> 00:18:45.000
And Claude was like, "Oh, I'll try to read that."

s197
00:18:45.000 --> 00:18:53.000
And you can actually see that because we've actually set up enforcement here uh Snappy actually blocked the access of of reading this file.

s198
00:18:53.000 --> 00:18:55.200
Um so yes, the agents are getting better.

s199
00:18:55.200 --> 00:19:01.080
They are not perfect, which is why I like having deterministic guardrails on your machine.

s200
00:19:01.080 --> 00:19:08.400
Um being able to set those guardrails on your on your machine and having visibility into what your agent is doing.

s201
00:19:08.400 --> 00:19:11.440
Um and what it's running, I I is pretty key.

s202
00:19:11.440 --> 00:19:14.320
Uh one of the big things is is how do we trust agents?

s203
00:19:14.320 --> 00:19:17.840
Um I want visibility, I want auditability.

s204
00:19:17.840 --> 00:19:19.960
Um and those are really key to me.

s205
00:19:19.960 --> 00:19:23.480
Um and so this is why I we started developing this.

s206
00:19:23.480 --> 00:19:27.360
Um in theory in in in the future you could actually hook this up.

s207
00:19:27.360 --> 00:19:31.800
You can you could actually download um you could download rule packs.

s208
00:19:31.800 --> 00:19:34.800
You could download connected to your sneak tenant and org.

s209
00:19:34.800 --> 00:19:37.800
Um you can update your own rate sheets, whatever you want.

s210
00:19:37.800 --> 00:19:42.360
Um but yeah, this is a little bit of a future vision of like what ADS could look like.

s211
00:19:42.360 --> 00:19:44.920
This is really saying like here's my development environment.

s212
00:19:44.920 --> 00:19:49.320
I could be running this on a cloud machine telling me everything that's happening on that cloud machine.

s213
00:19:49.320 --> 00:19:50.400
Give me that visibility.

s214
00:19:50.400 --> 00:19:51.520
Give me that auditability.

s215
00:19:51.520 --> 00:19:53.160
Give me that traceability.

s216
00:19:53.160 --> 00:20:00.160
Uh really important aspects in learning how to how to how to trust the agents and making sure that they're not going off the rail.

s217
00:20:01.600 --> 00:20:02.920
Cool.

s218
00:20:02.920 --> 00:20:05.200
Um and I think

s219
00:20:05.200 --> 00:20:06.760
I should have asked at the beginning.

s220
00:20:06.760 --> 00:20:09.680
Let me turn this mic on here.

s221
00:20:10.480 --> 00:20:11.200
That was awesome, Dan.

s222
00:20:11.200 --> 00:20:12.000
Thanks.

s223
00:20:12.000 --> 00:20:13.080
Uh should have asked at the beginning.

s224
00:20:13.080 --> 00:20:19.440
Of the folks are here, are you guys who who's on the security side of the house today as opposed to engineering?

s225
00:20:19.440 --> 00:20:19.960
Awesome.

s226
00:20:19.960 --> 00:20:20.440
Hands down.

s227
00:20:20.440 --> 00:20:23.520
And who who's an engineer uh in the room?

s228
00:20:23.520 --> 00:20:24.600
Cool.

s229
00:20:24.600 --> 00:20:31.880
So I think we're like this is a good acknowledgement of like the different audiences that I think these solutions are really trying to address.

s230
00:20:31.880 --> 00:20:35.280
I think if you ask the security folks in the room, they'd be like restrict everything.

s231
00:20:35.280 --> 00:20:38.120
Just like please do not let anything bad happen.

s232
00:20:38.120 --> 00:20:45.000
If you ask developers, you'd say any any false positive that causes kind of more noise in my workflow is just kind of hell on earth.

s233
00:20:45.000 --> 00:20:51.600
And so that's the that's the needle that we're ultimately looking to thread here um and why we're trying to come at this from kind of both sides.

s234
00:20:51.600 --> 00:21:06.760
Um and I think what Dan was really showing is how do we really really lean into the developer experience in this new agentic world um in a way that is going to make uh it still possible to achieve like all the productivity gains that I think everybody wants out of AI for software development

s235
00:21:06.760 --> 00:21:12.920
while still being able to trust and kind of sleep at night uh like the last presentation was talking about.

s236
00:21:15.560 --> 00:21:17.560
Yeah.

s237
00:21:21.520 --> 00:21:25.040
Uh we have talked about possibly throwing it up on our our Sneak Labs.

s238
00:21:25.040 --> 00:21:26.800
Uh it's still in development.

s239
00:21:26.800 --> 00:21:27.680
Uh come talk to me.

s240
00:21:27.680 --> 00:21:29.280
Come come by the Sneak booth afterwards.

s241
00:21:29.280 --> 00:21:33.000
I'll be there from uh till close this afternoon.

s242
00:21:33.000 --> 00:21:43.000
We'll at least get names and if it's if it's not today, which is probably not today, then we can at least see like once it's there we we'd love we'd love to get feedback once we can get something out there like that.

s243
00:21:43.000 --> 00:21:45.640
Do you expect Snappy

s244
00:21:46.120 --> 00:21:47.840
Can you repeat the question on the mic please?

s245
00:21:47.840 --> 00:21:52.720
Uh the question is do I expect Snappy to support things like Pi?

s246
00:21:55.080 --> 00:21:59.120
Um it's actually really well.

s247
00:21:59.280 --> 00:22:05.040
We will This is a tool that I have developed for my own personal use and I think it has a lot of value.

s248
00:22:05.040 --> 00:22:07.400
It will go where the market takes it.

s249
00:22:07.400 --> 00:22:08.520
And and that was kind of the idea.

s250
00:22:08.520 --> 00:22:10.400
Like it doesn't all have to be up in the cloud.

s251
00:22:10.400 --> 00:22:12.240
I want local visibility.

s252
00:22:12.240 --> 00:22:18.240
Um wherever the market takes it is where I expect it to go and and yeah.

s253
00:22:18.880 --> 00:22:23.480
Sounds like you might have a use case, so we'd love to love to connect after.

s254
00:22:35.760 --> 00:22:37.760
Yeah.

s255
00:22:40.120 --> 00:22:46.520
Yeah, so the question started with a lot of praise for what what we've done and then uh asked about the false positive rate um here.

s256
00:22:46.520 --> 00:22:50.680
Um and so far the the feedback overall has been has been good.

s257
00:22:50.680 --> 00:22:52.280
I like it's it's certainly not zero.

s258
00:22:52.280 --> 00:22:55.680
Like I don't think anybody who's playing in the space is claiming that it's zero.

s259
00:22:55.680 --> 00:23:04.360
Um I anecdotally like when I am using it cuz I have it installed on my machine like it does not I do not find it cumbersome and and bothersome.

s260
00:23:04.360 --> 00:23:10.640
I think I probably have one instance that I can remember like in the last month where I was like, "Oh, that that really was not a particular problem."

s261
00:23:10.640 --> 00:23:20.080
Uh but, we're a pretty aggressively right now working with design partners that include sometimes hundreds of developers within companies who might be doing a variety of different things to try to refine this.

s262
00:23:20.080 --> 00:23:26.280
So, I think it's going to continue to get better and better and hopefully like asymptotically approach approach zero.

s263
00:23:26.280 --> 00:23:32.120
But, I'd be shocked if we ever lived in a world where it was like absolute zero false positives for for for for any of the companies out there.

s264
00:23:32.120 --> 00:23:41.200
And if anybody here is working on solutions that are similar and you've got ideas on how you're solving that, would love to learn cuz I think like there's plenty for us to learn here, too.

s265
00:23:41.200 --> 00:23:43.440
Thank you.

s266
00:23:53.600 --> 00:24:02.920
I think the the question was, does this only work on on Does it only work on on cloud on quick code stored in the cloud or can it work on your local machine?

s267
00:24:02.920 --> 00:24:05.200
Is that the question?

s268
00:24:18.800 --> 00:24:22.360
Yeah, so different type types of of products that we have right there.

s269
00:24:22.360 --> 00:24:31.160
And I think Manosh mentioned in the first one that like burning down a backlog in a remediation agent, um currently not in the the tool that we showed today.

s270
00:24:31.160 --> 00:24:35.560
And Agility Development Security is more that that forward-facing stop the bleed going forward.

s271
00:24:35.560 --> 00:24:42.320
Um but, some of the things that we're releasing in Agility Development Security is a set of of skills, commands, hooks, and stuff like that.

s272
00:24:42.320 --> 00:24:52.640
And so, like we have a a sneak fix skill that you can pass in a known vulnerability to and it will actually use some of that sneak logic to actually like try to

s273
00:24:52.640 --> 00:25:00.680
understand the breakability, understand what changed, know what the good recommended fixes are, and guide the agent to a more effective uh fix.

s274
00:25:00.680 --> 00:25:10.240
So, what I didn't have time to demo in Manoj's keynote earlier was basically how we leverage that breakability as part of an input into is this something that I can fix

s275
00:25:10.240 --> 00:25:21.360
quickly and generally it's really easy to generate fixes, getting the approval of the code changes and merging them in like that's still the human loop kind of time time expensive type of a thing.

s276
00:25:21.360 --> 00:25:27.880
And so that is kind of that that that's a whole other area that we're working on under kind of the remediation umbrella.

s277
00:25:27.880 --> 00:25:32.080
Really cool stuff just didn't fit into this particular talk.

s278
00:25:32.080 --> 00:25:34.280
Thank you.

s279
00:25:36.720 --> 00:25:37.160
Yeah, sorry.

s280
00:25:37.160 --> 00:25:38.520
I can't see cuz of the lights.

s281
00:25:38.520 --> 00:25:41.240
we are blinded, yeah.

s282
00:25:58.520 --> 00:25:59.840
That is a fascinating question.

s283
00:25:59.840 --> 00:26:10.640
The question was how how would this potentially work any solution around guarding agent behavior when you're not talking about text-based exchanges but more sensor type data or other other things.

s284
00:26:10.640 --> 00:26:13.080
I don't I don't know.

s285
00:26:13.080 --> 00:26:15.640
Um Okay.

s286
00:26:15.640 --> 00:26:25.160
I mean if I think it's probably you know it's still it's it's a form of language that's being communicated right rather because just cuz it's not natural language doesn't mean it's not it's not language.

s287
00:26:25.160 --> 00:26:30.880
there might even be some advantages around kind of like a known structured schema for some of that data.

s288
00:26:30.880 --> 00:26:34.080
I don't know if schema is necessarily the right word but like schema adjacent.

s289
00:26:34.080 --> 00:26:36.080
Uh That's cool.

s290
00:26:36.080 --> 00:26:37.600
I've nobody's ever brought something up like that.

s291
00:26:37.600 --> 00:26:38.720
I would love I would love to talk.

s292
00:26:38.720 --> 00:26:43.400
I know I'm making a lot of meeting plans right after this but like I'll I'll be hanging out right outside.

s293
00:26:43.400 --> 00:26:45.560
And a lot of guardrails are still the same.

s294
00:26:45.560 --> 00:26:52.080
You still want you still may want to control access like what one sensor can affect or what one sensor can affect.

s295
00:26:52.080 --> 00:26:58.800
Um like so you're just talking about the input of data but there's a whole bunch of outputs of data and what the model can do and what else it can access.

s296
00:26:58.800 --> 00:27:10.040
Uh, so a lot of those guardrails um, and the format of those guardrails stay the same even if like that input language changes to sensor reading as opposed to natural language.

s297
00:27:11.560 --> 00:27:11.840
Cool.

s298
00:27:11.840 --> 00:27:14.360
I know we're holding you guys from lunch so thank you so much for the time.

s299
00:27:14.360 --> 00:27:17.120
We really appreciate it.

s300
00:27:29.963 --> 00:27:31.963
[music]