WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=mav15aW9lLM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/mav15aW9lLM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.920 --> 00:00:14.920
Okay.

s3
00:00:14.960 --> 00:00:16.000
Hello everybody.

s4
00:00:16.000 --> 00:00:21.920
My name is Chris Lovejoy and I'm a member of technical staff at Anterior and I work as a forward deployed engineer.

s5
00:00:21.920 --> 00:00:27.760
So I embed within enterprise organizations and help them get value from using AI agents.

s6
00:00:27.760 --> 00:00:31.600
And I previously worked at Anterior with Saul.

s7
00:00:31.600 --> 00:00:32.279
Hi everybody.

s8
00:00:32.279 --> 00:00:33.280
I'm Saul.

s9
00:00:33.280 --> 00:00:35.240
I'm VP of engineering at Anterior.

s10
00:00:35.240 --> 00:00:43.560
We're a New York based company selling AI uh agentic AI to US health insurance companies.

s11
00:00:43.560 --> 00:00:52.320
Um Chris and I have spent a lot of time building in enterprise and in health care enterprises particularly.

s12
00:00:52.320 --> 00:00:58.800
And health care is a very challenging place to develop and deploy AI.

s13
00:00:58.800 --> 00:01:12.600
Uh health [clears throat] care is so challenging because of the requirements around process and uh compliance, the regulatory requirements that that that are so important.

s14
00:01:12.600 --> 00:01:21.200
Also because of the uh direct real impact that your work has on people's lives, which is of course also what makes it so rewarding.

s15
00:01:21.200 --> 00:01:30.720
Um I think a lot of the learnings you can take from working in enterprise for health care, you can take to enterprise in other regulated industries.

s16
00:01:30.720 --> 00:01:39.520
Like finance, defense, government work, anywhere where process is so important and has to be followed.

s17
00:01:39.520 --> 00:01:46.800
In this talk we're going to talk about um some of the learnings that we've had and specifically we're going to talk about why enterprise tech stacks aren't ready for AI agents

s18
00:01:46.800 --> 00:01:51.640
and some of the primitives that we've built in the past in order to unlock them.

s19
00:01:51.640 --> 00:02:00.600
And to make this concrete, let's start by considering a scenario that might be familiar to many of you, which is the the enterprise proof of proof of concept, the enterprise POC.

s20
00:02:00.600 --> 00:02:07.240
And let's say we have identified a customer that we want to serve, and we've identified a priority use case with them.

s21
00:02:07.240 --> 00:02:09.320
So, obviously we're on the healthcare track here.

s22
00:02:09.320 --> 00:02:17.520
Let's consider a um a large health system and a use case that is some sort of administrative healthcare workflow.

s23
00:02:17.520 --> 00:02:23.920
So, you work with them, you scope out a POC, you define the metrics that you are going to care about, you're going to bench benchmark yourselves on.

s24
00:02:23.920 --> 00:02:32.480
Um you allocate two engineers, you spend 4 weeks building it, and um the actual build-out might look a little bit something like this.

s25
00:02:32.480 --> 00:02:35.840
An enterprise stack is very complicated.

s26
00:02:35.840 --> 00:02:44.000
It's much, much more than we're showing here, but generally you can have an application layer, a control plane layer, the data plane.

s27
00:02:44.000 --> 00:02:48.480
For your POC, you're going to need some access to the model provider as well.

s28
00:02:48.480 --> 00:02:52.959
And your POC is going to need access to data across all of these different planes.

s29
00:02:52.959 --> 00:02:57.120
It may be some in the data lake, some directly from the application layer, for example.

s30
00:02:57.120 --> 00:03:00.519
And so, you're going to deploy it something like this.

s31
00:03:00.519 --> 00:03:03.200
It's going to connect to all these different places.

s32
00:03:03.200 --> 00:03:05.239
There's going to be some offline data pulling.

s33
00:03:05.239 --> 00:03:08.000
There's going to be some maybe some online.

s34
00:03:08.000 --> 00:03:12.959
Generally, you'll get access to the data and push towards the results.

s35
00:03:14.080 --> 00:03:14.920
And so, things go well.

s36
00:03:14.920 --> 00:03:16.120
You you get great results.

s37
00:03:16.120 --> 00:03:18.560
The the AI performs you know, as you expected.

s38
00:03:18.560 --> 00:03:20.400
You you hit the performance metrics.

s39
00:03:20.400 --> 00:03:21.640
Um you know, it's fast.

s40
00:03:21.640 --> 00:03:22.959
It's relatively cheap.

s41
00:03:22.959 --> 00:03:27.840
And you hold a meeting, you present this to the relevant stakeholders, and everyone seems pretty happy.

s42
00:03:27.840 --> 00:03:35.640
Um so, you know, your chief of finance um in the in the company is very excited and wants to understand what's going to be the impact on the budget for next year.

s43
00:03:35.640 --> 00:03:40.720
Uh your chief medical officer is excited to tell his his colleagues, you know, how accurate his AI is.

s44
00:03:40.720 --> 00:03:44.519
Um and the head of sales asks, "Okay, when can we put powered by AI?

s45
00:03:44.519 --> 00:03:46.200
When can we put that on the websites?"

s46
00:03:46.200 --> 00:03:51.600
Um but the problem is that everyone here is assuming that the the hard part is done, that the AI was was the challenging part.

s47
00:03:51.600 --> 00:03:56.160
But actually, as we know, often getting things into production is really where the challenge lies.

s48
00:03:56.160 --> 00:04:04.840
Um and to get a bit more specific on what that challenge looks like, um you hold a meeting the next day, you bring in the relevant stakeholders to discuss productionizing

s49
00:04:04.840 --> 00:04:07.000
this proof of concept application.

s50
00:04:07.000 --> 00:04:11.760
And somebody raises their hand and says, "Um can I see the the audit trail for this?

s51
00:04:11.760 --> 00:04:17.720
Like for us, for compliance, it's critical that we can see every step, every action that the agent takes, every piece of data that it accesses.

s52
00:04:17.720 --> 00:04:19.239
Can can you give that to me?"

s53
00:04:19.239 --> 00:04:27.919
And you realize actually, you know, with the way things have been implemented in the initial POC, without these um kind of true integrations, that actually that's going to be quite challenging.

s54
00:04:27.919 --> 00:04:30.240
And then somebody else um pops up with some other questions.

s55
00:04:30.240 --> 00:04:33.800
So, somebody asks, "Okay, well actually, how is data sensitive data being handled here?

s56
00:04:33.800 --> 00:04:34.880
Um how's that being passed to the agents?

s57
00:04:34.880 --> 00:04:39.440
You know, we have a very strict boundary around where our data can go and where it can't go.

s58
00:04:39.440 --> 00:04:40.640
Um is it is this respecting that?

s59
00:04:40.640 --> 00:04:41.960
How does that look?"

s60
00:04:41.960 --> 00:04:44.680
And then your chief medical officer says, again, "Who's approving the decisions here?

s61
00:04:44.680 --> 00:04:53.520
Because we know in certain scenarios, we have to escalate to a clinician who will then uh you know, approve or or or not agree with what the agent is saying.

s62
00:04:53.520 --> 00:04:54.680
Um so, how does that happen?

s63
00:04:54.680 --> 00:04:56.480
Like what's the mechanism for that?"

s64
00:04:56.480 --> 00:04:59.000
And over the course of the meetings, you know, you can imagine you get more and more questions.

s65
00:04:59.000 --> 00:05:01.400
So, can untrusted data manipulate the model?

s66
00:05:01.400 --> 00:05:03.800
How do we know that the agent continues to perform well?

s67
00:05:03.800 --> 00:05:05.200
How do we deal with integrations?

s68
00:05:05.200 --> 00:05:09.000
How do we connect to Epic, to Salesforce, to the other kind of applications that we care about?

s69
00:05:09.000 --> 00:05:13.560
And for the purposes of this talk, we're going to focus on these four, the highlighted ones.

s70
00:05:13.560 --> 00:05:16.040
For the other two, feel free to come and chat to me and talk about these later.

s71
00:05:16.040 --> 00:05:20.200
We're very happy to talk, but um just in the interest of time, we'll stay focused.

s72
00:05:20.200 --> 00:05:23.800
And let's start with uh this one about the audit trail.

s73
00:05:23.800 --> 00:05:28.400
So, this is a question you're guaranteed to get from the security team.

s74
00:05:28.400 --> 00:05:31.080
They're going to want to see an audit trail.

s75
00:05:31.080 --> 00:05:36.919
And for programmers, an audit trail sounds very like a typical developer log you might have in DataDog.

s76
00:05:36.919 --> 00:05:39.440
Surely it's it's it's a similar kind of thing.

s77
00:05:39.440 --> 00:05:50.480
But for security frameworks that that exist in the real enterprise world, like SOC 2, HITRUST, HIPAA, an audit trail is is a bit more than that.

s78
00:05:50.480 --> 00:05:56.360
It it has to contain a complete record of absolutely every action that the agent took.

s79
00:05:56.360 --> 00:06:04.760
It has to contain all of the places where the agent accessed data, all of the authorization by which the agent did something.

s80
00:06:04.760 --> 00:06:09.440
It's it's this complete record in in a much more fundamental way.

s81
00:06:09.440 --> 00:06:13.880
And uh you one way of thinking about it is in a legal sense.

s82
00:06:13.880 --> 00:06:18.960
It say our agent's decisions came up in a court of law.

s83
00:06:18.960 --> 00:06:24.880
Could we show a justifiable chain of evidence for why the particular actions were taken by a decision?

s84
00:06:24.880 --> 00:06:29.080
And that's something that could easily happen within the health care context, for example.

s85
00:06:29.080 --> 00:06:38.000
When I think about architecting systems like this, I think often about what do I want to make easy?

s86
00:06:38.000 --> 00:06:47.800
What are But when I'm choosing my constraints, I'm saying, "Okay, these are the things I want my system to be easy and let that drive the trade-offs that that I'm going to make."

s87
00:06:47.800 --> 00:06:56.880
And a particular pattern that uh uh is used in lots of different industries, for example, in finance, is a transaction log.

s88
00:06:56.880 --> 00:07:05.640
An immutable record of events that store all of the transactions that happen throughout the system.

s89
00:07:05.640 --> 00:07:09.280
And this is append-only timestamp log.

s90
00:07:09.280 --> 00:07:10.640
It's complete.

s91
00:07:10.640 --> 00:07:14.440
So, this is your source of truth for all of the data of the system.

s92
00:07:14.440 --> 00:07:15.160
And it's unified.

s93
00:07:15.160 --> 00:07:21.160
So, there's only one source of truth across all of the different agents that you might have running in parallel, for example.

s94
00:07:21.160 --> 00:07:28.640
And architecting this way, the making this trade-off, uh means that auditability becomes trivial.

s95
00:07:28.640 --> 00:07:32.840
It falls out of your data storage paradigm that you've chosen.

s96
00:07:32.840 --> 00:07:43.480
It sort of is impossible not to be able to roll back time and and see exactly the state of the system at a a particular point in time and be able to uh provide that as an audit trail for what happened

s97
00:07:43.480 --> 00:07:46.160
uh uh each point in time.

s98
00:07:46.160 --> 00:07:48.960
And of course, these are trade-offs.

s99
00:07:48.960 --> 00:07:51.560
So, what's the trade-off you're making here?

s100
00:07:51.560 --> 00:07:58.760
I think we could say that for this kind of event logging or sometimes called event sourcing pattern, writes become very easy.

s101
00:07:58.760 --> 00:08:00.520
So, you just drop an event.

s102
00:08:00.520 --> 00:08:07.360
Reads become more difficult because you have to read through all of the events in order to reconstruct a view of what happened.

s103
00:08:07.360 --> 00:08:14.760
And there are patterns like caching and snapshots so you can bring to to to make that that simpler, but there always is more effort there.

s104
00:08:14.760 --> 00:08:26.000
Although, I have seen in in the healthcare context that actually, you're going to want different interpretations of the raw data that your agents recorded after the fact.

s105
00:08:26.000 --> 00:08:38.320
So, for example, it might be that more events happened and that changes the interpretation of the healthcare journey, and you want a different view of the the source of truth at that particular time.

s106
00:08:38.320 --> 00:08:46.800
And this pattern makes that easy because all of your views of the data are ephemeral computed projections of the event log.

s107
00:08:46.800 --> 00:08:48.600
Um okay.

s108
00:08:48.600 --> 00:08:55.920
Next, the compliance officer comes and is asking, "How is the sensitive data passed around the system?

s109
00:08:55.920 --> 00:08:58.720
What's the life cycle of data within our system?"

s110
00:08:58.720 --> 00:09:04.000
And within a healthcare context, as we all know, data means a lot.

s111
00:09:04.000 --> 00:09:07.839
It's PHI, protected or personal health information.

s112
00:09:07.839 --> 00:09:10.560
It's has legal restrictions around it.

s113
00:09:10.560 --> 00:09:14.520
Not just HIPAA, but other legal restrictions about the use of people's data.

s114
00:09:14.520 --> 00:09:27.760
You cannot have your agent, just as you cannot have humans, accessing and reading and utilizing healthcare data that they don't absolutely have a necessity to use at that that point in time for that particular

s115
00:09:27.760 --> 00:09:28.960
journey.

s116
00:09:28.960 --> 00:09:40.000
And so, again, architecturally, when I think about how am I storing data within a particular system, I would like to think, "What is the shape of the data, what kind of characteristics

s117
00:09:40.000 --> 00:09:42.280
does the data have?

s118
00:09:42.280 --> 00:09:46.080
For health care data, that might be that it's very complicated.

s119
00:09:46.080 --> 00:09:51.520
It doesn't follow strict hierarchical um relationships.

s120
00:09:51.520 --> 00:09:55.160
It's uh sometimes unstructured and it's sometimes structured.

s121
00:09:55.160 --> 00:09:56.200
It could be very large.

s122
00:09:56.200 --> 00:10:03.160
For example, health care data but one piece of health care data can easily be over a megabyte in size or or much more than that.

s123
00:10:03.160 --> 00:10:05.040
Uh it has strict access controls.

s124
00:10:05.040 --> 00:10:12.360
As we've been saying, the R back comes into play like uh both for humans and and then for agents downstream of that.

s125
00:10:12.360 --> 00:10:21.600
Uh it may even be we I've seen customers where they're not willing to have their health care data leave their own environment, leave their on prem VPC for example.

s126
00:10:21.600 --> 00:10:25.400
So, we have tangential access to their to their data.

s127
00:10:25.400 --> 00:10:30.360
And so, an architectural paradigm I might go to is object storage.

s128
00:10:30.360 --> 00:10:33.600
Schema driven object storage, I think is a good fit for this.

s129
00:10:33.600 --> 00:10:40.200
It's m- matches well with the choice of using event logging because you can separate the two.

s130
00:10:40.200 --> 00:10:53.200
So, the events we talked about as the record of what the agent is doing at any particular time only contain references to the schema driven blobs that are the storage of the actual health care data itself.

s131
00:10:53.200 --> 00:11:04.080
And uh it's important therefore that the health care data is stored immutably again so that you can always go back in time and reconstruct what data the agent had access to at that particular point in time.

s132
00:11:04.080 --> 00:11:17.800
This separation of events for what happened and object storage for the data that was used at that particular point in time has actually some some very useful benefits.

s133
00:11:17.800 --> 00:11:32.480
For example, with a system like this, it's possible for developers to go back and debug and have observability over what happened, what particular steps the agent took, why it did that, and and retrace the agent's steps

s134
00:11:32.480 --> 00:11:37.280
without having access to the personal health information itself.

s135
00:11:37.280 --> 00:11:46.120
Although because of the schema driven, they can see the shape of that data, they they can't and to be honest, often won't be able to be given access to that health care data.

s136
00:11:46.120 --> 00:11:52.400
So, you can separate out observability and orchestration and instrumentation from the health care data itself.

s137
00:11:52.400 --> 00:11:57.600
And this then has another benefit, which is zero trust.

s138
00:11:57.760 --> 00:12:03.000
It it the object storage becomes a place where you can apply zero trust principles.

s139
00:12:03.000 --> 00:12:15.480
Your agents can bear tokens and use those tokens to access the data at the point of use and not allow data to flow around the system as it likes.

s140
00:12:15.480 --> 00:12:22.480
This then leads into a mitigation for prompt injection for the lethal trifecta.

s141
00:12:22.480 --> 00:12:31.040
The way I think about the lethal trifecta is can I solve for the constraint if I have an agent at point A with access to this data?

s142
00:12:31.040 --> 00:12:36.200
Is it possible within my architecture for the agent to be also accessing data over here?

s143
00:12:36.200 --> 00:12:48.760
And zero trust principles, tokens are bad by the agents and object storage segregated from the event stream that has your orchestration logic gives you a place to be able to solve for that constraint.

s144
00:12:48.760 --> 00:12:56.240
It won't be possible for the agent to access data within the same process that that you've given it the the previous data.

s145
00:12:57.880 --> 00:13:01.960
Okay, so then it comes to how do you handle escalation?

s146
00:13:01.960 --> 00:13:10.000
And in many scenarios, you will want to be able to escalate the decision that an agent makes or an action that an agent makes to a human.

s147
00:13:10.000 --> 00:13:12.720
But one of the challenges here is that this is quite dynamic.

s148
00:13:12.720 --> 00:13:16.160
So, you don't know in advance when exactly perhaps the agent's going to escalate.

s149
00:13:16.160 --> 00:13:19.880
It could be that you're asking the AI to escalate when it's not sure.

s150
00:13:19.880 --> 00:13:28.640
Um it could be that you define some sort of rules in your system, maybe in a medical context, the treatments going above a certain threshold means that it needs to be escalated

s151
00:13:28.640 --> 00:13:30.240
uh for an approval.

s152
00:13:30.240 --> 00:13:30.260
But

s153
00:13:30.260 --> 00:13:30.440
[snorts]

s154
00:13:30.440 --> 00:13:34.160
this makes it very challenging because of this this inability to predict.

s155
00:13:34.160 --> 00:13:40.120
And a second challenge is also that humans and LLMs ultimately process context differently.

s156
00:13:40.120 --> 00:13:45.640
You know, LLMs will have no problem if you give them massive massive amounts of text, but humans that's not the case.

s157
00:13:45.640 --> 00:13:57.200
So, what we've seen is that one pattern that can work very well here is if in your platform you enforce you kind of a wider definition of agent which encompasses both LLMs

s158
00:13:57.200 --> 00:14:03.560
and humans, then you can make it such that any action that can be taken by an LLM could also be taken by a human.

s159
00:14:03.560 --> 00:14:10.800
And this is helpful because at any point in the kind of chain of actions that your agent is taking, it can escalate to a human, the human could perform that action,

s160
00:14:10.800 --> 00:14:16.840
and then any step downstream doesn't care about whether it was a human or an LLM that did those actions upstream.

s161
00:14:16.840 --> 00:14:31.600
Um and on the second point around the context, what this also makes much easier is that you can define methods that take the context, which has some kind of shared definition of context,

s162
00:14:31.600 --> 00:14:35.560
which is irrespective of whether it's a human or an LLM that's going to be accessing it.

s163
00:14:35.560 --> 00:14:44.240
And you can take those methods to then map into something that's agent friendly, like a prompt, or into something that's more human friendly, for example, a UI.

s164
00:14:46.960 --> 00:14:52.960
And then on this fourth and final question that we're going to talk about, um evals.

s165
00:14:53.040 --> 00:14:55.160
Obviously, you know, we hear a lot about evals.

s166
00:14:55.160 --> 00:15:02.160
We know that evals can be very helpful, that often they drive decision-making about the types of model you want to use, the type of approach you might want to use

s167
00:15:02.160 --> 00:15:03.440
within your product.

s168
00:15:03.440 --> 00:15:07.040
But we also know that evals can be pretty hard, and there's various factors here.

s169
00:15:07.040 --> 00:15:07.185
We

s170
00:15:07.185 --> 00:15:07.200
[snorts]

s171
00:15:07.200 --> 00:15:14.760
know that LLMs are not deterministic, so it can be quite tricky to pin down the precise change that led to some sort of change in outputs.

s172
00:15:14.760 --> 00:15:28.520
Um we also know that the data that you might put in an offline data set might not necessarily represent production data, and it could be that um maybe you sampled from data, but actually that sample isn't truly representative.

s173
00:15:28.520 --> 00:15:35.840
And then you also have drift of data over time, so maybe your offline data set is now out of date.

s174
00:15:36.920 --> 00:15:50.920
And what we found is that these three primitives that we've described described so far in the talk actually give you effective privacy preserving evals almost as a byproduct without needing to kind of bolt something onto the side of your

s175
00:15:50.920 --> 00:15:52.320
architecture.

s176
00:15:52.320 --> 00:15:57.280
So to make that more concrete, so the immutable ledger, what this means is that you can replay your actions.

s177
00:15:57.280 --> 00:16:04.960
So you can go back to any particular time, you know, in this kind of sequence of events, you can see the complete state of the system at that point in time.

s178
00:16:04.960 --> 00:16:07.680
And if you wanted to, you could then make very specific tweaks.

s179
00:16:07.680 --> 00:16:16.320
So you could tweak a prompt, you could tweak a model, you could tweak the code, and you can see the exact direct impact of that because you have all of that context.

s180
00:16:16.320 --> 00:16:30.640
Secondly, you have this human agent equivalency, which means that for any task, you could get both the agent, the LLM agent, and the human to perform it, and your difference is your eval, that gives you the eval scores.

s181
00:16:30.640 --> 00:16:39.960
And then finally, what the object storage enables you to do is to actually run these evals on production data including inside your customer's environment without actually ever exposing that data.

s182
00:16:39.960 --> 00:16:46.720
You can get your eval results without the sensitive data ever needing to come to where your agent is performing the work.

s183
00:16:48.200 --> 00:16:58.960
Right, so we've gone through four architectural principles that we found useful for building in healthcare and more generally in regulated environments for enterprise.

s184
00:16:58.960 --> 00:17:16.439
The immutable ledger of actions, the orchestration adjacent object storage, the human agent equivalency, and the way that with these three principles evals can emerge as a first-class property of the system rather than as something you attach onto the side.

s185
00:17:16.439 --> 00:17:30.040
I think one of the matters here is that I like to think about architecture as taking your constraints very seriously and thinking about what you want to be simple within the system and then choosing the trade-offs

s186
00:17:30.040 --> 00:17:31.320
for that.

s187
00:17:31.320 --> 00:17:37.920
And of course, alongside that, some things will become hard, but it's the things that are simple that are most important to you.

s188
00:17:37.920 --> 00:17:42.800
And that there are patterns that already exist across enterprises that solve for a lot of these things.

s189
00:17:42.800 --> 00:17:56.800
And sure, with AI, we need to combine them in new, sometimes radical ways and bring in other way other pieces, but there are patterns that have worked very well within finance, within defense, within big tech that that can be applied to this kind of system architecture.

s190
00:17:56.800 --> 00:18:11.160
And I'd say the takeaway is that where I've seen it go wrong is taking that initial POC, that um that point solution that showed so much promise and that that showed the high accuracy, for example,

s191
00:18:11.160 --> 00:18:16.440
and then trying to build up from it, strapping on the enterprise requirements as you come across them.

s192
00:18:16.440 --> 00:18:25.320
Okay, we need eval, we need uh security, we need auditability, and bolting these on as additions to the the the foundations of the POC.

s193
00:18:25.320 --> 00:18:32.800
You end up with something very brittle, something very hard to uh uh externalize and to generalize across different use cases.

s194
00:18:32.800 --> 00:18:49.400
But where I've seen it go well is if you take the constraints of a production-ready, scaled enterprise uh system seriously from the beginning and treat those as the architectural principles that you're going to build everything upon and then build back up towards that POC accuracy

s195
00:18:49.400 --> 00:18:52.160
using your new primitives.

s196
00:18:53.280 --> 00:18:54.320
Thank you for your attention.

s197
00:18:54.320 --> 00:18:55.783
Thank you.

s198
00:18:55.783 --> 00:18:57.783
[applause]

s199
00:19:11.503 --> 00:19:13.503
[music]