WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7T-gldvgfpo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7T-gldvgfpo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.640 --> 00:00:13.600
My name is Dan.

s3
00:00:13.600 --> 00:00:15.600
I am the creator of Open Pros.

s4
00:00:15.600 --> 00:00:25.400
This year is Raymond I'm joined by Raymond and Jose two fabulous teammates, co-workers, just phenomenal creators and engineers all around.

s5
00:00:25.400 --> 00:00:27.480
I'll introduce them a little bit in a bit.

s6
00:00:27.480 --> 00:00:33.640
Uh just a little bit of background about what we're what we're about to show you here.

s7
00:00:33.640 --> 00:00:50.240
Uh 5 years ago in April of 2021, I was named the best prompter in the Eleuther AI back rooms Discord, which was a skill at the time that I thought would be tremendously valuable for many years afterward

s8
00:00:50.240 --> 00:00:57.720
and actually it lasted I think around like two or three years maybe was the lifetime of of that skill.

s9
00:00:57.720 --> 00:01:03.680
And I say that because a lot obviously, as you all know, has changed in the last 5 years.

s10
00:01:03.680 --> 00:01:08.960
But one thing has not changed and I've been to a lot of workshops today.

s11
00:01:08.960 --> 00:01:12.560
We've got a lot of amazing talks coming up.

s12
00:01:12.560 --> 00:01:23.640
But the one thing that seems to be ever present in in all of the work that we do and and that actually doesn't change with the latest harness or the latest model

s13
00:01:23.640 --> 00:01:38.080
is we all still need to communicate our intent to the thing and no matter how smart the thing gets, there is still this challenge of how do I get out of my brain the thing that I want and and get it into the

s14
00:01:38.080 --> 00:01:42.320
super intelligence that is just going to carry it away and and do everything for me.

s15
00:01:42.320 --> 00:01:58.120
Uh Open Pros is is a is a language, it's a framework, it's a started as a project based on my own love of language and my own strong belief that language is this really powerful

s16
00:01:58.200 --> 00:02:07.160
force of creation and I mean that like I know that's a little bit cliche, but that's like in the very literal sense now with with the capabilities that we have.

s17
00:02:07.160 --> 00:02:12.000
You can write a sentence and the things just make that sentence true.

s18
00:02:12.000 --> 00:02:21.840
And so the question then becomes like what is the sentence that you write now that you have this tremendously powerful machine to make it true.

s19
00:02:21.840 --> 00:02:31.640
And the skill of prompting from 5 years ago looks very different from the skill of what we like to think of as contract creation or contract composition today.

s20
00:02:31.640 --> 00:02:41.480
And when you were dealing with a model that was just completing the the next token, there was there was good patterns and bad patterns for for prompting.

s21
00:02:41.480 --> 00:02:53.320
A few years later, you were filling out context by hand and you were taking rag results and and tool calls and stuff and search results and stuffing them into into the context window and you call that context engineering.

s22
00:02:53.320 --> 00:03:02.440
And then the harness started doing that for you and you were building a harness and and and you were you were you thought of the harness as this this machine that was stuffing context for you.

s23
00:03:02.440 --> 00:03:14.000
And and in the last 8 10 months, we've seen a number of papers come out that is just pushing everything up another layer of abstraction and and the harnesses themselves are are

s24
00:03:14.000 --> 00:03:15.600
changing out from under you.

s25
00:03:15.600 --> 00:03:27.480
We saw a great presentation by Anthropic this morning on on flows and that's this idea of this like self-modifying harness topology that is managing sub agents in a way that just was

s26
00:03:27.480 --> 00:03:29.280
not thinkable even just a year ago.

s27
00:03:29.280 --> 00:03:37.880
So, if if is solved and the context is now solved and now the harnesses are now solved, like what's kind of the next step here?

s28
00:03:37.880 --> 00:03:44.160
And that's what we'd like to talk a little bit about today and show you some of the stuff we've been we've been building in in support of that.

s29
00:03:44.160 --> 00:03:55.360
I like to think of it as as once you can name the thing that you want, any harness can self-shape, can fan out, do any sort of loop to get you there.

s30
00:03:55.360 --> 00:04:00.400
And the question is what are the good best practices for naming the thing that you want.

s31
00:04:00.400 --> 00:04:02.680
And and we think of that as as declarative.

s32
00:04:02.680 --> 00:04:06.360
We're we're going to borrow concepts from classical software engineering.

s33
00:04:06.360 --> 00:04:08.320
I'm a software engineer of 12 years.

s34
00:04:08.320 --> 00:04:11.600
Jose and Raymond have been around the block with this as well.

s35
00:04:11.600 --> 00:04:16.040
Jose was early Google Cloud and he's informed a number of the ways we think about this.

s36
00:04:16.040 --> 00:04:25.200
Raymond is just a phenomenal uh uh He told me to He told me to introduce himself as a as a vibe code extraordinaire, but that's way underselling his his talents.

s37
00:04:25.200 --> 00:04:32.560
He's a PhD and two-time founder and has been doing harness engineering for basically as long as anybody I know.

s38
00:04:32.560 --> 00:04:39.240
Um and and so so it's this idea of borrowing old engineering patterns and applying them into this new world.

s39
00:04:39.240 --> 00:04:41.720
We want to write our prompts compositionally.

s40
00:04:41.720 --> 00:04:45.840
We want to compose composition over inheritance.

s41
00:04:45.840 --> 00:04:51.440
We want to write our prompts declaratively rather than procedurally or imperatively.

s42
00:04:51.440 --> 00:05:00.160
We want to tell it what what to do, not how to do it because the harness in its infinite wisdom and its infinite intelligence is going to figure that out.

s43
00:05:00.160 --> 00:05:07.919
And if we tell it how to do the thing, now we're actually leaving cost and and intelligence on the table because we could run it for a cheaper model.

s44
00:05:07.919 --> 00:05:10.480
A more intelligent model can figure out how to do the thing.

s45
00:05:10.480 --> 00:05:16.440
So it's this idea of write what you want, declare what you want, the outcome, the goal, the end state.

s46
00:05:16.440 --> 00:05:18.480
We think of that as a contract.

s47
00:05:18.480 --> 00:05:31.200
And and write it in a way such that you can compose these because when you build a system, when when you build an open framework like this, when you build a system that has many parts, maybe across your company or across your team,

s48
00:05:31.200 --> 00:05:39.720
you want to do things in a way that allows things to elegantly compose and that lets you manage complexity in in a in a more local way.

s49
00:05:39.720 --> 00:05:47.480
Um with that, I'm going to hand it over to Raymond who's going to tell you a little bit about how it works and then we're going to show you some some demos after that.

s50
00:05:47.480 --> 00:05:50.000
We'll we'll we'll open it up to questions throughout.

s51
00:05:50.000 --> 00:05:55.440
So, and please do raise uh your hand if you have questions, interject, please.

s52
00:05:55.440 --> 00:05:56.440
Awesome.

s53
00:05:56.440 --> 00:05:57.400
Thank you, Dan.

s54
00:05:57.400 --> 00:06:04.840
Okay, so there um I wanted to just help you understand Open Pros and put it in a quadrant.

s55
00:06:04.840 --> 00:06:12.240
Um so, if there's only two things that you remember, the two things that I want you to remember are it's just markdown.

s56
00:06:12.240 --> 00:06:16.280
So, we like to sometimes call it it's a programming language for agents.

s57
00:06:16.280 --> 00:06:20.760
We sometimes call it a framework, but all of it is just text.

s58
00:06:20.760 --> 00:06:22.480
It's just one giant prompt.

s59
00:06:22.480 --> 00:06:28.480
Um and then the second thing to just help you figure out in your mental bucket where this fits is it's event driven.

s60
00:06:28.480 --> 00:06:29.720
It's not chat.

s61
00:06:29.720 --> 00:06:31.080
We're not chatting with these things.

s62
00:06:31.080 --> 00:06:37.560
Um I'll chat with an agent that will make Open Pros programs later, but if you only remember two things about us, that's it.

s63
00:06:37.560 --> 00:06:42.840
It's it's all text and we're doing these event-driven multi-agent workflows, not chatting.

s64
00:06:42.840 --> 00:06:47.400
Um you can it's open source, you can run it locally with NPX skills.

s65
00:06:47.400 --> 00:06:53.680
Uh I'll walk through the hosted version that I've built uh run.pros.md, and uh we also will do it for you.

s66
00:06:53.680 --> 00:06:56.720
If you want to reach out, uh we'll help you out.

s67
00:06:56.720 --> 00:07:03.800
So, uh I guess quick question, like how many people in here are from the the finance side of things?

s68
00:07:03.800 --> 00:07:04.919
Okay, that's good.

s69
00:07:04.919 --> 00:07:08.280
How many are like AI engineers or developers?

s70
00:07:08.280 --> 00:07:09.120
Okay.

s71
00:07:09.120 --> 00:07:17.560
Then how many people um are maybe more like PMs or like professionals but and you're using AI but you don't know how to code?

s72
00:07:17.560 --> 00:07:19.240
Okay, fewer but Okay, good.

s73
00:07:19.240 --> 00:07:21.680
So, I've got why should I care for those three audiences?

s74
00:07:21.680 --> 00:07:33.840
So, for the small regulated industries, one of the things that we've seen um some banks and uh other financial services companies be excited about is like what I said before, it's all text.

s75
00:07:33.840 --> 00:07:37.880
So, we don't need the procurement department, we don't need the IT department.

s76
00:07:37.880 --> 00:07:49.160
You can use whatever coding agent you want, whatever inference you want, whatever guardrails you already have, whatever stack you have, we can actually like help you um you know, create these multi-agent workflows without

s77
00:07:49.160 --> 00:07:51.800
deploying anything to you other than some markdown files.

s78
00:07:51.800 --> 00:07:57.520
Um so, that's one of the the angles or or benefits of it being all text.

s79
00:07:57.520 --> 00:08:04.400
Um for the folks that are uh not writing code themselves, uh basically, if you can read logical English, you can read Open Prose.

s80
00:08:04.400 --> 00:08:09.040
So, we're going to very specifically not get into the syntax of the language.

s81
00:08:09.040 --> 00:08:19.560
Technically, it is a programming language and it has a syntax, um but it's really meant to be more um read by people and edited by people rather than like handwritten

s82
00:08:19.560 --> 00:08:21.040
from scratch by people.

s83
00:08:21.040 --> 00:08:25.040
Like you we don't want users concerned with like learning the schema.

s84
00:08:25.040 --> 00:08:35.560
And so, you can kind of describe the work, have your agent write the Prose program, you know, that might give you some more ideas for how you might want to edit it, or you might see something you want to change,

s85
00:08:35.560 --> 00:08:39.719
uh and then that Prose program becomes the reusable workflow.

s86
00:08:39.719 --> 00:08:59.600
Um and then for the AI engineers and the developers, uh I think the angle where you might want to use this is it's you know, it's not it's a framework that's not a framework, and we'll talk a little bit about that in a second, but even the compiler for the framework, even the virtual machine for the framework,

s87
00:08:59.600 --> 00:09:08.160
is itself just a giant prompt, and you're literally like incepting cod code into compiling the program for you, incepting Codex into doing that.

s88
00:09:08.160 --> 00:09:18.480
Uh so, there's uh you can express very, very complex multi-agent workflows that uh things like the Anthropic um uh dynamic What do they call them?

s89
00:09:18.480 --> 00:09:22.360
Dynamic agent workflows um would probably be like an analogy.

s90
00:09:22.360 --> 00:09:27.200
You can do that in any coding agent in just our markdown contract.

s91
00:09:27.200 --> 00:09:27.839
Uh okay.

s92
00:09:27.839 --> 00:09:34.360
So, first on this idea that it's just markdown, um this can be a complete program, just as an example.

s93
00:09:34.360 --> 00:09:47.000
And again, I don't want to get uh so hung up in the syntax, um but you can basically say, "Hey, here's the goal and like here is the the contract that that the that this maintains.

s94
00:09:47.000 --> 00:09:50.040
Like here's the definition of what this contract maintains."

s95
00:09:50.040 --> 00:09:54.280
Um and again, that's just like a very silly simple one.

s96
00:09:54.280 --> 00:10:00.760
Um the uh this idea here of like, well, what is it that you can declare?

s97
00:10:00.760 --> 00:10:12.200
So, again, I don't want to get into the schema and the types and the syntax, but what you or your agent can declare on behalf of you with uh in the CLI is called the pros write function.

s98
00:10:12.200 --> 00:10:16.520
In our app, it's basically just the agent that drives the whole app for you.

s99
00:10:16.520 --> 00:10:18.200
You can make sub agents.

s100
00:10:18.200 --> 00:10:23.040
Uh you can assign specific skills uh to those specific sub agents.

s101
00:10:23.040 --> 00:10:25.400
You can have specific tools that are available.

s102
00:10:25.400 --> 00:10:27.680
So, for example, for search, I love Exa.

s103
00:10:27.680 --> 00:10:30.400
I think they're amazing and I don't want it doing regular web search.

s104
00:10:30.400 --> 00:10:33.360
I always want it using Exa cuz it's faster and cheaper for me.

s105
00:10:33.360 --> 00:10:35.120
You can say, "Here's the skill for that.

s106
00:10:35.120 --> 00:10:36.280
Here's the CLI for that."

s107
00:10:36.280 --> 00:10:39.400
If you can't find those things, just don't run the program.

s108
00:10:39.400 --> 00:10:41.680
Like we're not doing it the other way.

s109
00:10:41.680 --> 00:10:45.440
So, you might have skills that are specific to your company or to your org.

s110
00:10:45.440 --> 00:10:54.160
And then essentially, you can think of it as like wiring that program together and like if it doesn't have all the dependencies, like do not compile, right?

s111
00:10:54.160 --> 00:11:02.680
Um and then obviously that what are the inputs of this program, what are the outputs that this uh program has to deliver, this contract has to deliver.

s112
00:11:02.680 --> 00:11:07.880
Um and again, like again, this could be something if people want to nerd out and ask Dan about how he built it.

s113
00:11:07.880 --> 00:11:10.760
Like the runtime itself is just a giant prompt, too.

s114
00:11:10.760 --> 00:11:19.320
So, um you know, some of my favorite sentences in the uh in the skill or the prompt is like "You are the compiler."

s115
00:11:19.320 --> 00:11:21.080
This is not an analogy.

s116
00:11:21.080 --> 00:11:22.600
You are the compiler.

s117
00:11:22.600 --> 00:11:24.640
And uh it seems to work.

s118
00:11:24.640 --> 00:11:30.280
Um So, uh yeah, and then this idea of a event-driven.

s119
00:11:30.280 --> 00:11:38.960
So, we want to get out of this idea that everything is a conversation, that everything is a chat, um and build event-driven systems.

s120
00:11:38.960 --> 00:11:52.360
So, um again, like uh just to compare these different workflow trees, it's like something something happens that might be just like a cron job on a schedule, that might be a new issue is opened in GitHub, or a new PR

s121
00:11:52.360 --> 00:11:55.000
uh is filed, or you have some webhook system, right?

s122
00:11:55.000 --> 00:12:11.960
Um that you're I don't know, Stripe isn't working, whatever, then you can now um you know, trigger that workflow that you've expressed uh to go handle that entire thing end to end and deliver on the contract.

s123
00:12:12.440 --> 00:12:22.560
Um and so, this idea of like starting to build towards jobs, so you can set this up yourself with your own agents in your own system, and what I built um in run.prose.md

s124
00:12:22.560 --> 00:12:26.840
is um this idea of like a job is a contract plus a trigger.

s125
00:12:26.840 --> 00:12:36.360
So, again, the the trigger can be something super trivial like a schedule, and so here I'm basically saying like, "Okay, every day uh run this."

s126
00:12:36.360 --> 00:12:51.160
Um but uh right now, the two the two main triggers that we have implemented in the app as is are um our webhooks, which really uh inevitably cover a lot of different things, like a whole Slack app is just webhooks, GitHub just webhooks.

s127
00:12:51.160 --> 00:12:53.120
Um and cron jobs.

s128
00:12:53.120 --> 00:12:57.560
We're happy to add more if you want to give us feedback on ones that we can make easier for you.

s129
00:12:57.560 --> 00:13:03.240
Um and again, from from that trigger, then you triage uh these workflows.

s130
00:13:03.240 --> 00:13:13.400
So, uh you can start to build out these, you know, code factories from um creating these standing jobs um of the Prose programs.

s131
00:13:13.520 --> 00:13:22.960
Uh what I want to show and I'll get towards in a second is um one of the things that I've been building to make sure that's ready in time for this event is

s132
00:13:22.960 --> 00:13:34.920
um is webhooks and using uh GitHub uh events specifically, and I've been forcing myself to use the program to develop the program.

s133
00:13:34.920 --> 00:13:40.640
And so, one of the examples that we now have in the issue in the system is, well, there's two different main ones.

s134
00:13:40.640 --> 00:13:47.600
One is like any GitHub issue that comes in, you write a prose program for how it's going to try to address the issue.

s135
00:13:47.600 --> 00:13:50.480
So, you can make that as complex or as simple as you want.

s136
00:13:50.480 --> 00:13:52.960
You can run it could be a multi-agent workflow.

s137
00:13:52.960 --> 00:13:58.120
It could have specific, um, skills that it has to follow or use models.

s138
00:13:58.120 --> 00:14:02.880
Um, and so, basically the input is whatever that issue was, the output is a commit.

s139
00:14:02.880 --> 00:14:05.680
And it has to go all the way to getting that thing done for you.

s140
00:14:05.680 --> 00:14:11.080
Uh, and then similarly, uh, and I'll show this one more specifically is like doing it with, uh, PR reviews.

s141
00:14:11.080 --> 00:14:15.200
So, you can do this event-driven kind of adversarial review.

s142
00:14:15.200 --> 00:14:24.160
And I don't know if you guys heard, um, in one of the things I thought was interesting this morning in the Anthropic workshop where they're talking about um, the challenges with self-review.

s143
00:14:24.160 --> 00:14:28.640
So, like, you're you're in Claude Code and you're like, "All right.

s144
00:14:28.640 --> 00:14:34.520
Here, I'm going to let you push to production, but just could you just check like one more time for me?"

s145
00:14:34.520 --> 00:14:39.120
And like, it's it's biased to, you know, think that it did a good job.

s146
00:14:39.120 --> 00:14:40.800
And so, so are humans, right?

s147
00:14:40.800 --> 00:14:51.720
And so, um, I'll show a workflow where essentially the agents that are building this get this backpressure from the app itself um, through through pull requests.

s148
00:14:51.760 --> 00:14:53.200
Um, okay.

s149
00:14:53.200 --> 00:15:01.400
So, while we transition to this, are there any questions now while we switch over to the to the web app?

s150
00:15:02.280 --> 00:15:05.480
Please, no question is too silly.

s151
00:15:05.480 --> 00:15:06.480
Is that a hand or a phone?

s152
00:15:06.480 --> 00:15:09.280
I can't see the lights here.

s153
00:15:09.400 --> 00:15:10.240
Maybe none.

s154
00:15:10.240 --> 00:15:10.720
Yeah, go ahead.

s155
00:15:10.720 --> 00:15:11.770
One over there.

s156
00:15:11.770 --> 00:15:12.160
[clears throat]

s157
00:15:12.160 --> 00:15:13.400
It's a great question.

s158
00:15:13.400 --> 00:15:14.080
Uh, I'll field this.

s159
00:15:14.080 --> 00:15:15.120
Yeah, you should take that.

s160
00:15:15.120 --> 00:15:16.160
Yeah.

s161
00:15:16.160 --> 00:15:24.440
Uh, so, we actually have this idea of earlier I alluded to the what and the how being two two different two different ideas.

s162
00:15:24.440 --> 00:15:30.840
Declare the outcome and then the how is a little bit you you leave it open, let the agent figure it out.

s163
00:15:30.840 --> 00:15:38.160
On that spectrum, you actually do, to your point, at at certain places want to weigh in on the how.

s164
00:15:38.160 --> 00:15:44.880
And in fact, we have a a feature a syntax that lets you pin the how.

s165
00:15:44.880 --> 00:15:47.800
And so you can initially declare it very loose.

s166
00:15:47.800 --> 00:15:53.880
You can let a very intelligent agent run and figure out how to get to the thing that you've declared.

s167
00:15:53.880 --> 00:15:58.040
And then once it does that, you can say here's the agent trace.

s168
00:15:58.040 --> 00:16:04.840
Let's take this trace and we will sort of codify it or or I don't know, shrink wrap it into a how.

s169
00:16:04.840 --> 00:16:10.280
And what that allows us to do is actually drive down the the the cost of completing the workflow.

s170
00:16:10.280 --> 00:16:18.720
So you you start with a very expensive model, figures it out, and then you sort of, as you say, make it more specific, more deterministic.

s171
00:16:18.720 --> 00:16:26.520
So you're kind of like shifting the the the the the spectrum from totally at just a what to to how how to do it.

s172
00:16:26.520 --> 00:16:29.600
And and the cost comes down as you do that.

s173
00:16:29.600 --> 00:16:43.720
So when we say it's all markdown, that's where you can start it, but then you can move your way to to uh more specific procedural imperative style instructions as well.

s174
00:16:43.720 --> 00:16:45.560
Yeah, I guess a couple comments on that.

s175
00:16:45.560 --> 00:16:53.040
So one, we do have a system where you can take like a golden session and like the like say make this into a reusable workflow.

s176
00:16:53.040 --> 00:16:55.800
Like that was the best cloud code session I've ever had.

s177
00:16:55.800 --> 00:16:59.520
Like I wanted to do cuz that's this is we get this all the time, right?

s178
00:16:59.520 --> 00:17:04.720
And I think there's there's a part of it where like these things are non-deterministic.

s179
00:17:04.720 --> 00:17:16.880
I think a lot of the quote-unquote pinned and stable endpoints people have shown actually are changing as all these companies are compute constrained, but I also just think there's this like changing expectations thing, right?

s180
00:17:16.880 --> 00:17:25.240
Which is like the first time you tried like whatever it was, like Fable or whatever, like blew your blew your mind and you're like, I want whatever.

s181
00:17:25.240 --> 00:17:26.439
This workflow is so amazing.

s182
00:17:26.439 --> 00:17:27.439
I want this every time.

s183
00:17:27.439 --> 00:17:35.240
And then you kind of like boil the frog and like have just different expectations and all of a sudden doesn't happen again and like some of that is just reliability.

s184
00:17:35.240 --> 00:17:40.640
And so so one thing you can do is like take a really good session and like codify it.

s185
00:17:40.640 --> 00:17:49.640
But then the other thing, just to give Dan a little bit more credit of like building the framework is like it's not just literally like the markdowns you see are like

s186
00:17:49.640 --> 00:17:51.840
the tip of the iceberg of what it's seeing.

s187
00:17:51.840 --> 00:18:03.040
So like the last time I checked the full like system prompt or skill or however you there's I mean it is a skill it is a system prompt is like close to

s188
00:18:03.040 --> 00:18:05.880
70,000 tokens, 80,000 tokens of like

s189
00:18:05.880 --> 00:18:06.200
longer.

s190
00:18:06.200 --> 00:18:07.200
We're trying to clean it up.

s191
00:18:07.200 --> 00:18:07.880
We're trying to

s192
00:18:07.880 --> 00:18:10.840
Yeah, well now that the agents are smarter we can make it a little bit shorter.

s193
00:18:10.840 --> 00:18:22.760
But we basically have this full type system that's like we're overriding markdown with a very very clear type system and a very specific prompt for how the agent is going to deliver this incredibly complex multi-agent

s194
00:18:22.760 --> 00:18:34.640
um product on the out um but maybe to be explicit about where we don't mess with it is like um So like Claude code when it runs it it has its own system for sub-agents.

s195
00:18:34.640 --> 00:18:35.800
We don't want to mess with that.

s196
00:18:35.800 --> 00:18:44.120
We don't want where we don't want our prompt to be um in any way like specific to Claude code cuz Codex has a totally different like set of sub-agents.

s197
00:18:44.120 --> 00:18:49.232
So the the we're like I don't know, a meta harness, like

s198
00:18:49.232 --> 00:18:49.360
[laughter]

s199
00:18:49.360 --> 00:18:49.760
whatever.

s200
00:18:49.760 --> 00:18:51.680
We're one layer we want you to bring your own harness.

s201
00:18:51.680 --> 00:18:52.960
We don't want to be the harness.

s202
00:18:52.960 --> 00:18:56.480
We want to be this uh thing that is compiled by the harness.

s203
00:18:56.480 --> 00:18:58.280
So I guess that's a couple pieces of

s204
00:18:58.280 --> 00:18:59.680
Does that answer your question?

s205
00:18:59.680 --> 00:19:01.440
um All right.

s206
00:19:01.440 --> 00:19:07.440
So if you want to try it, it's um it's here the QR code run.prose.md uh the QR code goes to that.

s207
00:19:07.440 --> 00:19:09.200
If you want to start playing around with it.

s208
00:19:09.200 --> 00:19:18.800
Um what I want to do in um showcasing this is to uh show off a couple of things.

s209
00:19:18.800 --> 00:19:31.560
So, the um the first thing is that um the So, this is the This is a Prose program in the edit mode, right?

s210
00:19:31.560 --> 00:19:39.840
It is technically Markdown and Dan's overridden like the H3 headers with all the types of this thing.

s211
00:19:39.840 --> 00:19:50.200
But, getting someone to like learn that syntax and write it from scratch, it's kind of a not really the point.

s212
00:19:50.200 --> 00:20:04.120
Um and so, what we um what we want to do is So, even though everything that executes this program is here, and in fact, I've for the real nerds in the audience, you can see that

s213
00:20:04.120 --> 00:20:10.720
I've over I've taken the front matter of the Markdown file, and that is actually all of the stuff that's happening in the cloud.

s214
00:20:10.720 --> 00:20:21.000
So, I could take this program and export it as a pure Markdown file and go send it to my friend, and they'll it'll upload in their cloud this the exact same way

s215
00:20:21.000 --> 00:20:26.000
um to keep it like composable and keep it true to that idea of that we're pure Markdown.

s216
00:20:26.000 --> 00:20:32.640
Um anyways, the whole idea is like even though that is the thing and we'd want you to be reading this thing, it's not like the entry point.

s217
00:20:32.640 --> 00:20:42.040
So, what I've built instead is um an agent which hopefully will talk to me here.

s218
00:20:42.040 --> 00:20:43.600
Hi, it's Prose.

s219
00:20:43.600 --> 00:20:45.400
What are you hoping to get done today?

s220
00:20:45.400 --> 00:21:01.920
Okay, so uh every time I have a new pull request, I want you to um write me a Prose program in my run-prose repo that will review it as if you are a a grumpy

s221
00:21:01.920 --> 00:21:08.040
senior developer who doesn't like AI-generated code.

s222
00:21:08.040 --> 00:21:09.400
Sure, I can do that.

s223
00:21:09.400 --> 00:21:11.520
Hang on.

s224
00:21:11.520 --> 00:21:12.680
Okay.

s225
00:21:12.680 --> 00:21:15.160
So, I'm going to mute myself.

s226
00:21:15.160 --> 00:21:16.880
So, it's it's talking.

s227
00:21:16.880 --> 00:21:18.680
It's going to try to figure this out.

s228
00:21:18.680 --> 00:21:28.960
And the the whole idea is like anything So, anything that can charge money or spend money, you have to confirm.

s229
00:21:28.960 --> 00:21:31.160
I think that's like a best practice.

s230
00:21:31.160 --> 00:21:35.240
Um Oh, yeah.

s231
00:21:35.600 --> 00:21:35.960
Great.

s232
00:21:35.960 --> 00:21:41.640
So, I had to confirm cuz that that action costs money for it to So, that's the pros rate function.

s233
00:21:41.640 --> 00:21:48.240
So, it basically takes my query and is going to generate a new program um from that including all the wiring.

s234
00:21:48.240 --> 00:21:49.920
This step is going to take a little bit longer.

s235
00:21:49.920 --> 00:21:53.480
So, while that's running, I'll show like the manual way of doing that.

s236
00:21:53.480 --> 00:21:55.640
Which is in the jobs tab.

s237
00:21:55.640 --> 00:21:57.320
I don't know why this is cutting out.

s238
00:21:57.320 --> 00:22:01.680
Um in the jobs tab, we can go here, make a new job.

s239
00:22:01.680 --> 00:22:05.000
The the uh on a schedule is just a cron job.

s240
00:22:05.000 --> 00:22:07.320
We have a a web hook that's open-ended.

s241
00:22:07.320 --> 00:22:15.960
So, like you can have yourself or your agent um either use our specific web hook or like wire our thing to listen to your web hook.

s242
00:22:15.960 --> 00:22:19.040
But for Git GitHub, um so, I'll do the same one.

s243
00:22:19.040 --> 00:22:22.200
So, for GitHub pull requests, we make that easy.

s244
00:22:22.200 --> 00:22:24.800
Um we're going to do it in the run.pros.

s245
00:22:24.800 --> 00:22:27.600
We're going to uh I've just pre-wired this contract.

s246
00:22:27.600 --> 00:22:30.080
It's like, "Okay, you're going to review my thing."

s247
00:22:30.080 --> 00:22:33.760
And then I can pick, you know, what uh model that they have.

s248
00:22:33.760 --> 00:22:35.600
We'll we'll add more uh later.

s249
00:22:35.600 --> 00:22:40.920
Right now, we have the the like top three from uh OpenAI.

s250
00:22:40.920 --> 00:22:44.840
Uh and then I would just uh create that and deploy that.

s251
00:22:44.840 --> 00:22:53.680
So, then the other thing while this um this one's working that I want to show is how I've actually been using that as back pressure.

s252
00:22:53.680 --> 00:22:59.600
So, the main thing Don't get too hung up on the um whatever, the content up here.

s253
00:22:59.600 --> 00:23:05.680
But the thing that I want to show you here is like the the the volume.

s254
00:23:05.680 --> 00:23:15.800
So, like in whatever that is, like this entire thing is goes back to 3 days ago, but then if I go here, like these are all still from 3 days ago or 4 days ago.

s255
00:23:15.800 --> 00:23:25.560
Or so, like in the span of a few days, this is doing like dozens and dozens and dozens of PRs, and every time I'm asking for a new feature about

s256
00:23:25.560 --> 00:23:38.760
this GitHub was webhooks feature that I wanted to have ready for this event, uh every single one of those is firing off like a series of review agents where it's like so some of them are simple like okay, this one clearly seems

s257
00:23:38.760 --> 00:23:40.240
to have worked right away.

s258
00:23:40.240 --> 00:23:42.000
So, there's only one round.

s259
00:23:42.000 --> 00:23:53.200
Uh some of these had, you know, 14 rounds of review before uh it felt like the reviewer agents were like aligned and it let this thing pass.

s260
00:23:53.200 --> 00:24:04.440
Um and even just today, like I noticed something that I didn't love about um how it was doing it, and I was able to um you know, basically say okay,

s261
00:24:04.440 --> 00:24:07.480
um on our staging server, you know, don't mess with production.

s262
00:24:07.480 --> 00:24:08.600
I'm about to give a talk.

s263
00:24:08.600 --> 00:24:16.720
On the staging server, I want you to like change the things the exact ways in which this will get triggered, but you have to use the system yourself.

s264
00:24:16.720 --> 00:24:22.400
So, um like both our dev version of this and our prod version of this are giving different reviews.

s265
00:24:22.400 --> 00:24:36.240
Um and now the agent that's running uh here, uh which we can see all the different jobs in the the runs tab here, essentially like cannot consider its job to be complete until

s266
00:24:36.240 --> 00:24:43.400
it's like um a a like um like my developer agents have like have to take that feedback into account.

s267
00:24:43.400 --> 00:24:53.200
And just to emphasize like what's different about something like this from any other chat-based or or agent-based flow that you've seen is all of these are getting compared against a contract.

s268
00:24:53.200 --> 00:25:06.200
So, you could think of it as a fresh session every time is getting benchmarked against this contract that you've written declaratively, which as as Raymond alluded to earlier, gets the models out of this like self-confidence

s269
00:25:06.200 --> 00:25:09.720
loop where you don't want the same model checking the thing that was done.

s270
00:25:09.720 --> 00:25:16.480
So, you're spawning new sessions and and bouncing them off of a contract, which is I just a better uh pattern for for how to do it.

s271
00:25:16.480 --> 00:25:16.920
Yeah, go ahead.

s272
00:25:16.920 --> 00:25:18.040
Yeah.

s273
00:25:18.040 --> 00:25:22.160
But, doesn't [clears throat] it seem like a great opportunity for one-shot classifiers like Jeff?

s274
00:25:22.160 --> 00:25:22.800
Yeah.

s275
00:25:22.800 --> 00:25:25.160
So, they um

s276
00:25:25.160 --> 00:25:27.240
We have a lot of good stuff in the pipe.

s277
00:25:27.240 --> 00:25:31.200
So, so the idea of checking against a contract and and verifying it.

s278
00:25:31.200 --> 00:25:32.040
Sorry, Raymond.

s279
00:25:32.040 --> 00:25:32.240
Yeah.

s280
00:25:32.240 --> 00:25:44.480
Uh it it is super exciting on these decision models where uh you you have a contract, which is some ideal state, and then you have your messy real-world state, and you can apply a general class

s281
00:25:44.480 --> 00:25:48.520
decision model like like Jeff to say, is this is this passing or not?

s282
00:25:48.520 --> 00:25:49.280
So, I think

s283
00:25:49.280 --> 00:25:50.240
in this, but it's coming.

s284
00:25:50.240 --> 00:26:01.200
Yeah, and I think I think without um like Dan, again, we're going a lot off script, but it's a great question, and uh the great Jeff-ification is upon us.

s285
00:26:01.200 --> 00:26:03.480
So, we need to speak to this.

s286
00:26:03.480 --> 00:26:18.600
Um is uh I think Dan and I both been thinking about this a lot and realized that like um OpenPros itself, the language that's all markdown, I think does not have any place in

s287
00:26:18.600 --> 00:26:21.560
the Jeff land because they're not generative.

s288
00:26:21.560 --> 00:26:30.600
Like, so the whole idea of the lang- like it generating a language and understanding a language, like it uh it's not that.

s289
00:26:30.600 --> 00:26:39.880
But, I think the the two places where we think it would fit really well, and in particular with the hosted run.pros, is one on the back end.

s290
00:26:39.880 --> 00:26:46.200
So, the the downside or the weakness of OpenPros is it is you are letting the agent say, "I have met the contract.

s291
00:26:46.200 --> 00:26:47.200
Here it is."

s292
00:26:47.200 --> 00:26:47.520
Right?

s293
00:26:47.520 --> 00:27:00.720
And so, I'll often say to people uh like the intent behind OpenPros is similar to something like DSPy where you want to have like this abstraction layer over which you can express very complex

s294
00:27:00.720 --> 00:27:01.960
workflows.

s295
00:27:01.960 --> 00:27:04.560
The implementation is the exact opposite of DSPy.

s296
00:27:04.560 --> 00:27:10.240
DSPy is like every LLM call is a black box and you you isolate the crap out of that thing.

s297
00:27:10.240 --> 00:27:11.800
It's all deterministic code.

s298
00:27:11.800 --> 00:27:16.440
It's all typed interfaces to like only let the black box do that thing.

s299
00:27:16.440 --> 00:27:21.920
Dan's approach with open prose is the exact opposite which is like AI is getting smart enough to be the compiler.

s300
00:27:21.920 --> 00:27:25.120
Like let everything be in fuzzy land.

s301
00:27:25.120 --> 00:27:34.320
So the the the weakness of it though is like if you run it on Haku or you run it on Fable you're going to get a very different result and and you're also letting it say I'm done.

s302
00:27:34.320 --> 00:27:46.960
So I think a very natural place for like uh Jev or these new like decision models or whatever we want to call them system one models is like you were saying like as a verifier on the end of the job like very very fast and cheap.

s303
00:27:46.960 --> 00:27:52.520
Um and then the other thing that I want to build that's relevant to this this is like a router.

s304
00:27:52.520 --> 00:28:00.040
So like why would I have to go click the button that like oh issues go to the issue job but PRs go to the PR review job.

s305
00:28:00.040 --> 00:28:09.800
Why don't I just say like okay all the GitHub events that come in like you know solve the issues review the PRs right?

s306
00:28:09.800 --> 00:28:10.440
Another question.

s307
00:28:10.440 --> 00:28:11.960
Don't give them any more ideas about the game.

s308
00:28:11.960 --> 00:28:12.320
Okay.

s309
00:28:12.320 --> 00:28:13.849
I know I was going to say I was going to

s310
00:28:13.849 --> 00:28:14.080
[laughter]

s311
00:28:14.080 --> 00:28:15.120
Thank you.

s312
00:28:15.120 --> 00:28:19.440
It's just topical on the Twitter sphere at the exact moment that like why wouldn't you just RL this?

s313
00:28:19.440 --> 00:28:22.720
Like if I got all these beautiful runs here against the contract why am I not just RLing this?

s314
00:28:22.720 --> 00:28:30.760
So So one of the ideas back to the language question is one of the ideas of like why why are we doing this at all in language?

s315
00:28:30.760 --> 00:28:42.200
Like the original human side of this is like you can commit it to get and you can version control it or you can you know use it in notion or or Jira and and have anybody on your team using it.

s316
00:28:42.200 --> 00:28:49.840
But from from a engineering side it's the fact that it's all text is like perfect and beautiful for for RL.

s317
00:28:49.840 --> 00:28:59.480
So, we're not doing that right now, but like that's absolutely something we're talking with various prospects about is like once you have all of your stuff written as text,

s318
00:28:59.480 --> 00:29:00.280
this is like

s319
00:29:00.280 --> 00:29:00.680
Yeah.

s320
00:29:00.680 --> 00:29:02.720
a whole lot better for for RL.

s321
00:29:02.720 --> 00:29:13.040
Yeah, we haven't hit volumes where this is important, but like if you are hosting the model and you have this 100,000-ish token prefix, then you can like just it

s322
00:29:13.040 --> 00:29:17.440
or you like, you know, build these um more token efficient representations.

s323
00:29:17.440 --> 00:29:24.560
So, um Yeah, if someone wants to do a trillion tokens in Open Pros, we'll happily just it for you.

s324
00:29:24.560 --> 00:29:26.920
Um So, anyways, to go back, thank you.

s325
00:29:26.920 --> 00:29:28.640
Those are Those are really good ones.

s326
00:29:28.640 --> 00:29:32.440
We now have our grumpy senior pull request review.

s327
00:29:32.440 --> 00:29:36.000
Um and again, like the key idea is so the agent did all that for us.

s328
00:29:36.000 --> 00:29:46.880
The key idea here is um uh like the by deploying this job, like this will now start spending money when whenever we get a new PR, so I have to click that.

s329
00:29:46.880 --> 00:29:48.640
So, then I'm going to confirm that.

s330
00:29:48.640 --> 00:29:57.440
And then once this is deployed, it'll go when I go back over to my jobs, it will um it will look exactly like uh let's see.

s331
00:29:57.440 --> 00:30:02.640
So, if we just Oops, I did switch that.

s332
00:30:02.840 --> 00:30:09.720
Once this loads, we should see and the grumpy one is at the top or the bottom.

s333
00:30:10.200 --> 00:30:10.840
Yeah, here we go.

s334
00:30:10.840 --> 00:30:12.840
Grumpy AI skeptic PR review.

s335
00:30:12.840 --> 00:30:16.480
So, um you can see I was practicing earlier.

s336
00:30:16.480 --> 00:30:18.920
Um Okay.

s337
00:30:18.920 --> 00:30:22.400
So, let's see what in terms of this.

s338
00:30:22.400 --> 00:30:23.920
I think those are the main things.

s339
00:30:23.920 --> 00:30:31.360
Um Yeah, I'm in charge of run.pros.md, so if you run into any issues, don't tell my boss, you just send it straight to me.

s340
00:30:31.360 --> 00:30:34.760
Uh and my agents will uh take care of it for you.

s341
00:30:34.760 --> 00:30:42.760
Um but yeah, I'm really excited about this as just like a a simple, easy-to-use um um way for having these like event-based systems.

s342
00:30:42.760 --> 00:30:46.480
If you want to see other triggers or other features then that's really fun for me.

s343
00:30:46.480 --> 00:30:56.160
But again, you don't need to use the hosted thing at all and like you can uh you can use the um the skill instead NPX skills add open prose prose.

s344
00:30:56.160 --> 00:31:11.240
Um but now uh with that Dan is going to um go through a specific example that is for [clears throat] our finance friends um and we're going to show that one

s345
00:31:11.240 --> 00:31:19.360
uh in Codex and then he'll use that as an opportunity to talk a little bit more about how all this works under the hood.

s346
00:31:19.360 --> 00:31:19.800
Yep.

s347
00:31:19.800 --> 00:31:20.560
Thank you.

s348
00:31:20.560 --> 00:31:21.680
Sure.

s349
00:31:21.680 --> 00:31:26.680
Um so getting that working in the cloud is is interesting.

s350
00:31:26.680 --> 00:31:43.760
One of the more interesting use cases that we've been talking about with various customers and prospects is um you can this is this is a framework or a language or pattern that you can build in house and you can use you can run it on prem.

s351
00:31:43.760 --> 00:31:46.120
The whole idea here is that you can run it on prem.

s352
00:31:46.120 --> 00:31:48.720
You don't need to use our cloud version of it.

s353
00:31:48.720 --> 00:31:53.520
We'd love you to try it out in the cloud and and and and play with it get a get a feel for that.

s354
00:31:53.520 --> 00:32:04.760
Um but if this is something that you want to implement uh inside your organization the most interesting use case that we're seeing is on top of open models on top of your own harness bring your own harness

s355
00:32:04.760 --> 00:32:10.880
and then use this as kind of the application layer on on top of that and that lets you build this full stack in house.

s356
00:32:10.880 --> 00:32:13.720
You don't have to ship any of your data out to the labs.

s357
00:32:13.720 --> 00:32:24.080
More important than your data, you don't have to ship out your institutional We like to think of open prose actually as like institutional operational knowledge as documentation, right?

s358
00:32:24.080 --> 00:32:26.080
Think of it as like standard operating procedures.

s359
00:32:26.080 --> 00:32:32.320
So even more important than your data is you don't want to be shipping your standard operating procedure out to the big labs.

s360
00:32:32.320 --> 00:32:40.080
Um and so so this fits really nicely in this stack of of open open source application layer sits on top.

s361
00:32:40.080 --> 00:32:43.280
We we publish this open open core.

s362
00:32:43.280 --> 00:32:44.880
Uh it has a permissive license.

s363
00:32:44.880 --> 00:32:46.360
You can bring it in house and and use it.

s364
00:32:46.360 --> 00:32:49.520
We have a number of big companies who are are already doing that.

s365
00:32:49.520 --> 00:32:52.600
Um and and you can bring your own harness.

s366
00:32:52.600 --> 00:32:55.680
You can use it with the approved models that your company already has.

s367
00:32:55.680 --> 00:33:00.680
Uh so you don't have to feel like you're throwing data or your organization's knowledge over the wall.

s368
00:33:00.680 --> 00:33:06.440
Um this is just I mean a a we can get into this a little bit and I want to field more questions.

s369
00:33:06.440 --> 00:33:15.760
Um this is a a workflow that we're building out for a customer um that is interested in using OpenProse to document their quant models.

s370
00:33:15.760 --> 00:33:29.440
So uh for those of you who are familiar, um quant models have all sorts of regulatory um um commitments whereby every every 3 years or every 1 year or on certain event changes,

s371
00:33:29.440 --> 00:33:33.520
you need to update the the model documentation for these quant models.

s372
00:33:33.520 --> 00:33:42.440
So this is normally a process that we found in our experience that is uh human white collar, very smart person intensive, right?

s373
00:33:42.440 --> 00:33:50.840
These people uh maybe some of you probably spend a a share of your your year going through and and writing documentation.

s374
00:33:50.840 --> 00:34:06.640
That documents what your model's being used for, under what conditions was it tested, under um uh who who what other models reference it upstream, and a whole slew of other required These are obviously required by the regulators and your internal auditors and your internal compliance.

s375
00:34:06.640 --> 00:34:12.440
Um so this is this is a a flow that that we've found success running.

s376
00:34:12.440 --> 00:34:19.200
Uh it's it's perfect because in some ways the regulatory docs are like an OpenProse program, right?

s377
00:34:19.200 --> 00:34:23.399
And so this these are requirements written as prose.

s378
00:34:23.399 --> 00:34:29.879
Um and and so I'm going to kick off You can see here uh we're in Codex.

s379
00:34:29.879 --> 00:34:32.040
You You run this in Cloud Code.

s380
00:34:32.040 --> 00:34:33.440
You could run this on your own harness.

s381
00:34:33.440 --> 00:34:36.360
The whole point is that this is harness agnostic.

s382
00:34:36.360 --> 00:34:40.040
Uh so we we just say boot the kernel.

s383
00:34:40.040 --> 00:34:43.600
Uh and then pros run and then a specific program.

s384
00:34:43.600 --> 00:34:50.919
And so I'm going to kick this off and while it runs I can show you like what the kernel looks like uh and then what the program itself looks like.

s385
00:34:50.919 --> 00:34:54.159
Um just go back to Chrome.

s386
00:34:54.240 --> 00:34:54.720
And then

s387
00:34:54.720 --> 00:34:55.919
Right there.

s388
00:34:55.919 --> 00:34:58.040
It's uh that one.

s389
00:34:58.040 --> 00:34:59.960
So this is the kernel.

s390
00:34:59.960 --> 00:35:01.920
Um it's markdown.

s391
00:35:01.920 --> 00:35:05.080
It's not intended to be read by you which is why it's formatted like this.

s392
00:35:05.080 --> 00:35:07.960
We originally had it in a nice looking format and people complained.

s393
00:35:07.960 --> 00:35:09.760
They're like, "What is this?"

s394
00:35:09.760 --> 00:35:12.000
Uh it's meant to be read by the agents.

s395
00:35:12.000 --> 00:35:13.680
Uh and it's just markdown.

s396
00:35:13.680 --> 00:35:16.280
It's operating instructions for what to expect.

s397
00:35:16.280 --> 00:35:27.960
You could think of it a lot like a a bootloader or a resolution path, a an index uh system, a way of finding other documents that you that that the that the agent needs.

s398
00:35:27.960 --> 00:35:36.320
Um and then the program itself looks and again this is what I referenced in Codex looks very similar.

s399
00:35:36.320 --> 00:35:38.600
So this is again markdown.

s400
00:35:38.600 --> 00:35:45.800
It contains a whole slew of the requirements that the regulators and the compliance and the auditors are looking for.

s401
00:35:45.800 --> 00:35:54.760
Uh and this specifies, you know, uh the ideal state of of the end result is going to comply with this contract.

s402
00:35:54.760 --> 00:35:56.800
Uh and so those are the only two things you pass in.

s403
00:35:56.800 --> 00:35:59.160
You just pass in the kernel which is published publicly.

s404
00:35:59.160 --> 00:36:10.160
You can all download that right now uh and and and try it out and then write any program that you want, any contract that requires some complex um set of conditions to satisfy

s405
00:36:10.160 --> 00:36:18.400
and then the agents are just going to chug and and the you know, the better the harness, the better the model, the the faster and the cheaper that they will produce the output.

s406
00:36:18.400 --> 00:36:26.360
Uh but the idea is that that you're you're declaring the the result rather than declaring how to how to go about about doing it.

s407
00:36:26.360 --> 00:36:29.240
Um Let's see we can poke back and see.

s408
00:36:29.240 --> 00:36:34.280
We ran this with a a relatively cheap model here so that it would run faster.

s409
00:36:34.280 --> 00:36:36.200
Uh but it's still working.

s410
00:36:36.200 --> 00:36:40.160
Um Oh, it looks like it's finishing up.

s411
00:36:40.160 --> 00:36:42.400
So, this is I mean this is the document that it produced.

s412
00:36:42.400 --> 00:36:48.240
So, in this case we're we're we're um creating documentation for the sofa curve.

s413
00:36:48.240 --> 00:36:49.720
It's a public uh model.

s414
00:36:49.720 --> 00:36:54.760
So, we're not showing any any uh internal uh bank data here.

s415
00:36:54.760 --> 00:36:58.000
Um and this I don't know.

s416
00:36:58.000 --> 00:36:58.560
Raise a hand.

s417
00:36:58.560 --> 00:37:01.680
Anybody Are there any like quants in the room?

s418
00:37:01.680 --> 00:37:02.960
To review it?

s419
00:37:02.960 --> 00:37:05.520
How does this How does this look?

s420
00:37:05.520 --> 00:37:05.715
Yeah,

s421
00:37:05.715 --> 00:37:05.800
[laughter]

s422
00:37:05.800 --> 00:37:06.120
all right.

s423
00:37:06.120 --> 00:37:08.800
Well, we'll come we'll come we'll come find you afterwards.

s424
00:37:08.800 --> 00:37:16.400
Um but but so that you know this is this is this is documentation that has to be produced um regularly for every model.

s425
00:37:16.400 --> 00:37:20.040
A big bank has 4,000 or so models.

s426
00:37:20.040 --> 00:37:29.800
They have to be re re re-documented yearly and uh this is a this is a a a problem that is worth anywhere between 10 to 50 million dollars a year

s427
00:37:29.800 --> 00:37:37.240
uh for some of these bigger banks and and we're having great success in in um in doing it for them for for much cheaper.

s428
00:37:37.240 --> 00:37:50.000
Uh granted there are some some um Getting this through compliance and the auditors and the regulators is is another part of the battle obviously as I'm sure you're all familiar with but um

s429
00:37:50.000 --> 00:37:53.560
we we can benchmark this against historical results.

s430
00:37:53.560 --> 00:38:08.400
We can benchmark this against how long it takes the teams to do it uh and we have quant you know quants who are on the floor who are able to to rate these and and give us a sense of how how well the models are doing on this and we're finding that it's I mean quite shocking shockingly

s431
00:38:08.400 --> 00:38:09.160
good.

s432
00:38:09.160 --> 00:38:15.480
Um for for much cheaper than than traditional uh manual way of doing this.

s433
00:38:15.480 --> 00:38:16.360
Any questions?

s434
00:38:16.360 --> 00:38:16.840
Any like

s435
00:38:16.840 --> 00:38:19.560
the quants do the stuff that the quants are good at.

s436
00:38:19.560 --> 00:38:20.320
Making the models.

s437
00:38:20.320 --> 00:38:23.680
Not the regulatory paperwork which probably isn't very fun.

s438
00:38:23.680 --> 00:38:26.480
I'm not a quant, but

s439
00:38:26.936 --> 00:38:27.360
[snorts]

s440
00:38:27.360 --> 00:38:28.800
Any questions?

s441
00:38:28.800 --> 00:38:29.720
Great question.

s442
00:38:29.720 --> 00:38:31.160
Uh yes.

s443
00:38:31.160 --> 00:38:42.720
And so, actually, um Raymond showing off the chat agent a little bit as part of the kernel, as buried in the markdown at a at a path, we have uh best practices.

s444
00:38:42.720 --> 00:38:47.360
The These are the types of things you want to be doing when you're declaring out out outcomes.

s445
00:38:47.360 --> 00:38:51.120
Uh these are um and anti-patterns, things that you shouldn't be doing.

s446
00:38:51.120 --> 00:39:03.960
And so, yes, there is there is a a world in which you're not typing the whole contract itself, but you're iterating, and and that's what Raymond was showing uh with another agent

s447
00:39:03.960 --> 00:39:08.520
following our best practice guide on how to on on how to create the contract.

s448
00:39:08.520 --> 00:39:14.400
But, yeah, I mean, getting getting intent out of people's heads is like the hard part, and and I think this is a piece of it.

s449
00:39:14.400 --> 00:39:15.720
It's certainly not like the whole

s450
00:39:15.720 --> 00:39:16.480
Yeah.

s451
00:39:16.480 --> 00:39:20.000
I have a slightly like longer uh I don't want to make this too long.

s452
00:39:20.000 --> 00:39:20.880
We only have 5 minutes.

s453
00:39:20.880 --> 00:39:35.520
But, like, a slightly longer rant on this, which is that I do think that this is a big part of why chat like blew up so fast is like you do not need to know anything about what you're looking for, just start engaging with ChatGPT, and just like

s454
00:39:35.520 --> 00:39:41.040
the magic of generative AI and like having it like come back to you, right?

s455
00:39:41.040 --> 00:39:54.080
And so, like, if that's I don't I don't know that OpenProse is the best place if you're at that stage of something where you re you really don't even know the shape of what you're talking about, or you're not even sure

s456
00:39:54.080 --> 00:39:56.800
like what it is that you're looking for.

s457
00:39:56.800 --> 00:40:06.360
Um and like this go I'm like a very old-school DSPy user, and like my my joke about DSPy was like, "Oh, no, I have to know what I want now, right?"

s458
00:40:06.360 --> 00:40:11.560
Because like the whole idea of DSPy is like you have to you create the signature of the thing that you want.

s459
00:40:11.560 --> 00:40:16.160
And if you and you you you like see something like that, you're like, "This is so awesome."

s460
00:40:16.160 --> 00:40:21.160
And then you sit down, and you're like, oh I I actually don't know what I want to do I, right?

s461
00:40:21.160 --> 00:40:32.720
So, I think I think with the you know, Open Prose is a lot less strict than something like DS fire where you like you literally really needed to say the signature of the return object or nothing happens.

s462
00:40:32.720 --> 00:40:41.400
Um but yeah, I would say like the built-in agent hopefully can like help with some of like you can just turn on the voice and be like, what can you do?

s463
00:40:41.400 --> 00:40:51.840
Like and it can do almost anything you can do in the app like we you know, we have those hard-coded like confirm things for um for like anything that would cost money.

s464
00:40:51.840 --> 00:40:54.480
Obviously, we want people like click a button for that.

s465
00:40:54.480 --> 00:41:06.640
Um but yeah, I think I think it's um I don't know, I think it's also related to like the great jobification is like there are like chat isn't a great

s466
00:41:06.640 --> 00:41:22.480
um interface and agents that are like RLF to be convincing chat partners are not necessarily like the best type of AI for workflows that don't have human conversation as part of them at all.

s467
00:41:22.480 --> 00:41:22.880
Um

s468
00:41:22.880 --> 00:41:33.640
And let me let me interject, which is just use cases like we tell people you don't want to use this if if you're doing a one-off, if you are like chat is perfect for that.

s469
00:41:33.640 --> 00:41:36.040
You're if you're in exploratory mode.

s470
00:41:36.040 --> 00:41:39.360
What this is good for is something you need to do a lot.

s471
00:41:39.360 --> 00:41:44.280
It it needs to be reliable, it has to be repeatable, it needs to be auditable.

s472
00:41:44.280 --> 00:41:53.720
Those types of workflows are what what we're what Open Prose is built for and meant for and so there is an implicit like you do know what you need

s473
00:41:53.720 --> 00:41:54.760
in in here.

s474
00:41:54.760 --> 00:41:57.240
Oh, we got a question from a real quant.

s475
00:41:57.240 --> 00:42:01.400
And earlier I alluded to and I I wouldn't just call it inheritance, I would call it composition.

s476
00:42:01.400 --> 00:42:14.080
So, we we in fact a core feature of of the framework and and the the pattern itself is it's built to be composed and so yes, injecting multiple contracts

s477
00:42:14.080 --> 00:42:24.240
into the same uh run in in which the agent now has to satisfy many of them is is is a a core piece here.

s478
00:42:24.240 --> 00:42:30.120
The other half the event-driven half uh which maybe looks a little bit you might call it inheritance.

s479
00:42:30.120 --> 00:42:38.480
I I think of it more like um uh I don't know if you're familiar with like React React-style memoization where you have certain components.

s480
00:42:38.480 --> 00:42:40.120
It's still composition.

s481
00:42:40.120 --> 00:42:50.000
You have certain components that have upstream dependencies and when the data on on one of them changes, it's of course going to trickle down to everything below it.

s482
00:42:50.000 --> 00:43:02.120
Um that is another place where Jev is super exciting and and the decision model super exciting because we had built that system and we had agents trying to determine has the state changed

s483
00:43:02.120 --> 00:43:07.440
and if the state on upstream dependency changes, do we kick off downstream dependency?

s484
00:43:07.440 --> 00:43:17.880
Uh but that's very expensive to run and Jev becomes suddenly this very cheap like semantic classifier or semantic memoizer, which you could think of it as like a cache

s485
00:43:17.880 --> 00:43:22.920
and Jev is deciding when something is out of date and now it kicks off everything downstream from that.

s486
00:43:22.920 --> 00:43:27.320
So, that's another kind of core component of of the framework.

s487
00:43:27.320 --> 00:43:30.280
Minus the Jev part, we're still working we're still building that part in.

s488
00:43:30.280 --> 00:43:32.240
Yes, and there is no magic here.

s489
00:43:32.240 --> 00:43:33.240
It's all just English.

s490
00:43:33.240 --> 00:43:34.800
It's all just language.

s491
00:43:34.800 --> 00:43:40.000
Uh yeah, it's but yes, and that's where the composition composability comes in.

s492
00:43:40.000 --> 00:43:43.120
The other piece to your point is like it's not just a template.

s493
00:43:43.120 --> 00:43:53.600
So, like you could obviously even pre-LLMs, you could have um or very let's say not very early LLMs or even like encoders could like fill a template for you.

s494
00:43:53.600 --> 00:43:56.760
Like you like prefix the template and like fill the thing.

s495
00:43:56.760 --> 00:43:58.960
But here it's like, "Well, what can you template?"

s496
00:43:58.960 --> 00:44:02.920
And so, you could template the entire multi-agent workflow.

s497
00:44:02.920 --> 00:44:09.720
So, like this the simple example that I give is you could say like the output that I need is an article.

s498
00:44:09.720 --> 00:44:14.400
You know, and maybe there's it's got like this type of article, it's going to be this length, whatever, right?

s499
00:44:14.400 --> 00:44:16.200
Some constraints around that.

s500
00:44:16.200 --> 00:44:18.400
And the input is like the topic.

s501
00:44:18.400 --> 00:44:30.640
Like until you've got a good article for me, and the parent agent decides good, it decides when to exit, so that's where you're going to get a better result with a stronger model than a weaker one.

s502
00:44:30.640 --> 00:44:34.360
I've got a researcher sub-agent.

s503
00:44:34.360 --> 00:44:42.120
That agent has my access CLI and my access skill on how to use a CLI cuz I don't want it, you know, using any random web search.

s504
00:44:42.120 --> 00:44:50.600
I've got a writer sub-agent that has my special like here's the last 10 articles I wrote that are really good, and you I want you to write like this.

s505
00:44:50.600 --> 00:44:56.640
I've got a copy editor uh sub-agent, and that whole thing is just maybe

s506
00:44:56.640 --> 00:45:01.120
But the really important 100 lines of markdown, and it might take them 10 hours to do it.

s507
00:45:01.120 --> 00:45:08.080
important parts to emphasize there is is the contract composition where so you you you go to like, I mean, these things have been around forever.

s508
00:45:08.080 --> 00:45:12.240
Any Any Any workflow builder, drag-and-drop workflow builder.

s509
00:45:12.240 --> 00:45:27.520
That's a different way of thinking about this because when you think about this in terms of that like DAG directed acyclic graph of contracts, you can have intermediary steps that you almost think of them as like checkpoints or like

s510
00:45:27.520 --> 00:45:37.600
this intermediary step meets the contract, and the thing is going, and it hits this contract and fulfills this one, and then that causes the next one to keep going until it's fulfilled that contract.

s511
00:45:37.600 --> 00:45:49.840
So, you can you can still in some ways declare the how to get there just by composing these intermediary contracts along the way and still letting the agents be a little bit flexible on on how to do

s512
00:45:49.840 --> 00:45:51.200
achieve that intermediary step.

s513
00:45:51.200 --> 00:45:52.600
Yeah, and and the shape of it.

s514
00:45:52.600 --> 00:46:00.880
So, like one one example like when Claude um the dynamic workflows came out, they had this blog post that's like, here's the six types, right?

s515
00:46:00.880 --> 00:46:05.000
And it's like one of them's like fan out, fan in, and like a map reduce with sub agents.

s516
00:46:05.000 --> 00:46:11.600
One of them's like a adversarial review, one's like a loop until X condition is met, right?

s517
00:46:11.600 --> 00:46:17.640
And all of those things you can express in a prose program and then run them on on any agent.

s518
00:46:17.640 --> 00:46:26.520
So, you can like you can template the shape of the the multi-agent workflow um with no code.

s519
00:46:26.520 --> 00:46:35.760
Um and again, the whole it's only possible because the the agents are like good at managing themselves and because most harnesses have sub agents.

s520
00:46:35.760 --> 00:46:45.960
So, like if you're running plain vanilla pie, which has no sub agents at least last time I checked, you're going to want sub agents for it to do that more complex kind of workflow.

s521
00:46:45.960 --> 00:46:50.320
But, like essentially any harness that has one of the requirements,

s522
00:46:50.320 --> 00:46:50.800
Sub agents.

s523
00:46:50.800 --> 00:46:53.320
sub agents is pretty much the only one for most

s524
00:46:53.320 --> 00:46:54.000
context.

s525
00:46:54.000 --> 00:46:59.120
Yeah, like um like bash and sub agents are actually you don't even need bash.

s526
00:46:59.120 --> 00:47:02.440
We can run these in VM isolates in code mode, too.

s527
00:47:02.440 --> 00:47:02.960
So,

s528
00:47:02.960 --> 00:47:04.040
Any other questions?

s529
00:47:04.040 --> 00:47:06.000
We can give 10 minutes back to everyone.

s530
00:47:06.000 --> 00:47:18.400
Uh it is based on my many years of engineering experience and dealing with people who sometimes are not very good at expressing requirements and other people who are very good at expressing requirements.

s531
00:47:18.400 --> 00:47:30.480
And it's also, I mean, we we benchmark these internally and and run them against we we have sort of an evolutionary algorithm where we can shrink the kernel by seeing if if they can conform to that.

s532
00:47:30.480 --> 00:47:34.600
So, it's based on practice and empirical results.

s533
00:47:34.600 --> 00:47:35.600
Uh

s534
00:47:35.600 --> 00:47:49.320
Yeah, you can just have that as a spec and you could have like let's say you like um uh Matt Pocock's like uh spec, you know, you like Matt I built a funny one just to like troll him that was like auto Pocock and it basically

s535
00:47:49.320 --> 00:47:53.080
it was my way of proving that my way of wiring skills in works.

s536
00:47:53.080 --> 00:48:07.040
So, you basically just he has a workflow that's like grill me, you know, and then you ask all these questions and then it's like TDD and then all these other ones and you can put the whole thing in one prose program and just be like do Matt Pocock's

s537
00:48:07.040 --> 00:48:14.320
workflow and every sub-agent has to load at every stage of the program you have to load this the next skill in the sequence.

s538
00:48:14.320 --> 00:48:16.480
Um so yeah, you can absolutely like

s539
00:48:16.480 --> 00:48:23.440
do have people I I wasn't calling it spectrum driven development, but like almost by its very nature it is spectrum driven development.

s540
00:48:23.440 --> 00:48:24.840
I'd love you to try this out.

s541
00:48:24.840 --> 00:48:35.880
Try it or come find me after and like whatever spectrum driven development specification you have, we could compose with the boot kernel prose run and then we'll see how it does.

s542
00:48:35.880 --> 00:48:38.520
I I think it would do quite well actually.

s543
00:48:38.520 --> 00:48:40.520
That's a good good inspiration.

s544
00:48:40.520 --> 00:48:43.600
I haven't tried it with the spectrum driven development contracts.

s545
00:48:43.600 --> 00:48:49.720
Like I'd love to sit down and and and try it and see if it overrides it or if it starts colliding.

s546
00:48:49.720 --> 00:48:51.840
That'd be interesting like um

s547
00:48:51.840 --> 00:48:52.480
Here's Here's one

s548
00:48:52.480 --> 00:48:54.960
See how much different skills like throw it off.

s549
00:48:54.960 --> 00:49:01.600
Here's one thing we do have and software engineers using Open Prose to develop software.

s550
00:49:01.600 --> 00:49:14.040
Uh more of our users are using Open Prose to do ops and product management and um quant model documentation and non I don't know software engineering stuff.

s551
00:49:14.040 --> 00:49:22.040
Uh I think it's just the blend of like markdown documentation actually is more interesting to non-engineers um as a result.

s552
00:49:22.040 --> 00:49:23.240
Yeah.

s553
00:49:23.240 --> 00:49:25.640
Any other questions?

s554
00:49:25.760 --> 00:49:27.680
Thank you all.

s555
00:49:27.680 --> 00:49:28.494
Appreciate your time.

s556
00:49:28.494 --> 00:49:30.494
[applause]