I Don't Fear AI Slop. I Fear Untold Stories - Zack Proser, WorkOS
https://www.youtube.com/watch?v=1et3Lb_KLQo
AI Engineer · 18 min · 180 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Thank you all so much for being here.
[00:14] In this talk, I'm going to try to convince you that if you care about somebody and if you work with them regularly, even if you don't work with them regularly,
[00:21] you should build that person a factory.
[00:23] And we're going to talk about what that means and why.
[00:26] So, first, why do I say this?
[00:28] Um so, I'm an open-source developer, have been for 15 years now.
[00:31] But since my early teens, I was writing uh voraciously and constantly.
[00:36] It started off with pencil paper, pen, and notebooks.
[00:39] And then around 16, I was onto word processors.
[00:41] And in the course of my career in the last 15 years, I've worked at a bunch of startups and I've published stuff on on their blogs as an engineer while working there.
[00:49] And then I've constantly been publishing on my own site.
[00:51] So, I have a ton of essays and tutorials and things that I find interesting to kind of scratch my own itch.
[00:56] And then as part of that cycle, as soon as I scratch that itch, I like to just publish it and kind of share it with the world.
[01:03] And so, that is all just to say that I write sort of obsessively, kind of fanatically, and have been doing so for for quite some time.
[01:10] And we'll get into one of the posts that I I wrote here that has done consistently the best and that I wrote in 23 minutes.
[01:16] And we'll talk about uh how I did that, too.
[01:18] But I'm sure that some of you are already thinking uh this isn't, you know, this isn't an AI conference, isn't an AI talk.
[01:23] Uh certainly there is slop involved here.
[01:26] And uh I'm going to tell you that there is something that I'm far more concerned about than AI slop in particular.
[01:33] And I believe that if we uh bring the right intention to shaping the work and we judge the result and we hold ourselves and each other accountable, that it is actually possible
[01:41] to use AI to produce better stories than we can without it and to improve processes so that more of us can tell stories more effectively and more consistently.
[01:52] So, to get back to that post I was talking about, uh this is a post that I wrote about my own experience over the course of, you know, decade and a half running
[02:00] and self-publishing a tech blog.
[02:02] And I made some, you know, shared kind of what the experience was like and what it what it's like running that and keeping up the consistency.
[02:08] And I published this in I wrote the entire thing from didn't exist to written into markdown and and fully published in 23 minutes before LLMs.
[02:19] And it was after a particularly brutal bedtime routine putting down my two toddlers.
[02:24] And so inevitably what my colleagues and some of my friends would would constantly ask me is how do you write so fast?
[02:30] Like what are you doing and how how is this possible?
[02:32] You don't have more time in the day than I do.
[02:34] And when I thought about this, I realized that the answer is that I am always writing.
[02:38] And so to some degree, I'm kind of obsessively trying out new titles or ideas or sentences or how what stories might I tell or how might I develop that story so that it is
[02:47] the most compelling and impactful that it could possibly be.
[02:51] And as a result, I was essentially the once the only content factory that I knew personally.
[02:56] You know, just kind of, you know, doing my thing, working at in startups, being a developer, but then constantly publishing.
[03:02] And so I personally am not as worried about AI slop because I think there's a tremendous number of techniques that we can apply to to fix that and I think there's ways to solve that problem.
[03:13] I'm I'm significantly more concerned about all the stories that go untold.
[03:18] And so what I've noticed over the course of my career and working with a bunch of excellent and talented folks is that a lot of folks tell themselves, oh, I'm not a writer, I can't do that.
[03:26] A lot of folks feel fear and shame for various reasons, whatever the their experience might be or or their self-concept limits them from actually sharing the most impactful work and sharing it in a way that would be most impactful to them and their their future career.
[03:40] And so this is why I say that it's it's time to consider building factories of love and grace and to give them away.
[03:47] So we're going to develop what that actually means.
[03:49] So I am a engineer on the Applied AI team at WorkOS.
[03:54] I've been there for 2 years.
[03:56] About a year or so ago, we decided that we needed to increase publishing velocity again, not to slop cannon in the internet, but because we had multiple teams and folks that were doing excellent and innovative work constantly,
[04:09] and we were not consistently telling all those stories and kind of bringing, you know, our insight and stuff that we were doing to the marketplace.
[04:16] And so I built the initial what we call Blog Bot, right?
[04:19] And this is a system that was built on Cloudflare, and it runs in the cloud, and because of this sort of obsessive writing background that I had, what I was able to do was to think about what do I believe
[04:32] makes a good artifact in in this in this craft, if you will, right?
[04:36] And so what are all the things that over time and many, many repetitions I have come to believe personally make something more interesting, more engaging, you know, um to have better pedagogical value.
[04:48] And I just kind of stuffed those all into a box that made it automated, and then I handed that box to my colleagues.
[04:55] And what that ended up producing for them was this what I call asymmetric leverage now.
[05:00] So a colleague can come into a very simple interface in Slack, something that they already understand and use every day, and say, "At Blog Bot, we've got this transcript from an event,
[05:08] and I want you to produce an on-brand, you know, on-theme, you know, reaching the quality bar WorkOS blog post about it recapping it."
[05:17] And what they get out is, you know, a draft that is 85 to 90% of the way there.
[05:22] It means that they haven't had to manually and tediously assemble all the context from the Granola recordings or the images of the event or what folks said about it in Slack to prepare
[05:32] or the framing that we wanted to use.
[05:33] All of that is automatically sourced for them.
[05:36] And what they arrive at is a blog that is essentially ready to almost ready to go, but needs a couple more iterations and for a human to read all the way through and to put their name on it and to publish it.
[05:46] And so the nice thing about asymmetric leverage is that you can hide behind a very simple interface an incredibly powerful and even complex, you know, distributed system that has a data connector to every source of information that you care about in your company,
[05:59] custom tools, the ability to ask a human for approval to confirm facts, etc. We even have a special endpoint that has that does image generation that is um the outputs
[06:12] resemble exactly what our excellent and very talented design team would produce.
[06:16] And so that, you know, consistently doing this and allowing other people to run it does across the org increase publishing velocity.
[06:25] And so with just that initial draft or just that initial prompt where my colleague said, "Hey, we just had an event.
[06:30] We want to do a recap post about it.
[06:32] I have some other things on my plate right now, so why don't you start the draft, right?"
[06:35] And what they get back, what I mean by asymmetric leverage is this is what they get back, something substantial.
[06:40] It is not ready to one shot all the way to prod.
[06:42] Somebody has to take responsibility for it and read it all, but we have done or the system has done a significant amount of the work for them to assemble the the initial pieces.
[06:52] And then it is, more importantly, something that is solely in their power to adjust and tune to their liking without me being involved at all.
[07:01] So they don't need to, you know, ask me to do anything or anyone else that that might have a writing background.
[07:07] They can just make the changes that they want in a very simple way and kind of get their work done.
[07:11] And so, you know, this was fun to initially see after I published that this stuff started working and folks there were producing things.
[07:19] And I remember around this time I had a conversation with Michael Grinich, the founder of Work OS.
[07:24] We were talking about, you know, the pretty the debate that was happening kind of in the public square about, you know, slop canonning and and, you know, AI writing is all slop, etc.
[07:33] And at the same time the desire to move faster and for us as experienced developers, we were seeing it was very obvious that you could use these powerful tools to build things in days that used to take us months at a time.
[07:44] And so Michael used a metaphor and he showed me a YouTube video where a company was mass assembling symbols on a production line and, you know, producing them very, very rapidly
[07:54] and yet inspecting all of them.
[07:57] And initially, and he was trying to, you know, draw the the parallel to this is kind of what we're starting to experiment with with these this blog bot and other tools as well.
[08:05] And Michael said, "This screams craft to me."
[08:07] And I remember the initial time that we were kind of looking at that video together, I sort of flinched and didn't fully feel comfortable to accept that, "Yeah, I guess I can produce an articles even faster or content even faster."
[08:20] But Michael's point was that craft runs through the entire line.
[08:22] So, even though machines are often doing must much of the heavy lifting or even all the heavy lifting in some parts and producing units at a speed that humans could not approach,
[08:31] what has not left that company is a complete and profound dedication to craft and inspecting every single device as played by a musician that has been with the company for a long time and understands
[08:45] what the quality bar is and plays and he you can kind of see if you read his lips he's saying it's beautiful and then he hands it off and that one's good to go and he's putting,
[08:53] "I, this person, signed off on this symbol.
[08:56] It is ready to be produced under this brand name."
[08:59] So, they play, they listen, and they decide.
[09:01] They might be being produced at extreme speed, but the quality has not left the process.
[09:06] And this is the reason why Michael told me that this screams craft to me.
[09:10] And so, I was initially uncomfortable with that, but I went went back and kind of thought about my own journey in in writing and I I realized that I didn't disagree with him.
[09:18] And so, we published a recap post after the second or third version of blog posts was out and we had seen publication rates rise across the org and I had anecdotally, more importantly, seen
[09:27] colleagues that were doing excellent work but didn't always share that work were now regularly producing articles to say, "This is what we worked on.
[09:35] This is how it works.
[09:35] This is what it does."
[09:37] And so, I knew that it was starting to work.
[09:39] But we got one of the the first comments that we got online was from someone saying that this is cool that you did this, but now every single thing I read online, I flinch when I read it.
[09:48] I read it with a flinch.
[09:50] And I'm interrogating in my mind every sentence, "Was this AI?
[09:53] Was that one AI?
[09:54] Was this one human-written?
[09:55] Was that one?"
[09:56] And at a certain level, I understood this complaint initially, but it it also bothered me and and taken in conjunction with the conversation around the assembly line and craft,
[10:05] I decided that I kind of had to get my own thinking on this issue clear and straight.
[10:09] And so, of course the way that I do that is to write.
[10:13] And so I decided to look back and think about what do I think about the way I used to write and what it means to write as a person and now the fact that I can do it with an LLM and get to the the end goal that I want
[10:23] um you know, 6x faster with many more iterations.
[10:27] And I decided to borrow an extended metaphor from the blacksmithing world and say that in this case I am the spark who is still conscious, who's still dealing with people in the world experiencing friction and pain
[10:40] um and suffering and joy etc. And so I decide as a developer or a person with taste uh that this is something that's good or that I don't it's not good or I like it or I don't like it or actually this is an important and interesting story that is worth telling and sharing.
[10:55] And then I'm simultaneously the bellows.
[10:57] Now even if at this point in the process the LLM or the machinery might be doing the actual tedious um minutia for me I'm still watching what develops and saying this over here is is okay,
[11:10] this section is completely you know, um irrelevant just drop it.
[11:14] Uh all of these need to be developed but they need to be developed in this exact way and this is wrong, right?
[11:18] And so I'm continuously shaping the work closer and closer to where I want it to go.
[11:22] And then finally uh you know, something has to be done enough to at least initially publish it at a URL and so I am also the quench.
[11:30] I am the thing that you you know, drop into I'm so sorry if there's actual blacksmiths in this I'm as I torture this extended metaphor.
[11:37] But I am the thing that you drop your hot iron into the you know, into the water and it and it seals it and stops it and says publish,
[11:44] Zack Prosser said this and he stands behind all these words and these claims and these links and these specs and and this performance data.
[11:51] And so the finished work still has to earn my name and so the more that I thought about this the more that I felt like now I had a framework for kind of understanding how we're going to move forward and still produce quality in the future even if the velocity is increasing.
[12:03] And and furthermore the more I thought about this the more I realized that because of my background in producing a lot all the time and iterating constantly I feel that if the end user can't tell the difference between
[12:16] on the left me writing every single element by hand in the document and then printing it, and on the right me spending two and a half or three hours going through 14 iterations with an LLM and then saying no, not that, change this word, change that sentence like that.
[12:28] Okay, that's pretty close.
[12:29] And then hand editing a few more at the end myself, and then publishing it, there's effectively no no difference for the end user.
[12:37] You know, you either say that that thing that Zach Prosser produced is garbage, I'm going to read them again.
[12:41] That's fine.
[12:42] Uh or you say like that's okay and maybe next week I'll come back and give that person another try.
[12:47] And so I I realized that what I actually care about and what I think I would suggest we care about is that uh the real the real rubric here is is it true?
[12:55] Is it useful?
[12:57] Is it beautiful?
[12:57] And does it help anybody else?
[13:00] Because that's what we're actually going for if we're producing things that are for humans to review at all.
[13:04] And so uh you know, all of us in this room, whether we realize it or think of it or conceive of ourselves in that way or not, have something like this, some artifact or workflow that they are
[13:15] um absolutely an expert in.
[13:17] It could be an internal process, it could be um something extremely arcane or complex.
[13:22] And the the overall concept, regardless of where you work or the field and industry or what you're trying to do, is to create that into a tool initially, kind of get it working where you start to see
[13:32] something that you recognize and can produce.
[13:35] And then you let other folks try it and say uh if you try this and put your stuff through it, you know, uh are you able to get to the end goal that you're trying to get to?
[13:43] Are you able to get to your work uh your work done more completely and more effectively?
[13:48] And so I think that this is a broadly applicable pattern that we're going to see increasingly more and more.
[13:54] And uh you know, so the first factory that I built in the story essentially is for my colleagues.
[13:59] So more of where we can reach someone, people are publishing more about the excellent work that they're doing.
[14:03] But the experience, of course, with working with LLMs and the acceleration that I get from that as a software developer, then had me questioning uh in this new world, what if an audience that's even smaller is also worth building a factory for.
[14:16] What if uh two people were enough of an audience to build a particular factory?
[14:20] And so I applied the same idea again and uh got this fully working with uh Opus 55 extra high end in 4 hours.
[14:29] But this time my factory was for specifically two people.
[14:32] So uh I do a I've as I mentioned I have I'm a father to two toddlers and so I do a ton of bedtime story duties and uh I built this app so that it looks at the photos of their day and kind of finds what we were doing and then automatically sort of suggest them into a
[14:47] story that can be ready for bedtime and that cast them in it as well.
[14:51] And so their day becomes a story.
[14:53] Lessons or things that we did together they see again and can kind of see reflected in them.
[14:58] There's like animations and stuff and it'll kind of narrate to them.
[15:02] And uh this was a pretty trippy experience because I can say with confidence that this is the fastest successful playtest I've ever had in my entire career um from zero to
[15:12] working and again again again in 4 hours.
[15:15] So the first thing I added next was uh real-time voice control.
[15:18] So when they say again again again at the end it just plays it again.
[15:22] So again this is the same pattern though.
[15:24] This is another factory that I built that's fully working.
[15:26] It's codified with what I believe makes a good bedtime story what I have you know opinions about and have as a practitioner of bedtime stories for having served long enough to have opinions.
[15:36] And what comes out is exactly their content in their day and now they can literally go up to it and say I want a story about Captain America and I want a story about me and Captain America at the zoo
[15:47] and and then there you go and in a minute and a half like you have that full story.
[15:50] They go through it and they see themselves reflected in it.
[15:53] And so you know I I share all this because it's a moment of like incredible potential and advancement and anxiety and hand-wringing and uncertainty and fear and doubt.
[16:04] And I've experienced those things as you know as a visual artist going from oh I draw and I like to draw to oh why would I ever draw again
[16:13] um to the same thing for writing, same thing for software and then back to the other side.
[16:17] And so I just wanted to share that this is now the in some ways an antidote to that kind of fear and certainty doubt because I know that I like to produce these things, and I like to help other people produce those things,
[16:28] and I get immense joy and satisfaction out of seeing my colleagues who are doing excellent work but might not think of themselves as creators or writers now getting credit for and sharing their work with and the patterns that they've discovered with the world.
[16:40] And so, seeing my colleagues put increasingly more and more stories and technical deep dives and announcements through the first factory is incredibly satisfying.
[16:48] You know, seeing my kids produce their own stories that that literally talk through what we did in that day and what we learned, and you know, why we need to have soft hands and not slap our sis you know, sister etc.
[16:59] Um But again, most successful play tests I've ever done uh the the average uh request for book repeat is like three per story, and it's you know, the primary feedback is again again again.
[17:12] Uh so, now I get to enjoy what they make, and that's true for for both these factories.
[17:16] So, it's true for the factory that I work with with my colleagues with and then also at home.
[17:20] So, it's just because that I've done this one thing and made this type of artifact long enough to have opinions about what makes it interesting what makes it good,
[17:26] and now I allow the people that I care about to bring their own content and their own stories through it, and then that is where I'm getting my contentment and getting feedback from them about saying like hey, it would be even cooler if it did XYZ, and you're like okay, you're right.
[17:39] And so, the question that I would turn back to all of you in this room is who will you build a factory of loving grace for?
[17:45] So, who is somebody that you care about or a team that you care about?
[17:48] What is something that you specifically know by heart or are constantly told by colleagues or friends man, or you are so good at XYZ, or you have a real passion for XYZ.
[17:58] Can you Can you build a factory for it?
[18:00] If you're a developer, you know, go to town, and if if you're not, have an agent help you do it, and uh pour into that factory everything that you've learned uh from doing and from building,
[18:10] all of your strong uh strongly held opinions, and then hand it off to somebody else that doesn't normally produce that kind of stuff, and then see what they create,
[18:19] and uh that is why I hope I've convinced you that if you care about somebody, you should build them a factory.
[18:25] Thank you very much.
[18:26] [applause]