The Prompt Is Still a Punch Card - Ted Johnson, JoinIn AI

AI Engineer · 20 min · 319 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:07I'm sure that sometime in the last few hours most of you did this.
00:11You typed a request into a small box to a super intelligence and then you waited.
00:16You watched the cursor blink, maybe a little throbber cycled through clever gerunds like hullabalooing, tomfoolery, and philosophizing to hide the wait.
00:27Maybe it gave you what you wanted, maybe you rephrased it and tried again.
00:32It all felt completely normal.
00:34I want to spend the next 20 minutes making prompting feel unfamiliar and strange again.
00:40I'm Ted Johnson, co-founder of Join an AI.
00:43During my 25-year career building enterprise software, collaboration systems, and AI-enabled interfaces, I've always focused on human interaction.
00:53I've also been following AI for two decades, including back to the far less impressive GPT-1 and 2. And when ChatGPT arrived, I felt two things at once.
01:04First, and unsurprisingly, amazement, knowing the world would never be the same, followed by actually surprising disappointment I couldn't shake.
01:13This disappointment turned into an observation that started a company, Join an AI, and that I keep coming back to, which is why do we still have to learn AI?
01:23Why does something this powerful so often feel unnatural to use?
01:29Here's the path we'll take to answer that.
01:31We'll start with the most familiar computer interface and make it strange again.
01:35Then, I'll give you three key concepts.
01:38The channel, the physical transport that carries your intent, expression, the range and richness of meaning the channel can carry, and the protocol, the shape or rules of this exchange.
01:51I'll use those three concepts to show you that the prompt is our present-day punch card.
01:58We'll share examples of the ways interfaces could progress, and we'll wrap with some practical advice for AI and human-centered design.
02:06Everyone knows what this is, the keyboard.
02:09It's everywhere and it feels completely normal or natural.
02:14But it isn't.
02:14We all had to take lessons.
02:17We all had to practice.
02:18And I say this is someone who loves keyboards.
02:21But it seems that we've been trying to fix them as long as they've been around.
02:25People have tried more efficient layouts like the Dvorak or Colemak to save their fingers some work.
02:31A more extreme example, some keyboard enthusiasts refuse to squander their two digits on the spacebar giving them four or eight or 10 keys to press with that efficient thumb of theirs.
02:44And what do we even mean by the keyboard?
02:46Here's the patent drawing for the layout we use every day.
02:49This patent's from about 1860.
02:52And my personal favorite, it just as could have easily been the Hansen Writing Ball, which looks anything but like a way you want to talk to a superintelligence.
03:02We carry these legacies of an arbitrary input device designed under constraints that haven't existed for a century.
03:09And we put it between ourselves and the most capable machines ever built.
03:13Nobody alive chose it.
03:15We all inherited it.
03:16And then we stopped noticing.
03:18So, that's the first idea, the channel.
03:20The medium an interface gives you to work in.
03:23A keyboard is a channel.
03:24A microphone, channel.
03:27A screen, a punchcard, a prompt box, all channels.
03:30And channels matter because each one can physically carry a different kind of signal.
03:35For example, text is a stream of discrete symbols.
03:39Voice can carry timing, pitch, hesitation, and words.
03:42A diagram can carry spatial relationship all at once.
03:45But these are differences in what the medium can transmit.
03:50It's bandwidth, not the differences in the meaning.
03:53And carrying more signal isn't the same as the machine understanding
03:56[snorts]
03:57any of it.
03:58That's a separate question.
04:00That's the next idea.
04:02Humans use all these channels constantly without thinking.
04:05We never pick one channel and force everything through it.
04:08That would be absurd.
04:09And yet, that's what we ask people to do with machines over and over.
04:13Hold on to the word channel because here's the plot twist.
04:16With AI, the channel never really changed.
04:19You're still typing into a box, but what you were allowed to push through it was about to.
04:25For the first time, the computer channels carry rich, complete human language.
04:30Notice I said what it carries, not what it is.
04:34You're still typing into a box, you're still hitting the submit button.
04:37The keyboard didn't change.
04:39What improved is the range of what you're permitted to express through it.
04:43There's the second idea, expression.
04:46How much of what you actually communicate or mean will go through the interface?
04:51That's the second idea, expression.
04:53How much of what you actually communicate or mean will the interface let through?
04:58Here's an example of expression progress over time with computers.
05:03Starting with assembly, which gave you an instruction set, a few dozen opcodes.
05:08Then came the commands with the shell inputs, flags.
05:13Then modern programming languages gave you primitives that you could compose.
05:17While powerful, step-by-step, each one of these is a fixed vocabulary, requiring you to express your intent by choosing from a menu the machine will accept.
05:27Natural language blew that menu open.
05:29For the first time, you can say almost anything, the way you'd say it to another person.
05:34And on the expression one axis, the leap is real and enormous.
05:38There's an ocean of meaning in an ordinary human request, context, nuance, intent, all things we've never had to spell out to each other.
05:46For the first time, you can say almost anything the way you'd say it to another person.
05:51For the first time, a machine can take it in.
05:53So, here is what should bother us as engineers and designers as it's bothered and inspired me.
05:59The channels for computers have been the same for 15 years, some 180 years.
06:05Now, with AI, we've poured an ocean of expression into it.
06:08So, why does it feel like we're still sipping through a straw and struggling to learn how the AI thinks?
06:14Because there's a third idea underpinning the other two, and it's really the one that hasn't kept up, the protocol.
06:21The rules you follow and the shape of the interaction itself.
06:25Channel stayed the same.
06:27Expression exploded in the last 3 years with LLMs, but the protocol, prompting, is the protocol of a punch card.
06:36And the punch card's protocol is good old batch.
06:38Here's what punch card batch meant.
06:41You sat down away from the machine, carefully encoded your entire request in advance, carried your deck to the operator, you submitted the job, and then you waited, sometimes hours, sometimes overnight.
06:53Then you read the printout, found one thing that was wrong, fixed it, resubmitted it, and waited again.
06:59The machine never engaged with you while you were thinking.
07:04It engaged with the finished package after the fact.
07:08Now, let's look at the prompt.
07:10Assemble the whole request, submit it, wait, read what comes back.
07:14Something's off, assemble it again, submit it again, and wait.
07:19We have to acknowledge that there are features, interactive features improving this.
07:23You can ask for updates, you can ask for summaries of what was done.
07:27But in the end, it's still batch with interactive sprinkles.
07:30It's the same protocol.
07:32We shrank the wait time from overnight to a few seconds or few minutes, and the speed fooled us into thinking that it become interactive.
07:41It hasn't.
07:41It's still batch.
07:43You still package a complete turn before the machine is allowed to participate.
07:47We learn tricks, send tips to use code skills, or rewrite prompts a certain way to manage this.
07:54And speaking doesn't change it.
07:55Your voice just gets transcribed into the box and submitted.
07:58Shorter batch is still batch because the protocol is the part that did not advance.
08:03The protocol is the part we've had to learn.
08:06We just gave it a flattering name.
08:08We call it prompt engineering and treat it like it's a power user skill.
08:12Strip the label off and it's a set of rules for packaging up good old batch.
08:17For example, tell it to think step-by-step.
08:20Give it examples.
08:21Ask it to be an expert.
08:23Don't ask it to be an expert.
08:25Don't ask it that way.
08:26Paste more context.
08:27Paste less context.
08:29Only talk to it through markdown documents.
08:32We trade incantations.
08:33We've learned the magic words.
08:35That's the illusion.
08:36It feels like mastery, but it's the same sort of mastery a punch card operator had.
08:41Knowing exactly how to assemble the deck so the job wouldn't fail.
08:45Moreover, we've gotten good at prompting or these black boxes.
08:49And and that's the part that should bother us, not reassure us.
08:52None of this means prompts are bad.
08:54Punch cards weren't bad.
08:55Command lines aren't bad.
08:57They're brilliant solutions for constraints of their time.
09:00But that's the whole question.
09:01Is batch still the right protocol?
09:04Are we still pre-packaging our intent for a machine that no longer needs us to because it shouldn't need us anymore?
09:11It can ask a follow-up.
09:12It can clarify mid-thought.
09:14It can notice it's missing something and say so.
09:17It should be human conversational.
09:19Sherry Turkle at MIT puts it very well.
09:21Conversation is the most human and humanizing thing we do.
09:26It's one of humanity's superpowers.
09:28The capacity to engage and think is right there.
09:31And yet, we're still making people submit the deck and wait for the run.
09:36Even the punch card inheritance protocol in fact, batch came from the weaving loom.
09:41You set the whole pattern in advance, then ran the cloth.
09:45The punch card got reused on computers by default.
09:48We're still standing at the same moment again.
09:51AI could finally meet us in the middle of a thought, got handed a protocol of a loom.
09:56That's what I mean.
09:57The prompt is still a punch card, not because of how you encode it.
10:00The encoding is powerful and awesome.
10:03Because when the LLM is allowed to engage only after you've packaged a complete turn and submitted it.
10:10And this is where the mismatch bites.
10:11Model capacity is shooting straight up.
10:13Reasoning, speech, vision, memory, planning, all curving upwards.
10:18The interface protocol, flat.
10:20Still a box, still a submit button, still the human doing all the work around the LLM.
10:25The human still decides what context matters, still remembers what to ask, still chooses the timing, still notices the ambiguity, still repairs the output, still has to carefully engineer a prompt.
10:37But the intelligence feels magical.
10:39It's the interface that still feels like work.
10:41And when it feels like work, when the output's wrong, when the magic words don't land, people blame themselves.
10:48They decide they're bad at this.
10:49They're not specific enough.
10:51They don't get AI.
10:52I want to say as clearly as I can, it is not our fault.
10:56We are not bad at using AI.
10:58We are being asked to operate a brand new kind of intelligence through a protocol of a punch card.
11:03The mismatch isn't the user, it's the interface.
11:05In the race to enable AI, we shortcut the interface.
11:09Okay, let's make this concrete and familiar.
11:12A few weeks ago, my co-founder was using a Frontier company's voice mode.
11:17These are known as speech-to-speech models.
11:21He asked it a normal question, "When is the next Timberwolves game?"
11:25Fine.
11:26It answered it quickly.
11:27Then, he pretended I showed up as if to speak to me and said, "Hey Ted, come on in."
11:34He wasn't talking to the AI.
11:36But these models have no way to know that.
11:39So the AI did the only thing a prompt box can do.
11:43It took his speech as a turn and answered it.
11:46"Sure, I'm here.
11:48What's on your mind?"
11:49That's not a good answer, but it's not a dumb model.
11:53It answered the first question perfectly, but it's a protocol with exactly one slot.
11:59Your message, then it's reply.
12:01It has no concept of who's speaking, whether the words were even meant for it.
12:06And the frontier companies want to make strides as well.
12:09OpenAI released GPT real-time 2 in late May and started trying it for their voice mode as well.
12:16It backchannels now.
12:18It goes, "Mhm."
12:19and right.
12:20The little sounds we make to show we're listening actively.
12:23We're seeing the field is converging on the same conclusion we built our company on.
12:28The interface has to stop being batch and start participating.
12:33Others are working on real-time conversation as well.
12:37This is Nvidia's Personal Plex, a research model, not ours.
12:42Watch what happens when it gets interrupted.
12:44I've been thinking about starting a diet.
12:45Yeah, starting a diet can feel a bit daunting, but you could keep it simple.
12:50Focus on eating more veggies and fruits.
12:53Try to
12:53Before I forget, I signed up for a marathon.
12:56All right, congrats on signing up for the marathon.
12:58That's a big challenge.
12:59You've got a lot of time.
13:00Focus on building a solid base with regular long runs.
13:03Stay hydrated.
13:05Make sure you fuel right before and after, and don't forget to stretch and take care of your feet.
13:10Personal Plex stops.
13:11It yields.
13:13It picks the thread back up.
13:14That's real turn taking, listening and speaking at once in real time.
13:20You need to come visit me.
13:22Oh, okay.
13:23go into the city.
13:24Okay.
13:25Cuz that's the thing, like there's like the random spray paint, but then there's also like I'm not sure.
13:31People must commission them.
13:33Like these massive like mural spray paint pieces.
13:36Yeah, I think they do.
13:38And Persona Plex's back channels listens and lands where a person's would.
13:44Beyond conversational flow, there are lots of challenges and it's a complex problem.
13:49Making listening noises is not really the same as knowing who's in the room.
13:54These are not trained to tell that "Hey Ted" wasn't meant for it.
14:00But we are.
14:01We're working on improving the protocol to the models.
14:05By giving it a better understanding of human and group conversation.
14:10Good afternoon, everyone.
14:12Good afternoon, Sam.
14:14Hi, Jordan.
14:17Good afternoon.
14:18Good to see you both.
14:20Quick one, which requirement is this?
14:21Do we have an ID?
14:24This is our Q442, expense approvals.
14:28There, it answered a question.
14:30It only takes actions based on the utility-driven model.
14:35So, it creates goals to fulfill as it labels each of the participants' statements as a question, a proposal, an answer, and then only takes a turn when no one else is speaking
14:45or holding the floor.
14:47Right, we need users to approve requests faster.
14:50Yeah.
14:51The approval flow's too slow.
14:55What kind of requests, though?
14:58Expense approvals first.
15:00Access requests eventually.
15:09AI, hold that.
15:12Actually, let's pause.
15:14Expense approvals or a general approval workflow?
15:18Expense approvals first release.
15:21Access requests are future scope.
15:25That changes the data model.
15:27Good to know.
15:31AI, pull that up for everyone.
15:33Tracking determining who's the speaker is referring to is critical.
15:36In this case, it was easy with a direct reference to the AI, but it will happen again without a direct reference.
15:44Oh, I'd forgotten that was a rule.
15:46So, over 5,000 needs a second approver?
15:49Yep.
15:50Manager plus finance.
15:53So, a big one can't be a single tap.
15:56Right.
15:57Over the limit, it routes to a second approver.
16:00Agreed.
16:01Under five, one tap's fine.
16:03Works for me.
16:11Okay.
16:12Agreed.
16:13Expense approvals, 5,000 threshold.
16:16Right there, the AI resolved the scope objective.
16:19No one wrote the prompt, no one patched the turn and hit submit.
16:23The system was in the conversation, following it, understanding, and choosing its moment.
16:31AI, capture that for us.
16:34First release supports expense approvals only.
16:36Access requests are out of scope.
16:39Managers can approve or reject an expense right from a notification.
16:43And per the finance controls policy, anything over $5,000 routes to a second approver?
16:58Actually, make the threshold 10,000, not five.
17:03Want me to update the requirement to a 10,000 threshold?
17:06Yes.
17:09AI, is this room free after the meeting?
17:12Let me check.
17:13The room looks free until 3:00, but yes, it's yours until 3:00.
17:19And that's the difference between a smart machine behind the same old prompt and an interface that finally participates.
17:27Here's the mindset shift I want to leave you with.
17:29AI is not just an intelligence technology.
17:33It's increasingly becoming an interface technology.
17:36And if so, then book smart models alone are not enough.
17:41Stop picturing AI as a smarter machine hiding behind prompts, agents, loops, and all the old paradigms.
17:47We have to start seeing intelligence itself as a thing that can finally remove interface constraints and amplify human potential.
17:55For 75 years, humans adapted to the machine.
18:00It's syntax, it's forms, it's timing, it's batch.
18:03A system that can reason, listen, infer, adapt should be able to meet us partway, if not all the way.
18:11Instead, if AI is for users, then we should obsess about maximizing the interface.
18:16So, then the design question changes.
18:18It needs to become what burden are we still putting on humans only because the machine used to be too limited to carry that burden itself.
18:27Ask that question, and the whole interface space opens up.
18:31The answer isn't always chat.
18:33It isn't always voice, and not a wall of markdown.
18:36It's definitely not a decade-old set of digital constructs.
18:41The right answer is the affordance humans already use with each other.
18:45Communication, a question, a pause, a sketch, a checklist, a quiet aside, or saying nothing at all.
18:53An interface where timing and modality aren't the humans' job anymore.
18:58Where choosing the right channel at the right moment is done by the AI.
19:02And as a usability person, this is the part that excites me the most.
19:07When you take that burden off people, the the disappears and adoption follows.
19:13Computing has mostly been about to date improving how humans encode their intent for machines.
19:19The punch card, type a command, click a menu, use your thumb on an iPhone, write a prompt.
19:25Every step was progress and every step carried the old constraint forward into the next era.
19:30A translation tax, a precision tax, context tax, repair tax.
19:35AI is our chance to put those down.
19:38Not by making everything magical.
19:40Not by making everything voice.
19:41Not by replacing human judgment, but by making computers for once more fluent with us.
19:49Most talks and videos cover how to use or adopt AI.
19:53The deeper question is how AI intelligence changes the interface.
19:57Human conversation is the most human thing we do.
20:01Because if a machine can finally understand more of what we mean, then we can and should stop reshaping ourselves to be understood by it.
20:09Thank you.