{ "video": { "id": "JGtkTjn4120", "title": "Why AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis", "duration": 1153, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "JGtkTjn4120", "title": "Why AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis", "one_liner": "Individuals get 100x from coding-style agents but companies don't, because those agents push complexity and process onto each user; to fix that, encode opinions about how the agent should behave, then supervise those behaviours across trajectories with behavior specs, forking and simulations.", "summary": "Mitchell Troyanovsky, co-founder of Basis, which builds agents for end-to-end work at accounting firms, argues that organizations scale non-deterministic workers through top-down process, while coding-style agents leave the work's complexity and process up to each user. He calls these the competency gap and the process gap, and says closing them requires agents that are simple and reliable, which in turn requires being opinionated about how they behave. Because outcome-based evals can't cover the production distribution of long trajectories, he shows how to get more signal by supervising behaviour with behavior specs, forking and simulations. That signal builds trust through consistency and closes the loop toward self-improving agents in production.", "key_points": [ "Companies trust non-deterministic people through processes: leaders who don't know the day-to-day trust that following the process generalizes to good outcomes, even at the level of using approved software to extract bank data.", "Coding-style agents have two problems: a competency gap, where users must manage skills and memories and hold a theory of mind over the agent, and a process gap, where each user customizes how the work gets done instead of process coming from senior leadership down.", "The goal is reliability and consistency, not perfection: an agent can consistently be unable to do something and then improve, as a person does, but one that shows expertise and then misses something basic is not reliable.", "Without an opinion, 'extract PDF' has near-infinite valid approaches (a VLM, Python with an open-source OCR library, OCR providers, OCR checked by 1, 5 or 100 VLMs), so outcomes may be good but won't be reliable.", "Some opinions come from expertise and need no evals, such as reading the primary source before answering a tax question or taking a picture of a PowerPoint before returning it; opinions also trade off against other things, since the picture adds latency and may not be worth it if formatting errors occur only 1 in 100 times.", "Outcome-based evals (100 or 1,000 cases with ground truth or rubrics) can't prove generalization to hundreds of thousands of production situations, and decisions per trajectory grow exponentially as agents work over longer horizons.", "Basis's behavior spec is 'a product spec for the agent era': it records only the most important opinions, aligns the team, and serves as a rubric on any trajectory without ground truth by asking whether the behavior was relevant and, if so, whether the agent exhibited it. For example, explain a bank reconciliation difference rather than post a plug entry.", "Forking and simulations turn that signal into fixes: forking copies a trajectory at the decision point so you can ask the agent why or change its context, while simulations have a coding agent's sub-agent pretend to be the production agent (told not to cheat, given the real system prompt and skills), which is far worse than forking but much cheaper and faster, and which he used for a year and a half or more before Basis built forking." ], "takeaways": [ "Hide the agent's internals from users so that working with it feels like telling a co-worker what you want done, rather than asking them to manage skills, memories and when to use it.", "Decide how the agent should behave in key situations and enforce that consistently, because a customer shouldn't get one behaviour one time and another the next, even where the choice itself is subjective.", "Write behavior specs for your most important opinions, not every detail, and run them as rubrics across trajectories without ground truth to get far more signal than adding outcome evals can provide.", "Treat the context the agent sees as runtime training data: care about every token, and remember that behaviour comes from the system prompt, progressive disclosures, tool descriptions and model training combined.", "Debug behaviour failures by forking just after the bad decision to ask why or test a context change, and start today with sub-agent simulations, which require building nothing." ], "topics": [ "agents", "evals", "reliability", "behavior specs", "context engineering", "enterprise adoption", "accounting", "self-improving agents" ], "tools": [ "Basis", "Claude Code", "Codex", "Cowork", "ChatGPT" ], "quotes": [ { "text": "humans are already used to working with non-deterministic systems. It's just those systems are normally their co-workers, not their computers.", "at": "01:23", "url": "https://www.youtube.com/watch?v=JGtkTjn4120&t=83s" }, { "text": "The word prompt, I think, trivializes it a bit, right?", "at": "11:30", "url": "https://www.youtube.com/watch?v=JGtkTjn4120&t=690s" }, { "text": "this is going to sound silly, but I promise it's very effective", "at": "15:21", "url": "https://www.youtube.com/watch?v=JGtkTjn4120&t=921s" }, { "text": "Nobody wants their agent to move 37 their tax return.", "at": "18:29", "url": "https://www.youtube.com/watch?v=JGtkTjn4120&t=1109s" } ], "words": 3919 }, "summary_url": "/#JGtkTjn4120", "transcript": { "html": "/transcripts/JGtkTjn4120.html", "txt": "/transcripts/JGtkTjn4120.txt", "vtt": "/transcripts/JGtkTjn4120.vtt" } }