{ "video": { "id": "hd7TOvmyAxU", "title": "An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases", "duration": 1276, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "hd7TOvmyAxU", "title": "An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases", "one_liner": "Weights & Biases' agent Arya runs Karpathy's autoresearch loop on real GPUs, but the reusable lesson is how the team improves Arya itself: log 100% of traces to Weave, turn failures into YAML tasks with LLM and rule-based judges, run roughly 200 of them nightly, and promote a candidate only when it beats production (73% vs 72%).", "summary": "Tim Sweeney (principal engineer at Weights & Biases / CoreWeave) demos Arya, an agent built into the W&B UI. It orchestrates experiments through W&B Launch on a Kubernetes GPU cluster, finds patterns across hundreds of runs, and generates W&B reports and workspaces. He then walks through Arya's own architecture and improvement loop. Production traces go to Weave, where live LLM-judge 'signals', human review, and Arya's analysis of its own conversations become YAML tasks. Those tasks run as a nightly eval suite that drives go/no-go promotion decisions. He closes with advice for managers: invest in agent-oriented observability, treat tasks and evals as CI, keep humans as judges, and add value through context and tools rather than an over-engineered harness.", "key_points": [ "Live demo on Karpathy's autoresearch project (a simple LLM-training codebase): Arya has run 200+ training jobs, downloading the code, setting up launch jobs and GPUs, and iterating autonomously on both code and hyperparameters. For the new batch it chose hyperparameter changes over 'a big architecture swing' because that felt too risky.", "The live batch ran 12 experiments through a W&B Launch queue on a Kubernetes cluster. It scored 5.833 against the previous best of 5.831 from lunch, so it came just short of a live improvement.", "Asked to find patterns in the project, Arya reported that a new model family had emerged, that batch size was a high-leverage parameter, and that one architectural recipe looked promising. It also generated a W&B Report, including a parameter importance chart, and is tuned to build workspaces and plots.", "Arya launched in the W&B iOS app, so you can steer hyperparameter tuning jobs from your phone. 'Summarize' buttons across the W&B app start a chat scoped to whatever you're looking at.", "Architecture: web and iOS clients → API server → turn database → worker harness. The harness connects to a sandbox for shell and Python, an LLM provider (W&B Inference, e.g. GLM 5.2 or fine-tuned models), W&B Launch plus CoreWeave GPUs for jobs that run for days, and Weave for observability. 100% of traces are logged to Weave.", "Weave 'signals' are LLM judges running live on production traffic (user frustration, low-quality response, ask user), and each flag comes with the judge's reasoning. The team also uses Arya to analyze Arya's own conversations.", "Tasks are YAML 'unit tests for your model': a user prompt, an LLM judge for correctness, a second LLM judge for whether the insights are interesting, and a rule-based judge requiring a result within six tool calls. About 200 tasks form a nightly eval suite.", "Two nights before the talk, a candidate scored 73% on the eval suite against production's 72%, so it was due to be promoted that Friday. Evals feed a shared dashboard for go/no-go decisions and an 'improvement loop' (form hypotheses, implement candidate agents, analyze evals), with promotion through a registry." ], "takeaways": [ "Invest in agent-oriented observability: log sessions, turns, tool calls and feedback so you can catch 'behavioral bugs', which are neither exceptions nor performance problems.", "Treat tasks and evals as your CI. Put researchers on the same scrum team as engineers writing tasks, and use the metrics as real go/no-go gates for prompt and architecture changes.", "Keep humans as a necessary judge: use the product yourself and review the best and worst traces together as a team each week, because LLM judges miss some behavioral nuances.", "Before over-engineering the harness or memory, get the low-hanging fruit: give the agent context about your business domain, the primitives available to it, and your business data.", "Encode task expectations with multiple judges, including rule-based efficiency checks such as a maximum number of tool calls, and not just a correctness judge." ], "topics": [ "agents", "autoresearch", "evals", "observability", "llm-as-judge", "mlops", "experiment-tracking" ], "tools": [ "Weights & Biases", "CoreWeave", "Arya", "W&B Weave", "W&B Launch", "W&B Inference", "W&B Artifacts", "W&B Reports", "W&B iOS app", "CoreWeave sandbox", "Kubernetes", "Karpathy autoresearch", "GLM 5.2", "arXiv" ], "quotes": [ { "text": "what we're doing is using Arya to analyze Arya's own conversations to then make recommendations about how to improve Arya all within the UI.", "at": "14:55", "url": "https://www.youtube.com/watch?v=hd7TOvmyAxU&t=895s" }, { "text": "Not exceptions, not performance, but behavioral bugs.", "at": "19:10", "url": "https://www.youtube.com/watch?v=hd7TOvmyAxU&t=1150s" }, { "text": "tasks and evals are the new world of CI.", "at": "19:14", "url": "https://www.youtube.com/watch?v=hd7TOvmyAxU&t=1154s" }, { "text": "There are behavioral nuances that LLM will not catch.", "at": "19:33", "url": "https://www.youtube.com/watch?v=hd7TOvmyAxU&t=1173s" } ], "words": 4743 }, "summary_url": "/#hd7TOvmyAxU", "transcript": { "html": "/transcripts/hd7TOvmyAxU.html", "txt": "/transcripts/hd7TOvmyAxU.txt", "vtt": "/transcripts/hd7TOvmyAxU.vtt" } }