{ "video": { "id": "kiqubc5b5Yo", "title": "Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains — Brendan Rappazzo", "duration": 1207, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "kiqubc5b5Yo", "title": "Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains — Brendan Rappazzo", "one_liner": "Morgan Stanley's ~30-person PhD AI research group open-sourced AlphaLab, an auto-research agent (research → self-built eval → Kanban-style mass experimentation on a Slurm GPU cluster) and concluded that the harness itself will commoditize — the durable enterprise value is in building evals and environments.", "summary": "Brendan Rappazzo describes AlphaLab, a home-grown agentic harness Morgan Stanley built to automate quant research: you hand it a data path and a natural-language prediction goal, and it researches, builds its own backtest/eval, then runs mass experimentation via a strategist agent that files experiment 'cards' to worker agents. Version 1.0 (released early April, 40-page tech report plus open-sourced code) placed in the top 12% of an Nvidia-hosted Kaggle fine-tuning competition and has produced internal model improvements now moving through risk into production, but it also failed on hard problems and its design choices felt arbitrary. The 2.0 answer is to stop hand-designing the middle: impose a strict Kaggle-like environment/eval contract, build 10–20 careful environments as an RL signal, and let the LLM meta-optimize the harness itself. The headline claim is that general auto-research is becoming a commodity, so all of an enterprise's value comes from building environments.", "key_points": [ "The team is ~30 PhD AI researchers operating half-academic (papers, open source) and half-applied; many of their problems are 'Kaggle-shaped' — time-series data in, predict future values, well-calibrated — and being sell-side means less adversarial selection, which suits an auto-research framework. Motivating wins include hyperparameter churn on production algorithms, transferring a credit-bonds model to muni bonds, and low-hanging fruit at desks using no ML at all.", "It only felt possible from December 2025 onward, with Opus 4.5 plus harnesses like Claude Code and Codex making long-horizon tasks viable.", "AlphaLab runs three phases: (1) a research phase — 'almost like a super CLAUDE.md file or a super init' — driven by a to-do list scaffold so the agent can be re-prompted if it exits early, with markdown notes per item for later context querying, taking roughly 3–4 hours; (2) eval/backtest building; (3) mass experimentation.", "Eval building uses a builder agent plus two critics — one high-level (conceptual errors, forward leakage of information) and one programmatic (unit and integration tests) — looping back to the builder until all agents are satisfied.", "Mass experimentation is formulated as a Jira/Kanban board: a strategist agent files experiment cards into an 'to implement' column, worker agents write the code and Slurm config, submit to the GPU cluster, inspect training curves for under/overfit plus eval results, and write a postmortem back to the strategist — enabling self-evolution (e.g. three transformer variants flopped but XGBoost worked, so explore more tree methods). Users steer by cancelling cards, adding cards, or chatting with the strategist, with an ever-growing leaderboard scored on a held-out private validation set.", "No off-the-shelf agent framework — the harness is all their own code, largely written by Claude, using functional tool calling so it stays provider-agnostic across OpenAI, Anthropic and open-source models. Three main tools: full shell access, web search (mainly used in the research phase to read arXiv and technical blogs), and a Slurm abstraction where the model just declares 'I need 4 H100s and this many CPUs'.", "Results: in the paper they tested CUDA kernels, an academic traffic time-series dataset, and Karpathy-style LLM speedrunning, where AlphaLab found a better training config than a single-agent-in-a-loop baseline. On an Nvidia-hosted Kaggle competition to fine-tune NeMoTron into a reasoning model it reached the top 12% of submissions with only ~10 iterations because they joined late. Internally, several already-decent models handed to AlphaLab for churn yielded meaningful improvements now going through risk into production.", "AlphaLab 2.0 makes the eval strict and Kaggle-like: data and description in, harness in the middle, its only job is to submit containerized models, feedback via a public leaderboard score while the user sees a private held-out leaderboard. They've built 10–20 careful environments as a reinforcement-learning signal, are doing meta harness optimization (an LLM reads traces and results and improves the harness), and are collecting traces from open-source models to touch weights via GRPO and on-policy distillation — expecting the best system to be an orchestration of open and closed models optimized together.", "Human/enterprise expertise is encoded through environments, not prompts: proprietary data plus verifiable metrics ('maybe the easy part') plus qualitative rubrics that grade each rollout on how well it followed the firm's research process." ], "takeaways": [ "Start with a good eval before building any research agent — the speaker calls it a lesson he re-learns monthly; LLMs aren't malicious but make silly mistakes, and optimizing against a bad eval collapses the whole system.", "Harden the eval with adversarial multi-agent review: a builder plus a conceptual critic (looking for forward leakage) and a programmatic critic (unit/integration tests), looping until all agree.", "Impose a strict contract on your agent — data and description in, containerized model submissions out, public leaderboard score as feedback, private held-out validation for you — so the harness in the middle becomes swappable and measurable.", "Stop hand-tuning harness topology (two strategists? a debate?) by intuition; once you can measure, treat harness design as a verifiable loop and let the LLM meta-optimize it against traces.", "Consider writing your own harness from scratch rather than adopting an agent framework — with Claude Code writing the code you get max freedom, provider-agnostic functional tool calling, and you 'rise with the tide of the models' instead of having your scaffolding consumed by them.", "Invest expert time in building environments and qualitative research-process rubrics, not in the agent loop — general auto-research is commoditizing (he points at GLM 5.2), and environments are where enterprise value accrues." ], "topics": [ "agents", "evals", "environments", "multi-agent", "quant-research", "reinforcement-learning", "auto-research", "finance" ], "tools": [ "Morgan Stanley", "AlphaLab", "Claude", "Claude Code", "Opus 4.5", "Codex", "OpenAI", "Anthropic", "GitHub", "Slurm", "Nvidia H100", "XGBoost", "Kaggle", "Nvidia NeMoTron", "GLM 5.2", "arXiv", "Jira", "GRPO" ], "quotes": [ { "text": "LLMs aren't malicious, but they can make, you know, very silly mistakes and if you're optimizing against a bad a bad eval, the whole thing kind of falls apart.", "at": "09:54", "url": "https://www.youtube.com/watch?v=kiqubc5b5Yo&t=594s" }, { "text": "we wrote it all and really Claude wrote it all", "at": "06:45", "url": "https://www.youtube.com/watch?v=kiqubc5b5Yo&t=405s" }, { "text": "to me evals and environments are the same thing. It's just you train in environments.", "at": "16:44", "url": "https://www.youtube.com/watch?v=kiqubc5b5Yo&t=1004s" }, { "text": "all of your value as like an enterprise or human expert comes from building environments", "at": "19:13", "url": "https://www.youtube.com/watch?v=kiqubc5b5Yo&t=1153s" } ], "words": 4142 }, "summary_url": "/#kiqubc5b5Yo", "transcript": { "html": "/transcripts/kiqubc5b5Yo.html", "txt": "/transcripts/kiqubc5b5Yo.txt", "vtt": "/transcripts/kiqubc5b5Yo.vtt" } }