{ "video": { "id": "qFaKvbVc5cE", "title": "Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA", "duration": 3330, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "qFaKvbVc5cE", "title": "Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA", "one_liner": "To teach an agent search, NVIDIA turns 5–8-hop random walks over Wikidata into search riddles, has GPT-OSS-120B use Tavily over MCP to find the answers, then filters 50,000 seeds down to 7,000 high-quality SFT records; training Nemotron on that set took BrowseComp from 0% to 30%.", "summary": "Nathawani argues that any agent behaviour you want can be turned into a data problem. Looping on an LLM prompt fails because the output repeats itself, so he uses NVIDIA's open-source NeMo Data Designer to build synthetic data column by column as a DAG: seed columns, Python sampler columns for diversity, LLM text and structured columns, tool-calling columns, and LLM-as-judge columns. Three notebooks build up from basic trivia question-and-answer generation, to MCP tool use with Tavily, to end-to-end agentic search grounded in Wikidata hops. He stresses that evaluating synthetic data is the hard, unsolved part.", "key_points": [ "Data Designer (pip install data designer) is fully open source and has been used to create 6.5 trillion tokens of training data. It supports seed data (CSV, JSONL, Parquet), sampler and subsampler columns, text, structured, code and image generation, validation, preview mode (3 records), batch mode via data_designer.create, and pushing datasets straight to Hugging Face.", "Prompting an LLM in a loop fails on diversity: it starts repeating after 10–20 tries. It also lacks correlation between features, steerability, validation and reproducibility. Jinja-templated prompts filled from seed topics make every prompt unique.", "The search seed data comes from random walks of 5–8 hops (elsewhere he says 4–8) over the Wikidata graph, for example NVIDIA → Jensen → Oregon State University → Benton County → Thomas Hart Benton. Longer walks drift off topic. An LLM turns each walk into a riddle the model cannot answer from memory, because 'if we give an agent the chance to cheat, it's going to cheat.'", "Trajectories are generated with GPT-OSS-120B served locally on vLLM, using a Tavily search tool over MCP. The system prompt includes tool-usage guidelines that stop it overspending on searches, plus an in-context example. Seed creation is deterministic; only the search trajectory is agentic. This project was SFT post-training for Nemotron.", "Results: 50,000 seeds were filtered to 7,000 high-quality SFT records, and training on them moved BrowseComp from 0% to 30%. A separate SQL dataset moved Nemotron Nano on BIRD from 26% to 41%. On StructEval (JSON, YAML, XML, CSV, TOML), Nemotron distilled from GPT-OSS but beat GPT-OSS's score.", "Evaluation is the hard part. A deterministic matching verifier gave only 24%, which he expected. Seeds can go stale: the verifier accepted 'US' for who owns US Steel, but it was sold to Japan. He layers LLM-as-judge, human reviewers who cross-check the judge's verdicts, and real-time information checks.", "Nemotron Personas USA has 1 million personas built from census data with a probabilistic graphical model, with versions for Japan, India, Singapore and Brazil. It grounds synthetic data in real demographic distributions without leaking any individual's data; one example is that exactly 7 US under-18s earn more than $10M a year.", "For RL, you skip generating the golden solution and keep only the prompts, then use NeMo Gym reward functions with any RL training framework. Other open-source tools from his org: NeMo Curator, NeMo Anonymizer (for example, scrubbing API keys from agent logs) and NeMo Safe Synthesizer." ], "takeaways": [ "Put diversity before volume: Kimi K2 found 10 strategies for 1 epoch beats 1 strategy for 10 epochs. Inject diversity with Python sampler columns and grounded seed data rather than repeated prompting.", "Use the cheapest column type that gives a correct result. Never use an LLM to sample categories; save LLM calls for generating text.", "Generate more data than you need, then filter and verify hard (he kept 7,000 of 50,000). Iterate on preview records and read the data yourself before running a batch.", "Make agent prompts hard and riddle-like, grounded in multi-hop seeds, so the model has to call tools instead of answering from memory.", "Treat a data pipeline as version-controlled, shareable code. Layer several evaluation methods (LLM judge, human review of the judge, live fact-checks) rather than trusting one LLM-as-judge." ], "topics": [ "synthetic data", "agents", "search", "tool calling", "mcp", "evals", "post-training", "data quality" ], "tools": [ "NVIDIA NeMo Data Designer", "Nemotron", "Nemotron Nano", "Nemotron Ultra", "Nemotron Personas", "NeMo Gym", "NeMo Curator", "NeMo Anonymizer", "NeMo Safe Synthesizer", "Tavily", "Exa", "MCP", "vLLM", "GPT-OSS-120B", "Wikidata", "Wikipedia", "Hugging Face", "BrowseComp", "BIRD", "StructEval", "Kimi K2", "Codex", "Claude Code", "ChatGPT", "Claude", "Gemini" ], "quotes": [ { "text": "um well diversity diversity diversity that's the most important thing right", "at": "05:43", "url": "https://www.youtube.com/watch?v=qFaKvbVc5cE&t=343s" }, { "text": "Because if we give an agent the chance to cheat, it's going to cheat.", "at": "25:17", "url": "https://www.youtube.com/watch?v=qFaKvbVc5cE&t=1517s" }, { "text": "I started with 50,000 seeds and in the end I ended up only with 7,000 high quality endtoend SFT records", "at": "39:52", "url": "https://www.youtube.com/watch?v=qFaKvbVc5cE&t=2392s" }, { "text": "I think the best way to know if your data is any good is to actually look at the data.", "at": "46:33", "url": "https://www.youtube.com/watch?v=qFaKvbVc5cE&t=2793s" } ], "words": 10246 }, "summary_url": "/#qFaKvbVc5cE", "transcript": { "html": "/transcripts/qFaKvbVc5cE.html", "txt": "/transcripts/qFaKvbVc5cE.txt", "vtt": "/transcripts/qFaKvbVc5cE.vtt" } }