{
  "video": {
    "id": "UGjk3Uk0MxM",
    "title": "The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft",
    "duration": 3098,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "UGjk3Uk0MxM",
    "title": "The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft",
    "one_liner": "Benchmarks won't tell you how a model behaves in your app. Run the same small scenarios (counting, RAG grounding, image input, tool calls, agent loops) on every candidate model, then use automated evals and prompt optimisation such as DSPy+GEPA so a model swap doesn't mean rewriting prompts by hand.",
    "summary": "Pamela Fox ran a guided workshop comparing GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3, all deployed on Microsoft Foundry and called through the OpenAI Python SDK. Her argument is that new models arrive constantly and access can be cut off, so AI engineers need to be able to point their apps at a different model and still have them work. The same scenarios exposed real differences: counting, staying grounded in RAG, reading images, sticking to tool arguments, and how efficiently each model calls tools in an agent loop. She then moved from manual checks to string-check evals, LLM-as-a-judge, Microsoft Research's new Assert tool, and DSPy with the GEPA optimiser, which writes the prompt for you.",
    "key_points": [
      "Letter counting ('The elderly gentleman eagerly entered the elevator', 13 E's): GPT-5.5 and Kimi got it right, DeepSeek V4 Flash said 6 and Mistral said 8. Prompting Mistral to split the sentence into words, count per word, sum and show its reasoning fixed it.",
      "RAG grounding test with a carpenter-bee-only dataset asked about honey bees too. GPT-5.5, Kimi and DeepSeek admitted the sources had no honey bee information. Mistral answered from its weights anyway. Even after a prompt requiring 'no source provided' labels, it labelled the claims but still gave the information.",
      "Image input (aurochs vs unicorn, crocodiles vs alligators, cheapest plant in a small-font table screenshot): GPT-5.5 and Kimi got all three. Mistral called the crocodiles alligators and picked a $2.30 plant instead of the 58-cent one. DeepSeek V4 Flash is text-only. Asking the same question in Spanish made the task fail.",
      "Tool calling with a JSON-schema create_calendar_event function: she treats the schema's names, types and descriptions as prompt engineering. GPT-5.5 added the user as an attendee. DeepSeek did too and wrote 'virtual (Microsoft Teams)' for the location, which doesn't follow the argument constraints.",
      "In tool-calling loops the difference is efficiency. GPT-5.5 put a whole calculator word problem into one expression and one call, while DeepSeek needed four calls. Kimi sometimes gets the right answer and then re-checks itself with extra calls.",
      "LangChain v1 travel agent: GPT-5.5 called search_flights and search_hotels in parallel, then check_budget, then search_activities, which she called the most efficient agent possible here. Kimi didn't parallelise and made two extra calls. DeepSeek made four calls without parallelising. Mistral searched flights and hotels and then stopped.",
      "Evals: string comparison is enough to check a single tool call's name and arguments. Fuzzier output needs LLM-as-a-judge returning pass/fail plus a reason, given the query, the response and the context. Each case then costs at least two LLM calls, and RAG needs roughly 200–800 cases, so evals don't belong in every CI run.",
      "Microsoft Research's Assert (released about a week before the talk) turns a YAML spec into behaviour categories, stratified test scenarios and judging rubrics. On the travel planner, GPT-5.5 had the fewest violations, ahead of DeepSeek and Kimi."
    ],
    "takeaways": [
      "Before relying on a model, test it on your own scenarios rather than benchmarks. Check whether it admits what it doesn't know in RAG, whether it understands your kind of images, how well it sticks to tool arguments, and how many tool calls it needs. Include every language you support.",
      "Build evals that are easy to add cases to, and keep feeding real production user inputs into them so the evals don't drift from how people actually use the app. Pre-run app outputs so you can change the evaluator independently, and calibrate LLM judges against a domain expert's labels.",
      "Skip built-in coherence and fluency evaluators. Write custom evaluators (groundedness and relevance can be fine for RAG), and use something like Assert to bootstrap a test set before launch.",
      "Give models tools for date math and arithmetic instead of letting them work it out in their heads. When a tool call comes back with the wrong arguments, tighten the parameter descriptions to remove the ambiguity.",
      "For model independence, especially moving to smaller, cheaper models, consider DSPy with GEPA. You declare the spec and the checks in code, and the optimiser hill-climbs to a prompt for the target model (about 6 minutes for her Mistral example)."
    ],
    "topics": [
      "model comparison",
      "evals",
      "llm-as-judge",
      "rag",
      "tool calling",
      "agents",
      "prompt optimization",
      "multimodal"
    ],
    "tools": [
      "Microsoft Foundry",
      "OpenAI Python SDK",
      "OpenAI Responses API",
      "GPT-5.5",
      "GPT-5",
      "Kimi K2.6",
      "DeepSeek V4 Flash",
      "Mistral Large 3",
      "Anthropic (Messages API, citations)",
      "uv",
      "GitHub Codespaces",
      "LangChain v1",
      "Pydantic AI",
      "Microsoft Agent Framework",
      "Langfuse",
      "LangSmith",
      "Arize",
      "Azure AI Evaluation SDK",
      "OpenAI evals",
      "DeepEval",
      "promptfoo",
      "Assert (Microsoft Research)",
      "DSPy",
      "GEPA",
      "Ralph loop (Geoffrey Huntley)",
      "Hamel.dev",
      "Ollama",
      "Microsoft Teams",
      "Dropbox",
      "Shopify"
    ],
    "quotes": [
      {
        "text": "you don't really know how good a model is until you use it for your particular scenario",
        "at": "05:19",
        "url": "https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=319s"
      },
      {
        "text": "GB55 is very good at knowing what it doesn't know.",
        "at": "10:19",
        "url": "https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=619s"
      },
      {
        "text": "Never run those. Like LLMs are great at being coherent. You never have to check if an LLM is coherent.",
        "at": "44:43",
        "url": "https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2683s"
      },
      {
        "text": "Your spec is your code. The prompt is an output from this Dspy process",
        "at": "46:55",
        "url": "https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2815s"
      }
    ],
    "words": 11495
  },
  "summary_url": "/#UGjk3Uk0MxM",
  "transcript": {
    "html": "/transcripts/UGjk3Uk0MxM.html",
    "txt": "/transcripts/UGjk3Uk0MxM.txt",
    "vtt": "/transcripts/UGjk3Uk0MxM.vtt"
  }
}