Tribal Dungeons of Global Shipping: AI Agents at Global Scale — Dmitry Buykin, Maersk

AI Engineer · 12 min · 99 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:13Hello everyone.
  3. 00:14Um, this is a practitioner report uh from real production work.
  4. 00:18So, let's get into it.
  5. 00:21Um, I'll skip the generic uh yet another loop agent intro.
  6. 00:25This is about the hard part most agent demos skip.
  7. 00:30and about turning messy operational knowledge into something an agent can execute safely.
  8. 00:37This comes from real work uh in my company I'm working for supporting global shipping operations and grounded in production.
  9. 00:48On paper it's uh one workflow usually but uh in reality every shipment is an orchestration of many parallel state machines.
  10. 00:56While they agree the happy paths work the moment one drifts you get exception work.
  11. 01:06The easy majority is already automated in many companies.
  12. 01:10What's left is the long tail and more exceptions than system built uh to handle them.
  13. 01:18That tail is uh the expensive part.
  14. 01:23And then there's my favorite category.
  15. 01:25And it comes with a special uh plate here.
  16. 01:30See for EI builder dreams and their laptops.
  17. 01:35This what you can find outside of AI bubble in San Francisco.
  18. 01:42The signal process uh depends on many systems being coherent at once.
  19. 01:47If any step uh can't complete the happy path breaks and then it takes expert uh archist expert orchestration across uh multiple incomplete systems.
  20. 02:00All these uh variations um path pathways should be captured in SOPs.
  21. 02:06SOPs is a standard operating procedure common and regulated industries.
  22. 02:10So an expert and the model read them uh the same way.
  23. 02:16That gap is the hard part.
  24. 02:19Stable intent detection tool calls you can guarantee are safe integrating with legacy back ends and results evaluated with experts.
  25. 02:32Uh I call this uh tribal dungeons.
  26. 02:34Uh the knowledge exists but not in a form uh agent can execute and you can safely run a process.
  27. 02:43You can't safely run a process.
  28. 02:44The organization cannot represent standard legacy SOPs [clears throat] bunch of bunch of screenshots organized in sequence and but screenshots not uh a process.
  29. 02:59A legacy SOPs explain what a person sees and clicks.
  30. 03:03And an agent SOP needs a more complex uh setup, preconditions, uh decisions, identifiers, back end calls, validation, recovery, and evidence of uh successful execution.
  31. 03:21Experts own the what, agents own the how.
  32. 03:25And exception becomes a guardrail.
  33. 03:28Most of the effort is the translation and negotiation between them to align on common sense.
  34. 03:38Three parts here um in this architecture it's SOP memory uh organized as SOP corpus execution runtime and theme feedback capture.
  35. 03:49The agent loop is not the system.
  36. 03:52The refining loop around the agent is the system and it's the most complex part.
  37. 03:58Oh, sorry SAP is okay.
  38. 04:00It's this slide for UK.
  39. 04:03This is correct one.
  40. 04:05So and it's good illustration why the the same thing is means different and uh describing differently in different countries and it's creating a lot of variations between each country
  41. 04:19and that corpus is a asset the company company's process memory uh modified and aligned with every country um conditions and far bigger than than than runtime you could see the proportion 20 to1
  42. 04:36So and this is concurrently operating system and this is the scale we run in production today over 200 instances and spikes and latencies deviates from few minutes to up to 10 minutes.
  43. 04:56Um and mainly yeah the mainly main reason for it that u we depending on many legacy system which is uh so cannot be faster than agent loop itself.
  44. 05:11Expert time is the bottleneck.
  45. 05:14So the theme bench uh does the triage for us.
  46. 05:19It clusters the failures and hands back something you can act on.
  47. 05:23Not just look at look at it.
  48. 05:28The trace is the shared evidence that lets an expert and an engineer review the same case and agree on what happened.
  49. 05:39A correction only counts when it becomes an executable change.
  50. 05:43And that's the line between an opinion and a production fix.
  51. 05:52And and this is where quality comes from.
  52. 05:55not from vibes uh not from a bigger model from replaying real examples with u disabled rights to uh protect the production systems and checking whether behavior improved.
  53. 06:13You can see here on the uh cognitive proportion u or this effort ratio uh between each activity in our project.
  54. 06:22So usually uh pipe coding ends here.
  55. 06:27Here there ends um specdriven development because it cannot uh grow improve accuracy more than this stage on this scale.
  56. 06:39And this is uh where the real work starts.
  57. 06:44Nothing exotic.
  58. 06:44It's engineering common engineering sense applied at scale.
  59. 06:50So if uh you don't know all this uh terminology which developed over lastuh 30 years in software development argument to check because this is what every AI agent uh AI coding agent should know uh to help you
  60. 07:05develop reliable production systems and accuracy it's uh wasn't designed uh in one diagram up front it was earned one small correction at the time at the scale you see here.
  61. 07:23So we have over 100,000 corrections over last 9 months in the system when we developing it [clears throat] and this um heat maps uh turned thousands of traces into priorities.
  62. 07:41is how we keep experts and engineers uh looking at the same problems and prioritize where the the most beneficial work for them.
  63. 07:51Every cell is a group of tracked scenarios we have and uh usually to turn one block in red it's around one two months of force for the whole team
  64. 08:06whole team of engineers and also AI agents um the agent failed is uh where the investigation starts not where it ends each failure maps to a specific uh fix
  65. 08:22discovery needs agent freedom and production needs a cage.
  66. 08:27Uh the harness isn't there to give the agent more room.
  67. 08:30It's there to make the dumb mistakes impossible.
  68. 08:36So on this scale please be careful is not a guard guard.
  69. 08:40Uh if we have wrong workflow then classifier eval.
  70. 08:43If it's wrong right then right gate.
  71. 08:46If it's wrong assumption then it's a mere view.
  72. 08:49A preventive measure eliminates the unsafe path on critical paths.
  73. 08:56U review and approval stay in the loop.
  74. 08:59The engine engineering focus is uh to build safe hands offs and a trail you can trust.
  75. 09:08The real outcome uh wasn't the agent in the system.
  76. 09:13It was the [clears throat] methodology we built around it.
  77. 09:17If you want the blueprint, then it's uh these five moves.
  78. 09:22Make work representable.
  79. 09:23Make exe execution bounded.
  80. 09:26Make behavior observable for every agent and make correction cheap.
  81. 09:31And last thing is make improvement compound.
  82. 09:34So gradually systematically improve the quality of the system.
  83. 09:41AI native um operation is more than agents in workflow.
  84. 09:47It's a system that learns from what works and fold folds it back into code as new composite tools adapting to the applications and the people around it.
  85. 09:57The best AI models um oriented intelligence for us.
  86. 10:03The adaptive architecture we built is the asset, the final asset and we aggregating all um repeatable sequences of steps successful scenarios and uh merging them into bigger tools which uh
  87. 10:23combine the disproven scenarios into the reusable snippets by other agents.
  88. 10:28So and then um it's possible to roll out them not only for one country but for hundreds country in one go.
  89. 10:38So this is um um all for the talk and little time for questions and I'll be around afterwards.
  90. 10:46And the final reminder you know if you you know if you are AI builder if you emotionally attached to tools not MCPS we're not using MCPS because uh for us it's uh always
  91. 11:01not the best choice.
  92. 11:02So because all all systems usually really bloated and we have to distill responses and uh tune the tools through function calling uh to our agents then we can control
  93. 11:16quality of um our software and ensure that uh it's correctly processing assigned tasks.
  94. 11:27Thank you.
  95. 11:27Any questions?
  96. 11:33Okay, then um thanks for your attent u attention.
  97. 11:37Then I will be around so you can ask me questions if you want.
  98. 11:44[applause]
  99. 12:00[music]