WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=FLUoowDJg4I

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/FLUoowDJg4I.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:16.960 --> 00:00:17.880
Okay.

s3
00:00:17.880 --> 00:00:19.240
All right.

s4
00:00:19.240 --> 00:00:20.360
Hello everyone.

s5
00:00:20.360 --> 00:00:21.800
Thanks for coming by.

s6
00:00:21.800 --> 00:00:28.080
Today I'll talk about how I automate my own job at Hugging Face using agents.

s7
00:00:28.280 --> 00:00:29.720
Um short introduction.

s8
00:00:29.720 --> 00:00:34.640
I'm just uh Niels from Belgium, the land of beer, fries, and chocolate.

s9
00:00:34.640 --> 00:00:40.880
I studied at KU Leuven, and I'm a machine learning engineer at Hugging Face for 5 years now.

s10
00:00:42.080 --> 00:00:47.600
Uh today I'll talk about the community science team at Hugging Face, which is the team I'm part of.

s11
00:00:47.600 --> 00:00:52.280
Uh then I'll talk about how I automate large parts of the community science team.

s12
00:00:52.280 --> 00:00:58.240
And finally, I'll also discuss some other efforts uh that we do at Hugging Face.

s13
00:00:58.240 --> 00:01:02.000
So, let's start with the community science team at Hugging Face.

s14
00:01:02.000 --> 00:01:09.680
So, basically, this started when I was sent I was seeing like trending research uh passing by on GitHub.

s15
00:01:09.680 --> 00:01:15.880
And a lot of times when I saw new interesting work, um the weights were not available on Hugging Face, sadly.

s16
00:01:15.880 --> 00:01:20.920
Like, researchers use either Google Drive or they use GitHub releases.

s17
00:01:20.920 --> 00:01:26.920
They use Dropbox, they use Zenodo, or other servers to put their um artifacts on.

s18
00:01:26.920 --> 00:01:29.640
And this hurts uh discoverability of their work.

s19
00:01:29.640 --> 00:01:33.560
It's like not easily uh visible or discoverable.

s20
00:01:33.560 --> 00:01:39.040
And when I then open a GitHub issue to say like, "Actually, you could put your weights on Hugging Face for free."

s21
00:01:39.040 --> 00:01:46.360
Most of the time, people replied to me like, "Yeah, migrating the weights uh from Google Drive to Hugging Face actually makes perfect sense."

s22
00:01:46.360 --> 00:01:52.560
So, yeah, the community science team can also uh be described as the Google Drive to the hub team.

s23
00:01:52.560 --> 00:01:53.320
Um why?

s24
00:01:53.320 --> 00:01:59.520
Because on Hugging Face, we have these paper pages, uh and every single paper is from archive.

s25
00:01:59.520 --> 00:02:05.640
And then on the right side, you can basically list the linked artifacts, like the the linked models or data sets.

s26
00:02:05.640 --> 00:02:11.640
So, people can easily reproduce your paper or find the models or data sets.

s27
00:02:11.640 --> 00:02:14.000
So, yeah, you can see them on the right side.

s28
00:02:14.000 --> 00:02:25.240
Um this improves the discoverability of your work because we have these metadata tags or filters on the app, so you can easily find, for example, depth estimation model, an LLM if you're interested.

s29
00:02:25.240 --> 00:02:26.760
You can find them by language.

s30
00:02:26.760 --> 00:02:29.840
You can tag them with the library they are compatible with and so on.

s31
00:02:29.840 --> 00:02:32.880
So, this improves the discoverability of your work.

s32
00:02:32.880 --> 00:02:39.920
So, these are, yeah, the metadata tags that you can add to every single model on Hugging Face or every single data set.

s33
00:02:39.920 --> 00:02:48.360
So, yeah, this is like the main problem that we saw, like, lots of people, lots of researchers are like using third-party services to publish their work.

s34
00:02:48.360 --> 00:02:53.200
We have the Hugging Face platform, which is like a centralized place where people can find machine learning artifacts.

s35
00:02:53.200 --> 00:02:57.680
It also improves with documentation because you can add a model card or a data set card.

s36
00:02:57.680 --> 00:03:02.600
We have tooling, so you can easily upload or download stuff from Hugging Face.

s37
00:03:02.600 --> 00:03:06.280
And it might also help reach researchers in promoting their work.

s38
00:03:06.280 --> 00:03:12.720
So, it's basically a win-win both for researchers and then other people using the research.

s39
00:03:12.840 --> 00:03:15.760
So, yeah, these are the typical GitHub issues that I was opening.

s40
00:03:15.760 --> 00:03:17.280
I always had like the same template.

s41
00:03:17.280 --> 00:03:20.400
I just asked, "Could you please release this checkpoint on Hugging Face?

s42
00:03:20.400 --> 00:03:23.640
Could you please release this data set on Hugging Face?"

s43
00:03:23.640 --> 00:03:23.707
And

s44
00:03:23.707 --> 00:03:23.800
[snorts]

s45
00:03:23.800 --> 00:03:33.160
then I also opened PRs, pull requests on Hugging Face to add data set cards or model cards to improve the documentation of those artifacts.

s46
00:03:33.160 --> 00:03:35.440
But, there's a problem.

s47
00:03:35.440 --> 00:03:45.160
It's not really scalable for me to open all these GitHub issues or pull requests because every single day there are like hundreds of research papers coming out on archive, especially now with AI boom.

s48
00:03:45.160 --> 00:03:51.120
Um yeah, also NeurIPS, for example, a major AI conference, they are seeing a massive amount of papers.

s49
00:03:51.120 --> 00:03:52.840
So, can we automate this?

s50
00:03:52.840 --> 00:03:56.720
Can we scale the community science team with agents?

s51
00:03:56.720 --> 00:03:59.280
So, that's the second part of my talk.

s52
00:03:59.280 --> 00:04:05.680
How can we, yeah, scale this uh to a massive amount of research papers?

s53
00:04:05.800 --> 00:04:07.640
So, the idea is pretty simple.

s54
00:04:07.640 --> 00:04:17.760
Uh we should have an AI agent which can help me do this outreach to all these researchers which publish uh models or data sets uh as part of their research work.

s55
00:04:17.760 --> 00:04:21.600
And then, yeah, do the outreach in an automated way.

s56
00:04:21.600 --> 00:04:23.720
So, this is the typical workflow that I was following.

s57
00:04:23.720 --> 00:04:31.600
So, basically, whenever I saw a research paper, I first tried to find the GitHub URL uh of that paper, if it's available.

s58
00:04:31.600 --> 00:04:35.680
Then, I I read the readme of that GitHub file.

s59
00:04:35.680 --> 00:04:40.760
And then, I basically check if there's anything new interesting to be shared on Hugging Face.

s60
00:04:40.760 --> 00:04:42.960
Uh it could be that it's on Hugging Face already.

s61
00:04:42.960 --> 00:04:50.760
In that case, I check whether the model cards or data set cards already properly uh present, whether the metadata tags, for example, are there.

s62
00:04:50.760 --> 00:04:54.280
If uh not, then I will might open a pull request.

s63
00:04:54.280 --> 00:04:58.520
Otherwise, if the artifacts are not yet on Hugging Face, I open a GitHub issue.

s64
00:04:58.520 --> 00:05:00.720
And then, finally, I also follow up with the author.

s65
00:05:00.720 --> 00:05:05.480
So, that's kind of the workflow that I had to automate uh with agents.

s66
00:05:05.480 --> 00:05:07.600
And there are several ways to solve this.

s67
00:05:07.600 --> 00:05:10.600
Uh you could uh go with a workflow.

s68
00:05:10.600 --> 00:05:16.520
Uh these pictures are, by the way, taken from the blog post building effective agents by Anthropic, which is a really great read.

s69
00:05:16.520 --> 00:05:17.480
Uh read.

s70
00:05:17.480 --> 00:05:21.680
Um so, on the left side, you see, yeah, a workflow which is more deterministic.

s71
00:05:21.680 --> 00:05:28.200
You basically use LLM APIs within steps of a predefined path or pipeline, uh which is more predictable.

s72
00:05:28.200 --> 00:05:29.360
It's more deterministic.

s73
00:05:29.360 --> 00:05:30.960
You have more control over it.

s74
00:05:30.960 --> 00:05:33.160
Of course, it's less flex- flexible.

s75
00:05:33.160 --> 00:05:44.919
And then, on the other hand, you could have a fully fledged auto- autonomous agent, which is an LLM in a loop that calls tools until it's done, which is more flexible, but also less uh predictable.

s76
00:05:44.919 --> 00:05:48.680
Uh Uh, at the time, yeah, of course, it doesn't have to be a binary story.

s77
00:05:48.680 --> 00:05:55.480
You can have a workflow on one hand, you can have a fully autonomous agent on the other hand, but you you could of course also mix and match these type of things

s78
00:05:55.480 --> 00:05:57.640
uh, for your use case.

s79
00:05:57.640 --> 00:06:02.120
In my case, I went for um, a pretty deterministic workflow.

s80
00:06:02.120 --> 00:06:02.640
Uh, why?

s81
00:06:02.640 --> 00:06:10.600
Because at the time that I was building this, this was in 2024, was at the time that Anthropic uh, wrote their blog post building effective agents.

s82
00:06:10.600 --> 00:06:15.680
And there they actually said, "Try to avoid building agents if you really don't have to.

s83
00:06:15.680 --> 00:06:19.560
Start simple, start with a single LLM API.

s84
00:06:19.560 --> 00:06:20.880
Uh, avoid frameworks."

s85
00:06:20.880 --> 00:06:23.520
Uh, and actually I think those were great tips.

s86
00:06:23.520 --> 00:06:30.400
So, at the time I started building a workflow which basically replicated the workflow that I was doing when I was doing this outreach.

s87
00:06:30.400 --> 00:06:32.360
So, yeah, this is the whole uh, pipeline.

s88
00:06:32.360 --> 00:06:35.560
This is created using the Excalidraw MCP server in Cursor.

s89
00:06:35.560 --> 00:06:38.520
It's pretty nice to create a visualization of your code.

s90
00:06:38.520 --> 00:06:45.440
Uh, I'm not going to go into the details, but basically it just replicates um, the workflow that I was doing when doing the outreach.

s91
00:06:45.440 --> 00:06:52.000
And I use LLM APIs in then each of the steps without any framework, without any agent framework.

s92
00:06:52.000 --> 00:06:58.600
So, it made it quite uh, deterministic and I had a lot of control over uh, how this goes.

s93
00:06:58.880 --> 00:07:03.360
Um, in terms of deployment of this uh, workflow, it's a simple cron job.

s94
00:07:03.360 --> 00:07:06.000
So, cron is just something that runs regularly.

s95
00:07:06.000 --> 00:07:08.680
In my case, I run it once every night.

s96
00:07:08.680 --> 00:07:18.600
So, when I'm sleeping, there is this agent, but technically it's just a cron job, a Python script with an LLM API, which is going to read all these hundreds of archive papers,

s97
00:07:18.600 --> 00:07:23.840
uh, and then it might open GitHub issues or it might open pull requests on Hugging Face.

s98
00:07:23.840 --> 00:07:25.560
I'm using GitHub Actions for this.

s99
00:07:25.560 --> 00:07:33.600
Uh, I saw this very nice blog post free cron jobs with GitHub Actions, and actually it's probably the best entry point if you want to set up cron jobs,

s100
00:07:33.600 --> 00:07:40.040
um, because GitHub has a pretty generous tier if you want to get started with like putting simple cron jobs uh, up there.

s101
00:07:40.040 --> 00:07:45.600
And yeah, it makes it really easy for me in the UI to manage all these cron jobs.

s102
00:07:45.600 --> 00:07:52.440
So yeah, every night I have hundreds of uh GitHub issues being created.

s103
00:07:52.520 --> 00:07:56.240
For the tracing part, um I'm using LangFuse.

s104
00:07:56.240 --> 00:07:58.840
Uh yeah, LangFuse also has a a booth here.

s105
00:07:58.840 --> 00:08:01.920
Um LangFuse is pretty great.

s106
00:08:01.920 --> 00:08:10.920
Um I use it mostly for the tracing part, the observability part, just to see what is the LLM doing, what are the inputs, what are the outputs, what are the prompts,

s107
00:08:10.920 --> 00:08:14.360
how much does it cost, latency, and so on.

s108
00:08:14.360 --> 00:08:18.560
Um so yeah, uh I definitely recommend it.

s109
00:08:18.560 --> 00:08:30.520
Um but yeah, as my agents are opening so many GitHub issues every night, I then end up with a massive amount of unread GitHub notifications because people reply to those GitHub issues.

s110
00:08:30.520 --> 00:08:34.039
And that's a lot of work to then reply to all of those issues.

s111
00:08:34.039 --> 00:08:37.440
It's kind of like going through your mailbox.

s112
00:08:37.440 --> 00:08:44.120
So you could wonder, could we also um automate the follow-up to those GitHub uh issues?

s113
00:08:44.120 --> 00:08:51.600
Because initially I was still the GitHub issue creation was done uh by an agent, but I was still the one involved in then doing the follow-up.

s114
00:08:51.600 --> 00:08:57.240
Uh now a few months ago I also automated the the follow-up to those GitHub issues.

s115
00:08:57.240 --> 00:08:59.720
Again, you could think, how should you solve this?

s116
00:08:59.720 --> 00:09:08.680
Should you go for a more deterministic workflow or can you go for a fully autonomous agents, uh an LLM in a loop which runs with some tools and skills?

s117
00:09:08.680 --> 00:09:13.920
Um well, here I went for kind of a fully autonomous agents, uh so it's kind of flexible.

s118
00:09:13.920 --> 00:09:17.720
It's a bit less predictable, but it works quite well.

s119
00:09:17.720 --> 00:09:28.320
Um I went for this because uh in November of last year at AI Engineer in New York, there was a pretty nice workshop by Anthropic on the Claude agent SDK.

s120
00:09:28.320 --> 00:09:32.280
And there they were actually saying that agents might be better than workflows.

s121
00:09:32.280 --> 00:09:42.600
So they So they were kind of contradicting themselves, but they he said that models have become so good that you might actually now start to work with fully autonomous agents rather than a workflow.

s122
00:09:42.600 --> 00:09:49.880
So this is why I went with this approach and I actually am using the Claude agents SDK for this use case.

s123
00:09:49.880 --> 00:09:54.040
Uh there was another pretty nice talk by Cursor also at AI Engineer.

s124
00:09:54.040 --> 00:09:58.320
This was in the European version in London a few months ago.

s125
00:09:58.320 --> 00:10:07.000
There they talked about how they replaced 12,000 lines of custom code, pretty sophisticated workflow, with a very simple 200 lines of code skill.

s126
00:10:07.000 --> 00:10:18.640
Uh actually it's pretty similar for me like I can replace a lot of custom codes, thousands of lines of code, with nowadays just a simple agent with maybe a CLI

s127
00:10:18.640 --> 00:10:25.120
as a tool and a skill and that's it cuz the models have become so good.

s128
00:10:25.120 --> 00:10:30.200
So yeah, in terms of the architecture, this is a bit what it looks like.

s129
00:10:30.200 --> 00:10:37.200
Um so it's actually just the Claude agents SDK which is, I would say, a pretty good Python SDK for building an agent.

s130
00:10:37.200 --> 00:10:45.760
Initially I was using the Claude models, but then I since actually this week I'm using the GLM 5.2 model via Hugging Face inference providers.

s131
00:10:45.760 --> 00:10:55.120
So Hugging Face does offer a service which basically wraps a lot of inference providers like Together AI, Fireworks, Cerebras and so on.

s132
00:10:55.120 --> 00:10:58.839
So you can use a lot of open models in a unified way.

s133
00:10:58.839 --> 00:11:05.400
It's OpenAI compatible or Anthropic compatible and then I deploy this on Modal.

s134
00:11:05.400 --> 00:11:08.839
Modal is also present here today.

s135
00:11:08.839 --> 00:11:18.360
And it's mainly using Bash as a tool so the terminal to basically do Hugging Face commands because it's using the Hugging Face CLI quite a bit.

s136
00:11:18.360 --> 00:11:22.880
So I combine it with the Hugging Face CLI skill which is actually all it needs.

s137
00:11:22.880 --> 00:11:27.800
And then it might comment something on GitHub as a follow-up.

s138
00:11:27.800 --> 00:11:35.960
And it also actually does the posting on Slack because eventually I also want to see the final results on our Slack channel uh from Hugging Face.

s139
00:11:35.960 --> 00:11:44.000
So yeah, given that there's also a lot of hype on GLM 5.2 recently, for example, Cursor uh saw great performance on their Cursor bench.

s140
00:11:44.000 --> 00:11:49.240
Post-training bench is another one uh where it actually beats Opus 4.8 and it's cheaper.

s141
00:11:49.240 --> 00:11:55.800
So, yeah, there's no reason not to use GLM 5.2 uh especially given that I work at Hugging Face.

s142
00:11:55.800 --> 00:11:59.760
Um for the deployment, as I said before, I use Modal.

s143
00:11:59.760 --> 00:12:02.760
It's pretty great if you want to deploy agents.

s144
00:12:02.760 --> 00:12:05.760
Uh in my case, I'm using the batch processing feature.

s145
00:12:05.760 --> 00:12:09.400
So, they allow you to spin up a massive amount of containers all in parallel.

s146
00:12:09.400 --> 00:12:15.320
Every single container is basically one agent loop that is processing one GitHub issue.

s147
00:12:15.320 --> 00:12:17.960
Uh it's super easy to use, I have to say.

s148
00:12:17.960 --> 00:12:21.360
Um and the startups are also pretty fast.

s149
00:12:21.360 --> 00:12:29.280
So, I definitely recommend it if you're building uh agents that are like, for example, running in the background, running overnight, for example.

s150
00:12:29.280 --> 00:12:35.720
Um and then the way I invoke it, yeah, technically I could also just uh deploy this as a cron job.

s151
00:12:35.720 --> 00:12:38.000
Modal, for example, has support for this.

s152
00:12:38.000 --> 00:12:44.240
But typically, the follow-up on the GitHub issues, I still do that actually manually by invoking it as a skill.

s153
00:12:44.240 --> 00:12:47.680
So, I created a skill for this in Cursor.

s154
00:12:47.680 --> 00:12:50.480
Uh I call it process under it Modal.

s155
00:12:50.480 --> 00:13:00.680
And then what it's going to do is it's actually going to invoke an agent, in this case, Composer 2.5, which is like the agent that I'm mostly using in Cursor, which is again going to invoke all the other agents.

s156
00:13:00.680 --> 00:13:04.120
So, that's this is kind of the loop that people are talking about.

s157
00:13:04.120 --> 00:13:09.800
And then finally, it's going to post uh all the results on our Slack channel.

s158
00:13:09.800 --> 00:13:12.520
Uh so yeah, and this is actually what it just posts.

s159
00:13:12.520 --> 00:13:23.880
So, what it does is it basically just posts a huge amount of Hugging Face papers, uh which are these research papers which people can uh make available on Hugging Face because every time someone mentions it

s160
00:13:23.880 --> 00:13:26.920
in a model card or dataset card, we index it on the hub.

s161
00:13:26.920 --> 00:13:32.920
And then it just posts all the artifacts that people have been uploading based on the outreach that we do via GitHub.

s162
00:13:32.920 --> 00:13:36.640
Um so yeah, I do this still in a manual form.

s163
00:13:36.640 --> 00:13:43.920
So I just invoke the skill and then after a few minutes, these messages appear on our Slack channel.

s164
00:13:44.280 --> 00:13:52.000
Um yeah, I just included some fun results because to be honest, it's quite fun to see people interacting with the agents.

s165
00:13:52.000 --> 00:13:56.160
Um to be honest, I don't disclose that it's an agent.

s166
00:13:56.160 --> 00:13:56.440
Why?

s167
00:13:56.440 --> 00:14:00.960
Because I think if people know it's a bot, then they might quickly like close the issue.

s168
00:14:00.960 --> 00:14:05.720
And to be honest, they post exactly the same stuff as I was doing before manually.

s169
00:14:05.720 --> 00:14:09.760
So I don't actually see any reason to to do that.

s170
00:14:09.760 --> 00:14:12.320
Um so and then you see replies like this.

s171
00:14:12.320 --> 00:14:16.800
Hi Niels, thanks a lot for your suggestion and the clear guidance.

s172
00:14:16.800 --> 00:14:21.000
I actually also often times see people using an agent to reply to my agents.

s173
00:14:21.000 --> 00:14:23.640
So it's kind of the that internet nowadays.

s174
00:14:23.640 --> 00:14:27.160
Um but people yeah, make all their artifacts available on Hugging Face.

s175
00:14:27.160 --> 00:14:34.280
And out of the thousands of issues that are being created on Hugging Face, actually so far I've only had two negative comments.

s176
00:14:34.280 --> 00:14:37.200
One guy saying yeah, please close this slop.

s177
00:14:37.200 --> 00:14:38.320
So he closed the issue.

s178
00:14:38.320 --> 00:14:39.320
And then another one.

s179
00:14:39.320 --> 00:14:45.160
But most of the people they just say, yeah, actually it makes perfect sense to make my weights or my data sets available on Hugging Face.

s180
00:14:45.160 --> 00:14:47.160
Like why didn't I think of this?

s181
00:14:47.160 --> 00:14:51.040
Um so it's kind of a win-win I would say.

s182
00:14:51.120 --> 00:14:55.680
Uh I often times also post fun results on our Slack channel.

s183
00:14:55.680 --> 00:15:01.760
Like for example, one time someone a researcher from Apple sent me a DM like, I saw you reached out to me.

s184
00:15:01.760 --> 00:15:10.120
Yeah, technically it's my agent just posting a GitHub issue regarding publishing a new Apple uh the artifacts of an Apple paper on Hugging Face.

s185
00:15:10.120 --> 00:15:17.320
Or for example, it reaches out to Google DeepMind to publish um mathematics data sets.

s186
00:15:17.320 --> 00:15:29.680
Um so a lot of times like I receive emails, the one on the the side, where yeah, they want to publish a 400 GB data set on Hugging Face, but this was also my agent just opening GitHub issues.

s187
00:15:29.680 --> 00:15:32.320
Um Yeah, this is another fun result.

s188
00:15:32.320 --> 00:15:35.800
So, Paddle OCR, it's like a Chinese company.

s189
00:15:35.800 --> 00:15:44.200
They migrated all their OCR models to Hugging Face based on outreach by the agents that create issues for me.

s190
00:15:44.200 --> 00:15:47.800
So, yeah, it's pretty nice.

s191
00:15:47.800 --> 00:15:54.040
Another fun result is like when it when it completes the default template of model cards on Hugging Face.

s192
00:15:54.040 --> 00:16:04.680
So, Mac Mitchell, who also works at Hugging Face, she has a famous paper called model cards for model reporting, making sure that anyone documents their models in a proper way.

s193
00:16:04.680 --> 00:16:08.720
And so, we do provide this template, which you can see on the left side in the Git diff.

s194
00:16:08.720 --> 00:16:20.440
And then, the agent is just completing that template based on the content that it finds based on the paper, like the GitHub readme, the PDF itself, and so on.

s195
00:16:21.280 --> 00:16:28.680
Um yeah, it's also quite funny to see, for example, in this case that it included me in the model card.

s196
00:16:28.680 --> 00:16:32.839
It said, "Model card authors, Niels part of the Hugging Face community science team."

s197
00:16:32.839 --> 00:16:35.640
I never prompted it this way, but it's pretty fun to see.

s198
00:16:35.640 --> 00:16:39.000
Or people are replying, "Thank you for helping me fix my mistakes."

s199
00:16:39.000 --> 00:16:43.800
So, those are all done by the agents.

s200
00:16:44.480 --> 00:16:56.320
Uh I think the most popular GitHub issue that was created was this paper Tiny Recursive Models, which you might have seen, was quite trending both on Hugging Face, but also on Twitter.

s201
00:16:56.320 --> 00:17:02.800
So, yeah, more than 60 people actually upvoted that issue so that the model was released on Hugging Face.

s202
00:17:02.800 --> 00:17:04.360
So, this is again, I think, the win-win.

s203
00:17:04.360 --> 00:17:14.199
So, it's both a win for the researcher, making their research more discoverable on Hugging Face, but it's also, yeah, better for the people then who want to build on top of that research and want to

s204
00:17:14.199 --> 00:17:16.400
use them.

s205
00:17:16.560 --> 00:17:25.480
Uh so, yeah, I I hundreds of GitHub issues where I think I can show uh nice results um where people interact with the agents.

s206
00:17:25.480 --> 00:17:32.800
You might also wonder, yeah, how to avoid slop because you might think, okay, you have an agent uh spamming the whole internet with your GitHub issues.

s207
00:17:32.800 --> 00:17:34.600
Like, should you even do this?

s208
00:17:34.600 --> 00:17:36.680
Again, I already talked about the win-win.

s209
00:17:36.680 --> 00:17:48.920
Um, but a blog post that I highly recommend, if you want to avoid that your agent is just posting slop, is um the LLM Evils FAQ uh by Hamel Husain.

s210
00:17:48.920 --> 00:17:53.320
Uh, I would say he's like the main expert when it comes to LLM evaluation.

s211
00:17:53.320 --> 00:17:59.840
He also has like a a paid course, but he also publishes a lot of stuff for free online including this blog post.

s212
00:17:59.840 --> 00:18:06.080
So, I highly recommend to go through it if you want to learn more about how to evaluate your agents.

s213
00:18:06.280 --> 00:18:14.400
So, my conclusion would be um that open models are actually getting great, especially now with GLM 5.2. You have Deep Seek V4 and so on.

s214
00:18:14.400 --> 00:18:19.960
So, um yeah, we we are able to now replace closed-source models by open ones.

s215
00:18:19.960 --> 00:18:24.480
Uh, for my use case, I would say agents are actually better than uh workflows.

s216
00:18:24.480 --> 00:18:27.480
Uh, they only need a single CLI, which is the Hugging Face CLI.

s217
00:18:27.480 --> 00:18:32.800
They need a single skill, the Hugging Face CLI skill, and a sandbox, and that's all they need to do their work.

s218
00:18:32.800 --> 00:18:36.040
And finally, yeah, don't forget about evaluation.

s219
00:18:36.040 --> 00:18:44.440
Um Finally, uh I can also discuss some other efforts that we do as part of the community science uh team.

s220
00:18:44.440 --> 00:18:46.440
Um, very shortly.

s221
00:18:46.440 --> 00:18:49.880
Um, so, I have a Twitter account that I created.

s222
00:18:49.880 --> 00:18:51.720
It's called Daily Papers.

s223
00:18:51.720 --> 00:19:02.320
And it actually uses the exact same workflow as my agents behind the scenes to post interesting research papers on X. It uh recently crossed 90,000 followers without any involvement of me.

s224
00:19:02.320 --> 00:19:14.200
I just deployed this uh and it posts interesting research papers and artifacts from Hugging Face every 4 hours or every time someone uh releases something cool on the Hugging Face.

s225
00:19:14.200 --> 00:19:15.280
Um so, yeah.

s226
00:19:15.280 --> 00:19:21.160
And I have like Gemini determining the best visual to tweet or to include in the tweet.

s227
00:19:21.160 --> 00:19:29.240
Like for example, this recent tweet where it tweeted out that Nvidia released an optimized version of GLM 5.2 got more than 2,000 likes.

s228
00:19:29.240 --> 00:19:32.240
So, that's pretty cool to see.

s229
00:19:32.240 --> 00:19:41.600
And a final effort that I'm working on right now is a revival of Papers With Code, which is a website that once existed and then was acquired by Meta

s230
00:19:41.600 --> 00:19:43.680
and then sadly it died.

s231
00:19:43.680 --> 00:19:49.880
So, I'm I'm trying to re- revive it and making a research and state-of-the-art easier accessible.

s232
00:19:49.880 --> 00:19:53.120
Um for now it lives at paperswithcode.co.

s233
00:19:54.360 --> 00:19:55.120
Uh so, yeah.

s234
00:19:55.120 --> 00:19:57.320
You can find benchmarks over there.

s235
00:19:57.320 --> 00:20:01.680
For example, for OCR models, all OCR benches like popular benchmark.

s236
00:20:01.680 --> 00:20:12.120
But I'm also making it an educational resource so that people can learn about technical terms like mixed training uh on policy distillation and so on.

s237
00:20:12.120 --> 00:20:12.920
So, yeah.

s238
00:20:12.920 --> 00:20:14.440
That was it for my talk.

s239
00:20:14.440 --> 00:20:16.320
I hope you learned something.

s240
00:20:16.320 --> 00:20:18.969
Thanks all of you for your attention.

s241
00:20:18.969 --> 00:20:20.969
[applause]