WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-561cZmir5Q NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-561cZmir5Q.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.080 --> 00:00:13.880 Okay, let's get it started. s2 00:00:13.880 --> 00:00:15.560 Uh My name is Akram. s3 00:00:15.560 --> 00:00:18.120 I'm a machine learning engineer at Altos Labs. s4 00:00:18.120 --> 00:00:32.320 Altos Labs is a biotech startup and the goal is to restore cell health and resilience through cellular rejuvenation to inverse disease and disabilities that can happen throughout the life. s5 00:00:32.320 --> 00:00:37.280 And the title of my talk is uh from tokens to cells. s6 00:00:37.280 --> 00:00:51.000 And this is uh my kind of view as someone without bio background to kind of looking into the engineering challenges of foundation models for single-cell biology. s7 00:00:51.000 --> 00:00:54.640 And what I want to talk about first, what is single-cell? s8 00:00:54.640 --> 00:00:56.120 Why do we care about single-cell? s9 00:00:56.120 --> 00:00:57.800 How do How do we measure it? s10 00:00:57.800 --> 00:01:01.280 What are the problems with the data, getting the data? s11 00:01:01.280 --> 00:01:08.880 And then looking at current state of the art foundation models and then some takeaways in the end. s12 00:01:08.880 --> 00:01:12.720 Uh so, what is single-cell and why do we care about it? s13 00:01:12.720 --> 00:01:17.240 I would like to start with my favorite example, Yamanaka factor. s14 00:01:17.240 --> 00:01:22.320 In 2006, Shinya Yamanaka discovered four transcription factors. s15 00:01:22.320 --> 00:01:36.080 There are four specific type of a proteins that when they were overexpressed in a cell, uh they did it for in a cell which was age old skin cell, s16 00:01:36.080 --> 00:01:42.720 it could reprogram the skin cell old skin cell to embryonic stem cell like state. s17 00:01:42.720 --> 00:01:53.040 So, basically from cell type skin cell old, it could reprogram it back to a young embryonic stem cell type. s18 00:01:53.040 --> 00:02:07.520 Uh type and this was a breakthrough for biology and then got him Nobel Prize later in 2012 uh because of the application and the new chapters possibilities for medicine. s19 00:02:07.520 --> 00:02:13.480 From regenerative medicine for like to be able to kind of regenerate tissues or organs. s20 00:02:13.480 --> 00:02:22.520 Uh when we can like reprogram specific cell to any cell that we want to partially programming and aging application. s21 00:02:22.520 --> 00:02:29.480 For partial programming specifically is that we can also turn out that we can also only change the age of the cell. s22 00:02:29.480 --> 00:02:31.600 We don't have to change the type. s23 00:02:31.600 --> 00:02:46.720 So, by changing the age of the cell the goal is the hope is we can restore some of youthful function that we lose toward life as we age and then kind of like having this as a medicine maybe like an mRNA medicine s24 00:02:46.720 --> 00:02:51.000 uh for curing the disease that the we encounter as we age. s25 00:02:51.000 --> 00:02:54.120 And that's a one example. s26 00:02:54.120 --> 00:03:00.480 And then I think this year 2026 we got the first type of reprogramming medicine. s27 00:03:00.480 --> 00:03:06.677 I think it's OSK and uh it's going to be tested in a human. s28 00:03:06.677 --> 00:03:06.840 [clears throat] s29 00:03:06.840 --> 00:03:11.040 So, it took kind of like 20 20 years. s30 00:03:11.160 --> 00:03:22.360 And uh and the reason the other reason is that why we want to study a single cell is that we would like to model to have this unified holistic s31 00:03:22.360 --> 00:03:31.680 view of single cell to be able to model cell and from there hopefully we can model tissue and organ and then the entire human. s32 00:03:31.680 --> 00:03:43.880 And there are like projects like human cell atlas that actually started this effort this initiative and then they like you know they have they have mapped every single cell inside human. s33 00:03:43.880 --> 00:03:59.520 And the goal ultimately is something like maybe for a ultimately some after some long time we can actually actually model like model human body or there are terminologies like virtual cell, s34 00:03:59.520 --> 00:04:02.160 virtual tissue, virtual human, digital twins. s35 00:04:02.160 --> 00:04:09.959 They all saying the same thing that the more we can model this living organisms. s36 00:04:10.120 --> 00:04:18.600 The better we are in understanding our body and the how we can treat medicine, we can develop drugs. s37 00:04:19.560 --> 00:04:32.240 And the other the other problem is that so we you know, I think for this audience we know about Moore's law, the computer is getting like double every year. s38 00:04:32.240 --> 00:04:35.840 And then we have exact opposite on drug development. s39 00:04:35.840 --> 00:04:45.720 Basically, the number of drugs that developed each year is kind of declining, which is surprising with all the advances in technology, in AI. s40 00:04:45.720 --> 00:04:47.080 This is surprising to see. s41 00:04:47.080 --> 00:04:50.640 And then drug development is a field that failure is you know, very normal. s42 00:04:50.640 --> 00:04:57.320 Maybe the acceptance rate is kind of like you know, 5% or even less. s43 00:04:57.320 --> 00:05:08.800 And when we're looking at drug development pipeline from the early like research and development all the way to preclinical, clinical trial and then the final stage. s44 00:05:08.800 --> 00:05:16.320 The whole pipeline it could take up to 10 years easily and then it cost it can cost billions. s45 00:05:16.320 --> 00:05:27.680 And the goal here is that with the advances of AI and also like with this virtual cell, virtual organ, virtual human, the goal is that we can kind of like reduce s46 00:05:27.680 --> 00:05:30.760 this time, improve reduce this timeline. s47 00:05:30.760 --> 00:05:34.800 And and also we're looking at the entire timeline. s48 00:05:34.800 --> 00:05:43.680 Let's say if we only look at like research and development, the models like you know, protein design and the stuff, we might like save like you know, few years here, s49 00:05:43.680 --> 00:05:47.240 but at the end maybe it's not going to help for the entire pipeline. s50 00:05:47.240 --> 00:05:55.400 So, it's important to have innovation and breakthrough across all pipeline. s51 00:05:56.680 --> 00:06:03.240 Now that we know single cell is important, how can we measure single cell? s52 00:06:03.240 --> 00:06:05.760 So, looking at single, yeah, it's amazing. s53 00:06:05.760 --> 00:06:07.400 This is just one single cell. s54 00:06:07.400 --> 00:06:11.880 There is a lot going on inside that one little tiny organism. s55 00:06:11.880 --> 00:06:18.440 Uh there are different modalities and each modality is measure different things. s56 00:06:18.440 --> 00:06:25.480 Uh from genome, which is like, you know, DNA sequencing is almost, you know, same for all cells, to RNA sequence. s57 00:06:25.480 --> 00:06:33.240 RNA sequence or transcriptomic is like the profile of the genes that they are expressed at each time. s58 00:06:33.240 --> 00:06:42.240 And then that it's like a matrix and then basically cell it says for each cell uh what are the genes what are the genes how many genes are expressed s59 00:06:42.240 --> 00:06:43.840 and then in which quantity. s60 00:06:43.840 --> 00:06:50.960 And is like 20,000 K. It's it's 20,000 scale. s61 00:06:51.160 --> 00:06:54.640 And the other maybe modality is proteomics. s62 00:06:54.640 --> 00:07:03.600 Proteomics maybe is like what maybe we care about more like because the proteins are the one that they're doing most of the functions in our body. s63 00:07:03.600 --> 00:07:11.160 But the problem is that proteomics is a very hard to measure and you know, there are ways to measure it but it's a very low throughput. s64 00:07:11.160 --> 00:07:15.560 And morphology is about like the cell shape and structure. s65 00:07:15.560 --> 00:07:24.080 And there are like really good imaging technologies to kind of also measure to have like microscopic imaging from single cell. s66 00:07:24.080 --> 00:07:39.360 And um it's very I've also useful for a special special dimension kind of like, you know, knowing the place of like, you know, how the single cell is located within tissue. s67 00:07:39.360 --> 00:07:47.040 Um so, from all these modalities maybe the one that has been used mostly for foundation model training, I would say is RNA-seq. s68 00:07:47.040 --> 00:07:59.160 And there is an issue that the technology is is it's easier to measure and the technology is like because of like PC PCR technologies and the stuff, it's easier to a scale. s69 00:07:59.160 --> 00:08:10.000 So we have like usually data set for single cell RNA in the scale of like tens of tens of uh millions of cells to even like you know, I've heard 500 million s70 00:08:10.000 --> 00:08:13.720 cells and then kind of like even like there's a project 1 billion cells. s71 00:08:13.720 --> 00:08:17.480 But usually what they talk about is RNA-seq. s72 00:08:17.480 --> 00:08:26.880 And then also this is also a problem that if you really want to understand biology and then starting from single cell, we really need to kind of have technological s73 00:08:26.880 --> 00:08:30.480 advances in other dimensions as well. s74 00:08:30.480 --> 00:08:40.680 And now having said that, let's say we got like you know, gathered 1 billion RNA-seq, 1 billion cells, 1 billion samples of RNA-seq data. s75 00:08:40.680 --> 00:08:43.760 Is this good for foundation models that we can you know, train on? s76 00:08:43.760 --> 00:08:48.360 And then the the problem is uh it's it's very hard. s77 00:08:48.360 --> 00:08:54.800 The nature of the data is if you measure two identical cell at the they don't read the same. s78 00:08:54.800 --> 00:09:00.760 And then the problem is the problem is this is very very heterogeneous. s79 00:09:00.760 --> 00:09:13.640 I'll give you give an example here that usually cells they're living organism, they go through a lot of changes and cycles like from like you know, growing to uh kind of copying s80 00:09:13.640 --> 00:09:16.560 RNA to cell division. s81 00:09:16.560 --> 00:09:23.560 And usually some of these changes happens like in a burst, not like it's not something you know, continuous. s82 00:09:23.560 --> 00:09:30.760 And what we are measuring with current uh single cell sequencing is like a snapshot. s83 00:09:30.760 --> 00:09:36.160 We're taking a snapshots from from a whole movie. s84 00:09:36.480 --> 00:09:43.800 Uh now with that and then also there's their biological reason that the data is very noisy, it's very heterogeneous. s85 00:09:43.800 --> 00:09:53.960 Also, there is like technical reasons that usually if you measure in different labs, different machines, also the data would be the data wouldn't be the same. s86 00:09:53.960 --> 00:10:01.720 Now, let's see how what are look at some of the foundation models that we have and then see what they're doing with this uh, data. s87 00:10:01.720 --> 00:10:10.480 Um, and then also like maybe a small note here that given the nature that is very high dimensional is like multi-state. s88 00:10:10.480 --> 00:10:18.400 I think maybe, you know, some might argue that uh, a quantum computing would be there like kind of like natural fit for this application. s89 00:10:18.400 --> 00:10:21.400 And then, yeah, we don't know, it might be true. s90 00:10:21.400 --> 00:10:30.320 Uh, but you know, until we have quantum compute, for now we want to see what we can get with current state of the art AI. s91 00:10:30.320 --> 00:10:34.520 And now, uh, I want to start with like transformer-based models. s92 00:10:34.520 --> 00:10:43.720 And there have been really good like, you know, papers and models out there from the communities starting from like SCGPT and Informer. s93 00:10:43.720 --> 00:10:52.760 Uh, and what they want, they kind of like treat each uh, cell like a sentence and then genes like tokens. s94 00:10:52.760 --> 00:10:56.360 So, basically cell cell are built from genes. s95 00:10:56.360 --> 00:11:11.600 And what they do is like uh, uh, they um, like you know, BERT BERT style model like from language that kind of uh, masking some of the genes and they're trying s96 00:11:11.600 --> 00:11:19.280 having the model to predict that them predict that genes uh, that gene counts. s97 00:11:19.280 --> 00:11:22.680 So, looking at the data is like a matrix. s98 00:11:22.680 --> 00:11:28.800 Uh, and then for each cell we have like, you know, we have the count of uh, each genes. s99 00:11:28.800 --> 00:11:34.160 How many genes are, you know, how many genes are active, uh, per cell. s100 00:11:34.160 --> 00:11:45.640 And then it's trying to mask this to like, uh, attention, bidirectional attention across, and then kind of understand the relationship between, you know, between these genes. s101 00:11:45.640 --> 00:11:47.840 And, uh, this is how it's trained. s102 00:11:47.840 --> 00:11:54.880 And then like there are some, uh, uh, some some some downstream tasks that this model is used to. s103 00:11:54.880 --> 00:12:06.560 Uh, one is for, uh, so like, you know, predicting cell type, kind of predicting, uh, also, um, perturbation response modeling. s104 00:12:06.560 --> 00:12:19.680 Uh, but overall, when we look into these models, what they try to do, they try to kind of get the single cell data and then compress it in a kind of a latent vector and then do either decoding s105 00:12:19.680 --> 00:12:26.080 for like generative task or like classification for, uh, classic for other task. s106 00:12:26.080 --> 00:12:31.360 And then what happens is when you, uh, compress this data, we're losing a lot of information. s107 00:12:31.360 --> 00:12:36.080 It doesn't preserve the, it doesn't preserve the information. s108 00:12:36.080 --> 00:12:51.000 And then that's why when we're looking into this model, like, um, these models, we see that sometimes, sometimes like, uh, maybe, uh, simple linear models are on par or like sometimes even outperforming s109 00:12:51.000 --> 00:13:00.000 this, uh, these, uh, models that, you know, it's like, you know, complex models with a lot of compute is being used to train those. s110 00:13:00.000 --> 00:13:09.120 Um, then also we had like two papers last year at NeurIPS, we did like comprehensive benchmarking on uh, on these models. s111 00:13:09.120 --> 00:13:17.480 One is, uh, multimodal data for like imaging and RNA-seq, and the other one is for perturbation response modeling. s112 00:13:17.480 --> 00:13:27.720 So I have the links at the end of my talk, uh, go ahead and check them out, but like the final, the results, uh, the uh, what we found out was the s113 00:13:27.720 --> 00:13:41.080 Uh, that also is known in the community that even though these models are very expensive to train, uh, at the end they're not performing as well comparing to, you know, other domain language, s114 00:13:41.080 --> 00:13:43.960 uh, and and imaging. s115 00:13:43.960 --> 00:13:57.600 And this is our newer and then so transformer-based also we have like flow matching-based models that you you started to try to uh, predict the distribution of the data and s116 00:13:57.600 --> 00:14:03.560 uh, starting from like, you know, Gaussian noise and then trying to to match the distribution. s117 00:14:03.560 --> 00:14:16.080 And we have uh, this paper, uh, we have this model PrimeFlow uh, that is available on archive and it seems that these models they seem to do better. s118 00:14:16.080 --> 00:14:30.360 There is a better comparing to trans autoregressive-based models, um, because it's tries to kind of match the distribution predict the distribution rather than kind of, you know, understanding the s119 00:14:30.360 --> 00:14:38.800 understanding the data and compressing it into into uh, uh, into into a a latent vector. s120 00:14:38.800 --> 00:14:43.360 Um, and this is like some of the results that uh, we have. s121 00:14:43.360 --> 00:14:55.800 So, on the left we see like PrimeFlow it's the green the green dots are uh, ground truth label and then we see that PrimeFlow is kind of trying to match the distribution. s122 00:14:55.800 --> 00:15:07.040 Whereas other models like CPADR autoencoder-based is kind of like just mapping to the it's trying to predict the mean rather than understanding understanding the distribution. s123 00:15:07.040 --> 00:15:11.400 And then we have like MMD score at the bottom. s124 00:15:11.560 --> 00:15:22.920 Uh, so yeah, that was uh, my top three things to take away from this talk is that single cell is important and it's important to understand this. s125 00:15:22.920 --> 00:15:25.840 It has application for cellular rejuvenation. s126 00:15:25.840 --> 00:15:32.320 It's a path to like digital human and they're helping with drug development cycle. s127 00:15:32.320 --> 00:15:38.480 Then also we looked at at the different modalities to measure from single cell. s128 00:15:38.480 --> 00:15:47.280 RNA sequence data is the one that maybe it's we have it more available in a scale and also it's measuring something important in expression profile. s129 00:15:47.280 --> 00:16:02.080 But at the same at the same time that we would like to kind of get more data, work on the scale, but it's important that to work on quality of the data of the way that we are measuring this data to be a bit more realistic s130 00:16:02.080 --> 00:16:08.920 of the real organism than than just like you know scale the data the way it is. s131 00:16:08.920 --> 00:16:20.360 And then my final conclusion is that for now it seems flow matching models they're doing better for single cell data trying to match to the distribution. s132 00:16:20.360 --> 00:16:30.880 And also it to be able to scale these models that you know they can also do well on the data they haven't seen, they haven't trained on. s133 00:16:30.880 --> 00:16:38.920 We would need you know massive scaling scaling of the data I would say and then also the quality and the way that we measure data. s134 00:16:38.920 --> 00:16:45.360 And then with that yeah we have like three papers SCGENE scope, perturbation last year in NeurIPS and then PrimeFlow. s135 00:16:45.360 --> 00:16:48.640 You check them out and yeah, let me know if you have any questions. s136 00:16:48.640 --> 00:16:50.839 Thank you.