RAM usage from scVelo functions.

[EmilyMiddendorp]

Hi!
I am analysing a dataset of 87012 cells and reducing the number of genes as much as possible every step. I am doing the analysis on a Mac M1 silicon 8 cores with 16GB of RAM and on a Linux virtual machine 16 cores 32GB of RAM. In both machines, some processes are constantly killed. It seems like there are many RAM peaks were the code gets killed.
I was wondering what is the (estimated) RAM usage for scVelo and for how many cells? Is there a way to optimise the RAM use of scVelo?

[WeilerP]

@EmilyMiddendorp, there is no analysis on RAM usage I am aware of. Which functions exactly are causing problems? Are you experiencing problems prior to imputing the data? The imputed data is dense, i.e. will use up more memory. One major bottleneck is using scvelo.tools.recover_dynamics with multiple jobs. The issue will be dealt with in due time. ATM, the code base is unit tested first to not accidentally introduce changes when e.g. refactoring scvelo.tools.recover_dynamics.

[EmilyMiddendorp]

I had lots of troubles (still having) with velocity_graph.
I ran most of the computationally expensive processes on a more powerful virtual machine, and saved it on h5ad files. Right now I am trying to load some of this files on my 32GB VM, with the backend='r' option, but it still keeps using about 80% of my RAM so I can not do any further analysis. The h5ad file is about 10GB. Is there a way of working with these files that won't eat up my memory?

[WeilerP]

There's nothing we can do about the file size on our side. You can always remove all redundant entries and should make sure to use sparse matrices whenever possible.
I briefly skimmed through the function: With default values, you'll need the imputed spliced data, velocities, neighbor graph and the latent space representation you are working with (should be UMAP if you calculated it).