Optimal window size to reduce memory usage

[XieHongX]

Hi,

I am analysing a data set of 18+17 sample size and ~1.5 GB genome. I used block length 145bp because the genetic diversity of my species is lower. Then I want to use a window size of 500 as in your original paper, so run gimble windows with -w 150000 -s 30000. it turns out giving error message because of out of memory usage (there is 512 GB of memory in total!). Shall I increase the window size or decrease it in order to reduce memory usage?

Alternatively, is there a way to reduce memory usage while keeping the desired sample size?

Best,
Hongxin

[XieHongX]

The error message is:
...
gimble.py", line 1014, in tally_variation
mutuples = np.concatenate(
^^^^^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 188. GiB for an array with shape (5038150000, 5) and data type int64

[GertjanBisschop]

@DRL . I have no intuition here on whether there is a window size setting that will increase or reduce the memory usage. I feel like memory usage will rather depend on the number of blocks in total.

What I do see when looking at the code is the fact that we concatenate all blocks (here) with a window index effectively producing a copy of an existing array. We could easily process the array of blocks using for loops (without numpy logic) and instead numba-compile the little function with for-loop logic. Numba already is a dependency of gimble and this would seriously reduce the memory overhead. Is this worth opening an issue for @DRL?

[XieHongX]

Hi- I tried reducing the sliding window step, so reducing the total windows created, and that works. But I think the memory usage may still set a limit for the usage of gimble. For example when analysing species with even larger genomes (mammals for example)

[GertjanBisschop]

@XieHongX There is no quick fix to this however. Cleverly reducing memory usage would require some work from our end.

@KLohse is splitting your genomes prior to making blocks a potential issue for this? Ideally we would have function then that allows you to combine bSFS counts once counts have been generated for each of the separate genome chunks.