Vector memory exhausted when running methylKit

[yaaminiv]

I am running methylKit but ran into an error:

I thought I could use the R Studio server to run the code with more memory. When I tried opening each server, it seemed like they were frozen! I was unable to interact with R Studio through the server on roadrunner (I tried clicking things and nothing happened), and the R Studio server on emu was stuck for ~10 minutes trying to terminate R.

Any suggestions for using the servers, or any non-server workarounds/suggestions for processing my files?

[yaaminiv]

For reference, here's my script: https://github.com/RobertsLab/project-oyster-oa/blob/master/code/Haws/04-methylKit.Rmd#L105

In a previous iteration of the script, I was able to process my files without errors: https://yaaminiv.github.io/Hawaii-Gigas-Methylation-Analysis-Part7/

I'm testing the normalizeCoverage function within methylKit, which I think is causing the vector memory issue.

[sr320]

some odd filenames zr3644_1_R1_val_1_val_1_val_1_bismark_bt2_pe..CpG_report.merged_CpG_evidence.cov

[sr320]

@yaaminiv can you give me access to bams? I think this filetype does influence memory; deduplicated and sorted

[sr320]

found the bams

[yaaminiv]

Sorry for the delay! Odd filenames are because I trimmed 3 times (2 trimgalore adapter removal, 1 poly-G tail removal) and I never bothered to rename

[yaaminiv]

can you give me access to bams? I think this filetype does influence memory; deduplicated and sorted

Speaking of bams, I thought when we were working with the coral data that we decided to process the merged cov files with methylkit, and not bams, because they were already merged. Wouldn't the bams also be larger?

[kubu4]

When I tried opening each server, it seemed like they were frozen!

Can you please clarify this point? What do you mean "each server"? Meaning Emu and Roadrunner?

I logged into RStudio Server on both computers without issue (with my user account). This might suggest that you already had an existing RStudio Server session open from previous usage. Possibly tied into the default behavior where RStudio attempts to load the last used RStudio session. If the previous session had a lot of variables/dataframes (and, those things contained large amounts of data), it might cause the session to freeze and/or take a long time to open as it loads all of those things into memory.

I'll see if I can figure out how to disable this "feature" for you. Also, which user account are you using?

[yaaminiv]

moving the ~/.rstudio directory to ~/.rstudio-backup.

I wasn't able to move the directory itself to ~/.rstudio-backup

I moved the individual monitored and sessions directories:

When I open the R Studio server, it opens my previous script and is still frozen. Is there a different way I should be moving the directories?

[kubu4]

You have to delete the previous backup first (or, move to a differently named destination).

I went ahead and did this.

Try RStudio Server again.

[yaaminiv]

Still frozen on my script when I log into the srlab account

[kubu4]

You can load it now. I restarted RStudio Server. But, I'd refer to @sr320 's reply below RE: Mox.

[yaaminiv]

With respect to the original vector memory issue, I was able to run filterByCoverage + normalizeCoverage on the R Studio server (finally) without running into a memory issue. So at least that's good

[sr320]

is mox a viable option here? I have always had issues with R Server.

…

On Mon, Mar 8, 2021 at 1:13 PM Yaamini Venkataraman < ***@***.***> wrote: Still frozen on my script when I log into the srlab account — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1137 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB4PN6ALSQJSNK6CVWKLGLTCU4WZANCNFSM4YYMND7Q> .

[kubu4]

At this point, that'd be my suggestion as well - use R from command line instead. Might not even need Mox.

[sr320]

I presume the original error in question here is something you got from running on your local machine?

[kubu4]

It's basically the same thing as running an R script from any command line:

R name_of_R-script.R

So, inside an SBATCH script for Mox, you'd have all of your header junk specifying run time, directory, etc. Then, you'd have the line above which would call and execute an R script that you have somewhere.

[yaaminiv]

I was running it on genefish because I figured it would take a while and wanted to be able to do other things on my local machine, but I can try my laptop.

R studio server is unfrozen, but I'm running into different errors since I think I'm running a more updated version of methylKit. I'll see if I can troubleshoot it and get to the command that caused my initial error

[kubu4]

Alternatively, if you edit the top line of the SBATCH file to point to the R installation, then you can put the contents of your R script in the SBATCH file, just like you'd normally do:

Change this line:

#!/bin/bash

to

#!/gscratch/srlab/programs/R-3.6.2/bin/R

[sr320]

My 2 cents is make sure it goes error free on locally on your laptop before moving off to new platform (noting I get a earlier methylkit error trying to reproduce your code.

…

On Mon, Mar 8, 2021 at 1:33 PM kubu4 ***@***.***> wrote: It's basically the same thing as running an R script from any command line: R name_of_R-script.R So, inside an SBATCH script for Mox, you'd have all of your header junk specifying run time, directory, etc. Then, you'd have the line above which would call and execute an R script that you have somewhere. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1137 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB4PNZDBGTNQHBZ5OU5PULTCU6VTANCNFSM4YYMND7Q> .

[yaaminiv]

definitely. I'm trying to see if I can get error free on the R Studio Server, but I think I froze it again. I'll try on my local machine with a subset of files to check, then move to mox for the full dataset

[yaaminiv]

Update: was able to run filterByCoverage and normalizeCoverage with the Manchester data (8 samples) without running into a vector memory issue. So I think R was having a problem with the amount of data being processed at once (24 Hawaii samples).

Still working on debugging script with Manchester data, before running the same code on the Hawaii data on Mox.