You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Submit multiple experiments at the same time with a single bash script. Right now I am submitting single R script with the nohup Rscript "script.R" & option (batch and background mode).
I would like to be able to give a list of specs and then loop through them with a script that submits them all independently.
The text was updated successfully, but these errors were encountered:
I am working right now to generate a workflow that can work on almost "any" batch system (not just on Cranium). The reasons are many....My goal now is to have the workflow run on OSG-Condor, where I was able to get unlimited CPU/hrs of compoutation.
OSG (Open Science Grid) is perfect for the CBDA-SL jobs, since each job is short-burst (5-60 minutes), whether we run a simple binomial with the SuperLearner function, or a Multinomial with a CV.SuperLearner function. The LONI pipeline server (i.e., Cranium at usc.edu) is great, but limited to 3000 jobs to be submitted simultaneously. By using makeflow or Pegasus, we are working to recast the current LONI pipeline workflow into a more general platform that can potentially run on Condor (OSG), PBS (most medium size servers), slurm (XSEDE).
If this step works (it will !!), we can potentially submit ~1 million jobs (in 10K batches) almost at the same time.
Any suggestion is welcome....;-)
Submit multiple experiments at the same time with a single bash script. Right now I am submitting single R script with the nohup Rscript "script.R" & option (batch and background mode).
I would like to be able to give a list of specs and then loop through them with a script that submits them all independently.
The text was updated successfully, but these errors were encountered: