Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the submission of multiple experiments withing a single bash script #2

Open
simeonem opened this issue Jul 18, 2016 · 1 comment

Comments

@simeonem
Copy link
Member

Submit multiple experiments at the same time with a single bash script. Right now I am submitting single R script with the nohup Rscript "script.R" & option (batch and background mode).
I would like to be able to give a list of specs and then loop through them with a script that submits them all independently.

@simeonem
Copy link
Member Author

I am working right now to generate a workflow that can work on almost "any" batch system (not just on Cranium). The reasons are many....My goal now is to have the workflow run on OSG-Condor, where I was able to get unlimited CPU/hrs of compoutation.
OSG (Open Science Grid) is perfect for the CBDA-SL jobs, since each job is short-burst (5-60 minutes), whether we run a simple binomial with the SuperLearner function, or a Multinomial with a CV.SuperLearner function. The LONI pipeline server (i.e., Cranium at usc.edu) is great, but limited to 3000 jobs to be submitted simultaneously. By using makeflow or Pegasus, we are working to recast the current LONI pipeline workflow into a more general platform that can potentially run on Condor (OSG), PBS (most medium size servers), slurm (XSEDE).
If this step works (it will !!), we can potentially submit ~1 million jobs (in 10K batches) almost at the same time.
Any suggestion is welcome....;-)

simeonem added a commit that referenced this issue Jun 27, 2019
R script to generate and save as text file all the synthetic datasets used in the CBDA #2 manuscript.
simeonem added a commit that referenced this issue Jul 3, 2019
Updated pipeline workflow to download the datasets on Cranium used for the CBDA #2 manuyscript
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant