Merge pull request #57 from NIGMS/Update_AWSBatch

Update aws batch
NIGMS · Feb 21, 2025 · 6542311 · 6542311
2 parents 9b33b18 + 478d80e
commit 6542311
Show file tree

Hide file tree

Showing 15 changed files with 67 additions and 0 deletions.
diff --git a/docs/AWS-Batch-Setup.md b/docs/AWS-Batch-Setup.md
@@ -0,0 +1,52 @@
+# AWS Batch Setup
+
+## **0-Before You Begin**
+
+- Ensure you have a VPC, subnets, and security group set up before continuing the AWS Batch setup.
+- Attach following policies to your SageMaker notebook instance:
+
+![](../images/batch-policis.PNG)
+#### **1-Creating a Compute Environment**
+
+Naviate to **'AWS Batch'** in the console. Within AWS Batch on the left side menu navigate to **'Compute environments'**, select **'Create'**.  
+For this tutorial we will be working with EC2, select 'EC2'.
+- Select \"Managed\" for Orchestration type.
+- Enter a name for your compute environment
+- Under service role select "AWSServiceRoleForBatch"
+- Under Instance role select "AWSServiceRoleForBatch"
+- Click 'Next'
+
+![](../images/batch-compute1.PNG)
+
+We have set the fields in the image below to default. Click Next.
+
+![](../images/batch-compute2.PNG)
+
+For Network Configuration select your VPC, subnets, and security groups you would like to utilize. This will allow AWS Batch to create instances that can communicate with each other and have access to acceptable networks.
+
+![](../images/batch-compute3.PNG)
+
+The last step is to review all the configuration made to your compute environment. Once you are satisfied click "Create compute environment".
+
+![](../images/batch-compute4.PNG)
+## **3-Creating a Job Queue**
+
+Now that we have created a compute environment lets create a job queue. Job queues help Batch to stay organized by holding jobs until they can be scheduled to run in a compute environment.
+
+In the AWS Batch console, go to the left side menu, click "Job queues" and click "Create".
+
+- Set orchestration type to EC2
+- Give your queue a name and set its priority. for this tutorial we have se it to '1000' to have the highest priority
+- Associate the compute environment you created in the pervious step.
+- Review and create the job queue.
+
+![](../images/batch-queue1.PNG)
+## **4-Applying Permissions**
+
+For this step we are enabling AWS Batch permissions on EC2 clusters.
+
+- On the left side menu under 'Control settings' click 'Permissions'.
+- Next to Container insights click 'Edit'.
+- Using the toggles select with compute environment should have these permissions and click 'Save changes'.
+
+![](../images/batch-permission1.PNG)
diff --git a/docs/HowToLaunchAWSBatch.md b/docs/HowToLaunchAWSBatch.md
@@ -0,0 +1,15 @@
+## How to Launch AWS Batch for Nextflow Workflows
+
+If you would like to skip building in the cloud and just launch the stack please click the button below and follow the instructions to complete stack deployment. The deployment should take ~5 min and then the resources will be ready for use.
+
+[![Launch Stack](/images/images_for_creating_AWSBatch/LaunchStack.jpg)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-batch-nigms&templateURL=https://nigms-sandbox.s3.us-east-1.amazonaws.com/cf-templates/AWSBatch_template.yaml)
+
+Select `Next` at the bottom right. ![Launch Stack Select](/images/images_for_creating_AWSBatch/LaunchStack_select.png)
+
+Select `Next` again at the bottom right of the screen. ![Launch Stack Specify](/images/images_for_creating_AWSBatch/LaunchStack_specify.png)
+
+Select `I acknowledge` that AWS CloudFormation might create IAM resources and Next. ![Launch Stack Configure](/images/images_for_creating_AWSBatch/LaunchStack_configure.png)
+
+Review and select `Submit`. ![Launch Stack Review](/images/images_for_creating_AWSBatch/LaunchStack_review.png)
+
+When creating the SageMaker notebook please select the corresponding IAM role. The role name will start with aws-batch-nigms-SageMakerExecutionrole-*unique_identifier*. ![SageMaker IAM Role](/images/images_for_creating_AWSBatch/SageMaker_IAM_Role.png)
diff --git a/images/batch-compute1.PNG b/images/batch-compute1.PNG
diff --git a/images/batch-compute2.PNG b/images/batch-compute2.PNG
diff --git a/images/batch-compute3.PNG b/images/batch-compute3.PNG
diff --git a/images/batch-compute4.PNG b/images/batch-compute4.PNG
diff --git a/images/batch-permission1.PNG b/images/batch-permission1.PNG
diff --git a/images/batch-policis.PNG b/images/batch-policis.PNG
diff --git a/images/batch-queue1.PNG b/images/batch-queue1.PNG
diff --git a/images/images_for_creating_AWSBatch/LaunchStack.jpg b/images/images_for_creating_AWSBatch/LaunchStack.jpg
diff --git a/images/images_for_creating_AWSBatch/LaunchStack_configure.png b/images/images_for_creating_AWSBatch/LaunchStack_configure.png
diff --git a/images/images_for_creating_AWSBatch/LaunchStack_review.png b/images/images_for_creating_AWSBatch/LaunchStack_review.png
diff --git a/images/images_for_creating_AWSBatch/LaunchStack_select.png b/images/images_for_creating_AWSBatch/LaunchStack_select.png
diff --git a/images/images_for_creating_AWSBatch/LaunchStack_specify.png b/images/images_for_creating_AWSBatch/LaunchStack_specify.png
diff --git a/images/images_for_creating_AWSBatch/SageMaker_IAM_Role.png b/images/images_for_creating_AWSBatch/SageMaker_IAM_Role.png