Dealing with repetitive tasks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dealing with repetetive tasks in your terminal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For Loops\n",
    "### A 'for loop' is a bash programming languange statement which allows code to be repeatedly executed. Perfect for our metagenomic and genomic needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling each file within a folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Untitled.ipynb\n",
      "file1.txt\n",
      "file2.txt\n",
      "file3.txt\n",
      "file4.txt\n",
      "file5.txt\n",
      "merged_files.txt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ls\n",
    "# make sure the following files are in your current directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file1.txt\n",
      "file2.txt\n",
      "file3.txt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo file1.txt\n",
    "echo file2.txt\n",
    "echo file3.txt\n",
    "# we can call one file at a time but this is tedious this is where for loops come in handy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file1.txt\n",
      "file2.txt\n",
      "file3.txt\n",
      "file4.txt\n",
      "file5.txt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "for i in *.txt;\n",
    "do\n",
    "echo $i;\n",
    "done\n",
    "# See how easy it is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### But we can do more than just call files we can manipulate them too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Untitled.ipynb\n",
      "file1.txt\n",
      "file1.txt_newname.txt\n",
      "file2.txt\n",
      "file2.txt_newname.txt\n",
      "file3.txt\n",
      "file3.txt_newname.txt\n",
      "file4.txt\n",
      "file4.txt_newname.txt\n",
      "file5.txt\n",
      "file5.txt_newname.txt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "for i in *.txt;\n",
    "do\n",
    "cp $i ${i%}_newname.txt;\n",
    "done\n",
    "# See how easy it is\n",
    "ls\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neat but what if you want to the .txt in the original file to fasta without adding to the end of it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dealing with repetitive tasks.ipynb\n",
      "file1.fasta\n",
      "file1.txt\n",
      "file2.fasta\n",
      "file2.txt\n",
      "file3.fasta\n",
      "file3.txt\n",
      "file4.fasta\n",
      "file4.txt\n",
      "file5.fasta\n",
      "file5.txt\n",
      "merged_files\n",
      "sra_list.txt_newname\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "for i in *.txt;\n",
    "do\n",
    "cp $i ${i%.txt}.fasta;\n",
    "done\n",
    "# See how easy it is\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This can be manipulated to run with metagenomic scripts\n",
    "### For example, lets say that we want to run the taxonomic profiling program 'kraken' and have 20 fasta files we want to run. The script goes like this\n",
    "\n",
    "```perl kraken --db Bacteria file1.fasta >> file1.fasta.kraken```\n",
    "\n",
    "### Now imagine having to do this 20 times or 100, this is why loops are great."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#example script and this can be manipulated to whatever program you might want to run\n",
    "for f in *fasta;\n",
    "do\n",
    "perl kraken --db Bacteria $f >> ${f%}.kraken;\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  How to parse/read text files line by line?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/path/to/file1\n",
      "/path/to/file2\n",
      "/path/to/another/file3\n",
      "/path/to/file4\n",
      "/path/to/another/file5\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "cat merged_files.txt\n",
    "#simplest way to read line by line would be like this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### But what if we wanted to read the file from a script and assing that information line to a unix variable\n",
    "### While read loops are perfect when you wat to read an input file, line by line. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/path/to/file1\n",
      "/path/to/file2\n",
      "/path/to/another/file3\n",
      "/path/to/file4\n",
      "/path/to/another/file5\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "while read LINE;\n",
    "do \n",
    "echo $LINE;\n",
    "done < merged_files.txt\n",
    "# While there is a line to process, the loop body will be executed in this case or it cold be the name of a bunch of files \n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#just an example when wanting to download datasets from the SRA\n",
    "while read LINE ; \n",
    "do  \n",
    "fastq-dump --outdir . --gzip --skip-technical --readids --dumpbase --split-files --clip $LINE ; \n",
    "done < sra_list.txt\n",
    "\n",
    "#cold be a list of SRA you might want ot download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using multiple nodes/cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A great advantage of having a cluster is the multiple nodes that allows you to split these repetitive tasks into its own 'computer' allowing the analysis of a large number of files extreemly quickly. In this example we will pretend that we are working in the SDSU anthill cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Need to write a bash script\n",
    "### Each script starts with a \"shebang\" (#!) and you have to tell the shell which interpreter to run the rest of the script, in this case bash script.\n",
    "```#!/bin/bash```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#lets write a bash script first create the file you can use vim, nano or whatever I will use nano\n",
    "nano simple_script.sh\n",
    "# then new window opens copy and paste below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#!/bin/bash\n",
    "#$ -cwd \n",
    "\n",
    "# always comment your script so months from now you remember what it does\n",
    "# A simple script\n",
    "echo “Hey there” >> new.txt\n",
    "echo “Hi there”>> new.txt\n",
    "echo “Whoa there” >> new.txt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next you need to make this script executable\n",
    "### Chmod +x simple_script.sh\n",
    "##### to run just type ```./simple_script.sh```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now lets use it with a cluster. Simple job array A common problem is that you have a large number of jobs to run, and they are largely identical in terms of the command to run. For example, you may have 1000 data sets, and you want to run a single program on them, using the cluster. The naive solution is to somehow generate 1000 shell scripts, and submit them to the queue. This is not efficient, neither for you nor for the head node.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We will use SGE_TASK_ID: SGE_TASK_ID  is set to a unique number in a range that you define, and is incremented as you define it. Make a new file with all file names of interest in this case we want all files that end with fasta:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#!/bin/bash\n",
    "#$ -cwd\n",
    "\n",
    "FILE=$(head -n $SGE_TASK_ID fasta_files.txt | tail -n 1)\n",
    "\n",
    "cp $FILE ${FILE%}_newoutput.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "qsub -t 1-3:1 ./new_script.sh "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the -t flag when doing a qsub will be the the range of SGE_TASK_ID in this case it is set to every number form one to 3 and is incremented by 1. The reange can be any set of numbers you define."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}