examples/nipype_tutorial.ipynb

{
 "metadata": {
  "_draft": {
   "nbviewer_url": "gisting : nipype.ipynb\r\n"
  },
  "name": "nipype_tutorial"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Dissecting Nipype Workflows\n",
      "\n",
      "<center>\n",
      "Nipype team | contact: satra@mit.edu  |  nipy.org/nipype\n",
      "<br>\n",
      "(Hit Esc to get an overview)\n",
      "</center>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Contributors\n",
      "\n",
      "http://nipy.org/nipype/about.html#code-contributors\n",
      "\n",
      "# Funding\n",
      "\n",
      "- 1R03EB008673-01 from NIBIB, Satrajit Ghosh, Susan Whitfield-Gabrieli\n",
      "- 5R01MH081909-02 from NIMH, Mark Esposito\n",
      "- INCF\n",
      "\n",
      "# Conflict of interest\n",
      "\n",
      "<center>\n",
      "Satrajit Ghosh: TankThink Labs, LLC\n",
      "</center>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# What is Nipype?\n",
      "\n",
      "<center>\n",
      "<img src=\"https://raw.github.com/satra/intro2nipype/master/images/nipype.png\" width=\"50%\" />\n",
      "<br>\n",
      "created by: Arno Klein (www.mindboggle.info)\n",
      "</center>\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Make life a little easier\n",
      "\n",
      "<img src=\"https://raw.github.com/satra/intro2nipype/master/images/EDC.png\" />\n",
      "\n",
      "Poline _et al._ (2012)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Many workflow systems out there\n",
      "\n",
      "- [BioImage Suite](http://www.bioimagesuite.org/)\n",
      "- [BIRN Tools](https://wiki.birncommunity.org/x/LgFrAQ)\n",
      "- [BrainVisa](http://brainvisa.info)\n",
      "- [CambaFX](http://www-bmu.psychiatry.cam.ac.uk/software/)\n",
      "- [JIST for MIPAV](http://www.nitrc.org/projects/jist/)\n",
      "- [LONI pipeline](http://pipeline.loni.ucla.edu)\n",
      "- [MEVIS Lab](http://www.mevislab.de)\n",
      "- [PSOM](http://code.google.com/p/psom/)\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "source": [
      "<ul>\n",
      "<li><a href=\"http://www.bioimagesuite.org/\">BioImage Suite</a></li>\n",
      "<li><a href=\"https://wiki.birncommunity.org/x/LgFrAQ\">BIRN Tools</a></li>\n",
      "<li><a href=\"http://brainvisa.info\">BrainVisa</a></li>\n",
      "<li><a href=\"http://www-bmu.psychiatry.cam.ac.uk/software/\">CambaFX</a></li>\n",
      "<li><a href=\"http://www.nitrc.org/projects/jist/\">JIST for MIPAV</a></li>\n",
      "<li><a href=\"http://pipeline.loni.ucla.edu\">LONI pipeline</a></li>\n",
      "<li><a href=\"http://www.mevislab.de\">MEVIS Lab</a></li>\n",
      "<li><a href=\"http://code.google.com/p/psom/\">PSOM</a></li>\n",
      "</ul>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Solution requirements\n",
      "\n",
      "Coming at it from a developer's perspective, we needed something\n",
      "\n",
      "- lightweight\n",
      "- scriptable\n",
      "- provided formal, common semantics\n",
      "- allowed interactive exploration\n",
      "- supported efficient batch processing\n",
      "- enabled rapid algorithm prototyping\n",
      "- was flexible and adaptive\n",
      "- part of an ecosystem"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Existing technologies\n",
      "\n",
      "**shell scripting**:\n",
      "\n",
      "  Can be quick to do, and powerful, but only provides application specific \n",
      "  scalability, and not easy to port across different architectures.\n",
      "\n",
      "**make/CMake**:\n",
      "\n",
      "  Similar in concept to workflow execution in Nipype, but again limited by the\n",
      "  need for command line tools and flexibility in terms of scaling across\n",
      "  hardware architectures (although see [makeflow](http://nd.edu/~ccl/software/makeflow).\n",
      "\n",
      "**Octave/MATLAB**:\n",
      "\n",
      "  Integration with other tools is *ad hoc* (i.e., system call) and dataflow is\n",
      "  managed at a programmatic level. However, see [PSOM](http://code.google.com/p/psom/) which offers a nice\n",
      "  alternative to some aspects of Nipype for Octave/Matlab users.\n",
      "\n",
      "**Graphical options**: (e.g., [LONI Pipeline](http://pipeline.loni.ucla.edu), [VisTrails](http://www.vistrails.org/))\n",
      "\n",
      "  Are easy to use but reduces flexibility relative to scripting options."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "#Nipype architecture\n",
      "\n",
      "<img src=\"https://raw.github.com/satra/intro2nipype/master/images/arch.png\" width=\"100%\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "notes"
      }
     },
     "source": [
      "* **Interface**: Wraps a program or function\n",
      "\n",
      "- **Node/MapNode**: Wraps an `Interface` for use in a Workflow that provides\n",
      "  caching and other goodies (e.g., pseudo-sandbox)\n",
      "- **Workflow**: A *graph* or *forest of graphs* whose nodes are of type `Node`,\n",
      "  `MapNode` or `Workflow` and whose edges represent data flow\n",
      "\n",
      "* **Plugin**: A component that describes how a `Workflow` should be executed"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Software interfaces\n",
      "\n",
      "Currently supported (5-2-2013). `Click here for latest <http://www.mit.edu/~satra/nipype-nightly/documentation.html>`_\n",
      "\n",
      "- [AFNI](http://afni.nimh.nih.gov/afni)\n",
      "- [ANTS](http://www.picsl.upenn.edu/ANTS)\n",
      "- [BRAINS](http://www.psychiatry.uiowa.edu/mhcrc/IPLpages/BRAINS.htm)\n",
      "- [Camino](http://www.cs.ucl.ac.uk/research/medic/camino)\n",
      "- [Camino-TrackVis](http://www.nitrc.org/projects/camino-trackvis)\n",
      "- [ConnectomeViewerToolkit](http://www.connectomeviewer.org)\n",
      "- [dcm2nii](http://www.cabiatl.com/mricro/mricron/dcm2nii.html)\n",
      "- [Diffusion Toolkit](http://www.trackvis.org/dtk)\n",
      "- [FreeSurfer](http://freesurfer.net)\n",
      "- [FSL](http://www.fmrib.ox.ac.uk/fsl)\n",
      "- [MRtrx](http://www.brain.org.au/software/mrtrix/index.html)\n",
      "- [Nipy](http://nipy.org/nipy)\n",
      "- [Nitime](http://nipy.org/nitime)\n",
      "- [PyXNAT](http://github.com/pyxnat)\n",
      "- [Slicer](http://www.slicer.org)\n",
      "- [SPM](http://www.fil.ion.ucl.ac.uk/spm)\n",
      "\n",
      "Most used/contributed policy!\n",
      "\n",
      "Not all components of these packages are available."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "source": [
      "# Workflows\n",
      "\n",
      "- Properties:\n",
      "\n",
      "    - processing pipeline is a directed acyclic graph (DAG)\n",
      "    - nodes are processes\n",
      "    - edges represent data flow\n",
      "    - compact represenation for any process\n",
      "    - code and data separation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Execution Plugins\n",
      "\n",
      "Allows seamless execution across many architectures\n",
      "\n",
      "  - Local\n",
      "\n",
      "    - Serial\n",
      "    - Multicore\n",
      "\n",
      "  - Clusters\n",
      "\n",
      "    - HTCondor\n",
      "    - PBS/Torque/SGE/LSF (native and via IPython)\n",
      "    - SSH (via IPython)\n",
      "    - Soma Workflow"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Learn Nipype concepts in 10 easy steps\n",
      "\n",
      "\n",
      "1. Installing and testing the installation \n",
      "2. Working with interfaces\n",
      "3. Using Nipype caching\n",
      "4. Creating Nodes, MapNodes and Workflows\n",
      "5. Getting and saving data\n",
      "6. Using Iterables\n",
      "7. Function nodes\n",
      "8. Distributed computation\n",
      "9. Connecting to databases\n",
      "10. Execution configuration options"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Step 1. Installing Nipype\n",
      "\n",
      "## Scientific Python:\n",
      "\n",
      "* Debian/Ubuntu/Scientific Fedora\n",
      "* [Canopy from Enthought](https://www.enthought.com/products/canopy/)\n",
      "* [Anaconda from Contnuum Analytics](https://store.continuum.io/cshop/anaconda/)\n",
      "\n",
      "## Installing Nipype:\n",
      "\n",
      "* Available from [@NeuroDebian](http://neuro.debian.net/pkgs/python-nipype.html),\n",
      "  [@PyPI](http://pypi.python.org/pypi/nipype/), and\n",
      "  [@GitHub](http://github.com/nipy/nipype)\n",
      "  \n",
      "      - pip install nipype\n",
      "      - easy_install nipype\n",
      "      - sudo apt-get install python-nipype\n",
      "\n",
      "* Dependencies: networkx, nibabel, numpy, scipy, traits\n",
      "\n",
      "## Running Nipype ([Quickstart](http://nipy.org/nipype/quickstart.html)):\n",
      "\n",
      "* Ensure underlying tools are installed and accessible\n",
      "* Nipype **is a wrapper, not a substitute** for AFNI, ANTS, FreeSurfer, FSL, SPM,\n",
      "  NiPy, etc.,."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Step 1. Testing nipype"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "```\n",
      "$ ipython notebook\n",
      "```"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import nipype\n",
      "\n",
      "# Comment the following section to increase verbosity of output\n",
      "nipype.config.set('logging', 'workflow_level', 'CRITICAL')\n",
      "nipype.config.set('logging', 'interface_level', 'CRITICAL')\n",
      "nipype.logging.update_logging(nipype.config)\n",
      "\n",
      "nipype.test(verbose=0) # Increase verbosity parameter for more info"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "If all goes well you will see an OK:\n",
      "\n",
      "    ----------------------------------------------------------------------\n",
      "    Ran 2497 tests in 68.486s\n",
      "\n",
      "    OK (SKIP=13)\n",
      "\n",
      "The number of tests and time will vary depending on which interfaces you have installed on your system."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "nipype.get_info()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Environment and data setup\n",
      "\n",
      "Setting up your Ipython notebook environment and download some data to play with"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pylab inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Some preliminaries\n",
      "import os\n",
      "cwd = os.getcwd()\n",
      "tutorial_dir = '/software/temp/nipype-tutorial/ohbm/'\n",
      "if not os.path.exists(tutorial_dir):\n",
      "    os.mkdir(tutorial_dir)\n",
      "os.chdir(tutorial_dir)"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "-"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import urllib\n",
      "required_files = ['ds107/sub001/BOLD/task001_run001/bold.nii.gz',\n",
      "                  'ds107/sub001/BOLD/task001_run002/bold.nii.gz',\n",
      "                  'ds107/sub001/anatomy/highres001.nii.gz',\n",
      "                  'ds107/sub044/BOLD/task001_run001/bold.nii.gz',\n",
      "                  'ds107/sub044/BOLD/task001_run002/bold.nii.gz',\n",
      "                  'ds107/sub044/anatomy/highres001.nii.gz'\n",
      "                  ]\n",
      "base_url = 'http://openfmri.aws.amazon.com.s3.amazonaws.com/'\n",
      "for filepath in required_files:\n",
      "    file_location = os.path.join(tutorial_dir, filepath)\n",
      "    if not os.path.exists(file_location):\n",
      "        print('Retrieving: ' + file_location)\n",
      "        os.makedirs(os.path.dirname(file_location))\n",
      "        urllib.urlretrieve(base_url + filepath, file_location)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Step 2. Working with interfaces"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import nipype.algorithms"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.fsl import DTIFit\n",
      "from nipype.interfaces.spm import Realign"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Finding interface inputs and outputs and examples"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "DTIFit.help()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Realign.help()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Creating a directory for running interfaces"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "from shutil import copyfile\n",
      "library_dir = os.path.join(tutorial_dir, 'as_a_library')\n",
      "if not os.path.exists(library_dir):\n",
      "    os.mkdir(library_dir)\n",
      "os.chdir(library_dir)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "## Executing interfaces"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.freesurfer import MRIConvert\n",
      "convert = MRIConvert(in_file='../ds107/sub001/BOLD/task001_run001/bold.nii.gz',\n",
      "                     out_file='ds107.nii')\n",
      "print(convert.cmdline)\n",
      "results = convert.run(terminal_output='none')  # allatonce, stream (default), file"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "results.outputs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "convert = MRIConvert()\n",
      "convert.inputs.in_file='../ds107/sub001/BOLD/task001_run001/bold.nii.gz'\n",
      "convert.inputs.out_file='ds107.nii'\n",
      "convert.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "convert = MRIConvert()\n",
      "convert.run(in_file='../ds107/sub001/BOLD/task001_run001/bold.nii.gz',\n",
      "            out_file='ds107.nii')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "convert.inputs"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#### Look at only the defined inputs"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "results.inputs"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "-"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "### Experiment with other interfaces\n",
      "\n",
      "For example, run realignment with SPM"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.spm import Realign\n",
      "results1 = Realign(in_files='ds107.nii',\n",
      "                   register_to_mean=False).run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "And now use FSL"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.fsl import MCFLIRT\n",
      "results2 = MCFLIRT(in_file='ds107.nii', ref_vol=0,\n",
      "                   save_plots=True).run()"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "-"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "### Now we can look at some results"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print results1.runtime.duration, results2.runtime.duration\n",
      "subplot(211);plot(genfromtxt('ds107_mcf.nii.gz.par')[:, 3:]);title('FSL')\n",
      "subplot(212);plot(genfromtxt('rp_ds107.txt')[:,:3]);title('SPM')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### if i execute the MCFLIRT line again, well, it runs again!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Step 3. Nipype caching"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.caching import Memory\n",
      "mem = Memory('.')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Create `cacheable` objects"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "spm_realign = mem.cache(Realign)\n",
      "fsl_realign = mem.cache(MCFLIRT)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "### Execute interfaces"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "spm_results = spm_realign(in_files='ds107.nii', register_to_mean=False)\n",
      "fsl_results = fsl_realign(in_file='ds107.nii', ref_vol=0, save_plots=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "subplot(211);plot(genfromtxt(fsl_results.outputs.par_file)[:, 3:])\n",
      "subplot(212);plot(genfromtxt(spm_results.outputs.realignment_parameters)[:,:3])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "spm_results = spm_realign(in_files='ds107.nii', register_to_mean=False)\n",
      "fsl_results = fsl_realign(in_file='ds107.nii', ref_vol=0, save_plots=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# More caching"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from os.path import abspath as opap\n",
      "files = [opap('../ds107/sub001/BOLD/task001_run001/bold.nii.gz'),\n",
      "         opap('../ds107/sub001/BOLD/task001_run002/bold.nii.gz')]\n",
      "converter = mem.cache(MRIConvert)\n",
      "newfiles = []\n",
      "for idx, fname in enumerate(files):\n",
      "    newfiles.append(converter(in_file=fname,\n",
      "                              out_type='nii').outputs.out_file)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "os.chdir(tutorial_dir)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Step 4: Nodes, Mapnodes and workflows\n",
      "\n",
      "**Where:**"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.pipeline.engine import Node, MapNode, Workflow"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Node**:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "realign_spm = Node(Realign(), name='motion_correct')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Mapnode**:\n",
      "\n",
      "<img src=\"https://raw.github.com/satra/intro2nipype/master/images/mapnode.png\" width=\"30%\">"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "convert2nii = MapNode(MRIConvert(out_type='nii'),\n",
      "                      iterfield=['in_file'],\n",
      "                      name='convert2nii')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# \"Hello World\" of Nipype workflows"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "### Connect them up:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "realignflow = Workflow(name='realign_with_spm')\n",
      "realignflow.connect(convert2nii, 'out_file',\n",
      "                    realign_spm, 'in_files')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "convert2nii.inputs.in_file = files\n",
      "realign_spm.inputs.register_to_mean = False\n",
      "\n",
      "realignflow.base_dir = opap('.')\n",
      "realignflow.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Visualize the workflow"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "realignflow.write_graph()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython.core.display import Image\n",
      "Image('realign_with_spm/graph.dot.png')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "realignflow.write_graph(graph2use='orig')\n",
      "Image('realign_with_spm/graph_detailed.dot.png')"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Step 5. Getting and saving data\n",
      "\n",
      "### Instead of assigning data ourselves, let's *glob* it"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "os.chdir(tutorial_dir)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.io import DataGrabber, DataFinder\n",
      "ds = Node(DataGrabber(infields=['subject_id'], outfields=['func']),\n",
      "          name='datasource')\n",
      "ds.inputs.base_directory = opap('ds107')\n",
      "ds.inputs.template = '%s/BOLD/task001*/bold.nii.gz'\n",
      "ds.inputs.sort_filelist = True\n",
      "\n",
      "ds.inputs.subject_id = 'sub001'\n",
      "print ds.run().outputs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ds.inputs.subject_id = 'sub044'\n",
      "print ds.run().outputs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Multiple files\n",
      "\n",
      "### A little more practical usage"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ds = Node(DataGrabber(infields=['subject_id', 'task_id'],\n",
      "                              outfields=['func', 'anat']),\n",
      "          name='datasource')\n",
      "ds.inputs.base_directory = opap('ds107')\n",
      "ds.inputs.template = '*'\n",
      "ds.inputs.template_args = {'func': [['subject_id', 'task_id']],\n",
      "                           'anat': [['subject_id']]}\n",
      "ds.inputs.field_template = {'func': '%s/BOLD/task%03d*/bold.nii.gz',\n",
      "                            'anat': '%s/anatomy/highres001.nii.gz'}\n",
      "ds.inputs.sort_filelist = True\n",
      "ds.inputs.subject_id = 'sub001'\n",
      "ds.inputs.task_id = 1\n",
      "print ds.run().outputs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Connecting to computation"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "convert2nii = MapNode(MRIConvert(out_type='nii'),\n",
      "                      iterfield=['in_file'],\n",
      "                      name='convert2nii')\n",
      "\n",
      "realign_spm = Node(Realign(), name='motion_correct')\n",
      "realign_spm.inputs.register_to_mean = False\n",
      "\n",
      "connectedworkflow = Workflow(name='connectedtogether')\n",
      "connectedworkflow.base_dir = opap('working_dir')\n",
      "connectedworkflow.connect(ds, 'func', convert2nii, 'in_file')\n",
      "connectedworkflow.connect(convert2nii, 'out_file', realign_spm, 'in_files')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Data sinking\n",
      "\n",
      "###Take output computed in a workflow out of it."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.io import DataSink\n",
      "sinker = Node(DataSink(), name='sinker')\n",
      "sinker.inputs.base_directory = opap('output')\n",
      "connectedworkflow.connect(realign_spm, 'realigned_files',\n",
      "                          sinker, 'realigned')\n",
      "connectedworkflow.connect(realign_spm, 'realignment_parameters',\n",
      "                          sinker, 'realigned.@parameters')\n",
      "connectedworkflow.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### How to determine output location\n",
      "\n",
      "    'base_directory/container/parameterization/destloc/filename'\n",
      "    \n",
      "    destloc = string[[.[@]]string[[.[@]]string]] and\n",
      "    filename comes from the input to the connect statement."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "#Step 6: *iterables* - parametric execution\n",
      "\n",
      "**Workflow + iterables**: runs subgraph several times, attribute not input"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<img src=\"https://raw.github.com/satra/intro2nipype/master/images/iterables.png\" width=\"30%\">"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ds.iterables = ('subject_id', ['sub001', 'sub044'])\n",
      "connectedworkflow.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "#Putting it all together\n",
      "\n",
      "### iterables + MapNode + Node + Workflow + DataGrabber + DataSink"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "connectedworkflow.write_graph()\n",
      "Image('working_dir/connectedtogether/graph.dot.png')"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Step 7: The Function interface\n",
      "\n",
      "### The do anything you want card"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.utility import Function\n",
      "\n",
      "def myfunc(input1, input2):\n",
      "    \"\"\"Add and subtract two inputs\n",
      "    \"\"\"\n",
      "    return input1 + input2, input1 - input2\n",
      "\n",
      "calcfunc = Node(Function(input_names=['input1', 'input2'],\n",
      "                         output_names = ['sum', 'difference'],\n",
      "                         function=myfunc),\n",
      "                name='mycalc')\n",
      "calcfunc.inputs.input1 = 1\n",
      "calcfunc.inputs.input2 = 2\n",
      "res = calcfunc.run()\n",
      "print res.outputs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "#Step 8: Distributed computing\n",
      "\n",
      "### Normally calling run executes the workflow in series"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "connectedworkflow.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### but you can scale very easily\n",
      "\n",
      "For example, to use multiple cores on your local machine"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "connectedworkflow.run('MultiProc', plugin_args={'n_procs': 4})"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "# Or to other job managers\n",
      "\n",
      "```\n",
      "    >>> connectedworkflow.run('PBS', plugin_args={'qsub_args': '-q many'})\n",
      "    >>> connectedworkflow.run('SGE', plugin_args={'qsub_args': '-q many'})\n",
      "    >>> connectedworkflow.run('LSF', plugin_args={'qsub_args': '-q many'})\n",
      "    >>> connectedworkflow.run('Condor')\n",
      "    >>> connectedworkflow.run('IPython')\n",
      "```\n",
      "\n",
      "### or submit graphs as a whole\n",
      "\n",
      "```\n",
      "    >>> connectedworkflow.run('PBSGraph', plugin_args={'qsub_args': '-q many'})\n",
      "    >>> connectedworkflow.run('SGEGraph', plugin_args={'qsub_args': '-q many'})\n",
      "    >>> connectedworkflow.run('CondorDAGMan')\n",
      "```\n",
      "\n",
      "### Current Requirement: **shared filesystem**\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### You can also set node specific plugin arguments"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "```\n",
      "    >>> node.plugin_args = {'qsub_args': '-l nodes=1:ppn=3', 'overwrite': True}\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "#Step 9: Connecting to Databases"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.io import XNATSource\n",
      "from nipype.pipeline.engine import Node, Workflow\n",
      "from nipype.interfaces.fsl import BET\n",
      "\n",
      "dg = Node(XNATSource(infields=['subject_id'],\n",
      "                     outfields=['struct'],\n",
      "                     config='/Users/satra/xnat_configs/nitrc_ir_config'),\n",
      "          name='xnatsource')\n",
      "dg.inputs.query_template = ('/projects/fcon_1000/subjects/%s/experiments/xnat_E00001'\n",
      "                            '/scans/%s/resources/NIfTI/files')\n",
      "dg.inputs.query_template_args['struct'] = [['subject_id', 'anat_mprage_anonymized']]\n",
      "dg.inputs.subject_id = 'xnat_S00001'\n",
      "\n",
      "bet = Node(BET(), name='skull_stripper')\n",
      "\n",
      "wf = Workflow(name='testxnat')\n",
      "wf.base_dir = opap('xnattest')\n",
      "wf.connect(dg, 'struct', bet, 'in_file')\n",
      "wf.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "#Step 10: Configuration options\n",
      "\n",
      "[Configurable options](http://nipy.org/nipype/users/config_file.html) control workflow and node execution options\n",
      "\n",
      "At the global level:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype import config, logging\n",
      "\n",
      "config.enable_debug_mode()\n",
      "logging.update_logging(config)\n",
      "\n",
      "config.set('execution', 'stop_on_first_crash', 'true')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "At the workflow level:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "wf.config['execution']['hash_method'] = 'content'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Configurations can also be set at the node level."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "bet.config = {'execution': {'keep_unnecessary_outputs': 'true'}}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "wf.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Reusable workflows"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "config.set_default_config()\n",
      "logging.update_logging(config)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.workflows.fmri.fsl.preprocess import create_susan_smooth\n",
      "\n",
      "smooth = create_susan_smooth()\n",
      "smooth.inputs.inputnode.in_files = opap('output/realigned/_subject_id_sub044/rbold_out.nii')\n",
      "smooth.inputs.inputnode.fwhm = 5\n",
      "smooth.inputs.inputnode.mask_file = 'mask.nii'\n",
      "\n",
      "smooth.run() # Will error because mask.nii does not exist"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nipype.interfaces.fsl import BET, MeanImage, ImageMaths\n",
      "from nipype.pipeline.engine import Node\n",
      "\n",
      "\n",
      "remove_nan = Node(ImageMaths(op_string= '-nan'), name='nanremove')\n",
      "remove_nan.inputs.in_file = opap('output/realigned/_subject_id_sub044/rbold_out.nii')\n",
      "\n",
      "mi = Node(MeanImage(), name='mean')\n",
      "\n",
      "mask = Node(BET(mask=True), name='mask')\n",
      "\n",
      "wf = Workflow('reuse')\n",
      "wf.base_dir = opap('.')\n",
      "wf.connect(remove_nan, 'out_file', mi, 'in_file')\n",
      "wf.connect(mi, 'out_file', mask, 'in_file')\n",
      "wf.connect(mask, 'out_file', smooth, 'inputnode.mask_file')\n",
      "wf.connect(remove_nan, 'out_file', smooth, 'inputnode.in_files')\n",
      "\n",
      "wf.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "## Setting internal parameters of workflows"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(smooth.list_node_names())\n",
      "\n",
      "median = smooth.get_node('median')\n",
      "median.inputs.op_string = '-k %s -p 60'"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "-"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "wf.run()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "basedir = '/Users/satra/Dropbox/WORK/notebooks/'\n",
      "if os.path.exists(basedir):\n",
      "    os.chdir(basedir)"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}