Scribd
Upload
15 views

Python Lecture - 14

This document discusses using Python to generate and run Unix commands. It describes running processes in the foreground and background from the command line and Python. It provides examples of using Python to generate repetitive Unix commands that operate on multiple files, as well as generating shell scripts from Python with fixed and variable components. The document emphasizes that Python can be used to generate commands and scripts to run jobs sequentially or in parallel on a cluster in an automated fashion.

Uploaded by

let.me.g0.h0m3
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
15 views

Python Lecture - 14

This document discusses using Python to generate and run Unix commands. It describes running processes in the foreground and background from the command line and Python. It provides examples of using Python to generate repetitive Unix commands that operate on multiple files, as well as generating shell scripts from Python with fixed and variable components. The document emphasizes that Python can be used to generate commands and scripts to run jobs sequentially or in parallel on a cluster in an automated fashion.

Uploaded by

let.me.g0.h0m3
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Linking Python and

Unix
HORT 59000
Lecture 14
Instructor: Kranthi Varala
Controlling processes from command line

• Foreground: Default mode for running commands. The


shell waits on the process to finish.
• Process retains control of the command line.
• Key input is directed to the active process.
• Background: Process is initiated and pushed to the
background.
• Control of command-line is returned to the user.
• Key input and other interactions are no longer passed to the
process.
• Processes can be pushed to background at initiation using &
Controlling processes from Python

• Wait: Default mode for running commands. The python


interpreter waits on the process to finish.
• Process retains control of the shell.
• Both os.system() and subprocess.call() do this by default
• Submit and forget: Command is initiated in a separate
process.
• Processes can be pushed to background at initiation using &
• subprocess.Popen(), by default, does not wait for child
process to finish
Creating command strings vs. scripts

• Command strings: Use python to generate the command


string with a combination of fixed strings and variables.
• Submit command using os.system() or subprocess.call()
• Submit command using subprocess.Popen() if you need to
capture output.
• Scripts: Use file handle object to create a new script file with
commands and parameters embedded.
• subprocess.Popen() to submit the script as a job
• Remember to make shell scripts executable (chmod 755)
• PBS scripts need not be executable
• When submitting multiple jobs, creating scripts helps keep
track of the exact command and parameters used.
Generating UNIX commands from Python

• Python can be used to generate repetitive UNIX


commands that operate over multiples of a set
• For example, find a given sequence in all fastq files
• Key: A UNIX command is a string with fixed words,
such as command name, and variable words, such as
name of input file(s)

Fixed Variable
Generating shell scripts from Python
• Python can be used to generate shell scripts that differ
in few parameters
• For example, PBS scripts with different job parameters
• Key: A PBS script is made of fixed lines, such as PBS
parameters, module loads etc. and variable lines, such
as the lines specifying the input line.

Fixed

Variable
Generating UNIX commands from Python
Option 1 Option 2 Option 3
SRR039920 Run each download Generate commands Use Python to
command in Python. generate PBS script
SRR039921
individually. to run commands.
SRR039922
10 jobs have to be 1 script can run jobs 1 PBS job runs in
SRR039923
created manually. sequentially. background.
SRR039924
No record of Script keeps track of Script keeps track of
SRR039925 parameters give to parameters given to parameters given to
command. command. command.
SRR039926

SRR039927 Run time for Run time for Run time for
commads is commands is commands can be
SRR039928 cumulative and cumulative and cumulative and
needs user DOESN’T need user DOESN’T need user
SRR039929
attention. attention. attention.
Making system calls from python

• Remember to load a newer version of python:


• module load intel/14.0.2.144
• module load python/2.7.6
• Now, we can use python to make calls to the
system i.e., calls commands and scripts
available on the system command line.
• The ‘os’ and ‘subprocess’ module are the two
main ways to interact with the system
command line.
Generating UNIX commands from Python

• Example: Write a python script that calls fastq-dump on


a series of SRA IDs
• List of SRA IDs is in the file:
/scratch/scholar/k/kvarala/Week14/SRR_ids.txt
Generating shell scripts from Python

• Example 1: Write a python script that creates one PBS


job to fastq-dump a series of SRA IDs

• Example 2: Write a python script that creates one PBS


job for each SRA ID to fastq-dump the data
Final project specifications
• The previously provided final project goals will be used to
grade the quality of the final project submission.
• Tomorrow (April 25th) is final date to adjust your project
goals.
• The code for this project MUST be written in the Python
language and has to run on the Purdue Scholar cluster.
• The project submission HAS to include the input data set,
the student’s code and all instructions required to execute
the code.
• Project submission deadline is 12 PM on the May 5th, 2018.
• Final project is worth 40% of your grade.

You might also like