Astronomical Data Analysis with Python

Lecture 1
Yogesh Wadadekar
NCRA-TIFR

July-August 2010

Yogesh Wadadekar (NCRA-TIFR)

Topical course

1 / 33

What this course is about

prerequisites

Yogesh Wadadekar (NCRA-TIFR)

Topical course

2 / 33

What this course is about

prerequisites
URL: http://www.ncra.tifr.res.in/yogesh/python_course_2010

Yogesh Wadadekar (NCRA-TIFR)

Topical course

2 / 33

What this course is about

prerequisites
URL: http://www.ncra.tifr.res.in/yogesh/python_course_2010
please interrupt and ask questions at any time.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

2 / 33

What this course is about

prerequisites
URL: http://www.ncra.tifr.res.in/yogesh/python_course_2010
please interrupt and ask questions at any time.
programming background of the audience?

Yogesh Wadadekar (NCRA-TIFR)

Topical course

2 / 33

Why Python?
A powerful, general purpose programming language, yet easy to
learn. Strong, but optional, Object Oriented Programming support
Very large user and developer community, very extensive and
broad library base
Very extensible with C, C++, or Fortran, portable distribution
mechanisms available
Free; non-restrictive license; open source
fast becoming the standard scripting language for astronomy
very powerful array processing capabilities (numpy)
extensive documentation - Many books and on-line documentation
resources available (for the language and its libraries and
modules)

Yogesh Wadadekar (NCRA-TIFR)

Topical course

3 / 33

Why python?

Plotting (matplotlib)
framework (matplotlib) more extensible and general
Better font support and portability (only one way to do it too!)
Usable within many windowing framework (GTK, Tk, WX, Qt...)
backends
Standard plotting functionality independent of framework used
plots are embeddable within other GUIs
more powerful image handling (multiple simultaneous LUTS,
optional resampling/rescaling, alpha blending, etc)

Support for many widget systems for GUI development

superb database interfaces to all popular databases.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

4 / 33

Disadvantages of Python

More items to install separately (eased by yum and apt-get)

Not as well accepted in astronomical community (but support
clearly growing)
Scientic and numerical libraries not as mature; not as deep in
astronomical libraries and utilities
Array indexing convention backwards
Small array performance slower
No standard GUI run/debug tool e.g. like Eclipse for Java
Support for many widget systems (angst regarding which to
choose)

Yogesh Wadadekar (NCRA-TIFR)

Topical course

5 / 33

Comparison with FORTRAN/C/C++

huge amount of legacy code

compilers highly optimized for excellent runtime performance
but...
FORTRAN not really general purpose
relatively primitive datatypes
manual memory management
slow edit/compile/test cycle

Yogesh Wadadekar (NCRA-TIFR)

Topical course

6 / 33

Comparison with IDL/Matlab

Extremely popular
Interactive, great visualization, good libraries
but...
Not really general purpose
Vendor lock-in
fairly expensive, source code of core libraries not changeable.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

7 / 33

Comparison with Perl

quite popular locally - lots of code written

shares many of Pythons strengths
but...
Write Once, Read Never
just think and type, thats perl
I think that Just think and type, thats python is more appropriate.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

8 / 33

Python usage in optical astronomy

STScI PyRAF (IRAF) + additional Python only routines

ESO PyMIDAS (MIDAS)
STScI PyFITS (access to FITS files)
Astro-WISE (widefield imaging system)
Pyephem - solar system ephemeris
LSST will use Python/C++

Yogesh Wadadekar (NCRA-TIFR)

Topical course

9 / 33

Python usage in Radio astronomy

CasaPy (Casa) - AIPS++, default system for EVLA and ALMA

data analysis.
ParselTongue - call AIPS tasks from Python
PYGILDAS (GILDAS) - IRAM data analysis software ported to
Python
BoA (Bolometer Analysis Package) for LABOCA on APEX and
other bolometers
APECS (APEX control software)
KAT-7 CMS is in Python
Presto - pulsar search and analysis suite; most recent routines in
Python

Yogesh Wadadekar (NCRA-TIFR)

Topical course

10 / 33

Basic methodology

prepare a full working prototype of any application in Python,

without worrrying about speed

Yogesh Wadadekar (NCRA-TIFR)

Topical course

11 / 33

Basic methodology

prepare a full working prototype of any application in Python,

without worrrying about speed
profile the prototype

Yogesh Wadadekar (NCRA-TIFR)

Topical course

11 / 33

Basic methodology

prepare a full working prototype of any application in Python,

without worrrying about speed
profile the prototype
identify the slow parts of the Python code and rewrite them in a
compiled language like C/C++ for maximum speed.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

11 / 33

Basic methodology

prepare a full working prototype of any application in Python,

without worrrying about speed
profile the prototype
identify the slow parts of the Python code and rewrite them in a
compiled language like C/C++ for maximum speed.
write wrappers to interface the new compiled code with the Python
prototype. Repeat until required optimisation is achieved.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

11 / 33

Basic methodology

prepare a full working prototype of any application in Python,

without worrrying about speed
profile the prototype
identify the slow parts of the Python code and rewrite them in a
compiled language like C/C++ for maximum speed.
write wrappers to interface the new compiled code with the Python
prototype. Repeat until required optimisation is achieved.
If you are talking to devices - write a simulator first.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

11 / 33

Python for non-astronomers

full featured, high level programming language
very easy to learn -National Mission on Education through ICT is
sponsoring a large program (crores of rupees) to develop
computer education materials in Python for school and college
students (http://fossee.in/). Also, OLPCs Sugar environment is
written in Python.
powerful text processing capabilities - many sysadmins are
adopting it.
powerful interfaces to almost any database
web-friendly language - many frameworks available Django, Zope,
CherryPy, Trac for website CMS, wikis etc.
CERNs INDICO conference management system is all Python.
good plotting capabilities (see the latest casapy capabilities based
on matplotlib)
Yogesh Wadadekar (NCRA-TIFR)

Topical course

12 / 33

Python extensively used by Google

The Google build system is written in python. All of Googles
corporate code is checked into a repository and the dependency
and building of this code is managed by python.
Packaging. Google has an internal packaging format like RPM.
These packages are created using python.
Binary Data Pusher. This is the area where Alex Martelli is
working, on optimizing pushing bits between thousands of servers
Production servers. All monitoring, restarting and data collection
functionality is done with python
Reporting. Logs are analyzed and reports are generated using
Python.
A few services including code.google.com and google groups run
on Python. Most other front ends are in C++ (google.com) and
Java (gmail). All web services are built on top of a optimized http
server wrapped with SWIG.
Yogesh Wadadekar (NCRA-TIFR)

Topical course

13 / 33

If you are an absolute beginner to programming

Yogesh Wadadekar (NCRA-TIFR)

Topical course

14 / 33

Standard book for beginners - Lutz & Ascher

Yogesh Wadadekar (NCRA-TIFR)

Topical course

15 / 33

A new primer by Langtangen

Yogesh Wadadekar (NCRA-TIFR)

Topical course

16 / 33

Book for intermediate level

Yogesh Wadadekar (NCRA-TIFR)

Topical course

17 / 33

Python Cookbook

Yogesh Wadadekar (NCRA-TIFR)

Topical course

18 / 33

A Python quick reference by Martelli

Yogesh Wadadekar (NCRA-TIFR)

Topical course

19 / 33

Another reference book by Beazley

Yogesh Wadadekar (NCRA-TIFR)

Topical course

20 / 33

Python for Scientific Computing by Langtangen

Yogesh Wadadekar (NCRA-TIFR)

Topical course

21 / 33

Guide to Numpy by Travis Oliphant, now in public

domain

http://www.tramy.us/guidetoscipy.html

Yogesh Wadadekar (NCRA-TIFR)

Topical course

22 / 33

Plenty of other books are available

Search for Python programming on amazon.com throws up 429

items.
Having said this, I have not bought a Python book yet. Almost all the
books referred to above are now in the NCRA and GMRT libraries.
Many of them should also be available in the IUCAA library. Also, a lot
of documentation is online.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

23 / 33

Online Material

www.python.org

Yogesh Wadadekar (NCRA-TIFR)

Topical course

24 / 33

Online Material

www.python.org
Start with the Python tutorial - http://docs.python.org/tutorial/ we
will cover all of it and more in this course.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

24 / 33

Online Material

www.python.org
Start with the Python tutorial - http://docs.python.org/tutorial/ we
will cover all of it and more in this course.
Python for astronomers http://www.scipy.org/wikis/topical_software/Tutorial Excellent
tutorial by Greenfield and Jedrzejewski

Yogesh Wadadekar (NCRA-TIFR)

Topical course

24 / 33

Online Material

www.python.org
Start with the Python tutorial - http://docs.python.org/tutorial/ we
will cover all of it and more in this course.
Python for astronomers http://www.scipy.org/wikis/topical_software/Tutorial Excellent
tutorial by Greenfield and Jedrzejewski
SciPy conferences - http://conference.scipy.org - lots of interesting
talks (some with video versions)

Yogesh Wadadekar (NCRA-TIFR)

Topical course

24 / 33

Online Material

www.python.org
Start with the Python tutorial - http://docs.python.org/tutorial/ we
will cover all of it and more in this course.
Python for astronomers http://www.scipy.org/wikis/topical_software/Tutorial Excellent
tutorial by Greenfield and Jedrzejewski
SciPy conferences - http://conference.scipy.org - lots of interesting
talks (some with video versions)
Astropy mailing list - http://mail.scipy.org/mailman/listinfo/astropy

Yogesh Wadadekar (NCRA-TIFR)

Topical course

24 / 33

Python on MS Windows

Full distributions provided by Enthought and ActiveState. Enthought

distribution is better for scientific/technical computing.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

25 / 33

Python on Linux

Usually, Python is already installed. Type python in a terminal to

check it out.
On Redhat like distributions, the installer anaconda is written in
Python.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

26 / 33

GUIs for python development

IDLE is one that is distributed with Python.

emacs is a very good IDE if you willing to learn how to use it or
know it already.
Numerous other free and commercial IDEs are available. A
comprehensive list is available at
http://wiki.python.org/moin/PythonEditors

Yogesh Wadadekar (NCRA-TIFR)

Topical course

27 / 33

Python Version 2 or 3?

Python 3 is newer but Python 2 has more existing third party software.
For this reason, in this course, we will use Python 2 only. However,
within a year or so it should be possible to move to Python 3.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

28 / 33

Hello World program

$ python -c print "Hello World"

Hello World

Yogesh Wadadekar (NCRA-TIFR)

Topical course

29 / 33

Starting Python

simply type python at the command prompt

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for
more information.
>>>

Yogesh Wadadekar (NCRA-TIFR)

Topical course

30 / 33

Ipython - an enhanced python shell

simply type ipython at the command prompt. If it is not installed yum
install ipython
IPython 0.9.1 - An enhanced Interactive Python.
? -> Introduction and overview of IPythons
features.
%quickref -> Quick reference.
help -> Pythons own help system.
object? -> Details about object. ?object also
works, ?? prints more.
In [1]:
Ipython is the shell for casapy. It will also eventually become the
default shell for Pyraf.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

31 / 33

Numerical types: Integers

>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>

2
0
-4711
07, 022 # Octal tuple
0x9, 0xa, 0XF # Hexadecimal tuple
17 + 4 # Expression
0xa - 2
23 (2+3) # Power
7 / 2, 7 / -2 # Int division
from __future__ import division
7/2

Yogesh Wadadekar (NCRA-TIFR)

Topical course

32 / 33

Floats

>>>
>>>
>>>
>>>
>>>
>>>
>>>

2.3
-4.
0.1, .1
2.99E10, 6.62607e-27, -1e10
1.7 + .4
17. + 4
7./2., 7./2, 7/2.

Yogesh Wadadekar (NCRA-TIFR)

Topical course

33 / 33