Scribd
Upload
11 views26 pages

HackyHour - Python Tips & Tricks

The document provides a series of Python tips and tricks learned during a PhD, focusing on practical advice for managing Python environments, improving code efficiency, and utilizing object-oriented programming. Key recommendations include using package managers like Anaconda, leveraging Python as a wrapper for other languages, and optimizing I/O operations with appropriate modules. The session encourages collaboration and knowledge sharing among participants to enhance their coding practices.

Uploaded by

Shalin Shalin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
11 views26 pages

HackyHour - Python Tips & Tricks

The document provides a series of Python tips and tricks learned during a PhD, focusing on practical advice for managing Python environments, improving code efficiency, and utilizing object-oriented programming. Key recommendations include using package managers like Anaconda, leveraging Python as a wrapper for other languages, and optimizing I/O operations with appropriate modules. The session encourages collaboration and knowledge sharing among participants to enhance their coding practices.

Uploaded by

Shalin Shalin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 26

Python Tips & Tricks

Things I learnt during my PhD

Mike Laverick
Centre for eResearch
25/06/2020
Resources from the 25/06/20 session + chat
Presentation links
● https://uoa-eresearch.github.io/HackyHour/
● Stop using numpy.loadtxt: http://akuederle.com/stop-using-numpy-loadtxt
● Generators: https://realpython.com/introduction-to-python-generators/

Chat recommended links


● https://www.youtube.com/watch?v=8DvywoWv6fI&list=PLY9xW0dssvfYuTAVS7eNoqhLdcsu291Ll
● https://colab.research.google.com/
● https://runestone.academy/runestone/books/published/thinkcspy/index.html
● https://numpy.org/doc/stable/user/numpy-for-matlab-users.html
● http://swcarpentry.github.io/python-novice-gapminder/
● https://exercism.io/tracks/python
● https://online-learning.harvard.edu/course/using-python-research
What’s this session about?

● Providing some tips & tricks I picked up during my PhD

● Not a formal seminar or course / not a Python 101 or advanced session

● A quick ~25 minute presentation, followed by a 20-30 minute chat + Q&A

● Goal(s) : - Bring people together to discuss their work + coding difficulties


- Provide a (slack) space for people to share their coding knowledge
- To help save you some Python coding time and effort!
Tip #1

Use a package/
environment manager
Use a package/environment manager

● Have you ever had to install/upgrade/downgrade Python modules to run


a piece of code?

● Do you need to share code with other collaborators?

● Have you ever come back to some old code and can’t get it to run properly
again?

● Do you need to keep switching between Python 2 and Python 3 ???


Use a package/environment manager

Package/environment managers allow you to easily control which Python


modules and which version of that module are installed and available

Environment managers allow you to create groups of modules that you can
activate/enable for a given session.

They also allow you to import/export these “environments” for others to use
or to come back to later
Use a package/environment manager

There are a few to choose from:

Pip - Module installation


Virtualenv - Module installation, Environment manager
Anaconda - Module installation, Environment manager
Pyenv - “Environment manager” Manager

My personal preference is Anaconda, but they all do the job!


Tip #2

If you can’t do it
(quickly) in Python,
wrap Python around it
If you can’t beat them, join them!

Python is a “powerful” language in terms of it’s flexibility, support, modules

Toe-to-toe: cannot compete with Fortran/C in terms of pure computational speed

Fortunately, Python is a great “wrapper” language!


If you can’t beat them, join them!
Flexibility
Python is a “powerful” language in terms of it’s flexibility, support, modules

Toe-to-toe: cannot compete with Fortran/C in terms of pure computational speed

Speed up
Use subprocess module to run command line

1) You can execute other scripts/codes/languages/programs


2) Provide inputs + receive outputs
3) Spawn multiple subprocesses to work in the background

And much much more! Make sure to check it out!


Tip #3

I/O speedups
I/O speedups

There are many great and useful ways and modules to help load in your data:

Pandas, json, csv, to name but a few (there are many bespoke research file types)

numpy.loadtxt is not one of them!


I/O speedups

There are many great and useful ways and modules to help load in your data:

Pandas, json, csv, to name but a few (there are many bespoke research file types)

numpy.loadtxt is not one of them!

Example taken from http://akuederle.com/stop-using-numpy-loadtxt


I/O speedups: other solutions

Save data in binary formats for quick re-loading in the future:

numpy.load() on a compiled .npy array is far quicker than numpy.loadtxt()

Or even use the amazing Pickle module to save arbitrary data structures/objects
for future reloading. (Warning: does not work well with matplotlib, sorry!)
Final note on safe file reading/writing

Use “with open(“file”) as f:” instead of “open()”

You don’t need to explicitly close the file this way

Far safer if program crashes


Tip #4

Python is
object-oriented,
use objects!
Python is object-oriented, use objects

This point is very generic, and far too long to really cover properly here

But speaking from experience, a little time investment to switch to


Object-oriented thinking goes a long way in fast + easy Python
Python is object-oriented, use objects
You can put anything in a list: even functions, classes, and modules

Use dictionaries: being able to call your data using keywords rather than index
numbers is far easier for a human to remember, and can save needless iterations

Use classes (even!): You might not need to use these in your research, but they
are powerful for defining and creating sets of variables and methods, with both
common and instance-specific values/attributes

(Think of the common and unique properties that humans have... Now as a set of Python variables)
Tip #5

Efficiency, Efficiency,
Efficiency
Efficiency: a few examples w.r.t loops

Loops are unavoidably important for coding, yet often avoidably slow

Nested loops are particularly bad; time scales dramatically per extra loop

Fortunately, you can often speed up loops through sensible design


Efficiency: a few examples w.r.t loops

Think of how you structure your code:

● do I need to declare this variable in every iteration?

● Are there Built-in/Numpy/Scipy functions to do this more efficiently?

Don’t forget:

● for/while loops: you can use pass, continue, and break

My nested loop from 2 weeks to 30s thanks to these functions!


Efficiency: a few examples w.r.t loops

Do you need to loop over Gigabytes of input data?

Use a generator!

Normal loops need to load the entire list/array/object in memory before looping

Generators load on the fly, reducing RAM and speeding up iteration over large data

Check out this link: https://realpython.com/introduction-to-python-generators/


That’s all folks!

Now to hear from you...


Bonus Tip

How to write your code


neatly
Get a nice text editor + python linter + PEP8 style

Rule of thumb: if PEP8 makes it less readable - ignore PEP8

You might also like