Session 02 python basics

Editor's Notes

• #2: Today we’re looking at a programming language that’s used a lot by data scientists (and that many of the code examples will be written in). Some of the example code in the rest of these sessions will be in Python, so it will help for you to be able to read and understand what’s happening in that code.
• #3: So let’s begin. Here are the 6 things we’ll talk about today.
• #4: Before we talk about python, it helps to define it a bit.
• #5: See http://learnpythonthehardway.org/book/ and https://www.python.org/ for way more about Python.
• #6: We’ll cover the basics of Python in this session. The aim is to a) get you some useful tools, and b) not have you scared by something that looks like this. The toolset for this course (Python, R, offline tools) was chosen so you can still do something useful when you have no internet / have low bandwidth. There are many other tools out there if you have the bandwidth to run then, but at some point in your career you’ll find yourself somewhere like Arusha in Tanzania, desperately hoping that the page you need will load. We’ve based this toolkit on trying to minimise that experience. Plus it’s cool being able to do things for yourself!
• #7: That was your first python code. You opened up the ipython interpreter, then typed in ‘commands’ (‘1+2’) and got back outputs (‘3’). Then, because we’re tidy, you closed the python interpreter again. We met several important things here. You met objects: “1” and “2.5” are numbers (“1” is an integer number; “2.5” is a floating point number); and “‘hello world!’” is a string. You also met an operator: “+” and a function: print(). * Operators do basic things to pairs of objects (or sometimes to single objects). For instance, “+” adds together the objects to the left and right of it. You’ll meet arthmetic operators (+ add, - subtract, * multiply, / divide) and logical operators (of which more soon). * Functions are pieces of pre-written code that (usually) do something that needs to be done often, like sending a friendly message.
• #8: So you have some Python code - maybe written in these slides, maybe in a .py or .ipynb file. Let’s look at some of the ways you can run that code.
• #9: ‘python’ runs the python interpreter. ‘ipython’ runs a version of the python interpeter that’s slightly friendlier to new users, e.g. it has better help. ‘python helloworld.py’ runs the python interpreter on a file called ‘helloworld.py’. If that file contains a set of python commands (e.g. “print(“Hello World!”)”), you’ll see the output of those commands. You can do the same thing with ipython, e.g. ‘ipython helloworld.py’. There’s a file called helloworld.py in your course notes. Try running it, then editing it (any text editor will do, although some, like Sublimetext, are set up to point out any coding errors to you), and running it again. There are other ways to run Python code (e.g. from another program), but we won’t be doing that in this course.
• #10: Ipython notebooks are files that contain both formatted text and code; depending on how you set up your notebooks, they can run code written in Python, R or several other languages.
• #11: All the code examples (both Python and R) for this course are in iPython notebooks, so you’ll need to know how to open one. cd (“change directory”) is the terminal window command to change directory. “cd ~” takes you to your ‘top’ directory, where typing “ls” (mac) or “dir” (windows) will show you directories like Desktop, Downloads or Documents. “cd dir1/dir2/dir3” takes you ‘down’ directories, to dir3 which is below dir2 which is below dir1. This is the terminal equivalent of starting in one directory, then clicking on dir1, dir2 then dir3. “cd ..” takes you ‘up’ directories: for instance, “cd ..” in dir3 will take you to dir2. “pwd” (mac) or “dir” (windows) will show you which directory you are in at the moment. If you get lost, “cd ~”.
• #12: iPython starts with the Dashboard view. This lists all the files in the directory that you called iPython from, including directories. If you accidentally delete this view, open a web browser and type “localhost:8888” in the address window to get back to it. Things you need to know about in here: The “New” button on the top right hand side. Click on this, then “Python 3”, to create a new iPython notebook file. The file listing. Click on any file or directory to open it. The current directory (the grey bar above the files list): click on this to go back up from a directory.
• #13: And this is what you see when you open an iPython file. The grey boxes are called ‘cells’. You create a new cell by clicking ‘Insert’ on the top menu. You change the cell type (to ‘Markdown’, or ‘Code’) by clicking the box that currently says ‘Code’ You run a cell by clicking in the cell, then clicking ‘cell’ -> ‘run cells’. You change your file’s name by clicking the name to the right of jupyter You save your shiny new file by clicking ‘File’ -> ‘Save and Checkpoint’.
• #15: Don’t panic if your notebook looks like this when you open it. iPython has helpfully converted your markup cells into the way you’d see them if you printed the notebook. To edit any of these cells, click in the cell (you should then see the markdown code, ready to edit).
• #16: Variables are one of the basic building blocks of Python code. We’ll look at some of these - what they look like, what they can do.
• #17: But before we start, let’s talk about bugs. Bugs are errors that either stop your code from running, or make it run in ways that you weren’t expecting. You will meet many bugs when you’re coding. Don’t let them scare you: dealing with bugs is just one of the features of coding. You can do things to make this easier: for instance, if you use an editor like sublimetext to write your code, it has handy features like changing the colour of your code when you write something incorrectly. do this…
• #18: Coders leave messages for each other (and themselves) in their code: these are called comments. They’re there for humans to read, and are completely ignored by the python interpreter. The comments you’ll see in this course are mostly the ones that start with the ‘#’ symbol. Everything from the ‘#’ to the end of the line it’s on is a comment. You *might* see magic comments in python files, e.g. #!/usr/bin/python # -*- coding: utf-8 -*- These tell the interpreter which type of file this is (python), and which character set it uses (remember the problem with ‘“‘ earlier?). utf-8 is a very commonly-used character set.
• #19: Variables are places where you put objects that you don’t want to keep typing out. You do this because you don’t have to keep typing “Hello World” when you use it several times in your code it’s easier to read your code and see what’s going on it’s harder to make mistakes (e.g. if you put “10” everywhere in your code, and change your mind, then you might change some but not all of those “10”s to “11s”… if you put x=10 and then use x everywhere, you only have to change one thing). Variables have “types” that tell the interpreter what can be done with them (e.g. you can multiply numbers together, but you can’t do that with strings). Here are some common variable types: String: a set of characters, which could be letters, numbers or symbols. Strings can be as long as you like - in some datasets (e.g. reading in book texts) you’ll see strings thousands of characters long. Boolean: can be either True or False. Useful for when you want to do one thing if something is true (e.g. x == 10), and something else if it isn’t. Numbers: a number. This include integers (e.g. 10), and floating-point number (e.g. 15.523). Note how I reused the same variable name for two numbers. Try typing my_number = 10, then print(my_number), then my_number=15.523, then print(my_number) in ipython.
• #20: Strings are a bit special. A string is made out of characters (e.g. “H” or “r”), so it makes sense to ask things like: len(string): how many characters are there in this string? my_string[4]: what’s the 5th character in this string? (NB python counts from 0, not 1, e.g. my_string[0] is H).
• #21: Remember, functions are pieces of code that (often) do something that gets repeated lots. We already met the function “print()”. Here are some more functions. Functions generally have 3 parts: the function name (e.g. “len”), the function arguments (e.g. “my_string”) that tell the function what to do its thing on, and the the function return (e.g. stringlength), which is the variable (or variables) that the function puts its results into. You can use ‘string formatting’ to put other variables (numbers, other strings etc) into a string. Here, we’ve put a number (15.2) and a string (“Hello World!”) into a new string, and printed it. print(“{}”.format(x)) is very very useful for printing out more-complicated variables, where a simple print(x) will fail.
• #22: You can write your own functions using the keyword “def”. Here, we’ve created a function (my_function) that takes an argument called my_text, adds some more text to it, puts that new text into variable new_text, and returns new_text to the person who called the function. Then we use the function. We call my_function with the argument “sara likes “: my_text is now set to “sara likes “ before we add “bananas” to it. The function returns “sara likes bananas”, which is put into the variable new_sentence.
• #23: A collection is a group of python objects. We’re going to look at two collections here: lists, and dictionaries.
• #24: A list is an ordered set of python objects. Here, rowvals is a list that contains numbers. Lists are ordered, e.g. rowvals[3] will always give you the 4th object in rowvals (currently “6”). This might seem wrong to you - you may be asking why it doesn’t give you the 3rd object in the list. The reason is that all counting in Python starts at zero, so the “indices” for the objects in this list are 0,1,2,3,4,5,6,7 and 8. You can’t use an index larger than the last index in the list: typing rowvals[9] will give you an error message. But you will sometimes see negative indices. rowvals[3] and max(rowvals) will both return an object to you, that you can ‘assign’ to a new variable.
• #25: Some functions will change the object they’re applied to. list.sort() is one of these. rowvals.sort() changes the contents of rowvals itself. Try running this code to see how this happens.
• #26: So sometimes you want to do something to every item in a list. Repeating the same command for alist[0], alist[1], alist[2] and alist[3] would be tiresome (and even more tiresome if you have 1000 items in your list), so we use iterators. An iterator (e.g. “for”) selects each item in a list in turn, and applies your code to it. For instance, in the code above, the line “for item in alist” selects each object in alist, calls that object “item” then runs the code “print(item)” on it. Note the 4 spaces before print(item). This is how Python groups code together: the spaces tell the Python interpreter that print(item) and print(item+2) are run on the items from alist, but “print(I’m done now)” isn’t. This can be very annoying at first, but makes sense after a while. The enumerate function can be very useful when you’re iterating over lists. This allows you to use both the index of an item, and the item in your loops. An example of this is: for index, item in enumerate(alist): print(“the list item at index {} is {}”.format(index, item))
• #27: A python list is a collection that’s indexed by numbers (0,1,2,etc). A python dictionary is a container that’s indexed by anything you want to use. In this example, we’ve used ISO3166 country codes as indices, and the names of countries as the items in the list. Dictionaries are unordered, so iso3166[0] doesn’t make sense. But because we’ve created the indices SLE, NGA, LBR, we can use any of these to access the item connected with that index, e.g. iso3166[‘NGA’]. If we want to see all the indices (aka “keys”) in a dictionary, we use the “keys” function, e.g. iso3166.keys(). If we want to know if a key is in a dictionary, we can ask, for example using “‘NGA’ in iso3166”
• #28: We can also iterate over dictionary keys, as seen here. Don’t forget those 4 spaces! Speaking of spaces… you’ll notice that I’ve added spaces here and there to make the code more readable. You can do this in most places, but there are code format rules (“style guides”) that Python coders follow so they don’t confuse each other - most Python coders use PEP8 https://www.python.org/dev/peps/pep-0008/ - which, amongst other things, tells coders to use 4 (not 1,2,3,5 or anything else) spaces as indents. Some text editors, like Sublime Text (https://www.sublimetext.com), have ways to switch on these rules so the text editor colour-highlights any time you’re breaking a rule.
• #29: At some point, we’re going to need to pull data into our code: this might be input from the code’s user, or from a datafile, or from an API (application programming interface: more on that soon). Here’s how.
• #30: You might want to ask the user for input (e.g. the name of the csv file they want your code to process). The function “input()” does this.
• #31: First, a brief note about libraries. Places to look for libraries include: * https://docs.python.org/3/library/ (you already have these) * https://pypi.python.org/pypi
• #32: Here, we’re using the Python library “CSV” to read in the contents of a CSV file. First, we need to tell our code to use the CSV library (“import CSV”) Then we tell it which file to open (“open()”), and that we’re reading from (not writing to) a file (“‘r’”). We’ve opened the file for input, but that’s not enough if we want to treat this file as a CSV file (instead of a text file, or Excel, json, xml etc). To do this, we create a CSV reader object, using “csv.reader()”. This object (that we’ve called “csvin”) knows about commands like “next()” (get me the next row of data from the csv, and put it into a list object), and that “for row in csvin” will loop over each remaining row in the CSV file, and input it as a list. The rest is grabbing each row in the CSV file, and printing out that row. When you’re done, fin.close() closes the link between this program and the file pointed at by the variable ‘fin’. Later in the course, you'll see other, shorter, ways to read in a CSV file (dictreader, and pandas.read_csv), but this is development data, and the CSV library can often read in difficult CSV files that other methods fail on.
• #33: Sometimes we’ll want to process only some of the data that we see. In this case, we’re only interested in data about Liberia, so we test to see if the second entry in the row is ‘Liberia’. This test (if a == b) is called a conditional statement, because running the rows nested below the “if” statement (“nested” = using another 4 spaces, to show that the “print()” statement belongs to the “if” one) is conditional on whether the statement “row[1] == ‘Liberia’” is true. The conditions you’ll see in code include: a == b: a’s value is the same as b’s value a != b: a’s value is not the same as b’s value a < b: a is smaller than b a > b: a is bigger than b a<= b: a is smaller than or equal to b a >= b: a is bigger than or equal to b Beware the “==” in a conditional statement. “==” is used to test similarity; “=” is used to set the value of the thing on its left to the value of the thing on its right. Python will let you use either of these in a condition.
• #34: We’ve just skimmed through the basics of Python - enough so you don’t get lost when I start showing example code. If you want to know more, there are lots of courses listed in the course reading list; IMHO the best of these is Jeff Knupp’s Learn Python the Hard Way (available online for free).
• #35: Your exercises were all built into the class. But if you want more…
• #36: The iPython notebooks (that you downloaded from github) are there for you to learn more about the subjects in each class. Go play with them: take copies, edit the code and see what happens.
• #37: And make a list of those strange behaviours so we can talk about them in the next class.