{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Using functional programming in Python like a boss: Generators, Iterators and Decorators\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Features of functions\n", "1. First-class functions are objects and thus:\n", " * Can be assigned to variables\n", " * Can be stored in data structures\n", " * Can be used as parameters\n", " * Can be used as a return value\n", "2. Higher order functions:\n", " * Accept a function as an argument and/or return a function as a value.\n", " * Create composite functions from simpler functions.\n", " * Modify the behavior of existing functions.\n", "3. Pure functions:\n", " * Do not depend on hidden state, or equivalently depend only on their input.\n", " * Evaluation of the function does not cause side effects\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# A pure function\n", "def my_min(x, y):\n", " if x < y:\n", " return x\n", " else:\n", " return y\n", "\n", "\n", "# An impure function\n", "# 1) Depends on global variable, 2) Changes its input\n", "exponent = 2\n", "\n", "def my_powers(L):\n", " for i in range(len(L)):\n", " L[i] = L[i]**exponent\n", " return L" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "source": [ "# What can act as a function in Python?\n", "1. A function object, created with the `def` statement.\n", "2. A `lambda` anonymous function, restricted to an expression in single line.\n", "3. An instance of a class implementing the `__call__` function." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "def min_def(x, y):\n", " return x if x < y else y\n", "\n", "min_lambda = lambda x, y: x if x < y else y\n", "\n", "class MinClass:\n", " def __call__(self, x, y):\n", " return x if x < y else y\n", "\n", "min_class = MinClass()\n", "\n", "print(min_def(2,3) == min_lambda(2, 3) == min_class(2,3))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Common gotchas 1: Mutable Default Arguments\n", "\n", "* Python's default arguments are evaluated once when the function is **defined** and **not each time the function is called** (like say, in Ruby).\n", "* If you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.\n", "* Sometimes you can specifically \"exploit\" (read: use as intended) this behavior to maintain state between calls of a function. This is often done when writing a caching function." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "my_list: [12]\n", "my_other_list: [12, 42]\n", "my_list2: [12]\n", "my_other_list2: [42]\n" ] } ], "source": [ "def append_to(element, to=[]):\n", " to.append(element)\n", " return to\n", "\n", "my_list = append_to(12)\n", "print(\"my_list:\", my_list)\n", "my_other_list = append_to(42)\n", "print(\"my_other_list:\", my_other_list)\n", "\n", "def append_to2(element, to=None):\n", " if to is None:\n", " to = []\n", " to.append(element)\n", " return to\n", "\n", "my_list2 = append_to2(12)\n", "print(\"my_list2:\", my_list2)\n", "my_other_list2 = append_to2(42)\n", "print(\"my_other_list2:\", my_other_list2)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Common gotchas 2: Late Binding Closures\n", "\n", "1. A closure occurs when a function has access to a local variable from an enclosing scope that has finished its execution.\n", "2. Python's closures are *late binding*.\n", "3. Values of variables used in closures are looked up at the time the inner function is **called** and **not when it is defined**." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8\n", "8\n", "8\n", "8\n", "8\n" ] } ], "source": [ "def create_multipliers():\n", " multipliers = []\n", "\n", " for i in range(5):\n", " def multiplier(x):\n", " return i * x\n", " multipliers.append(multiplier)\n", "\n", " return multipliers\n", "\n", "for multiplier in create_multipliers():\n", " print(multiplier(2))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Higher order functions and decorators\n", "\n", "* Python functions are objects.\n", "* Can be defined in functions.\n", "* Can be assigned to variables.\n", "* Can be used as function parameters or returned from functions.\n", "* Decorators are syntactic sugar for higher order functions." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello world\n", "hello world\n" ] } ], "source": [ "# Higher order functions\n", "\n", "def makebold(fn):\n", " def wrapped():\n", " return \"\" + fn() + \"\"\n", " return wrapped\n", "\n", "def hello():\n", " return \"hello world\"\n", "\n", "print(hello())\n", "hello = makebold(hello)\n", "print(hello())" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello. args: ('world', 'pythess'), kwargs: {'where': 'soho'}\n" ] } ], "source": [ "# Decorated function with *args and **kewargs\n", "\n", "def makebold(fn):\n", " def wrapped(*args, **kwargs):\n", " return \"\" + fn(*args, **kwargs) + \"\"\n", " return wrapped\n", "\n", "@makebold # hello = makebold(hello)\n", "def hello(*args, **kwargs):\n", " return \"Hello. args: {}, kwargs: {}\".format(args, kwargs)\n", "\n", "print(hello('world', 'pythess', where='soho'))" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello. args: ('world', 'pythess'), kwargs: {'where': 'soho'}\n" ] } ], "source": [ "# Decorators can be combined\n", "\n", "def makeitalic(fn):\n", " def wrapped(*args, **kwargs):\n", " return \"\" + fn(*args, **kwargs) + \"\"\n", " return wrapped\n", "\n", "def makebold(fn):\n", " def wrapped(*args, **kwargs):\n", " return \"\" + fn(*args, **kwargs) + \"\"\n", " return wrapped\n", "\n", "@makeitalic\n", "@makebold # hello = makeitalic(makebold(hello))\n", "def hello(*args, **kwargs):\n", " return \"Hello. args: {}, kwargs: {}\".format(args, kwargs)\n", "\n", "print(hello('world', 'pythess', where='soho'))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello. args: ('world', 'pythess'), kwargs: {'where': 'soho'}\n" ] } ], "source": [ "# Decorators can be instances of callable classes\n", "\n", "class BoldMaker:\n", " def __init__(self, fn):\n", " self.fn = fn\n", " def __call__(self, *args, **kwargs):\n", " return \"\" + self.fn(*args, **kwargs) + \"\"\n", "\n", "@BoldMaker # hello = BoldMaker(hello)\n", "def hello(*args, **kwargs):\n", " return \"Hello. args: {}, kwargs: {}\".format(args, kwargs)\n", "\n", "# hello.__call__(*args, **kwargs)\n", "print(hello('world', 'pythess', where='soho'))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "

hello world

\n", "

hello world

\n", "

hello world

\n" ] } ], "source": [ "# Decorators can take arguments\n", "\n", "def enclose_in_tags(opening_tag, closing_tag): # returns a decorator\n", " def make_with_tags(fn): # returns a decorated function\n", " def wrapped(): # the function to be decorated (modified)\n", " return opening_tag + fn() + closing_tag\n", " return wrapped\n", " return make_with_tags\n", "\n", "# decorator function make_with_tags with the arguments in closure\n", "heading_decorator = enclose_in_tags('

', '

')\n", "paragraph_decorator = enclose_in_tags('

', '

')\n", "\n", "def hello():\n", " return \"hello world\"\n", "\n", "h1_hello = heading_decorator(hello)\n", "p_hello = paragraph_decorator(hello)\n", "h1_p_hello = heading_decorator(paragraph_decorator(hello))\n", "print(h1_hello())\n", "print(p_hello())\n", "print(h1_p_hello())" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "

hello world

\n", "

hello world

\n", "

hello world

\n" ] } ], "source": [ "# Decorators with arguments combined\n", "\n", "def enclose_in_tags(opening_tag, closing_tag):\n", " def make_with_tags(fn):\n", " def wrapped():\n", " return opening_tag + fn() + closing_tag\n", " return wrapped\n", " return make_with_tags\n", "\n", "# hello = enclose_in_tags('

', '

')(hello)\n", "@enclose_in_tags('

', '

') \n", "def hello():\n", " return \"hello world\"\n", "\n", "print(hello())\n", "\n", "# hello = enclose_in_tags('

', '

')(hello)\n", "@enclose_in_tags('

', '

')\n", "def hello():\n", " return \"hello world\"\n", "\n", "print(hello())\n", "\n", "# hello = enclose_in_tags('

', '

')(enclose_in_tags('

', '

')(hello))\n", "@enclose_in_tags('

', '

')\n", "@enclose_in_tags('

', '

')\n", "def hello():\n", " return \"hello world\"\n", "\n", "print(hello())" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "

hello world

\n" ] } ], "source": [ "# Decorators with arguments as instances of callable classes\n", "\n", "class TagEncloser:\n", " def __init__(self, opening_tag, closing_tag):\n", " self.opening_tag = opening_tag\n", " self.closing_tag = closing_tag\n", " def __call__(self, fn):\n", " def wrapped():\n", " return self.opening_tag + fn() + self.closing_tag\n", " return wrapped\n", "\n", "tag_h1 = TagEncloser('

', '

')\n", "tag_p = TagEncloser('

', '

')\n", "\n", "@tag_h1\n", "@tag_p\n", "def hello(): # hello = tag_h1(tag_p(hello))\n", " return \"hello world\"\n", "\n", "print(hello())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Iterables and Iterators\n", "\n", "1. **Iteration** is a general term for taking each item of something, one after another. Any time you use a loop, explicit or implicit, to go over a group of items, that is iteration.\n", "2. An **iterable** is an anything that can be looped over. It either:\n", " - Has an `__iter__` method which returns an **iterator** for that object when you call `iter()` on it, or implicitly in a for loop.\n", " - Defines a `__getitem__` method that can take sequential indexes starting from zero (and raises an `IndexError` when the indexes are no longer valid).\n", "3. An **iterator** is:\n", " * A stateful helper object which defines a `__next__` method and will produce the next value when you call ``next()`` on it. If there are no further items, it raises the `StopIteration` exception.\n", " * An object that is *self-iterable* (meaning that it has an `__iter__` method that returns `self`).\n", " \n", "Therefore: An *iterable* is an object from which we can get an *iterator*. An *iterator* is **always** an *iterable*. An *iterable* **is not always** an *iterator* but will always **return** an *iterator*." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\n", "2\n", "1\n", "\n", "\n" ] } ], "source": [ "x = [1, 2, 3] # this is an iterable\n", "y = iter(x) # an iterator of this iterable\n", "z = iter(x) # another iterator of this iterable\n", "\n", "print(next(y))\n", "print(next(y))\n", "print(next(z))\n", "print(type(x))\n", "print(type(y))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "* Here, `x` is the iter**able**, while `y` and `z` are two individual instances of an iterat**or**, producing values from the iterable x. Both `y` and `z` hold state, as you can see from the example. In this example, `x` is a data structure (a list), but that is not a requirement.\n", "* Often, for pragmatic reasons, iterable classes will implement both `__iter__()` and `__next__()` in the same class, and have `__iter__()` return `self`, which makes the class **both an iterable and its own iterator**. It is perfectly fine to return a different object as the iterator, though." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "source": [ "When you write:\n", "```python\n", "x = [1, 2, 3]\n", "for elem in x:\n", " ...\n", "```\n", "This is what actually happens:\n", "" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "c\n", "a\n", "t\n", "StopIteration raised\n" ] } ], "source": [ "s = 'cat' # s is an ITERABLE\n", " # s is a str object that is immutable\n", " # s has no state\n", " # s has a __getitem__() method \n", "\n", "t = iter(s) # t is an ITERATOR\n", " # t has state (it starts by pointing at the \"c\")\n", " # t has a __next__() method and an __iter__() method\n", " \n", "try:\n", " print(next(t)) # the next() function returns the next value and advances the state\n", " print(next(t)) # the next() function returns the next value and advances\n", " print(next(t)) # the next() function returns the next value and advances\n", " print(next(t)) # next() raises StopIteration to signal that iteration is complete\n", "except StopIteration as e:\n", " print(\"StopIteration raised\")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]\n", "[5, 4, 3, 2, 1, 'launch']\n" ] } ], "source": [ "class FibonacciIterator:\n", " \"\"\"\n", " Produces an arbitrary number of the Fibonacci numbers.\n", " Is an both an iterable and an iterator.\n", " \"\"\"\n", " def __init__(self):\n", " self.prev = 0\n", " self.curr = 1\n", " \n", " def __iter__(self):\n", " return self\n", " \n", " def __next__(self):\n", " self.prev, self.curr = self.curr, self.prev + self.curr\n", " return self.prev\n", "\n", "class Countdown:\n", " \"\"\"A simple iterable, NOT an iterator\"\"\"\n", " def __iter__(self):\n", " return iter([5, 4, 3, 2, 1, 'launch'])\n", " \n", "f = FibonacciIterator()\n", "print([next(f) for _i in range(10)])\n", "c = Countdown()\n", "print([i for i in c])" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "source": [ "# Generators\n", "A ***generator*** is a factory object that *lazily* produces values (i.e. *generates* values on demand). A generator is either:\n", "1. A **function** that contains the `yield` keyword (`yield expression`).\n", " - When this function is called, it **does not** execute, but returns a generator object.\n", " - The first time that `next()` is called, when this function encounters the `yield keyword` it suspends execution at that point, saves its context and returns to the caller along with any value in expression.\n", " - When the caller invokes `next()` again, execution of the function continues till another `yield` or `return` is encountered or end of function is reached.\n", "2. A **generator expression**, i.e. a syntactic construct for creating an anonymous generator object, following the form of mathematical set-builder notation. These are like list comprehensions but enclosed in `()` instead of `[]`.\n", " - The semantics of a generator expression are equivalent to creating an anonymous generator function and calling it.\n", "\n", "A *generator* is **always** an *iterator*, but not vice versa (*iterator* is a more general concept)." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "1\n", "[4, 9, 16, 25, 36, 49, 64, 81]\n", "0\n", "1\n", "[4, 9, 16, 25, 36, 49, 64, 81]\n" ] } ], "source": [ "# generator expression\n", "g = (x ** 2 for x in range(10))\n", "print(next(g))\n", "print(next(g))\n", "print([i for i in g])\n", "\n", "# generator function\n", "def _gen(exp):\n", " for x in exp:\n", " yield x ** 2\n", "\n", "g = _gen(range(10))\n", "print(next(g))\n", "print(next(g))\n", "print([i for i in g])" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ABCDE\n", "\n", "ABCDE\n", "\n", "ABCDE\n", "\n", "ABCDE\n", "\n" ] } ], "source": [ "# generator\n", "def it_gen(text):\n", " for ch in text:\n", " yield ch.upper()\n", "\n", "# generator expression\n", "def it_genexp(text):\n", " return (ch.upper() for ch in text)\n", "\n", "# iterator protocol (__iter__)\n", "class ItIter():\n", " def __init__(self, text):\n", " self.text = text\n", " self.index = 0\n", " def __iter__(self):\n", " return self\n", " def __next__(self):\n", " try:\n", " result = self.text[self.index].upper()\n", " except IndexError:\n", " raise StopIteration\n", " self.index += 1\n", " return result\n", "\n", "# iterator protocol (__getitem__)\n", "class ItGetItem():\n", " def __init__(self, text):\n", " self.text = text\n", " def __getitem__(self, index):\n", " result = self.text[index].upper()\n", " return result\n", "\n", "# an iterable of iterables (see what i did there? :P)\n", "for iterator in (it_gen, it_genexp, ItIter, ItGetItem):\n", " for ch in iterator('abcde'):\n", " print(ch, end='')\n", " print('\\n')" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[0, 1], [0, 1], [0, 1]], [[0, 1], [0, 1], [0, 1]], [[0, 1], [0, 1], [0, 1]], [[0, 1], [0, 1], [0, 1]]]\n", "[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]\n", "[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]\n" ] } ], "source": [ "import collections\n", "\n", "def flatten(container):\n", " \"\"\"Flatten an iterable of arbitrary depth.\"\"\"\n", " for item in container:\n", " if isinstance(item, collections.Iterable) and not isinstance(item, (str, bytes)):\n", " for element in flatten(item):\n", " yield element\n", " else:\n", " yield item\n", "\n", "d3 = [[[i for i in range(2)] for j in range(3)] for k in range(4)]\n", "print(d3)\n", "print([k for i in d3 for j in i for k in j])\n", "print([i for i in flatten(d3)])" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# More cool stuff\n", "* Partial functions, currying\n", "* Coroutines, `asyncio`, `yield from` \n", "* `functools`, `itertools` (standard library)\n", "* `toolz` (cool functional library)\n", "" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "source": [ "# References\n", "\n", "1. [K. Reitz - The Hitchhiker's Guide to Python](http://docs.python-guide.org/en/latest)\n", "* [PyToolz API Documentation](http://toolz.readthedocs.io/en/latest)\n", "* [StackOverflow - How to make a chain of function decorators in Python?](http://stackoverflow.com/questions/739654/how-to-make-a-chain-of-function-decorators-in-python)\n", "* [StackOverflow - What exactly are Python's iterator, iterable, and iteration protocols?](http://stackoverflow.com/questions/9884132/what-exactly-are-pythons-iterator-iterable-and-iteration-protocols)\n", "* [Bruce Eckel - Decorators I: Introduction to Python Decorators](http://www.artima.com/weblogs/viewpost.jsp?thread=240808)\n", "* [Bruce Eckel - Python Decorators II: Decorator Arguments](http://www.artima.com/weblogs/viewpost.jsp?thread=240845)\n", "* [V. Driessen - Iterables vs. Iterators vs. Generators](http://nvie.com/posts/iterators-vs-generators)\n", "* [Intermediate Pythonista - Introduction to Python Generators](http://intermediatepythonista.com/python-generators)\n", "* [David Beazley - Generator Tricks for Systems Programmers](http://www.dabeaz.com/generators-uk/)\n", "* D. Mertz - Functional Programming in Python, O' Reilly (2015)\n", "* S. Lott - Functional Python Programming, Packt Publishing (2015)" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }