Expert Python Programming - Third Edition: Become a master in Python by learning coding best practices and advanced programming concepts in Python 3.7, 3rd Edition

Expert Python Programming

Third Edition

Become a master in Python by learning coding best practices and advanced programming concepts in Python 3.7

Michał Jaworski

Tarek Ziadé

BIRMINGHAM - MUMBAI

Expert Python Programming Third Edition

Copyright © 2019 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Kunal Chaudhari

Acquisition Editor: Chaitanya Nair

Content Development Editor: Zeeyan Pinheiro

Technical Editor: Ketan Kamble

Copy Editor: Safis Editing

Project Coordinator: Vaidehi Sawant

Proofreader: Safis Editing

Indexer: Priyanka Dhadke

Graphics: Alishon Mendonsa

Production Coordinator: Shraddha Falebhai

First published: September 2008

Second edition: May 2016

Third edition: April 2019

Production reference: 1270419

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78980-889-6

www.packtpub.com

To my beloved wife, Oliwia, for her love, inspiration, and her endless patience.

To my loyal friends, Piotr, Daniel, and Paweł, for their support.

To my mother, for introducing me to the amazing world of programming.

                                                                                                 – Michał Jaworski

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

Improve your learning with Skill Plans built especially for you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

Packt.com

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Contributors

About the authors

Michał Jaworski has 10 years' of professional experience in Python. He has been in various roles at different companies, from an ordinary full-stack developer, through software architect, to VP of engineering in a fast-paced start-up company. He is currently a senior backend engineer at Showpad. He is highly experienced in designing high-performance distributed services. He is also an active contributor to many open source Python projects.

Tarek Ziadé is a Python developer located in the countryside near Dijon, France. He works at Mozilla in the services team. He founded a French Python user group called Afpy, and has written several books about Python in French and English. When he is not hacking on his computer or hanging out with his family, he's spending time between his two passions, running and playing the trumpet.

You can visit his personal blog (Fetchez le Python) and follow him on Twitter (tarek_ziade).

About the reviewer

Cody Jackson is a disabled military veteran, the founder of Socius Consulting, an IT and business management consulting company in San Antonio, and a co-founder of Top Men Technologies. He is currently employed at CACI International as the lead ICS/SCADA modeling and simulations engineer. He has been involved in the tech industry since 1994, when he joined the Navy as a nuclear chemist and radcon technician. Prior to CACI, he worked at ECPI University as a computer information systems adjunct professor. A self-taught Python programmer, he is the author of Learning to Program Using Python and Secret Recipes of the Python Ninja. He holds an Associate in Science degree, a Bachelor of Science degree, and a Master of Science degree.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Title Page

Copyright and Credits

Expert Python Programming Third Edition

Dedication

About Packt

Why subscribe?

Packt.com

Contributors

About the authors

About the reviewer

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Section 1: Before You Start

Current Status of Python

Technical requirements

Where are we now and where we are going to?

Why and how Python changes

Being up-to-date with changes ­by following PEP documents

Python 3 adoption at the time of writing this book

The main differences between Python 3 and Python 2

Why should I care?

The main syntax differences and common pitfalls

Syntax changes

Changes in the standard library

Changes in data types and collections and string literals

The popular tools and techniques used for maintaining cross-version compatibility

Not only CPython

Why should I care?

Stackless Python

Jython

IronPython

PyPy

MicroPython

Useful resources

Summary

Modern Python Development Environments

Technical requirements

Installing additional Python packages using pip

Isolating the runtime environment

Application-level isolation versus system-level isolation

Python's venv

venv versus virtualenv

System-level environment isolation

Virtual development environments using Vagrant

Virtual environments using Docker

Containerization versus virtualization

Writing your first Dockerfile

Running containers

Setting up complex environments

Useful Docker recipes for Python

Reducing the size of containers

Addressing services inside of a Compose environment

Communicating between multiple Compose environments

Popular productivity tools

Custom Python shells – ipython, bpython, ptpython, and so on

Setting up the PYTHONSTARTUP environment variable

IPython

bpython

ptpython

Incorporating shells in your own scripts and programs

Interactive debuggers

Summary

Section 2: Python Craftsmanship

Modern Syntax Elements - Below the Class Level

Technical requirements

Python's built-in types

Strings and bytes

Implementation details

String concatenation

Constant folding, the peephole optimizer, and the AST optimizer

String formatting with f-strings

Containers

Lists and tuples

Implementation details

List comprehensions

Other idioms

Dictionaries

Implementation details

Weaknesses and alternatives

Sets

Implementation details

Supplemental data types and containers

Specialized data containers from the collections module

Symbolic enumeration with the enum module

Advanced syntax

Iterators

Generators and yield statements

Decorators

General syntax and possible implementations

As a function

As a class

Parametrizing decorators

Introspection preserving decorators

Usage and useful examples

Argument checking

Caching

Proxy

Context provider

Context managers – the with statement

The general syntax and possible implementations

As a class

As a function – the contextlib module

Functional-style features of Python

What is functional programming?

Lambda functions

map(), filter(), and reduce()

Partial objects and partial() functions

Generator expressions

Function and variable annotations

The general syntax

The possible uses

Static type checking with mypy

Other syntax elements you may not know of yet

The for ... else ... statement

Keyword-only arguments

Summary

Modern Syntax Elements - Above the Class Level

Technical requirements

The protocols of the Python language – dunder methods and attributes

Reducing boilerplate with data classes

Subclassing built-in types

MRO and accessing methods from superclasses

Old-style classes and super in Python 2

Understanding Python's Method Resolution Order

Super pitfalls

Mixing super and explicit class calls

Heterogeneous arguments

Best practices

Advanced attribute access patterns

Descriptors

Real-life example – lazily evaluated attributes

Properties

Slots

Summary

Elements of Metaprogramming

Technical requirements

What is metaprogramming?

Decorators – a method of metaprogramming

Class decorators

Using __new__() for overriding the instance creation process

Metaclasses

The general syntax

New Python 3 syntax for metaclasses

Metaclass usage

Metaclass pitfalls

Code generation

exec, eval, and compile

Abstract syntax tree (AST)

Import hooks

Projects that use code generation patterns

Falcon's compiled router

Hy

Summary

Choosing Good Names

Technical requirements

PEP 8 and naming best practices

Why and when to follow PEP 8?

Beyond PEP 8 – Team-specific style guidelines

Naming styles

Variables

Constants

Naming and usage

Public and private variables

Functions and methods

The private controversy

Special methods

Arguments

Properties

Classes

Modules and packages

The naming guide

Using the has/is prefixes for Boolean elements

Using plurals for variables that are collections

Using explicit names for dictionaries

Avoid generic names and redundancy

Avoiding existing names

Best practices for arguments

Building arguments by iterative design

Trusting the arguments and your tests

Using *args and **kwargs magic arguments carefully

Class names

Module and package names

Useful tools

Pylint

pycodestyle and flake8

Summary

Writing a Package

Technical requirements

Creating a package

The confusing state of Python packaging tools

The current landscape of Python packaging thanks to PyPA

Tool recommendations

Project configuration

setup.py

setup.cfg

MANIFEST.in

Most important metadata

Trove classifiers

Common patterns

Automated inclusion of version string from package

README file

Managing dependencies

The custom setup command

Working with packages during development

setup.py install

Uninstalling packages

setup.py develop or pip -e

Namespace packages

Why is it useful?

PEP 420 - implicit namespace packages

Namespace packages in previous Python versions

Uploading a package

PyPI - Python Package Index

Uploading to PyPI - or other package index

.pypirc

Source packages versus built packages

sdist

bdist and wheels

Standalone executables

When standalone executables useful?

Popular tools

PyInstaller

cx_Freeze

py2exe and py2app

Security of Python code in executable packages

Making decompilation harder

Summary

Deploying the Code

Technical requirements

The Twelve-Factor App

Various approaches to deployment automation

Using Fabric for deployment automation 

Your own package index or index mirror

PyPI mirroring

Bundling additional resources with your Python package

Common conventions and practices

The filesystem hierarchy

Isolation

Using process supervision tools

Application code running in user space

Using reverse HTTP proxies

Reloading processes gracefully

Code instrumentation and monitoring

Logging errors – Sentry/Raven

Monitoring system and application metrics

Dealing with application logs

Basic low-level log practices

Tools for log processing

Summary

Python Extensions in Other Languages

Technical requirements

Differentiating between the C and C++ languages

Loading extensions in C or C++

The need to use extensions

Improving the performance in critical code sections

Integrating existing code written in different languages

Integrating third-party dynamic libraries

Creating custom datatypes

Writing extensions

Pure C extensions

A closer look at Python/C API

Calling and binding conventions

Exception handling

Releasing GIL

Reference counting

Writing extensions with Cython

Cython as a source-to-source compiler

Cython as a language

Challenges with using extensions

Additional complexity

Debugging

Interfacing with dynamic libraries without extensions

The ctypes module

Loading libraries

Calling C functions using ctypes

Passing Python functions as C callbacks

CFFI

Summary

Section 3: Quality over Quantity

Managing Code

Technical requirements

Working with a version control system

Centralized systems

Distributed systems

Distributed strategies

Centralized or distributed?

Use Git if you can

GitFlow and GitHub Flow

Setting up continuous development processes

Continuous integration

Testing every commit

Merge testing through CI

Matrix testing

Continuous delivery

Continuous deployment

Popular tools for continuous integration

Jenkins

Buildbot

Travis CI

GitLab CI

Choosing the right tool and common pitfalls

Problem 1 – Complex build strategies

Problem 2 – Long building time

Problem 3 – External job definitions

Problem 4 – Lack of isolation

Summary

Documenting Your Project

Technical requirements

The seven rules of technical writing

Write in two steps

Target the readership

Use a simple style

Limit the scope of information

Use realistic code examples

Use a light but sufficient approach

Use templates

Documentation as code

Using Python docstrings

Popular markup languages and styles for documentation

Popular documentation generators for Python libraries

Sphinx

Working with the index pages

Registering module helpers

Adding index markers

Cross-references

MkDocs

Documentation building and continuous integration

Documenting web APIs

Documentation as API prototype with API Blueprint

Self-documenting APIs with Swagger/OpenAPI

Building a well-organized documentation system

Building documentation portfolio

Design

Usage

Recipe

Tutorial

Module helper

Operations

Your very own documentation portfolio

Building a documentation landscape

Producer's layout

Consumer's layout

Summary

Test-Driven Development

Technical requirements

I don't test

Three simple steps of test-driven development

Preventing software regression

Improving code quality

Providing the best developer documentation

Producing robust code faster

What kind of tests?

Unit tests

Acceptance tests

Functional tests

Integration tests

Load and performance testing

Code quality testing

Python standard test tools

unittest

doctest

I do test

unittest pitfalls

unittest alternatives

nose

Test runner

Writing tests

Writing test fixtures

Integration with setuptools and plugin system

Wrap-up

py.test

Writing test fixtures

Disabling test functions and classes

Automated distributed tests

Wrap-up

Testing coverage

Fakes and mocks

Building a fake

Using mocks

Testing environment and dependency compatibility

Dependency matrix testing

Document-driven development

Writing a story

Summary

Section 4: Need for Speed

Optimization - Principles and Profiling Techniques

Technical requirements

The three rules of optimization

Making it work first

Working from the user's point of view

Keeping the code readable and maintainable

Optimization strategy

Looking for another culprit

Scaling the hardware

Writing a speed test

Finding bottlenecks

Profiling CPU usage

Macro-profiling

Micro-profiling

Profiling memory usage

How Python deals with memory

Profiling memory

objgraph

C code memory leaks

Profiling network usage

Tracing network transactions

Summary

Optimization - Some Powerful Techniques

Technical requirements

Defining complexity

Cyclomatic complexity

The big O notation

Reducing complexity by choosing proper data structures

Searching in a list

Using sets

Using collections

deque

defaultdict

namedtuple

Using architectural trade-offs

Using heuristics and approximation algorithms

Using task queues and delayed processing

Using probabilistic data structures

Caching

Deterministic caching

Non-deterministic caching

Cache services

Memcached

Summary

Concurrency

Technical requirements

Why concurrency?

Multithreading

What is multithreading?

How Python deals with threads

When should we use threading?

Building responsive interfaces

Delegating work

Multiuser applications

An example of a threaded application

Using one thread per item

Using a thread pool

Using two-way queues

Dealing with errors and rate limiting

Multiprocessing

The built-in multiprocessing module

Using process pools

Using multiprocessing.dummy as the multithreading interface

Asynchronous programming

Cooperative multitasking and asynchronous I/O

Python async and await keywords

asyncio in older versions of Python

A practical example of asynchronous programming

Integrating non-asynchronous code with async using futures

Executors and futures

Using executors in an event loop

Summary

Section 5: Technical Architecture

Event-Driven and Signal Programming

Technical requirements

What exactly is event-driven programming?

Event-driven != asynchronous

Event-driven programming in GUIs

Event-driven communication

Various styles of event-driven programming

Callback-based style

Subject-based style

Topic-based style

Event-driven architectures

Event and message queues

Summary

Useful Design Patterns

Technical requirements

Creational patterns

Singleton

Structural patterns

Adapter

Interfaces

Using zope.interface

Using function annotations and abstract base classes

Using collections.abc

Proxy

Facade

Behavioral patterns

Observer

Visitor

Template

Summary

reStructuredText Primer

reStructuredText

Section structure

Lists

Inline markup

Literal block

Links

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

Python is a dynamic programming language, used in a wide range of domains thanks to its simple yet powerful nature. Although writing Python code is easy, making it readable, reusable, and easy to maintain is challenging. Complete with best practices, useful tools, and standards implemented by professional Python developers, the third version of Expert Python Programming will help you overcome this challenge.

The book will start by taking you through the new features in Python 3.7. You'll learn the Python syntax and understand how to apply advanced object-oriented concepts and mechanisms. You'll also explore different approaches to implement metaprogramming. This book will guide you in following best naming practices when writing packages, and creating standalone executables easily, alongside using powerful tools such as buildout and virtualenv to deploy code on remote servers. You'll discover how to create useful Python extensions with C, C++, Cython, and Pyrex. Furthermore, learning about code management tools, writing clear documentation, and test-driven development will help you write clean code.

By the end of the book, you will have become an expert in writing efficient and maintainable Python code.

Who this book is for

This book is written for Python developers who wish to go further in mastering Python. And by developers, I mean mostly professionals, so programmers who write Python software for their living. This is because it focuses mostly on tools and practices that are crucial for creating performant, reliable, and maintainable software in Python.

It does not mean that hobbyists won't find anything interesting. This book should be great for anyone who is interested in learning advanced-level concepts with Python. Anyone who has basic Python skills should be able to follow the content of the book, although it might require some additional effort from less experienced programmers. It should also be a good introduction to Python 3.7 for those who are still a bit behind and continue to use Python version 2.7 or older.

Finally, the groups that should benefit most from reading this book are web developers and backend engineers. This is because of two topics featured in here that are especially important in their areas of work: reliable code deployments and concurrency.

What this book covers

Chapter 1, Current Status of Python, showcases the current state of the Python language and its community. We will see how Python is constantly changing, why it is changing, and also why these facts are important for anyone who wants to call themselves a Python professional. We will also take a look at the most popular and canonical ways for working on written in Python—popular productivity tools and conventions that are de facto standards now.

Chapter 2, Modern Python Development Environments, describes modern ways of setting up repeatable and consistent development environments for Python programmers. We will concentrate on two popular tools for environment isolation: virtualenv-type environments and Docker containers.

Chapter 3, Modern Syntax Elements – Below the Class Level, focuses on best practices for writing code in Python (language idioms) and also provides a summary of selected elements of Python syntax that may be new for intermediate Python users or those experienced with older versions of Python. We will also take a look at useful notes about internal CPython-type implementations and their computational complexities as a rationale for provided idioms.

Chapter 4, Modern Syntax Elements – Above the Class Level, covers more advanced object-oriented concepts and mechanisms available in Python.

Chapter 5, Elements of Metaprogramming, presents an overview of common approaches to metaprogramming available to Python programmers.

Chapter 6, Choosing Good Names, explains what is the most widely-adopted style guide for Python code (PEP-8) and when and why developers should follow it. We will also take a look at some of the author's general advice for naming things.

Chapter 7, Writing a Package, describes the current state of Python packaging and best practices for creating packages that are to be distributed as open source code in the Python Package Index (PyPI). We will also cover an often overlooked topic of Python – standalone executables.

Chapter 8, Deploying Code, presents some common lightweight tools for deploying Python code on remote servers. Deployment is one of the fields where Python shines are backends for web-based services and applications.

Chapter 9, Python Extensions in Other Languages, explains why writing extensions in C and C++ for Python can sometimes be a good solution and shows that it is not as hard as it seems, as long as the proper tools are used.

Chapter 10, Managing Code, describes how to properly manage a code base and why version control systems should be used. We will also leverage the power of version control systems (especially Git) in implementing continuous processes, such as continuous integration and continuous delivery.

Chapter 11, Documenting Your Project, describes the general rules for writing technical documentation that may be applied to software written in any language, and various tools that are especially useful for creating documentation of your Python code.

Chapter 12, Test-Driven Development, advocates the usage of test-driven development and provides more information on how to use popular Python tools designed for testing.

Chapter 13, Optimization – Principles and Profiling Techniques, discusses the most basic rules of optimization that every developer should be aware of. We will also learn how to identify application performance bottlenecks and use common profiling tools.

Chapter 14, Optimization – Some Powerful Techniques, shows how to use that knowledge to actually make your application run faster or be more efficient in terms of used resources.

Chapter 15, Concurrency, explains how to implement concurrency in Python using different approaches and libraries.

Chapter 16, Event-Driven and Signal Programming, describes what event-driven/signal programming is and how it relates to asynchronous programming and different concurrency models. We will present the various approaches to event-driven programming available to Python programmers, along with useful libraries that enable these patterns.

Chapter 17, Useful Design Patterns, implements a set of useful design patterns and example implementations in Python.

Appendix A, reStructuredText Primer, provides a brief tutorial on how to use reStructuredText markup language.

To get the most out of this book

This book is written for developers who work under any operating system for which Python 3 is available.

This is not a book for beginners, so I assume you have Python installed in your environment or know how to install it. Anyway, this book takes into account the fact that not everyone needs to be fully aware of the latest Python features or officially recommended tools. This is why the first chapter provides a recap on common utilities (such as virtual environments and pip) that are now considered standard tools of professional Python developers.

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

Log in or register at www.packt.com.

Select the SUPPORT tab.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Expert-Python-Programming-Third-Edition. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789808896_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: Any attempt to run the code that has such issues will immediately cause the interpreter to fail, raising a SyntaxError exception.

A block of code is set as follows:

print(hello world)

print goodbye python2

Any command-line input or output is written as follows:

$ python3 script.py

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

Section 1: Before You Start

This part prepares the user for the modern Python development routine. It explains how Python has changed over the last few years and what the common development tools used by modern Python programmers are.

The following chapters are included in this section:

Chapter 1, Current Status of Python

Chapter 2, Modern Python Development Environments

Current Status of Python

Python is amazing.

For a very long time, one of the most important virtues of Python was interoperability. No matter what operating system you or your customers were using, if a Python interpreter was available for that system, your software that was written in Python would work there. And, most importantly, your software would work the same way. However, that's not uncommon anymore. Modern languages such as Ruby and Java provide similar interoperability capabilities. But, interoperability isn't the most important quality of programming language nowadays. With the advent of cloud computing, web-based applications, and reliable virtualization software, it isn't that important to have a programming language that works the same no matter the operating system. What is still important is the tools that allow programmers to efficiently write reliable and maintainable software. Fortunately, Python is still one of the languages that allows programmers the most efficiency, and is definitely a smart choice for a company's primary development language.

Python stays relevant for so long because it is constantly evolving. This book is focused on the latest Python 3.7 version, and all code examples are written in this version of the language unless another version is explicitly mentioned. Because Python has a very long history, and there are still programmers using Python 2 on a daily basis, this book starts with a chapter that describes the current status quo of Python 3. In this chapter, you'll find how and why Python changes, and will learn how to write software that is compatible with both the historic and latest versions of Python.

In this chapter, we will cover the following topics:

Where are we now and where we are going to?

Why and how Python changes

Being up-to-date with changes to PEP documentation

Python 3 adoption at the time of writing this book

The main difference between Python 3 and Python 2

Not only CPython

Useful resources

Technical requirements

You can download the latest version of Python from https://www.python.org/downloads/ for this chapter.

Alternative Python interpreter implementations can be found at the following sites:

Stackless Python:https://github.com/stackless-dev/stackless

PyPy: https://pypy.org

Jython: https://www.jython.org

IronPython: https://ironpython.net

MicroPython: https://micropython.org

The code files for this chapter can be found at https://github.com/PacktPublishing/Expert-Python-Programming-Third-Edition/tree/master/chapter1.

Where are we now and where we are going to?

Python history starts somewhere in the late 1980s, but its 1.0 release date was in the year 1994. So, it isn't a young language. There could be a whole timeline of major Python releases mentioned here, but what really matters is a single date: Python 3.0—December 3, 2008.

At the time of writing, almost ten years have passed since the first Python 3 release. It is also seven years since the creation of PEP 404—the official document that un-released Python 2.8 and officially closed the 2.x branch. Although a lot of time has passed, there is a specific dichotomy in the Python community—while the language is developing very fast, there is a large group of its users that do not want to move forward with it.

Why and how Python changes

The answer is simple—Python changes because there is such a need. The competition does not sleep. Every few months, a new language pops out, out of nowhere, claiming to solve every problem of all its predecessors. Most projects like these lose the developers' attention very shortly, and their popularity is often driven by sudden hype.

This is a sign of some bigger problem. People design new languages because they find that existing ones do not solve their problems in the best way possible. It would be silly to not recognize such a need. Also, more and more widespread usage of Python shows that it could, and should, be improved on in many places.

Many improvements in Python are driven by the needs of particular fields where it is being used. The most significant one is web development. Thanks to the ever-increasing demand for speed and performance in this area, we've seen that ways to deal with concurrency in Python have been drastically improved over the time.

Other changes are simply caused by the age and maturity of the Python project. Throughout the years, it collected some of the clutter in the form of disorganized and redundant standard library modules, or some bad design decisions. First, the Python 3 release aimed to bring with it a major cleanup and refreshment to the language. Unfortunately, time showed that this plan backfired a bit. For a long time, Python 3 was treated by many developers only like a curiosity. Hopefully, this is changing.

Being up-to-date with changes ­by following PEP documents

The Python community has a well-established way of dealing with changes. While speculative Python language ideas are mostly discussed on specific mailing lists ([email protected]), nothing major ever gets changed without the existence of a new document, called a Python Enhancement Proposal (PEP).

It is a formalized document that describes, in detail, the proposal of change to be made in Python. It is also the starting point for the community discussion. The whole purpose, format, and workflow around these documents is also standardized in the form of a PEP—precisely the PEP 1 document (http://www.python.org/dev/peps/pep-0001).

PEP documentation is very important for Python, and, depending on the topic, they serve different purposes:

Informing: They summarize the information needed by core Python developers, and notify about Python release schedules

Standardizing: They provide code style, documentation, or other guidelines

Designing: They describe the proposed features

A list of all proposed PEPs are available in a living PEP 0 document (https://www.python.org/dev/peps/). Since they are easily accessible in one place, and the actual URL is also very easy to guess, they are usually referred to by the number in the book.

The PEP 0 document is a great source of information for those who are wondering what direction Python language is heading in, but do not have time to track every discussion on Python mailing lists. It shows which documents were already accepted but not yet implemented, and also which are still under consideration.

PEPs also serve additional purposes. Very often, people ask questions like the following:

Why does feature A work that way?

Why does Python not have feature B?

In most such cases, the extensive answer is already available in specific PEP documents where such a feature was already mentioned. There is a lot of PEP documentation describing Python language features that were proposed but not accepted. This documentation is left as a historical reference.

Python 3 adoption at the time of writing this book

So, thanks to new, exciting features, is Python 3 well adopted among its community? It's hard to say. The once-popular page, Python 3 Wall of Superpowers (https://python3wos.appspot.com), that tracked the compatibility of the most popular packages with the Python 3 branch was, at the beginning, named Python 3 Wall of Shame.

The site is no longer maintained, but in the list from the last time it was updated, on April 22, 2018, it shows that exactly 191 from 200 of the most popular Python packages at that time were compatible within Python 3. So, we can see that Python 3 seems to be finally well-adopted in the community of open source Python programmers. Still, this does not mean that all teams building their applications are finally using Python 3. At least, since most of the popular Python packages are available in Python 3, the popular excuse packages that we use have not been ported yet is no longer valid.

The main reason for such a situation is that porting the existing application from Python 2 to Python 3 is always a challenge. There are tools such as 2to3 that can perform automated code translation, but they do not assure that the result will be 100% correct. Also, such translated code may not perform as well as in its original form without manual adjustments. Moving existing complex code bases to Python 3 might involve tremendous effort, and a cost that some organizations may not be able to afford. Fortunately, such costs can be split over time. Some good software architecture design methodologies, such as service-oriented architecture or microservices, can help to achieve this goal gradually. New project components (services or microservices) can be written using the new technology, and existing ones can be ported one at a time.

In the long run, moving to Python 3 can have only beneficial effects on a project. According to PEP 404, there won't be another 2.8 release in the 2.x branch of Python, and the official end-of-life for Python 2 is scheduled for 2020. Until that time, we can expect only patch version updates for major security issues, but nothing more. Also, there may be a time in the future when all major projects, such as Django, Flask, and NumPy will drop any 2.x compatibility and will be available only in Python 3. Django has already made that step, and since version 2.0.0 was released, it no longer supports Python 2.7.

My personal opinion on this topic can be considered controversial. I think that the best incentive for the community would be to completely drop Python 2 support when creating new packages. This, of course, limits a range of such software, but may be the only right way to change the way of thinking in those who insist on sticking to Python 2.x.

We'll take a look at the main differences between Python 3 and Python 2 in the next section.

The main differences between Python 3 and Python 2

It has already been stated that Python 3 breaks backward compatibility with Python 2 on a syntax level. Still, it is not a complete redesign. Also, it does not cause every Python module written for some 2.x release to stop working under Python 3. It is possible to write completely cross-compatible code that will run on both major releases without additional tools or techniques, but usually it is possible only for simple applications.

Why should I care?

Despite my personal opinion on Python 2 compatibility that I exposed earlier in this chapter, it is impossible to simply forget about it at this time. There are still some useful packages that are really worth using, but are not likely to be ported in the very near future.

Also, sometimes, we may be constrained by the organization we work in. The existing legacy code may be so complex that porting it is not economically feasible. So, even if we decide to move on and live only in the Python 3 world from now on, it will be impossible to live completely without Python 2 for some time.

Nowadays, it is very hard to call yourself a professional developer without giving something back to the community. So, helping the open source developers add Python 3 compatibility to the existing packages is a good way to pay off the moral debt incurred by using them. This, of course, cannot be done without knowing the differences between Python 2 and Python 3. By the way, this is also a great exercise for those new to Python 3.

The main syntax differences and common pitfalls

The Python documentation is the best reference for differences between every Python release. However, for your convenience, this section summarizes the most important ones. This does not change the fact that the documentation is mandatory reading for those not familiar with Python 3 yet (see https://docs.python.org/3.0/whatsnew/3.0.html).

The breaking changes that were introduced by Python 3 can be generally divided into three groups:

Syntax changes, where some syntax elements were removed/changed and other elements were added

Changes in the standard library

Changes in datatypes and collections

Syntax changes

Syntax changes that make it difficult for the existing code to run are the easiest to spot—they will cause the code to not run at all. The Python 3 code that uses new syntax elements will fail to run on Python 2 and vice versa. The elements that were removed from official syntax will make Python 2 code visibly incompatible with Python 3. Any attempt to run the code that has such issues will immediately cause the interpreter to fail, raising a SyntaxError exception. Here is an example of the broken script that has exactly two statements, of which none will be executed due to the syntax error:

print(hello world)

print goodbye python2

Its actual result when run on Python 3 is as follows:

$ python3 script.py

  File script.py, line 2

    print goodbye python2

                          ^

  SyntaxError: Missing parentheses in call to 'print'

When it comes to new elements of Python 3 syntax, the total list of differences is a bit long, and any new Python 3.x release may add new elements of syntax that will raise such errors on earlier releases of Python (even on the same 3.x branch). The most important of them are covered in Chapter 2, Modern Python Development Environments, and Chapter 3, Modern Syntax Elements – Below the Class Level, so there is no need to list all of them here.

The list of things that used to work in Python 2 that will cause syntax or functional errors in Python 3 is shorter. Here are the most important backwards incompatible changes:

print is no longer a statement, but a function, so the parenthesis is now obligatory.

Catching exceptions changed from except exc, var to except exc as var.

The <> comparison operator has been removed in favor of !=.

from module import * (https://docs.python.org/3.0/reference/simple_stmts.html#import) is now allowed only on module level, and no longer inside the functions.

from .[module] import name is now the only accepted syntax for relative imports. All imports not starting with a dot character are interpreted as absolute imports.

The sorted() function and the list's sort() method no longer accept the cmp argument. The key argument should be used instead.

Division expressions on integers such as one half return floats. The truncating behavior is achieved through the // operator like 1//2. The good thing is that this can be used with floats too, so 5.0//2.0 == 2.0.

Changes in the standard library

Breaking changes in the standard library are the second easiest to catch after syntax changes. Each subsequent version of Python adds, deprecates, improves, or completely removes standard library modules. Such a process was also common in the older branches of Python (1.x and 2.x), so it does not come as a shock in Python 3. In most cases, depending on the module that was removed or reorganized (such as urlparse being moved to urllib.parse), it will raise exceptions on the import time just after it is interpreted. This makes such issues so easy to catch. In order to be sure that all such issues are covered, full test code coverage is essential. In some cases (for example, when using lazily loaded modules), the issues that are usually noticed at import time will not appear before some modules are used in code as function calls. This is why it is so important to make sure that every line of code is actually executed during tests suite.

Lazily loaded modules

A lazy loaded module is a module that is not loaded on import time. In Python, the import statements can be included inside functions, so an import will happen on function call and not on import time. In some cases, such loading of modules may be a reasonable choice, but in most cases, it is a workaround for poorly designed module structure (for example, to avoid circular imports). It is considered bad code smell and should be generally avoided. There is no justifiable reason to lazily load standard library modules. In well-structured code, all imports should be grouped at the top of module.

Changes in data types and collections and string literals

Changes in how Python represents datatypes and collections require the most effort when the developer tries to maintain compatibility or simply ports existing code to Python 3. While incompatible syntax or standard library changes are easily noticeable and often easy to fix, changes in collections and types are either non-obvious or require a lot of repetitive work. The list of such changes is long and the official documentation is the best reference.

Still, this section must cover the change in how string literals are treated in Python 3, because it seems to be the most controversial and discussed change in Python 3, despite being a very good move that makes things more explicit.

All string literals are now Unicode, and bytestring literals require b or B prefix. For Python 3.0 and 3.1, the old Unicode u prefix (like ufoo) is illegal and will raise a syntax error. Dropping off that prefix was the main reason for most of the controversies. It made it really hard to create code compatible with different branches of Python—Python in version 2.x relied on these prefixes in order to create Unicode literals. This prefix was brought back in Python 3.3 to ease the integration process, although it now lacks any syntactic meaning.

The popular tools and techniques used for maintaining cross-version compatibility

Maintaining compatibility between versions of Python is a challenge. It may add a lot of additional work depending on the size of the project, but is definitely doable and worth doing. For packages that are meant to be reused in many environments it is absolutely a must-have. Open source packages without well-defined and tested compatibility bounds are very unlikely to become popular, but closed third-party code that never leaves the company network can also greatly benefit from being tested in different environments.

It should be noted here that, while this part focuses mainly on compatibility between various versions of Python, these approaches apply for maintaining compatibility with external dependencies such as different package versions, binary libraries, systems, or external services.

The whole process can be divided into three main areas, ordered by their importance:

Defining and documenting target compatibility bounds and how they will be managed

Testing in every environment and with every dependency version declared as compatible

Implementing actual compatibility code

Declaration of what is considered compatible is the most important part of the whole process because it gives your code users (developers) the ability to have expectations and make assumptions on how it works and how it can change in the future. Our code can be used as a dependency in different projects that may also strive to manage compatibility, so the ability to reason how it behaves is crucial.

While this book tries to always give a few choices and not to give absolute recommendations on specific options, here is one of the few exceptions. The best way to define how compatibility may change in the future is by using proper approach to versioning numbers using Semantic Versioning (semver) (http://semver.org/). It describes a broadly accepted standard for marking scope of changes in code by the version specifier, consisting only of three numbers. It also gives some advice on how to handle deprecation policies. Here is an excerpt from its summary (licensed under Creative Commons - CC BY 3.0):

Given a version number MAJOR.MINOR.PATCH, increment:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards-compatible manner, and

PATCH version when you make backward-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

When it comes to testing the sad truth, that is, to be sure that code is compatible with every declared dependency version and in every environment (here Python version), it must be tested in every combination of these. This, of course, may not be possible when the project has a lot of dependencies, because the number of combinations grows rapidly with every new dependency version. So, typically some trade-off needs to be made so that running full compatibility tests does not need to take ages. The selection of tools that help testing in so-called matrixes is presented in Chapter 12, Test-Driven Development, which discusses testing in general.

The benefit of using projects that follow semver is that usually what needs to be tested are only major releases, because minor and patch releases are guaranteed to not include backwards incompatible changes. This is, of course, only true if such projects can be trusted to not break such a contract. Unfortunately, mistakes happen to everyone, and backwards incompatible changes happen in a lot of projects, even on patch versions. Still, since semver declares strict compatibility on minor and patch versions, breaking it is considered a bug, so it may be fixed in a patch release.

The implementation of the compatibility layer is the last, and also the least important, step of the process if the bounds of that compatibility are well-defined and rigorously tested. Still, there are some tools and techniques that every programmer interested in such a topic should know.

The most basic is Python's __future__ module. It backports some features from newer Python releases back into the older ones and takes the form of an import statement:

from __future__ import

Features provided by the future statements are syntax-related elements that cannot be easily handled by different means. This statement affects only the module where it was used. Here is an example of a Python 2.7 interactive session that brings Unicode literals from Python 3.0:

Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit

(Intel)] on win32

Type help, copyright, credits or license for more

information.

>>> type(foo)

  # old literals

>>> from __future__ import unicode_literals

>>> type(foo)

  # now is Unicode

Here is a list of all the available __future__ statement options that developers concerned with two-thirds compatibility should know:

division: This adds a Python 3 division operator (PEP 238)

absolute_import: This makes every form of an import statement not starting from dot character be interpreted as absolute imports (PEP 328)

print_function: This changes a print statement into a function call so that parentheses around print become mandatory (PEP 3112)

unicode_literals: This makes every string literal be interpreted as Unicode literals (PEP 3112)

A list of the __future__ statement options is very short, and it covers only a few syntax features. The other things that have changed, such as the metaclass syntax (which is an advanced feature that's covered in Chapter 5, Elements of Metaprogramming), are a lot harder to maintain. Reliable handling of multiple standard library reorganizations also cannot be solved by the future statements. Fortunately, there are some tools that aim to provide a consistent layer of ready-to-use compatibility code. The most well-known of these is Six (https://pypi.python.org/pypi/six/), which provides a whole common two-thirds compatibility boilerplate as a single module. The other promising, but slightly less popular, tool is the future module (http://python-future.org/).

In some situations, developers may not want to include additional dependencies in some small packages. A common practice is the additional module that gathers all the compatibility code, usually named compat.py. Here is an example of such compat modules taken from the python-gmaps project (https://github.com/swistakm/python-gmaps):

# -*- coding: utf-8 -*-

"This module provides compatibility layer for

selected things that have changed across Python versions.

"

import sys

if sys.version_info < (3, 0, 0):

    import urlparse  # noqa

    def is_string(s):

        Return True if given value is considered string

        return isinstance(s, basestring)

else:

    # note: urlparse was moved to urllib.parse in Python 3

    from urllib import parse as urlparse  # noqa

    def is_string(s):

        Return True if given value is considered string

        return isinstance(s, str)

Such compat.py modules are popular, even in projects that depend on Six for two-thirds compatibility, because it is a very convenient way to store code that handles compatibility with different versions of packages being used as dependencies.

In the next section, we'll take a look at what CPython is.

Not only CPython

The reference Python interpreter implementation is called CPython and, as its name suggests, it is written entirely in the C language. It was always C and probably will be still for a very long time. That's the implementation that most Python programmers choose because it is always up to date with the language specification and is the interpreter that most libraries are tested on. But, besides C, Python interpreter was written in a few other languages. Also, there are some modified versions of CPython interpreter available under different names and tailored exactly for some niche applications. Most of them are a few milestones behind CPython, but provide a great opportunity to use and promote the language in a specific environment.

In this section, we will discuss some of the most prominent and interesting alternative Python implementations. 

Why should I care?

There are plenty of alternative Python implementations available. The Python wiki page on that topic (https://wiki.python.org/moin/PythonImplementations) features dozens of different language variants, dialects, or implementations of Python interpreter built with something other than C. Some of them implement only a subset of the core language syntax, features, and built-in extensions, but there are at least a few that are almost fully compatible with CPython. The most important thing to know is that, while some of them are just toy projects or experiments, most of them were created to solve some real problems – problems that were either impossible to solve with CPython or required too much of the developer's effort.

Examples of such problems are as follows:

Running Python code on embedded systems

Integration with code written for runtime frameworks, such as Java or .NET, or in different languages

Running Python code in web browsers

The following sections provide a short description of, subjectively, the most popular and up-to-date choices that are currently available for