Parallel Programming with Python

Table of Contents

Parallel Programming with Python

Credits

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Contextualizing Parallel, Concurrent, and Distributed Programming

Why use parallel programming?

Exploring common forms of parallelization

Communicating in parallel programming

Understanding shared state

Understanding message passing

Identifying parallel programming problems

Deadlock

Starvation

Race conditions

Discovering Python's parallel programming tools

The Python threading module

The Python multiprocessing module

The parallel Python module

Celery – a distributed task queue

Taking care of Python GIL

Summary

2. Designing Parallel Algorithms

The divide and conquer technique

Using data decomposition

Decomposing tasks with pipeline

Processing and mapping

Identifying independent tasks

Identifying the tasks that require data exchange

Load balance

Summary

3. Identifying a Parallelizable Problem

Obtaining the highest Fibonacci value for multiple inputs

Crawling the Web

Summary

4. Using the threading and concurrent.futures Modules

Defining threads

Advantages and disadvantages of using threads

Understanding different kinds of threads

Defining the states of a thread

Choosing between threading and _thread

Using threading to obtain the Fibonacci series term with multiple inputs

Crawling the Web using the concurrent.futures module

Summary

5. Using Multiprocessing and ProcessPoolExecutor

Understanding the concept of a process

Understanding the process model

Defining the states of a process

Implementing multiprocessing communication

Using multiprocessing.Pipe

Understanding multiprocessing.Queue

Using multiprocessing to compute Fibonacci series terms with multiple inputs

Crawling the Web using ProcessPoolExecutor

Summary

6. Utilizing Parallel Python

Understanding interprocess communication

Exploring named pipes

Using named pipes with Python

Writing in a named pipe

Reading named pipes

Discovering PP

Using PP to calculate the Fibonacci series term on SMP architecture

Using PP to make a distributed Web crawler

Summary

7. Distributing Tasks with Celery

Understanding Celery

Why use Celery?

Understanding Celery's architecture

Working with tasks

Discovering message transport (broker)

Understanding workers

Understanding result backends

Setting up the environment

Setting up the client machine

Setting up the server machine

Dispatching a simple task

Using Celery to obtain a Fibonacci series term

Defining queues by task types

Using Celery to make a distributed Web crawler

Summary

8. Doing Things Asynchronously

Understanding blocking, nonblocking, and asynchronous operations

Understanding blocking operations

Understanding nonblocking operations

Understanding asynchronous operations

Understanding event loop

Polling functions

Using event loops

Using asyncio

Understanding coroutines and futures

Using coroutine and asyncio.Future

Using asyncio.Task

Using an incompatible library with asyncio

Summary

Index

Parallel Programming with Python

Parallel Programming with Python

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: June 2014

Production reference: 1180614

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78328-839-7

www.packtpub.com

Cover image by Lis Marie Martini (<[email protected]>)

Credits

Author

Jan Palach

Reviewers

Cyrus Dasadia

Wei Di

Michael Galloy

Ludovic Gasc

Kamran Hussain

Bruno Torres

Commissioning Editor

Rebecca Youé

Acquisition Editor

Llewellyn Rozario

Content Development Editor

Sankalp Pawar

Technical Editors

Novina Kewalramani

Humera Shaikh

Copy Editors

Roshni Banerjee

Sarang Chari

Gladson Monteiro

Project Coordinator

Lima Danti

Proofreaders

Simran Bhogal

Maria Gould

Paul Hindle

Indexers

Mehreen Deshmukh

Rekha Nair

Tejal Soni

Priya Subramani

Graphics

Disha Haria

Abhinash Sahu

Production Coordinator

Saiprasad Kadam

Cover Work

Saiprasad Kadam

About the Author

Jan Palach has been a software developer for 13 years, having worked with scientific visualization and backend for private companies using C++, Java, and Python technologies. Jan has a degree in Information Systems from Estácio de Sá University, Rio de Janeiro, Brazil, and a postgraduate degree in Software Development from Paraná State Federal Technological University. Currently, he works as a senior system analyst at a private company within the telecommunication sector implementing C++ systems; however, he likes to have fun experimenting with Python and Erlang—his two technological passions. Naturally curious, he loves challenges and learning new technologies, meeting new people, and learning about different cultures.

Acknowledgments

I had no idea how hard it could be to write a book with such a tight deadline among so many other things taking place in my life. I had to fit the writing into my routine, taking care of my family, karate lessons, work, Diablo III, and so on. The task was not easy; however, I got to the end of it hoping that I have generated quality content to please most readers, considering that I have focused on the most important thing based on my experience.

The list of people I would like to acknowledge is so long that I would need a book only for this. So, I would like to thank some people I have constant contact with and who, in a direct or indirect way, helped me throughout this quest.

My wife Anicieli Valeska de Miranda Pertile, the woman I chose to share my love with and gather toothbrushes with to the end of this life, who allowed me to have the time to create this book and did not let me give up when I thought I could not make it. My family has always been important to me during my growth as a human being and taught me the path of goodness.

I would like to thank Fanthiane Ketrin Wentz, who beyond being my best friend is also guiding me through the ways of martial arts, teaching me the values I will carry during a lifetime—a role model for me. Lis Marie Martini, dear friend who provided the cover for this book, and who is an incredible photographer and animal lover.

Big thanks to my former English teacher, reviser, and proofreader, Marina Melo, who helped along the writing of this book. Thanks to the reviewers and personal friends, Vitor Mazzi and Bruno Torres, who contributed a lot to my professional growth and still do.

Special thanks to Rodrigo Cacilhas, Bruno Bemfica, Rodrigo Delduca, Luiz Shigunov, Bruno Almeida Santos, Paulo Tesch (corujito), Luciano Palma, Felipe Cruz, and other people with whom I often talk to about technology. A special thanks to Turma B.

Big thanks to Guido Van Rossum for creating Python, which transformed programming into something pleasant; we need more of this stuff and less set/get.

About the Reviewers

Cyrus Dasadia has worked as a Linux system administrator for over a decade for organizations such as AOL and InMobi. He is currently developing CitoEngine, an open source alert management service written entirely in Python.

Wei Di is a research scientist at eBay Research Labs, focusing on advanced computer vision, data mining, and information retrieval technologies for large-scale e-commerce applications. Her interest covers large-scale data mining, machine learning in merchandising, data quality for e-commerce, search relevance, and ranking and recommender systems. She also has years of research experience in pattern recognition and image processing. She received her PhD from Purdue University in 2011 with focuses on data mining and image classification.

Michael Galloy works as a research mathematician for Tech-X Corporation involved in scientific visualizations using IDL and Python. Before that, he worked for five years teaching all levels of IDL programming and consulting for Research Systems, Inc. (now Exelis Visual Information Solutions). He is the author of Modern IDL (modernidl.idldev.com) and is the creator/maintainer