Python for Secret Agents

Table of Contents

Python for Secret Agents

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Our Espionage Toolkit

Getting the tools of the trade – Python 3.3

Windows secrets

Mac OS X secrets

Getting more tools – a text editor

Getting other developer tools

Getting a tool to get more Python components

Confirming our tools

How do we stop?

Using the help() system

Mac OS and GNU/Linux secrets

Windows secrets

Using the help mode

Background briefing – math and numbers

The usual culprits

The ivory tower of numbers

Integer numbers

Rational numbers

Floating-point numbers

Decimal numbers

Complex numbers

Outside the numbers

Assigning values to variables

Writing scripts and seeing output

Gathering user input

Handling exceptions

Looping and trying again

Handling text and strings

Converting between numbers and strings

Parsing strings

Organizing our software

Working with files and folders

Creating a file

Reading a file

Defining more complex logical conditions

Solving problems – recovering a lost password

Reading a word corpus

Reading a ZIP archive

Using brute-force search

Summary

2. Acquiring Intelligence Data

Accessing data from the Internet

Background briefing – the TCP/IP protocols

Using http.client for HTTP GET

Changing our client information

Using FTP in Python

Downloading a file via FTP

Using our FTP get() function

Using urllib for HTTP, FTP, or file access

Using urllib for FTP access

Using a REST API in Python

Getting simple REST data

Using more complex RESTful queries

Saving our data via JSON

Organizing collections of data

Using a Python list

Using list index operations

Using a Python tuple

Using generator expressions with list of tuples

Using a Python dictionary mapping

Using the dictionary access methods

Transforming sequences with generator functions

Using the defaultdict and counter mappings

Using a Python set

Using the for statement with a collection

Using Python operators on collections

Solving problems – currency conversion rates

Summary

3. Encoding Secret Messages with Steganography

Background briefing – handling file formats

Working with the OS filesystem

glob

os

Processing simple text files

Working with ZIP files

Working with JSON files

Working with CSV files

JPEG and PNG graphics – pixels and metadata

Using the Pillow library

Adding the required supporting libraries

GNU/Linux secrets

Mac OS X secrets

Windows secrets

Installing and confirming Pillow

Decoding and encoding image data

Manipulating images – resizing and thumbnails

Manipulating images – cropping

Manipulating images – enhancing

Manipulating images – filtering

Manipulating images – ImageOps

Some approaches to steganography

Getting the red-channel data

Extracting bytes from Unicode characters

Manipulating bits and bytes

Assembling the bits

Encoding the message

Decoding a message

Detecting and preventing tampering

Using hash totals to validate a file

Using a key with a digest

Solving problems – encrypting a message

Unpacking a message

Summary

4. Drops, Hideouts, Meetups, and Lairs

Background briefing – latitude, longitude, and GPS

Coping with GPS device limitations

Handling politics – borders, precincts, jurisdictions, and neighborhoods

Finding out where we are with geocoding services

Geocoding an address

Reverse geocoding a latitude-longitude point

How close? What direction?

Combining geocoding and haversine

Compressing data to make grid codes

Creating GeoRef codes

Decoding a GeoRef code

Creating Maidenhead grid codes

Decoding the Maidenhead grid codes

Creating natural area codes

Decoding natural area codes

Solving problems – closest good restaurant

Creating simple Python objects

Working with HTML web services – tools

Working with HTML web services – getting the page

Working with HTML web services – parsing a table

Making a simple Python object from columns of data

Enriching Python objects with geocodes

Enriching Python objects with health scores

Combining the pieces and parts

Working with clean data portals

Making a simple Python object from a JSON document

Combining different pieces and parts

Final steps

Understanding the data – schema and metadata

Summary

5. A Spymaster's More Sensitive Analyses

Creating statistical summaries

Parsing the raw data file

Finding an average value

Understanding generator expressions

Finding the value in the middle

Finding the most popular value

Creating Python modules and applications

Creating and using a module

Creating an application module

Creating a hybrid module

Creating our own classes of objects

Using a class definition

Comparisons and correlations

Computing the standard deviation

Computing a standardized score

Comparing a sequence and an iterable

Computing a coefficient of correlation

Writing high-quality software

Building a self-testing module and a test module

Creating more sophisticated tests

Adding doctest cases to a class definition

Solving problems – analyzing some interesting datasets

Getting some more data

Further research

Summary

Index

Python for Secret Agents

Python for Secret Agents

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: August 2014

Production reference: 1190814

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78398-042-0

www.packtpub.com

Cover image by Jarek Blaminsky (<[email protected]>)

Credits

Author

Steven F. Lott

Reviewers

Nick Glynn

Steven Heinrich

Sanjeev Jaiswal

Maurice HT Ling

Acquisition Editor

Sam Wood

Content Development Editor

Prachi Bisht

Technical Editors

Veena Pagare

Anand Singh

Copy Editors

Insiya Morbiwala

Sayanee Mukherjee

Deepa Nambiar

Karuna Narayanan

Adithi Shetty

Laxmi Subramanian

Project Coordinator

Sageer Parkar

Proofreaders

Ameesha Green

Samantha Lyon

Indexers

Hemangini Bari

Rekha Nair

Graphics

Sheetal Aute

Production Coordinator

Nilesh R. Mohite

Cover Work

Nilesh R. Mohite

About the Author

Steven F. Lott has been programming since the 70s, when computers were large, expensive, and rare. As a contract software developer and architect, he has worked on hundreds of projects, from very small to very large ones. He's been using Python to solve business problems for over 10 years.

He's particularly adept in struggling with gnarly data representation problems. He is also the author of Mastering Object-oriented Python, Packt Publishing.

He is currently a technomad who lives in various places on the East Coast of the U.S. His technology blog is located at http://slott-softwarearchitect.blogspot.com.

About the Reviewers

Nick Glynn is currently employed as a technical trainer and consultant delivering courses and expertise on Android, Python, and Linux at home in the UK and across the globe. He has a broad range of experience, from board bring-up, Linux driver development, and systems development to full stack deployments, web app development, and security hardening for both the Linux and Android platforms.

I would like to thank my family for their love and my beautiful baby girl, Inara, for always brightening my day.

Steven Heinrich is a full stack web developer who specializes in geospatial applications. He has a Bachelor's degree in Geographic Information Systems (GIS) and a Master's degree in GIS Management, both from Salisbury University. Steve likes to stay updated on new technologies and build a diverse portfolio.

Working on a wide array of projects allows him to work with a tech stack that best fits the unique needs of each project. This versatility and his desire to learn new technologies keeps Steve energized and motivated.

Besides enjoying spending time with his family, sports, and being a tech geek, he relaxes by sipping on one of the many homebrewed beers on tap in his garage.

Sanjeev Jaiswal is a computer graduate with 5 years of industrial experience. He basically uses Perl and GNU/Linux for his day-to-day work. He also teaches Drupal and WordPress CMS to bloggers. He has worked closely with top companies such as CSC, IBM, and Motorola Mobility. He has also developed an interest in web application penetration testing since 2013.

Sanjeev loves to impart his technical knowledge to engineering students and IT professionals and has been teaching since 2008. He founded Alien Coders (http://www.aliencoders.com/) based on the learning through sharing principle for computer science students and IT professionals in 2010 and has become a huge hit in India among engineering students.

He usually uploads technical videos on YouTube under the Alien Coders tag. He has a huge fan base on his site because of his simple yet effective way of teaching and his philanthropic nature towards students. You can follow him at http://www.facebook.com/aliencoders on Facebook and http://twitter.com/aliencoders on Twitter.

He has authored Instant PageSpeed Optimization, Packt Publishing, and is looking forward to author or review more books by Packt Publishing and other publishers.

I would like to thank my parents and my wife, Shalini Jaiswal, for their moral support in every phase of my life and growth.

I also give deep thanks and gratitude to my best friends, Ritesh Kamal and Ranjan Pandey, for their personal and professional help that is always available. It is because of them and some of my best friends, Rakesh Pandey, Rakesh Sharma, Abhishek Shukla, Anil Sharma, Santosh Vishwakarma, and others, that I have achieved a set of impossible goals in such a short time.

Maurice HT Ling (personal website: http://maurice.vodien.com) has completed his PhD in Bioinformatics and BSc (Hons) in Molecular and Cell Biology from The University of Melbourne. He is currently a research fellow at Nanyang Technological University, Singapore, and an honorary fellow at The University of Melbourne, Australia. Maurice is the chief editor of the journal, Computational and Mathematical Biology, iConcept Press Ltd.; the co-editor of the e-journal, The Python Papers; and the co-founder of the Python User Group (Singapore).

Recently, Maurice co-founded the first synthetic biology start-up in Singapore, AdvanceSyn Pte. Ltd., as Director and Chief Technology Officer. His research interests lie in life—biological life, artificial life, and artificial intelligence—using computer science and statistics as tools to understand life and its numerous aspects. In his free time, Maurice likes to read, enjoy a cup of coffee, write his personal journal, or philosophize on the various aspects of life.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why Subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

Every secret agent needs a good set of tools and gadgets. When an agent's missions involve gathering data, high-powered data processing is required. This book will provide you with the kinds of information processing tools that will help you gather, analyze, and communicate the data that the HQ demands.

Python allows agents to write simple scripts to gather data, perform sophisticated calculations, and produce useful results. Agents can also use Python to extract data from local files, HTTP web servers, and FTP file servers.

Python has numerous add-on packages. This book will explore just two: Pillow allows for sophisticated image conversion and manipulation, and BeautifulSoup allows an agent to extract data from HTML web pages. Agents with specific needs might have to explore the Python Natural Language Toolkit (NLTK), Numeric Python (NumPy), or even Scientific Python (SciPy).

What this book covers

Chapter 1, Our Espionage Toolkit, exposes the basics of installing and using Python. We'll write scripts to help agents work with foreign currency conversions and learn how an agent can recover a lost password from a ZIP archive.

Chapter 2, Acquiring Intelligence Data, shows how we can use Python to extract information from various kinds of file servers. Agents will learn how to work with different Internet protocols and use Representational State Transfer (REST) to interact with web services. This will include techniques to work with cyptocurrencies, such as bitcoins.

Chapter 3, Encoding Secret Messages with Steganography, shows how we can add the Pillow toolset to work with images. An agent with Pillow can create thumbnails and convert, crop, and enhance images. We'll also explore some steganography algorithms to bury our messages in an image file.

Chapter 4, Drops, Hideouts, Meetups, and Lairs, provides a closer look at geocoding and geolocation. This includes the use of web services to convert an address into a latitude and longitude. We'll also learn how to convert a latitude and longitude back into an address. We'll take a look at the haversine formula to get the correct distance between locations. We'll also look at some of the ways in which geographic locations can be represented for tidy storage and communication.

Chapter 5, A Spymaster's More Sensitive Analyses, shows how we can use Python for a basic data analysis. A good agent doesn't just spout facts and figures; a good agent does enough analysis to confirm that the data is realistic. Being able to examine the correlation between datasets is what creates a valuable intelligence asset.

What you need for this book

A secret agent needs a computer that they have administrative privileges over. We'll be installing additional software. Without the administrative password, they may have trouble installing Python 3, Pillow, or BeautifulSoup.

For agents using Windows, the packages we're looking at adding come prebuilt.

For agents using Linux, the developer tools are required. Linux has a complete suite of developer tools that are common. The Gnu C Compiler (GCC) is the backbone of these tools.

For agents using Mac OS X, the developer tool, Xcode (https://developer.apple.com/xcode/), is required. We'll also need to install a tool called homebrew (http://brew.sh) to help us add the Linux packages to Mac OS X.

Python 3 is available from the Python download page at https://www.python.org/download.

We'll download and install several things besides Python 3.3:

The setuptools package, which includes easy_install-3.3, will help us add packages. It can be downloaded from https://pypi.python.org/pypi/setuptools.

The PIP package will also help us install additional packages. Some experienced field agents prefer PIP over setuptools. It can be downloaded from https://pypi.python.org/pypi/pip/1.5.6.

The Pillow package will allow us to work with image files. It can be downloaded from https://pypi.python.org/pypi/Pillow/2.4.0.

The BeautifulSoup Version 4 package will allow us to work with HTML web pages. It can be downloaded from https://pypi.python.org/pypi/beautifulsoup4/4.3.2.

From this, we'll see how extensible Python is. Almost anything an agent might need may already be written and available through the Python Package Index (PyPi), which can be downloaded from https://pypi.python.org/pypi.

Who this book is for

This book is for secret agents who don't know much Python, but are comfortable installing new software and are ready to do some clever programming in Python. An agent who has never done any programming before might find some of this a bit advanced; a beginner's tutorial that covers the basics of Python may be helpful.

We'll expect that an agent using this book is comfortable with simple math. This involves the multiplication and division of currency conversions. It also includes polynomials, simple trigonometry, and a few statistical formulae.

We also expect the secret agents using this book to be doing their own investigations. The book's examples are designed to get the agent started down the road with developing interesting, useful applications. Each agent will have to explore further afield on their own.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text, package names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: The size_list variable is a sequence of eight-tuples built from the bytes of the encoded size.

A block of code is set as follows:

message_bytes= message.encode(UTF-8)

bits_list = list(to_bits(c) for c in message_bytes )

len_h, len_l = divmod( len(message_bytes), 256 )

size_list = [to_bits(len_h), to_bits(len_l)]

bit_sequence( size_list+bits_list )

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

w, h = ship.size

for p,m in enumerate( bit_sequence(size_list+bits_list) ):

    y, x = divmod( p, w )

    r, g, b = ship.getpixel( (x,y) )

r_new = (r & 0xfe) | m

    print( (r, g, b), m, (r_new, g, b) )

    ship.putpixel( (x,y), (r_new, g, b)  )

Any command-line input or output is written as follows:

$ python3.3 -m doctest ourfile.py

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: There is an Advanced Settings panel that defines these file associations.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.