Starting a package

D E V E L O P I N G P Y T H O N PA C K A G E S

James Fulton
Climate informatics researcher
Why build a package anyway?
To make your code easier to reuse.
To avoid lots of copying and pasting.

To keep your functions up to date.

To give your code to others.

DEVELOPING PYTHON PACKAGES

Course content
You will build a full package, and cover:

File layout

Import structure
Making your package installable

Adding licenses and READMEs

Style and unit tests for a high quality package

Registering and publishing your package to PyPI

Using package templates

DEVELOPING PYTHON PACKAGES

Scripts, modules, and packages
Script - A Python file which is run like python myscript.py .
Package - A directory full of Python code to be imported
e.g. numpy .

Subpackage - A smaller package inside a package

e.g. numpy.random and numpy.linalg .

Module - A Python file inside a package which stores the package code.
e.g. example coming in next 2 slide.

Library - Either a package, or a collection of packages.

e.g., the Python standard library ( math , os , datetime ,...)

DEVELOPING PYTHON PACKAGES

Directory tree of a package
Directory tree for simple package

mysimplepackage/
|-- simplemodule.py
|-- __init__.py

This directory, called mysimplepackage , is a Python Package

simplemodule.py contains all the package code

__init__.py marks this directory as a Python package

DEVELOPING PYTHON PACKAGES

Contents of simple package
__init__.py simplemodule.py

def cool_function():
...
Empty file return cool_result

...

def another_cool_function():
...
return another_cool_result

File with generalized functions and code.

DEVELOPING PYTHON PACKAGES

Subpackages
Directory tree for package with subpackages

mysklearn/
|-- __init__.py
|-- preprocessing
| |-- __init__.py
| |-- normalize.py
| |-- standardize.py
|-- regression
| |-- __init__.py
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

 Let's practice!
D E V E L O P I N G P Y T H O N PA C K A G E S
 Documentation
D E V E L O P I N G P Y T H O N PA C K A G E S

James Fulton
Climate informatics researcher
Why include documentation?
Helps your users use your code import numpy as np
help(np.sum)
Document each
Function ...
sum(a, axis=None, dtype=None, out=None)
Class Sum of array elements over a given axis.

Class method Parameters

----------
a : array_like
Elements to sum.
axis : None or int or tuple of ints, optional
Axis or axes along which a sum is performed.
The default, axis=None, will sum all of the
elements of the input array.
...

DEVELOPING PYTHON PACKAGES

Why include documentation?
Helps your users use your code import numpy as np
help(np.array)
Document each
Function ...
array(object, dtype=None, copy=True)
Class
Create an array.
Class method
Parameters
----------
object : array_like
An array, any object exposing the array
interface ...
dtype : data-type, optional
The desired data-type for the array.
copy : bool, optional
If true (default), then the object is copied.
...

DEVELOPING PYTHON PACKAGES

Why include documentation?
Helps your users use your code import numpy as np
x = np.array([1,2,3,4])
Document each help(x.mean)

Function
...
Class mean(...) method of numpy.ndarray instance
a.mean(axis=None, dtype=None, out=None)
Class method
Returns the average of the array elements
along given axis.

Refer to `numpy.mean` for full documentation.

...

DEVELOPING PYTHON PACKAGES

Function documentation
def count_words(filepath, words_list):
""" ... """

DEVELOPING PYTHON PACKAGES

Function documentation
def count_words(filepath, words_list):
"""Count the total number of times these words appear."""

DEVELOPING PYTHON PACKAGES

Function documentation
def count_words(filepath, words_list):
"""Count the total number of times these words appear.

The count is performed on a text file at the given location.

"""

DEVELOPING PYTHON PACKAGES

Function documentation
def count_words(filepath, words_list):
"""Count the total number of times these words appear.

The count is performed on a text file at the given location.

[explain what filepath and words_list are]

[what is returned]
"""

DEVELOPING PYTHON PACKAGES

Documentation style
Google documentation style NumPy style

"""Summary line. """Summary line.

Extended description of function. Extended description of function.

Args: Parameters
arg1 (int): Description of arg1 ----------
arg2 (str): Description of arg2 arg1 : int
Description of arg1 ...

reStructured text style Epytext style

"""Summary line. """Summary line.

Extended description of function. Extended description of function.

:param arg1: Description of arg1 @type arg1: int

:type arg1: int @param arg1: Description of arg1
:param arg2: Description of arg2 @type arg2: str
:type arg2: str @param arg2: Description of arg2

DEVELOPING PYTHON PACKAGES

NumPy documentation style
Popular in scientific Python packages like

numpy

scipy

pandas

sklearn

matplotlib

dask

etc.

DEVELOPING PYTHON PACKAGES

NumPy documentation style
import scipy
help(scipy.percentile)

percentile(a, q, axis=None, out=None, overwrite_input=False, interpolation='linear')

Compute the q-th percentile of the data along the specified axis.

Returns the q-th percentile(s) of the array elements.

Parameters
----------
a : array_like
Input array or object that can be converted to an array.

Other types include - int , float , bool , str , dict , numpy.array , etc.

DEVELOPING PYTHON PACKAGES

NumPy documentation style
import scipy
help(scipy.percentile)

percentile(a, q, axis=None, out=None, overwrite_input=False, interpolation='linear')

...
Parameters
----------
...
axis : {int, tuple of int, None}
...
interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}

List multiple types for parameter if appropriate

List accepted values if only a few valid options

DEVELOPING PYTHON PACKAGES

NumPy documentation style
import scipy
help(scipy.percentile)

percentile(a, q, axis=None, out=None, overwrite_input=False, interpolation='linear')

...
Returns
-------
percentile : scalar or ndarray
If `q` is a single percentile and `axis=None`, then the result
is a scalar. If multiple percentiles are given, first axis of
the result corresponds to the percentiles...
...

DEVELOPING PYTHON PACKAGES

 NumPy documentation style
Other sections

Raises

See Also

Notes

References

Examples

1 https://numpydoc.readthedocs.io/en/latest/format.html

DEVELOPING PYTHON PACKAGES

Documentation templates and style translation
pyment can be used to generate docstrings

Run from terminal

Any documentation style from

Google

Numpydoc

reST (i.e. reStructured-text)

Javadoc (i.e. epytext)

Modify documentation from one style to another

DEVELOPING PYTHON PACKAGES

Documentation templates and style translation
pyment -w -o numpydoc textanalysis.py

def count_words(filepath, words_list):

# Open the text file
...
return n

-w - overwrite file

-o numpydoc - output in NumPy style

DEVELOPING PYTHON PACKAGES

Documentation templates and style translation
pyment -w -o numpydoc textanalysis.py

def count_words(filepath, words_list):

"""

Parameters
----------
filepath :

words_list :

Returns
-------
type
"""

DEVELOPING PYTHON PACKAGES

Translate to Google style
pyment -w -o google textanalysis.py

def count_words(filepath, words_list):

"""Count the total number of times these words appear.

The count is performed on a text file at the given location.

Parameters
----------
filepath : str
Path to text file.
words_list : list of str
Count the total number of appearances of these words.

Returns
-------

DEVELOPING PYTHON PACKAGES

Translate to Google style
pyment -w -o google textanalysis.py

def count_words(filepath, words_list):

"""Count the total number of times these words appear.

The count is performed on a text file at the given location.

Args:
filepath(str): Path to text file.
words_list(list of str): Count the total number of appearances of these words.

Returns:

"""

DEVELOPING PYTHON PACKAGES

Package, subpackage and module documentation
mysklearn/__init__.py mysklearn/preprocessing/__init__.py

""" """
Linear regression for Python A subpackage for standard preprocessing operations.
============================ """

mysklearn is a complete package for implmenting

linear regression in python.

mysklearn/preprocessing/normalize.py

"""
A module for normalizing data.
"""

DEVELOPING PYTHON PACKAGES

 Let's practice!
D E V E L O P I N G P Y T H O N PA C K A G E S
 Structuring imports
D E V E L O P I N G P Y T H O N PA C K A G E S

James Fulton
Climate informatics researcher
Without package imports
import mysklearn Directory tree for package with subpackages

mysklearn/
help(mysklearn.preprocessing)
|-- __init__.py
|-- preprocessing
Traceback (most recent call last):
| |-- __init__.py
File "<stdin>", line 1, in <module>
| |-- normalize.py
AttributeError: module 'mysklearn' has no
| |-- standardize.py
attribute 'preprocessing'
|-- regression
| |-- __init__.py
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Without package imports
import mysklearn.preprocessing Directory tree for package with subpackages

mysklearn/
help(mysklearn.preprocessing)
|-- __init__.py
|-- preprocessing
Help on package mysklearn.preprocessing in
| |-- __init__.py
mysklearn:
| |-- normalize.py
| |-- standardize.py
NAME
|-- regression
mysklearn.preprocessing - A subpackage
| |-- __init__.py
for standard preprocessing operations.
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Without package imports
import mysklearn.preprocessing Directory tree for package with subpackages

mysklearn/
help(mysklearn.preprocessing.normalize)
|-- __init__.py
|-- preprocessing
Traceback (most recent call last):
| |-- __init__.py
File "<stdin>", line 1, in <module>
| |-- normalize.py
AttributeError: module
| |-- standardize.py
'mysklearn.preprocessing' has no attribute
|-- regression
'normalize'
| |-- __init__.py
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Without package imports
import mysklearn.preprocessing.normalize Directory tree for package with subpackages

mysklearn/
help(mysklearn.preprocessing.normalize)
|-- __init__.py
|-- preprocessing
Help on module mysklearn.preprocessing.normalize
| |-- __init__.py
in mysklearn.preprocessing:
| |-- normalize.py
| |-- standardize.py
NAME
|-- regression
mysklearn.preprocessing.normalize - A module
| |-- __init__.py
for normalizing data.
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Importing subpackages into packages
mysklearn/__init__.py Directory tree for package with subpackages

Absolute import mysklearn/

|-- __init__.py <--
from mysklearn import preprocessing |-- preprocessing
| |-- __init__.py
| |-- normalize.py
Used most - more explicit
| |-- standardize.py
Relative import |-- regression
| |-- __init__.py
| |-- regression.py
from . import preprocessing
|-- utils.py

Used sometimes - shorter and sometimes

simpler

DEVELOPING PYTHON PACKAGES

Importing modules
We imported preprocessing into mysklearn But preprocessing has no link to normalize

import mysklearn import mysklearn

help(mysklearn.preprocessing) help(mysklearn.preprocessing.normalize)

Help on package mysklearn.preprocessing in Traceback (most recent call last):

mysklearn: File "<stdin>", line 1, in <module>
AttributeError: module
NAME 'mysklearn.preprocessing' has no attribute
mysklearn.preprocessing - A subpackage 'normalize'
for standard preprocessing operations.

DEVELOPING PYTHON PACKAGES

Importing modules
mysklearn/preprocessing/__init__.py Directory tree for package with subpackages

Absolute import mysklearn/

|-- __init__.py
from mysklearn.preprocessing import normalize |-- preprocessing
| |-- __init__.py <--
| |-- normalize.py
Relative import
| |-- standardize.py
|-- regression
from . import normalize
| |-- __init__.py
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Restructuring imports
import mysklearn

help(mysklearn.preprocessing.normalize.normalize_data)

Help on function normalize_data in module

mysklearn.preprocessing.normalize:

normalize_data(x)
Normalize the data array.

DEVELOPING PYTHON PACKAGES

Import function into subpackage
mysklearn/preprocessing/__init__.py Directory tree for package with subpackages

Absolute import mysklearn/

|-- __init__.py
from mysklearn.preprocessing.normalize import \ |-- preprocessing
normalize_data | |-- __init__.py <--
| |-- normalize.py

Relative import | |-- standardize.py

|-- regression
| |-- __init__.py
from .normalize import normalize_data
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Import function into subpackage
import mysklearn

help(mysklearn.preprocessing.normalize_data)

Help on function normalize_data in module

mysklearn_imp.preprocessing.normalize:

normalize_data(x)
Normalize the data array.

DEVELOPING PYTHON PACKAGES

Importing between sibling modules
In normalize.py Directory tree for package with subpackages

Absolute import mysklearn/

|-- __init__.py
from mysklearn.preprocessing.funcs import ( |-- preprocessing
mymax, mymin | |-- __init__.py
) | |-- normalize.py <--
| |-- funcs.py

Relative import | |-- standardize.py

|-- regression
| |-- __init__.py
from .funcs import mymax, mymin
| |-- regression.py
|-- utils.py

DEVELOPING PYTHON PACKAGES

Importing between modules far apart
A custom exception MyException is in Directory tree for package with subpackages
utils.py
mysklearn/
In normalize.py , standardize.py and |-- __init__.py
|-- preprocessing
regression.py
| |-- __init__.py
| |-- normalize.py <--
Absolute import
| `-- standardize.py <--
|-- regression
from mysklearn.utils import MyException | |-- __init__.py
| |-- regression.py <--
Relative import `-- utils.py

from ..utils import MyException

DEVELOPING PYTHON PACKAGES

Relative import cheat sheet
from . import module
From current directory, import module

from .. import module

From one directory up, import module

from .module import function

From module in current directory, import function

from ..subpackage.module import function

From subpackage one directory up, from module in that subpackage, import function

DEVELOPING PYTHON PACKAGES

 Let's practice!
D E V E L O P I N G P Y T H O N PA C K A G E S