PART IV

Modules and CHAPTER 22 Modules: The Big Picture

Packages CHAPTER 23 Module Coding Basics

CHAPTER 24 Module Packages
CHAPTER 25 Advanced Module Topics

1
CHAPTER 22 Modules: The Big Picture

2
CHAPTER 22 Modules: The Big Picture
Python module:
● pack program code and data for reuse, each file is a module
● the highest-level program organization unit provides self-contained
namespaces that minimize variable name clashes

a.py
var_1= 10

b.py
var_1=10
import a
print(a.var_1)

3
CHAPTER 22 Python Program Architecture
A Python program consists of:
● 1 main top-level file
○ contains the main flow of control of your program (e.g., main function)
● n module files (n >= 0):
○ components used by the top-level file or other modules.

4
CHAPTER 22 Python Program Architecture
Imports and Attributes
# File b.py
def spam(text):
print(text, 'spam')

# File a.py
import b
main():
b.spam('gumby')

5
CHAPTER 22 Python Program Architecture
Standard Library Modules
● Python comes with a large collection of utility modules known as the standard
library
● Link: www.python.org

6
CHAPTER 22 Python Program Architecture
How Imports Work:
They are runtime operations that perform three steps the first time a program
imports a given file:
1. Find the module’s file. # (b.*: b.py b.pyc )
2. Compile it to byte code (if needed). #(b.py -> b.pyc; b.pyc: code b.py, version
python; b.pyc)
a. Compile: If the byte code file is older than the source file (i.e., if you’ve changed the source) or
was created by a different Python version
b. Don’t compile
3. Run the module’s code to build the objects it defines.

7
CHAPTER 22 Python Program Architecture
The Module Search Path
1. The home directory of the program: automatic, searched first, its files will
override modules of the same name in other directories;
2. PYTHONPATH directories (if set): configurable
a. used to extend the path to include user-defined and platform-specific source code directories,
b. import a file that is stored in a different directory from the file that imports it
3. Standard library directories: automatic.
4. The contents of any *.pth files (if present): configurable,
a. used to extend the path to include user-defined directories, e.g., myconfig.pth in Python install
directory (C:\Python33, C:\Python33\Lib\site-packages, etc.)
5. The site-packages home of third-party extensions: automatic

The concatenation of these five components becomes sys.path

8
CHAPTER 22 Python Program Architecture
The Module Search Path - Configuring the Search Path (e.g., Windows)

● PYTHONPATH:
c:\pycode\utilities;d:\pycode\package1

● C:\Python33\pydirs.pth
or C:\Python33\Lib\site-packages\pydirs.pth:
c:\pycode\utilities
d:\pycode\package1

9
CHAPTER 22 Python Program Architecture
Check sys.path
● Verify the search path settings
>>> import sys
>>> sys.path
● Modify the path only for current run
>>> sys.path.insert(0,'/path/to/mod_directory')
● PYTHONPATH and .pth files offer more permanent ways to modify the path

Module File Selection

● Python chooses the first file it can find on the search path that matches the
imported name.

10
CHAPTER 23 Module Coding Basics

11
CHAPTER 23 Module Coding Basics

● Module names become variable names in a program => should follow the normal variable
name rules
○ E.g.,
# if.py
X = 10
print(“if.py”)
# test.py
import if
print(if.X)
if true:
….
● Extension modules:
○ module by writing code in an external language such as C, C++, and others
○ like normal modules: accessed with import statements, and their functions and objects as module attributes.

12
CHAPTER 23 Module Coding Basics
Module Usage
● The import Statement
>>> import module1 # Get module as a whole (one or more)
>>> module1.printer('Hello world!') # Qualify to get names
Hello world!

● The from Statement

>>> from module1 import printer # Copy out a variable (one or more)
>>> printer('Hello world!') # No need to qualify name
Hello world!

● The from * Statement

>>> from module1 import * # Copy out _all_ variables
>>> printer('Hello world!')
Hello world!

13
CHAPTER 23 Module Coding Basics
Imports Happen Only Once
● By default Python imports just once per file, per process.
● For example:
# simple.py
print('hello')
spam = 1 # Initialize variable

# python
>>> import simple # First import: loads and runs file's code hello
>>> simple.spam # Assignment makes an attribute
1

>>> simple.spam = 2 # Change attribute in module

>>> import simple # Just fetches already loaded module
>>> simple.spam # Code wasn't rerun: attribute unchanged
2
14
CHAPTER 23 Module Coding Basics
import and from Are Assignments # small.py:
x = 1 # immutable (bat bien)
● executable statements
y = [1, 2] # mutable
● implicit assignments:
○ from assigns one or more names # python
to objects of the same names in >>> from small import x, y
another module >>> x = 42 # Changes local x only
○ import assigns an entire module >>> y[0] = 42 # Changes shared
object to a single name. mutable in place

>>> import small

>>> small.x
1
>>> small.y
[42, 2]

15
CHAPTER 23 Module Coding Basics
import and from Are Assignments - # small.py:
Cross-file name changes x = 1 # immutable (bat bien)
y = [1, 2] # mutable
● Copy two names out
● Changes my x only % python
>>> from small import x, y
● Get module name >>> x = 42
● Changes x in other module
>>> import small
>>> small.x = 42

16
CHAPTER 23 Module Coding Basics
import and from Equivalence

● Copy these two names out (only) from module import name1, name2

● Fetch the module object import module

● Copy names out by assignment name1 = module.name1
● Get rid of the module name name2 = module.name2
del module

17
CHAPTER 23 # M.py
def func():
Module Coding Basics ...do something...
# N.py
Potential Pitfalls of the from Statement def func():
...do something else…
This overwrites the one we fetched from M # O.py
Calls N.func only! from M import func
Solutions from N import func
func() # N.func()

# O.py
import M, N
M.func()
N.func()

# O.py
from M import func as mfunc
from N import func as nfunc
mfunc(); nfunc()
18
CHAPTER 23 Module Coding Basics
Module Namespaces - Files Generate Namespaces

● Module statements run on the first import.

● Top‐level assignments create module attributes.
● Module namespaces can be accessed via the attribute __dict__ or dir(M)
● Modules are a single scope (local is global)

19
CHAPTER 23 Module Coding Basics
Illustrate >>> import module2
● Top‐level assignments create starting to load...
module attributes done loading.
>>> module2.sys
% module2.py
<module 'sys' (built-in)>
print('starting to load...')
import sys >>> module2.name
name = 42 42
def func(): >>> module2.func
pass <function func at
class klass: 0x000000000222E7B8>
pass >>> module2.klass
print('done loading.') <class 'module2.klass'>

20
CHAPTER 23 Module Coding Basics
Namespace Dictionaries: __dict__ >>> list(module2.__dict__.keys())
● Module namespaces are stored ['__loader__', 'func', 'klass',
as dictionary objects '__builtins__', '__doc__',
● access via the module’s '__file__', '__name__', 'name',
__dict__ attribute '__package__', 'sys',
'__initializing__', '__cached__']

>>> dir(module2)

21
CHAPTER 23 Module Coding Basics
Imports Versus Scopes % moda.py
● Module code can never see X = 88
def f():
names in other modules, global X
unless they are explicitly X = 99
imported.
% modb.py
X = 11
import moda
moda.f()
print(X, moda.X)

>>> python modb.py

11 99

22
CHAPTER 23 Module Coding Basics
Namespace Nesting % mod1.py
X= 1
% mod3.py import mod2
X= 3 print(X, end=' ')
print(mod2.X, end=' ')
% mod2.py print(mod2.mod3.X)
X= 2
import mod3 >>> python mod1.py
print(X, end=' ')
print(mod3.X)

23
CHAPTER 23 Module Coding Basics
Reloading Modules - reload Basics import module
● reload is a function in Python ...change module.attributes
● reload is passed an existing module ...
object, not a new name. from imp import reload
● reload lives in a module in Python 3.X reload(module)
and must be imported itself. ...use module.attributes
○ Initial import …
○ Now, go change the module file …
○ Get reload itself (in 3.X)
○ Get updated exports

24
CHAPTER 23 Module Coding Basics
Reloading Modules - reload basic
● Reload runs a module file’s new code in the module’s current namespace.
● Top‐level assignments in the file replace names with new values.
● Reloads impact all clients that use import to fetch modules.
● Reloads impact future from clients only.
● Reloads apply to a single module only

25
CHAPTER 23 Module Coding Basics
% changer.py … back to the Python
message = "First version" interpreter …
def printer():
print(message)
>>> import changer
% python >>> changer.printer()
>>> import changer First version
>>> changer.printer()
First version >>> from imp import reload
>>> reload(changer)
...modify changer.py without stopping
<module 'changer' from
Python...
% changer.py as below:
'.\\changer.py'>
message = "After editing"
def printer(): >>> changer.printer()
print('reloaded:', message) reloaded: After editing

26
CHAPTER 24 Module Packages

27
Package Import Basics
● Imports are known as package imports
● The “dotted” path in the import statement is a relative path to the file mod.py

E.g.,
import dir1.dir2.mod

Directory hierarchy: dir1\dir2\mod.py

dir1\
dir2\
mod.py

28
Package __init__.py Files
● Each directory named within the path of a package import statement must
contain a file named __init__.py

E.g.,
import dir1.dir2.mod

Directory hierarchy: dir1\dir2\mod.py

dir1\
__init__.py
dir2\
__init__.py
mod.py
29
Package initialization file roles
__init__.py: a hook for package initialization-time actions
● automatically runs all the code in the __init__.py file when a package is first
accessed
● declares a directory as a package
● generates a module namespace for a directory
● from … import * statement behavior

30
Package Import Example
# dir1\__init__.py # First imports run init files
print('dir1 init') >>> import dir1.dir2.mod
x= 1 dir1 init
dir2 init
# dir1\dir2\__init__.py in mod.py
print('dir2 init')
y= 2 # Later imports do not run init
>>> import dir1.dir2.mod
# dir1\dir2\mod.py …
print('in mod.py')
z= 3

31
Package Import Example
An imported directory may be passed to >>> from imp import reload
reload to force re-execution of that
single item. >>> reload(dir1)
dir1 init
<module 'dir1' from
'.\\dir1\\__init__.py'>

>>> reload(dir1.dir2)
dir2 init
<module 'dir1.dir2' from
'.\\dir1\\dir2\\__init__.py'

32
Package Import Example
from Versus import with Packages >>> from dir1.dir2 import mod
dir1 init
dir2 init
● the from statement with in mod.py
>>> mod.z
packages avoids retyping the
3
paths at each access.
>>> from dir1.dir2.mod import z
>>> z
3

>>> import dir1.dir2.mod as mod

>>> mod.z
3
>>> from dir1.dir2.mod import z as modz
>>> modz
3
33
Why Use Package Imports?
● serve as an organizational tool:
○ If we organize files into subdirectories for functional areas, package imports make it more
obvious what role a module plays
● simplify your module search path: use explicit package imports for all your
cross-directory imports,
○ make those package imports relative to a common root directory
○ put the common root in PYTHONPATH and .pth file search path
● resolve ambiguities:
○ make explicit exactly which files we want to import

34
Why Use Package Imports?
system1\ root\
utilities.py system1\
main.py __init__.py
other.py utilities.py
main.py
other.py
system2\
system2\
utilities.py __init__.py
main.py utilities.py
other.py main.py
other.py
# myfile.py
import utilities system3\
utilities.function('spam') __init__.py
myfile.py

35
Why Use Package Imports?
root\ # myfile.py
system1\ import system1.utilities
__init__.py
utilities.py import system2.utilities
main.py system1.utilities.function('spam')
other.py system2.utilities.function('eggs')
system2\
__init__.py
utilities.py
main.py
other.py

system3\
__init__.py
myfile.py

36
Package Relative Imports
● Imports with dots: search for mypkg\
modules inside the package __init__.py
directory only main.py
● Imports without dots: look string.py
elsewhere on the sys.path
search path
from . import string
import string

37
Package Relative Imports
Code located in some module A.B.C can use any of these forms:
A/B/C test.py

from . import D # A.B.D

from .. import E # A.E

from .D import X # A.B.D.X

from ..E import X # A.E.X

38
Package Relative Imports
The Scope of Relative Imports:
● Relative imports apply to imports within packages only.
● Relative imports apply to the from statement only.

39
Package Relative Imports
Relative Imports in Action # C:\code>
● imports outside packages: >>> import string
>>> string
normal imports are still relative <module 'string' from
to the “home” directory 'C:\\Python33\\lib\\string.py'>
● package relative import syntax
# C:/code/string.py
is not even allowed in code that print('string' * 5)
is not in a file:
>>> import string
stringstringstringstringstring
>>> string
<module 'string' from '.\\string.py'>

>>> from . import string

SystemError: Parent module '' not
loaded, cannot perform relative import 40
Package Relative Imports
Relative Imports in Action

# C:\code\string.py
print('string' * 8)

# C:\code\pkg\spam.py
import eggs
print(eggs.X)

# C:\code\pkg\eggs.py
X = 99999
import string
print(string)

# C:\code>
>>> import pkg.spam
41
Package Relative Imports
Relative Imports in Action

# code\string.py
print('string' * 8)

# code\pkg\spam.py
from . import eggs
print(eggs.X)

# code\pkg\eggs.py
X = 99999
import string
print(string)

# C:\code>
>>> import pkg.spam
42
Package Relative Imports
Relative Imports in Action

# code\string.py
print('string' * 8)

# code\pkg\spam.py
from . import eggs
print(eggs.X)

# code\pkg\eggs.py
X = 99999
from . import string
print(string)

# C:\code>
>>> import pkg.spam
43
Package Relative Imports
Namespace Packages:
● Known as the virtual concatenation of individual directory components,
● Allows nested parts to be accessed through its name with normal imports

44
Package Relative Imports >>> sub.__path__
_NamespacePath(['C:\\code\\ns\\dir1\\
# C:\code\ns\dir1\sub\mod1.py sub', 'C:\\code\\ns\\dir2\\sub'])
print(r'dir1\sub\mod1')
>>> from sub import mod1
# C:\code\ns\dir2\sub\mod2.py dir1\sub\mod1
print(r'dir2\sub\mod2')
>>> import sub.mod2
set PYTHONPATH=C:\code\ns\dir1; dir2\sub\mod2
C:\code\ns\dir2
>>> mod1
>>> import sub <module 'sub.mod1' from
>>> sub 'C:\\code\\ns\\dir1\\sub\\mod1.py'>
<module 'sub' (namespace)>
>>> sub.mod2
<module 'sub.mod2' from
'C:\\code\\ns\\dir2\\sub\\mod2.py'>
45
Package Relative Imports
Namespace Package Nesting # ns\dir2\sub\lower\mod3.py
print(r'dir2\sub\lower\mod3')

>>> import sub.lower.mod3

dir2\sub\lower\mod3

>>> import sub

>>> import sub.mod2
dir2\sub\mod2
>>> import sub.lower.mod3
dir2\sub\lower\mod3
>>> sub.lower
46
CHAPTER 25 Advanced Module Topics

47
 Module Design Concepts
● You’re always in a module in Python.
● Minimize module coupling: global variables.
● Maximize module cohesion: unified purpose.
● Modules should rarely change other
modules’ variables.

Figure 25-1. Module execution environment.

48
Data Hiding in Modules
● Data hiding in modules is a # unders.py
convention, not a syntactical a, _b, c, _d = 1, 2, 3, 4
constraint >>> from unders import *
● Minimizing from * Damage: _X >>> a, c
and __all__; doesn’t work with (1, 3)
import statement >>> _b
○ _X identifies names not to be NameError: name '_b' is not defined
copied with a from * statement.
>>> import unders
>>> unders._b
2

49
Data Hiding in Modules # alls.py
__all__ = ['a', '_c']
a, b, _c, _d = 1, 2, 3, 4
● Minimizing from * Damage: _X
and __all__ >>> from alls import *
○ __all__ identifies names to be >>> a, _c
copied with a from * statement. (1, 3)
>>> b
NameError: name 'b' is not defined

>>> from alls import a, b, _c, _d

>>> a, b, _c, _d
(1, 2, 3, 4)

>>> import alls

>>> alls.a, alls.b, alls._c, alls._d
(1, 2, 3, 4)
50
Mixed Usage Modes: __name__ and __main__
● Import a file as a module and run it as a standalone program
○ If the file is being run as a top-level program file, __name__ is set to the string "__main__"
when it starts.
○ If the file is being imported instead, __name__ is set to the module’s name as known by its
clients.

51
Unit Tests with __name__
# minmax.py
print('I am:', __name__)

def minmax(test, *args):

res = args[0]
for arg in args[1:]:
if test(arg, res):
res = arg
return res

def lessthan(x, y): return x < y

def grtrthan(x, y): return x > y

print(minmax(lessthan, 4, 2, 1, 5, 6, 3))
print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))

# test_minmax.py
import minmax
print("test minmax")

52
Unit Tests with __name__
#minmax2.py
print('I am:', __name__)

def minmax(test, *args):

res = args[0]
for arg in args[1:]:
if test(arg, res):
res = arg
return res

def lessthan(x, y): return x < y

def grtrthan(x, y): return x > y

if __name__ == '__main__':
print(minmax(lessthan, 4, 2, 1, 5, 6, 3))
print(minmax(grtrthan, 4, 2, 1, 5, 6, 3))

# test_minmax.py
import minmax
print("test minmax")

53
The as Extension for import and from
import reallylongmodulename as name
name.func()

from module1 import utility as util1

from module2 import utility as util2
util1(); util2()

54
Importing Modules by Name String
● The program get the name of a >>> import 'string'
module as a string at runtime File "<stdin>", line 1 import
(e.g. from a user selection in a "string"
GUI, or a parse of an XML
document, for instance) # Run a string of code
○ Running Code Strings: exec, it >>> modname = 'string'
must compile the import statement
>>> exec('import ' + modname)
each time it runs, slowly
>>> string
<module 'string' from …>

55
 Importing Modules by Name String
● The program get the name of a >>> import 'string'
module as a string at runtime (e.g. File "<stdin>", line 1 import "string"
from a user selection in a GUI, or a
parse of an XML document) >>> modname = 'string'
○ Running Code Strings (exec) >>> string = __import__(modname)
○ Direct Calls: Two Options >>> string
■ __import__
■ importlib.import_module(...) >>> import importlib
>>> modname = 'string'
>>> string =
importlib.import_module(modname)
>>> string

56
Module Gotchas
● Statement Order Matters in Top-Level Code func1()
○ Code at the top level of a module file (not nested in a
function) runs when Python reaches it during an import def func1():
=> cannot reference names assigned lower in the file print(func2())
○ Code inside a function body doesn’t run until the
function is called => reference names anywhere in the
file func1()

def func2():
return "Hello"

func1()

57
Module Gotchas
● from Copies Names but Doesn’t Link # nested3.py
import nested1
# nested1.py
nested1.X = 88
X = 99
def printer(): nested1.printer()
print(X)
% python nested3.py
# nested2.py 88
from nested1 import X, printer
X = 88
printer()

% python nested2.py
99
58
Module Gotchas
● reload May Not Impact from Imports
from module import X
# before reload
# module.py from imp import reload
X = 10 reload(module)
print(X)
import module
# after reload print(X)
# module.py print(module.X)
X = 12
print(X)

59
Module Gotchas
● from * Can Obscure the Meaning of Variables

>>> from module1 import *

>>> from module2 import *
>>> from module3 import *
>>> . . .
>>> func() # func of which module?

60