Module 4
Module 4
MODULE 4
SYLLABUS:
Organizing Files: The shutil Module, Walking a Directory Tree, Compressing Files with the zipfile
Module, Project: Renaming Files with American-Style Dates to European-Style Dates,Project:
Backing Up a Folder into a ZIP File
Debugging: Raising Exceptions, Getting the Traceback as a String, Assertions, Logging, IDLE‟s
Debugger.
• In the previous chapter, you learned how to create and write to new files in Python.
• Your programs can also organize pre existing files on the hard drive.
• The shutil (or shell utilities) module has functions to let you copy, move, rename, and delete
files in your Python programs.
• To use the shutil functions, you will first need to use import shutil.
Copying Files and Folders
• The shutil module provides functions for copying files, as well as entire folders.
• Calling shutil. copy(source, destination) will copy the file at the path source to the folder at
thepath destination. (Both source and destination are strings.)
• If destination is a filename, it will be used as the new name of the copied file.
Example:
>>> import shutil, os
>>> os.chdir('E:\\')
>>> shutil.copy('E:\\spam.txt', 'E:\\delicious')
‘E:\\delicious\\spam.txt'
>>> shutil.copy('cheese.txt', 'E:\\delicious\\cheese2.txt')
‘E:\\delicious\\cheese2.txt‘ # gives the copied file the name cheese2.txt
• It will copy an entire folder and every folder and file contained in it.
Syntax:
shutil.copytree(source, destination)
1
lOMoARcPSD|38641732
- will copy the folder at the path source, along with all of its files and subfolders, to the folder at
the path destination.
- The source and destination parameters are both strings.
- The function returns a string of the path of the copied folder.
Example
>>> import shutil, os
>>> os.chdir(‘E:\\')
>>> shutil.copytree(‘E:\\bacon', ‘E:\\bacon_backup')
‘E:\\bacon_backup'
'C:\\eggs\\bacon.txt'
Example 2:
>>> shutil.move('C:\\bacon.txt', 'C:\\eggs\\new_bacon.txt')
'C:\\eggs\\new_bacon.txt'
2
lOMoARcPSD|38641732
Unlike range(), the os.walk() function will return three values on each iteration through the
loop:
1. A string of the current folder’s name
2. A list of strings of the folders in the current folder
3. A list of strings of the files in the current folder
Example 1:
import os
for i in os.walk(r'E:\SAMPLE'):
print(i)
Example 2:
for dirpath,dirnames,filename in os.walk(r'E:\SAMPLE'):
print(f'Root:{dirpath}\n'
f'Sub-Directories:{dirnames}\n'
f'Files:{filename}\n'
)
3
lOMoARcPSD|38641732
4
lOMoARcPSD|38641732
>>> exampleZip.namelist()
[‘file1.docx', ‘file2.docx', ‘file3.docx', ................. ,‘filen.docx']
>>> spam Info = exampleZip.getinfo('spam.txt')
>>> spamInfo = exampleZip.getinfo('EVALUATION SHEET_CHARCOAL ART .docx')
125686
>>> spamInfo.compress_size
3828
>>> 'Compressed file is %sx smaller!' % (round(spamInfo.file_size / spamInfo
.compress_size,2))
'Compressed file is 3.63x smaller!'
>>> exampleZip.close()
1.7.2 Extracting from ZIP Files
• The extractall() method for ZipFile objects extracts all the files and folders from a ZIP file into
the current working directory.
Example:
>>> import zip file, os
>>> os.chdir('E:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('data.zip')
>>> exampleZip.extractall()
>>> exampleZip.close()
Example:
>>> import zipfile
>>> newZip = zipfile. ZipFile ('E:\\new.zip', 'w')
>>> newZip.write('E:\\cheese.txt', compress type=zipfile.ZIP_DEFLATED)
>>> newZip.close()
5
lOMoARcPSD|38641732
Loop over each filename, using the regex to check whether it has a date.
If it has a date, rename the file with shutil.move().
For this project, open a new file editor window and save your code as renameDates.py
#! python3
# renameDates.py - Renames filenames with American MM-DD-YYYY date format
# to European DD-MM-YYYY.
*1 import shutil, os, re
# Create a regex that matches files with the American date format.
*2 datePattern = re.compile(r"""^(.*?) # all text before the date
((0|1)?\d)- # one or two digits for the month
((0|1|2|3)?\d)- # one or two digits for the day
((19|20)\d\d) # four digits for the year
(.*?)$ # all text after the date
*3 """, re.VERBOSE
# TODO: Loop over the files in the working directory.
# TODO: Skip files without a date.
# TODO: Get the different parts of the filename.
# TODO: Form the European-style filename.
# TODO: Get the full, absolute file paths.
# TODO: Rename the files.
Note: Write TODO comments for these steps in the source code to remind yourself to do them
later.
Explanation:
*1 The shutil.move() function can be used to rename files: Its arguments are the name of the
file to rename and the new filename. Because this function exists in the shutil module, you
must import that module .
*2 Before renaming the files, you need to identify which files you want to rename. Filenames
with dates such as spam4-4-1984.txt and 01-03-2014eggs.zip should be renamed, while
filenames without dates such as littlebrother.epub can be ignored. You can use a regular
expression to identify this pattern. After importing the re module at the top, call re.compile()
to create a Regex object .
*3 Passing re.VERBOSE for the second argument w will allow whitespace and comments in
the regex string to make it more readable
#! python3
6
lOMoARcPSD|38641732
Explanation:
*1 If the Match object returned from the search() method is None. Then the filename in
amerFilename does not match the regular expression.
*2 The continue statement will skip the rest of the loop and move on to the next filename.
*3 The various strings matched in the regular expression groups are stored in variables named
beforePart, monthPart, dayPart, yearPart, and afterPart .
#! python3
# renameDates.py - Renames filenames with American MM-DD-YYYY date format
# to European DD-MM-YYYY.
--snip—
# Form the European-style filename.
*1 euroFilename = beforePart + dayPart + '-' + monthPart + '-' + yearPart + afterPart
# Get the full, absolute file paths.
absWorkingDir = os.path.abspath('.')
amerFilename = os.path.join(absWorkingDir, amerFilename)
euroFilename = os.path.join(absWorkingDir, euroFilename)
Explanation:
*1 Store the concatenated string in a variable named euroFilename
7
lOMoARcPSD|38641732
*3 Then, pass the original filename in amerFilename and the new euroFilename variable to the
shutil.move() function to rename the file
*2 This program has the shutil.move() call commented out and instead prints the filenames that
will be renamed.
Say you’re working on a project whose files you keep in a folder named C:\AlsPythonBook.
You’re worried about losing your work, so you’d like to create ZIP file “snapshots” of the entire
folder. You’d like to keep different versions, so you want the ZIP file’s filename to increment
each time it is made; for example, AlsPythonBook_1.zip, AlsPythonBook_2.zip,
AlsPythonBook_3.zip, and so on. You could do this by hand, but it is rather annoying, and you
might accidentally misnumber the ZIP files’ names. It would be much simpler to run a program
that does this boring task for you.
Code:
For this project, open a new file editor window and save it as backupToZip.py
Step 1: Figure Out the ZIP File’s Name
#! python3
# backupToZip.py - Copies an entire folder and its contents into
# a ZIP file whose filename increments.
*1 import zipfile, os
def backupToZip(folder):
# Backup the entire contents of "folder" into a ZIP file.
folder = os.path.abspath(folder)
Explanation:
8
lOMoARcPSD|38641732
*1 Do the basics first: Add the shebang (#!) line, describe what the program does, and import
the zipfile and os modules.
*2 You can determine what N should be by checking whether delicious_1.zip already exists,
then checking whether delicious_2.zip already exists, and so on. Use a variable named number
for N
*3 and keep incrementing it inside the loop that calls os.path.exists() to check whether the file
exists
*4 Write TODO comments for these steps in the source code to remind yourself to do them
later.
backupToZip('C:\\delicious')
Explanation:
*1 Now that the new ZIP file’s name is stored in the zipFilename variable, you can call
zipfile.ZipFile() to actually create the ZIP file.
Note: Be sure to pass 'w' as the second argument so that the ZIP file is opened in write mode
Step 3: Walk the Directory Tree and Add to the ZIP File
#! python3
# backupToZip.py - Copies an entire folder and its contents into
# a ZIP file whose filename increments.
--snip—
# Walk the entire folder tree and compress the files in each folder.
*1 for foldername, subfolders, filenames in os.walk(folder):
print('Adding files in %s...' % (foldername))
# Add the current folder to the ZIP file.
9
lOMoARcPSD|38641732
*2 backupZip.write(foldername)
# Add all the files in this folder to the ZIP file.
*3 for filename in filenames:
newBase / os.path.basename(folder) + '_'
if filename.startswith(newBase) and filename.endswith('.zip')
continue # don't backup the backup ZIP files
backupZip.write(os.path.join(foldername, filename))
backupZip.close()
print('Done.')
backupToZip('C:\\delicious')
Explanation:
*1 You can use os.walk() in a for loop u, and on each iteration it will return the iteration’s
current folder name, the subfolders in that folder, and the filenames in that folder.
*2 In the for loop, the folder is added to the ZIP file
*3 The nested for loop can go through each filename in the filenames list.
10
lOMoARcPSD|38641732
Chapter 2: Debugging
Exceptions are raised with a raise statement. In code, a raise statement consists of the following:
• The raise keyword
• A call to the Exception() function
• A string with a helpful error message passed to the Exception() function
Example:
def boxPrint(symbol, width, height):
if len(symbol) != 1:
raise Exception('Symbol must be a single character string.')
if width <= 2:
raise Exception('Width must be greater than 2.')
if height <= 2: w
raise Exception('Height must be greater than 2.')
print(symbol * width)
for i in range(height - 2):
print(symbol + (' ' * (width - 2)) + symbol
print(symbol * width)
for sym, w, h in (('*', 4, 4), ('O', 20, 5), ('x', 1, 3), ('ZZ', 3, 3)):
try:
boxPrint(sym, w, h)
except Exception as err:
print('An exception happened: ' + str(err))
Output:
****
* *
* *
****
OOOOOOOOOOOOOOOOOOOO
O O
O O
O O
OOOOOOOOOOOOOOOOOOOO
An exception happened: Width must be greater than 2.
An exception happened: Symbol must be a single character string
11
lOMoARcPSD|38641732
The traceback includes the error message, the line number of the line that caused the error, and
the sequence of the function calls that led to the error.
This sequence of calls is called the call stack.
errorExample.py
def spam():
bacon()
def bacon():
raise Exception('This is the error message.')
spam()
When you run errorExample.py, the output will look like this
Traceback (most recent call last):
File "errorExample.py", line 7, in
spam()
File "errorExample.py", line 2, in spam
bacon()
File "errorExample.py", line 5, in bacon
raise Exception('This is the error message.')
Exception: This is the error message.
From the traceback, you can see that the error happened on line 5, in the bacon() function. This
particular call to bacon() came from line 2, in the spam() function, which in turn was called on
line 7.
>>> import traceback
>>> try:
raise Exception('This is the error message.')
except:
errorFile = open('errorInfo.txt', 'w')
errorFile.write(traceback.format_exc())
errorFile.close()
print('The traceback info was written to errorInfo.txt.')
116
The traceback info was written to errorInfo.txt.
The 116 is the return value from the write() method, since 116 characters were written to the
file.
The traceback text was written to errorInfo.txt (Below is the content stored in erriinfo.txt)
Traceback (most recent call last):
File "<pyshell#28>", line 2, in <module>
Exception: This is the error message.
2.3 Assertions
12
lOMoARcPSD|38641732
An assertion is a sanity check to make sure your code isn’t doing something obviously wrong.
In code, an assert statement consists of the following:
• The assert keyword
• A condition (that is, an expression that evaluates to True or False)
• A comma
• A string to display when the condition is False
Example:
>>> podBayDoorStatus = 'open'
>>> assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
>>> podBayDoorStatus = 'I\'m sorry, Dave. I\'m afraid I can't do that.''
>>> assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
Traceback (most recent call last):
File "<pyshell#28>", line 1, in <module>
assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
AssertionError: The pod bay doors need to be "open".
# keys 'ns' and 'ew', for the stoplights facing north-south and east-west, respectively
# The values at these keys will be one of the strings 'green', 'yellow', or 'red'.
market_2nd = {'ns': 'green', 'ew': 'red'}
mission_16th = {'ns': 'red', 'ew': 'green'}
def switchLights(stoplight):
for key in stoplight.keys():
if stoplight[key] == 'green':
stoplight[key] = 'yellow'
elif stoplight[key] == 'yellow':
stoplight[key] = 'red'
elif stoplight[key] == 'red':
stoplight[key] = 'green'
assert 'red' in stoplight.values(), 'Neither light is red! ' + str(stoplight)
switchLights(market_2nd)
Output:
Traceback (most recent call last):
File "carSim.py", line 14, in <module 4>
switchLights(market_2nd)
File "carSim.py", line 13, in switchLights
assert 'red' in stoplight.values(), 'Neither light is red! ' + str(stoplight)
AssertionError: Neither light is red! {'ns': 'yellow', 'ew': 'green'}
Disabling Assertions
13
lOMoARcPSD|38641732
2.4 Logging
If you’ve ever put a print() statement in your code to output some variable’s value while your
program is running, you’ve used a form of logging to debug your code.
Logging is a great way to understand what’s happening in your program and in what order its
happening.
Logging is a great way to understand what’s happening in your program and in what order its
happening.
Using the logging Module
To enable the logging module to display log messages on your screen as your program runs,
copy the following to the top of your program
basically,
when Python logs an event, it creates a LogRecord object that holds information about that
event.
The logging module’s basicConfig() function lets you specify what details about the LogRecord
object you want to see and how you want those details displayed.
Example
Open a new file editor window and enter the following code. It has a bug in it, but you will also
enter several log messages to help yourself figure out what is going wrong
import logging
logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s -
%(message)s')
logging.debug('Start of program')
def factorial(n):
logging.debug('Start of factorial(%s%%)' % (n))
total = 1
for i in range(n + 1):
total *= i
logging.debug('i is ' + str(i) + ', total is ' + str(total))
logging.debug('End of factorial(%s%%)' % (n))
return total
print(factorial(5))
14
lOMoARcPSD|38641732
logging.debug('End of program')
Output
================RESTART: C:/Users/admin/log.py ====================
2023-03-09 11:35:24,860 - DEBUG - Start of program
2023-03-09 11:35:24,870 - DEBUG - Start of factorial(5%)
2023-03-09 11:35:24,873 - DEBUG - i is 0, total is 0
2023-03-09 11:35:24,877 - DEBUG - i is 1, total is 0
2023-03-09 11:35:24,881 - DEBUG - i is 2, total is 0
2023-03-09 11:35:24,883 - DEBUG - i is 3, total is 0
2023-03-09 11:35:24,886 - DEBUG - i is 4, total is 0
2023-03-09 11:35:24,889 - DEBUG - i is 5, total is 0
2023-03-09 11:35:24,892 - DEBUG - End of factorial(5%)
0
2023-03-09 11:35:24,901 - DEBUG - End of program
Logging Levels
Level Logging Function Description
DEBUG logging.debug() The lowest level. Used for
small details. Usually you
care about these messages
only when diagnosing
problems.
INFO logging.info() Used to record information
on general events in your
program or confirm that
things are working at their
point in the program
WARNING logging.warning() Used to indicate a potential
problem that doesn’t
prevent the program from
15
lOMoARcPSD|38641732
Example:
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s -
%(message)s')
>>> logging.debug('Some debugging details.')
2015-05-18 19:04:26,901 - DEBUG - Some debugging details.
>>> logging.info('The logging module is working.')
2023-03-09 19:04:35,569 - INFO - The logging module is working.
>>> logging.warning('An error message is about to be logged.')
2023-03-09 19:04:56,843 - WARNING - An error message is about to be logged.
>>> logging.error('An error has occurred.')
2023-03-09 19:05:07,737 - ERROR - An error has occurred.
>>> logging.critical('The program is unable to recover!')
2023-03-09 19:05:45,794 - CRITICAL - The program is unable to recover!
Disabling Logging
After you’ve debugged your program, you probably don’t want all these log messages
cluttering the screen.
The logging.disable() function disables these so that you don’t have to go into your
program and remove all the logging calls by hand.
You simply pass logging.disable() a logging level, and it will suppress all log messages at
that level or lower.
Example:
>>> import logging
>>> logging.basicConfig(level=logging.INFO, format=' %(asctime)s - %(levelname)s -
%(message)s')
>>> logging.critical('Critical error! Critical error!')
2023-03-09 11:47:44,797 - CRITICAL - Critical error! Critical error!
>>> logging.disable(logging.CRITICAL)
16
lOMoARcPSD|38641732
Logging to a File
Instead of displaying the log messages to the screen, you can write them to a text file. The
logging.basicConfig() function takes a filename keyword argument, like so
Example:
import logging logging.basicConfig(filename='myProgramLog.txt', level=logging.DEBUG,
format=' %(asctime)s - %(levelname)s - %(message)s')
The debugger is a feature of IDLE that allows you to execute your program one line at a time.
The debugger will run a single line of code and then wait for you to tell it to continue.
To enable IDLE’s debugger, click DebugDebugger in the interactive shell window.
Go
Clicking the Go button will cause the program to execute normally until it terminates or reaches
a breakpoint.
Step
17
lOMoARcPSD|38641732
Clicking the Step button will cause the debugger to execute the next line of code and then pause
again.
Over
Clicking the Over button will execute the next line of code, similar to the Step button.
Out
Clicking the Out button will cause the debugger to execute lines of code at full speed until it
returns from the current function.
Quit
If you want to stop debugging entirely and not bother to continue executing the rest of the
program, click the Quit button.
The Quit button will immediately terminate the program.
2.5.1 Breakpoints
A breakpoint can be set on a specific line of code and forces the debugger to pause whenever the
program execution reaches that line.
Open a new file editor window and enter the following program, which simulates flipping a coin 1,000
times. Save it as coinFlip.py
Example: Flipping a coin for 1000 time.
import random
heads = 0
for i in range(1, 1001):
*1 if random. Rand int(0, 1) == 1:
heads = heads + 1
if i == 500:
*2 print('Halfway done!')
print('Heads came up ' + str(heads) + ' times.')
Output:
Halfway done!
Heads came up 517 times.
Explanation:
*1 The random. Rand int(0, 1) call will return 0 half of the time and 1 the other half of the time. This
canbe used to simulate a 50/50 coin flip where 1 represents heads.
*2 If you were interested in the value of heads at the halfway point of the program’s execution, when
500 of 1000 coin flips have been completed, you could instead just set a breakpoint on the line
print('Halfway done!')
Note:
To set a breakpoint, right-click the line in the file editor and select Set Breakpoint, as shown in Figure
below
If you want to remove a breakpoint, right-click the line in the file editor and select Clear Breakpoint
from the menu.
18
lOMoARcPSD|38641732
19