Scribd
Upload
27 views7 pages

Amazon S3 Buckets by Boto3

notes on Amazon S3 buckets by boto3

Uploaded by

Rohan Ashish
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
27 views7 pages

Amazon S3 Buckets by Boto3

notes on Amazon S3 buckets by boto3

Uploaded by

Rohan Ashish
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 7

Amazon S3 buckets

An Amazon S3 bucket is a storage location to hold files. S3 files are referred to as


objects.
This section describes how to use the AWS SDK for Python to perform common
operations on S3 buckets.

Create an Amazon S3 bucket


The name of an Amazon S3 bucket must be unique across all regions of the AWS
platform. The bucket can be located in a specific region to minimize latency or to
address regulatory requirements.

import logging

import boto3

from botocore.exceptions import ClientError

def create_bucket(bucket_name, region=None):

"""Create an S3 bucket in a specified region

If a region is not specified, the bucket is created in the S3 default

region (us-east-1).

:param bucket_name: Bucket to create

:param region: String region to create bucket in, e.g., 'us-west-2'

:return: True if bucket created, else False

"""
# Create bucket

try:

if region is None:

s3_client = boto3.client('s3')

s3_client.create_bucket(Bucket=bucket_name)

else:

s3_client = boto3.client('s3', region_name=region)

location = {'LocationConstraint': region}

s3_client.create_bucket(Bucket=bucket_name,

CreateBucketConfiguration=location)

except ClientError as e:

logging.error(e)

return False

return True

List existing buckets


List all the existing buckets for the AWS account.

# Retrieve the list of existing buckets

s3 = boto3.client('s3')

response = s3.list_buckets()

# Output the bucket names

print('Existing buckets:')

for bucket in response['Buckets']:

print(f' {bucket["Name"]}')
Uploading files
The AWS SDK for Python provides a pair of methods to upload a file to an S3 bucket.
The upload_file method accepts a file name, a bucket name, and an object name.
The method handles large files by splitting them into smaller chunks and uploading
each chunk in parallel.

import logging

import boto3

from botocore.exceptions import ClientError

import os

def upload_file(file_name, bucket, object_name=None):

"""Upload a file to an S3 bucket

:param file_name: File to upload

:param bucket: Bucket to upload to

:param object_name: S3 object name. If not specified then file_name is


used

:return: True if file was uploaded, else False

"""

# If S3 object_name was not specified, use file_name

if object_name is None:

object_name = os.path.basename(file_name)
# Upload the file

s3_client = boto3.client('s3')

try:

response = s3_client.upload_file(file_name, bucket, object_name)

except ClientError as e:

logging.error(e)

return False

return True

The upload_fileobj method accepts a readable file-like object. The file object must
be opened in binary mode, not text mode.

s3 = boto3.client('s3')

with open("FILE_NAME", "rb") as f:

s3.upload_fileobj(f, "BUCKET_NAME", "OBJECT_NAME")

The upload_file and upload_fileobj methods are provided by the


S3 Client, Bucket, and Object classes. The method functionality provided by each
class is identical. No benefits are gained by calling one class's method over another's.
Use whichever class is most convenient.

The ExtraArgs parameter


Both upload_file and upload_fileobj accept an optional ExtraArgs parameter that
can be used for various purposes. The list of valid ExtraArgs settings is specified in
the ALLOWED_UPLOAD_ARGS attribute of the S3Transfer object
at boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS.
The following ExtraArgs setting specifies metadata to attach to the S3 object.

s3.upload_file(

'FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME',

ExtraArgs={'Metadata': {'mykey': 'myvalue'}}

)
The following ExtraArgs setting assigns the canned ACL (access control list) value
'public-read' to the S3 object.

s3.upload_file(

'FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME',

ExtraArgs={'ACL': 'public-read'}

The ExtraArgs parameter can also be used to set custom or multiple ACLs.

s3.upload_file(

'FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME',

ExtraArgs={

'GrantRead':
'uri="http://acs.amazonaws.com/groups/global/AllUsers"',

'GrantFullControl': 'id="01234567890abcdefg"',

The Callback parameter


Both upload_file and upload_fileobj accept an optional Callback parameter. The
parameter references a class that the Python SDK invokes intermittently during the
transfer operation.
Invoking a Python class executes the class's __call__ method. For each invocation,
the class is passed the number of bytes transferred up to that point. This information
can be used to implement a progress monitor.
The following Callback setting instructs the Python SDK to create an instance of
the ProgressPercentage class. During the upload, the instance's __call__ method will
be invoked intermittently.

s3.upload_file(

'FILE_NAME', 'BUCKET_NAME', 'OBJECT_NAME',

Callback=ProgressPercentage('FILE_NAME')
)

An example implementation of the ProcessPercentage class is shown below.

import os

import sys

import threading

class ProgressPercentage(object):

def __init__(self, filename):

self._filename = filename

self._size = float(os.path.getsize(filename))

self._seen_so_far = 0

self._lock = threading.Lock()

def __call__(self, bytes_amount):

# To simplify, assume this is hooked up to a single filename

with self._lock:

self._seen_so_far += bytes_amount

percentage = (self._seen_so_far / self._size) * 100

sys.stdout.write(

"\r%s %s / %s (%.2f%%)" % (

self._filename, self._seen_so_far, self._size,

percentage))

sys.stdout.flush()
Downloading files
The methods provided by the AWS SDK for Python to download files are similar to
those provided to upload files.
The download_file method accepts the names of the bucket and object to download
and the filename to save the file to.

import boto3

s3 = boto3.client('s3')

s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')

The download_fileobj method accepts a writeable file-like object. The file object
must be opened in binary mode, not text mode.

s3 = boto3.client('s3')

with open('FILE_NAME', 'wb') as f:

s3.download_fileobj('BUCKET_NAME', 'OBJECT_NAME', f)

Like their upload cousins, the download methods are provided by the
S3 Client, Bucket, and Object classes, and each class provides identical functionality.
Use whichever class is convenient.
Also like the upload methods, the download methods support the
optional ExtraArgs and Callback parameters.
The list of valid ExtraArgs settings for the download methods is specified in
the ALLOWED_DOWNLOAD_ARGS attribute of the S3Transfer object
at boto3.s3.transfer.S3Transfer.ALLOWED_DOWNLOAD_ARGS.
The download method's Callback parameter is used for the same purpose as the
upload method's. The upload and download methods can both invoke the
same Callback class.

You might also like