Problem Statement: Use boto3 library in Python to paginate through multi-part upload objects of a S3 bucket from AWS Glue Data Catalog that is created in your account
Approach/Algorithm to solve this problem
Step 1: Import boto3 and botocore exceptions to handle exceptions.
Step 2: prefix_name, max_items, page_size and starting_token is optional parameter for this function while bucket_name is required parameters.
Prefix_name is the specific sub folders where user wants to paginate through
max_items denote the total number of records to return. If the number of available records > max_items, then a NextToken will be provided in the response to resume pagination.
page_size denotes the size of each page.
starting_token helps to paginate, and it uses Marker from a previous response.
Step 3: Create an AWS session using boto3 lib. Make sure region_name is mentioned in the default profile. If it is not mentioned, then explicitly pass the region_name while creating the session.
Step 4: Create an AWS client for S3.
Step 5: Create a paginator object that contains details of object versions of a S3 bucket using list_multipart_uploads.
Step 6: Call the paginate function and pass the max_items, page_size and starting_token as PaginationConfig parameter while bucket_name as Bucket parameter and prefix_name as Prefix.
Step 7: It returns the number of records based on max_size and page_size.
Step 8: Handle the generic exception if something went wrong while paginating.
Example Code
Use the following code to paginate through multipart upload of a S3 bucket created in user account −
import boto3
from botocore.exceptions import ClientError
def paginate_through_multipart_upload_s3_bucket(bucket_name, prefix_name=None, max_items=None:int,page_size=None:int, starting_token=None:string):
session = boto3.session.Session()
s3_client = session.client('s3')
try:
paginator = s3_client.get_paginator('list_objects')
response = paginator.paginate(Bucket=bucket_name, Prefix=prefix_name, PaginationConfig={
'MaxItems':max_items,
'PageSize':page_size,
'StartingToken':starting_token}
)
return response
except ClientError as e:
raise Exception("boto3 client error in paginate_through_multipart_upload_s3_bucket: " + e.__str__())
except Exception as e:
raise Exception("Unexpected error in paginate_through_multipart_upload_s3_bucket: " + e.__str__())
a = paginate_through_multipart_upload_s3_bucket('s3-test-bucket', 'testfolder',2,5)
print(*a)Output
{'ResponseMetadata': {'RequestId': 'YA9CGTAAX', 'HostId': '8dqJW******************', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '8dqJW*********************, 'x-amz-request-id': 'YA9CGTAAX', 'date': 'Sat, 03 Apr 2021 08:16:05 GMT', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'Bucket': 's3-test-bucket', 'KeyMarker': '', 'UploadIdMarker': '', 'NextKeyMarker': '', 'Prefix': 'testfolder', 'NextUploadIdMarker': '', 'MaxUploads': 5, 'IsTruncated': False, 'Uploads': [
{'UploadId': 'YADF**************LK25',
'Key': 'testfolder/testfilemultiupload.csv',
'Intiated':datetime(2021,1,2),
'StorageClass': 'STANDARD'
'Owner':{
'DisplayName': 'AmazonServicesJob'
'Id': '********************'
}
], 'CommonPrefixes': None}