Computer >> Computer tutorials >  >> Programming >> Python

How to use Boto3 to paginate through security configuration present in AWS Glue

In this article, we will see how to paginate through security configuration present in AWS Glue.

Example

Problem Statement: Use boto3 library in Python to paginate through security configuration from AWS Glue Data Catalog that is created in your account

Approach/Algorithm to solve this problem

  • Step 1: Import boto3 and botocore exceptions to handle exceptions.

  • Step 2: max_items, page_size and starting_token are the optional parameters for this function.

    • max_items denote the total number of records to return. If the number of available records > max_items, then a NextToken will be provided in the response to resume pagination.

    • page_size denotes the size of each page.

    • starting_token helps to paginate, and it uses NextToken from a previous response.

  • Step 3: Create an AWS session using boto3 lib. Make sure region_name is mentioned in the default profile. If it is not mentioned, then explicitly pass the region_name while creating the session.

  • Step 4: Create an AWS client for glue.

  • Step 5: Create a paginator object that contains the details of all crawlers using get_security_configurations

  • Step 6: Call the paginate function and pass the max_items, page_size and starting_token as PaginationConfig

  • Step 7: It returns the number of records based on max_size and page_size.

  • Step 8: Handle the generic exception if something went wrong while paginating.

Example Code

Use the following code to paginate through all security configuration created in user account −

import boto3
from botocore.exceptions import ClientError

def paginate_through_security_configuration(max_items=None:int,page_size=None:int, starting_token=None:string):
   session = boto3.session.Session()
   glue_client = session.client('glue')
   try:
   paginator = glue_client.get_paginator('get_security_configuration')
      response = paginator.paginate(PaginationConfig={
         'MaxItems':max_items,
         'PageSize':page_size,
         'StartingToken':starting_token}
      )
   return response
   except ClientError as e:
      raise Exception("boto3 client error in paginate_through_security_configuration: " + e.__str__())
   except Exception as e:
      raise Exception("Unexpected error in paginate_through_security_configuration: " + e.__str__())
a = paginate_through_security_configuration(2,5)
print(*a)

Output

{'SecurityConfigurations': [
{'Name': 'test-sc', 'CreatedTimeStamp': datetime.datetime(2020, 9, 24, 1, 53, 21, 265000, tzinfo=tzlocal()), 'EncryptionConfiguration': {'S3Encryption': [{'S3EncryptionMode': 'SSE-KMS', 'KmsKeyArn': 'arn:aws:kms:us-east-1:*************:key/***************'}]}},
{'Name': 'port-sc', 'CreatedTimeStamp': datetime.datetime(2020, 11, 6, 0, 38, 3, 753000, tzinfo=tzlocal()), 'EncryptionConfiguration': {'S3Encryption': [{'S3EncryptionMode': 'SSE-KMS', 'KmsKeyArn': 'arn:aws:kms:us-east-1:********:key/***************'}]}}],
'NextToken': '',
'ResponseMetadata': {'RequestId': **********, 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Fri, 02 Apr 2021 13:19:57 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '826', 'connection': 'keep-alive', 'x-amzn-requestid': *********}, 'RetryAttempts': 0}}