Problem Statement − Use boto3 library in Python to retrieve the metrics of a specified crawler.
Example − Retrieve the metrics of a specified crawler, crawler_for_s3_file_job.
Approach/Algorithm to solve this problem
Step 1 − Import boto3 and botocore exceptions to handle exceptions.
Step 2 − crawler_names is the mandatory parameter. It is a list so user can send one or many crawler names at a time to fetch metrics.
Step 3 − Create an AWS session using boto3 library. Make sure region_name is mentioned in default profile. If it is not mentioned, then explicitly pass the region_name while creating the session.
Step 4 − Create an AWS client for glue.
Step 5 − Now use get_crawler_metrics function and pass the crawler_names and CrawlerNameList parameter.
Step 6 − It returns the metrics of crawlers.
Step 7 − Handle the generic exception if something went wrong while checking the job.
Example
Use the following code to retrieve the metrics of a specified crawler −
import boto3 from botocore.exceptions import ClientError def retrieves_metrics_of_a_crawler(crawler_names:list) session = boto3.session.Session() glue_client = session.client('glue') try: crawler_details = glue_client.get_crawler_metrics(CrawlerNameList = crawler_names) return crawler_details except ClientError as e: raise Exception("boto3 client error in retrieves_metrics_of_a_crawler: " + e.__str__()) except Exception as e: raise Exception("Unexpected error in retrieves_metrics_of_a_crawler: " + e.__str__()) print(retrieves_metrics_of_a_crawler(["crawler_for_s3_file_job"]))
Output
{'CrawlerMetricsList': [{'CrawlerName': crawler_for_s3_file_job, 'TimeLeftSeconds': 0.0, 'StillEstimating': False, 'LastRuntimeSeconds': 79.673, 'MedianRuntimeSeconds': 79.673, 'TablesCreated': 1, 'TablesUpdated': 0, 'TablesDeleted': 0}], 'ResponseMetadata': {'RequestId': '680cf4ca-********0abe', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Sun, 28 Feb 2021 11:38:08 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '216', 'connection': 'keep-alive', 'x-amzn-requestid': '680cf4ca-******************0abe'}, 'RetryAttempts': 0}}