AWS Glue is a serverless ETL service that makes it easy to prepare data for analytics. It includes components like a data catalog to store metadata, databases and tables to organize data, crawlers and classifiers to extract metadata from sources, jobs to run ETL logic written in Python/Scala, and triggers to automate job execution. An IAM role is created to allow Glue access to S3. Source CSV files are uploaded to S3 and a crawler extracts metadata to populate the data catalog. A job is created to join tables, transform data, and load the results into a Redshift target table. Glue uses Spark under the hood and data can be converted to Spark DataFrames for advanced transformations when
AWS Glue is a serverless ETL service that makes it easy to prepare data for analytics. It includes components like a data catalog to store metadata, databases and tables to organize data, crawlers and classifiers to extract metadata from sources, jobs to run ETL logic written in Python/Scala, and triggers to automate job execution. An IAM role is created to allow Glue access to S3. Source CSV files are uploaded to S3 and a crawler extracts metadata to populate the data catalog. A job is created to join tables, transform data, and load the results into a Redshift target table. Glue uses Spark under the hood and data can be converted to Spark DataFrames for advanced transformations when
AWS Glue is a serverless ETL service that makes it easy to prepare data for analytics. It includes components like a data catalog to store metadata, databases and tables to organize data, crawlers and classifiers to extract metadata from sources, jobs to run ETL logic written in Python/Scala, and triggers to automate job execution. An IAM role is created to allow Glue access to S3. Source CSV files are uploaded to S3 and a crawler extracts metadata to populate the data catalog. A job is created to join tables, transform data, and load the results into a Redshift target table. Glue uses Spark under the hood and data can be converted to Spark DataFrames for advanced transformations when
AWS Glue is a serverless ETL service that makes it easy to prepare data for analytics. It includes components like a data catalog to store metadata, databases and tables to organize data, crawlers and classifiers to extract metadata from sources, jobs to run ETL logic written in Python/Scala, and triggers to automate job execution. An IAM role is created to allow Glue access to S3. Source CSV files are uploaded to S3 and a crawler extracts metadata to populate the data catalog. A job is created to join tables, transform data, and load the results into a Redshift target table. Glue uses Spark under the hood and data can be converted to Spark DataFrames for advanced transformations when
Download as DOCX, PDF, TXT or read online from Scribd
Download as docx, pdf, or txt
You are on page 1/ 10
AWS Glue
AWS Glue is a server less ETL (Extract, transform and load) service on AWS cloud. It makes it easy for to prepare data for analytics.
Components of AWS Glue
Data catalog: The data catalog holds the metadata and the structure of the data.
Database: It is used to create or access the database for the sources and targets.
Table: Create one or more tables in the database that can be used by the source and target.
Crawler and Classifier: A crawler is used to retrieve data from the source using built-in or custom classifiers. It creates/uses metadata tables that are pre-defined in the data catalog.
Job: A job is business logic that carries out an ETL task. Internally, Apache Spark with python or scal0061 language writes this business logic.
Trigger: A trigger starts the ETL job execution on-demand or at a specific time.
Development endpoint: It creates a development environment where the ETL job script can be tested, developed and debugged.
Create an IAM role to access AWS Glue + Amazon S3
Open the Amazon IAM console
Click on Roles in the left pane. Then click on Create Role. Choose the AWS service from Select type of trusted entity section Choose Glue service from “Choose the service that will use this role” section Choose Glue from “Select your use case” section Click on Next: Permissions Select AWSGlueServiceRole from the Attach Permissions Policies section. AWSGlueServiceRole policy contains permissions to access Glue, CloudWatch, EC2, S3, and IAM Click on Next:Tags. Leave the Add tags section blank and click on Next:Review Provide a name to identify the service role AWSGlueServiceRole-<role name>(for simplicity add prefix ‘AWSGlueServiceRole-’ in the role name) for the role Click on Create role Role with full access to AWS Glue and limited access to Amazon S3 has been created Upload source CSV files to Amazon S3:
On the Amazon S3 console, click on the Create a bucket where can store files and folders. Enter a bucket name, select a Region and click on Next. The remaining configuration settings for creating an S3 bucket are optional. Click Next to create S3 bucket. Create a new folder in bucket and upload the source CSV files.
Create a connection for the target database into Amazon Redshift:
Prerequisite: Must have an existing cluster, database name and user for the database in Amazon Redshift. In the AWS Glue console, click on the Add Connection in the left pane.
Select existing cluster in Amazon Redshift as the cluster for connection
Enter the database name, username, and password of the cluster and click Next Review the details of the connection and then click Finish
Amazon Redshift connection is now created and can be verified through the Test Connection.
Adding a crawler to create data catalog using Amazon S3 as a data source
On the left pane in the AWS Glue console, click on Crawlers -> Add Crawler
Enter the crawler name in the dialog box and click Next Choose S3 as the data store from the drop-down list Select the folder where your CSVs are stored in the Include path field
Select Choose an existing IAM role and select the previously created role name from the dropdown list of IAM roles and click Next
Leave the default option of Run on Demand and click Next
Choose an existing database. If do not have one, Click Add Database to create a new database on the fly.
Table prefixes are optional and left to the user to customer. The system would also create these automatically after running the crawler. Click Next.
Once are satisfied with the configuration, click on Finish
Can now run the crawler by clicking Run it now
Databases on the left pane let verify if the tables were created automatically by the crawler. Amazon Athena enables to view the data in the tables.
Add a Job to execute ETL work
In the left pane, Click on Job, then click on Add Job
Enter a name for the Job and then select an IAM role previously created for AWS Glue Select Spark for the Type and select Python or Scala. Can edit the number of DPU (Data processing unit) value in the Maximum capacity field of Security configuration, script libraries, and job parameters (optional). The remaining configuration is optional and default values would work fine. Click Next Choose a data source table from Choose a data source section. Can choose only a single data source. Click Next.
Choose a data target table from the list of tables. Either can create new tables or choose an existing one.
If haven’t created any target table, select Create tables in data target option
Enter a database name that must exist in the target data store. Click Next. Can map the columns of the source table with those of the target table. Click Save job and edit script.
Open the Python script by selecting the recently created job name. Click on Action -> Edit Script.
The left pane shows a visual representation of the ETL process. The right-hand pane shows the script code and just below that can see the logs of the running Job.
The script that performs extraction, transformation and loading process on AWS Glue.
import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job
Now, Apply transformation on the source tables. Can join both the tables on statecode column of tbl_syn_source_1_csv and code column of tbl_syn_source_2_csv.
Several transformations are available within AWS Glue such as RenameField, SelectField, Join, etc. Refer – https://docs.aws.amazon.com/glue/latest/dg/built-in-transforms.html.
import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job
## Read the data from Amazon S3 and have their structure in the data catalog. datasource1 = glueContext.create_dynamic_frame.from_catalog(database = "db_demo1", table_name = "tbl_syn_source_1_csv", transformation_ctx = "datasource1")
