1 Vishal F Script
1 Vishal F Script
Project 1
Client: KGL LOGISTICS
Description: KGL Logistics is a prominent logistics company headquartered in Kuwait,
specializing in integrated supply chain management services across the
Middle East. With over 40 years of experience, the company has
established itself as a key player in the logistics sector, offering a
comprehensive range of services tailored to meet diverse business needs.
Simplified Logistics.
This gold layer is where we make the final changes by converting the
validated data into fact and dimension tables, and we load
the data into ADLS in Parquet format.
So, our upstream data is in CSV format, and the downstream data is in
Parquet format.
After writing the final data to the Gold layer, we use Azure Event Grid
to create an event notification of that file and send it to the
handler. in our case handler is Snowflake Whenever
Snowflake gets a notification then Snowpipe will run. and
with the help of snow pipe
Sources of data
Data is generated from different applications like
1. warehouse management systems (WMS)-
2. enterprise resource planning (ERP):- : SAP ERP, Oracle ERP Cloud
3. transportation management systems (TMS):- : SAP Transportation
Management
4. customer feedback systems
5. environmental impact software:- : Enablon, Sphera
Types of data
Generally, data are ingested in the form of Flat Files & mostly those files in
the form of CSV.
1. Consolidation Hubs data file
2. Load Planning data file
3. Transport mode optimization data file
4. Scheduling and Coordination data file
5. Freight Documentation data file
6. Cost Management data file
7. Carrier Performance Evaluation data file
8. Environmental impact monitoring data file