Scribd
Upload
80 views3 pages

Airflow Documentation

Uploaded by

Y sudheer
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
80 views3 pages

Airflow Documentation

Uploaded by

Y sudheer
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

Airflow Documentation.

Airflow is a platform that lets you build and run workflows. A workflow is
represented as a DAG (a Directed Acyclic Graph), and contains individual pieces of
work called Tasks, arranged with dependencies and data flows taken into account.
Architecture of Airflow:-

An Airflow installation generally consists of the following components:

 A scheduler, which handles both triggering scheduled workflows, and


submitting Tasks to the executor to run.
 An executor, which handles running tasks. In the default Airflow
installation, this runs everything inside the scheduler, but most
production-suitable executors actually push task execution out
to workers.
 A webserver, which presents a handy user interface to inspect, trigger
and debug the behaviour of DAGs and tasks.
 A folder of DAG files, read by the scheduler and executor (and any
workers the executor has)
 A metadata database, used by the scheduler, executor and webserver
to store state.
Different Source systems in DataPipeline:-
1.AGP

2.AGC

3.FEP

4.WGS

5.WGS_ODS

1.AGP Pipeline screen shot:-

Requirement 1:- Need to change the bactch id for all the Json’s

Step:1 Go to this path /apps/fcs-python-modules/AGP/python and open the ami_audit.json file .

Step:2 And do the changes in batchID Parameter.

Step3 Go to putty and run below commands,

source /apps/fcs-python-modules/fcs2.0/bin/activate

python /apps/fcs-python-modules/python/datapipeline.py --job validation

Requirement 2: Getting Json file from Datapipe line team,

Step1: Add the file in /apps/fcs-python-modules/AGP/json_payload_AGP

Step2: Delete the /apps/fcs-python-modules/AGP/python config.ini file in the path.

Step3: Go to putty and run the below commands

source /apps/fcs-python-modules/fcs2.0/bin/activate

cd /apps/fcs-python-modules/AGP/python
python config_agp.py

Creation of Dags:-

Step 1: Go to the /apps/fcs-python-modules/airflow/dags path and create Yaml file with dag name,

Step2: Create the python file with .py extension and pass the path of yaml in DagFactory function.

You might also like