Scribd
Upload
11 views

1. Installing and Configuring Tools

This document provides a comprehensive step-by-step guide for installing and configuring Apache NiFi, Airflow, Elasticsearch, Kibana, PostgreSQL, and pgAdmin 4 on Windows 11. It includes detailed instructions for downloading, installing dependencies, configuring properties, and verifying installations for each tool. The guide ensures that users can successfully set up their data processing and management environment.

Uploaded by

rg.techprojects
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
11 views

1. Installing and Configuring Tools

This document provides a comprehensive step-by-step guide for installing and configuring Apache NiFi, Airflow, Elasticsearch, Kibana, PostgreSQL, and pgAdmin 4 on Windows 11. It includes detailed instructions for downloading, installing dependencies, configuring properties, and verifying installations for each tool. The guide ensures that users can successfully set up their data processing and management environment.

Uploaded by

rg.techprojects
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 5

1.

Installing and Configuring Tools

1.1 Install Apache NiFi & Airflow:

1. Download Apache NiFi and Airflow from official websites.


2. Install dependencies (Java for NiFi, Python for Airflow).
3. Configure NiFi by editing nifi.properties.
4. Configure Airflow by initializing the database (airflow db init).
5. Start NiFi (nifi.sh start) and Airflow Scheduler/Webserver (airflow scheduler
& airflow webserver).
6. Verify installations by accessing NiFi at http://localhost:8080 and Airflow at
http://localhost:8081.

1.2 Install Elasticsearch, Kibana, PostgreSQL, and pgAdmin 4:

1. Download and install Elasticsearch and Kibana.


2. Start Elasticsearch and Kibana services.
3. Verify Elasticsearch (http://localhost:9200) and Kibana
(http://localhost:5601).
4. Install PostgreSQL and pgAdmin 4.
5. Create databases and users in PostgreSQL using pgAdmin.

Here's a step-by-step guide for installing and configuring Apache NiFi, Airflow,
Elasticsearch, Kibana, PostgreSQL, and pgAdmin 4 on Windows 11:

✅ 1. Installing and Configuring Tools on


Windows 11

⚡ 1.1 Install Apache NiFi & Airflow


Step 1: Install Apache NiFi

✅ 1.1.1. Install Java (NiFi Dependency):

1. Download Java JDK 11 or later from Oracle or AdoptOpenJDK.


2. Install Java and set JAVA_HOME:
o Open System Properties → Environment Variables.
o Add new variable:

ini
CopyEdit
JAVA_HOME = C:\Program Files\Java\jdk-<version>

o Add %JAVA_HOME%\bin to the Path variable.


3. Verify installation:

bash
CopyEdit
java -version

✅ 1.1.2. Install Apache NiFi:

1. Download NiFi from Apache NiFi Downloads.


2. Extract the ZIP file to C:\nifi.
3. Configure NiFi:
o Navigate to:

makefile
CopyEdit
C:\nifi\conf\nifi.properties

o Set the following:

properties
CopyEdit
nifi.web.http.host=localhost
nifi.web.http.port=8080

4. Start NiFi:

bash
CopyEdit
C:\nifi\bin\nifi.bat start

5. Verify:
o Visit: http://localhost:8080

Step 2: Install Apache Airflow

✅ 1.1.3. Install Python (Airflow Dependency):

1. Download and install Python 3.8+ from Python.org.


2. Ensure pip is installed and added to Path.
3. Verify:

bash
CopyEdit
python --version
pip --version
✅ 1.1.4. Install Airflow Using pip:

1. Open Command Prompt or PowerShell.


2. Create a virtual environment (optional but recommended):

bash
CopyEdit
python -m venv airflow_env
airflow_env\Scripts\activate

3. Set Airflow constraints and install:

bash
CopyEdit
set AIRFLOW_VERSION=2.8.1
set PYTHON_VERSION=3.8
set
CONSTRAINT_URL=https://raw.githubusercontent.com/apache/airflow/const
raints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt

pip install "apache-airflow==%AIRFLOW_VERSION%" --constraint


%CONSTRAINT_URL%

✅ 1.1.5. Initialize Airflow Database:

bash
CopyEdit
airflow db init

✅ 1.1.6. Start Airflow Services:

1. Start Scheduler:

bash
CopyEdit
airflow scheduler

2. Open New Terminal and Start Webserver:

bash
CopyEdit
airflow webserver -p 8081

3. Verify:
o Visit: http://localhost:8081

⚡ 1.2 Install Elasticsearch, Kibana, PostgreSQL, and


pgAdmin 4
Step 3: Install Elasticsearch & Kibana

✅ 1.2.1. Install Elasticsearch:

1. Download Elasticsearch from Elastic Downloads.


2. Extract ZIP to C:\elasticsearch.
3. Configure Elasticsearch (C:\elasticsearch\config\elasticsearch.yml):

yaml
CopyEdit
network.host: localhost
http.port: 9200

4. Start Elasticsearch:

bash
CopyEdit
C:\elasticsearch\bin\elasticsearch.bat

5. Verify:
o Visit: http://localhost:9200

✅ 1.2.2. Install Kibana:

1. Download Kibana from Elastic Downloads.


2. Extract ZIP to C:\kibana.
3. Configure Kibana (C:\kibana\config\kibana.yml):

yaml
CopyEdit
server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

4. Start Kibana:

bash
CopyEdit
C:\kibana\bin\kibana.bat

5. Verify:
o Visit: http://localhost:5601

Step 4: Install PostgreSQL & pgAdmin 4

✅ 1.2.3. Install PostgreSQL:

1. Download PostgreSQL from PostgreSQL Downloads.


2. Run the installer and follow steps:
o Set a password for the postgres user.
o Default port: 5432
3. Verify installation:

bash
CopyEdit
psql -U postgres

✅ 1.2.4. Install pgAdmin 4:

1. Download from pgAdmin Downloads.


2. Install and launch pgAdmin.
3. Connect to PostgreSQL using:
o Host: localhost
o Port: 5432
o Username: postgres
o Password: (set during installation)

✅ 1.2.5. Create Databases & Users in pgAdmin:

1. In pgAdmin, right-click on Databases → Create → Database.


2. Create a new user:
o Navigate to Login/Group Roles → Create.
o Set permissions and assign a password.

You might also like