Scribd
Upload
17 views

Server Setup

The document outlines the setup of a server including functional user test, data store location, and tools installed in standalone mode. It describes the installation of Filebeat, Apache Kafka requiring Zookeeper, Apache Spark using Scala for an ETL pipeline with Kafka as the data source and Elasticsearch as the sink, Elasticsearch, and Kibana on the server. All tools were installed from their respective websites using provided documentation.

Uploaded by

Sateesh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
17 views

Server Setup

The document outlines the setup of a server including functional user test, data store location, and tools installed in standalone mode. It describes the installation of Filebeat, Apache Kafka requiring Zookeeper, Apache Spark using Scala for an ETL pipeline with Kafka as the data source and Elasticsearch as the sink, Elasticsearch, and Kibana on the server. All tools were installed from their respective websites using provided documentation.

Uploaded by

Sateesh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 2

Server Setup

Server info:
 Functional User: test
 With functional user we are not able to install tools using package manager.
 No tool is installed as service, need to start service again after server restart.
 Data store: /repo/test/
 All tools are installed in standalone mode.

System Design:

Tools:
 Filebeat 7.7
 Kafka 2.5
 Apache Spark 2.4
 Elasticsearch 7.7
 Kibana 7.7
 Java 1.8
 Scala 2.11
 SBT
Installation:
Filebeat
 https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html
 Run command: : ./filebeat -e
 Config file: filebeat.yml
Apache Kafka
 Kafka needs zookeeper for leader coordination
 https://kafka.apache.org/documentation/#quickstart
 Start zookeeper and then Kafka
 Broker port: 9200
Apache Spark
 The data pipeline is based on Spark structured streaming. The data source is Kafka
and sink are Elasticsearch.
 For development, we are using Scala programming language and SBT for
dependency management.
 For installation: https://spark.apache.org/docs/latest/spark-standalone.html
 Run program: sbt run and select respective class
Elasticsearch
 https://www.elastic.co/guide/en/elasticsearch/reference/7.7/targz.html
Kibana
 We have installed basic license.
 https://www.elastic.co/guide/en/kibana/7.7/targz.html

You might also like