Scribd
Upload
54 views9 pages

Setup Single Node Cluster

The document discusses three modes for setting up a Hive metastore: embedded Derby metastore, local Postgres metastore, and remote Postgres metastore on a different machine. It also provides steps for setting up the Spark web UI and application history server.

Uploaded by

praja
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
54 views9 pages

Setup Single Node Cluster

The document discusses three modes for setting up a Hive metastore: embedded Derby metastore, local Postgres metastore, and remote Postgres metastore on a different machine. It also provides steps for setting up the Spark web UI and application history server.

Uploaded by

praja
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

Single Node Cluster

Set up

Copyright [email protected] - All Rights Reserved 1


Subscribe at Learn-Spark.info
Learn-Spark.info
Sibaram Kumar
Sr Data Engineer | Spark | Big Data
Linkedin: in/SibaramKumar/
Copyright [email protected] - All Rights Reserved
Subscribe at Learn-Spark.info 3
3 Modes to Set up
Hive Metastore

Copyright [email protected] - All Rights Reserved 4


Subscribe at Learn-Spark.info
Mode #1

Embedded Derby Metastore :

Localhost

Hive Service JVM

Drive Metastore Derby

Copyright [email protected] - All Rights Reserved 5


Subscribe at Learn-Spark.info
Local Metastore : Mode # 2

Localhost

Hive Service JVM Postgress Service JVM

Drive Metastore

Postgress
Drive Metastore
//localhost:6432/metastore

Copyright [email protected] - All Rights Reserved 6


Subscribe at Learn-Spark.info
Remote Metastore: Mode #3

Localhost Different Machine(1.2.3.4)

Hive Service JVM

Metastore
Drive
Server JVM
Postgress
Metastore
Drive
Server JVM
/432/metastore //1.2.3.4:6432/metastore

Copyright [email protected] - All Rights Reserved 7


Subscribe at Learn-Spark.info
Localhost

Hive
Metastore

PostgreSQL
Image

PostgreSQL Client

Copyright [email protected] - All Rights Reserved 8


Subscribe at Learn-Spark.info
Set up Web UI:
1. Run a job using spark-submit Command.
2. Get the Port when the applications are submitted. <8088>
3. Set up the Firewall to allow port 8088 at our instance.
4. Open a Browser and test. <PublicIP>:8088
5. At this time, we are still unable to access the Application History.
6. Go to History URL and check the Port for Application History. <18080>
7. Also Initiate the Service : /opt/spark3/sbin/start-history-server.sh
8. Allow port 18080 in the Firewall at our instance.
9. Access the Application History. <PublicIP:18080>

Copyright [email protected] - All Rights Reserved 9


Subscribe at Learn-Spark.info

You might also like