Scribd
Upload
58 views6 pages

Managing Resources With Hadoop YARN

YARN manages resources in Hadoop clusters and schedules jobs across nodes. It consists of a ResourceManager and NodeManagers. Administrators configure YARN, submit applications like MapReduce or Spark, monitor progress via the ResourceManager UI, and can adjust resource allocation dynamically based on workload. YARN interacts with HDFS for storage and allows using components like MapReduce, Hive and Spark for processing and analyzing data.

Uploaded by

Kappa Digital
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
58 views6 pages

Managing Resources With Hadoop YARN

YARN manages resources in Hadoop clusters and schedules jobs across nodes. It consists of a ResourceManager and NodeManagers. Administrators configure YARN, submit applications like MapReduce or Spark, monitor progress via the ResourceManager UI, and can adjust resource allocation dynamically based on workload. YARN interacts with HDFS for storage and allows using components like MapReduce, Hive and Spark for processing and analyzing data.

Uploaded by

Kappa Digital
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 6

How to manage resources with hadoop

YARN and interacting with hadoop


ecosystem
the steps to manage resources with Hadoop YARN and interact
with the Hadoop ecosystem:

 Understand the YARN architecture:


YARN consists of two main components - ResourceManager (RM) and
NodeManager (NM). The RM manages resources in the cluster and schedules jobs,
while the NM runs on each node and manages resources on that node.
 Configure YARN:
Configure the yarn-site.xml file with the required parameters such as the maximum
and minimum resources per container, the number of containers per node, and the
allocation strategy.
Cont…………

 Submit applications:
Applications can be submitted to YARN using various frameworks such as
MapReduce, Spark, or Hive. These applications consist of one or more tasks that
YARN will allocate resources for execution.

Monitor applications:
YARN provides a web-based application called the Resource Manager UI to monitor
the progress of applications. It shows the status of running, completing, or failed
applications, as well as information about allocated resources.
Cont……….

 Allocate resources:
YARN automatically allocates resources to applications based on the configured
parameters. It considers the availability of resources, the requested resources by the
application, and the resource requirements specified in the yarn-site.xml file.
 Adjust resource allocation:
If required, you can adjust the resource allocation dynamically based on the
workload of the cluster. You can increase or decrease the number of containers and
adjust the memory and CPU allocation.
Cont……..
 Interact with the Hadoop ecosystem:
YARN allows you to interact with various components of the Hadoop ecosystem,
such as HDFS (Hadoop Distributed File System) for storing data, MapReduce for
processing data, and Hive or Spark for querying and analyzing data.
Conclusion

 By effectively managing resources with Hadoop YARN, you can optimize the
allocation of resources in your cluster and ensure efficient execution of
applications in the Hadoop ecosystem.

 MOST IMPORTANT ASPECT OF RESOURCE ALLOCATOR

You might also like