YARN- Empowering The Hadoop Functionalities

YARN (Yet Another Resource Navigator) was introduced in the second version of Hadoop and this is a technology to manage clusters. However, at the time of launch, Apache Software Foundation described it as a redesigned resource manager, but now it is known as a large-scale distributed operating system, which is used for Big data applications.

YARN provided new capabilities to Apache Hadoop by decoupling resource management and scheduling capabilities. Now, with the help of YARN, interactive queries can be run on Hadoop and data streaming can also be done at the same time. This article describes YARN in detail and it includes YARN architecture, its features, and other functionalities.

An Introduction to YARN                                                                         

YARN is a pre-requisite for Hadoop and provides security, data governance tools, resource management functionality across Hadoop clusters. YARN also extends the power of Hadoop by including new cost-effective processing, and linear-scale storage of beneficial technologies. A consistent framework is provided to developers and ISVs to write data, access applications which can run in Hadoop. Through following listed features YARN enhances the Hadoop capabilities:

1. Scalability: Processing power of data center keep on increasing rapidly. Resource manager of YARN keeps on focusing exclusively on scheduling and can manage many clusters and thousands of nodes as a result of which they can manage petabytes of data.
2. Compatibility: Without any disruption to the Hadoop1 processes all existing Map-Reduce applications can run YARN.
3. Cluster Utilization: Dynamic cluster allocation, used in YARN improves cluster utilization. It is far better than static Map Reduce technologies used in earlier Hadoop versions
4. Multi-tenancy: This feature improves ROI of the organization by allowing multiple engine access.

Components of YARN

YARN can split the job responsibilities of Job or Task Tracker into separate entities which are listed below:

• Resource Manager
• Application Manager
• Node Manager
• A dedicated container for every Node Manager

Here among above-listed components resource manager works as the master node of YARN and is responsible to take resource inventory and can run Scheduler like important and critical services. Resource managers can allocate required resources to the running applications.

Read: Hadoop Developer And Architect: Roles and Responsibilities

As it does not track and monitor the application status so it is a pure scheduler. So,the resource manager is basically used to manage clusters of distributed applications of Hadoop YARN.

Resource Manager works with an Application manager and node managers present on every node in the following way:

1. Resource manager instructions are followed by Node managers and they then manage the resources of a single node.
2. Application Managers can negotiate the resources of the resource manager and can coordinate with node managers by starting containers.

Here in this architecture of Resource manager again has four components with the help of which it executes its responsibilities perfectly. These components are:

• Resource Scheduler: Resource scheduler basically schedule all running activities of the applications and decide that which activity needs to run, when it must run, and for how long it will run?
• Application Master Liveness Monitor: Application master is responsible to launch, monitor and requesting containers for any specific application
• Node Manager Liveness Monitor: It tracks the available resources for data processing on its slave nodes and send on the basis of this information, they can send the reports to Resource Manager about all available resources.
• Several Event Handler: It handles and manages all events that happen during any application execution.

Resource management is a great feature which is present in YARN and it was launched to fulfill following tasks:

• To guarantee a fixed time to complete even the critical tasks
• Proper and reasonable cluster scheduling by allocating resources in a fair manner to each user group
• To prevent other users to access the clusters

A Walkthrough of Application Execution in YARN

By far it has been clear that YARN is responsible to run and schedule the applications so that they can complete their task in a particular time frame. Here it is obvious that any application runs or executes in following steps:

Read: How Long Does It Take To Learn hadoop?

• Submission of Application
• Application Master Instance Bootstrapping
• Execution of Application through application master instance

In the following diagram, a brief introduction of application execution is shown: Above shown steps are listed below for application execution:

1. To launch the specific application, master an application, and its specifications are submitted by the client.
2. Resource manager starts the application master to launch any specific application.
3. Application Master registers with Resource manager through which the client program can query resource manager and it can directly communicate with its application master.
4. During operation execution appropriate resource containers are negotiated by application master by following resource-request protocol.
5. After allocating appropriate container, application master launches it and provide its specification to the Node Manager. The launch specifications include all necessary information through which Application master can communicate properly with containers.
6. The code which executes within the container provides all information of the application like its status, progress status to application master through the application-specific.
7. During application execution direct communication between application master and program executes.
8. After completion of application execution, it deregisters itself with the resource manager and shuts down the process.

The complete process of Application Startup can also be represented in the following manner: As per above diagram, there are three below listed actors of YARN:

1. Job Submitter
2. Resource Manager
3. Node Manager

The complete process can be summarized as:

1. An application is submitted by the client
2. A container is allocated by Resource Manager
3. Resource Manager contacts related Node Manager
4. Containers are launched by Node Manager
5. Application Master is executed by Containers

A single application is responsible for the execution of Application Master. It asks for containers from resource scheduler and executes specific programs on the containers. So, we can say that Resource Manager is the core component of YARN and it occupies the role of Job tracker of MR version1. It is the central controlling authority for managing resources and allocating them to the proper and appropriate application. Through two main components named: Scheduler and ApplicationManager and it allocates resources to the applications.

RM and Components Interfacing

Following services are used for the interaction of Resource Manager and other components:

Read: Scala Tutorial Guide for Begginner

1. ClientService: All RPC interfaces are handled by this component like application submission, termination, clustering and other.
2. AdminService: This service ensures that all admin requests will be satisfied

• Resorce Tracker Service: All nodes information are forwarded to YARN scheduler

1. NodesListManager: All valid and excluded nodes are managed and are responsible to read host configuration files and track the application progress.

Final Words:

YARN has provided an exclusive feature to Hadoop system. In Hadoop version1, it was not able to manage the resources properly, and the user often finds it difficult to allocate resources properly. Through YARN now the scheduling and allocating the resource has become easier and the complete processing speed has been enhanced.

Through its various components, it can dynamically allocate various resources and schedule the application processing. For large volume data processing, it is quite necessary to manage the available resources properly so that every application can leverage them.

HDFS separation from Map Reduce has made the Hadoop environment more efficient and quicker. To know more about YARN and its capabilities you should join Hadoop training and certification program at JanBask right away.

FaceBook Twitter Google+ LinkedIn Pinterest Email