VCore Configuration In Hadoop

Just like memory, vcores, the abbreviation for virtual cores, is another type of resource in Hadoop cluster. It is the abstraction of the ability of CPU.

We can see from class 'org.apache.hadoop.yarn.api.records.Resource', which is the abstraction of resource in YARN, that memory and vcores are equally treated in Hadoop:

int getMemory();

void setMemory(int memory);

int getVirtualCores();

void setVirtualCores(int vCores);

Here are all the relative params for vcores in Hadoop configuration files:

Name	Description
mapreduce.map.cpu.vcores	The number of virtual cores required for each map task.
mapreduce.reduce.cpu.vcores	The number of virtual cores required for each reduce task.
yarn.app.mapreduce.am.resource.cpu-vcores	The number of virtual CPU cores the MR AppMaster needs.
yarn.scheduler.maximum-allocation-vcores	The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value.
yarn.scheduler.minimum-allocation-vcores	The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.
yarn.nodemanager.resource.cpu-vcores	Number of CPU cores that can be allocated for containers.

The first three params are configured in mapred-site.xml, the rest are in yarn-site.xml.

mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are easy to understand, which represents the number of vcores providing to each map or reduce task respectively.

yarn.app.mapreduce.am.resource.cpu-vcores stands for the number of vcores for MapReduce Application Master Node. (MR ApplicationMaster: A new framework for MR on YARN, which cooperates with NodeManager with the resource retrieved from ResourceManager)

The last three params is well-explained in the description section. As for yarn.nodemanager.resource.cpu-vcores, it should be set to the number of processors in a single node in most cases (One virtual core should correspond to one physical processor). At the same time, it is recommended that yarn.scheduler.maximum-allocation-vcores be set no more than yarn.nodemanager.resource.cpu-vcores.

For instance, if a single node has 24 processors, then an appropriate set of params can be set as follows:

Name	Value
mapreduce.map.cpu.vcores	1
mapreduce.reduce.cpu.vcores	1
yarn.app.mapreduce.am.resource.cpu-vcores	1
yarn.scheduler.maximum-allocation-vcores	24
yarn.scheduler.minimum-allocation-vcores	1
yarn.nodemanager.resource.cpu-vcores	24

Relative Posts:
·  Memory Configuration In Hadoop

© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu