Hadoop_Troubleshooting: 'AcpSubmitApps' In fair-scheduler.xml is Not Working.

After configuring "AclSubmitApps" for a specific queue in fair-scheduler.xml, I can still submit a hive task by user who is not in "AclSubmitApps" list, which is not expected according to the official document.

The configuration of my testmonitor queue:

<queue name="testmonitor">

  <minResources>10000 mb,20vcores</minResources>

  <maxResources>30000 mb,50vcores</maxResources>

  <maxRunningApps>50</maxRunningApps>

  <weight>1.0</weight>

  <schedulingPolicy>fair</schedulingPolicy>

  <aclSubmitApps>root</aclSubmitApps>

  <minSharePreemptionTimeout>300</minSharePreemptionTimeout>

</queue>

The monitoring status for my hive task:

The solution is simple and easy: Add "aclAdministerApps" to the queue correspondingly:

<queue name="testmonitor">

  <minResources>10000 mb,20vcores</minResources>

  <maxResources>30000 mb,50vcores</maxResources>

  <maxRunningApps>50</maxRunningApps>

  <weight>1.0</weight>

  <schedulingPolicy>fair</schedulingPolicy>

  <aclSubmitApps>root</aclSubmitApps>

  <aclAdministerApps>root</aclAdministerApps>

  <minSharePreemptionTimeout>300</minSharePreemptionTimeout>

</queue>

Then we can check the acl settings of queues via hadoop queue -showacls. In this time, the acl of queue `root.testmonitor` have neither SUBMIT_APPLICATIONS nor ADMINISTER_QUEUE.

When I submit a task in user monitor, The following exception is thrown as expected:

java.io.IOException: Failed to run job : User supertool cannot submit applications to queue root.mbmonitor
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:299)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
...

Honestly, I don't know exactly why we have to add "aclAdministerApps" in order to make it work, the official document says nothing about it, either. If anyone knows the essential reason to this solution, please leave a message, I'd really appreciate it :)


© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu