Log-Aggregation is a centralized management of logs in all NodeManager nodes provided by YARN. It will aggregate and upload finished container or task's log to HDFS. The related configurations are as follows:
Logs from NodeManager can be seen from the YARN monitor webpage:
References:
© 2014-2017 jason4zhu.blogspot.com All Rights Reserved
If transfering, please annotate the origin: Jason4Zhu
| name | value | description |
| yarn.log-aggregation-enable | false | Whether to enable log aggregation |
| yarn.log-aggregation.retain-seconds | -1 | How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node. |
| yarn.log-aggregation.retain-check-interval-seconds | -1 | How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node. |
| yarn.nodemanager.remote-app-log-dir | /tmp/logs | Where to aggregate logs to. |
| yarn.nodemanager.remote-app-log-dir-suffix | logs | The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam} |
Logs from NodeManager can be seen from the YARN monitor webpage:
References:
If transfering, please annotate the origin: Jason4Zhu
thanks for sharing such details about bigdata and hadoop. Big Data Hadoop Online Course
ReplyDeleteNice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating Big data training
ReplyDelete