Log-Aggregation is a centralized management of logs in all NodeManager nodes provided by YARN. It will aggregate and upload finished container or task's log to HDFS. The related configurations are as follows:
Logs from NodeManager can be seen from the YARN monitor webpage:
References:
© 2014-2017 jason4zhu.blogspot.com All Rights Reserved
If transfering, please annotate the origin: Jason4Zhu
| name | value | description |
| yarn.log-aggregation-enable | false | Whether to enable log aggregation |
| yarn.log-aggregation.retain-seconds | -1 | How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node. |
| yarn.log-aggregation.retain-check-interval-seconds | -1 | How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node. |
| yarn.nodemanager.remote-app-log-dir | /tmp/logs | Where to aggregate logs to. |
| yarn.nodemanager.remote-app-log-dir-suffix | logs | The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam} |
Logs from NodeManager can be seen from the YARN monitor webpage:
References:
If transfering, please annotate the origin: Jason4Zhu
