Friday, July 6, 2018

udf包污染导致tez任务在am端log4j stackoverlow问题排查过程

发现有log4j-over-slf4j,和slf4j-log4j12, 两者会循环引用,导致log4j相关的stackoverflow.

报错如下:
Exception in thread "main" java.lang.StackOverflowError
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:39)
at org.apache.log4j.LogManager.getLogger(LogManager.java:45)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.apache.log4j.Category.<init>(Category.java:57)
at org.apache.log4j.Logger.<init>(Logger.java:37)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:43)
at org.apache.log4j.LogManager.getLogger(LogManager.java:45)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.apache.log4j.Category.<init>(Category.java:57)
at org.apache.log4j.Logger.<init>(Logger.java:37)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:43)
at org.apache.log4j.LogManager.getLogger(LogManager.java:45)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.apache.log4j.Category.<init>(Category.java:57)
at org.apache.log4j.Logger.<init>(Logger.java:37)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:43)
at org.apache.log4j.LogManager.getLogger(LogManager.java:45)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.apache.log4j.Category.<init>(Category.java:57)
at org.apache.log4j.Logger.<init>(Logger.java:37)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:43)
at org.apache.log4j.LogManager.getLogger(LogManager.java:45)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.apache.log4j.Category.<init>(Category.java:57)
at org.apache.log4j.Logger.<init>(Logger.java:37)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:43)
at org.apache.log4j.LogManager.getLogger(LogManager.java:45)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)

exclude掉log4j-over-slf4j和log-to-slf4j即可。

反思:在udf中引用任何包后都要通过`mvn dependency:tree`和`jar -tf ...`命令查看引用的包包含的package和namespace,以防止后期的包污染问题扩散。

REF 

16 comments:

  1. Thanks for such a great article here. I was searching for something like this for quite a long time and at last, I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays. Well written article Thank You for Sharing with Us pmp training centers in chennai| pmp training in velachery | project management courses in chennai |pmp training in chennai | pmp training institute in chennai

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. I got what i am seraching from last few days in your Blog. I hope you will share more info about it. Please keep sharing.
    Laptop Service center in Ameerpet
    Dell Service center in Ameerpet
    HP Service center in Ameerpet
    Lenovo Service center in Ameerpet

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. thanks for sharing your interesting idea and thought is very nice and it's helpfull contant
    BA 3rd year Result 2019-20
    AMU BCOM 3rd Year Result 2020
    MDSU BCOM 3rd Year Result 2020

    ReplyDelete
  10. Hey, Wow all the posts are very informative for the people who visit this site. Good work! We also have a Website. Please feel free to visit our site. Thank you for sharing.
    Be Your Own Boss! If you're looking for a change in your work prospects, then let's prepare for your career from here!!!
    Self Employment | Women Development | Information Technology | Engineering Courses

    ReplyDelete
  11. I think you did an awesome job ba exam time table explaining it. Sure beats having to research bsc 3rd year time table it on my own. Thanks

    ReplyDelete
  12. Bollywood News in Hindi - Check out the latest Bollywood news, new Hindi movie reviews, box office collection updates and latest Hindi movie videos. Download free HD wallpapers of Bollywood celebrities and recent movies and much more on Bollywood Hungama.
    The Family Man Season 2
    Laal Singh Chaddha full movie

    ReplyDelete
  13. Awesome blog, thank you so much for sharing such an amazing information with us. Visit Import Globals for Vietnam Import and Export Data Services at Vietnam customs.
    Vietnam Import Data

    ReplyDelete