Deal With Error "A semaphore set has to be created but the system limit for the maximum number of semaphore sets has been exceeded"

There are some time that a specific node in our Hadoop cluster will be stuck at ssh connection procedure, whereas ping command is traversable. When this node recovers by itself, I look through the system messages in '/var/log/messages', where all of the content is:

A semaphore set has to be created but the system limit for the maximum number of semaphore sets has been exceeded

There are some commands that we could apply to check on semaphore status. You can run ipcs -s to list all of the semaphores currently allocated on your system and then use ipcrm -s <id> to remove a semaphore (if you're reasonably sure it's no longer needed). You might also want to track down the program that created them (using information from ipcs -s -i <id>) to make sure it's not leaking semaphores.

Alternatively, you could check the semaphore setting in sysctl(8) via issuing command `sysctl -a | grep kernel.sem`, it is equivalent to invoke command `ipcs -ls`, which would show descriptions to all values:

$ /sbin/sysctl -a | grep kernel.sem
kernel.sem = 250 32000 32 128

$ ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

The preceding error is related with the 'max semaphores system wide', thus we could increase this value by increasing either 'max number of arrays' or 'max semaphores per array'. However, there's something should be noticed, that when revising, we should always validate the equation: (max number of arrays)*(max semaphores per array)=(max semaphores system wide).

In my case, I changed the 'max number of arrays' to 1280, and 'max semaphores system wide' to 320000 correspondingly by appending "kernel.sem = 250 320000 32 1280" to '/etc/sysctl.conf' and execute `/sbin/sysctl -p`.



Reference:
1. Answer from ServerFault
2. Power Panel process fails to start - Parallels
3. 8.5. Setting Semaphore Parameters - RedHat
4. Setting the Kernel Parameters - SYBASE



© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu