What is Semaphore? Why it’s important for a Server Admin?

What is Semaphore?

Well, we need a discussion on Semaphore concept under Linux as it plays major role in a Multi Processing System. Some time we need to review the current Semaphore status on the server and need to kill unwanted/unused entries to free up memory allocation on the system. Okay, let’s start!

Semaphores are IPCs. IPC means Inter Process Communication Systems. These are used to allow different processes to communicate with each other.

See what wiki says;

In computer science, a semaphore is a variable or abstract data type used to control access to a common resource by multiple processes in a concurrent system such as a multi programming operating system.

Semaphores are used for communicating between the active processes of a certain application, says Apache or some other applications. As a Linux Server Admin, you must have faced a known common error with Semaphore while restarting the web server Apache. Yeah, the same “No space left on device” error!

I know you have the solution. Check this tutorial by Heba to get an idea on Semaphore error with Apache:

Apache error: No space left on device: mod_rewrite: Parent could not create RewriteLock.

Considering the scenario of Apache, Semaphores are used to communicate between the Apache parent and child processes. If Apache can’t write things down, then it can not communicate properly with all of the process it starts. So it won’t start the process.

Semaphores mark memory location as locked (not free) and may not release after completing the actual process. In most case, if the parent process dies without killing its child processes properly.

This condition can happen in a very busy server with high uptime.

Commonly showing errors are pasted below:

[error] (28)No space left on device: Cannot create SSLMutex

[crit] (28)No space left on device: mod_rewrite: Parent could not create RewriteLock file /usr/local/apache/logs/rewrite_lock Configuration Failed

[emerg] (28)No space left on device: Couldn't create accept lock
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[emerg] (28)No space left on device: Couldn't create accept lock

Yeah, we need to restart the Apache, however, the semaphores won’t start the Apache server. Okay, plan to kill them!!

Command to check Semaphore details

Dear Shell Lovers, the solution is there. Execute the following command to check the Semaphores details from the Linux command line.

ipcs -s

The above command will display the status of Semaphores on the server.

See the example pasted below

# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 256049154  nobody     600        1         
0x00000000 256081923  nobody     600        1         
0x00000000 266403844  nobody     600        1         
0x00000000 266436613  nobody     600        1         
0x00000000 266469382  nobody     600        1         
0x00000000 266502151  nobody     600        1         
0x00000000 266534920  nobody     600        1

Note the user and Semaphore ID for those processes listed on the above command. If Apache causing problem with Semaphore, you can see a lot of processes there with user apache (or nobody). We need to kill those processes to start Apache properly.

How to kill a Semaphore process?

To kill a Semaphore process, you can use the command “ipcrm” along with the switch -s. See the syntax pasted below:

ipcrm -s 

To kill all Semaphores listed for the user Apache, first you must know the user name of Apache server. Then execute the following command:

for i in `ipcs -s | grep nobody | awk {'print $2'}`; do (ipcrm -s $i); done

Or

for i in `ipcs -s | awk '/nobody/ {print $2}'`; do (ipcrm -s $i); done

Or

ipcs -s | grep nobody | awk '{print $2}' | xargs -n 1 ipcrm sem

Considering the user for Apache is nobody. It will remove all Semaphores for Apache user. Now you will be able start the Apache server normally.

How to list current Semaphore configuration details from command line?

Use the following command to print current Semaphore’s configuration details:

ipcs -l

See the sample output pasted below

# ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 10922
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

How to change the Semaphore parameters?

If the problem occurring continuously, we need to increase its value. This can be done by increasing the following entries in /etc/sysctl.conf

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

Then run sysctl -p to pick up the new changes.

Done!!!

I am sure you are now much familiar with Semaphore in Linux. Thanks!

Please let me know your suggestions as comments.

People also like to read..