Apache Error #28 – No space left on device

The Apache server on my EC2 instance stopped suddenly and refuses to restart. Repeating in the error log is this misleading error:

(28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

A quick check with df -h and repquota -v indicates that there are plenty of disk left and no user is over his quota. The culprit turns out to be left over semaphore arrays. To check whether they exist on your system, use

ipcs -s | grep apache

In my case this returns a shocking 128 left overs. To clear them, run

ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/s+/); print `ipcrm sem $a[1]`}'
Apache started without anymore complains.

What’s a semaphore? Wikipedia defines it as a protected variable or abstract data type that constitutes a classic method of controlling access by several processes to a common resource in a parallel programming environment. And a user at http://www.virtualmin.com/node/13303 suggest that virtualmin’s logrotate is casuing them to pile up:

logrotate is furiously trying to stop and restart httpd during the logrotate, which happens weekly at this time. I note that by default the logrotate set up here does a restart of apache, and when this happens over and over again in quick succession sometimes apache doesn’t finish dying before the next restart, and then the semaphores pile up until you run out of semaphore space.

That’s logical. Sounds like a short sleep after each log is archived might solve this issue to a certain extent, but that’s for another day.

Additional resource: http://carlosrivero.com/fix-apache—no-space-left-on-device-couldnt-create-accept-lock

Leave a Reply

Your email address will not be published. Required fields are marked *