How to fix WHM /Cpanel Apache Log Bug - Crashes Intermittently- for Linux newbies
I’m more of a PHP editor than a ’system admin’. I use a PC and am NOT a Linux operating system expert.
So, I hate hacking away at Linux. But, I learned a lesson about CPanel/WHM the other day:
WHM/Cpanel has a bug that prevents it from rotating the Apache logs properly.
This can then cause your site to crash if they get too big. So,
If :
- your WHM/CPanel server is crashing mysteriously, and
- the crashes are becoming more frequent over time, and
- your application worked fine on another server, and
- your hosting company is not being helpful
Then, it might be worth checking your Apache log files on your (linux) web server.
Any web developer worth their salt should be able to do this, so here’s how.
How to Check/Fix the CPanel Apache-Log Bug
Note: This post is about the WHM/Cpanel bug. So, ideally, you should have WHM or CPanel on your system for it to be relavent. If not, you can still be a voyer, i don’t mind.
Requirements:
- Basic ability to login to your server and run command line commands as root
- A quick read of some nice linux shell tutorial.
Tip: When hacking away at Linux commands I like to have a notepad open on my PC. Then, I write the command as text in my notepad first. Then, copy and paste it into the command line window. This means that (a) I can keep a record of what I have been getting up to; (b) I can go back to these notes long after I forgotten how to do the task in hand.
Login to SSH
First, login to the server’s command line window using an SSH or Telnet client. (N.B. SSH is far more secure).
Become the root user if you are not already.
Type “su“:
# su
This will ask you for the root password.
(The hash in the line above (”#“) merely represents the command line prompt in the examples in this blog post)
Check Your Apache Logs
We want to see what’s happening to our logs. So, navigate to the logs folder and see what we have got.
# cd /usr/local/apache/logs/
This, roughly in Windows Explorer terms, means “browse to the folder where Apache logs are usaually stored”. i.e. make the ‘current directory’ of the terminal become the directory named “/usr/local/apache/logs/”.
Tip: For most commands in Linux, if you are not sure what it does, you can type the word “man” then the name of command e.g “man cd“. This tells Linux to give you the low-down. You will see the “manual” for it. When the manual shows-up, you can scroll the text using your up and down arrow keys. Press the “Q” key on your keyboard to exit from the manual and go back to the command line prompt. An alternative way to see info on a command is to browse the web for the phrase “man <command>“. Some kind person would undoubtedly have put it on the web in easy-to-read format.
Looking At the Log Folder’s Contents
OK. So we are now in the “/usr/local/apache/logs/” directory.
Type the following to see what’s in that directory:
# ls -s -h -l -R
The ls (lower-case L, then S) command means “list the files in the current directory”. The options after the ls command mean the following:
- -s means show the size of the items in the directory
- -h means make the output human friendly (file sizes in easy-to-understand units like kb, Gb etc as opposed to ‘measures of blocks’)
- -l means something i cannot remember right now
- -R means look in directories recursively
I got output similar to this:
.:total 2.3G 131M access_log 0 audit_log 2.1G error_log 0 fpcgisock 4.0K httpd.pid 0 modsec_audit.log 0 modsec_debug_log8.0K ssl_engine_log 0 ssl_scache.dir4.0K ssl_scache.pag 24K suexec_log4.0K suexec_log.offset ...
Note:
- for me, the size of the error log is a MASSIVE 2.1G.
- there don’t appear to be any rotated log files
(indicated by the lack of files called access_log.0, access_log.1 etc)
OK, so now we suspect that our apache log files are NOT being rotated.
Side Note / Caveat - The advice that I got from my hosting company when i mentioned the log rotatation issue was as follows:
that “bug” with the cpanel logrotation is not actually a
bug but intended behaviour. The apache error logs should not contain
enough to require rotation and the xferlog should not be being used at
all. It doesn’t hurt to rotate them however but I would advise against
making manual changes to your cpanel jobs/scripts. It is much more
likely to break things than improve things.
Let’s Check if the Cpanel/WHM Apache-Log Bug Is the Cause
Its worth reading this article that offers a great introduction to logs, log rotation and log watching on Linux. In summary, it explains that Linux distributions include a tool called logrotate that rotates log files. Logrotate uses various configuration scripts to tell it how to do its stuff.
” /etc/logrotate.d/httpd” is commonly the name of the configuration script that defines “how Apache logs should be rotated”.
i.e its called “httpd” and it lives inside the “/etc/logrotate.d/” folder.
(I know…its a bit weird to see the .d on the end of a folder name, makes it look like a file…but…hey!)
Lets have a look inside that folder:
# cd /etc/logrotate.d
That changes to the directory named “/etc/logrotate.d/”
# ls
As you know, ls lists the contents of that directory (albeit without any extra options). You will see lots of files that have been dropped in there by many of the system’s packages during their installation/set-up.
Check to see if you can spot the “httpd” file.
If you CAN see it - then you are following my footsteps and we should take a look inside it to see if it is written correctly.
If you CAN’T see it - i never had that problem so have no idea how to help you on this.
Tip - Editing Files via SSH/Telnet: I use a tool called vim to look inside, edit and save text files from within my SSH terminal window. It is probably, but not definitely, installed on your Linux server already. To find out if you have it…type “vimtutor” at the command prompt. If it is installed properly on your server, a tutorial will pop-up telling you how to use it. Of course, if you already know how to edit files in your SSH window, then just use your preferred tool.
Some vim tips:
- i - insert
- ESC - to get out of edit mode
- :wq! - write and exit
Open up the apache logging configuration file with vim:
# vim httpd
this is what my /etc/logrotate.d/httpd file looked like:
/var/log/httpd/*log {
missingok
notifempty
sharedscripts
postrotate/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true endscript }
If yours look similar then you probably should fix it.
As this web page and this web page explain, there are two things wrong with this file:
- Issue 1 - The line
/var/log/httpd/*log
is saying,
“look for files, ending in log, that are in the /var/log/httpd/ directory”.
But, we know that they are in the /usr/local/apache/logs/directory. We saw them with our own eyes! - Issue 2 - The line
/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
is trying to use httpd.pid in /var/run/ .
But, on many servers its in /usr/local/apache/logs/
I know issue 1 is relevant to me, because I have just seen it.
We can check issue 2 by navigating to each directory on our system and seeing which one actually has httpd.pid in it.
So, exit from the vim window, (if it is still open), by typing : (colon) then Q, like:
:q
Now lets investigate whether or not our system matches the system implied by that config file:
# cd /var/run/
# ls
We should see output for the /var/run/ directory. Can we see any httpd.pid there?
Or is it in /usr/local/apache/logs/?:
# cd /usr/local/apache/logs/
# ls
For me, it was in the latter.
So I have convinced myself that these two random forum posters on the internet might have a valid point.
So, lets fix those lines according to their recommendations.
Editing the /etc/logrotate.d/httpd file
Lets go back to the log rotation config file for the Apache logs:
# cd /etc/logrotate.d
# ls
And open it up:
# vim httpd
Save a Backup
Its wise to copy the original contents of any important file that you edit on the server.
I do this by clicking on the icon in the top left hand corner of my Putty SSH tool.
Then selecting -> Copy All To clipboard
Then, pasting that into my windows notepad and saving it for posterity. There are probably better ways of doing it.
Write the New Config
I find it easier to write stuff in my desktop PC’s text file editor. Then, I copy and paste it into the vim/SSH window.
Most importantly we should fix those two lines to the appropriate values.
But while we are here, we may as well look into the other log rotation options possible and add any that take our fancy.
I decided to make my final , amended config look like this:
/usr/local/apache/logs/*log {
#MY OPTIONS ADDED #do it weekly weekly#number of generations of file before delete rotate 52 # compress gzip by default - delay actual compression til next rotation so it don't take ages compress delaycompress #to add a date to file YYYYMMDD e.g. 20101214 dateext #to prevent 50 GB data being emailed to me nomail #decided to leave this commented for now in case i use up my 52 rotations too quickly before i download them #size 10M missingok notifempty sharedscripts postrotate /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true endscript }
You can do what you think is fit for your requirements.
If you have edited this in notepad…just copy and paste it into vim. Check and tidy the file’s contents. Then save it and quit by typing:
:wq!
Do a trial run - to debug the Apache Log Rotation Config Script
OK. For sanity, lets check to see if the config script is going to work:
# logrotate -f -d /etc/logrotate.d/httpd
As our friend tells us, the logrotate tool command expects to be passed the path to the log config script to work with. So, we pass it the location of our newly edited file.
We also know, from the logrotate man page that we can pass
- the -d option to do a dry run of the process
- the -f option to force it to do the rotating, even if it a file is not due to be rotated.
The output of the logrotate -d command is pretty self-explanatory .
OK - Lets do it for Real
Before I do it for real. I want to go back to my logs folder and keep a record of what they look like. That way i can take a before and after snapshot, so I know what’s gwan on.
# cd /usr/local/apache/logs/# ls -s -h -l -R
Now do the actual proper command:
# logrotate -f -v /etc/logrotate.d/httpd
In this case,
- We pass the -f option to force it to rotate.
- But, we do NOT pass the -d option, ‘cos we aint debugging.
- We pass the -v option. So, we still get some verbose output.
Wow - so something should have happened then.
Lets see our logs folder now:
# cd /usr/local/apache/logs/# ls -s -h -l -R
You should now be looking at your newly rotated log files.
Wooo hooo!
There we go. Job done. Lesson learnt. Everyone’s’ happy.
There are a few more fixes that can be done while you’re at it.
Please comment if you spot any typos.
Here is the back ground to this story if your interested:
The Story of the Symptoms
I moved one of my web sites over from the PLESK web management platform to WHM/Cpanel. Shortly after the move, the server had started to begin hanging then crashing every week or so.
I’d check the site on a lazy Saturday afternoon, sigh, then reboot the server. Because this tended to happen at off-peak times, I suspected that some cron job might be causing problems.
Over time, the crashes became more frequent. I started to feel like this so-called automatic machine was requiring me to be constantly pushing it to make it work.
I took the advice of my hosting company who suggested that I adjust the :
WHM -> Service Configuration -> Apache Configuration -> Memory Usage Restrictions
(It amends Apache’s memory limit to that suitable for the amount of RAM in the server).
I also looked into adjusting the values for Apache’s the ‘max requests’ setting - but that seemed like a lot of tinkering (and sys admin researching) that I just could never find the time for. (Probably cos i should have been looking into max clients instead!).
This page is handy if you want to look into various server tweaks:
http://www.howtoforge.com/configuring_apache_for_maximum_performance
Anyway
The other day, things got worse. The site crashed. I rebooted. It crashed again. I rebooted. It crashed.
I had just made a minor refactoring of a major component. Despite taking extra precautions to make my changes, I suspected that I had let loose a gremlin in the web application code.
I also considered that it may have been caused by intensive database queries because the crash seemed to occur on pages that was heavy on the MySQL.
I really did not know what it was, so I took the site offline and then went to sleep (i really needed it).
The next day I woke up with a plan of action.
1 . Do a DB dump and backup everything else I could on the problematic server.
2. Run the dynamic web application from my test server, on my pc - then output as many flat files as logically possible
3. Upload as much of this cached content over to a folder in one of my other sites on a different servers.
4. Proxy any ‘Not Found’ requests from ‘troublesome server’ over to ‘live secondary server’ to be served as flat files.
5. Do some major tweaking and messing about on the troublesome server til i fixed the problem or gave up.
The idea was to keep a ‘jury rig’ site going whilst I looked into the issues.
So, I went onto step one, doing some backing up. Which is always wise if you are going to start tinkering.
I went beyond my normal backups (database dump, site logs) and decided to backup and download the Apache logs too.
This is when I noticed that my apache error log was 2 Gigabytes and the access log was massive too.
I am not a linux person or a system admin - but i knew that something was wrong. I inspected the files and noticed that they had covered the complete lifetime of the server. Hence, there was no log rotation going on.
I considered just renaming the log file, a sort of manual rotation, but had no idea what effect that would have. I did not know if it was going to make things worse. So, I did a bit of research into this issue - trying to learn about logging in Cpanel and WHM. Basically, asking things like “what is the -ssl_data_log in domlogs” and “is it OK to delete my Apache log files in WHM”.
During my searches I was seeing that more and more other people had been suffering from log problems with WHM and eventually discovered that the long running bug that Cpanel has - it does not rotate the apache logs properly.
After I ran the fix (listed above for those that just want answers), I put the site back up again and…. It worked.
So, lessons learned:
* Big logs can crash the server
* WHM /Cpanel has a bug where it does not rotate certain logs. You HAVE to fix it if your site is going to last online for a reasonable time.
