Over the last few weeks, I have been on a learning curve about Apache, MySQL, Memory, Virtual Memory and Swap files.
Here’s the thing:
I wasn’t really interested in any of them as already have a busy job, family and other stuff going on.
My problem though one of my VPS servers kept crashing every day or so. This was a journey believe me.
To the eye it looked like MySQL was crashing out all the time, Jetpack monitor would send me alerts saying site was down and I would see this:
Swap memory and using HTOP
I went down a few rabbit holes. Increasing swap memory seemed to be a popular option so I tried that. I increased RAM on my VPS as well, because my new swap file didn’t help.
It was hard for me to understand why MySQL kept crashing when not much seemed to be going on with it. The site on the server was perhaps getting 400-500 visits a day which is nothing really considering the specs of the server I was running.
After a few repeated times of it happening, and when I had a bit of time to investigate and use memory tools like HTOP, the light began turn on.
There was one key finding after my focus on MySQL crashing, it finally dawned on me that it was Apache getting overloaded (but not stopping), causing MySQL to be starved of memory and crashing.
Just to explain I am not a Linux guru so although I can Google stuff and eventually figure things out, it takes me some time. I am not your Linux Guru.
Now I had a clue, a reason to keep digging and head off on some more wild goose chases. I looked at and researched about the memory settings of Apache. I tried to tune my MPM_Prefork settings.
I read articles before going to sleep about tuning Apache, tuning MySQL, a true overwhelm of information.
I learned that often brute force attacks on xmlrpc.php could cause this kind of issue, but Jetpack was pretty good at stopping them.
Reading Apache Access logs
Finally I had some time to dig into my Apache access logs and look at what IP’s were hitting the site hard before it crashed and what they were doing.
What I saw coincided with what I had seen on HTOP and read about – a large number of Apache connections that were flooding the server and hitting xmlrpc.php.
An IP from Russia was hitting my site pretty hard just a minute or two before the alerts coming through from Jetpack saying my site had crashed.
Blocking out the offenders via Firewall
I had no idea of how to use a Firewall within the Google Cloud but it is pretty simple.
Head along to the Google Cloud Home page and look under VPC network – Firewall Rules:
From there you can make a rule to deny traffic that is coming from the offending IP or IP range:
Show the count of IP’s that have been hitting your server. You can look at any that stand out.
tail -n 500 /var/log/apache2/access.log | cut -d' ' -f1 | sort | uniq -c | sort -gr
Or do in real time
watch "tail -n 500 /var/log/apache2/access.log | cut -d' ' -f1 | Check the count of IP addresses connected to your server
That is pretty much it.
The server crashing has stopped for now. No doubt I will be hit again in the future but I have learned a lot more about where to look.
Next time I will just check my access logs and add the offenders to the Firewall.
I hope this helps you get to a satisfactory outcome a lot quicker than what it took me!