WordPress brute-force attack detection plugins comparison.

 

Update: We released a new set of benchmarks in October 2015.

 

Two benchmark tests will be performed against the 5 most popular security plugins whose description indicates that they provide protection against brute force attacks, and our plugin NinjaFirewall (WP edition).

The first test will use ApacheBench, the benchmarking tool part of the Apache HTTP server package. It will be a rather quick but totally brutal attack on the WordPress login page, originating from a single IP.
During the second (and much less brutal) test, a distributed attack against the login page will be simulated, from thousands of different IPs. It will be very similar to one of those large attacks that targeted WordPress sites lately.

The 5 plugins are:

  • All In One WP Security
  • Better WP Security
  • BulletProof Security
  • Limit Login Attempts
  • Wordfence Security

The test will also include:

  • NinjaFirewall (WP edition)
    • If you aren’t familiar with it, this is how it works:
Attacker => HTTP server => PHP => NinjaFirewall => WordPress
    • And this is how the above-mentioned plugins and, more generally, all WordPress plugins work:
Attacker => HTTP server => PHP => WordPress => Plugins
    NinjaFirewall sits between the attacker and WordPress. It can filter requests before they reach the blog. That makes it very suitable for detecting and, most important, for blocking brute-force attacks.
  • WordPress (no plugins)
    This is going to be a very interesting part of this article: testing WP alone, without any security plugin. Does it really need those plugins during an attack? We’ll see.

Two servers will be used:

The Victim

IP / Domain: 172.16.0.1 / www.testsite.com
CPU: Intel Core i5-3230M CPU @ 2.60GHz
RAM: 6GB
OS: Linux Debian 7
Kernel: 3.2.32-1 x86_64
HTTP server: Nginx, 1.2.1
PHP: 5.4.4-9 (FPM/FastCGI), no opcode cache
MySQL: 5.5.28
WordPress: 3.6.1
Sysstat version 10.0.5 (sar command)

The Attacker

IP: 172.16.0.200
CPU: Intel Core i3 M 390 @ 2.67GHz
RAM: 4GB
OS: Linux Debian 7
Kernel: 3.2.32-1 x86_64
Tools: ApacheBench v2.3, NinTechNet WP brute-force attack simulation Perl script

Connection

Ethernet crossover cable.

Plugins configuration

Each plugin will basically use the same configuration: try to detect and block an attack as soon as possible (>= 5 login attempts).

All In One WP Security (v2.5):
Default installation and click menu “WP Security > User Login”:

  • Enable Login Lockdown Feature: checked
  • Max Login Attempts: 5
  • Login Retry Time Period (min): 1
  • Time Length of Lockout (min): 2
  • Notify By Email*: keep it disabled

View screenshot

Better WP Security (v3.5.6):
Default installation and click menu “Security > Login Limits”:

  • Enable Login Limits: checked
  • Max Login Attempts Per Host: 5
  • Max Login Attempts Per User: 5
  • Login Time Period (minutes):5
  • Lockout Time Period (minutes): 2
  • Email Notifications : unchecked

View screenshot

BulletProof Security (v.49.2):
Default installation and click menu “BPS Security > Login Security”:

  • Max Login Attempts: 5
  • Automatic Lockout Time: 2

View screenshot

Limit Login Attempts (v1.7.1):
Default installation and click menu “Settings > Limit Login Attempts”:

  • 5 allowed retries
  • 2 minutes lockout
  • 10 lockouts increase lockout time to 24
  • Handle cookie login: No

View screenshot

NinjaFirewall (WP edition) (v1.1.1):
Default installation and click menu “NinjaFirewall > Firewall Policies”:

  • Do not scan traffic coming from localhost (127.0.0.1) and private IP address spaces: No

View screenshot #1

Menu “NinjaFirewall > Login Protection”:

  • Enable protection: yes
  • Protect the login page against: POST request attacks
  • Password-protect the login page for “2” minutes, if attack exceeds 5 requests within 20 seconds.
  • HTTP authentication: User: whatever4134, Password: fjiEJKSKWKW

View screenshot #2

Wordfence Security (v3.8.3):
Default installation and click menu “Wordfence > Options”:

  • Enable login security: enabled
  • Enable firewall: enabled
  • Enable Live Traffic View: disabled

View screenshot #1

Scroll down to “Login Security Options”:

  • Lock out after how many login failures: 5
  • Lock out after how many forgot password attempts: 5
  • Count failures over what time period: 5
  • Amount of time a user is locked out: 5 minutes

View screenshot #2

Like NinjaFirewall, Wordfence will ignore traffic coming from private IP addresses, but it does not seem to have any option to turn it off. Its /wp-content/plugins/wordfence/lib/wfLog.php file will have to be edited, and the following 11 lines of code from the isWhitelisted() function to be either deleted, or commented out:

//We now whitelist all RFC1918 IP addresses and loopback
if(strpos($IP, '127.') === 0 || strpos($IP, '10.') === 0 || strpos($IP, '192.168.') === 0 || strpos($IP, '172.') === 0){
   if(strpos($IP, '172.') === 0){
      $parts = explode('.', $IP);
      if($parts[1] >= 16 && $parts[1] <= 31){
         return true;
      }
   } else {
      return true;
   }
}

 

WordPress (v3.6.1):
Default installation. Only one user was created: admin (pass: StressTest)
The ‘admin’ user is important for the test: some of the plugins do not rely on the IP or an access to the login page, but rather on the user. For instance, BulletProof Security will block only if the user exists and has too many failed login attempts. If the user does not exist, it will simply do nothing at all against the attack.

Benchmarking methodology

The first test will record the number of requests per second and the response time, and the second one, the number of requests per second, the duration, the CPU load, memory usage, MySQL queries as well as the number of bytes returned by the Victim.

For each plugin, the same procedure will be followed:

  • Install and configure the plugin.
  • Delete the HTTP access log.
  • Restart Nginx, MySQL and PHP-FPM.
  • Get the current total number of queries from MySQL (SHOW STATUS WHERE variable_name='Queries';).
  • Run the sar command to collect system activity information every 5 seconds (sar 5 -rq).
  • Start the attack from the Attacker server.
  • Stop the sar command after the attack.
  • Get the new total number of queries sent to the DB using the above SQL command.
  • Collect the HTTP access log.
  • Disable and uninstall the plugin.

To ensure accuracy, each test will be performed 3 times.

First test

ApacheBench will run from the Attacker server with the following command:

# ab -t 60 -c 5 -p 'postdata.txt' -T 'application/x-www-form-urlencoded' -C 'wordpress_test_cookie=WP+Cookie+check' http://www.testsite.com/sites/wordpress/wp-login.php

Each request simulates an attempt to log in to the admin console. It includes all mandatory fields: WordPress wordpress_test_cookie cookie, credentials (user: admin, pass: bruteforce) and the POST payload inside a postdata.txt file:

log=admin&pwd=bruteforce&wp-submit=Log+In&redirect_to=http%3A%2F%2Fwww.testsite.com%2Fsites%2Fwordpress%2Fwp-admin%2F&testcookie=1

It will try to make 50,000 login attempts but, if the attack takes too long, it will stop after 60 seconds. 5 concurrent requests will be used. That will be quite brutal but will match the definition of a stress test:

The goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks.

The ApacheBench results can be downloaded at the bottom of this article.

The first graph shows the number of requests per second (RPS) that each plugin handled during the attack. The higher, the better:

At a rate of 1,520 requests per second, NinjaFirewall is the clear winner. In order to see the other results, we need to graph them separately:

They all processed the attack* at a rate of less than 30 RPS, BulletProof Security (whose 3 benchmarks show a very high standard deviation) and Wordfence Security being the two slowest. Note how WordPress alone outperformed 4 plugins.

*only NinjaFirewall processed the whole attack, i.e. 50,000 POST requests in 32 seconds. All others reached the 60-second limit and processed only +/-1,600 POST requests within that timeframe.

This second graph shows the response time. This is the time, in milliseconds (ms), it took for the server to process the request and send a reply. Obviously, small numbers are better:

With a 3ms response time, NinjaFirewall crushed its competitors (190ms on average).

Second test

This will be a distributed attack attempting to hack into the WordPress login page. It will be much less intensive than the previous one, hence very realistic.

According to the Brute Force Attacks Build WordPress Botnet article from the Krebs on Security blog, the last big attack against WordPress sites was carried out by 90,000+ IPs (servers and/or infected home computers). The blog’s article even includes a copy of the username/password list that the attackers may have been using for the attack. It contains 2,927 lines, with quite a lot of duplicates.
It will be used for the attack.

Simulating a large botnet is very easy, thanks to Nginx, by adding those 2 lines to its /etc/nginx/nginx.conf file:

set_real_ip_from 172.16.0.200;
real_ip_header X-Forwarded-For;

This tells Nginx to use the IP from the X-Forwarded-For header of each HTTP request coming from IP 172.16.0.200 (the Attacker). Nginx will forward it to PHP and then, to WordPress and its plugins.

The following simple Perl script will be used to run the attack: it will read the username/password list and will try each of the 2,927 combinations. Parameters, values and cookies are the same as those used for the first test.
Line 29, it generates a random IP for each request and adds it to the X-Forwarded-For field.

#!/usr/bin/perl
########################################################################
# WP brute-force attack simulation                                     #
#                                                                      #
# (c)2012-2013 NinTechNet                                              #
#                                                                      #
########################################################################
# See nintechnet.com/1.1.1/                                            #
########################################################################
# REVISION: 2013-09-23 23:51:15                                        #
########################################################################

use strict;
use Time::HiRes qw(gettimeofday tv_interval);
require LWP::UserAgent;

my $victim = 'http://www.testsite.com/sites/wordpress/wp-login.php';
my ( $user, $pass, $success, $error );
my $ua = LWP::UserAgent->new;
my $t0 = [gettimeofday];

open IN, "<passwords.txt" or die "Cannot find password file, aborting.\n";
while (  ) {
   ( $user, $pass) = split(':', $_);
   $ua->timeout(10);
   $ua->agent('Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2');
   $ua->default_header(
      'Cookie' => 'wordpress_test_cookie=WP+Cookie+check',
      'X-Forwarded-For' => join ".", map int rand 256, 1 .. 4
   );
   my $response = $ua->post(
      $victim,
      [
         'log'          => $user,
         'pwd'          => $pass,
         'wp-submit'    => 'Log In',
         'redirect_to'  => 'http%3A%2F%2Fwww.testsite.com%2Fsites%2Fwordpress%2Fwp-admin%2F',
         'testcookie'   => '1'
   ],
   );
   if ( $response->is_success ) {
      $success++;
   } else {
      $error++;
   }
}
close IN;
print "Total requests POSTed:\t" . ( $success + $error ) . "\n";
print "Total errors:\t\t" . $error . "\n";
print "Total time:\t\t" . tv_interval ( $t0, [gettimeofday]) . "\n";
exit;

 

Below is a sample of the HTTP access log showing the distributed attack with “spoofed” IPs:

141.198.86.76 - - [23/Sep/2013:11:21:48 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
173.64.20.223 - - [23/Sep/2013:11:21:48 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
162.249.143.157 - - [23/Sep/2013:11:21:48 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
151.20.33.174 - - [23/Sep/2013:11:21:48 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
210.93.209.167 - - [23/Sep/2013:11:21:48 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
142.168.104.49 - - [23/Sep/2013:11:21:48 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
122.65.255.220 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
223.93.203.133 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
114.188.20.53 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
26.171.193.140 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
117.168.98.26 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
31.3.22.232 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
85.224.74.118 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
163.108.233.192 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
251.72.201.12 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
93.78.133.49 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
112.53.25.146 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
33.199.125.74 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"
114.195.136.9 - - [23/Sep/2013:11:21:49 +0700] "POST /sites/wordpress/wp-login.php HTTP/1.1" 200 3336 "-" "Mozilla/5.0 (X11; Linux x86_64) Epiphany/3.4.2"

All HTTP logs can be downloaded at the bottom of this article.

The first graph shows the number of requests per second (RPS) during the attack. High numbers are better:

At a rate of 154 RPS, NinjaFirewall is again way ahead of all other plugins. Interesting enough, WordPress (without plugins) comes second with 11 RPS, more than twice as fast as Login Limit Attempts and Wordfence Security.

The next graph shows the total time it took, in seconds, to handle the whole attack. Small numbers are better:

It took NinjaFirewall 19 seconds to get through the whole attack, while WordPress (without plugins) required 276 seconds. Way behind, Limit Login Attempts took around 9 minutes (529 seconds) to perform the same work, and Wordfence Security exactly 625.13 seconds.

This graph shows the CPU load on the Victim‘s server (i5, 2 cores, 4-threaded processor). Low numbers are better:

Because NinjaFirewall processed the attack in only 19 seconds, attempting to measure its impact on the server load would be irrelevant.
Limit Login Attempts and Wordfence Security, once again, scored poorly by increasing the load much more than any other plugin.

The following graph shows the total amount of RAM, in megabytes, used by each plugins. The lower, the better:

This graph is interesting because it shows that Better WP Security and Wordfence Security used less RAM than WordPress without any plugins (43Mb vs 62Mb). With respectively 82Mb and 96Mb, BulletProof Security and Limit Login Attempts are far behind.
Here too, it would be irrelevant to attempt to graph the amount of RAM used by NinjaFirewall 19-second test.

Next graph displays the total number of queries (read/write) sent to the MySQL database during the whole attack. No query at all, or a low number, is better:

For better performance, NinjaFirewall does not send any query to the database during an attack. WordPress ranks 2nd with 32,234 queries sent (11 queries/POST request). Things start getting worrisome for Better WP Security with 56,461 queries, and totally critical for Wordfence Security which sent no less than 118,096 queries (40 queries/POST request). Looking at the database showed that Wordfence added 6,056 rows to the wp_options table.

The last graph shows the total amount of kilobytes returned by the Victim during the whole attack. The lower, the better:

NinjaFirewall returned a total of 142 Kb (44 bytes for each blocked request) while all other plugins returned around 9 MB (3 Kb/request), the same amount of data returned by WordPress without any plugin.

Recommendations

Username:

  • Do not use admin.

Password:

  • Do not use a word from the dictionary (whatever, blog, computer…).
  • Do not use a proper noun/name (michael, charlie, paris…).
  • Do not use a password that could be easily associated with you.
  • Use at least 10 alphanumeric characters, include non-alphabetic ones (dot, hash, comma etc…) and mix UPPER/lower cases.
  • Change it often.

Benchmark logs (raw)