One of our customers faced a large attack against his WordPress blog xmlrpc.php script. When I connected to his server, the CPU load was over 100. The problem is that I couldn’t install NinjaFirewall WP Edition, our Web Application Firewall for WordPress, because the blog was completely and utterly unresponsive.
However, having root access to the server I still had a few options left to quickly mitigate the attack. Blocking IP addresses at the firewall level was not one of them because the attack was distributed. I decided to filter the TCP packets and to search and block those whose payload contained a reference to the POST /xmlrpc.php HTTP request used in the attack.
I already wrote a long article about using the iptables command as well as some of its modules (Recent, String Match and TCP) to locate a specific packet in order to filter it. I recommend that you take the time to read it before going any further as I won’t provide too many explanations regarding the rules we are going to use below.
To show the effectiveness of the mitigation, here is a CPU load graph, taken from munin-node, showing the attack and its aftermath:
The attack started at 2:20PM and approximately fifteen minutes later, the CPU load reached 120 for the next forty minutes, then slightly went down to 110 and lasted for two hours. As soon a the iptables rules were added, it took less than ten minutes before the CPU load dropped down to 0.7. The site was back online and I was able to install NinjaFirewall (WP Edition) which is now blocking all attacks.
Saving & Restoring iptables rules
First of all, before editing the current firewall rules, it is a good idea to save them:
# iptables-save > /root/iptables.save
If you need to restore them, i.e., after the attack or if something goes wrong, simply run this command:
# iptables-restore < /root/iptables.save
Mitigation rules
This is the full set of rules needed to filter and block that kind of XML-RPC POST attack.
1) We exclude the loopback interface because we don’t want to filter it:
# iptables -I INPUT 1 -i lo -j ACCEPT
2) We create a new user-defined chain, named xmlrpc:
# iptables -N xmlrpc
3) All incoming TPC packets to port 80 must be redirected to our xmlrpc chain:
# iptables -I INPUT 2 -p tcp --dport 80 -j xmlrpc
4) We are looking for the first packet, which contains a SYN flag, of the three-way handshake TCP sequence. When we catch it, we create a list with the Recent module:
# iptables -I xmlrpc 1 -m recent -p tcp --syn --set
5) If we find the ACK packet from the same sequence, we update the list:
# iptables -I xmlrpc 2 -m recent -p tcp --tcp-flags PSH,SYN,ACK ACK --update
6) Now, the connection is established. We can scan the first PSH,ACK packet and search for the POST /xmlrpc.php substring: if we find it, we drop the connection, otherwise we remove the source address from our list because we don’t want or need to filter any subsequent packets from that connection:
# iptables -I xmlrpc 3 -m recent -p tcp --tcp-flags PSH,ACK PSH,ACK --remove -m string --to 70 --algo bm --string "POST /xmlrpc.php" -j DROP
Note: if your xmlrpc.php file is located inside a subfolder, e.g.
POST /blog/xmlrpc.php, change the above command accordingly. You may also need to increase the search length (--to 70) to match it.
Testing the rules
The curl command should return a timeout error message when attempting to send an HTTP POST request to the XML-RPC script:
$ curl -X POST -m 10 http://YOUR_BLOG/xmlrpc.php curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
Viewing blocked packets
If you want to see how many packets and bytes were blocked by the firewall:
# iptables -L xmlrpc -vn
Chain xmlrpc (1 references)
pkts bytes target prot opt in out source destination
415 24900 tcp -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: DEFAULT side: source mask: 255.255.255.255 tcp flags:0x17/0x02
2742 143k tcp -- * * 0.0.0.0/0 0.0.0.0/0 recent: UPDATE name: DEFAULT side: source mask: 255.255.255.255 tcp flags:0x1A/0x10
103 112k DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: DEFAULT side: source mask: 255.255.255.255 tcp flags:0x18/0x18 STRING match "POST /xmlrpc.php" ALGO name bm TO 70