Recently on one of our sites we had the issue that the server periodically was massively overloaded, and the website slowed to a crawl. The server had a load factor of over 75 – full load was 4 (yes, four), so it was overloaded with almost a factor 20. The reason? A bad bot. A specific bot was trying to index the site at a ferocious speed. This was enough to completely overload the otherwise proper sized webserver for the client.

Bots such as Semrush Bot, MJ12 Bot and DotBot just needlessly slow down your website.

216.244.66.242 - - [01/Oct/2021:10:11:51 +0000] "GET / HTTP/1.1" 301 255 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"

But aren’t bots good? Yes, most are. We all want our sites to be included in search engines, such as Google, Yahoo etc, however there are some indexing services which are NOT being used for search indexes. Instead the data are being used internally for their customers to show comparative data and in order to provide a service to their clients. This means that unless YOU are their client, you have absolutely ZERO benefit from this. One might even argue that you have a negative benefit, as it might allow competitor analysis on your site from your competitors.

So – why allow it? There is NO benefit to doing so for you, and you really should block them.

How to block them.

There are 2 simple methods (we recommend using both) to blocking them completely. One is to block them on your server itself. In order to do that, you can change your .htaccess for apache to prevent them (we can help you do that).

The other method is to block the site in CloudFlare (or whichever other CDN you’re using. You are using a CDN right?). You do this from their Firewall settings.

Once you have done this, you’re no longer providing free data to another company, and lower the change of your website being overrun by bad Robots.

You need not just multiple backups, you also need to actually TEST how to restore everything if disaster strikes. Have you tried this? You should. We’ve seen (and experienced) trying to restore a backup only to find it was maddeningly convoluted.

Disasters happen. They can happen at several levels. At the simplest level, someone deletes a post by accident. This is easy enough to recover from the backup. Or is it? The problem is the backup has the entire site, and what if there have been other changes since. Now it’s suddenly not that simple any longer, because if you restore the latest backup you might lose changes since then, or even worse – customer orders.

Let’s take it a step further – your site gets deleted. By accident or by malice, this can happen. What then? How do you get it up and running again fast? If you’re running a WordPress site, you need to set up WordPress again, install the program you took the backups with, and then… Wait, you didn’t save the backups on the webserver did you? If not you can just restore it from your offsite backup (here is where the offsite comes into play), but if the backup was just a copy on the server itself, your site or shop is now basically burned to the ground and gone.

What happens if the datacenter where it’s located burns? Rare, yes, but it’s happened before. So your backup was offsite, but still in the same datacenter? Then your data was burned to the ground as well.

For a multitude of reasons you should not trust a single backup location, or backup program – what happens if the files are defective due to a bug in the backup program, and all your backups are basically unreadable? You need backups at multiple levels, and locations.

It might sound overly complicated, and you might think we’re paranoid. Chances are you’ll be fine, and will never need it. But think of it like insurance. You’ll be really happen you have it, once disaster strikes.

So, how do we do backups in-house?

We do it at 3 levels basically – from the outside we take snapshots of the entire server within our hosting system. We can use those to near-instantly restore a complete site – even elsewhere if needed.

At the second level we do it in the server itself, where we take backups stored both on the server (for a quick restore), and copies stored remotely.

Finally we do it within WordPress (with for example UpdraftPlus) – this takes a copy of your website to 1+ offsite locations, and can do it frequently. This also allows us to do offsite development of a site, where we copy the site to the development server – do the work, and copy it back.

So yes, we use both several belts, and multiple suspenders. Once set up, it runs by itself (we still test it periodically to make sure the backups actually work), and our sites are protected against all but complete armageddon for what amounts to peanuts compared to the cost of redeveloping a site.

It has actually saved our butts more than once (but that’s an even longer story).