How does Burst detect bot traffic?

Did you know that nearly half of all internet traffic isn’t human? Bots are everywhere—some good, some bad. And if these bots are not filtered out, your website analytics might be telling a misleading story.

If you have looked at the raw requests that hit your server, you’ll know that a large part of those requests are just bots. Some studies show that almost 50% of internet traffic is bots. Some of them are useful bots that should visit your site (like Google), while others are purely to create fake traffic. Nevertheless, these bots are not what you want to show up in your statistics!

The normal bots, like Google etc, don’t try to hide themselves and have been pretty stable over the years in how they can be detected. These can be filtered out easily. However, the malicious kinds want to keep a low profile and may want to change their profile often.

Why filtering spam bots is important

Spam bots and malicious crawlers can significantly skew website analytics, leading to inaccurate data. By filtering out known spam referrers, website owners can ensure their analytics reflect actual human visitors rather than bot-generated traffic. This helps in making better decisions based on real user behavior.

How does Burst prevent bot traffic from cluttering your data?

Burst handles bots by implementing several filtering mechanisms. In JavaScript, we detect and filter out common bot user agents using the burst_is_user_agent function. Hits from Google, Bing, and many more. This way, over 200 types of bots are already detected in the JavaScript. Additionally, referrer spam bots are identified and excluded using the spammers.txt file after the hit reaches the server, in tracking.php, ensuring that spam traffic does not interfere with analytics. Over 2000 spammers are listed here.

Both lists are updated regularly, based on open source data.

Customizing the bot list

Need extra protection? You can manually add more bots to the blocklist using a simple WordPress filter. You can extend the referrer bot list by adding a filter with a simple mu-plugin:

function add_custom_spam_bots($ref_spam_list) {
    // Define your custom bots or spam referrers
    $custom_bots = [
        'badbot.example.com',
        'spammer-bot.net',
        'fake-traffic.xyz',
        'maliciouscrawler.org',
        'undesiredscraper.com'
    ];

    // Merge the custom bots with the existing list
    return array_merge($ref_spam_list, $custom_bots);
}