How do I detect and exclude robots / abnormal traffic?

We have certain measures in place to prevent robot traffic from polluting your data: an organisation that records all known robots provides us with a list (IAB/ABC spiders and bots list) on which we base our traffic exclusions so that bot traffic is not included alongside your other data in the interface.

When traffic is excluded, the exclusion is based on the IP addresses or user agent patterns of robots in the above list.

The IAB cannot however record all robots in existence. Some, especially the more recent ones that have not yet been listed, may bypass our exclusions and cause unusual spikes in your traffic. When this is the case, you can often recognize robots by their abnormal behaviour. Using various indicators, such as the following six, you can check for this behaviour amongst your visitors:

• Time spent/ pages: if it's short, it may indicate a crawler

• Page views/ visits: if it's high, same

• URLs (Referrer sites): a spike in visits from an unknown domain is suspicious

• Countries (Geolocation): a spike in visits from a country that doesn't usually feature in your stats is suspicious

• Towns (Geolocation): same

• Models (OS): a spike in visits from the same device model is suspicious

If you suspect abnormal traffic, please contact the Support Centre while providing context for your suspicions. We will be able to advise on the best solution to exclude it.

You can ask the support center to exclude traffic based on known IP addresses. However, we won’t be able to provide such information.

Please note that if you have IP anonymisation (e.g. with the CNIL Exemption), you will not be able to exclude a precise IP, but only the associated range. It is up to you whether to add this range as an exclusion or monitoring.

It is also possible to exclude traffic through the Exclusions available in Data Management. You could then base your exclusion on one or more criteria such as city, organisation, ISP, User-Agent, etc.