Bot detection for the Web CMP

Prev Next

Bot detection identifies automated crawlers and testing tools visiting your website and, when recognized, applies consent automatically instead of showing the CMP to them.

How it works

When a request's browser user agent matches a known bot signature, the CMP:

  • Applies implicit consent equivalent to Accept All to every configured Data Processing Service.

  • Does not display the consent banner.

  • Does not display the floating privacy button.

  • Skips the CMP's own analytics/tracking for that request.

This prevents bots from being blocked by (or skewing analytics from) your consent setup, while still allowing services like Googlebot or Lighthouse to load and evaluate your site normally.

Detected bots

The following user agents are currently recognized:

Search engine crawlers

  • Bingbot

  • BingPreview

  • DuckDuckBot

  • Googlebot

  • Googlebot-Image

  • Googlebot-Mobile

  • Googlebot-News

  • Googlebot-Video

  • Google Favicon

  • Google PP

  • Google Web Preview

  • Google-AdsBot

  • Google-AdsBot-Mobile

  • Google-Feedfetcher

  • Google-Read-Aloud

  • Google-SearchByImage

  • Google-Speaker

  • Google-Storebot

  • GoogleOther

  • Slurp

  • Yahoo! Slurp

  • YahooCacheSystem

  • YahooSeeker

SEO, performance & testing tools

  • Chrome-Lighthouse

  • Google PageSpeed Insights / Google Page Speed Insights

  • Google Search Console

  • Google-InspectionTool

  • Google-Structured-Data-Testing-Tool

  • HeadlessChrome

  • Lighthouse

  • Optimizer

  • Pagespeed

  • Playwright

  • SISTRIX

  • Yahoo! Site Explorer Feed Validator

AI crawlers & assistants

User agent

Operator

GPTBot

OpenAI

OAI-SearchBot

OpenAI

OAI-AdsBot

OpenAI

ChatGPT-User

OpenAI

ClaudeBot

Anthropic

Claude-User

Anthropic

Claude-SearchBot

Anthropic

anthropic-ai

Anthropic

PerplexityBot

Perplexity

Perplexity-User

Perplexity

Google-CloudVertexBot

Google

Amazonbot

Amazon

Amzn-SearchBot

Amazon

Amzn-User

Amazon

Applebot

Apple

meta-externalagent

Meta

meta-externalfetcher

Meta

meta-webindexer

Meta

FacebookBot

Meta

MistralAI-User

Mistral

MistralAI-Index

Mistral

MistralAI-Training

Mistral

cohere-ai

Cohere

cohere-training-data-crawler

Cohere

Bytespider

ByteDance

CCBot

Common Crawl

Diffbot

Diffbot

YouBot

You.com

DuckAssistBot

DuckDuckGo

PetalBot

Huawei

Timpibot

Timpi

Note

This list reflects the detection logic in the CMP SDK and may be extended over time.

Enabling bot detection

  1. In the Usercentrics Admin Interface, go to Configuration.

  2. Select the CMP Settings tab.

  3. Scroll to the Bot Detection section and toggle it on.