Bot detection¶
Bot detection identifies automated crawlers and testing tools visiting your website and, when recognized, applies consent automatically instead of showing the CMP to them.
How it works¶
When a request's browser user agent matches a known bot signature, the CMP:
- Applies implicit consent equivalent to Accept All to every configured Data Processing Service.
- Does not display the consent banner.
- Does not display the floating privacy button.
- Skips the CMP's own analytics/tracking for that request.
This prevents bots from being blocked by (or skewing analytics from) your consent setup, while still allowing services like Googlebot or Lighthouse to load and evaluate your site normally.
Detected bots¶
The following user agents are currently recognized:
Search engine crawlers¶
- Bingbot
- BingPreview
- DuckDuckBot
- Googlebot
- Googlebot-Image
- Googlebot-Mobile
- Googlebot-News
- Googlebot-Video
- Google Favicon
- Google PP
- Google Web Preview
- Google-AdsBot
- Google-AdsBot-Mobile
- Google-Feedfetcher
- Google-Read-Aloud
- Google-SearchByImage
- Google-Speaker
- Google-Storebot
- GoogleOther
- Slurp
- Yahoo! Slurp
- YahooCacheSystem
- YahooSeeker
SEO, performance & testing tools¶
- Chrome-Lighthouse
- Google PageSpeed Insights / Google Page Speed Insights
- Google Search Console
- Google-InspectionTool
- Google-Structured-Data-Testing-Tool
- HeadlessChrome
- Lighthouse
- Optimizer
- Pagespeed
- Playwright
- SISTRIX
- Yahoo! Site Explorer Feed Validator
AI crawlers & assistants¶
| User agent | Operator |
|---|---|
| GPTBot | OpenAI |
| OAI-SearchBot | OpenAI |
| OAI-AdsBot | OpenAI |
| ChatGPT-User | OpenAI |
| ClaudeBot | Anthropic |
| Claude-User | Anthropic |
| Claude-SearchBot | Anthropic |
| anthropic-ai | Anthropic |
| PerplexityBot | Perplexity |
| Perplexity-User | Perplexity |
| Google-CloudVertexBot | |
| Amazonbot | Amazon |
| Amzn-SearchBot | Amazon |
| Amzn-User | Amazon |
| Applebot | Apple |
| meta-externalagent | Meta |
| meta-externalfetcher | Meta |
| meta-webindexer | Meta |
| FacebookBot | Meta |
| MistralAI-User | Mistral |
| MistralAI-Index | Mistral |
| MistralAI-Training | Mistral |
| cohere-ai | Cohere |
| cohere-training-data-crawler | Cohere |
| Bytespider | ByteDance |
| CCBot | Common Crawl |
| Diffbot | Diffbot |
| YouBot | You.com |
| DuckAssistBot | DuckDuckGo |
| PetalBot | Huawei |
| Timpibot | Timpi |
This list reflects the detection logic in the CMP SDK and may be extended over time.
Enabling bot detection¶
- In the Usercentrics Admin Interface, go to Configuration.
- Select the CMP Settings tab.
- Scroll to the Bot Detection section and toggle it on.