AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
amazon-kendra
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
atlassian-bot
Awario
AzureAI-SearchBot
bedrockbot
bigsur.ai
Bravebot
Brightbot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-Code
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
Code
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
ExaBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-Agent
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-Gemini-CLI
Google-NotebookLM
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
HenkBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
NagetBot
netEstate Imprint Crawler
newsai
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
opencode
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
Spider
TavilyBot
Terra Cotta
TerraCotta
Thinkbot
TikTokSpider
Timpibot
Trae
TwinAgent
VelenPublicWebCrawler
WARDBot
Webzio-Extended
webzio-extended
wpbot
WRTNBot
YaK
YandexAdditional
YandexAdditionalBot
YouBot
ZanistaBot