فهرسة الإنترنت بواسطة الروبوت

Jetpack by WordPress.com (Jetpack 12.5; WordPress 6.1)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)

Jetpack by WordPress.com

Jetpack/12.1; WordPress/6.3; http://site61741366.com

Jetpack by WordPress.com (Jetpack 12.0; WordPress 6.3)

BuiltWith identifies the technologies used by websites and produces web technology, market, and lead intelligence.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj; 2fa6014a47) Chrome/124.0.0.0 Safari/537.36

serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/150.0.7871.186 Safari/537.36

Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.186 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

understory/0.1 (CT research; https://github.com/0x4b) +contact-via-repo

siteradar/0.1 contact: ops@yyai.studio

LeakIX scans internet-facing services and indexes exposed systems, service banners, leaks, and security findings.
Mozilla/5.0 (l9scan/2.0.933323e2033313e29393e2933313; +https://leakix.net)

Jetpack by WordPress.com (Jetpack 12.1; WordPress 6.4)

WordPress.com; https://wordpress.com

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 (Silvy X Ran; +https://silvyxran.love; +https://silver.inc)

Mozilla/5.0 (compatible; rust_sniffer/0.1; +https://github.com/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Googlebot/2.1 (+http://www.google.com/bot.html)

ByteDance operates content platforms including TikTok; Bytespider collects public web content for ByteDance services.
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

Mozilla/5.0 (compatible; DuckDuckBot/1.1; +http://duckduckgo.com/duckduckbot.html)

Mozilla/5.0 (compatible; GoogleOther; +https://developers.google.com/search/)

ucp-crawler/0.1 (+https://github.com/huckleberry-inc/ucp-crawler)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 AICrawler.store

Mozilla/5.0 (Windows NT 10.0.0; Win64; x64; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.6367.63 Chrome/124.0.6367.63 Not-A.Brand/99 Safari/537.36

Mozilla/5.0 (compatible; MJ12bot/v2.0.5; http://mj12bot.com/)

Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.128 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

Mozilla/5.0 (compatible; domains-distillery/1.0; +https://crawler.validus.ai)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Nessus SOAP v0.0.1 (Nessus.org)

UMD-gRPC-Research (academic gRPC measurement; https://www.cs.umd.edu/~stekin/scan/; contact: baddatalab@umd.edu)

httpx - Open-source project (github.com/projectdiscovery/httpx)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.124 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.114 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (compatible; heritrix/3.14.2-SNAPSHOT-2026-04-13T06:21:22Z +https://www.image-meta.com)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Jetpack/12.1; WordPress/6.1; http://site52012413.com

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/150.0.7871.46 Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko, Yokohama Institute of Information Security https://www.iisec.ac.jp) Chrome/124.0.0.0 Safari/537.36

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041117 Firefox/1.0 (Debian package 1.0-2.0.0.45.linspire0.4)

HTC-ST7377/1.59.502.3 (67150) Opera/9.50 (Windows NT 5.1; U; en) UP.Link/6.3.1.17.0

Mozilla/5.0 (compatible; FlowIQLabsBot/1.0; +https://flowiq-labs.com/scanning-info)

Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 +https://forestengine.net/#opt-out

Jetpack/12.5; WordPress/6.4; http://site74961882.com

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +https://www.google.com/bot.html) Safari/537.36

Jetpack by WordPress.com (Jetpack 12.1; WordPress 6.2)

Mozilla/5.0 (compatible; Infrawatch/1.0; +https://infrawat.ch/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.46 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/149.0.7827.200 Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1 CibraxScanner/1.0 (+https://cibrax.com/scanner)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.7827.200 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Wget/1.21.4 (HelpMeEscapeFromBelarus@proton.me)

Mozilla/5.0 (compatible; Exabot/3.0; http://www.exabot.com/go/robot)

FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)

Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE90-1/07.24.0.3; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413 UP.Link/6.2.3.18.0

Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )

MOT-V177/0.1.75 UP.Browser/6.2.3.9.c.12 (GUI) MMP/2.0 UP.Link/6.3.1.13.0

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)

Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)

msnbot/1.0 ( http://search.msn.com/msnbot.htm)

SonyEricssonW810i/R4EA Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0

WebZIP/3.5 (http://www.spidersoft.com)

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)

Mozilla/5.0 (Linux; Android 5.1.1; vivo Y31L.RastaMod™_Version) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.143 Mobile Safari/537.36

Bloglines/3.1 (http://www.bloglines.com)

Mozilla/5.0 (X11; Linux 3.8-6.dmz.1-liquorix-686) KHTML/4.8.4 (like Gecko) Konqueror/4.8

Mozilla/5.0 (compatible; GeniusDataBot/1.0; +https://geniusdata.com)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; DomainScores/2.1; +https://www.domainscores.net/headers)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

TLS tester from https://testssl.sh/

BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)

Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)

Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

NokiaN70-1/5.0609.2.0.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.1.13.0

Mozilla/5.0 (Linux; U; Android 8.1.0; zh-CN; EML-AL00 Build/HUAWEIEML-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.108 baidu.sogo.uc.UCBrowser/11.9.4.974 UWS/2.13.1.48 Mobile Safari/537.36 AliApp(DingTalk/4.5.11) com.alibaba.android.rimet/10487439 Channel/227200 language/zh-CN

MOT-L7v/08.B7.5DR MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

SEC-SGHX210/1.0 UP.Link/6.3.1.13.0

Yandex operates a search engine; YandexBot discovers and indexes public pages for its search services.
Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)

msnbot/0.11 ( http://search.msn.com/msnbot.htm)

everyfeed-spider/2.0 (http://www.everyfeed.com)

Gregarius/0.5.2 ( http://devlog.gregarius.net/docs/ua)

msnbot/1.1 ( http://search.msn.com/msnbot.htm)

BlackBerry9530/4.7.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 UP.Link/6.3.1.20.0

msnbot-media/1.1 ( http://search.msn.com/msnbot.htm)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)

Googlebot/2.1 ( http://www.googlebot.com/bot.html)

SonyEricssonK800i/R1CB Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

Mozilla/5.0 (Linux; Android 8.1.0; LM-Q710.FG) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Mobile Safari/537.36

SonyEricssonZ800/R1Y Browser/SEMC-Browser/4.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) baidu.sogo.uc.Chrome/36.0.1985.125 Safari/537.36

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
AdsBot-Google ( http://www.google.com/adsbot.html)

Gaisbot/3.0 (robot@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php)

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

check-host.cc Intel Platform (https://check-host.cc/about/scan-infra; abuse@check-host.cc)

Mozilla/5.0 (compatible; SparixEmailScraper/1.0; +https://sparix.ai)

Mozilla/5.0 (compatible; wpbot/1.4; +https://forms.gle/ajBaxygz9jSR8p8G9)

2ip bot/1.1 (+https://2ip.io/bot/)

Mozilla/5.0 (compatible; Dataprovider.com)

Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 AISearchIndex.space

Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Googlebot-Image/1.0

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/148.0.7778.96 Safari/537.36

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

BuiltWith identifies the technologies used by websites and produces web technology, market, and lead intelligence.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

Example.com is a reserved documentation domain, so a scanner using it in its user agent does not identify its real operator.
Mozilla/5.0 (compatible; CMS-Checker/1.0; +https://example.com)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Linux; Android 13; Pixel 4a (5G) Build/TQ2A.230505.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/112.0.5615.136 Mobile Safari/537.36 GoogleApp/14.16.27.29.arm64 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Android 13; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Yandex operates a search engine; YandexBot discovers and indexes public pages for its search services.
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)

Baidu is a major Chinese search engine; Baiduspider crawls pages for its web search index.
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)

Google Cloud provides cloud computing and Vertex AI; Google-CloudVertexBot retrieves pages for Vertex AI agent features.
Mozilla/5.0 (compatible; Google-CloudVertexBot; +https://cloud.google.com/vertex-ai-bot)

DeepSeek develops large language models; DeepSeekBot retrieves public web content for its search and AI services.
Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

CheckMark Network crawls the web to support trademark, domain, ecommerce, image, and brand-monitoring services.
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Googlebot-News

CyberConvoy Scout performs identifiable internet scanning to discover publicly reachable services and security exposures.
Mozilla/5.0 (compatible; CyberConvoyScout/1.0; +https://scout.cyberconvoy.co)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

DomainsBot provides domain and business intelligence; Pandalytics gathers public web data for market and domain analysis.
Pandalytics/2.0 (https://domainsbot.com/pandalytics/)

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Driftnet operates this domain to discover and measure publicly exposed network services without attempting to log in.
Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

ByteDance operates content platforms including TikTok; Bytespider collects public web content for ByteDance services.
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-Web/1.0; +https://www.anthropic.com)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (compatible; Applebot/0.1; +http://www.apple.com/go/applebot)

Cohere develops enterprise language and retrieval models; its crawler collects public content for AI products and models.
Mozilla/5.0 (compatible; cohere-ai/1.0; +https://cohere.com)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +mailto:support@anthropic.com

Common Crawl operates CCBot and publishes an open repository of web crawl data for research and analysis.
CCBot/2.0 (https://commoncrawl.org/faq/)

Recorded Future collects and analyzes internet data to produce threat intelligence; its crawler inventories public assets.
RecordedFuture Global Inventory Crawler

Censys continuously scans public internet services and certificates to build an internet-wide asset and exposure index.
Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)

IPIP.net's security scanner probes public services and collects HTTP banner information for internet asset analysis.
HTTP Banner Detection (https://security.ipip.net)

Sogou is a Chinese search service; Sogou web spider discovers and indexes public pages for its search engine.
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

VisionHeight performs ethical internet scanning to map adversary infrastructure and provide pre-attack threat intelligence.
visionheight.com/scan Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36

Nokia Deepfield's Genome crawler maps public IP addresses, applications, and services to provide internet security context.
Mozilla/5.0 (compatible; GenomeCrawlerd/1.0; +https://www.nokia.com/genomecrawler)

Modat combines large-scale internet scanning with infrastructure context to identify exposures and emerging cyber threats.
Mozilla/5.0 (compatible; ModatScanner/1.2; +https://modat.io/)

Scrapy is an open-source Python framework used to build web crawlers and data-extraction applications.
Scrapy/2.16.0 (+https://scrapy.org)

Palo Alto Networks Cortex Xpanse scans internet-facing systems to discover assets and exposures for attack-surface management.
Hello from Palo Alto Networks, find out more about our scans in https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

LeakIX scans internet-facing services and indexes exposed systems, service banners, leaks, and security findings.
Mozilla/5.0 (l9scan/2.0.035323e2634323e2430313e233; +https://leakix.net)

Example.com is a reserved documentation domain, so a scanner using it in its user agent does not identify its real operator.
Mozilla/5.0 (compatible; CT-WP-Scanner/1.0; +https://example.com/bot)