# robots.txt for tenablyai.com / www.tenablyai.com # # Served by the "tendably" Fly.io app (review-dashboard). The CDN domain # cdn.tenablyai.com has its own robots.txt at gs://tenably_ai_images/robots.txt # (source: reviewprocessor/cdn/robots.txt). Keep both files permissive toward # social-platform scrapers — Facebook, Instagram, LinkedIn, X, TikTok, and # Pinterest fetch hosted media URLs server-side and honor robots.txt, and a # malformed file can block every outbound post. # Default: allow everything for SEO crawlers. We used to ship # `Disallow: /tenably_ai_images/` etc. here, but those paths never existed on # this host (they were bucket-name fragments) so the rules were no-ops with # confusing intent; removing them prevents future debuggers from chasing a # red herring. User-agent: * Disallow: # Explicit allow for social platform scrapers that fetch hosted media URLs. # Redundant given `*` above, but guards against individual crawler parser # quirks — Meta's scraper in particular has been observed rejecting hosted # media with "Restricted by robots.txt" when the rules group it sees is not # in the RFC-9309 canonical form. User-agent: facebookexternalhit Disallow: User-agent: facebookcatalog Disallow: User-agent: meta-externalagent Disallow: User-agent: Facebot Disallow: User-agent: Twitterbot Disallow: User-agent: LinkedInBot Disallow: User-agent: TikTokBot Disallow: User-agent: Pinterestbot Disallow: # ───────────────────────────────────────────────────────────────────────────── # AI ANSWER ENGINES # # Tenably is a marketing SaaS: we want the maximum possible citation surface, so # the policy is ALLOW EVERYTHING that can produce a citation, a brand mention, or # model familiarity with the product. # # RFC 9309 TRAP: a crawler that matches a NAMED group below ignores the # `User-agent: *` group ENTIRELY. Every named group here is allow-all, so today # that is harmless — but if anyone ever adds a Disallow or Crawl-delay to `*`, it # will NOT apply to any bot named below. It must be added to each group. # UA tokens match case-insensitively, so one line per bot is enough. # # REALITY CHECK: this file was already permissive via `User-agent: * / Disallow:`. # Naming bots is defensive documentation, not the thing that made the site # visible. The PRERENDER (prerender.mjs) is what fixed visibility — before it, # every page served an empty
and no crawler could read anything. # # Legend: [CITE] can link/cite us · [TRAIN] model corpus · [AGENT] user-triggered # (!) does not actually honour robots.txt — listed for documentation only # ── OpenAI (ChatGPT, ChatGPT Search, Atlas) ───────────────────────────────── User-agent: OAI-SearchBot Disallow: # [CITE] The one that matters. Decides what ChatGPT Search and Atlas can cite. User-agent: ChatGPT-User Disallow: # [AGENT](!) Fetches a page when a user opens/pastes a link in ChatGPT. User-agent: GPTBot Disallow: # [TRAIN] Model corpus — how ChatGPT knows Tenably exists without searching. User-agent: OAI-AdsBot Disallow: # [CITE] Validates ad landing pages; needed if we ever buy OpenAI ads. # ── Anthropic (Claude) ────────────────────────────────────────────────────── User-agent: Claude-SearchBot Disallow: # [CITE] Builds Claude's web-search index. This is Anthropic's citation bot. User-agent: Claude-User Disallow: # [AGENT] Fetches a page to answer a live Claude question. User-agent: ClaudeBot Disallow: # [TRAIN] Claude model corpus. User-agent: anthropic-ai Disallow: # [TRAIN] Legacy Anthropic token, still seen in logs. # ── Google (AI Overviews, AI Mode, Gemini) ────────────────────────────────── # IMPORTANT AND COUNTERINTUITIVE: Google-Extended does NOT gate AI Overviews or # AI Mode. Google states it "does not impact a site's inclusion in Google Search # nor is it used as a ranking signal" — it controls Gemini TRAINING and grounding # in the Gemini apps. AI Overviews and AI Mode are gated by GOOGLEBOT plus # snippet eligibility. Which means: never add nosnippet / max-snippet:0 / # noarchive to a marketing page, or we silently vanish from AI Overviews while # still ranking in the blue links. prerender.mjs hard-fails the build if one # of those tags ever appears on a prerendered page. User-agent: Googlebot Disallow: # [CITE] Search, AI Overviews and AI Mode — the whole Google AI surface. User-agent: Google-Extended Disallow: # [TRAIN] Gemini training + grounding inside the Gemini apps. User-agent: GoogleOther Disallow: # [TRAIN] Google product-team research crawls. User-agent: Google-NotebookLM Disallow: # [AGENT] Someone added tenablyai.com as a NotebookLM source. # ── Microsoft (Bing, Copilot) ─────────────────────────────────────────────── # Copilot has NO crawler of its own — it answers off the Bing index, so bingbot # IS the Copilot bot. Bing is also the index ChatGPT's live search leans on, # which makes this the highest-leverage group in the file after OpenAI's. # Do not add NOARCHIVE/NOCACHE to marketing pages: NOARCHIVE removes a page from # Copilot answers outright. User-agent: bingbot Disallow: # [CITE] Bing Search AND Microsoft Copilot answers. User-agent: msnbot Disallow: # [CITE] Legacy Bing token. User-agent: AdIdxBot Disallow: # [CITE] Bing Ads landing-page validation. # ── Perplexity ────────────────────────────────────────────────────────────── User-agent: PerplexityBot Disallow: # [CITE] Builds Perplexity's index — its citation bot. User-agent: Perplexity-User Disallow: # [AGENT](!) Visits a page to answer a live question. # ── Apple (Siri, Spotlight, Apple Intelligence) ───────────────────────────── # Applebot — NOT Applebot-Extended — is the citation bot. Applebot-Extended is a # training opt-out only. Since Apple's 2026-06 rewrite, Applebot-crawled content # also powers Siri and Apple Intelligence answers, gated by `nosnippet`. User-agent: Applebot Disallow: # [CITE] Siri, Spotlight, Safari and Apple Intelligence answers. User-agent: Applebot-Extended Disallow: # [TRAIN] Apple foundation models. # ── Meta AI (WhatsApp, Instagram, Facebook) ───────────────────────────────── User-agent: meta-webindexer Disallow: # [CITE] The Meta bot that decides Meta AI search citations. User-agent: meta-externalfetcher Disallow: # [AGENT](!) User-requested single-link fetch. # (meta-externalagent and Facebot are already allowed in the social block above.) # ── Amazon (Alexa) ────────────────────────────────────────────────────────── User-agent: Amzn-SearchBot Disallow: # [CITE] Alexa answers. Not training. User-agent: Amazonbot Disallow: # [TRAIN] Amazon product + model improvement. # ── Mistral (Le Chat) ─────────────────────────────────────────────────────── User-agent: MistralAI-Index Disallow: # [CITE] Mistral's search index; explicitly not training. User-agent: MistralAI-User Disallow: # [AGENT] Le Chat visits a page and links the source. # ── Second-tier answer engines (small reach, zero cost to allow) ──────────── User-agent: DuckAssistBot Disallow: # [CITE] DuckDuckGo AI-assisted answers. User-agent: YouBot Disallow: # [CITE] You.com AI search. User-agent: kagi-fetcher Disallow: # [CITE] Kagi's AI suite. User-agent: ExaSearchBot Disallow: # [CITE] Exa — the retrieval API behind many third-party AI agents. User-agent: TavilyBot Disallow: # [CITE] Tavily — powers a large share of agent search calls. User-agent: PhindBot Disallow: # [CITE] Phind (developer-facing answers). User-agent: cohere-ai Disallow: # [AGENT] Cohere retrieval on a user prompt. # ── Open training corpora (no citations, but they seed open-weight models) ── User-agent: CCBot Disallow: # [TRAIN] Common Crawl — feeds most open-weight models. Cheap brand presence. User-agent: AI2Bot Disallow: # [TRAIN] Allen Institute open models. # ── DELIBERATELY NOT LISTED — do not "complete" this list ─────────────────── # xAI / Grok No robots.txt token exists. xAI publishes no crawler docs and # Grok retrieval reportedly uses rotating residential IPs with # spoofed browser UAs. Any "xAI-Bot"/"GrokBot" line you find in a # blog listicle is invented. There is nothing to allow or block. # DeepSeek No published user-agent. Same situation. # ChatGPT Atlas No token of its own; governed by OAI-SearchBot above. # Bytespider ByteDance. Documented as NOT honouring robots.txt and crawls # aggressively. A line here would be decorative. If it ever costs # real bandwidth, block it at the Fly/WAF layer instead. # Diffbot, Brightbot, omgili, Webzio-Extended, ImagesiftBot, SemrushBot-OCOB # Data RESELLERS and SEO crawlers. No answer surface cites us, so # allowing them is egress cost with no visibility upside. # ───────────────────────────────────────────────────────────────────────────── # IndexNow keys. BOTH files are hosted on purpose — IndexNow allows many key # files per host, and keeping the old one valid means earlier submissions are # not retroactively rejected. # active (from bing.com/indexnow/getstarted, 2026-08-18): # https://www.tenablyai.com/4e9381d51f404ffbbd1d61ee1bd14203.txt # previous (self-generated, still served): # https://www.tenablyai.com/6ef0d06ba40454674f844e2aa2d5e518.txt # Each URL must return 200 with the bare key as its body or every push from # `npm run indexnow` is rejected. A key file only goes live after a DEPLOY — # adding it to public/ mid-build is not enough. # (Comment only — IndexNow has no robots.txt directive.) # # AI ingest manifest: /llms.txt (index) + /llms-full.txt (prose mirror). # Status: NOT honoured by any major AI vendor. Google's AI-features guidance says # outright that llms.txt is not needed for AI Overviews or AI Mode. Keep it (it # costs nothing and is useful for agent workflows) but expect no citation lift. # ───────────────────────────────────────────────────────────────────────────── # Sitemap — global directive, kept last so it is never mistaken for part of a # preceding user-agent group. Sitemap: https://www.tenablyai.com/sitemap.xml