Skip to content
Docs · Scanner access

Our scanner announces itself. Here's how to let it in.

When CiteAgentic audits a page, the request identifies itself honestly in the User-Agent header. Some firewalls and CDNs block automated requests by default, which shows up in your audit as pages we couldn't fetch. The fix is a one-line allowlist rule, not a workaround.

Run a scan
Applies to on-page audits of your own site. AI engines use their own crawlers. See our guide to allowing those.

How our scanner identifies itself

Every request from the CiteAgentic scanner carries this User-Agent:

Mozilla/5.0 (compatible; CiteAgenticBot/1.0; +https://citeagentic.com/docs/scanner)

Earlier versions identified as Launchy/1.0. If you allowlisted that token before, keep the rule or update it; matching CiteAgenticBot covers all current requests.

The +https://citeagentic.com link is there so any server administrator who sees the request can find out who we are. We fetch only public pages of sites our customers have asked us to audit, we respect the page's robots directives for our purposes, and we cap how much we fetch per page and per scan.

What we will never do

Some tools try to slip past security controls by pretending to be a normal browser or, worse, by impersonating Google's or OpenAI's crawlers. We don't, as a hard policy:

  • No spoofed User-Agents. We never present ourselves as Googlebot, GPTBot, a human browser session, or anything we aren't.
  • No WAF evasion. If your firewall challenges or blocks our request, we record "couldn't fetch" honestly and tell you. We don't rotate IPs or solve challenges to get around it.
  • No fetching behind logins. Public pages only.

This matters beyond principle: impersonating verified crawlers gets IPs blacklisted and produces audit data your real visitors and real AI crawlers would never see. An audit is only useful if it sees what the world sees.

Why a scan says "couldn't fetch"

If your audit shows pages we couldn't collect, one of these is almost always the cause:

  • CDN or WAF bot rules: Cloudflare, Akamai, AWS WAF and similar block or challenge unrecognised automated agents by default.
  • JavaScript challenges: interstitial pages ("Checking your browser…") that require executing a challenge before serving content.
  • IP-based rate limiting or blocking: security rules that throttle or reject repeated automated requests.
  • User-Agent filtering: rules that reject any request whose agent string isn't a known browser.

None of these mean your site is broken. They mean your security layer is doing its job against unknown bots, and our scanner is, deliberately, an announced bot.

How to allowlist the scanner

Add one allow rule in your CDN/WAF for requests whose User-Agent contains:

CiteAgenticBot

or, more strictly, the full token CiteAgenticBot/1.0; +https://citeagentic.com/docs/scanner.

  • Cloudflare: Security → WAF → Custom rules → create a rule with expression (http.user_agent contains "CiteAgenticBot") and action Skip (bot fight mode / managed challenge).
  • Akamai: add a match target on the User-Agent header containing CiteAgenticBot with a bypass action in your bot category.
  • AWS WAF: add a string-match rule on the user-agent header with action Allow, placed above your bot-control rule group.
  • Other stacks: any rule engine that can match a request header can allow the same token.

Re-run your scan afterwards. Previously blocked pages should collect normally. If they still fail, the block may be IP-based; contact us and we'll share the details your administrator needs.

See exactly which pages were blocked

Every CiteAgentic audit lists pages it couldn't fetch and why, instead of silently skipping them. Run a scan and check the Technical Accessibility section.

Frequently asked questions

Does CiteAgentic bypass Cloudflare or other firewalls?
No. If a firewall blocks or challenges our scanner, the audit reports the page as not collectable and tells you why. The supported fix is allowlisting our announced User-Agent, never evasion, IP rotation, or challenge solving.
What User-Agent does the CiteAgentic scanner use?
Mozilla/5.0 (compatible; CiteAgenticBot/1.0; +https://citeagentic.com/docs/scanner). Earlier versions identified as Launchy/1.0. The URL in the agent string identifies us to any server administrator reviewing logs.
Why does my audit say some pages could not be fetched?
Your CDN or firewall is blocking unrecognised automated agents, commonly Cloudflare or Akamai bot rules, JavaScript challenges, IP rate limits, or User-Agent filtering. Allowlist the CiteAgenticBot agent token and re-run the scan.
Is allowlisting the scanner a security risk?
The rule only admits requests carrying our agent string to your public pages, which are already publicly readable. It does not disable your WAF for browsers or other bots. Strict setups can scope the rule to the full token including our URL.
Does the scanner affect how AI engines see my site?
No. ChatGPT, Perplexity and the others use their own crawlers (OAI-SearchBot, PerplexityBot and friends) with their own access rules. Our guide on AI crawler access covers allowing those separately.