SiteBrainBot

SiteBrainBot is SiteBrain’s crawler. It only visits a website when its owner (or someone acting on their behalf) creates a chatbot for it on SiteBrain, to read the public content used to answer questions.

User agent

User-Agent: SiteBrainBot/1.0 (+https://sitebrain.io.vn/bot/)

Behaviour

• Follows robots.txt (including Crawl-delay) and <meta name="robots" content="noindex">.
• Reads public HTML pages only; skips login, cart, admin pages and files.
• Page limits per plan and low concurrency keep the load on your server minimal.
• Re-reads at most once a week (or when the chatbot owner asks).

Blocking SiteBrainBot

User-agent: SiteBrainBot
Disallow: /

Allowing SiteBrainBot (Cloudflare and firewalls)

If the dashboard says "No page could be read" (usually HTTP 403 or 503), the website may be blocking bots. The most common cause is Cloudflare Bot Fight Mode or AI-bot blocking, because SiteBrainBot runs on Cloudflare Workers.

• Cloudflare (Pro plan and above): Security → WAF → Custom rules → create a rule with the condition User Agent contains "SiteBrainBot" and the action Skip, skipping Bot Fight Mode / Super Bot Fight Mode and the managed rules.
• Cloudflare (Free plan): Bot Fight Mode has no exceptions. Turn it off in Security → Bots while SiteBrain reads the site, then turn it back on. If AI-bot blocking is on, allow SiteBrainBot there in the same way.
• Other firewalls or security plugins (Wordfence etc.): add the user agent SiteBrainBot to the allow list.

Then click Refresh content in the dashboard so SiteBrain reads the site again.

Blocked content is removed from the chatbot at the next crawl. If you own a website and don’t want third parties to build a chatbot from it, email support@sitebrain.io.vn — we handle requests within two business days.