Feature docs · Website security · Module: robots
robots.txt is a Guardbee scan module under website security. Fetches and analyzes robots.txt.
What does it check?
Fetches and analyzes robots.txt. Results appear in the dashboard and scan detail as score, findings, and recommended actions.
Business impact
A bad robots.txt can disclose sensitive paths or hurt SEO.
Guardbee presents risk, impact, and priority — so product, legal, and engineering share one screen.
How to run it
- Create a Guardbee account (14-day free trial)
- Add a brand / URL
- Select robots.txt under Website security
- Run the scan; optionally explain findings with BeeAI
Related docs
Run this module now
Use robots.txt with no scan quota. Start with a 14-day free trial.
Frequently asked questions
Is robots.txt required?
No. Pick modules as needed; there is no scan quota — limits are active brands.
Is scanning destructive?
No. Guardbee uses controlled, read-only, limited requests.
Does it work with BeeAI?
Yes. After a scan, BeeAI can summarize findings in business language.