Discovery Surface: robots.txt, sitemap.xml, security.txt, and Technology Fingerprinting

Feature docs · Discovery surface

Website security analysis does not start with finding vulnerabilities alone. First you need to understand the discovery surface an attacker can already see.

At Guardbee, the first step is analyzing publicly reachable information to reveal how the application is structured, which technologies it uses, and how security teams can be contacted. This passive analysis produces useful security signals without harming the target system.

What is the discovery surface?

The discovery surface is the set of technical details that are publicly reachable on the internet.

An attacker often looks at these first:

• robots.txt
• sitemap.xml
• security.txt
• HTTP headers
• HTML source
• JavaScript files
• Framework fingerprints
• CDN details
• Server technology
• DNS records

These details may not be vulnerabilities by themselves, but they can shape the direction of an attack.

Guardbee analyzes them automatically and reports findings by risk level.

1. robots.txt analysis

robots.txt tells search engines which directories should not be crawled.

Example:

User-agent: *
Disallow: /admin/
Disallow: /backup/
Disallow: /test/
Disallow: /old/

Even though this is written for crawlers, it also creates useful hints for an attacker.

For example:

• /admin — may be an admin panel.
• /backup — may expose backup files.
• /test — may be a development environment.

Listing these paths does not mean they are vulnerable, but it makes discovery much easier.

Guardbee:

• downloads robots.txt
• analyzes Allow / Disallow rules
• classifies paths that look sensitive
• produces a security risk score

2. sitemap.xml analysis

A sitemap is a list of URLs on the site.

Example:

<url>
   <loc>https://example.com/admin</loc>
</url>
<url>
   <loc>https://example.com/api/docs</loc>
</url>
<url>
   <loc>https://example.com/internal</loc>
</url>

Forgotten test pages, old admin panels, or API documentation sometimes still appear here.

Guardbee:

• finds sitemap files
• follows nested sitemaps
• extracts URLs
• flags sensitive endpoints
• reports pages that should not be indexed

3. security.txt checks

security.txt is the standard file that tells security researchers who to contact when they find a vulnerability.

Location:

https://site.com/.well-known/security.txt

Example:

Contact: mailto:security@example.com
Expires: 2027-01-01
Policy: https://example.com/security

Many organizations still do not publish this file.

Missing it is not a critical vulnerability by itself, but it is an important maturity signal.

Guardbee:

• checks whether security.txt exists
• validates RFC 9116 alignment
• reviews the Contact field
• checks the Expires date
• reports missing required fields

4. Technology fingerprinting

An attacker tries to identify the target stack. For example:

• Apache, Nginx, IIS
• Cloudflare
• Next.js, Angular, React, Vue
• WordPress
• PHP, ASP.NET, Node.js
• Laravel, Django

Knowing the stack makes it easier to research known vulnerabilities.

Guardbee analyzes signals without running an active attack. Some of them:

HTTP headers

• Server
• X-Powered-By
• Via
• CF-Ray

Example:

Server: nginx
X-Powered-By: PHP/8.1

HTML source

Meta tags:

<meta name="generator" …>

Framework scripts and traces:

• _next, nuxt, webpack, vite
• react, angular

JavaScript files

File names:

• main.js, vendor.js, runtime.js

Webpack signatures, Angular bundles, React hydration code, and Vue runtime files are also evaluated.

Cookie names

Examples:

• PHPSESSID
• ASP.NET_SessionId
• JSESSIONID
• connect.sid

These names often hint at the underlying stack.

HTTP response behavior

Examples:

• 404 behavior
• redirect patterns
• cache headers
• compression
• ETag
• HSTS

These can indicate CDN or reverse-proxy usage.

How does Guardbee do this?

Guardbee uses a fully passive discovery approach.

Process order:

1. Validates the domain.
2. Downloads robots.txt.
3. Discovers sitemap.xml files.
4. Checks security.txt.
5. Analyzes HTTP headers.
6. Inspects HTML source.
7. Evaluates JavaScript files.
8. Matches technology signatures.
9. Scores security and privacy signals.
10. Reports results in business language.

No attack is performed against the target, no brute-force is applied, and no malicious requests are sent.

How can you test it manually?

Many checks can also be done by hand.

robots.txt

https://site.com/robots.txt

Check:

• Are admin directories listed?
• Are test folders listed?
• Are old directories still present?

sitemap.xml

https://site.com/sitemap.xml

Check:

• Are admin pages present?
• Are test URLs indexed?
• Is API documentation published?

security.txt

https://site.com/.well-known/security.txt

Check:

• Does the file exist?
• Is the contact information correct?
• Has the expiry date passed?

HTTP headers

From a terminal:

curl -I https://site.com

Inspect:

• Server
• X-Powered-By
• Strict-Transport-Security
• Content-Security-Policy
• Referrer-Policy
• Permissions-Policy

Technology detection

Using browser developer tools:

• Inspect response headers.
• View HTML source.
• Check JavaScript files.
• Review cookie names.
• Analyze requests in the Network tab.

Best practices

• Do not advertise sensitive directory names in robots.txt.
• Keep sitemaps limited to publicly intended pages.
• Publish an RFC 9116-aligned security.txt.
• Remove unnecessary Server and X-Powered-By headers.
• Avoid outdated framework versions.
• Configure security headers correctly (HSTS, CSP, Referrer-Policy, and similar).
• Run technology fingerprint scans regularly to catch unexpected stack changes.

Conclusion

Discovery surface analysis lets you see the same information an attacker can gather in the first few minutes. Passive signals such as robots.txt, sitemap.xml, security.txt, and technology fingerprints may not be vulnerabilities on their own, but misconfigurations can narrow an attacker’s focus and increase risk.

Guardbee automatically analyzes this surface to evaluate visible technologies, configuration gaps, and security maturity. That means you see not only what was found, but why it matters and how to improve it—in clear language.

Related guide: sensitive paths.

Frequently asked questions

Is security.txt mandatory?

No. It is recommended for responsible disclosure and is a maturity signal. Guardbee checks RFC 9116 alignment plus Contact / Expires fields.

Do paths listed in robots.txt or a sitemap mean they are vulnerable?

No. Listing makes discovery easier; accessibility and vulnerability still need separate checks.

Is technology fingerprinting an active attack?

No. Guardbee works from passive signals (headers, HTML, JS, cookies, and similar). It does not brute-force or send malicious requests.

Is fingerprinting for competitor intelligence?

No. It is for seeing your own brand’s externally visible technology and spotting unexpected changes.