IT lexicon › Web › robots.txt

robots.txt

Web På svenska → Updated: 2026-05-23

A text file at a site's root telling web crawlers (Googlebot, Bingbot) what they can and can't index.

Disallow: /admin/. Honour-based — ignored by malicious bots. Not security, just etiquette. Also not indexing protection: use the noindex meta tag for that.

← Back to the lexicon