robots.txt is a directive, not enforcement — well-behaved crawlers respect it, bad actors ignore it. Use for: blocking admin areas, excluding staging environments, pointing to your sitemap. Never rely on robots.txt to hide sensitive data — use authentication.