User-agent: * # all robots Disallow: /admin/ Disallow: /db/ Disallow: /includes/ Disallow: /intranet/ Disallow: /login/ Disallow: /public/