User-agent: * # don't crawl dynamic pages Disallow: /ws/cgi/