User-agent: * Crawl-delay: 10 Disallow: /invoice/* Disallow: /review-trip/* Disallow: /invoice/* Disallow: /media/attachments/* Disallow: /about-test Disallow: */media/ Disallow: */pdf/ Disallow: /booking-form/ Disallow: /admin/ User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: serpstatbot Disallow: / User-Agent: MJ12bot Disallow: / User-agent: PetalBot Disallow: / User-Agent: aspiegelbot Disallow: / User-Agent: Seekport Crawler Disallow: / User-agent: barkrowler Disallow: / User-agent: Barkrowler/0.9 Disallow: / User-agent: clark-crawler2/Nutch-1.19-SNAPSHOT Disallow: / User-agent: SeekportBot Disallow: / User-agent: clark-crawler2 Disallow: / User-agent: Orbbot Disallow: / User-agent: GeedoBot Disallow: / User-agent: Bytespider Disallow: / User-agent: ZoominfoBot Disallow: /