# robots.txt for https://boldautotransport.com/ # Updated: 2026-04-23 — Phase 0 SEO deployment # --------------------------------------------------------------- # Default: all well-behaved search engines welcome # --------------------------------------------------------------- User-agent: * Disallow: /cpresources/ Disallow: /vendor/ Disallow: /.env Disallow: /cache/ Disallow: /admin/ Disallow: /actions/ Disallow: /cp/ Disallow: /index.php?* Disallow: /*?utm_* Disallow: /*?fbclid=* Disallow: /*?gclid=* # --------------------------------------------------------------- # AI search engines — explicitly welcome # (these crawlers feed ChatGPT, Claude, Perplexity, Gemini, Copilot, etc.) # --------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: CCBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: FacebookBot Allow: / # --------------------------------------------------------------- # Scraper blocklist (low-signal, high-cost crawlers that burn bandwidth) # NOTE: SemrushBot + AhrefsBot are explicitly allowed — we use them for # our own rank/backlink intelligence. Don't blindfold ourselves. # --------------------------------------------------------------- User-agent: Bytespider Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / # --------------------------------------------------------------- # Sitemaps # --------------------------------------------------------------- Sitemap: https://boldautotransport.com/sitemap.xml Sitemap: https://boldautotransport.com/sitemaps-1-sitemap.xml Sitemap: https://boldautotransport.com/sitemap-routes.xml