Sitemap: https://tryremotely.com/sitemaps/sitemap.xml.gz # Machine-readable entry points (not directives — robots.txt is just the first # file a crawler fetches): # https://tryremotely.com/llms.txt — what Remotely is for, and when to use it # https://tryremotely.com/openapi.json — OpenAPI spec for the public JSON API # https://tryremotely.com/api — human-readable API documentation # AI Crawlers — explicitly allowed User-agent: GPTBot Allow: / Allow: /api/v1 Allow: /api/openapi Disallow: /admin/ Disallow: /api/ Disallow: /recruit/ Disallow: /talent/ User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Amazonbot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Cohere-ai Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: Perplexity-User Allow: / User-agent: CCBot Allow: / User-agent: Meta-ExternalAgent Allow: / # Default rules User-agent: * Allow: / # Longer rules win, so these override the Disallow below. Allow: /api/v1 Allow: /api/openapi Disallow: /admin/ Disallow: /api/ Disallow: /apply/ Disallow: /subscribe/ Disallow: /job-alerts/ Disallow: /recruit/ Disallow: /talent/ Disallow: /job-listings/*/favorite Disallow: /job-listings/*/unfavorite Disallow: /tools/*/new Disallow: /tools/*/create Disallow: /*?*utm_* Disallow: /*?*fbclid* Disallow: /*?*gclid* # Filter refinements: these query-string params never represent a canonical # facet page (the controller already noindexes them). Block crawling so they # don't waste crawl budget — noindex alone still gets the page fetched. Disallow: /*?*sort_by= Disallow: /*?*keyword= Disallow: /*?*exclude_keyword= Disallow: /*?*min_salary= Disallow: /*?*max_salary= User-agent: SemrushBot Disallow: / User-agent: SemrushBot-BA Disallow: / # Majestic - SEO User-agent: MJ12bot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / # Seozoom - SEO User-Agent: ZoomBot Disallow: / User-agent: sistrix Disallow: / User-agent: serpstatbot Disallow: / User-agent: MozBot Disallow: / # https://moz.com/help/moz-procedures/crawlers/rogerbot User-agent: rogerbot Disallow: / # https://moz.com/help/moz-procedures/crawlers/dotbot User-agent: dotbot Disallow: / User-agent: DataForSeoBot Disallow: /