Skip to content
agentusehub
Sign in+ Submit
Sales, Marketing & SupportAny

Audit cleaning-company sites for AI crawler access

An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.

Read the experience
Time saved
Not reported
Setup time
Not reported
Running cost
Not reported
Cost range
Not reported

Use case overview

An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.

Read the original report ↗

Reported experience

  1. What they did

    The authors took domains with "cleaning" in the name from the Tranco top-1M list and filtered them down to 18 service firms. For each site they called two Site Check tools. One read robots.txt, headers and redirects for 14 named AI crawlers. The other checked 8 page tags. Site selection and filtering were done by hand from page titles.

  2. How it went

    9 of 18 sites returned their home page. Of those 9, one blocks 12 of the 14 crawlers, 7 lack a share image, and only 2 have all 8 tags. The other 9 sites gave a 403, a CDN error or no answer.

  3. Worth knowing

    A refused or failed fetch doesn't show what AI bots receive, and firewall or CDN rules can block bots that robots.txt allows. Treat 9 sites as a snapshot, not a survey.

Example prompts

Try with Claude Code
Here is a list of [number] company websites: [list of domains]. For each one, fetch robots.txt and report whether it allows or blocks major AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Also check the homepage for meta tags that affect AI or search access, and flag any site that returns its homepage instead of a real robots.txt. Finish with a single table of results and a short list of the biggest gaps.

Related use cases