Block unnecessary web crawlers from accessing your website by modifying the robots.txt file.

I've built a small website that gets randomly crawled by search engines every day.

So, I created a robots.txt file to block all those useless, ridiculous spiders.

This is the first step.

User-agent: DotBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SplitSignalBot
Disallow: /

User-agent: SemrushBot-OCOB
Disallow: /

User-agent: SemrushBot-FT
Disallow: /

User-agent: RyteBot
Disallow: /

User-agent: SemrushBot-ESI
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: TrendictionBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Anthropic-ai
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: *
Disallow: 
Crawl-delay: 10

Step 2: When the silly spider visits my website, it returns a direct 403 response.

Pseudo-static code – I'm using an Nginx environment.

# 屏蔽冷门爬虫（不遵守 robots.txt 或只采集不贡献流量的）
if ($http_user_agent ~* "(DotBot|SemrushBot|SiteAuditBot|SemrushBot-BA|SemrushBot-SI|SemrushBot-SWA|SplitSignalBot|SemrushBot-OCOB|SemrushBot-FT|RyteBot|SemrushBot-ESI|MJ12bot|AhrefsBot|BLEXBot|TrendictionBot|ClaudeBot|Anthropic-ai|GPTBot|Amazonbot|PetalBot)") {
    return 403;
}

 List Explanation

• DotBotMoz's web crawler.

• Semrush All-in-One Package: Includes the main crawler and various specialized clones for different tools (SemrushBot、SiteAuditBot、SemrushBot-BA   etc.), single-use full-seal.

• MJ12bot、AhrefsBot、BLEXBot、TrendictionBotThese are renowned international SEO/data analysis crawlers that solely collect data without driving any traffic.

• ClaudeBot、Anthropic-ai、GPTBotUsing AI-powered web crawlers – if your server load was previously at 100%, it could be because they were running a massive crawling campaign.

• Amazonbot、PetalBotOther commercial web crawlers are equally useless.

Why use 403 instead of 502?

• 403 ForbiddenA clear "Deny Access" signal will inform the web crawler that access is prohibited; as a result, it will typically stop crawling.

• 502 Bad GatewayThis indicates a gateway error, which may lead the crawler to misinterpret it as a temporary server outage; as a result, the crawler may repeatedly retry the request, thereby increasing the system load.