How to block specific web crawlers on a website? How to block GPTBot or ClaudeBot crawlers?

Recently, our website has been overwhelmed by a surge in crawls from two types of web crawlers: GPTBot and ClaudeBot.

The server was under heavy load throughout the entire day (over the course of one month, it handled 7 TB of traffic – absolutely insane!).

Upon reviewing the access logs, it turned out that these two types of web crawlers were the ones causing the trouble.

So, I came up with a solution and blocked it.

Add the following code to your website's pseudo-static routing rules; if the request is from a GPTBot or ClaudeBot spider, simply return a 502 error.

if ($http_user_agent ~* "GPTBot|ClaudeBot") {
    return 502;
}

Subsequently, I added both categories of spider IPs to the firewall's blocklist.

The website has been fully restored to normal operation.