Issue ACL setup for Google website crawl

[Thomas Oryon]

Server operating system version

Redhat linux

Plesk version and microupdate number

18.0.74

Hello Support,

We would like to setup ACL (Access Control Limit) to stop/block the Google to crawl the website. Also, once we setup this option, the domain which was already added and upcoming domain add will also should not crawl by Google. Is that possible to setup? If yes, please let us know the steps to setup this.

Awaiting your reply.

 

[scsa20]

Why not just configure robots.txt? Google does honors whatever is configured on robots.txt. You can learn more over at https://www.cloudflare.com/learning/bots/what-is-robots-txt/ and you can utilize Robots.txt Generator

 

[Thomas Oryon]

Hello Support,

We can't add robots.txt for each domain name. We wish to block server wide. Can you let us know how to setup this?

Awaiting your reply

 

[scsa20]

You're best bet would still use robots.txt unless you want to edit your firewall rules to block all of Googlebot's IP addresses. See Verify Requests from Google Crawlers and Fetchers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers for list.

 

[Thomas Oryon]

Hello Support,

Thanks for the update. Can you let us know the steps to enable the robots.txt in server wide?

Awaiting your reply.

 

[scsa20]

robots.txt cannot be set server wide, it needs to be done per domain, otherwise just block the IP address at the firewall level (check the link I provided originally to get the list of IP addresses used by the crawlers).

 

[Thomas Oryon]

Hello Support,

Thanks for the update. Can you please confirm that it was not possible to setup ACL for the website crawl as server wide because you have suggested the robots.txt which can be setup under each domain name.

Also, can you please confirm that whether any alternate option available to setup the Google crawl block for server wide?

Awaiting your reply.

 

[Maarten]

You need to use custom server-wide web server configuration files, as described in this guide:

https://support.plesk.com/hc/en-us/articles/12377559696151-How-to-create-custom-Apache-and-nginx-configuration-files-for-all-domains-in-Plesk

In short, the steps are:

• Create a custom Apache or Nginx include file on server level (not per domain).
• Add rules to block the Google crawler based on the User-Agent.
• Restart the web server so the configuration is applied to all domains.

Example for Nginx, blocking Googlebot:

if ($http_user_agent ~* "googlebot") {
    return 403;
}

This configuration file is automatically included for every domain, which makes it a server-wide crawl block.

Important notes:

• There is no GUI option in Plesk for this.
• This affects all websites hosted on the server.
• Blocking Googlebot globally can have a serious SEO impact for all subscriptions.

It’s strongly recommended to test this on a single subscription first before applying it globally.