Simple sites: add a new option to site privacy settings

[ariel-maidana]

What

With the advent of LLM and bots that crawl the web to train them, we've added new rules to simple sites' robots.txt. This has resulted in many unhappy users, who are seeing how their sites cannot be accessed by Bingbot and other search crawlers: #83341

While it's not possible to allow simple site users to customize their robots.txt, we could add a fourth mode to the Site Privacy section of the General Settings screen in Calypso, allowing users to remove any restrictions from their robots.txt file.

This setting could be checked by default, or we could phrase it differently ("Allow AI crawlers...") and leave it unchecked.

Why

This change would allow users of simple to have more control over the way their site can be accessed by crawlers, improving their user experience.

Many users are not worried about their intellectual property or whether or not their content is used to train AIs, and are much more worried about whether or not they show up in all search engines (business owners are the most typical case, though not the only one).

How

No response

[fredrikekelund]

I say #87267 resolved this. We now have a Prevent third-party sharing for SITE option for Simple and Atomic sites that disallows a list of AI bots through robots.txt when enabled.