Exclude site from ChatGPT scraping

Your second example prevents all user-agents (crawlers) from indexing your site. So, in theory, that should be enough as long as OpenAI respects the robots.txt standard. It’s not clear to me if they do, so to be on the safe side, you might want to add ChatGPT-User and GPTBot explicitly:

User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /

The robots.txt file can be added via a Custom Theme, sparsley documented here.

4 Likes