why even bother with a robots.txt file anymore
We just pushed our new technical documentation live and the server logs immediately lit up with unknown IP addresses pulling the entire directory down. Tech forums keep repeating the same advice: update your robots.txt file and add Disallow directives for GPTBot and ClaudeBot. Does relying…