Daenney 4604224c4d
[chore] Update our robots.txt (#3033)
This syncs our copy with the current state of the ai.robots.txt
repository. Upstream has tightened their scope to be AI-only, whereas
before it included a bunch of SEO and "web intelligence" marketing
stuff. I've kept those but moved them into their own section.
2024-06-23 15:34:21 +02:00
..