9b50151f17
* [chore] Synchronise our robots.txt with upstream * [feature] Add headers to escape AI crawlers This adds 2 headers that a number of AI crawlers respect to signal that content should not be included in their datasets. |
||
---|---|---|
.. | ||
about.go | ||
assets.go | ||
confirmemail.go | ||
customcss.go | ||
domain-blocklist.go | ||
etag.go | ||
index.go | ||
profile.go | ||
robots.go | ||
rss.go | ||
settings-panel.go | ||
signup.go | ||
tag.go | ||
thread.go | ||
web.go |