GoToSocial/internal/web
Daenney 0cce2c0838
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers

* [feature] Block the other OpenAI crawler

* [feature] Block Common Crawl crawler

This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.

Fixes: #2240

* [feature] Block Omgilikebot

Used by some shady big web data engine company.

* [feature] Block Meta's language model crawler

* [feature] Block well-known.dev crawler
2023-09-30 20:44:57 +01:00
..
about.go [chore] Prefer JSON errors in API endpoints (#1766) 2023-05-12 10:16:41 +02:00
assets.go [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
base.go [chore] Prefer JSON errors in API endpoints (#1766) 2023-05-12 10:16:41 +02:00
confirmemail.go [chore] Prefer JSON errors in API endpoints (#1766) 2023-05-12 10:16:41 +02:00
customcss.go [chore] Prefer JSON errors in API endpoints (#1766) 2023-05-12 10:16:41 +02:00
domain-blocklist.go [chore] Prefer JSON errors in API endpoints (#1766) 2023-05-12 10:16:41 +02:00
etag.go [feature] simpler cache size configuration (#2051) 2023-08-03 11:34:35 +02:00
opengraph.go [bugfix] Use custom bluemonday policy to disallow inline img tags (#2100) 2023-08-11 14:40:11 +02:00
opengraph_test.go [bugfix] Fix invalid og:description on account w/ empty note (#1733) 2023-05-03 16:18:34 +02:00
profile.go [bugfix] Reorder web view logic, other small fixes (#1954) 2023-07-07 14:58:53 +02:00
robots.go [feature] Block a bunch of "AI" crawlers (#2239) 2023-09-30 20:44:57 +01:00
rss.go [bugfix] Tidy up rss feed serving; don't error on empty feed (#1970) 2023-07-10 17:05:59 +02:00
settings-panel.go [chore] Prefer JSON errors in API endpoints (#1766) 2023-05-12 10:16:41 +02:00
tag.go [feature] Hashtag federation (in/out), hashtag client API endpoints (#2032) 2023-07-31 15:47:35 +02:00
thread.go [bugfix] Reorder web view logic, other small fixes (#1954) 2023-07-07 14:58:53 +02:00
web.go [feature] Hashtag federation (in/out), hashtag client API endpoints (#2032) 2023-07-31 15:47:35 +02:00