GoToSocial/internal
Daenney 0cce2c0838
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers

* [feature] Block the other OpenAI crawler

* [feature] Block Common Crawl crawler

This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.

Fixes: #2240

* [feature] Block Omgilikebot

Used by some shady big web data engine company.

* [feature] Block Meta's language model crawler

* [feature] Block well-known.dev crawler
2023-09-30 20:44:57 +01:00
..
ap [bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
api [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
cache [chore] fix typo in slice.go (#2219) 2023-09-23 18:42:34 +02:00
cleaner [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
config [feature] Implement explicit domain allows + allowlist federation mode (#2200) 2023-09-21 12:12:04 +02:00
db [bugfix] Fix paging for empty items (#2236) 2023-09-29 15:31:10 +02:00
email [bugfix] add From to email header (#1717) 2023-04-29 11:47:14 +02:00
federation [bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
gotosocial [bugfix] fix double firing bun.DB query hooks (#2124) 2023-08-17 17:26:21 +01:00
gtscontext [chore] ensure worker contexts have request ID (#2120) 2023-08-15 17:01:01 +01:00
gtserror [feature] List replies policy, refactor async workers (#2087) 2023-08-09 19:14:33 +02:00
gtsmodel [feature] Implement explicit domain allows + allowlist federation mode (#2200) 2023-09-21 12:12:04 +02:00
httpclient [bugfix] only set content-length AFTER rewinding body bytes (#2086) 2023-08-08 12:45:29 +01:00
id [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
iotools [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
log [feature] Make log format configurable (#2130) 2023-08-21 19:07:55 +01:00
media [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
messages [feature] List replies policy, refactor async workers (#2087) 2023-08-09 19:14:33 +02:00
middleware [feature] Add rate limit exceptions option, use ISO8601 for rate limit reset (#2151) 2023-08-23 14:32:27 +02:00
oauth [chore] Enable S256 code challenge (#2224) 2023-09-28 11:21:19 +02:00
oidc [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
paging [feature] add paging to AP following / followers endpoints (#2198) 2023-09-20 16:49:46 +01:00
processing [bugfix] Fix paging for empty items (#2236) 2023-09-29 15:31:10 +02:00
regexes [feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
router [bugfix] handle HEAD requests more elegantly (#2055) 2023-08-02 10:28:20 +02:00
state [feature] Add List functionality (#1802) 2023-05-25 10:37:38 +02:00
storage [feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs (#2134) 2023-08-20 13:35:55 +02:00
stream [feature] Add List functionality (#1802) 2023-05-25 10:37:38 +02:00
text [chore/bugfix] Deinterface text.Formatter, allow underscores in hashtags (#2233) 2023-09-29 10:39:56 +02:00
timeline [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
tracing [feature] Support OTLP HTTP, drop Jaeger (#2184) 2023-09-07 13:20:37 +02:00
trans [chore] Improve copyright header handling (#1608) 2023-03-12 16:00:57 +01:00
transport [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
typeutils [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
uris [feature] Hashtag federation (in/out), hashtag client API endpoints (#2032) 2023-07-31 15:47:35 +02:00
util [bugfix] Fix paging for empty items (#2236) 2023-09-29 15:31:10 +02:00
validate [feature] Allow full BCP 47 in language inputs (#2067) 2023-08-07 10:25:54 +02:00
visibility [bugfix] Move follow.show_reblogs check further up to avoid showing unwanted reblogs in home timeline (#2234) 2023-09-29 10:39:35 +02:00
web [feature] Block a bunch of "AI" crawlers (#2239) 2023-09-30 20:44:57 +01:00
workers [feature] Store admin actions in the db, prevent conflicting actions (#2167) 2023-09-04 14:55:17 +01:00