GoToSocial/internal
Daenney 0cce2c0838
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers

* [feature] Block the other OpenAI crawler

* [feature] Block Common Crawl crawler

This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.

Fixes: #2240

* [feature] Block Omgilikebot

Used by some shady big web data engine company.

* [feature] Block Meta's language model crawler

* [feature] Block well-known.dev crawler
2023-09-30 20:44:57 +01:00
..
ap [bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
api [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
cache [chore] fix typo in slice.go (#2219) 2023-09-23 18:42:34 +02:00
cleaner [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
config [feature] Implement explicit domain allows + allowlist federation mode (#2200) 2023-09-21 12:12:04 +02:00
db [bugfix] Fix paging for empty items (#2236) 2023-09-29 15:31:10 +02:00
email
federation [bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
gotosocial [bugfix] fix double firing bun.DB query hooks (#2124) 2023-08-17 17:26:21 +01:00
gtscontext [chore] ensure worker contexts have request ID (#2120) 2023-08-15 17:01:01 +01:00
gtserror [feature] List replies policy, refactor async workers (#2087) 2023-08-09 19:14:33 +02:00
gtsmodel [feature] Implement explicit domain allows + allowlist federation mode (#2200) 2023-09-21 12:12:04 +02:00
httpclient
id
iotools
log [feature] Make log format configurable (#2130) 2023-08-21 19:07:55 +01:00
media [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
messages [feature] List replies policy, refactor async workers (#2087) 2023-08-09 19:14:33 +02:00
middleware [feature] Add rate limit exceptions option, use ISO8601 for rate limit reset (#2151) 2023-08-23 14:32:27 +02:00
oauth [chore] Enable S256 code challenge (#2224) 2023-09-28 11:21:19 +02:00
oidc
paging [feature] add paging to AP following / followers endpoints (#2198) 2023-09-20 16:49:46 +01:00
processing [bugfix] Fix paging for empty items (#2236) 2023-09-29 15:31:10 +02:00
regexes
router
state
storage [feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs (#2134) 2023-08-20 13:35:55 +02:00
stream
text [chore/bugfix] Deinterface text.Formatter, allow underscores in hashtags (#2233) 2023-09-29 10:39:56 +02:00
timeline [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
tracing [feature] Support OTLP HTTP, drop Jaeger (#2184) 2023-09-07 13:20:37 +02:00
trans
transport [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
typeutils [chore] deinterface the typeutils.Converter and update to use state structure (#2217) 2023-09-23 18:44:11 +02:00
uris
util [bugfix] Fix paging for empty items (#2236) 2023-09-29 15:31:10 +02:00
validate
visibility [bugfix] Move follow.show_reblogs check further up to avoid showing unwanted reblogs in home timeline (#2234) 2023-09-29 10:39:35 +02:00
web [feature] Block a bunch of "AI" crawlers (#2239) 2023-09-30 20:44:57 +01:00
workers [feature] Store admin actions in the db, prevent conflicting actions (#2167) 2023-09-04 14:55:17 +01:00