Commit Graph

89 Commits

Author SHA1 Message Date
tobi 2ddeb9b514
[chore] Add envparsing test for advanced-header-filter-mode (#3051) 2024-06-28 15:28:00 +02:00
Vyr Cossont 5e2d4fdb19
[feature] User muting (#2960)
* User muting

* Address review feedback

* Rename uniqueness constraint on user_mutes to match convention

* Remove unused account_id from where clause

* Add UserMute to NewTestDB

* Update test/envparsing.sh with new and fixed cache stuff

* Address tobi's review comments

* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired

* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 16:38:02 +00:00
kim 5dcc954072
[feature] do not uncache status / emoji media if attached status is bookmarked (#2956)
* do not uncache status / emoji media if attached status is bookmarked

* add status bookmark and bookmark IDs caches

* update status bookmark tests

* move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache

* fix envparsing.sh test
2024-06-06 11:44:43 +01:00
kim 1e7b32490d
[experiment] add alternative wasm sqlite3 implementation available via build-tag (#2863)
This allows for building GoToSocial with [SQLite transpiled to WASM](https://github.com/ncruces/go-sqlite3) and accessed through [Wazero](https://wazero.io/).
2024-05-27 17:46:15 +02:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types (#2838)
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
Vyr Cossont fc3741365c
[bugfix] Fix Swagger spec and add test script (#2698)
* Add Swagger spec test script

* Fix Swagger spec errors not related to statuses with polls

* Add API tests that post a status with a poll

* Fix creating a status with a poll from form params

* Fix Swagger spec errors related to statuses with polls (this is the last error)

* Fix Swagger spec warnings not related to unused definitions

* Suppress a duplicate list update params definition that was somehow causing wrong param names

* Add Swagger test to CI

- updates Drone config
- vendorizes go-swagger
- fixes a file extension issue that caused the test script to generate JSON instead of YAML with the vendorized version

* Put `Sample: ` on its own line everywhere

* Remove unused id param from emojiCategoriesGet

* Add 5 more pairs of profile fields to account update API Swagger

* Remove Swagger prefix from dummy fields

It makes the generated code look weird

* Manually annotate params for statusCreate operation

* Fix all remaining Swagger spec warnings

- Change some models into operation parameters
- Ignore models that already correspond to manually documented operation parameters but can't be trivially changed (those with file fields)

* Documented that creating a status with scheduled_at isn't implemented yet

* sign drone.yml

* Fix filter API Swagger errors

* fixup! Fix filter API Swagger errors

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-03-06 18:05:45 +01:00
tobi b22e213e15
[feature/chore] Add Move database functions + cache (#2647)
* [feature/chore] Add Move database functions + cache

* add move mem ratio to envparsing.sh

* update comment
2024-03-06 11:18:57 +01:00
Vyr Cossont 61a2b91f45
[feature] Filters v1 (#2594)
* Implement client-side v1 filters

* Exclude linter false positives

* Update test/envparsing.sh

* Fix minor Swagger, style, and Bun usage issues

* Regenerate Swagger

* De-generify filter keywords

* Remove updating filter statuses

This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered.

* Add a test for filter statuses specifically

* De-generify filter statuses

* Inline FilterEntry

* Use vertical style for Bun operations consistently

* Add comment on Filter DB interface

* Remove GoLand linter control comments

Our existing linters should catch these, or they don't matter very much

* Reduce memory ratio for filters
2024-03-06 11:15:58 +01:00
kim 90dcc008c5
[feature] add script to test import / export cycle of a gotosocial instance (#2693)
* add script to test import / export cycle of a gotosocial instance

* use suffix for setting mktemp file suffix
2024-02-27 16:29:56 +01:00
9p4 9bf448be7a
[feature/oidc] Add support for very basic RBAC (#2642)
* Add support for very basic RBAC

* Add some small tests for allowedGroup and adminGroup

* Switch to table-driven tests
2024-02-27 16:07:29 +01:00
tobi 9cadc764b3
[feature] Add experimental `instance-federation-spam-filter` option (#2685)
* [chore] Move `visibility` to `filter/visibility`

* [feature] Add experimental instance-federation-spam-filter option
2024-02-27 12:22:05 +00:00
tobi 4b0eefbcc9
[chore] Increase default max image description to 1500 chars, collapse cw char count into status (#2682)
* [chore] Make default max image description 1500 chars, collapse cw char count into status

* oops

* tests
2024-02-23 18:28:09 +00:00
kim 8ebb7775a3
[feature] request blocking by http headers (#2409) 2023-12-18 14:18:25 +00:00
Tsuribori 1ba3e14b36
[feature] Initial Prometheus metrics implementation (#2334)
* feat: Initial OTEL metrics

* docs: add metrics documentation

* fix: metrics endpoint conditional check

* feat: metrics endpoint basic auth

* fix: make metrics-auth-enabled default false

* fix: go fmt helpers.gen.go

* fix: add metric-related env vars to envparsing.sh

* fix: metrics docs

* fix: metrics related stuff in envparsing.sh

* fix: metrics docs

* chore: metrics docs wording

* fix: metrics stuff in envparsing?

* bump otel versions

---------

Co-authored-by: Tsuribori <user@acertaindebian>
Co-authored-by: Tsuribori <none@example.org>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-11-20 16:43:55 +01:00
tobi fc02d3c6f7
[feature] Set/show instance language(s); show post language on frontend (#2362)
* update go text, include text/display

* [feature] Set instance langs, show post lang on frontend

* go fmt

* WebGet

* set language for whole article, don't use FA icon

* mention instance languages + other optional config vars

* little tweak

* put languages in config properly

* warn log language parse

* change some naming around

* tidy up validate a bit

* lint

* rename LanguageTmpl in template
2023-11-17 11:35:28 +01:00
kim e9e5dc5a40
[feature] add support for polls + receiving federated status edits (#2330) 2023-11-08 14:32:17 +00:00
tobi 4dc0547dc0
[feature] Customizable media cleaner schedule (#2304) 2023-10-30 17:35:11 +00:00
tobi c7b6cd7770
[feature] Status thread mute/unmute functionality (#2278)
* add db models + functions for keeping track of threads

* give em the old linty testy

* create, remove, check mutes

* swagger

* testerino

* test mute/unmute via api

* add info log about new index creation

* thread + allow muting of any remote statuses that mention a local account

* IsStatusThreadMutedBy -> IsThreadMutedByAccount

* use common processing functions in status processor

* set = NULL

* favee!

* get rekt darlings, darlings get rekt

* testrig please, have mercy muy liege
2023-10-25 15:04:53 +01:00
tobi 183eaa5b29
[feature] Implement explicit domain allows + allowlist federation mode (#2200)
* love like winter! wohoah, wohoah

* domain allow side effects

* tests! logging! unallow!

* document federation modes

* linty linterson

* test

* further adventures in documentation

* finish up domain block documentation (i think)

* change wording a wee little bit

* docs, example

* consolidate shared domainPermission code

* call mode once

* fetch federation mode within domain blocked func

* read domain perm import in streaming manner

* don't use pointer to slice for domain perms

* don't bother copying blocks + allows before deleting

* admonish!

* change wording just a scooch

* update docs
2023-09-21 12:12:04 +02:00
Daenney 14ef098099
[feature] Support OTLP HTTP, drop Jaeger (#2184)
* [feature] Add http trace exporter, drop Jaeger

Jaeger supports ingesting traces using the OpenTelemetry gRPC or HTTP
methods. The Jaeger project has deprecated the old jaeger transport.

* Add support for submitting traces over HTTP
* Drop support for the old Jaeger protocol
* Upgrade the trace libraries to v1.17

Fixes: #2176
Fixes: #2179
2023-09-07 13:20:37 +02:00
rdelaage 7b48437f17
[feature] list commands for both attachment and emojis (#2121)
* [feature] list commands for both attachment and emojis

* use fewer commands, provide `local-only` and `remote-only` as filters

* envparsing

---------

Co-authored-by: Romain de Laage <romain.delaage@rdelaage.ovh>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-23 18:01:16 +02:00
tobi 8f38dc2e7f
[feature] Add rate limit exceptions option, use ISO8601 for rate limit reset (#2151)
* start updating rate limiting, add exceptions

* tests, comments, tidying up

* add rate limiting exceptions to example config

* envparsing

* nolint

* apply kimbediff

* add examples
2023-08-23 14:32:27 +02:00
Daenney 4ae16bce8c
[feature] Make log format configurable (#2130)
* [feature] Don't emit timestamp in log lines

When running gotosocial with a service manager like systemd, or a
container runtime, the associated log driver usually emits timestamps
itself. In those cases, having the extra timestamp from our own log
lines ends up being a bit noisy and when centrally ingesting logs is
duplicate information.

This introduces a configuration flag that allows disabling emitting the
timestamp. It's only wired up for "daemonised" processes, meaning server
and testrig.

* [chore] Add docs for log-timestamp

* [feature] Simplify timestamp handling

Co-Authored-By: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>

* [chore] Less escaped double-quotes

* [chore] Fix help string

---------

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-08-21 19:07:55 +01:00
tobi 1e2db7a32f
[feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs (#2134)
* [feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs

* env parsing tests, my coy mistress
2023-08-20 13:35:55 +02:00
kim 815b5291e0
[bugfix] fix inconsistent calculated cache sizes (#2115)
* use calculated exampleTime instead of `time.Now()` to ensure no locale data, retweak cache ratios

* update envparsing test

* update default cache memory to 100MiB

* fix envparsing with latest cache target default

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-14 17:08:19 +01:00
kim 91cbcd589e
[performance] remove last of relational queries to instead rely on caches (#2091) 2023-08-10 15:08:41 +01:00
kim 3920bc87d1
[bugfix] don't accept unrelated statuses (#2078)
Co-authored-by: Daenney <daenney@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-08 12:26:34 +01:00
kim 9a291dea84
[performance] add caching of status fave, boost of, in reply to ID lists (#2060) 2023-08-04 12:28:33 +01:00
kim 00adf18c24
[feature] simpler cache size configuration (#2051)
* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-03 11:34:35 +02:00
tobi 2be83fdca5
[feature] Allow users to skip http client tls verification for testing purposes (with appropriately loud warnings) (#2052) 2023-08-01 18:50:17 +01:00
tobi 2796a2e82f
[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)
* update go-fed

* do the things

* remove unused columns from tags

* update to latest lingo from main

* further tag shenanigans

* serve stub page at tag endpoint

* we did it lads

* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)

* swagger docs

* document hashtag usage + federation

* instanceGet

* don't bother parsing tag href

* rename whereStartsWith -> whereStartsLike

* remove GetOrCreateTag

* dont cache status tag timelineability
2023-07-31 15:47:35 +02:00
kim ed2477ebea
[performance] cache follow, follow request and block ID lists (#2027) 2023-07-31 11:25:29 +01:00
Vyr Cossont b874e9251e
[feature] Implement markers API (#1989)
* Implement markers API

Fixes #1856

* Correct import grouping in markers files

* Regenerate Swagger for markers API

* Shorten names for readability

* Cache markers for 6 hours

* Update DB ref

* Update envparsing.sh
2023-07-29 12:49:14 +02:00
Vyr Cossont 22ac4607a1
[feature] Support setting private notes on accounts (#1982)
* Support setting private notes on accounts

* Reformat comment whitespace

* Add missing license headers

* Use apiutil.ParseID

* Rename Note model and cache to AccountNote

* Update golden cache config in test/envparsing.sh

* Rename gtsmodel/note.go to gtsmodel/accountnote.go

* Update AccountNote uniqueness constraint name

Now has same prefix as other indexes on this table.

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2023-07-27 10:30:39 +02:00
kim 9eff0d46e4
[feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
Daenney f8f0312042
[feature] Report Masto version in /api/v1/instance (#1977) 2023-07-21 18:49:13 +01:00
tobi 2a99df0588
[feature] enable + document explicit IP dialer allowing/denying (#1950)
* [feature] enable + document explicit IP dialer allowing/denying

* lord have mercy

* allee jonge

* shortcut check ipv6 prefixes

* comment

* separate httpclient_test, export Sanitizer
2023-07-07 16:17:39 +02:00
tobi e70bf8a6c8
[chore/bugfix] Domain block tidying up, Implement first pass of `207 Multi-Status` (#1886)
* [chore/refactor] update domain block processing

* expose domain block import errors a lil better

* move/remove unused query keys
2023-07-07 11:34:12 +02:00
tobi fc524f8cf1
[feature] Configurable custom css length (#1827)
* [feature] Make accounts custom css length configurable

* test custom css validation
2023-05-25 15:18:15 +02:00
tobi f5c004d67d
[feature] Add List functionality (#1802)
* start working on lists

* further list work

* test list db functions nicely

* more work on lists

* peepoopeepoo

* poke

* start list timeline func

* we're getting there lads

* couldn't be me working on stuff... could it?

* hook up handlers

* fiddling

* weeee

* woah

* screaming, pissing

* fix streaming being a whiny baby

* lint, small test fix, swagger

* tidying up, testing

* fucked! by the linter

* move timelines to state like a boss

* add timeline start to tests using state

* invalidate lists
2023-05-25 10:37:38 +02:00
Daenney 107237c8e8
[feature] Make client IP logging configurable (#1799) 2023-05-21 16:12:47 +01:00
Dominik Süß 6392e00653
feat: initial tracing support (#1623) 2023-05-09 18:19:48 +01:00
tobi cbb9e2d3f0
[chore/performance] Make sender multiplier configurable (#1750) 2023-05-08 18:03:38 +01:00
kim de6e3e5f2a
[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility

* invalidate visibility cache items on account / status deletes

* fix requester ID passed to visibility cache nil ptr

* de-interface caches, fix home / public timeline caching + visibility

* finish adding code comments for visibility filter

* fix angry goconst linter warnings

* actually finish adding filter visibility code comments for timeline functions

* move home timeline status author check to after visibility

* remove now-unused code

* add more code comments

* add TODO code comment, update printed cache start names

* update printed cache names on stop

* start adding separate follow(request) delete db functions, add specific visibility cache tests

* add relationship type caching

* fix getting local account follows / followed-bys, other small codebase improvements

* simplify invalidation using cache hooks, add more GetAccountBy___() functions

* fix boosting to return 404 if not boostable but no error (to not leak status ID)

* remove dead code

* improved placement of cache invalidation

* update license headers

* add example follow, follow-request config entries

* add example visibility cache configuration to config file

* use specific PutFollowRequest() instead of just Put()

* add tests for all GetAccountBy()

* add GetBlockBy() tests

* update block to check primitive fields

* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests

* fix copy-pasted code

* update envparsing test

* whitespace

* fix bun struct tag

* add license header to gtscontext

* fix old license header

* improved error creation to not use fmt.Errorf() when not needed

* fix various rebase conflicts, fix account test

* remove commented-out code, fix-up mention caching

* fix mention select bun statement

* ensure mention target account populated, pass in context to customrenderer logging

* remove more uncommented code, fix typeutil test

* add statusfave database model caching

* add status fave cache configuration

* add status fave cache example config

* woops, catch missed error. nice catch linter!

* add back testrig panic on nil db

* update example configuration to match defaults, slight tweak to cache configuration defaults

* update envparsing test with new defaults

* fetch followingget to use the follow target account

* use accounnt.IsLocal() instead of empty domain check

* use constants for the cache visibility type check

* use bun.In() for notification type restriction in db query

* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())

* use bun query building for nested select statements to ensure working with postgres

* update public timeline future status checks to match visibility filter

* same as previous, for home timeline

* update public timeline tests to dynamically check for appropriate statuses

* migrate accounts to allow unique constraint on public_key

* provide minimal account with publicKey

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-28 14:03:14 +01:00
tobi 7db81cde44
[feature] Email notifications for new / closed moderation reports (#1628)
* start fiddling about with email sending to allow multiple recipients

* do some fiddling

* notifs working

* notify on closed report

* finishing up

* envparsing

* use strings.ContainsAny
2023-03-19 13:11:46 +01:00
Daenney e397272fe8
[feature] Discover webfinger through host-meta (#1588)
* [feature] Discover webfinger through host-meta

This implements a fallback for discovering the webfinger endpoint in
case the /.well-known/webfinger endpoint wasn't properly redirected.
Some instances do this because the recommendation used to be to use
host-meta for the webfinger redirect in the before times.

Closes #1558.

* [bug] Ensure we only ever update cache on success

* [chore] Move finger tests to their own place

This adds a test suite for transport and moves the finger cache tests
into there instead of abusing the search test suite.

* [chore] cleanup the test a bit more

We don't really need a separate function for the oddly located webfinger
response as we check the full URL string anyway

* Address review comments

* [chore] update config example

* [chore] access DB only through state in controller
2023-03-08 12:57:41 +00:00
Daenney 65aeeb4442
[chore] Print human readable config (#1589)
This changes the config action to print the config in a more human
readable format, indented by 4 spaces and with newlines.

Thanks to this, we can now reasonably construct some JSON in the
envparsing shell script, without needing utilities like jd. It does
assume cat is available in order to not change the shebang to bash.

With the expected JSON now being one key per line it should make it much
easier for multiple PRs that change something around config not to
collide in merge conflicts.
2023-03-04 20:56:50 +00:00