Commit Graph

443 Commits

Author SHA1 Message Date
kim 21bb324156
[chore] media and emoji refactoring (#3000)
* start updating media manager interface ready for storing attachments / emoji right away

* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()

* remove now unused media workers

* fix tests and issues

* fix another test!

* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues

* fix more tests

* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis

* whoops, rebase issue

* remove kim's whacky experiments

* do some reshuffling, ensure emoji uri gets set

* ensure marked as not cached on cleanup

* tweaks to media / emoji processing to handle context canceled better

* ensure newly fetched emojis actually get set in returned slice

* use different varnames to be a bit more obvious

* move emoji refresh rate limiting to dereferencer

* add exported dereferencer functions for remote media, use these for recaching in processor

* add check for nil attachment in updateAttachment()

* remove unused emoji and media fields + columns

* see previous commit

* fix old migrations expecting image_updated_at to exists (from copies of old models)

* remove freshness checking code (seems to be broken...)

* fix error arg causing nil ptr exception

* finish documentating functions with comments, slight tweaks to media / emoji deref error logic

* remove some extra unneeded boolean checking

* finish writing documentation (code comments) for exported media manager methods

* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot

* move doesColumnExist() to util.go in migrations package
2024-06-26 16:01:16 +01:00
tobi 4ce5c37df5
[bugfix] rename `include_types[]` to `types[]` (#3023) 2024-06-18 18:18:35 +02:00
tobi d2b3d37724
[feature/frontend] Reports frontend v2 (#3022)
* use apiutil + paging in admin processor+handlers

* we're making it happen

* fix little whoopsie

* styling for report list

* don't youuuu forget about meee don't don't don't don't

* last bits

* sanitize content before showing in report statuses

* update report docs
2024-06-18 18:18:00 +02:00
Vyr Cossont b08c1bd0cb
[feature] Implement types[] param for notifications (#3009)
Counterpart of exclude_types[].

Also updates Swagger spec for types[] to use the correct param name and enumerate possible values.

Fixes #3003
2024-06-17 20:50:50 +01:00
Vyr Cossont b789fe2bc7
[feature] filter API v2: Restore keywords_attributes and statuses_attributes (#2995)
These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
2024-06-14 10:11:41 +02:00
Vyr Cossont dcf825bd87
[bugfix] Correct Swagger path for poll voting (#2996)
Updates the Swagger spec to match the router.

Fixes #2994
2024-06-12 11:16:46 +02:00
tobi 611f9de39b
[bugfix] Deref stats async, serve stub collections if handshaking (#2990)
* [bugfix] Deref stats async, allow peek if handshaking

* don't return totalItems when handshaking or hiding collections

* use GetLimit()

* use StubAccountStats
2024-06-11 11:54:59 +02:00
tobi ce3b8aacf7
[chore] Warn about email/password change when using OIDC (#2975)
* [chore] Warn about email/password change when using OIDC

* go fmt
2024-06-07 15:21:57 +01:00
Vyr Cossont afcfa48a7d
[feature] Implement filters_changed stream event (#2972) 2024-06-07 10:51:13 +02:00
Vyr Cossont e605788b4b
[bugfix] Filter fixes (#2971)
* Add Filter.Expired convenience method, consistent with mutes

* Add missing Swagger for filter keyword/status delete and for 403s on moved accounts
2024-06-06 20:16:20 +02:00
Vyr Cossont 5e2d4fdb19
[feature] User muting (#2960)
* User muting

* Address review feedback

* Rename uniqueness constraint on user_mutes to match convention

* Remove unused account_id from where clause

* Add UserMute to NewTestDB

* Update test/envparsing.sh with new and fixed cache stuff

* Address tobi's review comments

* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired

* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 16:38:02 +00:00
tobi bcda048eab
[feature] Self-serve email change for users (#2957)
* [feature] Email change

* frontend stuff for changing email

* docs

* tests etc

* differentiate more clearly between local user+account and account

* populate user
2024-06-06 14:43:25 +01:00
tobi 6f26b32ec3
[bugfix] Don't nil emojis + fields on blocked accounts (#2968)
* [bugfix] Don't nil emojis + fields on blocked accounts

* comment

* swagger
2024-06-06 11:22:16 +01:00
tobi d79c2f2688
[feature/frontend] Add debug sections to settings panel (#2950)
* [feature/frontend] Add debug sections to settings panel

* max-width

* swagger
2024-06-03 11:20:53 +02:00
Vyr Cossont 04bcde08a1
[feature] Add from: search operator and account_id query param (#2943)
* Add from: search operator

* Fix whitespace in Swagger YAML comment

* Move query parsing into its own method

* Document search

* Clarify post search scope
2024-05-31 12:57:42 +02:00
Vyr Cossont 61a8d36255
[feature] Implement Filter API v2 (#2936)
* Use correct entity name

* We support server-side filters now

* Document filter v1 methods that can throw a 409

* Validate v1 filter phrase as filter title

* Always check v1 filter API status codes in tests

* Document keyword minimum requirement on filter API v1

* Make it possible to specify filter keyword update columns per filter keyword

* Implement v2 filter API

* Fix lint and tests

* Update Swagger spec

* Fix filter update test

* Update Swagger spec *correctly*

* Update actual files Swagger spec was generated from

* Remove keywords_attributes and statuses_attributes

* Add test for serialization of empty filter

* More helpful messages when object is owned by wrong account
2024-05-31 12:55:56 +02:00
Vyr Cossont 975e92b7f1
[feature] Implement profile API (#2926)
* Implement profile API

This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header.

See: https://docs.joinmastodon.org/methods/profile/

* Move profile media methods to media processor

* Remove check for moved account
2024-05-29 12:57:44 +02:00
tobi f9a4a6120d
[feature] Debug admin endpoint to clear caches (#2940)
* [feature] Debug admin endpoint to clear caches

* go fmt
2024-05-29 12:56:17 +02:00
tobi a276b1ca06
[feature/frontend] Let admins send test email to validate SMTP config (#2934)
* [feature/frontend] Let admins send test email to validate SMTP config

* wee
2024-05-27 17:03:54 +00:00
kim 1e7b32490d
[experiment] add alternative wasm sqlite3 implementation available via build-tag (#2863)
This allows for building GoToSocial with [SQLite transpiled to WASM](https://github.com/ncruces/go-sqlite3) and accessed through [Wazero](https://wazero.io/).
2024-05-27 17:46:15 +02:00
kim 3d3e99ae52
[performance] update storage backend and make use of seek syscall when available (#2924)
* update to use go-storage/ instead of go-store/v2/storage/

* pull in latest version from codeberg

* remove test output 😇

* add code comments

* set the exclusive bit when creating new files in disk config

* bump to actual release version

* bump to v0.1.1 (tis a simple no-logic change)

* update readme

* only use a temporary read seeker when decoding video if required (should only be S3 now)

* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()

* update to use seek call in serveFileRange()
2024-05-22 11:46:24 +02:00
Vyr Cossont 45f4afe60e
feature: filters v2 server-side warning/hiding (#2793)
* Remove dead code

* Filter statuses when converting to frontend representation

* status.filtered is an array

* Make matching case-insensitive

* Remove TODOs that don't need to be done now

* Add missing filter check for notification

* lint: rename ErrHideStatus

* APIFilterActionToFilterAction not used yet

* swaggerino docseroni

* Address review comments

* Add apimodel.FilterActionNone

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-05-06 12:49:08 +01:00
tobi 6171dcbe51
[feature] Add HTTP header permission section to frontend (#2893)
* [feature] Add HTTP header filter section to frontend

* tweak naming a bit
2024-05-05 11:47:22 +00:00
tobi 725a21b027
[feature] Page through accounts as moderator (#2881)
* [feature] Page through accounts as moderator

* aaaaa

* use COLLATE "C" for Postgres to ensure same ordering as SQLite

* fix typo, test paging up

* don't show moderation / info for our instance acct
2024-05-01 14:11:22 +01:00
kim a8254a40e7
[bugfix] further paging mishaps (#2884)
* FURTHER paging shenanigans 🥲

* remove cursor logic from ToLinkURL()

* fix up paging tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-04-30 16:22:23 +02:00
kim 4f87ef246c
[bugfix] paging rel links (#2883)
* fix paging so it uses correct cursor query parameter name

* improved code comment

* whoops, flip the cursoring 🤦

* fix the broken test
2024-04-30 12:19:33 +02:00
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
tobi b7c629a18a
[bugfix] Fix incorrect field name for status source, add helpful message (#2854)
* [bugfix] Fix incorrect field name for status source, add helpful message

* swagger

* yyammm
2024-04-18 13:22:55 +02:00
tobi 431505b3e4
[feature] Stub conversations endpoint (#2853) 2024-04-18 12:36:02 +02:00
tobi 34d0159ad5
[feature] Stub account mutes endpoint (#2852)
* [feature] Stub account mutes endpoint

* swagger? i barely know 'er!
2024-04-18 09:59:47 +01:00
tobi cef9924d9a
[feature] Status source endpoint (#2848)
* [feature] statusSource endpoint

* finish up
2024-04-17 13:49:20 +01:00
tobi ef16919d4a
[feature] Stub status history endpoint (#2847) 2024-04-17 13:06:49 +01:00
tobi 8cf685fbe9
[bugfix] Fix minor API issue w/ boosted statuses (#2846) 2024-04-17 12:41:40 +02:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types (#2838)
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00
tobi 89e0cfd874
[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)
* update settings panels, add pending overview + approve/deny functions

* add admin accounts get, approve, reject

* send approved/rejected emails

* use signup URL

* docs!

* email

* swagger

* web linting

* fix email tests

* wee lil fixerinos

* use new paging logic for GetAccounts() series of admin endpoints, small changes to query building

* shuffle useAccountIDIn check *before* adding to query

* fix parse from toot react error

* use `netip.Addr`

* put valid slices in globals

* optimistic updates for account state

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-04-13 13:25:10 +02:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
kim a483bd9e38
[performance] massively improved ActivityPub delivery worker efficiency (#2812)
* add delivery worker type that pulls from queue to httpclient package

* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller

* hook up queue deletion logic

* support deleting queued http requests by target ID

* don't index APRequest by hostname in the queue

* use gorun

* use the original context's values when wrapping msg type as delivery{}

* actually log in the AP delivery worker ...

* add uncommitted changes

* use errors.AsV2()

* use errorsv2.AsV2()

* finish adding some code comments, add bad host handling to delivery workers

* slightly tweak deliveryworkerpool API, use advanced sender multiplier

* remove PopCtx() method, let others instead rely on Wait()

* shuffle things around to move delivery stuff into transport/ subpkg

* remove dead code

* formatting

* validate request before queueing for delivery

* finish adding code comments, fix up backoff code

* finish adding more code comments

* clamp minimum no. senders to 1

* add start/stop logging to delivery worker, some slight changes

* remove double logging

* use worker ptrs

* expose the embedded log fields in httpclient.Request{}

* ensure request context values are preserved when updating ctx

* add delivery worker tests

* fix linter issues

* ensure delivery worker gets inited in testrig

* fix tests to delivering messages to check worker delivery queue

* update error type to use ptr instead of value receiver

* fix test calling Workers{}.Start() instead of testrig.StartWorkers()

* update docs for advanced-sender-multiplier

* update to the latest activity library version

* add comment about not using httptest.Server{}
2024-04-11 11:45:35 +02:00
kim 83e7847cdf
fix possible nil panic (#2809) 2024-04-04 16:45:59 +01:00
kim 15ede4c1ea
[bugfix] improved authenticate post inbox error handling (#2803)
* improved PostInboxScheme() error handling / logging in case of failed auth

* dumbass kim. returning err instead of errWithCode...

* add checks for the slightly changed error handling in tests, add notes in codebase about the odd way of working
2024-04-03 13:57:07 +01:00
tobi f05874be30
[feature] Option to hide followers/following (#2788) 2024-04-02 10:42:24 +01:00
tobi 8953f57d88
[feature] User-selectable preset CSS themes for accounts (#2777)
* [feature] User-selectable preset themes

* docs, more theme stuff

* lint, tests

* fix css name

* correct some little issues

* add another theme

* fix poll background

* okay last theme i swear

* make retrieval of apimodel themes more conventional

* preallocate stylesheet slices
2024-03-25 17:32:24 +00:00
tobi b7b42e832a
[feature] Add healthcheck endpoints `/livez` and `/readyz` (#2783)
* [feature] Add healthcheck endpoints `/livez` and `/readyz`

* use select that returns no data
2024-03-25 17:05:14 +00:00
tobi 36f79e650c
[bugfix] Avoid empty public/local timeline queries (#2784) 2024-03-25 16:20:45 +00:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
Vyr Cossont 4a56f31047
[chore] Swagger: add missing paging params to bookmarks list (#2759)
No code changes.
2024-03-15 11:43:24 +01:00
Vyr Cossont efbc8151db
[chore] Swagger: correct names and formats for several array params (#2758)
No code changes.
2024-03-15 11:31:58 +01:00
tobi 92bf1f779b
[chore] Expose move endpoint again, small settings panel fixes (#2752) 2024-03-13 14:45:50 +01:00
tobi ab2d063fcb
[feature] Process outgoing Move from clientAPI (#2750)
* prevent moved accounts from taking create-type actions

* update move logic

* federate move out

* indicate on web profile when an account has moved

* [docs] Add migration docs section

* lock while checking + setting move state

* use redirectFollowers func for clientAPI as well

* comment typo

* linter? i barely know 'er!

* Update internal/uris/uri.go

Co-authored-by: Daenney <daenney@users.noreply.github.com>

* add a couple tests for move

* fix little mistake exposed by tests (thanks tests)

* ensure Move marked as successful

* attach shared util funcs to struct

* lock whole account when doing move

* move moving check to after error check

* replace repeated text with error func

* linterrrrrr!!!!

* catch self follow case

---------

Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-13 13:53:29 +01:00
Vyr Cossont 13b9fd5f92
[bugfix]: Add missing Link headers in Swagger spec (#2751)
* Adds Link headers to all endpoints that use it for paging
* Fixes the return type for listing what accounts someone follows
2024-03-13 12:23:26 +01:00