Commit Graph

15 Commits

Author SHA1 Message Date
tobi 1cdc163276
[performance] Don't retry/backoff invalid http requests that will never succeed (#609)
* add httpguts (ew)

* add ValidateRequest err wrapping logic

* don't retry on unrecoverable errors

* i am very clever
2022-05-26 13:38:41 +02:00
kim 223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
tobi e63b653199
[performance] Add dereference shortcuts to avoid making http calls to self (#430)
* update transport (controller) to allow shortcuts

* go fmt

* expose underlying sig transport to allow test sigs
2022-03-15 15:01:19 +01:00
tsmethurst c157b1b20b rework data function to provide filesize 2022-01-23 14:41:58 +01:00
tsmethurst 589bb9df02 pass reader around instead of []byte 2022-01-16 18:52:55 +01:00
tsmethurst 113f9d9ab4 pass a function into the manager, start work on emoji 2022-01-11 17:49:14 +01:00
tsmethurst f61c3ddcf7 compiling now 2022-01-08 17:17:01 +01:00
tobi ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi 09ef9e639e
move to ssb gofed fork (#298) 2021-11-13 17:29:43 +01:00
R. Aidan Campbell 083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi 2dc9fc1626
Pg to bun (#148)
* start moving to bun

* changing more stuff

* more

* and yet more

* tests passing

* seems stable now

* more big changes

* small fix

* little fixes
2021-08-25 15:34:33 +02:00
Tobi Smethurst 87cf621e21
Remote instance dereferencing (#70)
Remote instances are now dereferenced when they post to an inbox on a GtS instance.

    Dereferencing will be done first by checking the /api/v1/instance endpoint of an instance.
    If that doesn't work, /.well-known/nodeinfo will be checked.
    If that doesn't work, only a minimal representation of the instance will be stored.

A new field was added to the Instance database model. To create it:

alter table instances add column contact_account_username text;
2021-06-27 16:52:18 +02:00
tsmethurst 0fe853b1ee first implementation of search feature 2021-05-29 19:36:54 +02:00
Tobi Smethurst d839f27c30
Follows and relationships (#27)
* Follows -- create and undo, both remote and local
* Statuses -- federate new posts, including media, attachments, CWs and image descriptions.
2021-05-21 15:48:26 +02:00
Tobi Smethurst 6cd033449f
Refine statuses (#26)
Remote media is now dereferenced and attached properly to incoming federated statuses.
    Mentions are now dereferenced and attached properly to incoming federated statuses.
    Small fixes to status visibility.
    Allow URL params for filtering statuses:

	// ExcludeRepliesKey is for specifying whether to exclude replies in a list of returned statuses by an account.
      	// PinnedKey is for specifying whether to include pinned statuses in a list of returned statuses by an account.
      	// MaxIDKey is for specifying the maximum ID of the status to retrieve.
      	// MediaOnlyKey is for specifying that only statuses with media should be returned in a list of returned statuses by an account.

    Add endpoint for fetching an account's statuses.
2021-05-17 19:06:58 +02:00