Commit Graph

80 Commits

Author SHA1 Message Date
Daenney dcab555a6b
[chore] Update robots.txt (#2856)
This updates the robots.txt based on the list of the ai.robots.txt
repository. We can look at automating that at some point.

It's worth pointing out that some robots, namely the ones by Bytedance,
are known to ignore robots.txt entirely.
2024-04-22 11:01:37 +02:00
tobi 89e0cfd874
[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)
* update settings panels, add pending overview + approve/deny functions

* add admin accounts get, approve, reject

* send approved/rejected emails

* use signup URL

* docs!

* email

* swagger

* web linting

* fix email tests

* wee lil fixerinos

* use new paging logic for GetAccounts() series of admin endpoints, small changes to query building

* shuffle useAccountIDIn check *before* adding to query

* fix parse from toot react error

* use `netip.Addr`

* put valid slices in globals

* optimistic updates for account state

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-04-13 13:25:10 +02:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
tobi 8953f57d88
[feature] User-selectable preset CSS themes for accounts (#2777)
* [feature] User-selectable preset themes

* docs, more theme stuff

* lint, tests

* fix css name

* correct some little issues

* add another theme

* fix poll background

* okay last theme i swear

* make retrieval of apimodel themes more conventional

* preallocate stylesheet slices
2024-03-25 17:32:24 +00:00
Daenney 6528592dd2
[feature] Block Amazonbot (#2692)
Blocks the Amazon crawler bot.

Closes: #2686
2024-02-27 13:25:08 +00:00
tobi 5aca09b59e
[chore] Rename frontend.tmpl to settings.tmpl, remove unused "lightgray" class (#2674) 2024-02-21 16:11:11 +01:00
tobi 0ff52b71f2
[chore] Refactor HTML templates and CSS (#2480)
* [chore] Refactor HTML templates and CSS

* eslint

* ignore "Local"

* rss tests

* fiddle with OG just a tiny bit

* dick around with polls a bit more so SR stops saying "clickable"

* remove break

* oh lord

* don't lazy load avatar

* fix ogmeta tests

* clean up some cruft

* catch remaining calls to c.HTML

* fix error rendering + stack overflow in tag

* allow templating attributes

* fix indent

* set aria-hidden on status complementary content, since it's already present in the label anyway

* tidy up templating calls a little

* try to make styling a bit more consistent + readable

* fix up some remaining CSS issues

* fix up reports
2023-12-27 11:23:52 +01:00
tobi 0e2c342191
[bugfix/chore] `Announce` reliability updates (#2405)
* [bugfix/chore] `Announce` updates

* test update

* fix tests

* TestParseAnnounce

* update comments

* don't lock/unlock, change function signature

* naming stuff

* don't check domain block twice

* UnwrapIfBoost

* beep boop
2023-12-01 15:27:15 +01:00
kim 74700cc803
[performance] http response encoding / writing improvements (#2374) 2023-11-27 14:00:57 +00:00
tobi 2b9cf56f56
[chore/bugfix] Fix double gzip on prometheus endpoint (#2383)
* [chore] Move "/metrics" into separate API module

* use our own gzip middleware for prom
2023-11-23 19:10:51 +01:00
Tsuribori 1ba3e14b36
[feature] Initial Prometheus metrics implementation (#2334)
* feat: Initial OTEL metrics

* docs: add metrics documentation

* fix: metrics endpoint conditional check

* feat: metrics endpoint basic auth

* fix: make metrics-auth-enabled default false

* fix: go fmt helpers.gen.go

* fix: add metric-related env vars to envparsing.sh

* fix: metrics docs

* fix: metrics related stuff in envparsing.sh

* fix: metrics docs

* chore: metrics docs wording

* fix: metrics stuff in envparsing?

* bump otel versions

---------

Co-authored-by: Tsuribori <user@acertaindebian>
Co-authored-by: Tsuribori <none@example.org>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-11-20 16:43:55 +01:00
tobi fc02d3c6f7
[feature] Set/show instance language(s); show post language on frontend (#2362)
* update go text, include text/display

* [feature] Set instance langs, show post lang on frontend

* go fmt

* WebGet

* set language for whole article, don't use FA icon

* mention instance languages + other optional config vars

* little tweak

* put languages in config properly

* warn log language parse

* change some naming around

* tidy up validate a bit

* lint

* rename LanguageTmpl in template
2023-11-17 11:35:28 +01:00
tobi 8d0c017cf2
[feature/performance] Wrap incoming HTTP requests in timeout handler (#2353)
* deinterface router, start messing about with deadlines

* weeeee

* thanks linter (thinter)

* write Connection: close when timing out requests

* update wording

* don't replace req

* don't bother with fancy Cause functions (I'll use them one day...)
2023-11-13 19:48:51 +01:00
tobi ba9d6b467a
[feature] Media attachment placeholders (#2331)
* [feature] Use placeholders for unknown media types

* fix read of underreported small files

* switch to reduce nesting

* simplify cleanup
2023-11-10 19:29:26 +01:00
Daenney 0cce2c0838
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers

* [feature] Block the other OpenAI crawler

* [feature] Block Common Crawl crawler

This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.

Fixes: #2240

* [feature] Block Omgilikebot

Used by some shady big web data engine company.

* [feature] Block Meta's language model crawler

* [feature] Block well-known.dev crawler
2023-09-30 20:44:57 +01:00
tobi dc96562b40
[bugfix] Use custom bluemonday policy to disallow inline img tags (#2100) 2023-08-11 14:40:11 +02:00
tobi 4b05dcde43
[chore] Update robots.txt, give chatgpt the middle finger (#2085) 2023-08-08 13:16:34 +02:00
kim 00adf18c24
[feature] simpler cache size configuration (#2051)
* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-03 11:34:35 +02:00
tobi 2796a2e82f
[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)
* update go-fed

* do the things

* remove unused columns from tags

* update to latest lingo from main

* further tag shenanigans

* serve stub page at tag endpoint

* we did it lads

* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)

* swagger docs

* document hashtag usage + federation

* instanceGet

* don't bother parsing tag href

* rename whereStartsWith -> whereStartsLike

* remove GetOrCreateTag

* dont cache status tag timelineability
2023-07-31 15:47:35 +02:00
kim 5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
tobi 12b6cdcd8c
[bugfix] Set Vary header correctly on cache-control (#1988)
* [bugfix] Set Vary header correctly on cache-control

* Prefer activitypub types on AP endpoints

* use immutable on file server, vary by range

* vary auth on Accept
2023-07-13 21:27:25 +02:00
tobi ca5492b65f
[bugfix] Tidy up rss feed serving; don't error on empty feed (#1970)
* [bugfix] Tidy up rss feed serving; don't error on empty feed

* fall back to account creation time as rss feed update time

* return feed early when account has no eligible statuses
2023-07-10 17:05:59 +02:00
tobi ac564c1862
[bugfix] Reorder web view logic, other small fixes (#1954) 2023-07-07 14:58:53 +02:00
tobi 24fbdf2b0a
[chore] Refactor AP authentication, other small bits of tidying up (#1874) 2023-06-13 15:47:56 +01:00
Daenney e91cabb704
[bugfix] Fix NegotiateFormat with multiple accept headers (#1797)
* [bugfix] Fix NegotiateAccept with multi accept

There's a bug in Gin's NegotiateFormat that doesn't handle the presence
of multilpe accept headers. This lifts the code from the PR @tsmethurst
sent a year ago to Gin into our codebase to fix the issue.

* [bugfix] Concat accept header in webfinger

Some implementations bug out when there's multiple accept headers,
including Gin (see 7050112af1). But things
seem to work reliably with a single accept header with multiple parts.

Fixes: #1793
2023-05-16 15:08:45 +02:00
Natsu Kagami ba5a464ca5
[chore] Prefer JSON errors in API endpoints (#1766)
* Default to JSON over HTML for error handling

* Change the default error display for web endpoints to html
2023-05-12 10:16:41 +02:00
f0x52 9cc9ffc5a7
[frontend] Profiles with fields & more (#1764)
* redesign status template

* separate index page styling

* redesign profile template

* fix header styling/wrapping

* remove old spoiler js

* fix status cw button wrapping

* fix status info variables

* profile responsiveness, accessibility tweaks

* fix variable use, mobile

* remove duplicate id's

* rss icon, fix indent

* fix toot border-radius

* fix toot spacing

* emojify and html profile fields

* refactor (sensitive) media rendering

* plaintext profile fields

* bundle plyr icon svg

* only pause video when switching photoswipe slides

* yarn upgrade

* profile fields formatting

* replace uglifyify with @browserify updated fork

* fix profile field templating (yet again)

* fix React classes

* testrig: add testing profile field for admin user

* fix sensitive media interactions

* Revert "testrig: add testing profile field for admin user"

This reverts commit 80490c183e.

* settings interface wrapping

* fix reported toot styling

* add role to profile sr-only text

* comment fallback rule

* remove currently unused image description lacking indicator
2023-05-11 17:46:32 +02:00
tobi a6ec2a5bc2
[bugfix] Fix invalid og:description on account w/ empty note (#1733) 2023-05-03 16:18:34 +02:00
Daenney 5e2bf0bdca
[chore] Improve copyright header handling (#1608)
* [chore] Remove years from all license headers

Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.

In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.

* [chore] Add license header check

This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.

* [chore] Add missing license headers

* [chore] Further updates to license header

* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
2023-03-12 16:00:57 +01:00
tobi a772d4d98e
[chore] Fix opengraph properties (#1611) 2023-03-12 14:14:33 +01:00
tobi 24cec4e7aa
[feature] Federate pinned posts (aka `featuredCollection`) in and out (#1560)
* start fiddling

* the ol' fiddle + update

* start working on fetching statuses

* poopy doopy doo where r u uwu

* further adventures in featuring statuses

* finishing up

* fmt

* simply status unpin loop

* move empty featured check back to caller function

* remove unnecessary log.WithContext calls

* remove unnecessary IsIRI() checks

* add explanatory comment about status URIs

* change log level to error

* better test names
2023-03-01 17:52:44 +00:00
tobi c27b4d7ed0
[feature] Client API endpoints + v. basic web view for pinned posts (#1547)
* implement status pin client api + web handler

* make test names + comments more descriptive

* don't use separate table for status pins

* remove unused add + remove checking

* tidy up + add some more tests
2023-02-25 12:16:30 +00:00
Daenney 074f352709
[chore] improve opengraph descripiton tag (#1550)
This changes parseDescription to properly encode things to be safe for
usage without removing things like backslashes that may be relevant.

* text.SanitizePlaintext already calls html.UnescapeString so we don't
  have to do that
* Replace \n with space early
* Remove duplicate white-space by splitting on fields and joining
* HTML-escape the string we have
* For extra certainty, encode the backslash as &bsol;

Fixes #1549
2023-02-22 21:36:18 +00:00
tobi b6fbdc66c1
[chore] Deinterface processor and subprocessors (#1501)
* [chore] Deinterface processor and subprocessors

* expose subprocessors via function calls

* missing license header
2023-02-22 16:05:26 +01:00
f0x52 b6143c9ab8
[feature] About page (#1495)
* about page basics

* more info, styling

* update emoji sizing on about page contact card
2023-02-20 16:29:29 +01:00
Daenney 68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi 4e4da19720
[bugfix] Use SignatureCheck middleware for web profile endpoints too (#1451) 2023-02-07 14:57:09 +01:00
tobi 382512a5a6
[feature] Implement `/api/v2/instance` endpoint (#1409)
* interim: start adding /api/v2/instance

* finish up
2023-02-02 14:08:13 +01:00
f0x52 17eecfb6d9
[feature] Public list of suspended domains (#1362)
* basic rendered domain blocklist (unauthenticated!)

* style basic domain block list

* better formatting for domain blocklist

* add opt-in config option for showing suspended domains

* format/linter

* re-use InstancePeersGet for web-accessible domain blocklist

* reword explanation, border styling

* always attach blocklist handler, update error message

* domain blocklist error message grammar
2023-01-25 18:06:41 +01:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
kim 71dfea7e47
[chore] shuffle middleware to split rate limitting into client/s2s/fileserver, share gzip middleware globally (#1290)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-03 11:50:59 +01:00
tobi 941893a774
[chore] The Big Middleware and API Refactor (tm) (#1250)
* interim commit: start refactoring middlewares into package under router

* another interim commit, this is becoming a big job

* another fucking massive interim commit

* refactor bookmarks to new style

* ambassador, wiz zeze commits you are spoiling uz

* she compiles, we're getting there

* we're just normal men; we're just innocent men

* apiutil

* whoopsie

* i'm glad noone reads commit msgs haha :blob_sweat:

* use that weirdo go-bytesize library for maxMultipartMemory

* fix media module paths
2023-01-02 12:10:50 +00:00
tobi 1d24c1c283
[bugfix] Use null for empty api status `language` (#1268)
* [bugfix] Use null for empty api status `language`

* update swagger docs
2022-12-15 09:41:49 +01:00
tobi 1281a9fd4a
[chore/frontend] remove references to unused bundle.js (#1128) 2022-11-23 15:32:57 +01:00
kim 8598dea98b
[chore] update database caching library (#1040)
* convert most of the caches to use result.Cache{}

* add caching of emojis

* fix issues causing failing tests

* update go-cache/v2 instances with v3

* fix getnotification

* add a note about the left-in StatusCreate comment

* update EmojiCategory db access to use new result.Cache{}

* fix possible panic in getstatusparents

* further proof that kim is not stinky
2022-11-15 18:45:15 +00:00
Adam Harvey ee6a0a1e15
[bugfix] use the landing page user directly (#956)
If set, the landing page user configuration value is used as a Gin
context parameter, which seems incorrect, since a normal request isn't
going to have a parameter named after an arbitrarily configured user.
Instead, the user name should be used directly when building the
redirect URL.
2022-11-05 08:39:36 +01:00
Mia Heidenstedt f3fc040c2e
[feature] Allow user to show instead of landing page on / (#922)
* configurable user instead of landing page

* rename DefaultUser to LandingPageUser

* code review changes and fix tests

* try to fix pipeline error

* code review changes

* code review changes

* code review changes

* remove unnecessary line
2022-11-03 13:15:31 +01:00
f0x52 c4c713988a
[frontend] change bundler to skulk (#942)
* replace web bundler with skulk

* upgrade skulk

* add license
2022-11-02 16:31:43 +01:00
tobi 80663061d8
[feature] Add opt-in RSS feed for account's latest Public posts (#897)
* start adding rss functionality

* add gorilla/feeds dependency

* first bash at building rss feed
still needs work, this is an interim commit

* tidy up a bit

* add publicOnly option to GetAccountLastPosted

* implement rss endpoint

* fix test

* add initial user docs for rss

* update rss logo

* docs update

* add rssFeed to frontend

* feed -> feed.rss

* enableRSS

* increase rss logo size a lil bit

* add rss toggle

* move emojify to text package

* fiddle with rss feed formatting

* add Text field to test statuses

* move status to rss item to typeconverter

* update bun schema for enablerss

* simplify 304 checking

* assume account not rss

* update tests

* update swagger docs

* allow more characters in title, trim nicer

* update last posted to be more consistent
2022-10-08 14:00:39 +02:00
f0x52 5249294a16
[chore] Bundler restructure (#880)
* re-structure bundler, settings panel files

* add more info logging

* tidy up CSS syntax errors

* split into lib/ files

* livereloading server

* fix factor function for production builds

* remove testing console.log

* default to production env, saves 300kb bundle size
2022-10-03 16:46:38 +02:00