Commit Graph

228 Commits

Author SHA1 Message Date
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
kim c67bbe5ba0
update to set requesting account when deleting status (#2849) 2024-04-17 16:54:40 +01:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim a483bd9e38
[performance] massively improved ActivityPub delivery worker efficiency (#2812)
* add delivery worker type that pulls from queue to httpclient package

* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller

* hook up queue deletion logic

* support deleting queued http requests by target ID

* don't index APRequest by hostname in the queue

* use gorun

* use the original context's values when wrapping msg type as delivery{}

* actually log in the AP delivery worker ...

* add uncommitted changes

* use errors.AsV2()

* use errorsv2.AsV2()

* finish adding some code comments, add bad host handling to delivery workers

* slightly tweak deliveryworkerpool API, use advanced sender multiplier

* remove PopCtx() method, let others instead rely on Wait()

* shuffle things around to move delivery stuff into transport/ subpkg

* remove dead code

* formatting

* validate request before queueing for delivery

* finish adding code comments, fix up backoff code

* finish adding more code comments

* clamp minimum no. senders to 1

* add start/stop logging to delivery worker, some slight changes

* remove double logging

* use worker ptrs

* expose the embedded log fields in httpclient.Request{}

* ensure request context values are preserved when updating ctx

* add delivery worker tests

* fix linter issues

* ensure delivery worker gets inited in testrig

* fix tests to delivering messages to check worker delivery queue

* update error type to use ptr instead of value receiver

* fix test calling Workers{}.Start() instead of testrig.StartWorkers()

* update docs for advanced-sender-multiplier

* update to the latest activity library version

* add comment about not using httptest.Server{}
2024-04-11 11:45:35 +02:00
Daenney 8ed1b8142c
[bugfix] Sort follows chronologically (#2801)
The id on the follows table is not a ULID, but a random ID. Sorting on
them results in a completely random order. Instead, sort on created_at,
which sould result in a stable and intended sort order.

Fixes: #2769

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2024-04-03 14:06:39 +01:00
kim 15ede4c1ea
[bugfix] improved authenticate post inbox error handling (#2803)
* improved PostInboxScheme() error handling / logging in case of failed auth

* dumbass kim. returning err instead of errWithCode...

* add checks for the slightly changed error handling in tests, add notes in codebase about the odd way of working
2024-04-03 13:57:07 +01:00
tobi 4bbdef02f1
[chore] Try to parse public key as both Actor + bare key (#2710)
* [chore] Try to parse public key as both Actor + bare key

* fix weird test fail
2024-04-02 11:30:10 +01:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
kim 459e75a9db
[bugfix] only check replyTo visibility during permission checks IF status is local (#2757) 2024-03-14 17:55:35 +01:00
tobi 1bcdf1da3b
[feature] Process incoming `Move` activity (#2724)
* [feature] Process incoming account Move activity

* fix targetAcct typo

* put move origin account on fMsg

* shift more move functionality back to the worker fn

* simplify error logic
2024-03-12 14:34:08 +00:00
kim d85727e184
[bugfix] check remote status permissibility (#2703)
* add more stringent checks for remote status permissibility

* add check for inreplyto of a remote status being a boost

* do not permit inReplyTo boost wrapper statuses

* change comment wording

* fix calls to NewFederator()

* add code comments for NotPermitted() and SetNotPermitted()

* improve comment

* check that existing != nil before attempting delete

* ensure replying account isn't suspended

* use a debug log instead of info. check for boost using ID

* shorten log string length. make info level

* add note that replying to boost wrapper status shouldn't be able to happen anyways

* update to use onFail() function
2024-03-04 12:30:12 +00:00
tobi 9cadc764b3
[feature] Add experimental `instance-federation-spam-filter` option (#2685)
* [chore] Move `visibility` to `filter/visibility`

* [feature] Add experimental instance-federation-spam-filter option
2024-02-27 12:22:05 +00:00
kim 1d51e3c8d6
[bugfix] 2643 bug search for account url doesnt always work when redirected (#2673)
* update activity library so dereferencer returns full response and checks *final* link to allow for redirects

* temporarily add bodged fixed library

* remove unused code

* update getAccountFeatured() to use dereferenceCollectionPage()

* make sure to release map

* perform a 2nd decode to ensure reader is empty after primary decode

* add comment explaining choice of using Decode() instead of Unmarshal()

* update embedded activity library to latest matching https://github.com/superseriousbusiness/activity/pull/21

* add checks to look for changed URI and re-check database if redirected

* update max iteration count to 512, add checks during dereferenceAncestors() for indirect URLs

* remove doubled-up code

* fix use of status instead of current

* use URIs for checking equality for security

* use the latest known URI for boost_of_uri in case original was an indirect

* add dereferenceCollection() function for dereferenceAccountFeatured()

* pull in latest github.com/superseriousbusiness/activity version (and remove the bodge!!)

* fix typo in code comments

* update decodeType() to accept a readcloser and handle body closing

* switch to checking using BoostOfID and add note why not using BoostOfURI

* ensure InReplyTo gets unset when deleting status parent in case currently stubbed

* add tests for Collection and CollectionPage iterators
2024-02-23 16:24:40 +01:00
tobi 40f9eefc21
[bugfix] Fix dereferencing ancestors on new status create (#2652)
* [bugfix] Pass `latest` to dereferenceThread instead of barebones status

* only mark status orphaned if visibility suggests parent is really deleted

* tone down "not deref'd" warnings, since they represent a legit visibility situation

* remove FAQ entry for "status not deref'd yet"
2024-02-18 09:49:40 +00:00
tobi d3f35e8eba
[chore] refactor extractFromCtx a bit (#2646) 2024-02-17 15:20:39 +01:00
kim 2bafd7daf5
[bugfix] add stricter checks during all stages of dereferencing remote AS objects (#2639)
* add stricter checks during all stages of dereferencing remote AS objects

* a comment
2024-02-14 12:13:38 +01:00
tobi e890169e6f
use pointer for freshness window (#2614) 2024-02-09 14:24:49 +00:00
tobi 7a7746701d
[chore] Move `DoOnce` func wrapper to util (#2613) 2024-02-09 11:38:51 +00:00
tobi b6fe8e7a5b
[bugfix] Ensure activities sender always = activities actor (#2608) 2024-02-06 12:59:37 +01:00
tobi b614d33c40
[feature] Try HTTP signature validation with and without query params for incoming requests (#2591)
* [feature] Verify signatures both with + without query params

* Bump to tagged version
2024-01-31 14:15:28 +00:00
kim 0f7a2024c3
[bugfix] parent status replied to status not dereferenced sometimes (#2587)
* much simplified DereferenceStatusAncestors(), also handles edge cases now

* perform status acceptibility check before handling even as forward

* don't further dereference ancestors if they're up to date

* call enrichStatusSafely() directly to ensure we get error messages

* change getStatusByURI() semantics to return error + old model on failed update, fix deref ancestor to check for staleness before refetch

* perform a nil-check on the status.Local variable, in case it hasn't been set on new status attempting refresh

* more consistently set returned parent status, don't check if updated

* only home-timeline statuses if explicitly visible AND not explicitly invisible!

* fix broken test now that status acceptibility checks happen on forwarded statuses
2024-01-31 13:29:47 +00:00
tobi e3052e8c82
[bugfix] Don't return Account or Status if new and dereferencing failed, other small fixes (#2563)
* tidy up account, status, webfingering logic a wee bit

* go fmt

* invert published check

* alter resp initialization

* get Published from account in typeutils

* don't instantiate error for no darn good reason

* shadow err

* don't repeat error codes in wrapped errors

* don't wrap error unnecessarily
2024-01-26 14:17:10 +01:00
tobi 9d80f7fd68
[feature] Allow "charset=utf8" in incoming AP POST requests (#2564)
* [feature] Allow "charset=utf8" in incoming AP POST requests

* changed my mind

* document POSTing to a GtS inbox

* correct link
2024-01-22 14:33:01 +00:00
kim 7ec1e1332e
[performance] overhaul struct (+ result) caching library for simplicity, performance and multiple-result lookups (#2535)
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial

* use actual go-structr release version (not just commit hash)

* revert go toolchain changes (damn you go for auto changing this)

* fix go mod woes

* ensure %w is used in calls to errs.Appendf()

* fix error checking

* fix possible panic

* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop

* fix copy-paste artifact... 😇

* fix all comment copy-paste artifacts

* remove dropID() function, now we can just use slices.DeleteFunc()

* use util.Deduplicate() instead of collate(), move collate to util

* move orderByIDs() to util package and "generify"

* add a util.DeleteIf() function, use this to delete entries on failed population

* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)

* add note about how collate differs from deduplicate
2024-01-19 12:57:29 +00:00
kim 67e11a1a61
[chore] chore rationalise http return codes for activitypub handlers (#2540)
* some small code fixups and changes

* add check in ResolveIncomingActivity for transient activity types (i.e. activity ID is nil)

* update test to handle new transient behaviour
2024-01-18 17:11:13 +01:00
kim 10660e566d
[bugfix] misc dereferencer fixes (#2475)
* only perform status-up-to-date checks if no statusable has been provided

* copy over the same style of freshness checking from status deref -> accounts

* change some var names

* check for empty account domain
2024-01-09 10:42:39 +01:00
kim f4fcffc8b5
[bugfix] use a much shorter refresh limit for statuses with polls (#2453)
* specifically use a much shorter refresh limit for statuses with polls

* allow specifying whether status must be upToDate in calls to Get(Visible)?TargetStatusBy_(), limit force refresh to 5 minute cooldown

* remove the PollID check from statusUpToDate()

* remove unnecessary force flag checks

* remove unused field

* check refresh status error

* use argument name 'refresh' instead of 'upToDate' to better fit with the codebase

* add statuses_poll_id_idx

* remove the definitely-not copy-pasted comment i accidentally typed out in full

* only synchronously refresh if the refresh flag is provided, otherwise do async

* fix wrong force value being provided for async

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2023-12-15 15:24:39 +01:00
tobi 0e2c342191
[bugfix/chore] `Announce` reliability updates (#2405)
* [bugfix/chore] `Announce` updates

* test update

* fix tests

* TestParseAnnounce

* update comments

* don't lock/unlock, change function signature

* naming stuff

* don't check domain block twice

* UnwrapIfBoost

* beep boop
2023-12-01 15:27:15 +01:00
kim eb170003b8
[bugfix] return 400 Bad Request on more cases of malformed AS data (#2399) 2023-11-30 16:22:34 +00:00
kim 5fd2e427bb
[bugfix] always go through status parent dereferencing on isNew, even on data-race (#2402)
* no need to deref status author account, will already be deref'd during previous getStatusByAP{IRI,Model}()

* don't unset the isNew flag on dereference data race

* improved code comment
2023-11-30 12:32:45 +01:00
kim 42d8011ff4
[chore/security] refactor AuthenticateFederatedRequest() to handle account deref + suspension checks (#2371)
* refactor AuthenticateFederatedRequest() to handle account suspension + fetching of owner

* small fixups

* small changes

* revert to 'IsEitherBlocked' instead of just 'IsBlocked" :grimace:

* update code comment to indicate that AuthenticateFederatedRequest() will handle account + instance dereferencing
2023-11-21 11:35:30 +01:00
tobi 4ee436e98a
[bugfix] process account delete side effects in serial, not in parallel (#2360)
* [bugfix] process account delete side effects in serial, not in parallel

* StartWorkers / StartNoopWorkers for tests

* undo testrig trace logging

* log errors instead of immediately returning
2023-11-14 15:57:25 +01:00
kim deaea100c3
[bugfix] support endless polls, and misskey's' method of inferring expiry in closed polls (#2349) 2023-11-11 10:15:04 +00:00
tobi ba9d6b467a
[feature] Media attachment placeholders (#2331)
* [feature] Use placeholders for unknown media types

* fix read of underreported small files

* switch to reduce nesting

* simplify cleanup
2023-11-10 19:29:26 +01:00
tobi 7ce3a1e6f3
[bugfix] Don't try to update suspended accounts (#2348)
* [bugfix] Don't try to update suspended accounts

* bail early if requesting account suspended
2023-11-10 17:16:58 +01:00
kim e9e5dc5a40
[feature] add support for polls + receiving federated status edits (#2330) 2023-11-08 14:32:17 +00:00
kim 41435a6c4e
[feature] support canceling scheduled tasks, some federation API performance improvements (#2329) 2023-11-04 20:21:20 +00:00
kim ce71a5a790
[feature] add per-uri dereferencer locks (#2291) 2023-10-31 11:12:22 +00:00
tobi 51d0a0bba5
[bugfix] Relax `Mention` parsing, allowing either href or name (#2320) 2023-10-31 11:05:17 +00:00
tobi c7b6cd7770
[feature] Status thread mute/unmute functionality (#2278)
* add db models + functions for keeping track of threads

* give em the old linty testy

* create, remove, check mutes

* swagger

* testerino

* test mute/unmute via api

* add info log about new index creation

* thread + allow muting of any remote statuses that mention a local account

* IsStatusThreadMutedBy -> IsThreadMutedByAccount

* use common processing functions in status processor

* set = NULL

* favee!

* get rekt darlings, darlings get rekt

* testrig please, have mercy muy liege
2023-10-25 15:04:53 +01:00
kim 69ba9a79a1
[chore] de-interface{} the federator and dereferencer structs (#2285)
* de-interface{} the federator and dereferencer structs

* fix broken type signatures
2023-10-23 10:58:13 +01:00
kim c6e00afc7c
[feature] tentatively start adding polls support (#2249) 2023-10-04 13:09:42 +01:00
kim 297b6eeaaa
[chore] internal/ap: add pollable AS types, code reformatting, general niceties (#2248) 2023-10-03 14:59:30 +01:00
kim 9f9fcf743d
[bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
kim 8f67dd583d
[chore] deinterface the typeutils.Converter and update to use state structure (#2217)
* update typeconverter to use state structure

* deinterface the typeutils.TypeConverter -> typeutils.Converter

* finish copying over old type converter code comments

* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
2023-09-23 18:44:11 +02:00
kim fc11deeb83
[feature] add paging to AP following / followers endpoints (#2198) 2023-09-20 16:49:46 +01:00
kim 7293d6029b
[feature] add paging to account follows, followers and follow requests endpoints (#2186) 2023-09-12 14:00:35 +01:00
tobi 4b594516ec
[feature] Allow admins to expire remote public keys; refetch expired keys on demand (#2183) 2023-09-12 10:43:12 +01:00
kim 916c6d07ba
[bugfix] fix checks for deref the same status descendants / ascendants (#2181) 2023-09-05 11:22:02 +01:00
tobi a1768a83e0
[bugfix] Suppress 'errNoEntries' warnings from InboxForwarding function call (#2102) 2023-08-11 14:17:36 +01:00