162f798df7
simplify (and hand-write) query in refresh_account_with_oldest_post.
...
.#166
this takes about 1/5th the time of the old query. it's still kinda slow
2020-12-29 21:49:27 +01:00
c5c4b72c6f
ensure fetch_acc is rescheduled properly when doing historic fetch
...
the unique flag on the task means we can't reschedule it for right now
but we must wait for the current one to end. this commit introduces a 1
second delay before the task is ran again, which should be enough
2019-09-13 03:07:58 +02:00
249842ed9f
fix newer twitter accounts not fetching new posts after the initial fetch
...
twitter returns the post on the max_id boundary, which mastodon does not
do. the code that decides to move from historical fetch to a new batch
assumes that no posts will be returned at all when we're done, but in
twitter's case that will never happen, or not until the oldest post is
deleted. this change updates that code to ignore any posts returned that
match either max_id or since_id
2019-08-28 11:32:19 +02:00
f01b5b1511
guhhhhhhh. i forgot an import
2019-07-19 05:48:43 +02:00
7fad5ea458
shorten frequency at which we refresh the most stale posts
2019-07-16 18:50:25 +00:00
36860b6bf7
ignore posts that are not known to match policy
...
until now, delete_from_account would only ignore posts that were too
young, but would fetch and refresh posts that didn't match
fave/media/DM policies. this was fine most of the time but would result
in a lot of refreshes if those policies were very restrictive. now these
policies are respected when selecting candidate posts for deletion
closes #174
2019-05-11 20:21:43 +02:00
2b39a61442
increased task frequency for refreshes. decreased for bookkeeping
2019-03-11 12:18:51 +01:00
0f3df6ad24
remove leftover debug print, standardise task prints
2019-03-11 12:12:04 +01:00
249fab7014
make fetch_acc unique
...
now that it doesn't carry state info in its arguments, it's safe to skip
dupes
2019-03-11 12:08:47 +01:00
fc06355bca
Merge pull request #170 from codl/fix-fetch-acc
...
fix crash in fetch_acc when user has no posts
2019-03-11 03:19:14 +01:00
1430361763
Merge pull request #169 from codl/fix-deadlock
...
disable autoflush in refresh_posts, fixing deadlocks. closes GH-19
2019-03-11 03:18:58 +01:00
6d6184f3d8
disable autoflush in refresh_posts, fixing deadlocks. closes GH-19
2019-03-11 03:10:32 +01:00
84089f8a40
refresh_account: increment backoff if something crashes
...
this is kind of a hotfix for the thing with pleroma returning bad error
messages crashing mastodon.py. but it's good practice anyway
2019-03-11 03:10:06 +01:00
84e0ad6b1f
fix crash in fetch_acc when user has no posts
2019-03-09 15:56:07 +01:00
dfcc9287b8
check for existence before refreshing oldest post/account
...
this only happens when you don't have any accounts yet and it is a bad
look the first time you set up a new forget install
2019-03-08 01:24:13 +01:00
4054ebfffd
Merge branch 'master' into fix-refresh
2019-02-24 16:35:44 +01:00
d5e0b43c9e
fix: posts not getting refreshed
...
turns out i was not touching updated_at, so i'd always be refreshing
the same posts unless one of them actually changed
this is embarassing, there are posts in the DB that haven't been
refreshed since 2017
2019-02-24 16:28:59 +01:00
a6c5361138
more robust fetching. closes #13
...
this also refactors libforget so neither the twitter or mastodon lib
insert posts directly
2019-02-24 15:40:41 +01:00
c54390801c
code quality fixes
...
also ran yapf on tasks.py
2018-08-14 03:51:14 +02:00
b3de8e10ac
add exponential backoff
2018-07-06 01:44:30 +02:00
e2f1ca1732
replace messy version stuff with versioneer
2018-01-31 23:16:54 +01:00
51e070af3d
three-way favourite policy. closes #18
2018-01-03 18:00:59 +01:00
b11ffbbc11
more knob twiddling
2017-12-28 15:30:02 +01:00
e6a582431d
twiddle knob
2017-12-28 13:22:15 +01:00
8b7db5cce2
simplify mastodon instance scoring
2017-12-28 13:12:55 +01:00
9574476491
OOPS WHOOPS uh oh
2017-12-28 03:25:27 +01:00
ada719faeb
add ability to keep only posts without media
...
closes #16
2017-12-28 03:02:08 +01:00
c0c1591dfa
shorten delay between delete queue ups
2017-12-27 22:01:45 +01:00
f2cdc30c9e
fix #17 ignore fav on retweets
...
im not sure how it works on mastodon so i just took the safe option
2017-12-27 21:25:09 +01:00
8422b6d89b
add soft and hard time limits to every task
2017-09-30 20:44:27 +02:00
a6870d3775
codacy
2017-09-24 23:54:03 +02:00
fd0bd49bdd
lol oops
2017-09-21 02:33:00 +02:00
fb65bcf0ac
dont open one redis client per task 👀
2017-09-21 02:27:30 +02:00
0d7c1af13d
twiddle knob
2017-09-21 02:19:42 +02:00
0ad980f499
twiddle knob
2017-09-21 01:25:25 +02:00
c135dc793e
rename lib to libforget
2017-09-20 23:04:44 +02:00
89e35786d0
oops!!!!!
2017-09-20 14:44:01 +02:00
06c0de29da
oops
2017-09-20 14:42:45 +02:00
6bce7b1d04
make sure only one of each task runs at once
2017-09-20 14:39:31 +02:00
b0b17ae39f
oops
2017-09-20 12:58:16 +02:00
53c2e8a023
another measure to avoid hitting rate limits
2017-09-20 12:41:10 +02:00
02b564283f
twiddle the knob
2017-09-17 19:11:45 +02:00
34b6f622a1
oops! oops. forgot to commit transaction
2017-09-17 15:21:48 +02:00
614db2964d
update mastodon instance score based on active accounts
...
this is far more interesting than updating every time someone logs in,
since users don't have a reason to log in very often
2017-09-17 15:12:46 +02:00
7c8c49a2ae
an attempt at preventing hitting rate limits on mastodon
...
most tasks don't need to be retried if they have failed, they will be
queued up again in the next batch
2017-09-16 20:08:06 +02:00
bde08c6c49
pylint
2017-09-06 13:08:06 +02:00
840f56a769
send PermanentErrors to sentry
2017-09-05 13:01:33 +02:00
e8dbcb1d14
oops!!!!!
2017-09-05 00:08:33 +02:00
edf7732e67
overhaul and abstract errors in service libs
...
also add support for making an account dormant if there is a permanent
error
2017-09-04 22:15:05 +02:00
5e22a0531f
capitalise service names
2017-09-04 22:15:04 +02:00
codl