61700f3dd9
fix deps
2022-03-01 00:59:55 +01:00
1c65cd2556
also update date when updating batch end
2022-02-27 13:20:24 +01:00
a85095cd00
remove fk from fetch_current_batch_end_id
...
an attempt at fixing #584
2022-02-27 13:12:47 +01:00
9d2147e905
Merge branch 'misskey'
2022-02-27 00:38:03 +01:00
b54a26cf8f
fix exception when no posts are in database
...
Instead of using Query.one use Query.one_or_none. The check after
setting the variable suggests this was the intended behaviour instead
of throwing errors if there are no posts in the database.
2021-12-09 18:57:00 +01:00
623a7e4415
fix all kinds of mistakes
2021-11-10 12:33:55 +01:00
ed1c42d30d
WIP: implement misskey API
2021-11-09 10:07:56 +01:00
162f798df7
simplify (and hand-write) query in refresh_account_with_oldest_post.
...
.#166
this takes about 1/5th the time of the old query. it's still kinda slow
2020-12-29 21:49:27 +01:00
c5c4b72c6f
ensure fetch_acc is rescheduled properly when doing historic fetch
...
the unique flag on the task means we can't reschedule it for right now
but we must wait for the current one to end. this commit introduces a 1
second delay before the task is ran again, which should be enough
2019-09-13 03:07:58 +02:00
249842ed9f
fix newer twitter accounts not fetching new posts after the initial fetch
...
twitter returns the post on the max_id boundary, which mastodon does not
do. the code that decides to move from historical fetch to a new batch
assumes that no posts will be returned at all when we're done, but in
twitter's case that will never happen, or not until the oldest post is
deleted. this change updates that code to ignore any posts returned that
match either max_id or since_id
2019-08-28 11:32:19 +02:00
f01b5b1511
guhhhhhhh. i forgot an import
2019-07-19 05:48:43 +02:00
7fad5ea458
shorten frequency at which we refresh the most stale posts
2019-07-16 18:50:25 +00:00
36860b6bf7
ignore posts that are not known to match policy
...
until now, delete_from_account would only ignore posts that were too
young, but would fetch and refresh posts that didn't match
fave/media/DM policies. this was fine most of the time but would result
in a lot of refreshes if those policies were very restrictive. now these
policies are respected when selecting candidate posts for deletion
closes #174
2019-05-11 20:21:43 +02:00
2b39a61442
increased task frequency for refreshes. decreased for bookkeeping
2019-03-11 12:18:51 +01:00
0f3df6ad24
remove leftover debug print, standardise task prints
2019-03-11 12:12:04 +01:00
249fab7014
make fetch_acc unique
...
now that it doesn't carry state info in its arguments, it's safe to skip
dupes
2019-03-11 12:08:47 +01:00
fc06355bca
Merge pull request #170 from codl/fix-fetch-acc
...
fix crash in fetch_acc when user has no posts
2019-03-11 03:19:14 +01:00
1430361763
Merge pull request #169 from codl/fix-deadlock
...
disable autoflush in refresh_posts, fixing deadlocks. closes GH-19
2019-03-11 03:18:58 +01:00
6d6184f3d8
disable autoflush in refresh_posts, fixing deadlocks. closes GH-19
2019-03-11 03:10:32 +01:00
84089f8a40
refresh_account: increment backoff if something crashes
...
this is kind of a hotfix for the thing with pleroma returning bad error
messages crashing mastodon.py. but it's good practice anyway
2019-03-11 03:10:06 +01:00
84e0ad6b1f
fix crash in fetch_acc when user has no posts
2019-03-09 15:56:07 +01:00
dfcc9287b8
check for existence before refreshing oldest post/account
...
this only happens when you don't have any accounts yet and it is a bad
look the first time you set up a new forget install
2019-03-08 01:24:13 +01:00
4054ebfffd
Merge branch 'master' into fix-refresh
2019-02-24 16:35:44 +01:00
d5e0b43c9e
fix: posts not getting refreshed
...
turns out i was not touching updated_at, so i'd always be refreshing
the same posts unless one of them actually changed
this is embarassing, there are posts in the DB that haven't been
refreshed since 2017
2019-02-24 16:28:59 +01:00
a6c5361138
more robust fetching. closes #13
...
this also refactors libforget so neither the twitter or mastodon lib
insert posts directly
2019-02-24 15:40:41 +01:00
c54390801c
code quality fixes
...
also ran yapf on tasks.py
2018-08-14 03:51:14 +02:00
b3de8e10ac
add exponential backoff
2018-07-06 01:44:30 +02:00
e2f1ca1732
replace messy version stuff with versioneer
2018-01-31 23:16:54 +01:00
51e070af3d
three-way favourite policy. closes #18
2018-01-03 18:00:59 +01:00
b11ffbbc11
more knob twiddling
2017-12-28 15:30:02 +01:00
e6a582431d
twiddle knob
2017-12-28 13:22:15 +01:00
8b7db5cce2
simplify mastodon instance scoring
2017-12-28 13:12:55 +01:00
9574476491
OOPS WHOOPS uh oh
2017-12-28 03:25:27 +01:00
ada719faeb
add ability to keep only posts without media
...
closes #16
2017-12-28 03:02:08 +01:00
c0c1591dfa
shorten delay between delete queue ups
2017-12-27 22:01:45 +01:00
f2cdc30c9e
fix #17 ignore fav on retweets
...
im not sure how it works on mastodon so i just took the safe option
2017-12-27 21:25:09 +01:00
8422b6d89b
add soft and hard time limits to every task
2017-09-30 20:44:27 +02:00
a6870d3775
codacy
2017-09-24 23:54:03 +02:00
fd0bd49bdd
lol oops
2017-09-21 02:33:00 +02:00
fb65bcf0ac
dont open one redis client per task 👀
2017-09-21 02:27:30 +02:00
0d7c1af13d
twiddle knob
2017-09-21 02:19:42 +02:00
0ad980f499
twiddle knob
2017-09-21 01:25:25 +02:00
c135dc793e
rename lib to libforget
2017-09-20 23:04:44 +02:00
89e35786d0
oops!!!!!
2017-09-20 14:44:01 +02:00
06c0de29da
oops
2017-09-20 14:42:45 +02:00
6bce7b1d04
make sure only one of each task runs at once
2017-09-20 14:39:31 +02:00
b0b17ae39f
oops
2017-09-20 12:58:16 +02:00
53c2e8a023
another measure to avoid hitting rate limits
2017-09-20 12:41:10 +02:00
02b564283f
twiddle the knob
2017-09-17 19:11:45 +02:00
34b6f622a1
oops! oops. forgot to commit transaction
2017-09-17 15:21:48 +02:00
codl
Johann150