Commit Graph

40 Commits

Author SHA1 Message Date
Giacomo Leidi e381c1b522
Migrate to importlib and update some dependencies (#189)
* Migrate to importlib.

* Update CI
2024-01-28 21:08:17 +01:00
Giacomo Leidi bf3170cb6f
Decouple DB instantiation from logger instantion. (#188) 2023-07-11 22:24:56 +02:00
Giacomo Leidi 6bd2d606df
Store notifications. (#180) 2023-05-22 13:00:37 +02:00
Giacomo Leidi 3874acf247
Failures (#183)
* Update beautifulsoup4 to 4.11.

* Update docker image.

* Add type annotations.

* Publish where you can and notify errors.

* Add tests for partial publishing.
2023-05-22 12:53:32 +02:00
Simone Robutti 370e00d187
decouple dataclasses from models (#181)
* fixed parsing bug

* implemented events and publications endpoints

split endpoints by entity

removed credentials

* add pagination (#179)

* added pagination

* integrated pagination with tortoise

* added test for publications

* removed converter file

* moved publications to dataclasses module

* implemented import pattern on dataclasses to prevent circular imports

* removed redundant fetch

* removed unused query

* split build_publications

* split failed_publications

* removed redundant query functions

* split publication retrieve

* split all read functions

* removed redundant write function

* fixed lock
2022-12-11 14:15:04 +01:00
Simone Robutti ddc706e201
Add publication crud (#178)
* fixed parsing bug

* implemented events and publications endpoints

split endpoints by entity

removed credentials

* add pagination (#179)

* added pagination

* integrated pagination with tortoise

* added test for publications

* removed converter file

* updated dependencies
2022-12-07 21:46:57 +01:00
Giacomo Leidi cf9ffd2149 query: Add get_event_publications. 2022-10-26 00:43:25 +02:00
Simone Robutti 44340fde8f
Introduced FastAPI and Postgres support (#173)
* added poc

* added check for sqlite db

* added events test

* draft docker-compose-test.yml

* improved docker-compose

* added support for postgres migrations

* add documentation

* added some qol to migrations

* added migration generation script

* removed settings.toml

* waiting for postgress in script

* commented script

* added sample web config

* fixed tests

* mock memory db

* reviewed PR
2022-10-14 22:11:27 +02:00
Simone Robutti 63a30bb483
refactor coordinators (#171)
* introduced CommandConfig object

* added dry_run for start command

* added test to start_dry_run

* added dry_run recap

* fixed import

* improved printing

* fixed zulip debug info

* improved recap dry-run print

* moved coordinator classes into dedicated package

* removed unused init

* divided event coordinators from recap coordinators

* added some docstrings
2022-09-24 16:46:03 +02:00
Giacomo Leidi 4ce6c7b171
Update dependencies. (#170)
* Update Guix commit.

* Update dependencies

* Drop CONNECTION_NAME hack.
2022-07-26 22:23:47 +02:00
Simone Robutti 9d71ef36b9
add dry run (#169)
* introduced CommandConfig object

* added dry_run for start command

* added test to start_dry_run

* added dry_run recap

* fixed import

* improved printing

* fixed zulip debug info

* improved recap dry-run print
2022-07-12 07:39:56 +02:00
Giacomo Leidi b66c94c8a2
Add publish command. (#167)
* Add publish command.

* publish: Add tests.

* Add list-platforms and test-configuration.

* Update Guix dependencies.

* Move publishing of events and publications to retry.
2022-05-17 23:14:52 +02:00
Giacomo Leidi 002399161d
Pull publish (#157)
* Add pull command.

* Add publish command.

* Enforce start's old semantics.

* Add pull test cases:

- Pull with no events anywhere
- Pull with one event on mobilizon and none on the db
- Pull with one event on mobilizon and another on the db
- Pull with one event on mobilizon and the same event on the db

* Add pull tests:

- Sequence: pull->start
- Sequence: start->pull
- Sequence: pull->new event on mobilizon -> pull

* manifest.scm: Add cloc.

* query: read: Prefer return list to generators.

* tests: pull: Actually test that models are written to the DB.
2022-03-22 21:16:34 +01:00
Simone Robutti 1217b17326
fix event retry (#152)
* added twitter error handling

* added facebook tests

* added header format test

* added multiple newlines check

* added test list command

* fixed commands structure

* fixed event retry

* fixed publication retry

* added publication tests

* removed unused option

* fixed list begin/end window

* added test retry failures

* linting

* refactored sender

* added timezone freeze

* fixed facebook-sdk and beatifulsoup errors
2022-03-06 10:41:02 +01:00
Giacomo Leidi 529f83825e
Pave the way for using only EventPublications outside of storage module. (#149)
* Move `EventPublication.from_orm` to `storage.query.converter`.

Co-authored-by: Simone Robutti <simone.robutti@protonmail.com>
2022-02-23 17:26:13 +01:00
Simone Robutti f0a7449336
rework maintenance cli (#140)
* added default

* moved object before verb

* added event retry

* added publication_retry

* renamed inspect to list

* fixed retry publication

* fixed retry event and structure
2022-02-14 21:10:27 +01:00
magowiz 45802ecbdd
Feature/optimizing download (#118)
* add column last_update_time

* save last_update_time in event db record

* use id + updatedAt for comparison instead of mobilizon_id, this will treat updated events like new ones

* rework event selection/comparison to include unpublished with updates to ones need to be saved

* added update for unpublished events

* tests: test_update: Add create_unpublished_events tests.

* Move `MobilizonEvent.to_model` to `storage.query`

* Move `MobilizonEvent.from_model` to `storage.query`

* Move `MobilizonEvent.compute_status` to `storage.query`

* Move `publishers.exception.EventNotFound` to `storage.query.exceptions`
2022-02-09 00:54:56 +01:00
Giacomo Leidi 1f1ff2e5c2
Debug image (#131)
* config: get_settings_files_paths: Return first existing path.

* tests: Set SETTINGS_FOR_DYNACONF at import time.

* config: get_settings_files_paths: Log config path.

* manifest.scm: Add docker-compose.

* build_docker_image.sh: Allow building debug image.

* storage: db: Use the same configuration for Tortoise and Aerich.

* Distribuite migration queries.

* storage: implement_db_changes: Use pkg_resources.
2022-01-26 10:11:16 +01:00
Simone Robutti 5804b55dd8
Fix markdown cropping (#128)
* fixed error log

* changed telegram publisher to use HTML format

* fixed dataclass update issue

* docker: Add python-telegram-bot.

* docker: mobilizon-reshare: Reenable non deterministic tests.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2022-01-18 01:39:00 +01:00
Simone Robutti 5bce70b4a1
retry command (#119)
* added failed publication retrieval

* completed retry command

* added more error handling

* added atomic to query
2022-01-04 21:35:43 +01:00
magowiz 86e698271c add intermediate variable since aerich doesn't play nice with functions as configuration entrypoint 2021-12-18 12:54:09 +01:00
Simone Robutti 31da51b69b
Merge pull request #110 from magowiz/feature/aerich
Feature/aerich
2021-12-14 22:54:50 +01:00
magowiz e836e51ee4 fixed exit on migration exception 2021-12-14 22:37:49 +01:00
magowiz c27a18cc17 completed chain, handling exception 2021-12-07 16:58:12 +01:00
magowiz e6f25178a5 handle aerich migrations not found 2021-12-07 12:10:32 +01:00
magowiz 14bd3afa5b move aerich conf to storage.db 2021-12-05 18:37:35 +01:00
magowiz 6cb1f3eedd changed name of __implement_db_changes__ to _implement_db_changes and using logging facility for printing starting update database 2021-12-05 17:11:24 +01:00
magowiz f77e11fd9e add aenrich with migration initialized and auto-upgrade on start 2021-12-04 19:34:00 +01:00
magowiz 1f27f2f78c add aerich upgrade before init 2021-12-04 17:55:43 +01:00
Giacomo Leidi 6f8f96d5b6 cli: inspect: Refactor command.
Now the inspect command can display informations about
different kind of objects including events and publications.

This patch also changes the cli to output columnar values
suitable for further processing with standard Unix
tool, such as awk.

$ mobilizon-reshare.sh inspect publication -s completed | awk '{ print  }' | sort | uniq -c
[2021-12-01 01:05:55,321] [20] [INFO] Tortoise-ORM shutdown
      2 mastodon
      2 telegram
      2 zulip
2021-12-01 01:13:59 +01:00
Giacomo Leidi 64c9d168c3 storage: publications_with_status: Return a list[Publication]. 2021-11-30 00:05:11 +01:00
Giacomo Leidi 1efa191771
Query refactoring (#102)
* Rename query modules.

* storage: save_publication_report: Create publications.

* Remove placeholder PublicationStatus.UNSAVED

* Minor fixes.
2021-11-24 23:58:06 +01:00
Giacomo Leidi ed8f752fe6
Make logs a little bit more informative. (#100) 2021-11-20 15:53:38 +01:00
Simone Robutti 4dc1e4080a
fix publisher deactivation (#93)
* filtering publications with inactive publishers

* filtering publications with inactive publishers

* WIP: Generate publications at runtime.

TODO:
- change `MobilizonEvent.compute_status`'s contract and break everything
- while we're at it we should remove `PublicationStatus.WAITING`
- test `storage.query.create_publications_for_publishers`

* cli: inspect_events: Unnest if-then-else.

* publishers: abstract: Remove `EventPublication.make`.

* fixed tests

* split query.py file

* added tests for get_unpublished_events

* added tests

* more tests

* added start test

* main: start: Remove filter_publications_with_inactive_publishers.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-11 15:18:04 +01:00
Simone Robutti e16dd19a7c
simplified validation (#91) 2021-10-24 21:32:28 +02:00
Simone Robutti 5e171216d2
event recap (#69)
* added basic recap feature (no error handling)

* introduced abstractpublication

* extracted base reports

* added error report to recap

* added test

* added docs
2021-10-16 01:25:45 +02:00
Giacomo Leidi 6654d13ed0
publishers: Add Zulip publisher. (#62)
* - publishers: Add Zulip publisher.
- README.md.: Add Zulip.
- LICENSE: `fold -s LICENSE | sed 's/ *$//g' > l; mv l LICENSE`

* tests: publishers: Test Zulip publisher.

* publishers: templates: Update Zulip template.

* zulip: Add more tests.

* release.sh: Safer publication.

* tests: publishers: Move zulip specific fixtures to test_zulip.py
2021-09-26 20:12:06 +02:00
tinoilcotechino 489e3d66ad
Fix PublisherCoordinatorReport generation. (#54)
* Fix PublisherCoordinatorReport generation.

Without this patch we are given *either* the failed or the successful
publication reports. We actually need both, this patch implements the
merge.

* Reformat

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-28 13:17:39 +02:00
Simone Robutti 2197e07213
notify failure (#52)
* fixed visualization

* simplified tests

* split into files

* refactored test expected publications

* split update tests

* expanded specifications and tests

* added event_status window tests

* fixed 'all' command

* renamed everything

* fixed uppercase

* refactored main and publisher to add notifications

* tested report successful

* added tests to publisher coordinator

* added more coordinator tests

* test coordinator success
2021-08-27 23:45:24 +02:00
Simone Robutti 2c8063cf4a
rename everything (#50)
* fixed visualization

* simplified tests

* split into files

* refactored test expected publications

* split update tests

* expanded specifications and tests

* added event_status window tests

* fixed 'all' command

* renamed everything

* fixed uppercase
2021-08-16 10:49:52 +02:00