Commit Graph

35 Commits

Author SHA1 Message Date
Giacomo Leidi 6bd2d606df
Store notifications. (#180) 2023-05-22 13:00:37 +02:00
Simone Robutti 370e00d187
decouple dataclasses from models (#181)
* fixed parsing bug

* implemented events and publications endpoints

split endpoints by entity

removed credentials

* add pagination (#179)

* added pagination

* integrated pagination with tortoise

* added test for publications

* removed converter file

* moved publications to dataclasses module

* implemented import pattern on dataclasses to prevent circular imports

* removed redundant fetch

* removed unused query

* split build_publications

* split failed_publications

* removed redundant query functions

* split publication retrieve

* split all read functions

* removed redundant write function

* fixed lock
2022-12-11 14:15:04 +01:00
Simone Robutti ddc706e201
Add publication crud (#178)
* fixed parsing bug

* implemented events and publications endpoints

split endpoints by entity

removed credentials

* add pagination (#179)

* added pagination

* integrated pagination with tortoise

* added test for publications

* removed converter file

* updated dependencies
2022-12-07 21:46:57 +01:00
Simone Robutti 44340fde8f
Introduced FastAPI and Postgres support (#173)
* added poc

* added check for sqlite db

* added events test

* draft docker-compose-test.yml

* improved docker-compose

* added support for postgres migrations

* add documentation

* added some qol to migrations

* added migration generation script

* removed settings.toml

* waiting for postgress in script

* commented script

* added sample web config

* fixed tests

* mock memory db

* reviewed PR
2022-10-14 22:11:27 +02:00
Simone Robutti 63a30bb483
refactor coordinators (#171)
* introduced CommandConfig object

* added dry_run for start command

* added test to start_dry_run

* added dry_run recap

* fixed import

* improved printing

* fixed zulip debug info

* improved recap dry-run print

* moved coordinator classes into dedicated package

* removed unused init

* divided event coordinators from recap coordinators

* added some docstrings
2022-09-24 16:46:03 +02:00
Giacomo Leidi 4ce6c7b171
Update dependencies. (#170)
* Update Guix commit.

* Update dependencies

* Drop CONNECTION_NAME hack.
2022-07-26 22:23:47 +02:00
Simone Robutti 9d71ef36b9
add dry run (#169)
* introduced CommandConfig object

* added dry_run for start command

* added test to start_dry_run

* added dry_run recap

* fixed import

* improved printing

* fixed zulip debug info

* improved recap dry-run print
2022-07-12 07:39:56 +02:00
Giacomo Leidi b66c94c8a2
Add publish command. (#167)
* Add publish command.

* publish: Add tests.

* Add list-platforms and test-configuration.

* Update Guix dependencies.

* Move publishing of events and publications to retry.
2022-05-17 23:14:52 +02:00
Giacomo Leidi 002399161d
Pull publish (#157)
* Add pull command.

* Add publish command.

* Enforce start's old semantics.

* Add pull test cases:

- Pull with no events anywhere
- Pull with one event on mobilizon and none on the db
- Pull with one event on mobilizon and another on the db
- Pull with one event on mobilizon and the same event on the db

* Add pull tests:

- Sequence: pull->start
- Sequence: start->pull
- Sequence: pull->new event on mobilizon -> pull

* manifest.scm: Add cloc.

* query: read: Prefer return list to generators.

* tests: pull: Actually test that models are written to the DB.
2022-03-22 21:16:34 +01:00
Simone Robutti 1217b17326
fix event retry (#152)
* added twitter error handling

* added facebook tests

* added header format test

* added multiple newlines check

* added test list command

* fixed commands structure

* fixed event retry

* fixed publication retry

* added publication tests

* removed unused option

* fixed list begin/end window

* added test retry failures

* linting

* refactored sender

* added timezone freeze

* fixed facebook-sdk and beatifulsoup errors
2022-03-06 10:41:02 +01:00
Simone Robutti dae9dfd889 set timezone to CUT 2022-03-04 10:56:20 +01:00
Simone Robutti cb0fe7b5fc
set timezone to Rome (#153) 2022-03-04 10:45:11 +01:00
Giacomo Leidi 529f83825e
Pave the way for using only EventPublications outside of storage module. (#149)
* Move `EventPublication.from_orm` to `storage.query.converter`.

Co-authored-by: Simone Robutti <simone.robutti@protonmail.com>
2022-02-23 17:26:13 +01:00
Simone Robutti f0a7449336
rework maintenance cli (#140)
* added default

* moved object before verb

* added event retry

* added publication_retry

* renamed inspect to list

* fixed retry publication

* fixed retry event and structure
2022-02-14 21:10:27 +01:00
magowiz 45802ecbdd
Feature/optimizing download (#118)
* add column last_update_time

* save last_update_time in event db record

* use id + updatedAt for comparison instead of mobilizon_id, this will treat updated events like new ones

* rework event selection/comparison to include unpublished with updates to ones need to be saved

* added update for unpublished events

* tests: test_update: Add create_unpublished_events tests.

* Move `MobilizonEvent.to_model` to `storage.query`

* Move `MobilizonEvent.from_model` to `storage.query`

* Move `MobilizonEvent.compute_status` to `storage.query`

* Move `publishers.exception.EventNotFound` to `storage.query.exceptions`
2022-02-09 00:54:56 +01:00
Giacomo Leidi 1f1ff2e5c2
Debug image (#131)
* config: get_settings_files_paths: Return first existing path.

* tests: Set SETTINGS_FOR_DYNACONF at import time.

* config: get_settings_files_paths: Log config path.

* manifest.scm: Add docker-compose.

* build_docker_image.sh: Allow building debug image.

* storage: db: Use the same configuration for Tortoise and Aerich.

* Distribuite migration queries.

* storage: implement_db_changes: Use pkg_resources.
2022-01-26 10:11:16 +01:00
Simone Robutti a0a1d43fa0
rework config (#121)
* simplified config behavior

* temp

* removed redundant tests

* removed publication window

* removed settings_file cli option

* add pre_test code, in order to set environment variables

* Revert "add pre_test code, in order to set environment variables"

This reverts commit 0d25f9313a.

Co-authored-by: magowiz <magowiz@gmail.com>
Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2022-01-08 00:54:27 +01:00
Simone Robutti 5bce70b4a1
retry command (#119)
* added failed publication retrieval

* completed retry command

* added more error handling

* added atomic to query
2022-01-04 21:35:43 +01:00
Giacomo Leidi 1efa191771
Query refactoring (#102)
* Rename query modules.

* storage: save_publication_report: Create publications.

* Remove placeholder PublicationStatus.UNSAVED

* Minor fixes.
2021-11-24 23:58:06 +01:00
Simone Robutti 2476686c33
added facebook publisher (#99)
* added facebook publisher

* mobilizon-reshare.git: [propagated-inputs]: Add python-sdk-facebook.

This package definition has been generated with `guix import pypi -r
facebook-sdk`.

* mobilizon-reshare.git: [propagated-inputs]: Use python-facebook-sdk.git.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-20 15:40:10 +01:00
Simone Robutti 5335ed8cc3
command tests (#97)
* filtering publications with inactive publishers

* filtering publications with inactive publishers

* WIP: Generate publications at runtime.

TODO:
- change `MobilizonEvent.compute_status`'s contract and break everything
- while we're at it we should remove `PublicationStatus.WAITING`
- test `storage.query.create_publications_for_publishers`

* cli: inspect_events: Unnest if-then-else.

* publishers: abstract: Remove `EventPublication.make`.

* fixed tests

* split query.py file

* added tests for get_unpublished_events

* added tests

* more tests

* added start test

* refactored start test

* added test start with db event

* added test recap

* added failed publication test

* added format test

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-11 16:20:50 +01:00
Simone Robutti 4dc1e4080a
fix publisher deactivation (#93)
* filtering publications with inactive publishers

* filtering publications with inactive publishers

* WIP: Generate publications at runtime.

TODO:
- change `MobilizonEvent.compute_status`'s contract and break everything
- while we're at it we should remove `PublicationStatus.WAITING`
- test `storage.query.create_publications_for_publishers`

* cli: inspect_events: Unnest if-then-else.

* publishers: abstract: Remove `EventPublication.make`.

* fixed tests

* split query.py file

* added tests for get_unpublished_events

* added tests

* more tests

* added start test

* main: start: Remove filter_publications_with_inactive_publishers.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-11 15:18:04 +01:00
Giacomo Leidi 6654d13ed0
publishers: Add Zulip publisher. (#62)
* - publishers: Add Zulip publisher.
- README.md.: Add Zulip.
- LICENSE: `fold -s LICENSE | sed 's/ *$//g' > l; mv l LICENSE`

* tests: publishers: Test Zulip publisher.

* publishers: templates: Update Zulip template.

* zulip: Add more tests.

* release.sh: Safer publication.

* tests: publishers: Move zulip specific fixtures to test_zulip.py
2021-09-26 20:12:06 +02:00
Simone Robutti 360740eae0
format event cli (#61)
* added get_formatted_event

* fixed cli settings

* added format command

* refactored inspect
2021-09-09 23:04:19 +02:00
tinoilcotechino 394d84b0a5
settings.toml: Move secrets to dedicated file. (#60)
Configuration files are checked in this order:

      1. CLI argument
      2. `$MOBILIZION_RESHARE_SETTINGS_FILE` environment variable;
      3. User configuration directory. On Linux that's `$XDG_CONFIG_HOME/mobilizon_reshare/<mobilizon-reshare-version>`;
      4. System configuration directory. On Linux that's the first element in `$XDG_CONFIG_DIRS` + `/mobilizon_reshare/<mobilizon-reshare-version>`.
      5. The default configuration distributed with the package.

The first available configuration file will be loaded. Secrets can be set either
via environment variables (such as `export MOBILIZON_RESHARE_PUBLISHER_TELEGRAM_TOKEN="my-telegram-token"` )
or by setting the `$SECRETS_FOR_DYNACONF` environment variable to a `secrets.(toml|yaml|ini|json)` file.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-09-07 23:39:58 +02:00
tinoilcotechino 09124b6358
Make `Event.mobilizon_id` a proper `UUIDField`. (#59)
This fixes #38.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-29 14:51:52 +02:00
tinoilcotechino 489e3d66ad
Fix PublisherCoordinatorReport generation. (#54)
* Fix PublisherCoordinatorReport generation.

Without this patch we are given *either* the failed or the successful
publication reports. We actually need both, this patch implements the
merge.

* Reformat

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-28 13:17:39 +02:00
Simone Robutti 2c8063cf4a
rename everything (#50)
* fixed visualization

* simplified tests

* split into files

* refactored test expected publications

* split update tests

* expanded specifications and tests

* added event_status window tests

* fixed 'all' command

* renamed everything

* fixed uppercase
2021-08-16 10:49:52 +02:00
SlyK182 b75f0ff057
Updated statuses management, tests (#41)
* Updated statuses management, tests

* storage: query: Generalize event loading logic.

* reformat

* storage: query: Rename load_events to prefetch_event_relations.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-07-15 18:13:11 +02:00
Simone Robutti a33a1d7b3e removed duplicated settings 2021-07-08 10:31:28 +02:00
Simone Robutti 40919a994f
publication window (#24)
* added get events stub

* added event parsing

* removed datetime

* added config and tests

* fixed response format in test

* more tests

* added error handling for request

* improved dummy data

* added get_unpublished_events

* added test

* removed last_accessed

* mobilizon_url moved to fixture

* added config comments

* moved mobilizon group to config

* added method

* removed non-working test settings

* added publishing window logic and tests

* added inner/outer window check and mocking

* improved fixture

* fixed outer window

* fixed tests
2021-07-07 11:19:37 +02:00
tinoilcotechino 6cea51bcab
Publications (#28)
* mobilizon_bots: query: Add create_publisher.

* Move PublicationStatus to models.publication.

* Move NotificationStatus to models.notification.

* storage: query: Add events_with_status.

* storage: query: Add get_unpublished_events.

* storage: query: Add create_unpublished events.

This function takes care of looking into the database
to store only the events whose  is not
already present.

* event: event: Support multiple publications.

This patch changes the public interface of `MobilizonEvent` to
support multiple publications. This mainly entails two changes:

  - When instancing a `MobilizonEvent`  from an `Event` model
    the `publication_status` will be computed by looking at
    the statuses of all the related publications.
  - Now the `publication_time` is a `dict[str, Arrow]`. This
    enables tracking multiple social platforms publication time.

* Update main.

* Minor fixes

* Better definition of MobilizonEvent.publication_status.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-07-05 23:07:12 +02:00
tinoilcotechino c129edd1e5
Add models (#18) 2021-05-31 01:11:50 +02:00
Simone Robutti fcd5116d1c
download events (#19)
* added get events stub

* added event parsing

* removed datetime

* added config and tests

* fixed response format in test

* more tests

* added error handling for request

* improved dummy data

* added get_unpublished_events

* added test

* removed last_accessed

* mobilizon_url moved to fixture

* added config comments

* moved mobilizon group to config
2021-05-30 21:47:36 +02:00
Simone Robutti 4f194ab01c added tests for strategy 2021-05-05 14:29:13 +02:00