62 Commits

Author SHA1 Message Date
Simone Robutti
f0a7449336
rework maintenance cli (#140)
* added default

* moved object before verb

* added event retry

* added publication_retry

* renamed inspect to list

* fixed retry publication

* fixed retry event and structure
2022-02-14 21:10:27 +01:00
magowiz
45802ecbdd
Feature/optimizing download (#118)
* add column last_update_time

* save last_update_time in event db record

* use id + updatedAt for comparison instead of mobilizon_id, this will treat updated events like new ones

* rework event selection/comparison to include unpublished with updates to ones need to be saved

* added update for unpublished events

* tests: test_update: Add create_unpublished_events tests.

* Move `MobilizonEvent.to_model` to `storage.query`

* Move `MobilizonEvent.from_model` to `storage.query`

* Move `MobilizonEvent.compute_status` to `storage.query`

* Move `publishers.exception.EventNotFound` to `storage.query.exceptions`
2022-02-09 00:54:56 +01:00
Giacomo Leidi
ad8647704d
Add event name to notifications. (#136) 2022-02-05 18:46:48 +01:00
Giacomo Leidi
1f1ff2e5c2
Debug image (#131)
* config: get_settings_files_paths: Return first existing path.

* tests: Set SETTINGS_FOR_DYNACONF at import time.

* config: get_settings_files_paths: Log config path.

* manifest.scm: Add docker-compose.

* build_docker_image.sh: Allow building debug image.

* storage: db: Use the same configuration for Tortoise and Aerich.

* Distribuite migration queries.

* storage: implement_db_changes: Use pkg_resources.
2022-01-26 10:11:16 +01:00
Simone Robutti
994102c937
added timezone_sensitive marker (#129) 2022-01-20 11:30:11 +01:00
Simone Robutti
5804b55dd8
Fix markdown cropping (#128)
* fixed error log

* changed telegram publisher to use HTML format

* fixed dataclass update issue

* docker: Add python-telegram-bot.

* docker: mobilizon-reshare: Reenable non deterministic tests.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2022-01-18 01:39:00 +01:00
Simone Robutti
a0a1d43fa0
rework config (#121)
* simplified config behavior

* temp

* removed redundant tests

* removed publication window

* removed settings_file cli option

* add pre_test code, in order to set environment variables

* Revert "add pre_test code, in order to set environment variables"

This reverts commit 0d25f9313a5a791b0c0ae5be63fa08aced69ba28.

Co-authored-by: magowiz <magowiz@gmail.com>
Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2022-01-08 00:54:27 +01:00
Simone Robutti
5bce70b4a1
retry command (#119)
* added failed publication retrieval

* completed retry command

* added more error handling

* added atomic to query
2022-01-04 21:35:43 +01:00
Giacomo Leidi
43d653f37c
Update guix package definition after core-updates merge. (#122) 2022-01-03 20:14:31 +01:00
Simone Robutti
94904a8917
Merge pull request #109 from Tech-Workers-Coalition-Italia/inspect
cli: inspect: Refactor command.
2021-12-11 11:17:59 +01:00
Simone Robutti
7c39d94b48 added new test 2021-12-05 16:50:37 +01:00
Giacomo Leidi
6f8f96d5b6 cli: inspect: Refactor command.
Now the inspect command can display informations about
different kind of objects including events and publications.

This patch also changes the cli to output columnar values
suitable for further processing with standard Unix
tool, such as awk.

$ mobilizon-reshare.sh inspect publication -s completed | awk '{ print  }' | sort | uniq -c
[2021-12-01 01:05:55,321] [20] [INFO] Tortoise-ORM shutdown
      2 mastodon
      2 telegram
      2 zulip
2021-12-01 01:13:59 +01:00
Giacomo Leidi
64c9d168c3 storage: publications_with_status: Return a list[Publication]. 2021-11-30 00:05:11 +01:00
Giacomo Leidi
fee794be48 test: puslishers: Adapt Zulip tests to HTTPResponseError. 2021-11-28 17:25:43 +01:00
Giacomo Leidi
f8bbd1df41
Fix publication exception handling for notification. (#105) 2021-11-27 23:31:44 +01:00
Giacomo Leidi
1efa191771
Query refactoring (#102)
* Rename query modules.

* storage: save_publication_report: Create publications.

* Remove placeholder PublicationStatus.UNSAVED

* Minor fixes.
2021-11-24 23:58:06 +01:00
Simone Robutti
2476686c33
added facebook publisher (#99)
* added facebook publisher

* mobilizon-reshare.git: [propagated-inputs]: Add python-sdk-facebook.

This package definition has been generated with `guix import pypi -r
facebook-sdk`.

* mobilizon-reshare.git: [propagated-inputs]: Use python-facebook-sdk.git.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-20 15:40:10 +01:00
Giacomo Leidi
f12992c5d9
publishers: zulip: Let the user specify their instance. (#95) 2021-11-11 16:25:09 +01:00
Simone Robutti
5335ed8cc3
command tests (#97)
* filtering publications with inactive publishers

* filtering publications with inactive publishers

* WIP: Generate publications at runtime.

TODO:
- change `MobilizonEvent.compute_status`'s contract and break everything
- while we're at it we should remove `PublicationStatus.WAITING`
- test `storage.query.create_publications_for_publishers`

* cli: inspect_events: Unnest if-then-else.

* publishers: abstract: Remove `EventPublication.make`.

* fixed tests

* split query.py file

* added tests for get_unpublished_events

* added tests

* more tests

* added start test

* refactored start test

* added test start with db event

* added test recap

* added failed publication test

* added format test

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-11 16:20:50 +01:00
Simone Robutti
4dc1e4080a
fix publisher deactivation (#93)
* filtering publications with inactive publishers

* filtering publications with inactive publishers

* WIP: Generate publications at runtime.

TODO:
- change `MobilizonEvent.compute_status`'s contract and break everything
- while we're at it we should remove `PublicationStatus.WAITING`
- test `storage.query.create_publications_for_publishers`

* cli: inspect_events: Unnest if-then-else.

* publishers: abstract: Remove `EventPublication.make`.

* fixed tests

* split query.py file

* added tests for get_unpublished_events

* added tests

* more tests

* added start test

* main: start: Remove filter_publications_with_inactive_publishers.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-11-11 15:18:04 +01:00
Simone Robutti
41e82f5035
better error handling (#92)
* safe logging of notifier failure to send

* added test
2021-10-24 21:43:09 +02:00
Simone Robutti
e16dd19a7c
simplified validation (#91) 2021-10-24 21:32:28 +02:00
Giacomo Leidi
6430de4a84
Add Mastodon publisher. (#83)
* Add Mastodon publisher.

This commit enables publishing on Mastodon and tries to define the
minimal requirements for adding a new platform to Mobilizon Reshare.

* publishers: exceptions: Add HTTPError.

* platforms: mastodon: Make toot length customizable.
2021-10-20 00:08:58 +02:00
Simone Robutti
8de70ef857
mobilizon link in template (#80)
* added basic recap feature (no error handling)

* introduced abstractpublication

* extracted base reports

* added error report to recap

* added test

* added docs

* implemented publisher and formatter

* fixed API for recap

* removed redundant config validation

* added config sample

* added mobilizon link to templates

* added link format to telegram

* added mobilizon link to recap

* fixed config and emoji
2021-10-19 07:35:18 +02:00
Simone Robutti
489d41179e
recap header (#79)
* added basic recap feature (no error handling)

* introduced abstractpublication

* extracted base reports

* added error report to recap

* added test

* added docs

* implemented publisher and formatter

* fixed API for recap

* removed redundant config validation

* added config sample

* added active publisher test

* added recap header template
2021-10-17 14:09:24 +02:00
Simone Robutti
71b65342b9
fix telegram active (#78)
* added basic recap feature (no error handling)

* introduced abstractpublication

* extracted base reports

* added error report to recap

* added test

* added docs

* implemented publisher and formatter

* fixed API for recap

* removed redundant config validation

* added config sample

* added active publisher test
2021-10-17 14:05:25 +02:00
Simone Robutti
5e171216d2
event recap (#69)
* added basic recap feature (no error handling)

* introduced abstractpublication

* extracted base reports

* added error report to recap

* added test

* added docs
2021-10-16 01:25:45 +02:00
Simone Robutti
b61a2c5c3c
More tests (#67)
* decoupled notifiers from event

* stub

* publishers working

* fixed format CLI

* fixed unit tests

* renamed abstractnotifier

* added another excluded character

* restored bundled secrets file

* test telegram escape

* tested telegram event validation

* added telegram response validation

* added pragma

* added zulip response validation test
2021-10-05 15:32:07 +02:00
Simone Robutti
bc212e7801
Check length (#66)
* decoupled notifiers from event

* stub

* publishers working

* fixed format CLI

* fixed unit tests

* renamed abstractnotifier

* added another excluded character

* restored bundled secrets file
2021-10-03 13:19:37 +02:00
Simone Robutti
b6b2402767
Refactor publication (#65)
* decoupled notifiers from event

* stub

* publishers working

* fixed format CLI

* fixed unit tests

* renamed abstractnotifier

* added another excluded character
2021-10-02 18:09:03 +02:00
Simone Robutti
3258e1b1c9
Telegram format (#64)
* improved template

* improved location format
2021-09-27 09:20:01 +02:00
Giacomo Leidi
6654d13ed0
publishers: Add Zulip publisher. (#62)
* - publishers: Add Zulip publisher.
- README.md.: Add Zulip.
- LICENSE: `fold -s LICENSE | sed 's/ *$//g' > l; mv l LICENSE`

* tests: publishers: Test Zulip publisher.

* publishers: templates: Update Zulip template.

* zulip: Add more tests.

* release.sh: Safer publication.

* tests: publishers: Move zulip specific fixtures to test_zulip.py
2021-09-26 20:12:06 +02:00
Simone Robutti
360740eae0
format event cli (#61)
* added get_formatted_event

* fixed cli settings

* added format command

* refactored inspect
2021-09-09 23:04:19 +02:00
tinoilcotechino
394d84b0a5
settings.toml: Move secrets to dedicated file. (#60)
Configuration files are checked in this order:

      1. CLI argument
      2. `$MOBILIZION_RESHARE_SETTINGS_FILE` environment variable;
      3. User configuration directory. On Linux that's `$XDG_CONFIG_HOME/mobilizon_reshare/<mobilizon-reshare-version>`;
      4. System configuration directory. On Linux that's the first element in `$XDG_CONFIG_DIRS` + `/mobilizon_reshare/<mobilizon-reshare-version>`.
      5. The default configuration distributed with the package.

The first available configuration file will be loaded. Secrets can be set either
via environment variables (such as `export MOBILIZON_RESHARE_PUBLISHER_TELEGRAM_TOKEN="my-telegram-token"` )
or by setting the `$SECRETS_FOR_DYNACONF` environment variable to a `secrets.(toml|yaml|ini|json)` file.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-09-07 23:39:58 +02:00
tinoilcotechino
09124b6358
Make Event.mobilizon_id a proper UUIDField. (#59)
This fixes #38.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-29 14:51:52 +02:00
tinoilcotechino
489e3d66ad
Fix PublisherCoordinatorReport generation. (#54)
* Fix PublisherCoordinatorReport generation.

Without this patch we are given *either* the failed or the successful
publication reports. We actually need both, this patch implements the
merge.

* Reformat

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-28 13:17:39 +02:00
Simone Robutti
2197e07213
notify failure (#52)
* fixed visualization

* simplified tests

* split into files

* refactored test expected publications

* split update tests

* expanded specifications and tests

* added event_status window tests

* fixed 'all' command

* renamed everything

* fixed uppercase

* refactored main and publisher to add notifications

* tested report successful

* added tests to publisher coordinator

* added more coordinator tests

* test coordinator success
2021-08-27 23:45:24 +02:00
Simone Robutti
2c8063cf4a
rename everything (#50)
* fixed visualization

* simplified tests

* split into files

* refactored test expected publications

* split update tests

* expanded specifications and tests

* added event_status window tests

* fixed 'all' command

* renamed everything

* fixed uppercase
2021-08-16 10:49:52 +02:00
Simone Robutti
450e5b0044
fix cli (#49)
* fixed visualization

* simplified tests

* split into files

* refactored test expected publications

* split update tests

* expanded specifications and tests

* added event_status window tests

* fixed 'all' command
2021-08-14 18:23:55 +02:00
tinoilcotechino
41317f062d
Publication report (#46)
* publishers: coordinator: Change `PublisherReport` to `PublicationReport`.

* publishers: AbstractNotifier: Add `AbstractNotifier.get_name`.

* models: Publication: Add `reason`.

* storage: query: Add `get_mobilizon_event_publications`.

* tests: query: Refactor models generation.

* storage: query: Add `save_publication_report`.

* Track publication ids during the publishing process.

This patch changes the PublisherCoordinator to keep track
of the different Publications it's performing. This also
enables multiple publications for the same publisher.

* tests: storage: Add some tests.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-05 00:29:50 +02:00
Simone Robutti
880a34115f
extend cli (#48)
* fixed teardown

* Updated statuses management, tests (#41)

* Updated statuses management, tests

* storage: query: Generalize event loading logic.

* reformat

* storage: query: Rename load_events to prefetch_event_relations.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>

* fixed test_window_no_event

* added strategy tests

* removed unused code

* added config tests

* added more config tests

* refactored tests

* updated pytest

* added inspect_all

* temp

* added colors

* storage: events_with_status: Use `EventPublicationStatus`.

* storage: events_with_status: Enable closed ranges.

* added time window to CLI

Co-authored-by: SlyK182 <60148777+SlyK182@users.noreply.github.com>
Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-08-04 18:53:58 +02:00
Simone Robutti
929e3aa78e
added format to markdown (#43)
* added format to markdown

* comments

* added markdown tests
2021-07-21 09:13:32 +02:00
Simone Robutti
8a27de8981
add tests (#44)
* fixed test_window_no_event

* added strategy tests

* removed unused code

* added config tests

* added more config tests

* refactored tests

* updated pytest
2021-07-18 18:23:30 +02:00
SlyK182
b75f0ff057
Updated statuses management, tests (#41)
* Updated statuses management, tests

* storage: query: Generalize event loading logic.

* reformat

* storage: query: Rename load_events to prefetch_event_relations.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-07-15 18:13:11 +02:00
Simone Robutti
9578f18078
first run (#36)
* added settings as CLI option

* added group missing error

* comments

* testing empty events

* fixed transaction

* main is working

* added logging and fixed publication

* refactored some publication code

* removed redundant field

* storage: query: Work around https://github.com/tortoise/tortoise-orm/issues/419 .

* storage: query: Update testing environment check.

* review

* tests: models: Test `MobilizonEvent.compute_status`.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-07-12 22:17:49 +02:00
Simone Robutti
a33a1d7b3e removed duplicated settings 2021-07-08 10:31:28 +02:00
Simone Robutti
d89659735d
added event selection function (#25) 2021-07-07 11:41:13 +02:00
Simone Robutti
40919a994f
publication window (#24)
* added get events stub

* added event parsing

* removed datetime

* added config and tests

* fixed response format in test

* more tests

* added error handling for request

* improved dummy data

* added get_unpublished_events

* added test

* removed last_accessed

* mobilizon_url moved to fixture

* added config comments

* moved mobilizon group to config

* added method

* removed non-working test settings

* added publishing window logic and tests

* added inner/outer window check and mocking

* improved fixture

* fixed outer window

* fixed tests
2021-07-07 11:19:37 +02:00
tinoilcotechino
6cea51bcab
Publications (#28)
* mobilizon_bots: query: Add create_publisher.

* Move PublicationStatus to models.publication.

* Move NotificationStatus to models.notification.

* storage: query: Add events_with_status.

* storage: query: Add get_unpublished_events.

* storage: query: Add create_unpublished events.

This function takes care of looking into the database
to store only the events whose  is not
already present.

* event: event: Support multiple publications.

This patch changes the public interface of `MobilizonEvent` to
support multiple publications. This mainly entails two changes:

  - When instancing a `MobilizonEvent`  from an `Event` model
    the `publication_status` will be computed by looking at
    the statuses of all the related publications.
  - Now the `publication_time` is a `dict[str, Arrow]`. This
    enables tracking multiple social platforms publication time.

* Update main.

* Minor fixes

* Better definition of MobilizonEvent.publication_status.

Co-authored-by: Giacomo Leidi <goodoldpaul@autistici.org>
2021-07-05 23:07:12 +02:00
SlyK182
de3e503cd6
Notification system (#13)
* Fixed typo

* Added proper exceptions for publishers

* Updated publishers logics: using messengers to allow publications without events

* Added new exception 'InvalidSettings'

* Updated notifiers and publishers: retrieve credentials and destination from config, use only message/event in constructors, lesser refactoring

* Updated publishers' tests

* Updated notifiers and publishers config for Telegram service, updated settings.toml file

* Updated 'post()' method and removed useless config checks for Telegram bot

* Added jinja2 template management for message formatting methods

* Manage message validation in publishers' run() method

* Better config management for notifiers and publishers

* Lesser update to logging and error-raising management (#23)
2021-07-05 21:44:11 +02:00