scraper

resources/docs/Documentation.md

@@ -12,6 +12,7 @@
     * [Gmail](#gmail)
     * [Feedly](#feedly)
     * [Labels](Labels.md)
+    * [Notifications](#notifications)
     * [Downloading files](#downloading-files)
     * [External tools](#external-tools)
     * [AdBlock](#adblock)
@@ -140,7 +141,7 @@ Interpreter must be provided in all cases, arguments do not have to be. For exam
 
 Note that the above examples are cross-platform and you can use the exact same command on Windows, Linux or Mac OS X, if your operating system is properly configured.
 
-RSS Guard offers [placeholder](#data-placeholder) `%data%` which is automatically replaced with full path to RSS Guard's [user data folder](Documentation.md#portable-user-data), allowing you to make your configuration fully portable. You can, therefore, use something like this as source script line: `bash#%data%/scripts/download-feed.sh`.
+RSS Guard offers [placeholder](#data-placeholder) `%data%` which is automatically replaced with full path to RSS Guard's [user data folder](#portable-user-data), allowing you to make your configuration fully portable. You can, therefore, use something like this as source script line: `bash#%data%/scripts/download-feed.sh`.
 
 Also, working directory of process executing the script is set to RSS Guard's user data folder.
 
@@ -191,6 +192,15 @@ There are two big downsides of using `developer access token`:
 * It expires after one month and must be manually renewed.
 * It allows maximum of 250 API calls per day.
 
+## Notifications
+RSS Guard allows you to configure behavior of desktop notifications. There is a number of events to be configured:
+* new messages downloaded,
+* downloading of messages started,
+* login OAuth tokens refreshed,
+* ...
+
+Your notification can also play `.wav` sounds which you can place under your [user data folder](#portable-user-data) and use them via special [placeholder](#data-placeholder). Other audio formats are not supported.
+
 ## Downloading files
 RSS Guard offers simple embedded file downloader.
 
@@ -328,6 +338,7 @@ command.
 RSS Guard stores its data and settings in single folder. What exact folder it is is described [here](#portable-user-data). RSS Guard allows you to use the folder programmatically in some special contexts via `%data%` placeholder. You can use this placeholder in these RSS Guard contexts:
 * Contents of your [message filters](Message-filters.md) - you can therefore place some scripts under your user data folder and include it via `JavaScript` into your message filter.
 * `source` and `post-process script` attributes of for [scraping](#websites-scraping) feed - you can use the placeholder to load scripts to generate/process feed from user data folder.
+* Notifications also support the placeholder in path to audio files which are to be played when some event happens. For example you could place audio files in your data folder and then use them in notification with `%data%\audio\new-messages.wav`. See more about notifications [here](#notifications).
 
 ## Cleaning database
 Your RSS Guard's database can grow really big over time, therefore you might need to do its cleanup regularly. There is a dialog `Cleanup database` in `Tools` menu to do just that for you, but note that RSS Guard should run just fine even with tens of thousands of messages.
@@ -347,6 +358,8 @@ This is _fully-portable mode_. Check `About RSS Guard -> Resources` dialog to fi
 
 RSS Guard on Linux, Android or Mac OS automatically uses non-portable user data location, so that it is in line with [XDG](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) standard.
 
+You can use your base user data folder on some places in RSS Guard via special [placeholder](#data-placeholder).
+
 ## Downloading new messages
 Here is the rough workflow which is performed when you hit `Feeds & categories -> Update all items` or `Feeds & categories -> Update selected items`. At that point of time this happens:
 1. RSS Guard creates a list of all/selected feeds.

resources/scripts/scrapers/wiki-qotd.py

@@ -0,0 +1,39 @@
+# Downloads all quotes of the day.
+
+import urllib.request
+import json
+from bs4 import BeautifulSoup
+
+url = "https://en.wikiquote.org/wiki/Wikiquote:Quote_of_the_day/Complete_list"
+response = urllib.request.urlopen(url)
+text =  response.read().decode("utf-8")
+
+soup = BeautifulSoup(text, 'html.parser')
+lists = soup.find_all("ul")
+items = list()
+json_feed = "{{\"title\": {title}, \"items\": [{items}]}}"
+
+for lst in lists:
+  try:
+    last_link = lst.find_all("a")[-1]
+    quote_link = last_link.get("href")
+
+    if quote_link.startswith("/"):
+      quote_link = "https://en.wikiquote.org" + quote_link
+
+    quote_author = last_link.get_text()
+    quote_text = lst.find("li").decode_contents()
+    quote_heading = (quote_text[:75] + '...') if len(quote_text) > 75 else quote_text
+    quote_text = "<span>" + quote_text + "</span>"
+
+    items.append("{{\"title\": {title}, \"authors\": [{{\"name\": {author}}}], \"content_html\": {html}, \"url\": {url}, \"date_published\": {date}}}".format(
+      title = json.dumps(quote_heading),
+      html = json.dumps(quote_text),
+      url = json.dumps(quote_link),
+      author = json.dumps(quote_author),
+      date = json.dumps("2020-12-31T08:00:00")))
+  except:
+    continue
+
+json_feed = json_feed.format(title = json.dumps(soup.title.text), items = ", ".join(items))
+print(json_feed)

src/librssguard/gui/dialogs/formaddaccount.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>405</width>
-    <height>354</height>
+    <height>409</height>
    </rect>
   </property>
   <property name="windowTitle">

src/librssguard/librssguard.pro

@@ -120,6 +120,8 @@ HEADERS += core/feeddownloader.h \
            miscellaneous/iofactory.h \
            miscellaneous/localization.h \
            miscellaneous/mutex.h \
+           miscellaneous/notification.h \
+           miscellaneous/notificationfactory.h \
            miscellaneous/regexfactory.h \
            miscellaneous/settings.h \
            miscellaneous/settingsproperties.h \
@@ -298,6 +300,8 @@ SOURCES += core/feeddownloader.cpp \
            miscellaneous/iofactory.cpp \
            miscellaneous/localization.cpp \
            miscellaneous/mutex.cpp \
+           miscellaneous/notification.cpp \
+           miscellaneous/notificationfactory.cpp \
            miscellaneous/regexfactory.cpp \
            miscellaneous/settings.cpp \
            miscellaneous/skinfactory.cpp \

src/librssguard/miscellaneous/application.cpp

@@ -15,6 +15,7 @@
 #include "miscellaneous/iconfactory.h"
 #include "miscellaneous/iofactory.h"
 #include "miscellaneous/mutex.h"
+#include "miscellaneous/notificationfactory.h"
 #include "network-web/webfactory.h"
 #include "services/abstract/serviceroot.h"
 #include "services/owncloud/owncloudserviceentrypoint.h"
@@ -54,6 +55,7 @@ Application::Application(const QString& id, int& argc, char** argv)
   m_icons = new IconFactory(this);
   m_database = new DatabaseFactory(this);
   m_downloadManager = nullptr;
+  m_notifications = new NotificationFactory(this);
   m_shouldRestart = false;
 
   determineFirstRuns();

src/librssguard/miscellaneous/application.h

@@ -35,6 +35,7 @@ class QAction;
 class Mutex;
 class QWebEngineDownloadItem;
 class WebFactory;
+class NotificationFactory;
 
 class RSSGUARD_DLLSPEC Application : public QtSingleApplication {
   Q_OBJECT
@@ -175,6 +176,7 @@ class RSSGUARD_DLLSPEC Application : public QtSingleApplication {
     IconFactory* m_icons;
     DatabaseFactory* m_database;
     DownloadManager* m_downloadManager;
+    NotificationFactory* m_notifications;
     bool m_shouldRestart;
     bool m_firstRunEver;
     bool m_firstRunCurrentVersion;

src/librssguard/miscellaneous/notification.cpp

@@ -0,0 +1,21 @@
+// For license of this file, see <project-root-folder>/LICENSE.md.
+
+#include "miscellaneous/notification.h"
+
+Notification::Notification() {}
+
+Notification::Event Notification::event() const {
+  return m_event;
+}
+
+void Notification::setEvent(const Event& event) {
+  m_event = event;
+}
+
+QString Notification::soundPath() const {
+  return m_soundPath;
+}
+
+void Notification::setSoundPath(const QString& sound_path) {
+  m_soundPath = sound_path;
+}

src/librssguard/miscellaneous/notification.h

@@ -0,0 +1,39 @@
+// For license of this file, see <project-root-folder>/LICENSE.md.
+
+#ifndef NOTIFICATION_H
+#define NOTIFICATION_H
+
+#include <QString>
+
+class Notification {
+  public:
+    enum class Event {
+      // New (unread) messages were downloaded for some feed.
+      NewMessagesDownloaded = 1,
+
+      // RSS Guard started downloading messages for some feed.
+      MesssagesDownloadStarted = 2,
+
+      // Login tokens were successfuly refreshed.
+      // NOTE: This is primarily used in accounts which use
+      // OAuth or similar mechanism.
+      LoginDataRefreshed = 4
+    };
+
+    explicit Notification();
+
+    Event event() const;
+    void setEvent(const Event& event);
+
+    // Returns full path to audio file which should be played when notification
+    // is launched.
+    // NOTE: This property supports "%data%" placeholder.
+    QString soundPath() const;
+    void setSoundPath(const QString& sound_path);
+
+  private:
+    Event m_event;
+    QString m_soundPath;
+};
+
+#endif // NOTIFICATION_H

src/librssguard/miscellaneous/notificationfactory.cpp

@@ -0,0 +1,27 @@
+// For license of this file, see <project-root-folder>/LICENSE.md.
+
+#include "miscellaneous/notificationfactory.h"
+
+#include "3rd-party/boolinq/boolinq.h"
+#include "definitions/definitions.h"
+#include "exceptions/applicationexception.h"
+#include "miscellaneous/settings.h"
+
+NotificationFactory::NotificationFactory(QObject* parent) : QObject(parent) {}
+
+Notification NotificationFactory::notificationForEvent(Notification::Event event) const {
+  auto good_n = boolinq::from(m_notifications).where([event](const Notification& n) {
+    return n.event() == event;
+  });
+
+  if (good_n.count() <= 0) {
+    throw ApplicationException(QSL("notification for event %1 was not found").arg(QString::number(int(event))));
+  }
+  else {
+    return good_n.first();
+  }
+}
+
+void NotificationFactory::load(Settings* settings) {}
+
+void NotificationFactory::save(const QList<Notification> new_notifications, Settings* settings) {}

src/librssguard/miscellaneous/notificationfactory.h

@@ -0,0 +1,31 @@
+// For license of this file, see <project-root-folder>/LICENSE.md.
+
+#ifndef NOTIFICATIONFACTORY_H
+#define NOTIFICATIONFACTORY_H
+
+#include <QObject>
+
+#include "miscellaneous/notification.h"
+
+class Settings;
+
+class NotificationFactory : public QObject {
+  Q_OBJECT
+
+  public:
+    explicit NotificationFactory(QObject* parent = nullptr);
+
+    Notification notificationForEvent(Notification::Event event) const;
+
+  public slots:
+
+    // Load saved notifications from settings
+    void load(Settings* settings);
+    void save(const QList<Notification> new_notifications, Settings* settings);
+
+  private:
+    QList<Notification> m_notifications = {};
+
+};
+
+#endif // NOTIFICATIONFACTORY_H

src/librssguard/services/abstract/gui/formaccountdetails.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>500</width>
-    <height>450</height>
+    <height>515</height>
    </rect>
   </property>
   <layout class="QFormLayout" name="formLayout_2">