FreshRSS/rssguard
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

preliminary implementation of #410

Browse Source
This commit is contained in:
Martin Rotter 2021-05-11 08:08:00 +02:00
parent adec28ae8e
commit ef1b32bb52
8 changed files with 112 additions and 69 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
resources/desktop/com.github.rssguard.appdata.xml
View File

@ -30,7 +30,7 @@
<url type="donation">https://martinrotter.github.io/donate/</url>
<content_rating type="oars-1.1" />
<releases>
<release version="3.9.2" date="2021-05-10"/>
<release version="3.9.2" date="2021-05-11"/>
</releases>
<content_rating type="oars-1.0">
<content_attribute id="violence-cartoon">none</content_attribute>

34
resources/docs/Documentation.md
View File

@ -7,8 +7,8 @@
* [Supported feed formats and online feed services](Feed-formats.md)
* [Message filtering](Message-filters.md)
* [Database backends](#database-backends)
* [Google Reader API](#google-reader-api)
* [Websites scraping](#websites-scraping)
* [Google Reader API](#google-reader-api)
* [Gmail](#gmail)
* [Feedly](#feedly)
* [Labels](Labels.md)
@ -109,19 +109,6 @@ MariaDB (MySQL) backend is there for users, who want to store their data in a ce
For database-related configuration see `Settings -> Data storage` dialog.
## Google Reader API
There is a plugin which offers synchronization with services using Google Reader API. Plugin was so far tested with FreshRSS, Reedah, The Old Reader and Bazqux. All Google Reader API enabled services should work.
Note that Inoreader has its own separate plugin, because it uses OAuth as authentication method.
Google Reader API integration in RSS Guard offers a way to set custom service endpoint even if you select service which is not self-hosted such as Bazqux, providing all users with greater flexibility and freedom.
<img src="images/greader-api-settings.png">
Note that even when all Google Reader API enabled services should follow the API, there are still some minor differences, primarily because Google Reader API has no strict documentation to follow and some services do not offer some features.
For example The Old Reader does not seem to offer tags/labels functionality, therefore tags/labels in RSS Guard are not synchronized, but you can still use offline labels.
## Websites scraping
> **Only proceed if you consider yourself to be a power user and you know what you are doing!**
@ -169,6 +156,19 @@ Typical post-processing filter might do things like advanced CSS formatting, loc
It's completely up to you if you decide to only use script as `Source` of the script or separate your custom functionality between `Source` script and `Post-process` script. Sometimes you might need different `Source` scripts for different online sources and the same `Post-process` script and vice versa.
## Google Reader API
There is a plugin which offers synchronization with services using Google Reader API. Plugin was so far tested with FreshRSS, Reedah, The Old Reader and Bazqux. All Google Reader API enabled services should work.
Note that Inoreader has its own separate plugin, because it uses OAuth as authentication method.
Google Reader API integration in RSS Guard offers a way to set custom service endpoint even if you select service which is not self-hosted such as Bazqux, providing all users with greater flexibility and freedom.
<img src="images/greader-api-settings.png">
Note that even when all Google Reader API enabled services should follow the API, there are still some minor differences, primarily because Google Reader API has no strict documentation to follow and some services do not offer some features.
For example The Old Reader does not seem to offer tags/labels functionality, therefore tags/labels in RSS Guard are not synchronized, but you can still use offline labels.
## Gmail
RSS Guard includes Gmail plugin, which allows users to receive and send e-mail messages in a very simple fashion. Plugin uses [Gmail API](https://developers.google.com/gmail/api) and offers some e-mail client-like features:
* Sending e-mail messages.
@ -213,6 +213,12 @@ RSS Guard allows you to define a set of custom tools which you can subsequently
You need to have have [Node.js](https://nodejs.org) with [NPM](https://www.npmjs.com) (which is usually included in Node.js installer) installed to have ad-blocking in RSS Guard working. Also, the implementation requires additional [npm](https://www.npmjs.com) modules to be installed. You see that list of needed modules near the top of [this](https://github.com/martinrotter/rssguard/blob/master/resources/scripts/adblock/adblock-server.js) file.
I understand that the above installation of needed dependencies is not trivial, but it is necessary evil to have up-to-date and modern implementation of AdBlock in RSS Guard. Previous `C++`-based implementation was buggy, quite slow and hard to maintain.
You can find elaborated lists of AdBlock rules [here](https://easylist.to). You can just copy direct hyperlinks to those lists and paste them into "Filter lists" textbox as seen below. Remember to always separate individual links with newlines. Same applies to "Custom filters" where you can insert individual filters, for example [filter](https://adblockplus.org/filter-cheatsheet) "idnes" to block all URLs with "idnes" in them.
<img src="images/adblock.png" width="90%">
The way ad-blocking internally works is that RSS Guard starts local HTTP browser which provides ad-blocking API, which is subsequently called by RSS Guard. There is some caching done in between, which speeds up some ad-blocking decisions.
## GUI tweaking

BIN
resources/docs/images/adblock.png Executable file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

2
resources/scripts/7za

@ -1 +1 @@
Subproject commit 9c10723bfbaf6cb85107d6ee16e0324e9e487749
Subproject commit 47f4125753452eff8800dbd6600c5a05540b15d9

33
src/librssguard/network-web/adblock/adblockdialog.cpp
View File

@ -7,6 +7,7 @@
#include "definitions/definitions.h"
#include "exceptions/applicationexception.h"
#include "gui/guiutilities.h"
#include "gui/messagebox.h"
#include "miscellaneous/application.h"
#include "miscellaneous/iconfactory.h"
#include "network-web/webfactory.h"
@ -32,6 +33,13 @@ AdBlockDialog::AdBlockDialog(QWidget* parent)
connect(m_ui.m_cbEnable, &QCheckBox::toggled, this, &AdBlockDialog::enableAdBlock);
connect(m_ui.m_buttonBox, &QDialogButtonBox::rejected, this, &AdBlockDialog::saveAndClose);
m_ui.m_lblTestResult->label()->setWordWrap(true);
m_ui.m_btnHelp->setIcon(qApp->icons()->fromTheme(QSL("help-about")));
m_ui.m_btnTest->setIcon(qApp->icons()->fromTheme(QSL("media-playback-start")));
m_ui.m_lblTestResult->setStatus(WidgetWithStatus::StatusType::Information,
tr("No test executed yet."),
tr("No test executed yet."));
load();
m_ui.m_buttonBox->setFocus();
}
@ -39,7 +47,23 @@ AdBlockDialog::AdBlockDialog(QWidget* parent)
void AdBlockDialog::saveAndClose() {
m_manager->setFilterLists(m_ui.m_txtPredefined->toPlainText().split(QSL("\n")));
m_manager->setCustomFilters(m_ui.m_txtCustom->toPlainText().split(QSL("\n")));
m_manager->updateUnifiedFiltersFile();
try {
m_manager->updateUnifiedFiltersFile();
}
catch (const ApplicationException& ex) {
qCriticalNN << LOGSEC_ADBLOCK
<< "Failed to write unified filters to file or re-start server, error:"
<< QUOTE_W_SPACE_DOT(ex.message());
MessageBox::show(this,
QMessageBox::Icon::Critical,
tr("Cannot enable AdBlock"),
tr("There is some error in AdBlock component and it cannot be enabled. "
"Check error message below (or application debug log) for more information."),
{},
ex.message());
}
close();
}
@ -54,7 +78,9 @@ void AdBlockDialog::enableAdBlock(bool enable) {
void AdBlockDialog::testConfiguration() {
try {
m_manager->testConfiguration();
m_manager->setFilterLists(m_ui.m_txtPredefined->toPlainText().split(QSL("\n")));
m_manager->setCustomFilters(m_ui.m_txtCustom->toPlainText().split(QSL("\n")));
m_manager->updateUnifiedFiltersFile();
m_ui.m_lblTestResult->setStatus(WidgetWithStatus::StatusType::Ok, tr("You are good to go."), tr("OK!"));
}
catch (const ApplicationException& ex) {
@ -63,7 +89,8 @@ void AdBlockDialog::testConfiguration() {
<< QUOTE_W_SPACE_DOT(ex.message());
m_ui.m_lblTestResult->setStatus(WidgetWithStatus::StatusType::Error,
tr("There is error, check application log for more details and "
"head to online documentation."), tr("ERROR!"));
"head to online documentation.\n\nError: %1").arg(ex.message()),
tr("ERROR!"));
}
}

10
src/librssguard/network-web/adblock/adblockdialog.ui
View File

@ -58,11 +58,11 @@
<item row="3" column="0" colspan="2">
<widget class="QTabWidget" name="m_tcSubscriptions">
<property name="currentIndex">
<number>1</number>
<number>0</number>
</property>
<widget class="QWidget" name="m_tabPredefinedLists">
<attribute name="title">
<string>Filter lists (list per line)</string>
<string>Filter lists</string>
</attribute>
<layout class="QVBoxLayout" name="verticalLayout_3">
<item>
@ -113,7 +113,11 @@
</widget>
</item>
<item row="2" column="0" colspan="2">
<widget class="LabelWithStatus" name="m_lblTestResult" native="true"/>
<widget class="LabelWithStatus" name="m_lblTestResult" native="true">
<property name="layoutDirection">
<enum>Qt::RightToLeft</enum>
</property>
</widget>
</item>
</layout>
</widget>

95
src/librssguard/network-web/adblock/adblockmanager.cpp
View File

@ -25,15 +25,15 @@
#include <QWebEngineProfile>
AdBlockManager::AdBlockManager(QObject* parent)
: QObject(parent), m_loaded(false), m_enabled(false), m_interceptor(new AdBlockUrlInterceptor(this)) {
: QObject(parent), m_loaded(false), m_enabled(false), m_interceptor(new AdBlockUrlInterceptor(this)),
m_serverProcess(nullptr) {
m_adblockIcon = new AdBlockIcon(this);
m_adblockIcon->setObjectName(QSL("m_adblockIconAction"));
m_unifiedFiltersFile = qApp->userDataFolder() + QDir::separator() + QSL("adblock-unified-filters.txt");
m_serverProcess = new QProcess(this);
}
AdBlockManager::~AdBlockManager() {
if (m_serverProcess->state() == QProcess::ProcessState::Running) {
if (m_serverProcess != nullptr && m_serverProcess->state() == QProcess::ProcessState::Running) {
m_serverProcess->kill();
}
}
@ -50,7 +50,7 @@ BlockingResult AdBlockManager::block(const AdblockRequestInfo& request) const {
return { false };
}
else {
if (m_serverProcess->state() == QProcess::ProcessState::Running) {
if (m_serverProcess != nullptr && m_serverProcess->state() == QProcess::ProcessState::Running) {
try {
auto result = askServerIfBlocked(url_string);
@ -93,7 +93,23 @@ void AdBlockManager::load(bool initial_load) {
}
if (m_enabled) {
updateUnifiedFiltersFile();
try {
updateUnifiedFiltersFile();
}
catch (const ApplicationException& ex) {
qCriticalNN << LOGSEC_ADBLOCK
<< "Failed to write unified filters to file or re-start server, error:"
<< QUOTE_W_SPACE_DOT(ex.message());
qApp->showGuiMessage(tr("AdBlock needs to be configured"),
tr("AdBlock component is not configured properly."),
QSystemTrayIcon::MessageIcon::Warning,
nullptr,
true,
[=]() {
showDialog();
});
}
}
}
@ -105,12 +121,8 @@ bool AdBlockManager::canRunOnScheme(const QString& scheme) const {
return !(scheme == QSL("file") || scheme == QSL("qrc") || scheme == QSL("data") || scheme == QSL("abp"));
}
void AdBlockManager::testConfiguration() {
// Just try to run testing JS program to see if all dependecies are installed.
}
QString AdBlockManager::elementHidingRulesForDomain(const QUrl& url) const {
if (m_serverProcess->state() == QProcess::ProcessState::Running) {
if (m_serverProcess != nullptr && m_serverProcess->state() == QProcess::ProcessState::Running) {
try {
auto result = askServerForCosmeticRules(url.toString());
@ -241,16 +253,7 @@ QString AdBlockManager::askServerForCosmeticRules(const QString& url) const {
}
}
void AdBlockManager::restartServer(int port) {
if (m_serverProcess->state() == QProcess::ProcessState::Running) {
m_serverProcess->kill();
if (!m_serverProcess->waitForFinished(1000)) {
m_serverProcess->deleteLater();
m_serverProcess = new QProcess(this);
}
}
QProcess* AdBlockManager::restartServer(int port) {
QString temp_server = QDir::toNativeSeparators(IOFactory::getSystemFolder(QStandardPaths::StandardLocation::TempLocation)) +
QDir::separator() +
QSL("adblock-server.js");
@ -259,21 +262,23 @@ void AdBlockManager::restartServer(int port) {
qWarningNN << LOGSEC_ADBLOCK << "Failed to copy server file to TEMP.";
}
QProcess* proc = new QProcess(this);
#if defined(Q_OS_WIN)
m_serverProcess->setProgram(QSL("node.exe"));
proc->setProgram(QSL("node.exe"));
#else
m_serverProcess->setProgram(QSL("node"));
proc->setProgram(QSL("node"));
#endif
m_serverProcess->setArguments({
proc->setArguments({
QDir::toNativeSeparators(temp_server),
QString::number(port),
QDir::toNativeSeparators(m_unifiedFiltersFile)
});
m_serverProcess->setProcessEnvironment(QProcessEnvironment::systemEnvironment());
proc->setProcessEnvironment(QProcessEnvironment::systemEnvironment());
auto pe = m_serverProcess->processEnvironment();
auto pe = proc->processEnvironment();
QString node_path =
#if defined(Q_OS_WIN)
pe.value(QSL("APPDATA")) +
@ -291,14 +296,18 @@ void AdBlockManager::restartServer(int port) {
pe.insert(QSL("NODE_PATH"), node_path);
}
m_serverProcess->setProcessEnvironment(pe);
m_serverProcess->setProcessChannelMode(QProcess::ProcessChannelMode::ForwardedErrorChannel);
proc->setProcessEnvironment(pe);
proc->setProcessChannelMode(QProcess::ProcessChannelMode::ForwardedErrorChannel);
if (!m_serverProcess->open()) {
qWarningNN << LOGSEC_ADBLOCK << "Failed to start server.";
if (!proc->open()) {
auto ers = proc->errorString();
proc->deleteLater();
throw ApplicationException(ers);
}
else {
qDebugNN << LOGSEC_ADBLOCK << "Started server.";
return proc;
}
}
@ -312,6 +321,10 @@ void AdBlockManager::updateUnifiedFiltersFile() {
// Download filters one by one and append.
for (const QString& filter_list_url : qAsConst(filter_lists)) {
if (filter_list_url.simplified().isEmpty()) {
continue;
}
QByteArray out;
auto res = NetworkFactory::performNetworkOperation(filter_list_url,
2000,
@ -328,11 +341,7 @@ void AdBlockManager::updateUnifiedFiltersFile() {
<< QUOTE_W_SPACE_DOT(filter_list_url);
}
else {
qWarningNN << LOGSEC_ADBLOCK
<< "Failed to download list of filters"
<< QUOTE_W_SPACE(filter_list_url)
<< "with error"
<< QUOTE_W_SPACE_DOT(res.first);
throw NetworkException(res.first, tr("failed to download filter list '%1'").arg(filter_list_url));
}
}
@ -343,16 +352,16 @@ void AdBlockManager::updateUnifiedFiltersFile() {
QDir::separator() +
QSL("adblock.filters");
try {
IOFactory::writeFile(m_unifiedFiltersFile, unified_contents.toUtf8());
IOFactory::writeFile(m_unifiedFiltersFile, unified_contents.toUtf8());
if (m_enabled) {
restartServer(ADBLOCK_SERVER_PORT);
if (m_enabled) {
if (m_serverProcess != nullptr && m_serverProcess->state() == QProcess::ProcessState::Running) {
m_serverProcess->kill();
m_serverProcess->waitForFinished(1000);
m_serverProcess->deleteLater();
m_serverProcess = nullptr;
}
}
catch (const ApplicationException& ex) {
qCriticalNN << LOGSEC_ADBLOCK
<< "Failed to write unified filters to file, error:"
<< QUOTE_W_SPACE_DOT(ex.message());
m_serverProcess = restartServer(ADBLOCK_SERVER_PORT);
}
}

5
src/librssguard/network-web/adblock/adblockmanager.h
View File

@ -37,8 +37,6 @@ class AdBlockManager : public QObject {
bool canRunOnScheme(const QString& scheme) const;
AdBlockIcon* adBlockIcon() const;
void testConfiguration();
// General methods for adblocking.
BlockingResult block(const AdblockRequestInfo& request) const;
QString elementHidingRulesForDomain(const QUrl& url) const;
@ -62,8 +60,7 @@ class AdBlockManager : public QObject {
private:
BlockingResult askServerIfBlocked(const QString& url) const;
QString askServerForCosmeticRules(const QString& url) const;
void restartServer(int port);
QProcess* restartServer(int port);
private:
bool m_loaded;