FreshRSS/rssguard
1
0
Fork 0
mirror of https://github.com/martinrotter/rssguard.git synced 2025-01-31 17:44:52 +01:00
Code Issues Packages Projects Releases Wiki Activity

Fixed #265.

Browse Source
This commit is contained in:
Martin Rotter 2021-02-03 10:41:59 +01:00
parent c53b4bb970
commit a94c0167f4
14 changed files with 491 additions and 298 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
resources/desktop/com.github.rssguard.appdata.xml
View File

@ -30,7 +30,7 @@
<url type="donation">https://martinrotter.github.io/donate/</url>
<content_rating type="oars-1.1" />
<releases>
<release version="3.8.4" date="2021-02-02"/>
<release version="3.8.4" date="2021-02-03"/>
</releases>
<content_rating type="oars-1.0">
<content_attribute id="violence-cartoon">none</content_attribute>

33
resources/docs/Feed-formats.md
View File

@ -4,17 +4,31 @@ RSS Guard is a modular application which supports plugins. It offers well-mainta
* [Tiny Tiny RSS](https://tt-rss.org) plugin: Adds ability to synchronize messages with TT-RSS instances, either self-hosted or via 3rd-party external service.
* [Inoreader](https://www.inoreader.com) plugin: Adds ability to synchronize messages with Inoreader. All you need to do is create free account on their website and start rocking.
* [Nextcloud News](https://apps.nextcloud.com/apps/news) plugin: Nextcloud News is a Nextcloud app which adds feed reader abilities into your Nextcloud instances.
* Google Reader API plugin: This plugin was added in RSS Guard 3.9.0 and offers two-way synchronization with services which implement Google Reader API. At this point, plugin was tested and works with Bazqux, The Old Reader and FreshRSS.
* [Google Reader API](https://rss-sync.github.io/Open-Reader-API/resources/#unofficial-google-reader-documentation) plugin: This plugin was added in RSS Guard 3.9.0 and offers two-way synchronization with services which implement Google Reader API. At this point, plugin was tested and works with Bazqux, The Old Reader and FreshRSS.
* [Gmail](https://www.google.com/gmail) plugin: Yes, you are reading it right. RSS Guard can be used as very lightweight and simple e-mail client. This plugins uses [Gmail API](https://developers.google.com/gmail/api) and offers even e-mail sending.
All plugins share almost all core RSS Guard's features, including labels, recycle bins, podcasts fetching or newspaper view. They are implemented in a very transparent way, making it easy to maintain them or add new ones.
> All plugins share almost all core RSS Guard's features, including labels, recycle bins, podcasts fetching or newspaper view. They are implemented in a very transparent way, making it easy to maintain them or add new ones.
Usually, plugins have some exclusive functionality, for example Gmail plugin allows user to send e-mail messages. This extra functionality is always accessible via plugin's context menu and also via main menu.
Usually, plugins have some exclusive functionality, for example Gmail plugin allows user to send e-mail messages or reply to existing ones. This extra functionality is always accessible via plugin's context menu and also via main menu.
<img src="images/gmail-context-menu.png" width="80%">
If there is interest in other plugins, you might write one yourself or if many people are interested then I might write it for you, even commercially if we make proper arrangements.
## Plugin API
RSS Guard offers simple `C++` API for creating new service plugins. All base API classes are in folder [`abstract`](https://github.com/martinrotter/rssguard/tree/master/src/librssguard/services/abstract). User must subclass and implement all interface classes:
| Class | Purpose |
|-------|---------|
| `ServiceEntryPoint` | Very base class which provides basic information about the plugin name, author etc. It also provides methods which are called when new account should be created and when existing accounts should be loaded from database. |
| `ServiceRoot` | This is the core "account" class which represents account node in feed's list and offers interface for all critical functionality of a plugin, including handlers which are called when starting/stoping a plugin, marking messages read/unread/starred/deleted, (de)assigning labels etc. |
API is reasonably simple to understand but relatively large. Sane default behavior is employed where it makes sense.
Perhaps the best approach to use when writing new plugin is to copy [existing](https://github.com/martinrotter/rssguard/tree/master/src/librssguard/services/greader) one and start from there.
Note that RSS Guard can support loading of plugins from external libraries (dll, so, etc.) but the functionality must be polished because so far all plugins are directly bundled into the app as no one really requested run-time loading of plugins so far.
## Features found exclusively in `standard RSS` plugin
Standard plugin in RSS Guard offers some features which are specific to it. Of course it supports all news syndication formats which are nowadays used:
* RSS 0.90, 0.91, 0.92, 1.0 (also known as *RDF*), 2.0.
@ -30,10 +44,11 @@ OPML files can be exported/imported in simple dialog.
You just select output file (in case of OPML export), check desired feeds and hit `Export to file`.
### Websites scraping and other related advanced features
RSS Guard 3.9.0+ offers extra advanced features which were inspired by [Liferea](https://lzone.de/liferea/).
### Websites scraping and related advanced features
**Only proceed if you consider yourself as power user and you know you are doing!**
> **Only proceed if you consider yourself as power user and you know you are doing!**
RSS Guard 3.9.0+ offers extra advanced features which are inspired by [Liferea](https://lzone.de/liferea/).
You can select source type of each feed. If you select `URL`, then RSS Guard simply downloads feed file from given location.
@ -43,13 +58,15 @@ However, if you choose `Script` option, then you cannot provide URL of your feed
Any errors in your script must be written to **error output**.
Note that you must provide full execution line to your custom script, including interpreter binary path and name and all that must be written in special format `<interpreter>#<arguments>`. The `#` character is there to separate interpreter from its arguments. Interpreter must be provided in all cases, arguments do not have to be. For example `bash.exe#` is valid execution line, as well as `bash#-C "cat feed.atom"`. Some examples of valid and tested execution lines are:
Note that you must provide full execution line to your custom script, including interpreter binary path and name and all that must be written in special format `<interpreter>#<arguments>`. The `#` character is there to separate interpreter from its arguments.
Interpreter must be provided in all cases, arguments do not have to be. For example `bash.exe#` is valid execution line, as well as `bash#-C "cat feed.atom"`. Note the difference in interpreter's binary name suffix. Some examples of valid and tested execution lines are:
| Command | Explanation |
|---------|-------------|
| `bash#-c "curl 'https://github.com/martinrotter.atom'"` | Downloads ATOM feed file with Bash and Curl. |
| `Powershell#"Invoke-WebRequest 'https://github.com/martinrotter.atom' | Select-Object -ExpandProperty Content"` | Downloads ATOM feed file with Powershell. |
| `php#tweeper.php https://twitter.com/NSACareers` | Downloads RSS feed file with [Tweeper](https://git.ao2.it/tweeper.git/). Tweeper is utility which is able to produce RSS feed from Twitter. |
| `php#tweeper.php -v 0 https://twitter.com/NSACareers` | Downloads RSS feed file with [Tweeper](https://git.ao2.it/tweeper.git/). Tweeper is utility which is able to produce RSS feed from Twitter. |
<img src="images/scrape-source.png" width="50%">

2
resources/scripts/7za

@ -1 +1 @@
Subproject commit 9c10723bfbaf6cb85107d6ee16e0324e9e487749
Subproject commit 47f4125753452eff8800dbd6600c5a05540b15d9

4
src/librssguard/exceptions/applicationexception.cpp
View File

@ -7,3 +7,7 @@ ApplicationException::ApplicationException(QString message) : m_message(std::mov
QString ApplicationException::message() const {
return m_message;
}
void ApplicationException::setMessage(const QString& message) {
m_message = message;
}

3
src/librssguard/exceptions/applicationexception.h
View File

@ -11,6 +11,9 @@ class ApplicationException {
QString message() const;
protected:
void setMessage(const QString& message);
private:
QString m_message;
};

39
src/librssguard/exceptions/scriptexception.cpp Executable file
View File

@ -0,0 +1,39 @@
// For license of this file, see <project-root-folder>/LICENSE.md.
#include "exceptions/scriptexception.h"
#include "definitions/definitions.h"
ScriptException::ScriptException(Reason reason, QString message) : ApplicationException(message), m_reason(reason) {
if (message.isEmpty()) {
setMessage(messageForReason(reason));
}
else if (reason == ScriptException::Reason::InterpreterError ||
reason == ScriptException::Reason::OtherError) {
setMessage(messageForReason(reason) + QSL(": '%1'").arg(message));
}
}
ScriptException::Reason ScriptException::reason() const {
return m_reason;
}
QString ScriptException::messageForReason(ScriptException::Reason reason) const {
switch (reason) {
case ScriptException::Reason::ExecutionLineInvalid:
return tr("script line is not well-formed");
case ScriptException::Reason::InterpreterError:
return tr("script threw an error");
case ScriptException::Reason::InterpreterNotFound:
return tr("script's interpreter was not found");
case ScriptException::Reason::InterpreterTimeout:
return tr("script execution took too long");
case ScriptException::Reason::OtherError:
default:
return tr("unknown error");
}
}

33
src/librssguard/exceptions/scriptexception.h Executable file
View File

@ -0,0 +1,33 @@
// For license of this file, see <project-root-folder>/LICENSE.md.
#ifndef SCRIPTEXCEPTION_H
#define SCRIPTEXCEPTION_H
#include "exceptions/applicationexception.h"
#include <QCoreApplication>
class ScriptException : public ApplicationException {
Q_DECLARE_TR_FUNCTIONS(ScriptException)
public:
enum class Reason {
ExecutionLineInvalid,
InterpreterNotFound,
InterpreterError,
InterpreterTimeout,
OtherError
};
explicit ScriptException(Reason reason = Reason::OtherError, QString message = QString());
Reason reason() const;
private:
QString messageForReason(Reason reason) const;
private:
Reason m_reason;
};
#endif // SCRIPTEXCEPTION_H

2
src/librssguard/librssguard.pro
View File

@ -50,6 +50,7 @@ HEADERS += core/feeddownloader.h \
exceptions/applicationexception.h \
exceptions/filteringexception.h \
exceptions/ioexception.h \
exceptions/scriptexception.h \
gui/baselineedit.h \
gui/basetoolbar.h \
gui/colortoolbutton.h \
@ -221,6 +222,7 @@ SOURCES += core/feeddownloader.cpp \
exceptions/applicationexception.cpp \
exceptions/filteringexception.cpp \
exceptions/ioexception.cpp \
exceptions/scriptexception.cpp \
gui/baselineedit.cpp \
gui/basetoolbar.cpp \
gui/colortoolbutton.cpp \

8
src/librssguard/services/standard/gui/formstandardfeeddetails.cpp
View File

@ -47,13 +47,17 @@ int FormStandardFeedDetails::addEditFeed(StandardFeed* input_feed, RootItem* par
}
void FormStandardFeedDetails::guessFeed() {
m_standardFeedDetails->guessFeed(m_standardFeedDetails->m_ui.m_txtSource->lineEdit()->text(),
m_standardFeedDetails->guessFeed(m_standardFeedDetails->sourceType(),
m_standardFeedDetails->m_ui.m_txtSource->lineEdit()->text(),
m_standardFeedDetails->m_ui.m_txtPostProcessScript->lineEdit()->text(),
m_authDetails->m_txtUsername->lineEdit()->text(),
m_authDetails->m_txtPassword->lineEdit()->text());
}
void FormStandardFeedDetails::guessIconOnly() {
m_standardFeedDetails->guessIconOnly(m_standardFeedDetails->m_ui.m_txtSource->lineEdit()->text(),
m_standardFeedDetails->guessIconOnly(m_standardFeedDetails->sourceType(),
m_standardFeedDetails->m_ui.m_txtSource->lineEdit()->text(),
m_standardFeedDetails->m_ui.m_txtPostProcessScript->lineEdit()->text(),
m_authDetails->m_txtUsername->lineEdit()->text(),
m_authDetails->m_txtPassword->lineEdit()->text());
}

62
src/librssguard/services/standard/gui/standardfeeddetails.cpp
View File

@ -107,53 +107,63 @@ StandardFeedDetails::StandardFeedDetails(QWidget* parent) : QWidget(parent) {
onPostProcessScriptChanged({});
}
void StandardFeedDetails::guessIconOnly(const QString& url, const QString& username,
void StandardFeedDetails::guessIconOnly(StandardFeed::SourceType source_type, const QString& source,
const QString& post_process_script, const QString& username,
const QString& password, const QNetworkProxy& custom_proxy) {
QPair<StandardFeed*, QNetworkReply::NetworkError> result = StandardFeed::guessFeed(url,
username,
password,
custom_proxy);
bool result;
StandardFeed* metadata = StandardFeed::guessFeed(source_type,
source,
post_process_script,
&result,
username,
password,
custom_proxy);
if (result.first != nullptr) {
if (metadata != nullptr) {
// Icon or whole feed was guessed.
m_ui.m_btnIcon->setIcon(result.first->icon());
m_ui.m_btnIcon->setIcon(metadata->icon());
if (result.second == QNetworkReply::NoError) {
if (result) {
m_ui.m_lblFetchMetadata->setStatus(WidgetWithStatus::StatusType::Ok,
tr("Icon fetched successfully."),
tr("Icon metadata fetched."));
}
else {
m_ui.m_lblFetchMetadata->setStatus(WidgetWithStatus::StatusType::Warning,
tr("Result: %1.").arg(NetworkFactory::networkErrorText(result.second)),
tr("Icon metadata not fetched."),
tr("Icon metadata not fetched."));
}
// Remove temporary feed object.
delete result.first;
delete metadata;
}
else {
// No feed guessed, even no icon available.
m_ui.m_lblFetchMetadata->setStatus(WidgetWithStatus::StatusType::Error,
tr("Error: %1.").arg(NetworkFactory::networkErrorText(result.second)),
tr("No icon fetched."),
tr("No icon fetched."));
}
}
void StandardFeedDetails::guessFeed(const QString& url, const QString& username,
void StandardFeedDetails::guessFeed(StandardFeed::SourceType source_type, const QString& source,
const QString& post_process_script, const QString& username,
const QString& password, const QNetworkProxy& custom_proxy) {
QPair<StandardFeed*, QNetworkReply::NetworkError> result = StandardFeed::guessFeed(url,
username,
password,
custom_proxy);
bool result;
StandardFeed* metadata = StandardFeed::guessFeed(source_type,
source,
post_process_script,
&result,
username,
password,
custom_proxy);
if (result.first != nullptr) {
if (metadata != nullptr) {
// Icon or whole feed was guessed.
m_ui.m_btnIcon->setIcon(result.first->icon());
m_ui.m_txtTitle->lineEdit()->setText(result.first->title());
m_ui.m_txtDescription->lineEdit()->setText(result.first->description());
m_ui.m_cmbType->setCurrentIndex(m_ui.m_cmbType->findData(QVariant::fromValue((int) result.first->type())));
int encoding_index = m_ui.m_cmbEncoding->findText(result.first->encoding(), Qt::MatchFixedString);
m_ui.m_btnIcon->setIcon(metadata->icon());
m_ui.m_txtTitle->lineEdit()->setText(metadata->title());
m_ui.m_txtDescription->lineEdit()->setText(metadata->description());
m_ui.m_cmbType->setCurrentIndex(m_ui.m_cmbType->findData(QVariant::fromValue((int) metadata->type())));
int encoding_index = m_ui.m_cmbEncoding->findText(metadata->encoding(), Qt::MatchFlag::MatchFixedString);
if (encoding_index >= 0) {
m_ui.m_cmbEncoding->setCurrentIndex(encoding_index);
@ -162,24 +172,24 @@ void StandardFeedDetails::guessFeed(const QString& url, const QString& username,
m_ui.m_cmbEncoding->setCurrentIndex(m_ui.m_cmbEncoding->findText(DEFAULT_FEED_ENCODING, Qt::MatchFixedString));
}
if (result.second == QNetworkReply::NoError) {
if (result) {
m_ui.m_lblFetchMetadata->setStatus(WidgetWithStatus::StatusType::Ok,
tr("All metadata fetched successfully."),
tr("Feed and icon metadata fetched."));
}
else {
m_ui.m_lblFetchMetadata->setStatus(WidgetWithStatus::StatusType::Warning,
tr("Result: %1.").arg(NetworkFactory::networkErrorText(result.second)),
tr("Feed or icon metadata not fetched."),
tr("Feed or icon metadata not fetched."));
}
// Remove temporary feed object.
delete result.first;
delete metadata;
}
else {
// No feed guessed, even no icon available.
m_ui.m_lblFetchMetadata->setStatus(WidgetWithStatus::StatusType::Error,
tr("Error: %1.").arg(NetworkFactory::networkErrorText(result.second)),
tr("No metadata fetched."),
tr("No metadata fetched."));
}
}

8
src/librssguard/services/standard/gui/standardfeeddetails.h
View File

@ -23,11 +23,15 @@ class StandardFeedDetails : public QWidget {
explicit StandardFeedDetails(QWidget* parent = nullptr);
private slots:
void guessIconOnly(const QString& url,
void guessIconOnly(StandardFeed::SourceType source_type,
const QString& source,
const QString& post_process_script,
const QString& username,
const QString& password,
const QNetworkProxy& custom_proxy = QNetworkProxy::ProxyType::DefaultProxy);
void guessFeed(const QString& url,
void guessFeed(StandardFeed::SourceType source_type,
const QString& source,
const QString& post_process_script,
const QString& username,
const QString& password,
const QNetworkProxy& custom_proxy = QNetworkProxy::ProxyType::DefaultProxy);

549
src/librssguard/services/standard/standardfeed.cpp
View File

@ -5,6 +5,7 @@
#include "core/feedsmodel.h"
#include "definitions/definitions.h"
#include "exceptions/applicationexception.h"
#include "exceptions/scriptexception.h"
#include "gui/feedmessageviewer.h"
#include "gui/feedsview.h"
#include "miscellaneous/databasequeries.h"
@ -132,22 +133,28 @@ QString StandardFeed::sourceTypeToString(StandardFeed::SourceType type) {
}
void StandardFeed::fetchMetadataForItself() {
QPair<StandardFeed*, QNetworkReply::NetworkError> metadata = guessFeed(url(),
username(),
password(),
getParentServiceRoot()->networkProxy());
bool result;
StandardFeed* metadata = guessFeed(sourceType(),
url(),
postProcessScript(),
&result,
username(),
password(),
getParentServiceRoot()->networkProxy());
if (metadata.first != nullptr && metadata.second == QNetworkReply::NetworkError::NoError) {
if (metadata != nullptr && result) {
// Some properties are not updated when new metadata are fetched.
metadata.first->setParent(parent());
metadata.first->setUrl(url());
metadata.first->setPasswordProtected(passwordProtected());
metadata.first->setUsername(username());
metadata.first->setPassword(password());
metadata.first->setAutoUpdateType(autoUpdateType());
metadata.first->setAutoUpdateInitialInterval(autoUpdateInitialInterval());
editItself(metadata.first);
delete metadata.first;
metadata->setParent(parent());
metadata->setUrl(url());
metadata->setPasswordProtected(passwordProtected());
metadata->setUsername(username());
metadata->setPassword(password());
metadata->setAutoUpdateType(autoUpdateType());
metadata->setAutoUpdateInitialInterval(autoUpdateInitialInterval());
metadata->setPostProcessScript(postProcessScript());
metadata->setSourceType(sourceType());
editItself(metadata);
delete metadata;
// Notify the model about fact, that it needs to reload new information about
// this item, particularly the icon.
@ -155,8 +162,8 @@ void StandardFeed::fetchMetadataForItself() {
}
else {
qApp->showGuiMessage(tr("Metadata not fetched"),
tr("Metadata was not fetched because: %1.").arg(NetworkFactory::networkErrorText(metadata.second)),
QSystemTrayIcon::Critical);
tr("Metadata was not fetched."),
QSystemTrayIcon::MessageIcon::Critical);
}
}
@ -176,195 +183,253 @@ void StandardFeed::setSourceType(const SourceType& source_type) {
m_sourceType = source_type;
}
QPair<StandardFeed*, QNetworkReply::NetworkError> StandardFeed::guessFeed(const QString& url,
const QString& username,
const QString& password,
const QNetworkProxy& custom_proxy) {
QPair<StandardFeed*, QNetworkReply::NetworkError> result;
result.first = nullptr;
StandardFeed* StandardFeed::guessFeed(StandardFeed::SourceType source_type,
const QString& source,
const QString& post_process_script,
bool* result,
const QString& username,
const QString& password,
const QNetworkProxy& custom_proxy) {
auto timeout = qApp->settings()->value(GROUP(Feeds),
SETTING(Feeds::UpdateTimeout)).toInt();
QByteArray feed_contents;
QList<QPair<QByteArray, QByteArray>> headers;
QList<QString> icon_possible_locations;
QString content_type;
headers << NetworkFactory::generateBasicAuthHeader(username, password);
NetworkResult network_result = NetworkFactory::performNetworkOperation(url,
qApp->settings()->value(GROUP(Feeds),
SETTING(Feeds::UpdateTimeout)).toInt(),
QByteArray(),
feed_contents,
QNetworkAccessManager::Operation::GetOperation,
headers,
false,
{},
{},
custom_proxy);
if (source_type == StandardFeed::SourceType::Url) {
QList<QPair<QByteArray, QByteArray>> headers = { NetworkFactory::generateBasicAuthHeader(username, password) };
NetworkResult network_result = NetworkFactory::performNetworkOperation(source,
timeout,
QByteArray(),
feed_contents,
QNetworkAccessManager::Operation::GetOperation,
headers,
false,
{},
{},
custom_proxy);
result.second = network_result.first;
content_type = network_result.second.toString();
if (result.second == QNetworkReply::NoError || !feed_contents.isEmpty()) {
if (result.first == nullptr) {
result.first = new StandardFeed();
if (network_result.first != QNetworkReply::NetworkError::NoError) {
*result = false;
return nullptr;
}
QList<QString> icon_possible_locations;
icon_possible_locations.append(source);
}
else {
qDebugNN << LOGSEC_CORE
<< "Running custom script for guessing"
<< QUOTE_W_SPACE(source)
<< "to obtain feed data.";
icon_possible_locations.append(url);
if (network_result.second.toString().contains(QSL("json"), Qt::CaseSensitivity::CaseInsensitive)) {
// We have JSON feed.
result.first->setEncoding(DEFAULT_FEED_ENCODING);
result.first->setType(Type::Json);
QJsonDocument json = QJsonDocument::fromJson(feed_contents);
result.first->setTitle(json.object()["title"].toString());
result.first->setDescription(json.object()["description"].toString());
auto icon = json.object()["icon"].toString();
if (icon.isEmpty()) {
icon = json.object()["favicon"].toString();
}
if (!icon.isEmpty()) {
icon_possible_locations.prepend(icon);
}
// Use script to generate feed file.
try {
feed_contents = generateFeedFileWithScript(source, timeout).toUtf8();
}
else {
// Feed XML was obtained, now we need to try to guess
// its encoding before we can read further data.
QString xml_schema_encoding;
QString xml_contents_encoded;
QString enc = QRegularExpression(QSL("encoding=\"([A-Z0-9\\-]+)\""),
QRegularExpression::PatternOption::CaseInsensitiveOption).match(feed_contents).captured(1);
catch (const ScriptException& ex) {
qCriticalNN << LOGSEC_CORE
<< "Custom script for generating feed file failed during guessing:"
<< QUOTE_W_SPACE_DOT(ex.message());
if (!enc.isEmpty()) {
// Some "encoding" attribute was found get the encoding
// out of it.
xml_schema_encoding = enc;
}
QTextCodec* custom_codec = QTextCodec::codecForName(xml_schema_encoding.toLocal8Bit());
if (custom_codec != nullptr) {
// Feed encoding was probably guessed.
xml_contents_encoded = custom_codec->toUnicode(feed_contents);
result.first->setEncoding(xml_schema_encoding);
}
else {
// Feed encoding probably not guessed, set it as
// default.
xml_contents_encoded = feed_contents;
result.first->setEncoding(DEFAULT_FEED_ENCODING);
}
// Feed XML was obtained, guess it now.
QDomDocument xml_document;
QString error_msg;
int error_line, error_column;
if (!xml_document.setContent(xml_contents_encoded,
&error_msg,
&error_line,
&error_column)) {
qDebugNN << LOGSEC_CORE
<< "XML of feed" << QUOTE_W_SPACE(url) << "is not valid and cannot be loaded. "
<< "Error:" << QUOTE_W_SPACE(error_msg) << "(line " << error_line
<< ", column " << error_column << ").";
result.second = QNetworkReply::NetworkError::UnknownContentError;
// XML is invalid, exit.
return result;
}
QDomElement root_element = xml_document.documentElement();
QString root_tag_name = root_element.tagName();
if (root_tag_name == QL1S("rdf:RDF")) {
// We found RDF feed.
QDomElement channel_element = root_element.namedItem(QSL("channel")).toElement();
result.first->setType(Type::Rdf);
result.first->setTitle(channel_element.namedItem(QSL("title")).toElement().text());
result.first->setDescription(channel_element.namedItem(QSL("description")).toElement().text());
QString source_link = channel_element.namedItem(QSL("link")).toElement().text();
if (!source_link.isEmpty()) {
icon_possible_locations.prepend(source_link);
}
}
else if (root_tag_name == QL1S("rss")) {
// We found RSS 0.91/0.92/0.93/2.0/2.0.1 feed.
QString rss_type = root_element.attribute("version", "2.0");
if (rss_type == QL1S("0.91") || rss_type == QL1S("0.92") || rss_type == QL1S("0.93")) {
result.first->setType(Type::Rss0X);
}
else {
result.first->setType(Type::Rss2X);
}
QDomElement channel_element = root_element.namedItem(QSL("channel")).toElement();
result.first->setTitle(channel_element.namedItem(QSL("title")).toElement().text());
result.first->setDescription(channel_element.namedItem(QSL("description")).toElement().text());
QString icon_link = channel_element.namedItem(QSL("image")).toElement().text();
if (!icon_link.isEmpty()) {
icon_possible_locations.prepend(icon_link);
}
QString source_link = channel_element.namedItem(QSL("link")).toElement().text();
if (!source_link.isEmpty()) {
icon_possible_locations.prepend(source_link);
}
}
else if (root_tag_name == QL1S("feed")) {
// We found ATOM feed.
result.first->setType(Type::Atom10);
result.first->setTitle(root_element.namedItem(QSL("title")).toElement().text());
result.first->setDescription(root_element.namedItem(QSL("subtitle")).toElement().text());
QString icon_link = root_element.namedItem(QSL("icon")).toElement().text();
if (!icon_link.isEmpty()) {
icon_possible_locations.prepend(icon_link);
}
QString logo_link = root_element.namedItem(QSL("logo")).toElement().text();
if (!logo_link.isEmpty()) {
icon_possible_locations.prepend(logo_link);
}
QString source_link = root_element.namedItem(QSL("link")).toElement().text();
if (!source_link.isEmpty()) {
icon_possible_locations.prepend(source_link);
}
}
else {
// File was downloaded and it really was XML file
// but feed format was NOT recognized.
result.second = QNetworkReply::NetworkError::UnknownContentError;
}
}
// Try to obtain icon.
QIcon icon_data;
if ((result.second = NetworkFactory::downloadIcon(icon_possible_locations,
DOWNLOAD_TIMEOUT,
icon_data,
custom_proxy)) == QNetworkReply::NoError) {
// Icon for feed was downloaded and is stored now in _icon_data.
result.first->setIcon(icon_data);
*result = false;
return nullptr;
}
}
return result;
if (!post_process_script.simplified().isEmpty()) {
qDebugNN << LOGSEC_CORE
<< "Post-processing obtained feed data with custom script for guessing"
<< QUOTE_W_SPACE_DOT(post_process_script);
try {
feed_contents = postProcessFeedFileWithScript(post_process_script,
feed_contents,
timeout).toUtf8();
}
catch (const ScriptException& ex) {
qCriticalNN << LOGSEC_CORE
<< "Post-processing script for feed file for guessing failed:"
<< QUOTE_W_SPACE_DOT(ex.message());
*result = false;
return nullptr;
}
}
StandardFeed* feed = nullptr;
if (content_type.contains(QSL("json"), Qt::CaseSensitivity::CaseInsensitive)) {
feed = new StandardFeed();
// We have JSON feed.
feed->setEncoding(DEFAULT_FEED_ENCODING);
feed->setType(Type::Json);
QJsonDocument json = QJsonDocument::fromJson(feed_contents);
feed->setTitle(json.object()["title"].toString());
feed->setDescription(json.object()["description"].toString());
auto icon = json.object()["icon"].toString();
if (icon.isEmpty()) {
icon = json.object()["favicon"].toString();
}
if (!icon.isEmpty()) {
icon_possible_locations.prepend(icon);
}
}
else {
// Feed XML was obtained, now we need to try to guess
// its encoding before we can read further data.
QString xml_schema_encoding;
QString xml_contents_encoded;
QString enc = QRegularExpression(QSL("encoding=\"([A-Z0-9\\-]+)\""),
QRegularExpression::PatternOption::CaseInsensitiveOption)
.match(feed_contents)
.captured(1);
if (!enc.isEmpty()) {
// Some "encoding" attribute was found get the encoding
// out of it.
xml_schema_encoding = enc;
}
QTextCodec* custom_codec = QTextCodec::codecForName(xml_schema_encoding.toLocal8Bit());
QString encod;
if (custom_codec != nullptr) {
// Feed encoding was probably guessed.
xml_contents_encoded = custom_codec->toUnicode(feed_contents);
encod = xml_schema_encoding;
}
else {
// Feed encoding probably not guessed, set it as
// default.
xml_contents_encoded = feed_contents;
encod = DEFAULT_FEED_ENCODING;
}
// Feed XML was obtained, guess it now.
QDomDocument xml_document;
QString error_msg;
int error_line, error_column;
if (!xml_document.setContent(xml_contents_encoded,
&error_msg,
&error_line,
&error_column)) {
qDebugNN << LOGSEC_CORE
<< "XML of feed" << QUOTE_W_SPACE(source) << "is not valid and cannot be loaded. "
<< "Error:" << QUOTE_W_SPACE(error_msg) << "(line " << error_line
<< ", column " << error_column << ").";
*result = false;
return nullptr;
}
feed = new StandardFeed();
feed->setEncoding(encod);
QDomElement root_element = xml_document.documentElement();
QString root_tag_name = root_element.tagName();
if (root_tag_name == QL1S("rdf:RDF")) {
// We found RDF feed.
QDomElement channel_element = root_element.namedItem(QSL("channel")).toElement();
feed->setType(Type::Rdf);
feed->setTitle(channel_element.namedItem(QSL("title")).toElement().text());
feed->setDescription(channel_element.namedItem(QSL("description")).toElement().text());
QString source_link = channel_element.namedItem(QSL("link")).toElement().text();
if (!source_link.isEmpty()) {
icon_possible_locations.prepend(source_link);
}
}
else if (root_tag_name == QL1S("rss")) {
// We found RSS 0.91/0.92/0.93/2.0/2.0.1 feed.
QString rss_type = root_element.attribute("version", "2.0");
if (rss_type == QL1S("0.91") || rss_type == QL1S("0.92") || rss_type == QL1S("0.93")) {
feed->setType(Type::Rss0X);
}
else {
feed->setType(Type::Rss2X);
}
QDomElement channel_element = root_element.namedItem(QSL("channel")).toElement();
feed->setTitle(channel_element.namedItem(QSL("title")).toElement().text());
feed->setDescription(channel_element.namedItem(QSL("description")).toElement().text());
QString icon_link = channel_element.namedItem(QSL("image")).toElement().text();
QString icon_url_link = channel_element.namedItem(QSL("image")).namedItem(QSL("url")).toElement().text();
if (!icon_url_link.isEmpty()) {
icon_possible_locations.prepend(icon_url_link);
}
else if (!icon_link.isEmpty()) {
icon_possible_locations.prepend(icon_link);
}
QString source_link = channel_element.namedItem(QSL("link")).toElement().text();
if (!source_link.isEmpty()) {
icon_possible_locations.append(source_link);
}
}
else if (root_tag_name == QL1S("feed")) {
// We found ATOM feed.
feed->setType(Type::Atom10);
feed->setTitle(root_element.namedItem(QSL("title")).toElement().text());
feed->setDescription(root_element.namedItem(QSL("subtitle")).toElement().text());
QString icon_link = root_element.namedItem(QSL("icon")).toElement().text();
if (!icon_link.isEmpty()) {
icon_possible_locations.prepend(icon_link);
}
QString logo_link = root_element.namedItem(QSL("logo")).toElement().text();
if (!logo_link.isEmpty()) {
icon_possible_locations.prepend(logo_link);
}
QString source_link = root_element.namedItem(QSL("link")).toElement().text();
if (!source_link.isEmpty()) {
icon_possible_locations.prepend(source_link);
}
}
else {
// File was downloaded and it really was XML file
// but feed format was NOT recognized.
feed->deleteLater();
*result = false;
return nullptr;
}
}
// Try to obtain icon.
QIcon icon_data;
if (NetworkFactory::downloadIcon(icon_possible_locations,
DOWNLOAD_TIMEOUT,
icon_data,
custom_proxy) == QNetworkReply::NetworkError::NoError) {
// Icon for feed was downloaded and is stored now in _icon_data.
feed->setIcon(icon_data);
}
*result = true;
return feed;
}
Qt::ItemFlags StandardFeed::additionalFlags() const {
@ -530,7 +595,7 @@ QList<Message> StandardFeed::obtainNewMessages(bool* error_during_obtaining) {
try {
formatted_feed_contents = generateFeedFileWithScript(url(), download_timeout);
}
catch (const ApplicationException& ex) {
catch (const ScriptException& ex) {
qCriticalNN << LOGSEC_CORE
<< "Custom script for generating feed file failed:"
<< QUOTE_W_SPACE_DOT(ex.message());
@ -551,9 +616,9 @@ QList<Message> StandardFeed::obtainNewMessages(bool* error_during_obtaining) {
formatted_feed_contents,
download_timeout);
}
catch (const ApplicationException& ex) {
catch (const ScriptException& ex) {
qCriticalNN << LOGSEC_CORE
<< "Post-processing script failed:"
<< "Post-processing script for feed file failed:"
<< QUOTE_W_SPACE_DOT(ex.message());
setStatus(Status::OtherError);
@ -593,27 +658,48 @@ QList<Message> StandardFeed::obtainNewMessages(bool* error_during_obtaining) {
QPair<QString, QString> StandardFeed::prepareExecutionLine(const QString& execution_line) {
auto split_exec = execution_line.split('#', Qt::SplitBehaviorFlags::KeepEmptyParts);
if (split_exec.size() != 2) {
throw ScriptException(ScriptException::Reason::ExecutionLineInvalid);
}
auto user_data_folder = qApp->userDataFolder();
return { split_exec[0].replace(EXECUTION_LINE_USER_DATA_PLACEHOLDER, user_data_folder),
split_exec[1].replace(EXECUTION_LINE_USER_DATA_PLACEHOLDER, user_data_folder) };
}
QString StandardFeed::generateFeedFileWithScript(const QString& execution_line, int run_timeout) {
auto prepared_query = prepareExecutionLine(execution_line);
QString StandardFeed::runScriptProcess(const QPair<QString, QString>& cmd_args, const QString& working_directory,
int run_timeout, bool provide_input, const QString& input) {
QProcess process;
process.setWorkingDirectory(qApp->userDataFolder());
process.setProgram(prepared_query.first);
if (provide_input) {
process.setInputChannelMode(QProcess::InputChannelMode::ManagedInputChannel);
}
process.setProcessChannelMode(QProcess::ProcessChannelMode::SeparateChannels);
process.setWorkingDirectory(working_directory);
process.setProgram(cmd_args.first);
#if defined(Q_OS_WIN)
process.setNativeArguments(prepared_query.second);
process.setNativeArguments(cmd_args.second);
#else
process.setArguments({ prepared_query.second });
process.setArguments({ cmd_args.second });
#endif
if (!process.open() || process.error() == QProcess::ProcessError::FailedToStart) {
throw ApplicationException(QSL("process failed to start"));
if (!process.open()) {
switch (process.error()) {
case QProcess::ProcessError::FailedToStart:
throw ScriptException(ScriptException::Reason::InterpreterNotFound);
default:
break;
}
}
if (provide_input) {
process.write(input.toUtf8());
process.closeWriteChannel();
}
if (process.waitForFinished(run_timeout)) {
@ -624,55 +710,30 @@ QString StandardFeed::generateFeedFileWithScript(const QString& execution_line,
else {
process.kill();
auto raw_error = process.readAllStandardError();
auto raw_error = process.readAllStandardError().simplified();
if (raw_error.simplified().isEmpty()) {
throw ApplicationException(QSL("process failed to finish properly"));
}
else {
throw ApplicationException(QString(raw_error));
switch (process.error()) {
case QProcess::ProcessError::Timedout:
throw ScriptException(ScriptException::Reason::InterpreterTimeout);
default:
throw ScriptException(ScriptException::Reason::InterpreterError, raw_error);
}
}
}
QString StandardFeed::postProcessFeedFileWithScript(const QString& execution_line, const QString raw_feed_data, int run_timeout) {
QString StandardFeed::generateFeedFileWithScript(const QString& execution_line, int run_timeout) {
auto prepared_query = prepareExecutionLine(execution_line);
QProcess process;
process.setInputChannelMode(QProcess::InputChannelMode::ManagedInputChannel);
process.setWorkingDirectory(qApp->userDataFolder());
process.setProgram(prepared_query.first);
return runScriptProcess(prepared_query, qApp->userDataFolder(), run_timeout, false);
}
#if defined(Q_OS_WIN)
process.setNativeArguments(prepared_query.second);
#else
process.setArguments({ prepared_query.second });
#endif
QString StandardFeed::postProcessFeedFileWithScript(const QString& execution_line,
const QString raw_feed_data,
int run_timeout) {
auto prepared_query = prepareExecutionLine(execution_line);
if (!process.open() || process.error() == QProcess::ProcessError::FailedToStart) {
throw ApplicationException(QSL("process failed to start"));
}
process.write(raw_feed_data.toUtf8());
process.closeWriteChannel();
if (process.waitForFinished(run_timeout)) {
auto raw_output = process.readAllStandardOutput();
return raw_output;
}
else {
process.kill();
auto raw_error = process.readAllStandardError();
if (raw_error.simplified().isEmpty()) {
throw ApplicationException(QSL("process failed to finish properly"));
}
else {
throw ApplicationException(QString(raw_error));
}
}
return runScriptProcess(prepared_query, qApp->userDataFolder(), run_timeout, true, raw_feed_data);
}
QNetworkReply::NetworkError StandardFeed::networkError() const {

15
src/librssguard/services/standard/standardfeed.h
View File

@ -87,10 +87,13 @@ class StandardFeed : public Feed {
// Returns pointer to guessed feed (if at least partially
// guessed) and retrieved error/status code from network layer
// or NULL feed.
static QPair<StandardFeed*, QNetworkReply::NetworkError> guessFeed(const QString& url,
const QString& username = QString(),
const QString& password = QString(),
const QNetworkProxy& custom_proxy = QNetworkProxy::ProxyType::DefaultProxy);
static StandardFeed* guessFeed(SourceType source_type,
const QString& url,
const QString& post_process_script,
bool* result,
const QString& username = QString(),
const QString& password = QString(),
const QNetworkProxy& custom_proxy = QNetworkProxy::ProxyType::DefaultProxy);
// Converts particular feed type to string.
static QString typeToString(Type type);
@ -99,6 +102,10 @@ class StandardFeed : public Feed {
public slots:
void fetchMetadataForItself();
private:
static QString runScriptProcess(const QPair<QString, QString>& cmd_args, const QString& working_directory,
int run_timeout, bool provide_input, const QString& input = {});
private:
SourceType m_sourceType;
Type m_type;

29
src/librssguard/services/standard/standardfeedsimportexportmodel.cpp
View File

@ -185,13 +185,18 @@ void FeedsImportExportModel::importAsOPML20(const QByteArray& data, bool fetch_m
QString feed_url = child_element.attribute(QSL("xmlUrl"));
if (!feed_url.isEmpty()) {
QPair<StandardFeed*, QNetworkReply::NetworkError> guessed;
StandardFeed* guessed;
bool result;
if (fetch_metadata_online &&
(guessed = StandardFeed::guessFeed(feed_url, {}, {}, custom_proxy)).second == QNetworkReply::NoError) {
(guessed = StandardFeed::guessFeed(StandardFeed::SourceType::Url,
feed_url,
{}, &result, {}, {},
custom_proxy)) != nullptr &&
result) {
// We should obtain fresh metadata from online feed source.
guessed.first->setUrl(feed_url);
active_model_item->appendChild(guessed.first);
guessed->setUrl(feed_url);
active_model_item->appendChild(guessed);
succeded++;
}
else {
@ -224,7 +229,7 @@ void FeedsImportExportModel::importAsOPML20(const QByteArray& data, bool fetch_m
active_model_item->appendChild(new_feed);
if (fetch_metadata_online && guessed.second != QNetworkReply::NoError) {
if (fetch_metadata_online && result) {
failed++;
}
else {
@ -304,12 +309,16 @@ void FeedsImportExportModel::importAsTxtURLPerLine(const QByteArray& data, bool
for (const QByteArray& url : urls) {
if (!url.isEmpty()) {
QPair<StandardFeed*, QNetworkReply::NetworkError> guessed;
StandardFeed* guessed;
bool result;
if (fetch_metadata_online &&
(guessed = StandardFeed::guessFeed(url, {}, {}, custom_proxy)).second == QNetworkReply::NoError) {
guessed.first->setUrl(url);
root_item->appendChild(guessed.first);
(guessed = StandardFeed::guessFeed(StandardFeed::SourceType::Url,
url, {}, &result, {}, {},
custom_proxy)) != nullptr &&
result) {
guessed->setUrl(url);
root_item->appendChild(guessed);
succeded++;
}
else {
@ -322,7 +331,7 @@ void FeedsImportExportModel::importAsTxtURLPerLine(const QByteArray& data, bool
feed->setEncoding(DEFAULT_FEED_ENCODING);
root_item->appendChild(feed);
if (fetch_metadata_online && guessed.second != QNetworkReply::NoError) {
if (fetch_metadata_online && result) {
failed++;
}
else {