Commit Graph

39 Commits

Author SHA1 Message Date
Noémi Ványi f00d9e0ec4
Pick minor fixes from searxng (#3251)
* [fix] Rename ccengine engine to openverse

The CC engine was merged with WordPress and renamed to Openverse

Source: https://wordpress.org/news/2021/05/welcome-to-openverse/

* [fix] ccengine engine - avoid unwanted redirects

api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:

    /v1/images? -->/ v1/images/?

otherwise it redirects, here is the debug log:

    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
    WARNING searx.engines.openverse       : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] FutureWarning from lxml

Just in case if content is None, the original code will skip extract_text(), and
just append the None value to 'content'. So just add allow_none=True, and this
will return None without raising a ValueError in extract_text().

* [enh] Add pagination to Brave

Also added ```&spellcheck=1``` because now it is disabled by default, not returning any ```suggestion_xpath```.

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: capric98 <42015599+capric98@users.noreply.github.com>
Co-authored-by: Allen <64094914+allendema@users.noreply.github.com>
2022-06-06 00:01:27 +02:00
Eric Zhang b7d91c9c95
yahoo engine - don't lump all search suggestions together (#3208) 2022-04-13 21:00:54 +02:00
Markus Heiser 4d36aee57b [fix] engine - yahoo: rewrite and fix issues
Languages are supported by mapping the language to a domain.  If domain is not
found in :py:obj:`lang2domain` URL ``<lang>.search.yahoo.com`` is used.

Closes #3020

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 20:30:10 +01:00
Alexandre Flament ca93a01844 [mod] dynamically set language_support variable
The language_support variable is set to True by default,
and set to False in only 5 engines.

Except the documentation and the /config URL, this variable is not used.

This commit remove the variable definition in the engines, and
set value according to supported_languages length: False when the length is 0,
True otherwise.

Close #2485
2021-02-01 17:10:37 +01:00
Alexandre Flament a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
2021-01-14 20:57:17 +01:00
Alexandre Flament 2006eb4680 [mod] move extract_text, extract_url to searx.utils 2020-10-02 18:13:56 +02:00
Dalf 1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Markus Heiser 5293e58032 [fix] yahoo engine - changed content_xpath
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-25 21:45:42 +02:00
Dalf 85b3723345 [mod] speed optimization
compile XPath only once
avoid redundant call to urlparse
get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
2019-11-15 09:33:15 +01:00
Noémi Ványi b63d645a52 Revert "remove 'all' option from search languages"
This reverts commit 4d1770398a.
2019-01-07 21:19:00 +01:00
Marc Abonce Seguin 772c048d01 refactor engine's search language handling
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
2018-03-27 00:08:03 -06:00
marc 4d1770398a remove 'all' option from search languages 2017-12-06 01:20:15 -06:00
Adam Tauber 52e615dede [enh] py3 compatibility 2017-05-15 12:02:30 +02:00
marc af35eee10b tests for _fetch_supported_languages in engines
and refactor method to make it testable without making requests
2016-12-15 00:40:21 -06:00
marc f62ce21f50 [mod] fetch supported languages for several engines
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
2016-12-13 19:58:10 -06:00
marc 149802c569 [enh] add supported_languages on engines and auto-generate languages.py 2016-12-13 19:32:00 -06:00
Noémi Ványi b034356825 add year filter to engines with time range support && tests
Following engines does not support "Last year":
 * Bing News
 * DeviantArt
 * DuckDuckGo
 * Yahoo
 * YouTube (noapi)
2016-12-11 16:58:31 +01:00
Adam Tauber 350a84520d [fix] time range detection 2016-07-26 00:28:48 +02:00
Noemi Vanyi a7c8d5882c fix pep8 2016-07-25 23:28:14 +02:00
Noemi Vanyi 93c0c49e9a add time range search with yahoo 2016-07-25 23:19:46 +02:00
Adam Tauber 2f7752b410 [enh] display number of results 2016-06-28 00:06:50 +02:00
Alexandre Flament 4689fe341c update versions.cfg to use the current up-to-date packages 2015-05-02 15:45:17 +02:00
dalf 57996b12fc [fix] update yahoo engine according to the web site changes 2015-02-20 12:34:13 +01:00
Cqoicebordel 9f13af8d3c Yahoo's unit test 2015-02-04 20:41:40 +01:00
Thomas Pointhuber a508d540ac [fix] pep8 2014-12-16 17:26:16 +01:00
dalf 7c13d630e4 [fix] pep8 : engines (errors E121, E127, E128 and E501 still exist) 2014-12-07 16:37:56 +01:00
Dalf a4ffeddce1 [fix] yahoo engines: parse_url doesn't throw an exception with not tracking URL 2014-09-07 18:42:56 +02:00
Thomas Pointhuber 03db970e6a fix yahoo engines and add comments 2014-09-01 16:17:29 +02:00
Adam Tauber 01c2eeb8ff [fix] indexing ++ url extraction 2014-03-21 16:36:13 +01:00
Adam Tauber faed14b2c6 [fix] #55 2014-03-18 10:06:14 +01:00
Adam Tauber 3acdf3f9a1 [fix] pep8 2014-03-10 16:46:11 +01:00
Adam Tauber 0fc481f47e [fix] yahoo url extraction 2014-03-08 19:09:03 +01:00
Adam Tauber ee1ffbc87f [fix] yahoo engine url extraction 2014-03-04 14:19:59 +01:00
Adam Tauber 4627788834 [fix] yahoo url mods 2014-02-27 10:59:39 +01:00
Gabor Nagy 22fd0746d9 Improves PEP8 compatibility. 2014-02-05 20:24:31 +01:00
asciimoo 3a2b9a8682 [fix] pep8 2014-01-31 05:11:37 +01:00
asciimoo 68a0832524 [enh] search language support upadtes 2014-01-31 05:10:49 +01:00
asciimoo 72793725e2 [fix] print removed 2014-01-30 02:36:15 +01:00
asciimoo 642986c2e7 [enh] yahoo engine added 2014-01-30 02:33:24 +01:00