searx/searx/engines
Noémi Ványi 05fe2ee093
pick engine fixes (#3306)
* [fix] google engine: results XPath

* [fix] google & youtube - set EU consent cookie

This change the previous bypass method for Google consent using
``ucbcb=1`` (6face215b8) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] https://github.com/iv-org/invidious/pull/2207
[2] https://github.com/TeamNewPipe/NewPipeExtractor/issues/592

Closes: https://github.com/searxng/searxng/issues/1432

* [fix] sjp engine - convert enginename to a latin1 compliance name

The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] engine tineye: handle 422 response of not supported img format

Closes: https://github.com/searxng/searxng/issues/1449
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* bypass google consent with ucbcb=1

* [mod] Adds Lingva translate engine

Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.

* openstreetmap engine: return the localized named.

For example: display "Tokyo" instead of "東京都" when the language is English.

* [fix] engines/openstreetmap.py typo: user_langage --> user_language

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* Wikidata engine: ignore dummy entities

* Wikidata engine: minor change of the SPARQL request

The engine can be slow especially when the query won't return any answer.
See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Emilien Devos <contact@emiliendevos.be>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Emilien Devos <github@emiliendevos.be>
Co-authored-by: ta <alt3753.7@gmail.com>
Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-07-30 21:45:07 +02:00
..
1337x.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
__init__.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
ahmia.py Update onion engines to v3 (#2904) 2021-08-02 20:03:55 +02:00
apkmirror.py [fix] APKMirror engine - update xpath selectors and fix img_src 2021-03-09 08:34:57 +01:00
archlinux.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
arxiv.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
bandcamp.py [fix] xpath expressions to grap all items from bandcamp's response 2021-04-15 08:52:11 +02:00
base.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
bing.py fix pylint error in bing engine 2022-07-01 13:12:21 +02:00
bing_images.py Fix paging of Bing Images 2021-02-22 21:19:34 +01:00
bing_news.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
bing_videos.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
btdigg.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
command.py Bump pylint from 2.10.2 to 2.12.2 (#3124) 2022-01-15 20:23:09 +01:00
core.py add core.ac.uk engine 2021-04-29 22:43:52 +02:00
currency_convert.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
dailymotion.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
deezer.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
deviantart.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
dictzone.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
digbt.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
digg.py Fix digg engine (#3150) 2022-01-30 16:41:53 +01:00
doku.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
duckduckgo.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
duckduckgo_definitions.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
duckduckgo_images.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
duden.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
dummy-offline.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
dummy.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
ebay.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
elasticsearch.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
emojipedia.py Add engine for Emojipedia (#3278) 2022-07-28 21:45:07 +02:00
etools.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
fdroid.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
flickr.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
flickr_noapi.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
framalibre.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
freesound.py Add Freesound engine (#2596) 2021-03-01 08:52:36 +01:00
frinkiac.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
genius.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
gentoo.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
gigablast.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
github.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
google.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
google_images.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
google_news.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
google_play_apps.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
google_scholar.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
google_videos.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
imdb.py Add IMDB support (#2980) 2021-10-02 13:41:38 +02:00
ina.py [fix] make ina engine compatible with the new response json 2021-03-25 01:20:41 +01:00
invidious.py [fix] Update about section of Invidious and Rumble + Change filtron error wording (#2959) 2021-10-10 16:44:21 +02:00
json_engine.py [mod] json_engine: add content_html_to_text and title_html_to_text 2021-02-10 16:42:11 +01:00
kickass.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
lingva.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
loc.py Add Library of Congress engine 2021-02-09 12:45:39 +01:00
mediathekviewweb.py [enh] add engine MediathekViewWeb (API) 2021-02-09 13:08:01 +01:00
mediawiki.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
meilisearch.py Bump pylint from 2.9.6 to 2.10.2 2021-08-22 20:53:12 +02:00
microsoft_academic.py fix Microsoft Academic engine 2021-03-15 20:21:28 +01:00
mixcloud.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
mongodb.py [fix] pep8 2021-07-05 19:57:53 +02:00
mysql_server.py make port configurable in MySQL engine 2022-01-11 22:49:53 +01:00
nyaa.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
onesearch.py Remove debug log from onesearch engine 2021-11-17 15:15:17 -03:00
opensemantic.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
openstreetmap.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
openverse.py Pick minor fixes from searxng (#3251) 2022-06-06 00:01:27 +02:00
pdbe.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
peertube.py Improve peertube searching 2021-02-13 19:47:33 +01:00
photon.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
piratebay.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
postgresql.py Fix remote PEP8 errors as well 2021-04-29 22:05:31 +02:00
prowlarr.py New files engine: Prowlarr (#3118) 2022-01-15 19:18:15 +01:00
pubmed.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
qwant.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
recoll.py [feat] recoll: paged json support 2021-02-07 10:05:35 -03:00
reddit.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
redis_server.py [fix] initialize redis engine at the right time 2021-07-05 19:40:51 +02:00
rumble.py [fix] Update about section of Invidious and Rumble + Change filtron error wording (#2959) 2021-10-10 16:44:21 +02:00
scanr_structures.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
searchcode_code.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
searx_engine.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
semantic_scholar.py [fix] engine: Semantic Scholar (Science) // rework & fix 2022-03-05 20:59:11 +01:00
sepiasearch.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
seznam.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
sjp.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
solidtorrents.py Fix URL to solidtorrent result page 2021-04-28 23:57:54 +02:00
solr.py Add Solr engine 2021-03-13 21:18:09 +01:00
soundcloud.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
spotify.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
springer.py [enh] Add Springer Nature engine 2021-04-29 22:43:52 +02:00
sqlite.py Add sqlite engine to pylint 2021-05-13 21:47:38 +02:00
stackexchange.py [fix] style of stackexchange engine 2021-10-02 13:25:50 +02:00
stackoverflow.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
startpage.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
tineye.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
tokyotoshokan.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
torrentz.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
translated.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
tvmaze.py Add TVmaze engine (#3246) 2022-06-05 23:36:04 +02:00
unsplash.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
vimeo.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
wikidata.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
wikipedia.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
wolframalpha_api.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
wolframalpha_noapi.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
wordnik.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
www1x.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
xpath.py Pick pass cookies from searxng (#3252) 2022-06-06 00:18:33 +02:00
xpath_flex.py feature: new engine xpath_flex (#3119) 2022-01-11 22:44:19 +01:00
yacy.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
yahoo.py Pick minor fixes from searxng (#3251) 2022-06-06 00:01:27 +02:00
yahoo_news.py [fix] remove unused import from yahoo-news engine 2021-03-14 15:13:57 +01:00
yggtorrent.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
youtube_api.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
youtube_noapi.py pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00