searx/searx
Noémi Ványi 05fe2ee093
pick engine fixes (#3306)
* [fix] google engine: results XPath

* [fix] google & youtube - set EU consent cookie

This change the previous bypass method for Google consent using
``ucbcb=1`` (6face215b8) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] https://github.com/iv-org/invidious/pull/2207
[2] https://github.com/TeamNewPipe/NewPipeExtractor/issues/592

Closes: https://github.com/searxng/searxng/issues/1432

* [fix] sjp engine - convert enginename to a latin1 compliance name

The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] engine tineye: handle 422 response of not supported img format

Closes: https://github.com/searxng/searxng/issues/1449
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* bypass google consent with ucbcb=1

* [mod] Adds Lingva translate engine

Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.

* openstreetmap engine: return the localized named.

For example: display "Tokyo" instead of "東京都" when the language is English.

* [fix] engines/openstreetmap.py typo: user_langage --> user_language

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* Wikidata engine: ignore dummy entities

* Wikidata engine: minor change of the SPARQL request

The engine can be slow especially when the query won't return any answer.
See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Emilien Devos <contact@emiliendevos.be>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Emilien Devos <github@emiliendevos.be>
Co-authored-by: ta <alt3753.7@gmail.com>
Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-07-30 21:45:07 +02:00
..
answerers [fix] answers: don't crash when the query is an empty string 2021-03-01 10:52:39 +01:00
data Update searx.data - update_wikidata_units.py (#3271) 2022-07-01 13:33:24 +02:00
engines pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
metrology Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
plugins [enh] Add Pagination to Wiby / Fix Kaufland (#3000) 2021-10-10 20:52:06 +02:00
search Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
shared [fix] checker: don't run the checker when uwsgi is not properly configured 2021-01-13 14:07:39 +01:00
static [fix] keyboard hints for category tabs (#1187) (#3276) 2022-07-12 16:23:44 +02:00
templates [fix] keyboard hints for category tabs (#1187) (#3276) 2022-07-12 16:23:44 +02:00
translations [enh] update translations from transifex 2021-03-27 19:10:54 +01:00
__init__.py Provide better error message if settings.yml cannot be loaded 2022-03-17 20:34:50 +01:00
autocomplete.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
exceptions.py [enh] add raise_for_httperror 2020-12-11 14:37:08 +01:00
external_bang.py [mod] add utils/fetch_external_bangs.py 2021-02-24 18:48:36 +01:00
external_urls.py [enh] openstreetmap / map template: improve results 2021-10-02 14:57:30 +02:00
languages.py Update searx.data - update_languages.py 2021-03-05 10:56:46 +00:00
poolrequests.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
preferences.py Bump pylint from 2.9.6 to 2.10.2 2021-08-22 20:53:12 +02:00
query.py [enh] autocomplete refactoring, autocomplete on external bangs 2021-03-01 19:12:32 +01:00
raise_for_httperror.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
results.py Add scheme to img_src and thumbnail_url if missing from URL 2022-01-22 11:59:21 +01:00
settings.yml pick engine fixes (#3306) 2022-07-30 21:45:07 +02:00
settings_loader.py Provide better error message if settings.yml cannot be loaded 2022-03-17 20:34:50 +01:00
settings_robot.yml Use oadoi.org as default_doi_resolver 2021-04-29 22:43:52 +02:00
testing.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
utils.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
version.py [enh] release v1.0.0 2021-03-27 20:30:08 +01:00
webadapter.py [fix] typo in searx.webadapter.parse_lang 2021-09-19 17:37:35 +02:00
webapp.py Remove `httpx` and use `requests` instead (#3305) 2022-07-30 20:56:56 +02:00
webutils.py Updated webutils.highlight_content to ignore double-quotes when highlighting query parts 2021-02-08 23:58:54 -05:00