Commit Graph

4489 Commits

Author SHA1 Message Date
Noémi Ványi 54697a8705 Fix online dictionaries 2022-07-30 21:54:24 +02:00
Noémi Ványi 05fe2ee093
pick engine fixes (#3306)
* [fix] google engine: results XPath

* [fix] google & youtube - set EU consent cookie

This change the previous bypass method for Google consent using
``ucbcb=1`` (6face215b8) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] https://github.com/iv-org/invidious/pull/2207
[2] https://github.com/TeamNewPipe/NewPipeExtractor/issues/592

Closes: https://github.com/searxng/searxng/issues/1432

* [fix] sjp engine - convert enginename to a latin1 compliance name

The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] engine tineye: handle 422 response of not supported img format

Closes: https://github.com/searxng/searxng/issues/1449
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* bypass google consent with ucbcb=1

* [mod] Adds Lingva translate engine

Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.

* openstreetmap engine: return the localized named.

For example: display "Tokyo" instead of "東京都" when the language is English.

* [fix] engines/openstreetmap.py typo: user_langage --> user_language

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* Wikidata engine: ignore dummy entities

* Wikidata engine: minor change of the SPARQL request

The engine can be slow especially when the query won't return any answer.
See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Emilien Devos <contact@emiliendevos.be>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Emilien Devos <github@emiliendevos.be>
Co-authored-by: ta <alt3753.7@gmail.com>
Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-07-30 21:45:07 +02:00
Noémi Ványi 85034b49ef
Remove `httpx` and use `requests` instead (#3305)
## What does this PR do?

This PR prepares for removing `httpx`, and reverts back to `requests`.

## Why is this change important?

`httpx` hasn't proven itself to be faster or better than `requests`. On the other
hand it has caused issues on Windows.

=============================================
Please update your environment to use requests instead of httpx.
=============================================
2022-07-30 20:56:56 +02:00
james-still 210e59c68c
Add engine for Emojipedia (#3278) 2022-07-28 21:45:07 +02:00
Ben Collerson 16d43fe8d4
[fix] keyboard hints for category tabs (#1187) (#3276)
Category tabs issue: hints don't see category tabs
[qutebrowser] #1187
2022-07-12 16:23:44 +02:00
Noémi Ványi a2c7cf4b8a Reword readme again 2022-07-05 09:44:52 +02:00
Noémi Ványi 2e9d69cee4 challange -> challenge in readme 2022-07-03 21:56:34 +02:00
Noémi Ványi 4a92c6e7d9
Fix headers in README 2022-07-03 20:20:45 +02:00
Noémi Ványi f51bc5c648 Update readme with comparison with searxng
Closes #3269
2022-07-03 20:12:26 +02:00
searx-bot 28d5347aef
Update searx.data - update_wikidata_units.py (#3271)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-07-01 13:33:24 +02:00
searx-bot a7a70f67a9
Update searx.data - update_currencies.py (#3272)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-07-01 13:23:32 +02:00
searx-bot 9354df795c
Update searx.data - update_ahmia_blacklist.py (#3273)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-07-01 13:23:28 +02:00
searx-bot 7cbd35cc75
Update searx.data - update_firefox_version.py (#3274)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-07-01 13:23:02 +02:00
Noémi Ványi 7bb499cb1e fix pylint error in bing engine 2022-07-01 13:12:21 +02:00
Adam Tauber a3ad9f9b34 [fix] use chrome ua to quickfix bing result urls - closes #3239 2022-06-06 14:34:56 +02:00
dependabot[bot] 8a19442897
Bump jinja2 from 3.1.1 to 3.1.2 (#3253)
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.1 to 3.1.2.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.1...3.1.2)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 00:30:30 +02:00
dependabot[bot] fedbea2c92
Bump certifi from 2021.10.8 to 2022.5.18.1 (#3254)
Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.10.8 to 2022.5.18.1.
- [Release notes](https://github.com/certifi/python-certifi/releases)
- [Commits](https://github.com/certifi/python-certifi/compare/2021.10.08...2022.05.18.1)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 00:30:08 +02:00
searx-bot 763d0826ec
Update searx.data - update_ahmia_blacklist.py (#3242)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-06-06 00:23:23 +02:00
dependabot[bot] 738606a277
Bump httpx[http2] from 0.21.3 to 0.23.0 (#3238)
Bumps [httpx[http2]](https://github.com/encode/httpx) from 0.21.3 to 0.23.0.
- [Release notes](https://github.com/encode/httpx/releases)
- [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/encode/httpx/compare/0.21.3...0.23.0)

---
updated-dependencies:
- dependency-name: httpx[http2]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 00:23:09 +02:00
dependabot[bot] a29bc166a6
Bump httpx-socks[asyncio] from 0.7.2 to 0.7.4 (#3237)
Bumps [httpx-socks[asyncio]](https://github.com/romis2012/httpx-socks) from 0.7.2 to 0.7.4.
- [Release notes](https://github.com/romis2012/httpx-socks/releases)
- [Commits](https://github.com/romis2012/httpx-socks/compare/v0.7.2...v0.7.4)

---
updated-dependencies:
- dependency-name: httpx-socks[asyncio]
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 00:22:44 +02:00
Noémi Ványi 2719fd2526
Pick pass cookies from searxng (#3252)
* [enh] Allow passing headers/cookies from settings.yml

Example:

   - engine: xpath
   - search_url: example.org
   - headers: {'example_header': 'example_header'}
   - cookies: {'safesearch': 'off'}

* [fix[ Update only cookies/headers

* [enh] XPath engine - add time range support

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [enh] XPath engine - add time safe-search support

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Co-authored-by: Allen <64094914+allendema@users.noreply.github.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-06 00:18:33 +02:00
Noémi Ványi f00d9e0ec4
Pick minor fixes from searxng (#3251)
* [fix] Rename ccengine engine to openverse

The CC engine was merged with WordPress and renamed to Openverse

Source: https://wordpress.org/news/2021/05/welcome-to-openverse/

* [fix] ccengine engine - avoid unwanted redirects

api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:

    /v1/images? -->/ v1/images/?

otherwise it redirects, here is the debug log:

    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
    WARNING searx.engines.openverse       : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] FutureWarning from lxml

Just in case if content is None, the original code will skip extract_text(), and
just append the None value to 'content'. So just add allow_none=True, and this
will return None without raising a ValueError in extract_text().

* [enh] Add pagination to Brave

Also added ```&spellcheck=1``` because now it is disabled by default, not returning any ```suggestion_xpath```.

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: capric98 <42015599+capric98@users.noreply.github.com>
Co-authored-by: Allen <64094914+allendema@users.noreply.github.com>
2022-06-06 00:01:27 +02:00
dependabot[bot] 8ee980979a
Bump lxml from 4.7.1 to 4.9.0 (#3249)
Bumps [lxml](https://github.com/lxml/lxml) from 4.7.1 to 4.9.0.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.7.1...lxml-4.9.0)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-05 23:47:08 +02:00
liimee a3e41c3cd6
Add TVmaze engine (#3246) 2022-06-05 23:36:04 +02:00
Noémi Ványi f0b1c9bbcc
Updated version of "Ddg safe search" PR (#3247)
* fix safe search with ddg engine

* fix unused imports

* extract title from htmlextractor

Co-authored-by: Nivesh Krishna <nivesh@e.email>
2022-06-02 21:36:04 +02:00
searx-bot 6ffa70d879
Update searx.data - update_wikidata_units.py (#3222)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-05-24 21:08:52 +02:00
searx-bot 81b8bf3fe0
Update searx.data - update_firefox_version.py (#3223)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-05-24 21:08:36 +02:00
nathannaveen 260949ed48
chore: Set permissions for GitHub actions (#3225)
Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much.

- Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions

https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

[Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)

Signed-off-by: nathannaveen <42319948+nathannaveen@users.noreply.github.com>
2022-05-24 21:07:23 +02:00
searx-bot f522f92250
Update searx.data - update_currencies.py (#3203)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-04-13 21:11:59 +02:00
searx-bot 3a2a153cb8
Update searx.data - update_firefox_version.py (#3202)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-04-13 21:10:18 +02:00
searx-bot a87555755d
Update searx.data - update_ahmia_blacklist.py (#3201)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-04-13 21:09:55 +02:00
searx-bot ddb9870acf
Update searx.data - update_wikidata_units.py (#3200)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-04-13 21:09:32 +02:00
Eric Zhang b7d91c9c95
yahoo engine - don't lump all search suggestions together (#3208) 2022-04-13 21:00:54 +02:00
Noémi Ványi ba95fd570b
Merge pull request #3209 from kvch/update-flask
Update flask and jinja2 to fix build
2022-04-13 20:55:30 +02:00
Markus Heiser 3abf620418 [fix] issue when upgrading from werkzeug v2.0.3 to v2.1.0
In v2.1.0 werkzeug [1] fixed an issue [2] to keep relative redirect locations by
default [3].  Since relative locations are returned, we need to fix out test
cases to avoid AssertionErrors like this one::

    ======================================================================
    FAIL: test_index_html_get (tests.unit.test_webapp.ViewsTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/runner/work/searxng/searxng/tests/unit/test_webapp.py", line 105, in test_index_html_get
      self.assertEqual(result.location, 'http://localhost/search?q=test')
    AssertionError: '/search?q=test' != 'http://localhost/search?q=test'
    - /search?q=test
    + http://localhost/search?q=test

[1] https://werkzeug.palletsprojects.com/
[2] https://github.com/pallets/werkzeug/issues/2352 fixed in
[3] https://github.com/pallets/werkzeug/pull/2354

Related-to: https://github.com/searxng/searxng/pull/1039#issuecomment-1085538288
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-13 20:49:42 +02:00
Noémi Ványi bca820589e Update flask and jinja2 2022-04-13 20:45:24 +02:00
Noémi Ványi 03eb9c2461 Provide better error message if settings.yml cannot be loaded
Closes #3184
2022-03-17 20:34:50 +01:00
Markus Heiser f231d79a5d [fix] engine: Semantic Scholar (Science) // rework & fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-05 20:59:11 +01:00
Noémi Ványi c56f2f1d6b Skip result in Semantic Scholar engine if URL is missing 2022-03-03 22:06:04 +01:00
searx-bot e2ab703f3e
Update searx.data - update_firefox_version.py (#3171)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-03-02 22:29:00 +01:00
searx-bot c9777de0d5
Update searx.data - update_wikidata_units.py (#3170)
Co-authored-by: dalf <dalf@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-03-02 22:23:06 +01:00
Marc Abonce Seguin c9e6d9f5f6
Fix Tor proxy when using httpx 0.21.x (#3165)
## What does this PR do?

This should fix #3164.
The problem is that `httpx` keeps making breaking changes to their library, so we just have to adjust the code a little bit to make it work with the new version of the library.


## Related issues
Closes  #3164
2022-03-01 20:21:25 +01:00
Noémi Ványi 0669bfd7a5
Fix issues in network after updating httpx to 0.21.x (#3169)
* [mod] upgrade httpx 0.21.2

httpx 0.21.2 and httpcore 0.14.4 fix multiple issues:
* https://github.com/encode/httpx/releases/tag/0.21.2
* https://github.com/encode/httpcore/releases/tag/0.14.4

so most of the workarounds in searx.network have been removed.

* pick even more changes from searxng

Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-02-28 22:05:20 +01:00
searx-bot 0248777f95
Update searx.data - update_ahmia_blacklist.py (#3158)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-02-11 21:24:50 +01:00
searx-bot 22ecae7d48
Update searx.data - update_currencies.py (#3157)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-02-11 21:24:43 +01:00
searx-bot fa2ad3cb03
Update searx.data - update_wikidata_units.py (#3156)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-02-11 21:24:26 +01:00
searx-bot bf021c538d
Update searx.data - update_firefox_version.py (#3155)
Co-authored-by: dalf <dalf@users.noreply.github.com>
2022-02-11 21:24:12 +01:00
israelyago 3fd18ab51b
Fix digg engine (#3150) 2022-01-30 16:41:53 +01:00
Noémi Ványi a164585118 Add extra features to Gigablast engine:
* fast can be enabled to results are returned quicker
* collection can be configured
* search_type can be changed to images or news

Closes #3078
2022-01-22 19:14:45 +01:00
iko 01e28757d3
Fixed Hoogle engine (#3146) 2022-01-22 18:22:24 +01:00