devol
/
searx
mirror of https://github.com/searx/searx
1
0
Fork 0
Code Issues Releases Wiki Activity
4,304 Commits 19 Branches 18 Tags 56 MiB
Commit Graph

2833 Commits

Author SHA1 Message Date
Noémi Ványi fd1e49c9b6
Merge branch 'master' into dependabot/pip/master/pylint-2.10.2 2021-08-22 21:11:12 +02:00
dependabot[bot] e271d6d1e1 Bump pylint from 2.9.6 to 2.10.2 
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.10.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.10.2)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-08-22 20:53:12 +02:00
Markus Heiser 2484e7473a [fix] /config add missing GIT_BRANCH value 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-08-22 18:58:03 +02:00
Noémi Ványi 23b3b56a06 add filter=0 to Google engine for more results 
Closes #2944
 2021-08-21 16:29:20 +02:00
dependabot[bot] 0810263bdd
Bump pylint from 2.8.2 to 2.9.6 (#2934) 
* Bump pylint from 2.8.2 to 2.9.6

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.8.2 to 2.9.6.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.8.2...v2.9.6)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update preferences.py to comply with new pep8

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Noémi Ványi <sitbackandwait@gmail.com>
 2021-08-12 21:50:48 +02:00
Finn b1e5ff3cf7
[fix] Update xpaths for petalsearch.com (#2932) 2021-08-12 18:27:15 +02:00
Noémi Ványi 3d2a5061db Fix unicode error when using saved preferences URL 
Closes #2917
Closes #2856
 2021-08-07 16:33:47 +02:00
Allen be401469d2
[mod] Apply HTTPS where possible + fix small typo (#2922) 2021-08-07 14:56:01 +02:00
Allen 76606e7372
[enh] Add whaleslide.com (#2861) 
Upstream example: https://whaleslide.com/search/web/runlevels

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
 2021-08-03 21:31:08 +02:00
Noémi Ványi dc99fec831 [fix] add new list element to petal so it does not overwrite previous engines 2021-08-03 21:21:28 +02:00
searx-bot 291b154a61
Update searx.data - update_currencies.py (#2916) 
Co-authored-by: dalf <dalf@users.noreply.github.com>
 2021-08-03 21:17:05 +02:00
Allen ef246b7661
[enh] Add Petalsearch.com engine (xpath) (#2897) 
* [enh] Add Petalsearch.com engine (xpath)

Upstream example query:
https://petalsearch.com/search?query=Felis%20lynx%20Linnaeus&sregion=de&pn=1

* Fix Petalsearch title_xpath

Fixes the title_xpath from Petalsearch which got changed in the meantime.
 2021-08-03 21:15:34 +02:00
Allen 36cf794cfa [enh] Add woxikon.de synonyme (xpath) 
Upstream example query:
https://synonyme.woxikon.de/synonyme/test.php
 2021-08-02 21:58:17 +02:00
Allen e83c5fd0fd [enh] Add gpodder.net (JSON) 
Upstream query example:
https://gpodder.net/search.json?q=linux
 2021-08-02 21:56:59 +02:00
Allen ec4e48e5df [enh] Add pagination support for Library Genesis 
[enh] Add pagination support for Library Genesis and add it to "files" category too.
 2021-08-02 21:52:12 +02:00
Émilien Devos ee443d9739 Fix google images 
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010

Closes #2914
 2021-08-02 20:14:54 +02:00
Noémi Ványi bb724cabf9
Merge pull request #2898 from allendema/ask.com 
[enh] Add ask.com engine
 2021-08-02 20:12:49 +02:00
Samuel Dudík 13a608b0b8
Fix Seznam engine (#2905) 
## What does this PR do?

Fixes the Seznam engine by updating XPath strings.

## Why is this change important?

Without this PR Seznam returns no results.

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
 2021-08-02 20:09:31 +02:00
Marc Abonce Seguin a5839a66d6
Update onion engines to v3 (#2904) 
downgrade httpx:
PR https://github.com/encode/httpx/pull/1522
made some breaking changes in AsyncHTTPTransport that affect
our code in https://github.com/searx/searx/blob/master/searx/network/client.py

remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
 2021-08-02 20:03:55 +02:00
Allen 1d60d4253a
[enh] Add Kaufland.de 
thumbnail_xpath not working currently.
 2021-07-29 13:34:58 +02:00
Allen 573f91143c
[enh] Add ask.com engine 2021-07-19 18:24:39 +02:00
Adam Tauber a8988885a5 [fix] pep8 2021-07-05 19:57:53 +02:00
Adam Tauber 198aad43e0 [enh] add mongodb offline engine 2021-07-05 19:47:15 +02:00
Adam Tauber b2ed65eb6d [fix] initialize redis engine at the right time 2021-07-05 19:40:51 +02:00
Adam Tauber 2af7c60598 [mod] disable wikimini engine by default 2021-07-05 19:40:51 +02:00
Markus Heiser cbc50a9bc5 [fix] google-news engine - KeyError: 'hl in request 
Since we added

- 1c67b6aec [enh] google engine: supports "default language"

there is a KeyError: 'hl in request,error pattern::

    ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl'
    Traceback (most recent call last):
      File "searx/search/processors/online.py", line 144, in search
        search_results = self._search_basic(query, params)
      File "searx/search/processors/online.py", line 118, in _search_basic
        self.engine.request(query, params)
      File "searx/engines/google_news.py", line 97, in request
        if lang_info['hl'] == 'en':
      KeyError: 'hl'

Closes: https://github.com/searxng/searxng/issues/154
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-07-03 16:53:31 +02:00
Markus Heiser 98a63058e5 [fix] google answers: normalize space of the answers. 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-07-03 16:53:31 +02:00
Markus Heiser 412677d495 [mod] google engine: reduce mobile UI parameters to what is needed 
Reverse engineering shows that not all of the parameters used by google's mobile
UI (aka "more results" button) are needed [1].

[1] https://github.com/searxng/searxng/pull/160#issuecomment-865013625

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-07-03 16:53:31 +02:00
Alexandre Flament 8bf216eab6 [mod] google: add "use_mobile_ui" parameter to use mobile endpoint. 
disable by default, it has to be enabled in settings.yml

related to  #159
 2021-07-03 16:53:31 +02:00
Alexandre Flament 3863f5a83f [enh] google engine: supports "default language" 
Same behaviour behaviour than Whoogle [1].  Only the google engine with the
"Default language" choice "(all)"" is changed by this patch.

When searching for a locate place, the result are in the expect language,
without missing results [2]:

  > When a language is not specified, the language interpretation is left up to
  > Google to decide how the search results should be delivered.

The query parameters are copied from Whoogle.  With the ``all`` language:

- add parameter ``source=lnt``
- don't use parameter ``lr``
- don't add a ``Accept-Language`` HTTP header.

The new signature of function ``get_lang_info()`` is:

    lang_info = get_lang_info(params, lang_list, custom_aliases, supported_any_language)

Argument ``supported_any_language`` is True for google.py and False for the other
google engines.  With this patch the function now returns:

- query parameters: ``lang_info['params']``
- HTTP headers: ``lang_info['headers']``
- and as before this patch:
  - ``lang_info['subdomain']``
  - ``lang_info['country']``
  - ``lang_info['language']``

[1] https://github.com/benbusby/whoogle-search
[2] https://github.com/benbusby/whoogle-search/releases/tag/v0.5.4
 2021-07-03 16:53:31 +02:00
dalf 66206bfb36 Update searx.data - update_ahmia_blacklist.py 2021-07-03 15:31:09 +02:00
dalf 5bfcc120ba Update searx.data - update_wikidata_units.py 2021-07-03 15:21:20 +02:00
dalf 5c57f83ac6 Update searx.data - update_currencies.py 2021-07-03 15:21:01 +02:00
dalf 9e7a68480c Update searx.data - update_currencies.py 2021-06-03 22:13:13 +02:00
dalf 89acf2462a Update searx.data - update_wikidata_units.py 2021-06-03 22:03:36 +02:00
dalf 3033b3297f Update searx.data - update_firefox_version.py 2021-06-03 21:48:45 +02:00
dalf fc5973cf95 Update searx.data - update_ahmia_blacklist.py 2021-06-03 21:44:22 +02:00
Jordan Webb 60ad4118d6
Add json_engine configuration for ManKier 2021-05-31 13:54:50 -05:00
Noémi Ványi 0267563970
Merge pull request #2830 from jordemort/pypi 
Add xpath configuration for PyPI
 2021-05-31 20:36:43 +02:00
Adam Tauber c8d2b5eb34 [doc] add info about redis engine dependency 2021-05-30 19:25:26 +02:00
Adam Tauber 01a8a5814a [fix] pylint 2021-05-30 19:25:03 +02:00
Adam Tauber ea7ccf2422 [fix] correct kv template formatting and remove internal data 2021-05-30 19:20:46 +02:00
Adam Tauber 97269be680 [enh] add redis offline engine 2021-05-30 19:20:17 +02:00
Jordan Webb 66d06b05fe
Add xpath configuration for PyPI 2021-05-28 16:32:32 -05:00
Allen 28e4ef9173
Adds Dogpile as an engine (#2822) 
* Add Dogpile engine

Example Query:
[https://www.dogpile.com/serp?q=streisand+effect](https://www.dogpile.com/serp?q=streisand+effect)

* Remove double "engines: " text
 2021-05-26 23:39:32 +02:00
Noémi Ványi c486adf8f7 Minor fixes to wikimini engine 2021-05-26 23:34:25 +02:00
LL Productions FR bed044cc62 Add Wikimini 
Fixed a mistake

Update settings.yml

Squashed 3 commits that is for adding Wikimini to Searx
 2021-05-22 21:38:00 +02:00
Markus Heiser 0647b34b1d [fix] engine archive is - search_url has been changed 
BTW: set soft_max_redirects from *archive is* by 1, to prevent logging::

    DEBUG:httpx._client:HTTP Request: GET https://archive.is/search/?q=www.python.org "HTTP/2 302 Found"
    DEBUG:httpx._client:HTTP Request: GET https://archive.is/www.python.org "HTTP/2 200 OK"
    DEBUG:searx:archive is: ErrorContext('searx/search/processors/online.py', 110, 'count_error(self.engine_name,', None, '1 redirects, maximum: 0', ('200', 'OK', 'archive.is')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-22 20:42:24 +02:00
Markus Heiser 650a1c0b89 [enh] xpath engine - add request parameter 'soft_max_redirects' 
Make 'soft_max_redirects' configurable per Xpath engine::

    - name : <engine-name>
      engine : xpath
      soft_max_redirects: 1
      ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-22 20:41:36 +02:00
Adam Tauber 9b5415ea2f [mod] disable /stats page by default to prevent potential data leak 2021-05-13 22:21:46 +02:00