devol
/
searx
mirror of https://github.com/searx/searx
1
0
Fork 0
Code Issues Releases Wiki Activity
4,267 Commits 19 Branches 18 Tags 56 MiB
Commit Graph

4267 Commits

Author SHA1 Message Date
Allen e83c5fd0fd [enh] Add gpodder.net (JSON) 
Upstream query example:
https://gpodder.net/search.json?q=linux
 2021-08-02 21:56:59 +02:00
Allen ec4e48e5df [enh] Add pagination support for Library Genesis 
[enh] Add pagination support for Library Genesis and add it to "files" category too.
 2021-08-02 21:52:12 +02:00
Émilien Devos ee443d9739 Fix google images 
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010

Closes #2914
 2021-08-02 20:14:54 +02:00
Noémi Ványi bb724cabf9
Merge pull request #2898 from allendema/ask.com 
[enh] Add ask.com engine
 2021-08-02 20:12:49 +02:00
Samuel Dudík 13a608b0b8
Fix Seznam engine (#2905) 
## What does this PR do?

Fixes the Seznam engine by updating XPath strings.

## Why is this change important?

Without this PR Seznam returns no results.

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
 2021-08-02 20:09:31 +02:00
Marc Abonce Seguin a5839a66d6
Update onion engines to v3 (#2904) 
downgrade httpx:
PR https://github.com/encode/httpx/pull/1522
made some breaking changes in AsyncHTTPTransport that affect
our code in https://github.com/searx/searx/blob/master/searx/network/client.py

remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
 2021-08-02 20:03:55 +02:00
Noémi Ványi 05b9ceddd5
Merge pull request #2915 from allendema/kaufland 
[enh] Add Kaufland.de
 2021-08-02 20:00:09 +02:00
Allen 1d60d4253a
[enh] Add Kaufland.de 
thumbnail_xpath not working currently.
 2021-07-29 13:34:58 +02:00
Allen 573f91143c
[enh] Add ask.com engine 2021-07-19 18:24:39 +02:00
Noémi Ványi 3b192e6387 Add blog post about NoSQL datastores 2021-07-16 12:08:55 +02:00
Adam Tauber a8988885a5 [fix] pep8 2021-07-05 19:57:53 +02:00
Adam Tauber 198aad43e0 [enh] add mongodb offline engine 2021-07-05 19:47:15 +02:00
Adam Tauber b2ed65eb6d [fix] initialize redis engine at the right time 2021-07-05 19:40:51 +02:00
Adam Tauber 2af7c60598 [mod] disable wikimini engine by default 2021-07-05 19:40:51 +02:00
Markus Heiser cbc50a9bc5 [fix] google-news engine - KeyError: 'hl in request 
Since we added

- 1c67b6aec [enh] google engine: supports "default language"

there is a KeyError: 'hl in request,error pattern::

    ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl'
    Traceback (most recent call last):
      File "searx/search/processors/online.py", line 144, in search
        search_results = self._search_basic(query, params)
      File "searx/search/processors/online.py", line 118, in _search_basic
        self.engine.request(query, params)
      File "searx/engines/google_news.py", line 97, in request
        if lang_info['hl'] == 'en':
      KeyError: 'hl'

Closes: https://github.com/searxng/searxng/issues/154
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-07-03 16:53:31 +02:00
Markus Heiser 98a63058e5 [fix] google answers: normalize space of the answers. 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-07-03 16:53:31 +02:00
Markus Heiser 412677d495 [mod] google engine: reduce mobile UI parameters to what is needed 
Reverse engineering shows that not all of the parameters used by google's mobile
UI (aka "more results" button) are needed [1].

[1] https://github.com/searxng/searxng/pull/160#issuecomment-865013625

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-07-03 16:53:31 +02:00
Alexandre Flament 8bf216eab6 [mod] google: add "use_mobile_ui" parameter to use mobile endpoint. 
disable by default, it has to be enabled in settings.yml

related to  #159
 2021-07-03 16:53:31 +02:00
Alexandre Flament 3863f5a83f [enh] google engine: supports "default language" 
Same behaviour behaviour than Whoogle [1].  Only the google engine with the
"Default language" choice "(all)"" is changed by this patch.

When searching for a locate place, the result are in the expect language,
without missing results [2]:

  > When a language is not specified, the language interpretation is left up to
  > Google to decide how the search results should be delivered.

The query parameters are copied from Whoogle.  With the ``all`` language:

- add parameter ``source=lnt``
- don't use parameter ``lr``
- don't add a ``Accept-Language`` HTTP header.

The new signature of function ``get_lang_info()`` is:

    lang_info = get_lang_info(params, lang_list, custom_aliases, supported_any_language)

Argument ``supported_any_language`` is True for google.py and False for the other
google engines.  With this patch the function now returns:

- query parameters: ``lang_info['params']``
- HTTP headers: ``lang_info['headers']``
- and as before this patch:
  - ``lang_info['subdomain']``
  - ``lang_info['country']``
  - ``lang_info['language']``

[1] https://github.com/benbusby/whoogle-search
[2] https://github.com/benbusby/whoogle-search/releases/tag/v0.5.4
 2021-07-03 16:53:31 +02:00
dependabot[bot] e628d75727 Bump httpx[http2] from 0.17.1 to 0.18.2 
Bumps [httpx[http2]](https://github.com/encode/httpx) from 0.17.1 to 0.18.2.
- [Release notes](https://github.com/encode/httpx/releases)
- [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/encode/httpx/compare/0.17.1...0.18.2)

---
updated-dependencies:
- dependency-name: httpx[http2]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-07-03 15:45:33 +02:00
dependabot[bot] 75e85f2a38 Bump sphinx-tabs from 2.1.0 to 3.1.0 
Bumps [sphinx-tabs](https://github.com/executablebooks/sphinx-tabs) from 2.1.0 to 3.1.0.
- [Release notes](https://github.com/executablebooks/sphinx-tabs/releases)
- [Changelog](https://github.com/executablebooks/sphinx-tabs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/executablebooks/sphinx-tabs/compare/v2.1.0...v3.1.0)

---
updated-dependencies:
- dependency-name: sphinx-tabs
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-07-03 15:37:10 +02:00
dalf 66206bfb36 Update searx.data - update_ahmia_blacklist.py 2021-07-03 15:31:09 +02:00
dalf 5bfcc120ba Update searx.data - update_wikidata_units.py 2021-07-03 15:21:20 +02:00
dalf 5c57f83ac6 Update searx.data - update_currencies.py 2021-07-03 15:21:01 +02:00
Noémi Ványi c5d63a5c97
Merge pull request #2837 from searx/update_data_update_currencies.py 
Update searx.data - update_currencies.py
 2021-06-03 22:17:29 +02:00
dalf 9e7a68480c Update searx.data - update_currencies.py 2021-06-03 22:13:13 +02:00
Noémi Ványi 2db2dfa874
Merge pull request #2836 from searx/update_data_update_wikidata_units.py 
Update searx.data - update_wikidata_units.py
 2021-06-03 22:12:38 +02:00
dalf 89acf2462a Update searx.data - update_wikidata_units.py 2021-06-03 22:03:36 +02:00
Noémi Ványi 0d10ad5602
Merge pull request #2835 from searx/update_data_update_firefox_version.py 
Update searx.data - update_firefox_version.py
 2021-06-03 21:56:31 +02:00
dalf 3033b3297f Update searx.data - update_firefox_version.py 2021-06-03 21:48:45 +02:00
Noémi Ványi 6b738021f7
Merge pull request #2834 from searx/update_data_update_ahmia_blacklist.py 
Update searx.data - update_ahmia_blacklist.py
 2021-06-03 21:48:08 +02:00
dalf fc5973cf95 Update searx.data - update_ahmia_blacklist.py 2021-06-03 21:44:22 +02:00
Noémi Ványi d89b42879b
Merge pull request #2829 from jordemort/mankier 
Add json_engine configuration for ManKier
 2021-06-03 21:40:37 +02:00
Jordan Webb 60ad4118d6
Add json_engine configuration for ManKier 2021-05-31 13:54:50 -05:00
Noémi Ványi 0267563970
Merge pull request #2830 from jordemort/pypi 
Add xpath configuration for PyPI
 2021-05-31 20:36:43 +02:00
Adam Tauber c8d2b5eb34 [doc] add info about redis engine dependency 2021-05-30 19:25:26 +02:00
Adam Tauber 01a8a5814a [fix] pylint 2021-05-30 19:25:03 +02:00
Adam Tauber ea7ccf2422 [fix] correct kv template formatting and remove internal data 2021-05-30 19:20:46 +02:00
Adam Tauber 97269be680 [enh] add redis offline engine 2021-05-30 19:20:17 +02:00
Jordan Webb 66d06b05fe
Add xpath configuration for PyPI 2021-05-28 16:32:32 -05:00
Noémi Ványi 22a79a4896 Add blog post about SQL servers 2021-05-27 10:29:07 +02:00
Allen 28e4ef9173
Adds Dogpile as an engine (#2822) 
* Add Dogpile engine

Example Query:
[https://www.dogpile.com/serp?q=streisand+effect](https://www.dogpile.com/serp?q=streisand+effect)

* Remove double "engines: " text
 2021-05-26 23:39:32 +02:00
Noémi Ványi c486adf8f7 Minor fixes to wikimini engine 2021-05-26 23:34:25 +02:00
LL Productions FR bed044cc62 Add Wikimini 
Fixed a mistake

Update settings.yml

Squashed 3 commits that is for adding Wikimini to Searx
 2021-05-22 21:38:00 +02:00
Markus Heiser 0647b34b1d [fix] engine archive is - search_url has been changed 
BTW: set soft_max_redirects from *archive is* by 1, to prevent logging::

    DEBUG:httpx._client:HTTP Request: GET https://archive.is/search/?q=www.python.org "HTTP/2 302 Found"
    DEBUG:httpx._client:HTTP Request: GET https://archive.is/www.python.org "HTTP/2 200 OK"
    DEBUG:searx:archive is: ErrorContext('searx/search/processors/online.py', 110, 'count_error(self.engine_name,', None, '1 redirects, maximum: 0', ('200', 'OK', 'archive.is')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-22 20:42:24 +02:00
Markus Heiser 650a1c0b89 [enh] xpath engine - add request parameter 'soft_max_redirects' 
Make 'soft_max_redirects' configurable per Xpath engine::

    - name : <engine-name>
      engine : xpath
      soft_max_redirects: 1
      ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-22 20:41:36 +02:00
Noémi Ványi cafd4cb4f8 Follow up /stats changes in unit test 2021-05-15 21:29:04 +02:00
Adam Tauber 9b5415ea2f [mod] disable /stats page by default to prevent potential data leak 2021-05-13 22:21:46 +02:00
Adam Tauber 6cd3bf376f [fix] activate pylint only for the tests 2021-05-13 22:14:55 +02:00
Noémi Ványi 0313797dfd Add sqlite engine to pylint 2021-05-13 21:47:38 +02:00