Commit Graph

472 Commits

Author SHA1 Message Date
Kian-Meng Ang 629ebb426f
Fix typos (#3366)
Found via `codespell -S ./searx/translations,./searx/data,./searx/static -L ans,te,fo,doubleclick,tthe,dum`
2022-09-29 23:06:59 +02:00
Adam Tauber 2222caec22 [enh] add omnom engine 2022-09-20 23:04:25 +02:00
Rob 9e8995e13d
[fix] brave engine: no search results (#3361)
There were never any results from the Brave search engine
so fixed url_xpath and now Brave search results are working.
2022-09-18 19:32:23 +02:00
Elena Poelman ca27e91594
New search engine: IPFS search (#3218)
* Feat: initial support for the ipfs-search engine

* Feat: enable paging for ipfs search

* Make ipfs-search code more readable

* added support for images, music and video to the ipfs search engine

* FIX: redefined some variables that where redefining built-ins

* adjust code so it works on older python versions

* Feat: add support for time ranges
2022-09-07 22:13:19 +02:00
Ben Collerson 465bbd4402
[enh] archive.today option for results page (#3308)
Adds an option to the oscar theme that puts an archive.today link next to the web.archive.org cache link. In preferences change 'Show archive.today links' to 'On'
2022-07-31 16:37:00 +02:00
Ben Collerson 78a87caa0f
[fix] make autofocus configurable (#1984) (#3285)
* [fix] make autofocus configurable (#1984)

"Results page: having text cursor still in search field is annoying #1984"

Allows autofocus on the results page to be configured either in the UI
or in the searx server config.

* fix commented code

Co-authored-by: Ben Collerson <benc@benon.com>
2022-07-30 22:28:41 +02:00
Noémi Ványi 05fe2ee093
pick engine fixes (#3306)
* [fix] google engine: results XPath

* [fix] google & youtube - set EU consent cookie

This change the previous bypass method for Google consent using
``ucbcb=1`` (6face215b8) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] https://github.com/iv-org/invidious/pull/2207
[2] https://github.com/TeamNewPipe/NewPipeExtractor/issues/592

Closes: https://github.com/searxng/searxng/issues/1432

* [fix] sjp engine - convert enginename to a latin1 compliance name

The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] engine tineye: handle 422 response of not supported img format

Closes: https://github.com/searxng/searxng/issues/1449
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* bypass google consent with ucbcb=1

* [mod] Adds Lingva translate engine

Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.

* openstreetmap engine: return the localized named.

For example: display "Tokyo" instead of "東京都" when the language is English.

* [fix] engines/openstreetmap.py typo: user_langage --> user_language

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* Wikidata engine: ignore dummy entities

* Wikidata engine: minor change of the SPARQL request

The engine can be slow especially when the query won't return any answer.
See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Emilien Devos <contact@emiliendevos.be>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Emilien Devos <github@emiliendevos.be>
Co-authored-by: ta <alt3753.7@gmail.com>
Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-07-30 21:45:07 +02:00
Noémi Ványi 85034b49ef
Remove `httpx` and use `requests` instead (#3305)
## What does this PR do?

This PR prepares for removing `httpx`, and reverts back to `requests`.

## Why is this change important?

`httpx` hasn't proven itself to be faster or better than `requests`. On the other
hand it has caused issues on Windows.

=============================================
Please update your environment to use requests instead of httpx.
=============================================
2022-07-30 20:56:56 +02:00
james-still 210e59c68c
Add engine for Emojipedia (#3278) 2022-07-28 21:45:07 +02:00
Noémi Ványi f00d9e0ec4
Pick minor fixes from searxng (#3251)
* [fix] Rename ccengine engine to openverse

The CC engine was merged with WordPress and renamed to Openverse

Source: https://wordpress.org/news/2021/05/welcome-to-openverse/

* [fix] ccengine engine - avoid unwanted redirects

api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:

    /v1/images? -->/ v1/images/?

otherwise it redirects, here is the debug log:

    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
    WARNING searx.engines.openverse       : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

* [fix] FutureWarning from lxml

Just in case if content is None, the original code will skip extract_text(), and
just append the None value to 'content'. So just add allow_none=True, and this
will return None without raising a ValueError in extract_text().

* [enh] Add pagination to Brave

Also added ```&spellcheck=1``` because now it is disabled by default, not returning any ```suggestion_xpath```.

Co-authored-by: Léon Tiekötter <leon@tiekoetter.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: capric98 <42015599+capric98@users.noreply.github.com>
Co-authored-by: Allen <64094914+allendema@users.noreply.github.com>
2022-06-06 00:01:27 +02:00
liimee a3e41c3cd6
Add TVmaze engine (#3246) 2022-06-05 23:36:04 +02:00
Markus Heiser f231d79a5d [fix] engine: Semantic Scholar (Science) // rework & fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-05 20:59:11 +01:00
Noémi Ványi a164585118 Add extra features to Gigablast engine:
* fast can be enabled to results are returned quicker
* collection can be configured
* search_type can be changed to images or news

Closes #3078
2022-01-22 19:14:45 +01:00
iko 01e28757d3
Fixed Hoogle engine (#3146) 2022-01-22 18:22:24 +01:00
Allen 0c351ea364
[enh] Add Tineye reverse image search (#3040)
* [enh] Add Tineye reverse image search 

Other optional parametesr:

"&sort=crawl_date" can be appended to search_string to sort results by date.
"&domain=example.org" can be implemented to search_string to get results from just one domain.

Public instances could get relatively fast timed-out for 3600s.

* [enh] Add TIneye to settings.yml 

Check if that's the right shortcut.

* [mod] Fix checks

* [mod] Try to fix checks

* [mod] Use Four spaces for indentation

And set paging back to True

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-01-22 12:15:19 +01:00
Dario Nuevo 1a18adcc16
New files engine: Prowlarr (#3118)
## What does this PR do?

Gives the user the possibility to search their own prowlarr instances.

Info: https://wiki.servarr.com/en/prowlarr
Github: https://github.com/Prowlarr/Prowlarr

## Why is this change important?

Prowlarr searchs multiple upstream search providers, thus allows to use that functionality through searx.
2022-01-15 19:18:15 +01:00
Allen 0c2165324d
[Fix] Add suggestions + Fix xpaths (#3082)
* [mod] Add Suggestion to Petalsearch

* [Fix] Changed xpath for Petalsearch
2022-01-07 21:49:08 +01:00
Émilien Devos 8cde08ded2
Disable onesearch by default (#3099)
onesearch is not available everywhere and thus display an error by default in searx
2022-01-07 21:42:51 +01:00
israelyago b90616a25f
Remove categories from onesearch config
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2021-11-18 08:19:19 -03:00
israelyago 6b3915a2dc
Removed paging from onesearch config
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2021-11-18 08:18:50 -03:00
israelyago 0d28fd2efe
Merge branch 'master' into onesearch-engine 2021-11-17 15:27:11 -03:00
Israel Yago Pereira 4b785677d8 Onesearch pagination 2021-11-17 15:14:43 -03:00
Israel Yago Pereira 258c6fbd5a Onesearch engine without pagination 2021-11-17 15:14:43 -03:00
Israel Yago Pereira 8e00249633 WIP: onesearch engine 2021-11-17 15:14:43 -03:00
Finn 8c3454fd1b
[enh] Improve ranking based on language (#3053)
Add configurable setting to rank search results higher when part of the
domain (e.g. 'en' in 'en.wikipedia.org' or 'de' in 'beispiel.de')
matches the selected search language. Does not apply to e.g. 'be' in
'youtube.com'.

Closes #206
2021-11-15 20:31:22 +01:00
Markus Heiser 3205785059 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-10 21:08:48 +02:00
Allen b0888c6ca3
[enh] Add Pagination to Wiby / Fix Kaufland (#3000)
* [fix] Fix Kaufland engine

Changed Xpath expressions

* [enh] Remove tracking params from Kaufland results

* [enh] Add pagination to Wiby

* [fix] Properly select title_xpath

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2021-10-10 20:52:06 +02:00
Allen d91f1ede61
[mod] Fix Libgen + Uncomment Ebay and Urbandictiorany (#2986)
- Change Libgen provider and use https by default.
- Umcomment Urbandictionary but disable it by default, it is working.
- Uncomment Ebay as it is working correctly.
(For ebay in the future: base_url should be changed from settings.yml  just like peertube or invidious)
2021-10-10 15:42:43 +02:00
Paul Alcock bb34685dfa
Add IMDB support (#2980)
Closes #1145
2021-10-02 13:41:38 +02:00
Markus Heiser 0b3488158b [mod] engines - add superuser.com (Stack Exchange API)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:48:39 +02:00
Markus Heiser a130c7c7a3 [mod] engines - add askubuntu.com (Stack Exchange API)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:47:11 +02:00
Markus Heiser 5e84d670a2 [mod] replace old stackoverflow engine by Stack Exchange API v2.3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:43:51 +02:00
Markus Heiser e88bea53eb [enh] add suggestions to brave engine
Suggestions should be added too.

    suggestion_xpath: //div[@class="text-gray h6"]/a

You can try it with:

    !brave recurzuoin

Suggested-by: @allendema in https://github.com/searx/searx/issues/2857#issuecomment-904837023
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-13 21:27:04 +02:00
Markus Heiser 7075fc1324 [fix] brave engine: shows descriptions with their correct URLs
BTW add about section to the YAML configuration

It now shows descriptions with their correct URLs when there are videos in the
search results, pulling content_xpath from snippet-description instead of
snippet-content.

Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-13 21:27:02 +02:00
Markus Heiser 9afb845a00 [enh] add Brave-Search engine (XPath)
Add https://search.brave.com which was requested by @kaonashi696 [1].
This patch was suggested by @eagle-dogtooth [2].

[1] https://github.com/searx/searx/issues/2857
[2] https://github.com/searx/searx/issues/2857#issuecomment-867199241

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-13 21:26:05 +02:00
Allen 7e3a30940b
[mod] Change shortcut to azl 2021-09-05 20:52:11 +02:00
Allen 8b6d5a0e5b
[enh] Add azlyrics.com
Upstream example query: https://search.azlyrics.com/search.php?q=The+verbal+acupunture&w=lyrics&p=1
2021-08-31 01:46:55 +02:00
Finn b1e5ff3cf7
[fix] Update xpaths for petalsearch.com (#2932) 2021-08-12 18:27:15 +02:00
Allen be401469d2
[mod] Apply HTTPS where possible + fix small typo (#2922) 2021-08-07 14:56:01 +02:00
Allen 76606e7372
[enh] Add whaleslide.com (#2861)
Upstream example: https://whaleslide.com/search/web/runlevels

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2021-08-03 21:31:08 +02:00
Noémi Ványi dc99fec831 [fix] add new list element to petal so it does not overwrite previous engines 2021-08-03 21:21:28 +02:00
Allen ef246b7661
[enh] Add Petalsearch.com engine (xpath) (#2897)
* [enh] Add Petalsearch.com engine (xpath)

Upstream example query:
https://petalsearch.com/search?query=Felis%20lynx%20Linnaeus&sregion=de&pn=1

* Fix Petalsearch title_xpath

Fixes the title_xpath from Petalsearch which got changed in the meantime.
2021-08-03 21:15:34 +02:00
Allen 36cf794cfa [enh] Add woxikon.de synonyme (xpath)
Upstream example query:
https://synonyme.woxikon.de/synonyme/test.php
2021-08-02 21:58:17 +02:00
Allen e83c5fd0fd [enh] Add gpodder.net (JSON)
Upstream query example:
https://gpodder.net/search.json?q=linux
2021-08-02 21:56:59 +02:00
Allen ec4e48e5df [enh] Add pagination support for Library Genesis
[enh] Add pagination support for Library Genesis and add it to "files" category too.
2021-08-02 21:52:12 +02:00
Noémi Ványi bb724cabf9
Merge pull request #2898 from allendema/ask.com
[enh] Add ask.com engine
2021-08-02 20:12:49 +02:00
Marc Abonce Seguin a5839a66d6
Update onion engines to v3 (#2904)
downgrade httpx:
PR https://github.com/encode/httpx/pull/1522
made some breaking changes in AsyncHTTPTransport that affect
our code in https://github.com/searx/searx/blob/master/searx/network/client.py

remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
2021-08-02 20:03:55 +02:00
Allen 1d60d4253a
[enh] Add Kaufland.de
thumbnail_xpath not working currently.
2021-07-29 13:34:58 +02:00
Allen 573f91143c
[enh] Add ask.com engine 2021-07-19 18:24:39 +02:00
Adam Tauber 198aad43e0 [enh] add mongodb offline engine 2021-07-05 19:47:15 +02:00