devol
/
searx
mirror of https://github.com/searx/searx
1
0
Fork 0
Code Issues Releases Wiki Activity
3,975 Commits 19 Branches 18 Tags 56 MiB
Commit Graph

241 Commits

Author SHA1 Message Date
Adam Tauber 8e727ac77f [fix] update csv unit test 2020-03-13 01:05:02 +01:00
Noémi Ványi 99435381a8 [enh] introduce private engines 
This PR adds a new setting to engines named `tokens`.
It expects a list of tokens which lets searx validate
if the request should be accepted or not.
 2020-02-08 11:47:39 +01:00
Adam Tauber 52ccaa7acc [mod] remove useless engine unit tests 
These tests are not able to detect engine errors if the upstream
site changes.
 2019-12-21 21:15:09 +01:00
Marc Abonce Seguin ccaf6ca02c [fix] update xpaths for new google results page 2019-12-07 16:37:24 -07:00
Marc Abonce Seguin 9299355570 add seedpeer again 2019-11-24 22:01:44 -07:00
Adam Tauber 3c425f09c1 [fix] remove useless engine tests 2019-10-16 15:32:21 +02:00
Adam Tauber e74bdf8429 [fix] engine test 2019-10-14 15:09:25 +02:00
Léo Bourrel 88261e111c Fix bing engine results count (#1387) 
This PR fixes the result count from bing which was throwing an (hidden) error and add a validation to avoid reading more results than avalaible.

For example :
If there is 100 results from some search and we try to get results from 120 to 130, Bing will send back the results from 0 to 10 and no error. If we compare results count with the first parameter of the request we can avoid this "invalid" results.
 2019-08-05 16:15:40 +02:00
Dalf fcc9587ee9 [fix] fdroid engine 2019-08-05 15:44:02 +02:00
Dalf 9ff5001816 [fix] arxiv engine 2019-08-05 15:43:01 +02:00
Alexandre Flament 333e54943d
[fix] fix monkey patch in test_webapp.py (#1667) 
at the end of test_webapp.py, the monkey patch of searx.search.Search was not revert which lead to side effects on other tests
close #1663
 2019-08-03 13:23:36 +02:00
Alexandre Flament 72029d27de
[enh] Add timeout limit per request (#1640) 
The new url parameter "timeout_limit" set timeout limit defined in second.
Example "timeout_limit=1.5" means the timeout limit is 1.5 seconds.

In addition, the query can start with <[number] to set the timeout limit.

For number between 0 and 99, the unit is the second :
Example: "<30 searx" means the timeout limit is 3 seconds

For number above 100, the unit is the millisecond:
Example: "<850 searx" means the timeout is 850 milliseconds.

In addition, there is a new optional setting: outgoing.max_request_timeout.
If not set, the user timeout can't go above searx configuration (as before: the max timeout of selected engine for a query).

If the value is set, the user can set a timeout between 0 and max_request_timeout using
<[number] or timeout_limit query parameter.

Related to #1077
Updated version of PR #1413 from @isj-privacore
 2019-08-02 13:50:51 +02:00
Alexandre Flament 2179079a91
[fix] fix flickr_noapi decoding (#1655) 
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
 2019-08-02 13:37:13 +02:00
Dalf 6e0285b2db [fix] wikidata engine: faster processing, remove one HTTP redirection. 
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection)
* url_detail: remove the disabletidy=1 deprecated parameter
* Add eval_xpath function: compile once for all xpath.
* Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}')
* Create an etree.HTMLParser() instead of using the global one (see #1575)
 2019-07-29 07:39:39 +02:00
Frank de Lange cbc5e13275 [enh] flickr_noapi: use complete JSON data block, add 'content', 'img_format', 'source', etc. (#1571) 
Fetch complete JSON data block, use legend to extract images. 
Unquote urlencoded strings.
Add image description as 'content'. 
Add 'img_format' and 'source' data (needs PR #1567 to enable this data to be displayed). 
Show images which lack ownerid instead of discarding them.
 2019-07-28 10:42:00 +02:00
Frank de Lange 204a2cbbf0 [fix] bing_videos (#1579) 
use JSON where possible, compose 'content' using all available data, use correct 'url' (direct to source instead of redirect through bing)
 2019-07-27 17:49:30 +02:00
Frank de Lange 11fc9913e9 [enh] bing_images: use data from embedded JSON to improve results (e.g. real page title) (#1568) 
use data from embedded JSON to improve results (e.g. real page title), add image format and source info (see PR #1567), improve paging logic (it now works)
 2019-07-27 08:22:02 +02:00
volth eb182df132 [mod] restore btdigg engine as btdig.com (#1515) 2019-07-25 08:40:48 +02:00
rachmadani haryono 3b1122c5fa [fix] fix duden engine (#1594) 2019-07-25 08:17:45 +02:00
Alexandre Flament 554a21e1d0
[enh] Add Server-Timing header (#1637) 
Server Timing specification: https://www.w3.org/TR/server-timing/

In the browser Dev Tools, focus on the main request, there are the responses per engine in the Timing tab.
 2019-07-17 10:38:45 +02:00
rachmadani haryono ec88fb8a0f [fix] secret_key can be bytes instead of a string (#1602) 
Fix #1600
In settings.yml, the secret_key can be written as string or as base64 encoded data using !!binary notation.
 2019-07-17 10:09:09 +02:00
rachmadani haryono 8f44014627 [fix] preference query parameter decoding (#1599) 
Fix issue #1598
 2019-07-17 09:42:40 +02:00
rachmadani haryono ac357b12e3
Merge branch 'master' into feature/fix-config 2019-05-28 19:16:58 +08:00
Dalf ffe0972f91 Remove some engines : subtitleseeker, seedpeer, swisscows 
http://www.subtitleseeker.com and http://www.seedpeer.eu don't exist anymore.
https://swisscows.ch/ has change : the engine needs to be updated
 2019-05-28 04:06:35 +02:00
rachmadaniHaryono 9afc1b1e83 new: dev: test for config api 2019-05-18 18:58:05 +08:00
Marc Abonce Seguin 3e1c2153f7 [fix] duckduckgo images requests 2019-04-13 00:38:37 -05:00
Marc Abonce Seguin f2d49a6971 [fix] get youtube results from js object 
Results are not appearing in the html document anymore,
instead they are found inside an object embedded in a script.
 2019-03-26 21:09:15 -06:00
d-tux f1814079f0
Merge branch 'master' into engines/unsplash 2019-01-14 13:40:57 +01:00
Marc Abonce Seguin 626a8e9ac9 [fix] unicode error with WolframAlpha API engine 2019-01-08 21:02:23 -06:00
d-tux 329172f66e
Merge branch 'master' into engines/unsplash 2019-01-08 09:24:45 +01:00
Noémi Ványi 97351a2c72 fix after rebase 2019-01-07 21:28:58 +01:00
Noémi Ványi b63d645a52 Revert "remove 'all' option from search languages" 
This reverts commit 4d1770398a.
 2019-01-07 21:19:00 +01:00
Marc Abonce Seguin 0169b63e84 [fix] fetch google's supported languages 2019-01-06 21:31:45 -06:00
Marc Abonce Seguin 5568f24d6c [fix] check language aliases when setting search language 2019-01-06 20:31:57 -06:00
Noémi Ványi 1ea56576dc
Merge branch 'master' into devel_google_videos 2019-01-04 22:27:05 +01:00
Noémi Ványi 5b81f7c2cc
Merge branch 'master' into bugfix_startpage 2019-01-04 22:03:00 +01:00
Venca24 0e493db2fb [fix] google videos test 2019-01-04 16:04:05 +01:00
Léo Bourrel 73ec2ba74e Update bing image engine with new image json format 2018-12-19 11:12:25 +01:00
Michael Pfitzner 83a06d276d change rest of test_startpage.py 2018-12-14 22:29:57 +01:00
Michael Pfitzner 5a20d9ecd1 change the html in test_startpage.py to look more like the actual startpage site 2018-12-14 22:18:21 +01:00
Venca24 cee15f0375 [fix] google videos test 2018-11-22 13:18:18 +01:00
Denis Wernert b9ada93b3a Removes what looks like tracking parameters 2018-10-08 14:56:20 +02:00
Denis Wernert ee07a5e750 Adds a unit test for the unsplash engine 2018-10-08 14:01:55 +02:00
Adam Tauber baacfac32b [fix] remove publicly unavailable 500px engine - #1338 2018-08-19 15:06:59 +02:00
dadosch b575f898c0 duden.de engine 2018-08-18 19:24:02 +02:00
Léo Bourrel 0a37f90990 Fix wikidata tests with updated path to media 2018-07-06 11:15:43 +02:00
Adam Tauber a3b83b73cb [fix] remove obsolete test 2018-06-14 11:47:42 +02:00
Noémi Ványi 7fd7bd75ff follow up tests 2018-05-21 18:15:49 +02:00
rinpatch 059c2ccb95
Fix acgsou encoding error on python2 and add unicode characters to unit testt 2018-05-05 23:41:13 +03:00
rinpatch b7eb05dac6
Fix unit test url 2018-05-03 15:20:48 +03:00
rinpatch a79c676f27
Remove Japanese characters 2018-04-27 17:38:52 +03:00
rinpatch 1ac0c90037
Fix unit test 2018-04-27 17:33:23 +03:00
rinpatch dcc9fdb47f
Added unit test 2018-04-27 15:36:15 +03:00
rinpatch 06e070aee2 Fix string length 2018-04-23 12:55:25 +03:00
rinpatch c40d8e1d15 Unit test 2018-04-23 11:02:32 +03:00
Marc Abonce Seguin f7f9c50393 [fix] force English results in Google when using en-US 2018-04-18 23:29:48 -05:00
Marc Abonce Seguin 9687786226 update unit tests for google news and wikidata 2018-04-08 23:31:24 -05:00
Marc Abonce Seguin 772c048d01 refactor engine's search language handling 
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
 2018-03-27 00:08:03 -06:00
Noémi Ványi 3ef8533f4d fix unit tests 2018-02-10 20:03:55 +01:00
marc 4d1770398a remove 'all' option from search languages 2017-12-06 01:20:15 -06:00
Adam Tauber 9e79ad4bde [fix] pdbe test 2017-12-01 20:58:40 +01:00
Adam Tauber 060c797739 [fix] change domain name in google engine test 2017-11-22 23:06:24 +01:00
pyrrh0n1c a1d1a50149 Fixed unittests. 2017-11-07 14:49:10 +00:00
pyrrh0n1c 4340c0b16c Fixed typos. 2017-11-07 14:14:20 +00:00
Adam Tauber 00a7041daa
Merge pull request #1068 from Apply55gx/genius 
Continuation of PR #1004 (seems abandoned)
 2017-11-03 20:21:25 +01:00
Noémi Ványi 5954a8e16a minor fix of BASE engine 2017-11-01 17:02:38 +01:00
jibe-b df0d915806 [add] pubmed engine 2017-11-01 17:02:38 +01:00
Noémi Ványi 9c2b7a82f0 minor fixes of arxiv 
Closes #1050
 2017-11-01 14:22:22 +01:00
jibe-b 3e3672e079 [add] arxiv engine 2017-11-01 11:54:16 +01:00
Apply55gx d800e3fcfa Merge pull request #1 from asciimoo/master 
-
 2017-10-25 10:44:28 +02:00
marc 54f0ab1f3b test fetch langauges function on bing images 2017-10-10 18:37:24 -05:00
marc 44085e31d0 update engines_languages.json and languages.py 
Also, fix fetch_languages.py so it can run on python3.
 2017-10-10 16:53:28 -05:00
marc a524dbb823 [fix] language support for bing images and videos 2017-10-10 16:49:49 -05:00
jibe-b d081eee3f8 [add] unit_test for base engine 2017-09-22 22:26:37 +02:00
misnyo c3232b0e1a Merge branch 'master' into nyaa 2017-09-04 17:48:25 +02:00
misnyo 01330f71cd [fix] nyaa.si fixed 2017-08-31 21:32:30 +02:00
misnyo 3182ba7069 [fix] google news dom xpath fix 2017-08-31 17:48:07 +02:00
Adam Tauber e74aaa781e Merge pull request #1016 from misnyo/blekko 
[mod]blekko images removed
 2017-08-30 21:48:36 +02:00
misnyo 0607b167f8 [fix] faroo json api and image layout fixed 2017-08-30 21:09:03 +02:00
misnyo f139f9cfdb [mod]blekko images removed 2017-08-30 16:47:56 +02:00
misnyo bf3f9a91fa [fix]torrentz unit_tests timezone fix 2017-08-29 17:54:41 +02:00
misnyo 2098e1b7ff [fix]torrentz unit_tests timezone fix 2017-08-25 01:11:02 +02:00
misnyo d14e2781b2 [fix]torrentz search engine fixed for new version 2017-08-25 00:52:35 +02:00
woorst 636b760196 remove unicode characters 2017-08-21 11:45:23 -05:00
woorst 62b2f79ce7 fix line lengths for pep8 standards 2017-08-20 21:10:51 -05:00
woorst 2434c29dc5 New engine: Genius (lyrics) 2017-08-19 19:05:17 -05:00
marc 3ca9cad927 add bing videos engine 2017-08-05 20:35:57 -05:00
marc 856dfc3018 add google videos 2017-07-26 15:49:16 +02:00
Noémi Ványi 2242000bd4 change unresponsive_engines to a set to eliminate duplication of errors 2017-07-20 13:30:56 +02:00
Noémi Ványi 243d3e4298 show engine errors in infobox && add new error alert 2017-07-12 16:29:11 +02:00
marc 405e5c8f24 [fix] duckduckgo images doesn't fail with countryless language 2017-07-10 18:41:50 +02:00
marc 3b950929b1 [fix] fix duckduckgo's offset 
First page now starts with 0 offset,
rather than starting on the 30th result.

DuckDuckGo returns 30 results on each page.
 2017-05-27 21:12:39 -05:00
marc c65a409f0d add duckduckgo images engine 2017-05-23 20:07:09 +02:00
Adam Tauber 8db527c1d2 [fix] use raw response with etree.parsefromstring - Unicode strings with encoding declaration are not supported 2017-05-22 15:36:52 +02:00
marc 9ee8e552da [fix] bing images 2017-05-17 15:45:55 +02:00
Alexandre Flament f5128c7cb9 [mod] add/modify image fetching for bing_news, qwant and twitter engines 2017-05-15 17:28:21 +02:00
Adam Tauber 52e615dede [enh] py3 compatibility 2017-05-15 12:02:30 +02:00
François Revol 45d15bd6f0 Add framalibre engine 
framalibre.org is a catalogue of Free Software, edited by Framasoft.

For now we pass the thumbnail as img_src as it doesn't seem to be used
for IT...
 2017-03-22 00:18:23 +01:00
marc 805fb02ed1 add language support for qwant 
closes issue #863
 2017-03-18 23:44:21 +01:00
Alexandre Flament e67dfaaac7 Merge branch 'master' into flask_perimeter 2017-01-27 13:47:59 +00:00