1 Milestones
Alexandre Flament edited this page 2021-01-17 18:06:35 +01:00

Searx milestones

Milestone 1.1 - Clean up

Milestone 1.2 - async

  • Replace requests by httpx or aiohttp

See https://github.com/searx/searx/pull/1856: use httpx instead of requests. Drop source ip rotation, proxy support in this milestone.

Milestone 1.4 - network

  • IP rotation per engine (for now it is per request even on different engines).
  • allow to specify an IP range (useful for IPv6). Related to searx#1034 (would be better to detect IPv6 support to avoid maintenance, dnspython can help).
  • allow to specify a list of proxy.
  • being able to define a retry policy.
  • what will take time here is test, test, test.

Milestone 1.5 - better statistics about the engines

Updated version of PR measure response time with more details. #447 (see #162#issuecomment-76623027 ).

Records accurate statistics, display graphics about them (produce graph on the server to avoid javascript usage.).

Milestone 1.6 - infoboxes engines

The wikidata engine is the default engine to display infoboxes. Unfortunately, it is slow, the duckduckgo_definition is faster but requires some work to provide more user friendly informations.

  • improve response time of the wikidata engine. Define helpers:
    • load all property name translations at load time (one SPARQL request).
    • build one big SPARQL request template at load time.
    • use the big SPARQL request instead of asking for the HTML version.
    • parse JSON with a functionToApply[propertyName]
  • improve the results of the ddd engine:
    • define common data_type

Milestone 1.7 - better engine framework

  • Apply this : #302#issuecomment-565828553 (the issue, but not the comment, is included in the version 1.0). See this gist : https://gist.github.com/dalf/3c3904699153a741f27842d8ea30b449
  • #1802 : Engine code: describe which XPath can fail, which must not. The idea: if an engine fails, we should know why: missing XPath result, missing JSON result, internal error, unexpect data, etc... --> if I sum up: the purpose is it create a better framework / toolset for the engines. It will take some times to review all the engines and find what kind of error to report (the purpose is to not fix them, but to be able to report the errors). For example: issue a warning if there is a unexpected HTTP redirect.
  • Integrate https://github.com/searx/searx-checker into searx: see #1559 : Add some code directly into the engine to make sure that they are working as expecting. For example, list some request that should work, and the expected results. Most probably it should be code rather that data because each engine behaves differently. So CI can include a report.
  • Expose the errors to a public API so searx-stats2 can collect them (for example: this XPath in this engine fails 40% of the time). Triple check that everything is anonymous.

Milestone 1.8 - framework for the the on_result plugins

Related issue: #2080

The on_result plugins can define some triggers: searx calls the "on_result" functions only when the host match.

Milestone 1.9 - autocomplete

  • #392 : include answers in the autocomplete results.
  • autocomplete with the external bangs.

Milestone 1.10 - data upgrade & build process

Related issues:

Check with different the searx package maintainers.

Milestone 1.11 - upgrade oscar theme

  • Upgrade to the dependencies (jquery, bootstrap, leaftlet, etc...)
  • Drop IE support
  • Optimize some of the HTML <--- see performance on FF, Chrome, mobile, desktop:

Documentation & packaging milestones

Milestone 2.1 - common configuration files

Find a way to have a reference configuration. Currently about the filtron configuration, there are 3 versions:

The purpose is to ensure the default setup is secured (HTTP headers, know working filtron configuration, etc...). If it is updated, it is updated everywhere.

Same can be done about the reverse proxy configuration / HTTP headers.