0 Brainstorming: IDE & database
Alexandre Flament edited this page 2020-07-16 18:29:08 +02:00

The engines will keep failing. More checking at runtime can help to detect the errors, but it won't solve the issues. We lack time and hands to review all the changes.

Searx online IDE

Currently, to update an engine you have: git clone searx, install the dependencies, edit some files, make run, git commit, create a pull request. Same about the review: get the code, checks the different options, approve / comment.

What if we could make it more simple for new contributors ? Like editing a Jupyter notebook ?

More precisly, the idea is to have the "edit this file" button of Github on steroid:

  • starts an online IDE with autocompletion.
  • lxml and other dependencies are availabled.
  • there is a "run" button which allow to try the code (with the all the parameters).
  • a save button creates a PR in the git repository.
  • like github comments the code is automaticaly saved in the browser.

For the json and xpath engines won't even need an online IDE: just some text inputs, and a "try" button. Later, a tool similar to wuzz can help for example.


  • Python can run inside a browser using WASM, but it may not work as expected. So, a Python inside a container would do the job:
  • The idea, here is to provide an online sandbox to try searx engine online. To avoid abuse, a Github OAuth anthentification would be required (so there is no need for an additional account).
  • The IDE: fork of Jupyter or and monaco with language server ?
  • The result can be saved as a PR in the git repository.
  • Some engines reuse code from other engines. Example: the google engine, duckduckgo_images.

Searx engine database

Looking at the issue What is the data lifecycle ? #2052, there is another way to deal with that.

Why not make something similar to wikidata but only for searx: one entry per engine (and entries for currencies, DOI). The code could even be one field on a entry.

From the engine developer point of view:

  • a "page" per engine (similar to https://www.wikidata.org/wiki/Q12805 for example, but only with the Searx revelant data).
  • there are different fields, like engine name, bang shortcut name, categories, etc...
  • the code is one the field. An online IDE allows to edit the code and try it online.
  • save

Someone reviews the change request, and approve /deny.

Authentification can use OAuth from github.

The purpose here is to simply the maintenance/edit process to the maximum.

Disclaimers. I'm aware that:

  • Even if a framework like django can help a lot, it still a lot of work.
  • The 10 "write/update engine" tasks may turn into more than 10 "review" tasks.

Database model

Class Engine:

  • official URL
  • bang URL
  • autocomplete URL
  • Category / Subcategory
  • Short name
  • Engine name
  • favicon.ico
  • engine definition

Class Engine definition

  • version
  • default timeout
  • languages (equivalent of searx/data/engines_languages.json)

Class Engine XPath(Engine definition)

  • search_url
  • url_xpath
  • title_xpath
  • content_xpath

Class Engine JSON(Engine definition)

  • search_url
  • url_query
  • content_query
  • title_query
  • paging
  • suggestion_query
  • results_query

Class Engine Python(Engine definition)

  • code

Class Engine DSL(Engine definition)

  • code

Database storage

The storage can be a public git repository, so the backup is free.


Scenario - An user adds a new engine

A user adds a search engine:

  • URL (mandatory)
  • Category (and Subcategory as DuckDuckGo) (optional)
  • Searx name & shortcut (optional)

A backend looks for:

  • opensearch.xml or sitelinks search box --> free external bang update/addition
  • favicon.ico
  • web site name

Add to engine definition:

  • search URL
  • autocomplete URL
  • favicon
  • name

Scenario - An user edits an existing engine

Scenario - An user adds XPath engine

Scenario - An user moderate an change