diff --git a/.buildinfo b/.buildinfo new file mode 100644 index 00000000..a6604560 --- /dev/null +++ b/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 136e2f2a5b4be30afcaa05306b334941 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/404.html b/404.html index a5aa6972..31addbfb 100644 --- a/404.html +++ b/404.html @@ -1 +1,103 @@ - + + + + + + + + + Page Not Found — Searx Documentation (Searx-1.0.0.tex) + + + + + + + + + + + + + + + +
+
+
+
+ +

Page Not Found

+

+ The page you requested does not exist. You may have followed a bad + link, or the page may have been moved or removed. +

+ Go to the overview or + search. + +

+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst b/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst index 39cd9f67..1817504c 100644 --- a/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst +++ b/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst @@ -15,8 +15,8 @@ generated and deployed at :docs:`github.io <.>`. For build prerequisites read :ref:`docs build`. The source files of Searx's documentation are located at :origin:`docs`. Sphinx -assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs-live -` to build HTML while editing. +assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs.live +` to build HTML while editing. .. sidebar:: Further reading @@ -1276,13 +1276,12 @@ Templating .. sidebar:: Build environment - All *generic-doc* tasks are running in the :ref:`build environment `. + All *generic-doc* tasks are running in the :ref:`make install`. Templating is suitable for documentation which is created generic at the build -time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`build -environment ` (with searx modules installed). We use this e.g. to -build chapter: :ref:`engines generic`. Below the jinja directive from the +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with searx modules installed). We use this e.g. to build chapter: +:ref:`engines generic`. Below the jinja directive from the :origin:`docs/admin/engines.rst` is shown: .. literalinclude:: ../admin/engines.rst diff --git a/_sources/admin/buildhosts.rst.txt b/_sources/admin/buildhosts.rst.txt index 1f6eb472..e23327ba 100644 --- a/_sources/admin/buildhosts.rst.txt +++ b/_sources/admin/buildhosts.rst.txt @@ -49,9 +49,9 @@ Build docs - dvisvgm_ Most of the sphinx requirements are installed from :origin:`setup.py` and the -docs can be build from scratch with ``make docs``. For better math and image -processing additional packages are needed. The XeTeX_ needed not only for PDF -creation, its also needed for :ref:`math` when HTML output is build. +docs can be build from scratch with ``make docs.html``. For better math and +image processing additional packages are needed. The XeTeX_ needed not only for +PDF creation, its also needed for :ref:`math` when HTML output is build. To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered as images (``sphinx.ext.imgmath`` extension). @@ -64,7 +64,7 @@ to ``imgmath``: :start-after: # sphinx.ext.imgmath setup :end-before: # sphinx.ext.imgmath setup END -If your docs build (``make docs``) shows warnings like this:: +If your docs build (``make docs.html``) shows warnings like this:: WARNING: dot(1) not found, for better output quality install \ graphviz from https://www.graphviz.org diff --git a/_sources/admin/installation-docker.rst.txt b/_sources/admin/installation-docker.rst.txt index c5f54b49..13d21bc4 100644 --- a/_sources/admin/installation-docker.rst.txt +++ b/_sources/admin/installation-docker.rst.txt @@ -51,7 +51,7 @@ It's also possible to build searx from the embedded Dockerfile. git clone https://github.com/searx/searx.git cd searx - make docker + make docker.build Public instance diff --git a/_sources/blog/lxcdev-202006.rst.txt b/_sources/blog/lxcdev-202006.rst.txt index 2bea1980..b53501da 100644 --- a/_sources/blog/lxcdev-202006.rst.txt +++ b/_sources/blog/lxcdev-202006.rst.txt @@ -31,7 +31,7 @@ might fail in some aspects we should not overlook. The environment in which we run all our development processes matters! -The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but they do not have access to all prerequisites. For example, there may have dependencies on packages that are installed on the developer's desktop, but usually are not preinstalled on a server or client system. Another examples @@ -356,7 +356,7 @@ daily usage: .. code:: sh $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - make docs + make docs.html .. _blog-lxcdev-202006 abstract: @@ -407,7 +407,7 @@ To get remarks from the suite of the archlinux container we can use: ... [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ - [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ + [searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] ... diff --git a/_sources/dev/contribution_guide.rst.txt b/_sources/dev/contribution_guide.rst.txt index cffbbcae..ed1c223c 100644 --- a/_sources/dev/contribution_guide.rst.txt +++ b/_sources/dev/contribution_guide.rst.txt @@ -132,11 +132,11 @@ Here is an example which makes a complete rebuild: .. code:: sh - $ make docs-clean docs + $ make docs.clean docs.html ... The HTML pages are in dist/docs. -.. _make docs-live: +.. _make docs.live: live build ---------- @@ -144,19 +144,19 @@ live build .. _sphinx-autobuild: https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md -.. sidebar:: docs-clean +.. sidebar:: docs.clean It is recommended to assert a complete rebuild before deploying (use - ``docs-clean``). + ``docs.clean``). Live build is like WYSIWYG. If you want to edit the documentation, its -recommended to use. The Makefile target ``docs-live`` builds the docs, opens +recommended to use. The Makefile target ``docs.live`` builds the docs, opens URL in your favorite browser and rebuilds every time a reST file has been changed. .. code:: sh - $ make docs-live + $ make docs.live ... The HTML pages are in dist/docs. ... Serving on http://0.0.0.0:8000 @@ -169,7 +169,7 @@ argument. E.g to find and use a free port, use: .. code:: sh - $ SPHINXOPTS="--port 0" make docs-live + $ SPHINXOPTS="--port 0" make docs.live ... ... Serving on http://0.0.0.0:50593 ... @@ -180,21 +180,10 @@ argument. E.g to find and use a free port, use: deploy on github.io ------------------- -To deploy documentation at :docs:`github.io <.>` use Makefile target -:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub -folder ``gh-pages``, cleans it, copies the doc build into and runs all the -needed git add, commit and push: +To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make +docs.gh-pages`, which builds the documentation and runs all the needed git add, +commit and push: .. code:: sh - $ make docs-clean gh-pages - ... - SPHINX docs --> file://<...>/dist/docs - The HTML pages are in dist/docs. - ... - Cloning into 'gh-pages' ... - ... - cd gh-pages; git checkout gh-pages >/dev/null - Switched to a new branch 'gh-pages' - ... - doc available at --> https://searx.github.io/searx + $ make docs.clean docs.gh-pages diff --git a/_sources/dev/makefile.rst.txt b/_sources/dev/makefile.rst.txt index f9385592..870b5d49 100644 --- a/_sources/dev/makefile.rst.txt +++ b/_sources/dev/makefile.rst.txt @@ -1,33 +1,33 @@ .. _makefile: -================ -Makefile Targets -================ +======== +Makefile +======== .. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction .. sidebar:: build environment - Before looking deeper at the targets, first read about :ref:`make pyenv`. + Before looking deeper at the targets, first read about :ref:`make + install`. To install system requirements follow :ref:`buildhosts`. -With the aim to simplify development cycles, started with :pull:`1756` a -``Makefile`` based boilerplate was added. If you are not familiar with -Makefiles, we recommend to read gnu-make_ introduction. +All relevant build tasks are implemented in :origin:`manage.sh` and for CI or +IDE integration a small ``Makefile`` wrapper is available. If you are not +familiar with Makefiles, we recommend to read gnu-make_ introduction. The usage is simple, just type ``make {target-name}`` to *build* a target. Calling the ``help`` target gives a first overview (``make help``): .. program-output:: bash -c "cd ..; make --no-print-directory help" - .. contents:: Contents :depth: 2 :local: :backlinks: entry -.. _make pyenv: +.. _make install: Python environment ================== @@ -36,31 +36,42 @@ Python environment ``source ./local/py3/bin/activate`` -With Makefile we do no longer need to build up the virtualenv manually (as -described in the :ref:`devquickstart` guide). Jump into your git working tree -and release a ``make pyenv``: - -.. code:: sh +We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a ``make install`` to get a virtualenv with a +*developer install* of searx (:origin:`setup.py`). :: $ cd ~/searx-clone - $ make pyenv - PYENV usage: source ./local/py3/bin/activate + $ make install + PYENV [virtualenv] installing ./requirements*.txt into local/py3 ... + PYENV OK + PYENV [install] pip install -e 'searx[test]' + ... + Successfully installed argparse-1.4.0 searx + BUILDENV INFO:searx:load the default settings from ./searx/settings.yml + BUILDENV INFO:searx:Initialisation done + BUILDENV build utils/brand.env -With target ``pyenv`` a development environment (aka virtualenv) was build up in -``./local/py3/``. To make a *developer install* of searx (:origin:`setup.py`) -into this environment, use make target ``install``: - -.. code:: sh +If you release ``make install`` multiple times the installation will only +rebuild if the sha256 sum of the *requirement files* fails. With other words: +the check fails if you edit the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`). :: $ make install - PYENV usage: source ./local/py3/bin/activate - PYENV using virtualenv from ./local/py3 - PYENV install . - -You have never to think about intermediate targets like ``pyenv`` or -``install``, the ``Makefile`` chains them as requisites. Just run your main -target. + PYENV OK + PYENV [virtualenv] requirements.sha256 failed + [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt + [virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt + ... + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV OK + PYENV [install] pip install -e 'searx[test]' + ... + Successfully installed argparse-1.4.0 searx + BUILDENV INFO:searx:load the default settings from ./searx/settings.yml + BUILDENV INFO:searx:Initialisation done + BUILDENV build utils/brand.env .. sidebar:: drop environment @@ -68,10 +79,7 @@ target. ` first. If you think, something goes wrong with your ./local environment or you change -the :origin:`setup.py` file (or the requirements listed in -:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call -:ref:`make clean`. - +the :origin:`setup.py` file, you have to call :ref:`make clean`. .. _make run: @@ -81,77 +89,44 @@ the :origin:`setup.py` file (or the requirements listed in To get up a running a developer instance simply call ``make run``. This enables *debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` instance, disables *debug* option again and opens the URL in your favorite WEB -browser (:man:`xdg-open`): +browser (:man:`xdg-open`):: -.. code:: sh - - $ make run - PYENV usage: source ./local/py3/bin/activate - PYENV install . - ./local/py3/bin/python ./searx/webapp.py - ... - INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit) - ... + $ make run + PYENV OK + SEARX_DEBUG=1 ./manage.sh pyenv.cmd python ./searx/webapp.py + ... + INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit) .. _make clean: ``make clean`` ============== -Drop all intermediate files, all builds, but keep sources untouched. Includes -target ``pyclean`` which drops ./local environment. Before calling ``make -clean`` stop all processes using :ref:`make pyenv`. - -.. code:: sh +Drop all intermediate files, all builds, but keep sources untouched. Before +calling ``make clean`` stop all processes using :ref:`make install`. :: $ make clean - CLEAN pyclean - CLEAN clean + CLEAN pyenv + PYENV [virtualenv] drop ./local/py3 + CLEAN docs -- ./build/docs ./dist/docs + CLEAN locally installed npm dependencies + CLEAN test stuff + CLEAN common files .. _make docs: -``make docs docs-live docs-clean`` -================================== +``make docs docs.autobuild docs.clean`` +======================================= -We describe the usage of the ``doc*`` targets in the :ref:`How to contribute / +We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / Documentation ` section. If you want to edit the documentation -read our :ref:`make docs-live` section. If you are working in your own brand, +read our :ref:`make docs.live` section. If you are working in your own brand, adjust your :ref:`settings global`. -.. _make books: +.. _make docs.gh-pages: -``make books/{name}.html books/{name}.pdf`` -=========================================== - -.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html -.. _XeTeX: https://tug.org/xetex/ - -.. sidebar:: info - - To build PDF a XeTeX_ is needed, see :ref:`buildhosts`. - - -The ``books/{name}.*`` targets are building *books*. A *book* is a -sub-directory containing a ``conf.py`` file. One example is the user handbook -which can deployed separately (:origin:`docs/user/conf.py`). Such ``conf.py`` -do inherit from :origin:`docs/conf.py` and overwrite values to fit *book's* -needs. - -With the help of Intersphinx_ (:ref:`reST smart ref`) the links to searx’s -documentation outside of the book will be bound by the object inventory of -``DOCS_URL``. Take into account that URLs will be picked from the inventary at -documentation's build time. - -Use ``make docs-help`` to see which books available: - -.. program-output:: bash -c "cd ..; make --no-print-directory docs-help" - :ellipsis: 0,-6 - - -.. _make gh-pages: - -``make gh-pages`` -================= +``make docs.gh-pages`` +====================== To deploy on github.io first adjust your :ref:`settings global`. For any further read :ref:`deploy on github.io`. @@ -161,37 +136,66 @@ further read :ref:`deploy on github.io`. ``make test`` ============= -Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does -additional :ref:`pylint checks `. You can run tests selective, -e.g.: - -.. code:: sh +Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit`` +and ``test.robot``. You can run tests selective, e.g.:: $ make test.pep8 test.unit test.sh - . ./local/py3/bin/activate; ./manage.sh pep8_check - [!] Running pep8 check - . ./local/py3/bin/activate; ./manage.sh unit_tests - [!] Running unit tests + TEST test.pep8 OK + ... + TEST test.unit OK + ... + TEST test.sh OK -.. _make pylint: +.. _make test.sh: -``make pylint`` -=============== +``make test.sh`` +================ + +:ref:`sh lint` / if you have changed some bash scripting run this test before +commit. + +.. _make test.pylint: + +``make test.pylint`` +==================== .. _Pylint: https://www.pylint.org/ -Before commiting its recommend to do some (more) linting. Pylint_ is known as -one of the best source-code, bug and quality checker for the Python programming -language. Pylint_ is not yet a quality gate within our searx project (like -:ref:`test.pep8 ` it is), but Pylint_ can help to improve code -quality anyway. The pylint profile we use at searx project is found in -project's root folder :origin:`.pylintrc`. +Pylint_ is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile we use at searx project is +found in project's root folder :origin:`.pylintrc`. -Code quality is a ongoing process. Don't try to fix all messages from Pylint, -run Pylint and check if your changed lines are bringing up new messages. If so, -fix it. By this, code quality gets incremental better and if there comes the -day, the linting is balanced out, we might decide to add Pylint as a quality -gate. +.. _make search.checker: + +``search.checker.{engine name}`` +================================ + +To check all engines:: + + make search.checker + +To check a engine with whitespace in the name like *google news* replace space +by underline:: + + make search.checker.google_news + +To see HTTP requests and more use SEARX_DEBUG:: + + make SEARX_DEBUG=1 search.checker.google_news + +.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection + +To filter out HTTP redirects (3xx_):: + + make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]" + ... + Engine google news Checking + https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- ``make pybuild`` @@ -200,9 +204,7 @@ gate. .. _PyPi: https://pypi.org/ .. _twine: https://twine.readthedocs.io/en/latest/ -Build Python packages in ``./dist/py``. - -.. code:: sh +Build Python packages in ``./dist/py``:: $ make pybuild ... @@ -210,9 +212,11 @@ Build Python packages in ``./dist/py``. running sdist running egg_info ... - $ ls ./dist/py/ - searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz + running bdist_wheel -To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs -twine_ to be installed. Since you are not the owner of :pypi:`searx` you will -never need the latter. + $ ls ./dist + searx-0.18.0-py3-none-any.whl searx-0.18.0.tar.gz + +To upload packages to PyPi_, there is also a ``pypi.upload`` target (to test use +``pypi.upload.test``). Since you are not the owner of :pypi:`searx` you will +never need to upload. diff --git a/_sources/dev/reST.rst.txt b/_sources/dev/reST.rst.txt index 39cd9f67..1817504c 100644 --- a/_sources/dev/reST.rst.txt +++ b/_sources/dev/reST.rst.txt @@ -15,8 +15,8 @@ generated and deployed at :docs:`github.io <.>`. For build prerequisites read :ref:`docs build`. The source files of Searx's documentation are located at :origin:`docs`. Sphinx -assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs-live -` to build HTML while editing. +assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs.live +` to build HTML while editing. .. sidebar:: Further reading @@ -1276,13 +1276,12 @@ Templating .. sidebar:: Build environment - All *generic-doc* tasks are running in the :ref:`build environment `. + All *generic-doc* tasks are running in the :ref:`make install`. Templating is suitable for documentation which is created generic at the build -time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`build -environment ` (with searx modules installed). We use this e.g. to -build chapter: :ref:`engines generic`. Below the jinja directive from the +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with searx modules installed). We use this e.g. to build chapter: +:ref:`engines generic`. Below the jinja directive from the :origin:`docs/admin/engines.rst` is shown: .. literalinclude:: ../admin/engines.rst diff --git a/_sources/utils/lxc.sh.rst.txt b/_sources/utils/lxc.sh.rst.txt index 114d0911..56bac267 100644 --- a/_sources/utils/lxc.sh.rst.txt +++ b/_sources/utils/lxc.sh.rst.txt @@ -119,15 +119,15 @@ of coffee).:: To build (live) documentation inside a archlinux_ container:: - sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live + sudo -H ./utils/lxc.sh cmd searx-archlinux make docs.clean docs.live ... [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080 To get IP of the container and the port number *live docs* is listening:: - $ sudo ./utils/lxc.sh show suite | grep docs-live + $ sudo ./utils/lxc.sh show suite | grep docs.live ... - [searx-archlinux] INFO: (eth0) docs-live: http://n.n.n.12:8080/ + [searx-archlinux] INFO: (eth0) docs.live: http://n.n.n.12:8080/ .. _lxc.sh help: diff --git a/admin/buildhosts.html b/admin/buildhosts.html index 7ba8bae1..b2b8a4e4 100644 --- a/admin/buildhosts.html +++ b/admin/buildhosts.html @@ -124,9 +124,9 @@ on build hosts (see git://setup.py and the -docs can be build from scratch with make docs. For better math and image -processing additional packages are needed. The XeTeX needed not only for PDF -creation, its also needed for Math equations when HTML output is build.

+docs can be build from scratch with make docs.html. For better math and +image processing additional packages are needed. The XeTeX needed not only for +PDF creation, its also needed for Math equations when HTML output is build.

To be able to do Math support for HTML outputs in Sphinx without CDNs, the math are rendered as images (sphinx.ext.imgmath extension).

Here is the extract from the git://docs/conf.py file, setting math renderer @@ -136,7 +136,7 @@ to imgmathimgmath_font_size = 14 -

If your docs build (make docs) shows warnings like this:

+

If your docs build (make docs.html) shows warnings like this:

WARNING: dot(1) not found, for better output quality install \
          graphviz from https://www.graphviz.org
 ..
diff --git a/admin/installation-docker.html b/admin/installation-docker.html
index ea3d4e2f..c8004d23 100644
--- a/admin/installation-docker.html
+++ b/admin/installation-docker.html
@@ -106,7 +106,7 @@ Volume:
 

It’s also possible to build searx from the embedded Dockerfile.

git clone https://github.com/searx/searx.git
 cd searx
-make docker
+make docker.build
 
diff --git a/blog/lxcdev-202006.html b/blog/lxcdev-202006.html index 5753d527..b5e0015e 100644 --- a/blog/lxcdev-202006.html +++ b/blog/lxcdev-202006.html @@ -77,7 +77,7 @@ might fail in some aspects we should not overlook.

The environment in which we run all our development processes matters!

-

The Makefile Targets and the Python environment encapsulate a lot for us, but they +

The Makefile and the Python environment encapsulate a lot for us, but they do not have access to all prerequisites. For example, there may have dependencies on packages that are installed on the developer’s desktop, but usually are not preinstalled on a server or client system. Another examples @@ -301,7 +301,7 @@ daily usage:

./utils/searx.sh inspect service
-

Run Makefile Targets, e.g. to test inside the container:

+

Run Makefile, e.g. to test inside the container:

$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
   make test
 
@@ -313,7 +313,7 @@ daily usage:

To build the docs on a buildhost Buildhosts:

$ sudo -H ./utils/lxc.sh cmd searx-archlinux \
-  make docs
+  make docs.html
 
@@ -348,7 +348,7 @@ $ systemctl restart uwsgi@searx ... [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ -[searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ +[searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] ... diff --git a/dev/contribution_guide.html b/dev/contribution_guide.html index 03958fff..124aa41a 100644 --- a/dev/contribution_guide.html +++ b/dev/contribution_guide.html @@ -159,9 +159,9 @@ code.

The documentation is built using Sphinx. So in order to be able to generate the required files, you have to install it on your system. Much easier, use -our Makefile Targets.

+our Makefile.

Here is an example which makes a complete rebuild:

-
$ make docs-clean docs
+
$ make docs.clean docs.html
 ...
 The HTML pages are in dist/docs.
 
@@ -169,15 +169,15 @@ The HTML pages are in dist/docs.

live build

Live build is like WYSIWYG. If you want to edit the documentation, its -recommended to use. The Makefile target docs-live builds the docs, opens +recommended to use. The Makefile target docs.live builds the docs, opens URL in your favorite browser and rebuilds every time a reST file has been changed.

-
$ make docs-live
+
$ make docs.live
 ...
 The HTML pages are in dist/docs.
 ... Serving on http://0.0.0.0:8000
@@ -188,7 +188,7 @@ The HTML pages are in dist/docs.
 $(SPHINXOPTS) to pass arguments to the sphinx-autobuild command.  Except
 option --host (which is always set to 0.0.0.0) you can pass any
 argument.  E.g to find and use a free port, use:

-
$ SPHINXOPTS="--port 0" make docs-live
+
$ SPHINXOPTS="--port 0" make docs.live
 ...
 ... Serving on http://0.0.0.0:50593
 ...
@@ -197,21 +197,9 @@ argument.  E.g to find and use a free port, use:

deploy on github.io

-

To deploy documentation at github.io use Makefile target -make gh-pages, which will builds the documentation, clones searx into a sub -folder gh-pages, cleans it, copies the doc build into and runs all the -needed git add, commit and push:

-
$ make docs-clean gh-pages
-...
-SPHINX    docs --> file://<...>/dist/docs
-The HTML pages are in dist/docs.
-...
-Cloning into 'gh-pages' ...
-...
-cd gh-pages; git checkout gh-pages >/dev/null
-Switched to a new branch 'gh-pages'
-...
-doc available at --> https://searx.github.io/searx
+

To deploy documentation at github.io use Makefile target make docs.gh-pages, which builds the documentation and runs all the needed git add, +commit and push:

+
$ make docs.clean docs.gh-pages
 
diff --git a/dev/index.html b/dev/index.html index 723adbfc..bf9b83a2 100644 --- a/dev/index.html +++ b/dev/index.html @@ -82,15 +82,16 @@
  • Update translations
  • -
  • Makefile Targets
  • @@ -47,196 +47,248 @@
    -
    -

    Makefile Targets

    +
    +

    Makefile

    -

    With the aim to simplify development cycles, started with PR 1756 a -Makefile based boilerplate was added. If you are not familiar with -Makefiles, we recommend to read gnu-make introduction.

    +

    All relevant build tasks are implemented in git://manage.sh and for CI or +IDE integration a small Makefile wrapper is available. If you are not +familiar with Makefiles, we recommend to read gnu-make introduction.

    The usage is simple, just type make {target-name} to build a target. Calling the help target gives a first overview (make help):

    -
      test      - run developer tests
    -  docs      - build documentation
    -  docs-live - autobuild HTML documentation while editing
    -  run       - run developer instance
    -  install   - developer install (./local)
    -  uninstall - uninstall (./local)
    -  gh-pages  - build docs & deploy on gh-pages branch
    -  clean     - drop builds and environments
    -  project   - re-build generic files of the searx project
    -  buildenv  - re-build environment files (aka brand)
    -  themes    - re-build build the source of the themes
    -  docker    - build Docker image
    -  node.env  - download & install npm dependencies locally
    -
    -  make V=0|1 [targets] 0 => quiet build (default), 1 => verbose build
    -  make V=2   [targets] 2 => give reason for rebuild of target
    -
    -to get more help:  make help-all
    +
    buildenv
    +  rebuild ./utils/brand.env
    +babel.compile
    +  pybabel compile ./searx/translations
    +data.*
    +  all       : update searx/languages.py and ./data/*
    +  languages : update searx/data/engines_languages.json & searx/languages.py
    +  useragents: update searx/data/useragents.json with the most recent versions of Firefox.
    +docs.*
    +  html      : build HTML documentation
    +  live      : autobuild HTML documentation while editing
    +  gh-pages  : deploy on gh-pages branch
    +  prebuild  : build reST include files (./build/docs/includes)
    +  clean     : clean documentation build
    +docker
    +  build     : build docker image
    +  push      : build and push docker image
    +gecko.driver
    +  download & install geckodriver if not already installed (required for
    +  robot_tests)
    +node.*
    +  env       : download & install npm dependencies locally
    +  clean     : drop npm installations
    +py.*
    +  build     : Build python packages at ./dist
    +  clean     : delete virtualenv and intermediate py files
    +pyenv.* :
    +  install   : developer install of searx into virtualenv
    +  uninstall : uninstall developer installation
    +  cmd ...   : run command ... in virtualenv
    +  OK        : test if virtualenv is OK
    +pypi.upload:
    +  Upload python packages to PyPi (to test use pypi.upload.test)
    +test.* :
    +  pylint    : lint PYLINT_FILES, searx/engines, searx & tests
    +  pep8      : pycodestyle (pep8) for all files except PYLINT_FILES
    +  unit      : run unit tests
    +  coverage  : run unit tests with coverage
    +  robot     : run robot test
    +  clean     : clean intermediate test stuff
    +themes.* :
    +  all       : build all themes
    +  oscar     : build oscar theme
    +  simple    : build simple theme
    +----
    +run            - run developer instance
    +install        - developer install of searx into virtualenv
    +uninstall      - uninstall developer installation
    +clean          - clean up working tree
    +search.checker - check search engines
    +test           - run shell & CI tests
    +test.sh        - test shell scripts
    +ci.test        - run CI tests
     
    -

    Python environment

    +

    Python environment

    -

    With Makefile we do no longer need to build up the virtualenv manually (as -described in the Development Quickstart guide). Jump into your git working tree -and release a make pyenv:

    -
    $ cd ~/searx-clone
    -$ make pyenv
    -PYENV     usage: source ./local/py3/bin/activate
    +

    We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a make install to get a virtualenv with a +developer install of searx (git://setup.py).

    +
    $ cd ~/searx-clone
    +$ make install
    +PYENV     [virtualenv] installing ./requirements*.txt into local/py3
     ...
    +PYENV     OK
    +PYENV     [install] pip install -e 'searx[test]'
    +...
    +Successfully installed argparse-1.4.0 searx
    +BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
    +BUILDENV  INFO:searx:Initialisation done
    +BUILDENV  build utils/brand.env
     
    -

    With target pyenv a development environment (aka virtualenv) was build up in -./local/py3/. To make a developer install of searx (git://setup.py) -into this environment, use make target install:

    -
    $ make install
    -PYENV     usage: source ./local/py3/bin/activate
    -PYENV     using virtualenv from ./local/py3
    -PYENV     install .
    +

    If you release make install multiple times the installation will only +rebuild if the sha256 sum of the requirement files fails. With other words: +the check fails if you edit the requirements listed in +git://requirements-dev.txt and git://requirements.txt).

    +
    $ make install
    +PYENV     OK
    +PYENV     [virtualenv] requirements.sha256 failed
    +          [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e...  ./requirements-dev.txt
    +          [virtualenv] - 471efef6c73558e391c3adb35f4...  ./requirements.txt
    +...
    +PYENV     [virtualenv] installing ./requirements*.txt into local/py3
    +...
    +PYENV     OK
    +PYENV     [install] pip install -e 'searx[test]'
    +...
    +Successfully installed argparse-1.4.0 searx
    +BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
    +BUILDENV  INFO:searx:Initialisation done
    +BUILDENV  build utils/brand.env
     
    -

    You have never to think about intermediate targets like pyenv or -install, the Makefile chains them as requisites. Just run your main -target.

    If you think, something goes wrong with your ./local environment or you change -the git://setup.py file (or the requirements listed in -git://requirements-dev.txt and git://requirements.txt), you have to call -make clean.

    +the git://setup.py file, you have to call make clean.

    -

    make run

    +

    make run

    To get up a running a developer instance simply call make run. This enables debug option in git://searx/settings.yml, starts a ./searx/webapp.py instance, disables debug option again and opens the URL in your favorite WEB browser (xdg-open):

    -
    $ make run
    -PYENV     usage: source ./local/py3/bin/activate
    -PYENV     install .
    -./local/py3/bin/python ./searx/webapp.py
    -...
    -INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
    +
    $ make run
    +PYENV     OK
    +SEARX_DEBUG=1 ./manage.sh pyenv.cmd python ./searx/webapp.py
     ...
    +INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
     
    -

    make clean

    -

    Drop all intermediate files, all builds, but keep sources untouched. Includes -target pyclean which drops ./local environment. Before calling make -clean stop all processes using Python environment.

    -
    $ make clean
    -CLEAN     pyclean
    -CLEAN     clean
    +

    make clean

    +

    Drop all intermediate files, all builds, but keep sources untouched. Before +calling make clean stop all processes using Python environment.

    +
    $ make clean
    +CLEAN     pyenv
    +PYENV     [virtualenv] drop ./local/py3
    +CLEAN     docs -- ./build/docs ./dist/docs
    +CLEAN     locally installed npm dependencies
    +CLEAN     test stuff
    +CLEAN     common files
     
    -
    -

    make docs docs-live docs-clean

    -

    We describe the usage of the doc* targets in the How to contribute / +

    +

    make docs docs.autobuild docs.clean

    +

    We describe the usage of the doc.* targets in the How to contribute / Documentation section. If you want to edit the documentation read our live build section. If you are working in your own brand, adjust your Global Settings.

    -
    -

    make books/{name}.html books/{name}.pdf

    - -

    The books/{name}.* targets are building books. A book is a -sub-directory containing a conf.py file. One example is the user handbook -which can deployed separately (git://docs/user/conf.py). Such conf.py -do inherit from git://docs/conf.py and overwrite values to fit book’s -needs.

    -

    With the help of Intersphinx (Smart refs) the links to searx’s -documentation outside of the book will be bound by the object inventory of -DOCS_URL. Take into account that URLs will be picked from the inventary at -documentation’s build time.

    -

    Use make docs-help to see which books available:

    -
    ...
    -  books/{name}.html : build only the HTML of document {name}
    -  valid values for books/{name}.html are:
    -    books/user.html
    -  books/{name}.pdf : build only the PDF of document {name}
    -  valid values for books/{name}.pdf are:
    -    books/user.pdf
    -
    -
    -
    -
    -

    make gh-pages

    +
    +

    make docs.gh-pages

    To deploy on github.io first adjust your Global Settings. For any further read deploy on github.io.

    -

    make test

    -

    Runs a series of tests: test.pep8, test.unit, test.robot and does -additional pylint checks. You can run tests selective, -e.g.:

    -
    $ make test.pep8 test.unit test.sh
    -. ./local/py3/bin/activate; ./manage.sh pep8_check
    -[!] Running pep8 check
    -. ./local/py3/bin/activate; ./manage.sh unit_tests
    -[!] Running unit tests
    +

    make test

    +

    Runs a series of tests: make test.pylint, test.pep8, test.unit +and test.robot. You can run tests selective, e.g.:

    +
    $ make test.pep8 test.unit test.sh
    +TEST      test.pep8 OK
    +...
    +TEST      test.unit OK
    +...
    +TEST      test.sh OK
     
    -
    -

    make pylint

    -

    Before commiting its recommend to do some (more) linting. Pylint is known as -one of the best source-code, bug and quality checker for the Python programming -language. Pylint is not yet a quality gate within our searx project (like -test.pep8 it is), but Pylint can help to improve code -quality anyway. The pylint profile we use at searx project is found in -project’s root folder git://.pylintrc.

    -

    Code quality is a ongoing process. Don’t try to fix all messages from Pylint, -run Pylint and check if your changed lines are bringing up new messages. If so, -fix it. By this, code quality gets incremental better and if there comes the -day, the linting is balanced out, we might decide to add Pylint as a quality -gate.

    +
    +

    make test.sh

    +

    Lint shell scripts / if you have changed some bash scripting run this test before +commit.

    +
    +
    +

    make test.pylint

    +

    Pylint is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile we use at searx project is +found in project’s root folder git://.pylintrc.

    +
    +
    +

    search.checker.{engine name}

    +

    To check all engines:

    +
    make search.checker
    +
    +
    +

    To check a engine with whitespace in the name like google news replace space +by underline:

    +
    make search.checker.google_news
    +
    +
    +

    To see HTTP requests and more use SEARX_DEBUG:

    +
    make SEARX_DEBUG=1 search.checker.google_news
    +
    +
    +

    To filter out HTTP redirects (3xx):

    +
    make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
    +...
    +Engine google news                   Checking
    +https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
    +https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
    +--
    +https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
    +https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
    +--
    +
    +
    -

    make pybuild

    -

    Build Python packages in ./dist/py.

    -
    $ make pybuild
    +

    make pybuild

    +

    Build Python packages in ./dist/py:

    +
    $ make pybuild
     ...
     BUILD     pybuild
     running sdist
     running egg_info
     ...
    -$ ls  ./dist/py/
    -searx-0.15.0-py3-none-any.whl  searx-0.15.0.tar.gz
    +running bdist_wheel
    +
    +$ ls  ./dist
    +searx-0.18.0-py3-none-any.whl  searx-0.18.0.tar.gz
     
    -

    To upload packages to PyPi, there is also a upload-pypi target. It needs -twine to be installed. Since you are not the owner of PyPi: searx you will -never need the latter.

    +

    To upload packages to PyPi, there is also a pypi.upload target (to test use +pypi.upload.test). Since you are not the owner of PyPi: searx you will +never need to upload.

    diff --git a/dev/reST.html b/dev/reST.html index 4c311d9a..592364a8 100644 --- a/dev/reST.html +++ b/dev/reST.html @@ -17,7 +17,7 @@ - + @@ -34,7 +34,7 @@ next |
  • - previous |
  • @@ -59,7 +59,7 @@ documentation, with the builders from the github.io. For build prerequisites read Build docs.

    The source files of Searx’s documentation are located at git://docs. Sphinx -assumes source files to be encoded in UTF-8 by defaul. Run make docs-live to build HTML while editing.

    +assumes source files to be encoded in UTF-8 by defaul. Run make docs.live to build HTML while editing.

    To build (live) documentation inside a archlinux container:

    -
    sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live
    +
    sudo -H ./utils/lxc.sh cmd searx-archlinux make docs.clean docs.live
     ...
     [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
     

    To get IP of the container and the port number live docs is listening:

    -
    $ sudo ./utils/lxc.sh show suite | grep docs-live
    +
    $ sudo ./utils/lxc.sh show suite | grep docs.live
     ...
    -[searx-archlinux]  INFO:  (eth0) docs-live:  http://n.n.n.12:8080/
    +[searx-archlinux]  INFO:  (eth0) docs.live:  http://n.n.n.12:8080/
     
    @@ -194,7 +194,7 @@ install :base: prepare LXC; install basic packages :suite: install LXC searx suite into all (or <name>) containers -LXC suite: searx --> http://fv-az121-840/searx +LXC suite: searx --> http://fv-az180-407/searx suite includes searx, morty & filtron suite images: ubu1804 ubu2004 ubu2010 fedora33 archlinux centos7 diff --git a/utils/morty.sh.html b/utils/morty.sh.html index 82b2dca5..0d56b531 100644 --- a/utils/morty.sh.html +++ b/utils/morty.sh.html @@ -186,22 +186,22 @@ inspect service option set one of the available options :new-key: set new morty key -apache : http://fv-az121-840/morty/ +apache : http://fv-az180-407/morty/ :install: apache site with a reverse proxy (ProxyPass) :remove: apache site morty.conf -nginx (http://fv-az121-840/morty/) +nginx (http://fv-az180-407/morty/) :install: nginx site with a reverse proxy (ProxyPass) :remove: nginx site morty.conf If needed, set the environment variables in the '.config.sh' file:: - PUBLIC_URL_MORTY: http://fv-az121-840/morty/ + PUBLIC_URL_MORTY: http://fv-az180-407/morty/ MORTY_LISTEN: 127.0.0.1:3000 SERVICE_USER: morty To activate result and image proxy in searx, edit settings.yml (read: https://searx.github.io/searx/admin/morty.html):: result_proxy: - url : http://fv-az121-840/morty/ + url : http://fv-az180-407/morty/ server: image_proxy : True
    diff --git a/utils/searx.sh.html b/utils/searx.sh.html index 05611205..d5c82d0b 100644 --- a/utils/searx.sh.html +++ b/utils/searx.sh.html @@ -113,8 +113,8 @@ apache searx settings: /etc/searx/settings.yml If needed, set PUBLIC_URL of your WEB service in the '.config.sh' file:: - PUBLIC_URL : http://fv-az121-840/searx - SEARX_INSTANCE_NAME : searx@fv-az121-840 + PUBLIC_URL : http://fv-az180-407/searx + SEARX_INSTANCE_NAME : searx@fv-az180-407 SERVICE_USER : searx SEARX_INTERNAL_HTTP : http://127.0.0.1:8888