devol
/
searx
mirror of https://github.com/searx/searx
1
0
Fork 0
Code Issues Releases Wiki Activity

[doc] modify docs to fit to the new build boilerplate 

Signed-off-by: Markus Heiser <markus@darmarit.de>
This commit is contained in:
Markus Heiser 2021-02-27 18:31:27 +01:00
parent f55babc23c
commit 34abad95df
7 changed files with 149 additions and 157 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/admin/buildhosts.rst
View File

@ -49,9 +49,9 @@ Build docs
- dvisvgm_ - dvisvgm_
Most of the sphinx requirements are installed from :origin:`setup.py` and the Most of the sphinx requirements are installed from :origin:`setup.py` and the
docs can be build from scratch with ``make docs``. For better math and image docs can be build from scratch with ``make docs.html``. For better math and
processing additional packages are needed. The XeTeX_ needed not only for PDF image processing additional packages are needed. The XeTeX_ needed not only for
creation, its also needed for :ref:`math` when HTML output is build. PDF creation, its also needed for :ref:`math` when HTML output is build.
To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered
as images (``sphinx.ext.imgmath`` extension). as images (``sphinx.ext.imgmath`` extension).
@ -64,7 +64,7 @@ to ``imgmath``:
:start-after: # sphinx.ext.imgmath setup :start-after: # sphinx.ext.imgmath setup
:end-before: # sphinx.ext.imgmath setup END :end-before: # sphinx.ext.imgmath setup END
If your docs build (``make docs``) shows warnings like this:: If your docs build (``make docs.html``) shows warnings like this::
WARNING: dot(1) not found, for better output quality install \ WARNING: dot(1) not found, for better output quality install \
graphviz from https://www.graphviz.org graphviz from https://www.graphviz.org

2
docs/admin/installation-docker.rst
View File

@ -51,7 +51,7 @@ It's also possible to build searx from the embedded Dockerfile.
git clone https://github.com/searx/searx.git git clone https://github.com/searx/searx.git
cd searx cd searx
make docker make docker.build
Public instance Public instance

6
docs/blog/lxcdev-202006.rst
View File

@ -31,7 +31,7 @@ might fail in some aspects we should not overlook.
The environment in which we run all our development processes matters! The environment in which we run all our development processes matters!
The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but they
do not have access to all prerequisites. For example, there may have do not have access to all prerequisites. For example, there may have
dependencies on packages that are installed on the developer's desktop, but dependencies on packages that are installed on the developer's desktop, but
usually are not preinstalled on a server or client system. Another examples usually are not preinstalled on a server or client system. Another examples
@ -356,7 +356,7 @@ daily usage:
.. code:: sh .. code:: sh
$ sudo -H ./utils/lxc.sh cmd searx-archlinux \ $ sudo -H ./utils/lxc.sh cmd searx-archlinux \
make docs make docs.html
.. _blog-lxcdev-202006 abstract: .. _blog-lxcdev-202006 abstract:
@ -407,7 +407,7 @@ To get remarks from the suite of the archlinux container we can use:
... ...
[searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx
[searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/
[searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ [searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/
[searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65]
... ...

33
docs/dev/contribution_guide.rst
View File

@ -132,11 +132,11 @@ Here is an example which makes a complete rebuild:
.. code:: sh .. code:: sh
$ make docs-clean docs $ make docs.clean docs.html
... ...
The HTML pages are in dist/docs. The HTML pages are in dist/docs.
.. _make docs-live: .. _make docs.live:
live build live build
---------- ----------
@ -144,19 +144,19 @@ live build
.. _sphinx-autobuild: .. _sphinx-autobuild:
https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md
.. sidebar:: docs-clean .. sidebar:: docs.clean
It is recommended to assert a complete rebuild before deploying (use It is recommended to assert a complete rebuild before deploying (use
``docs-clean``). ``docs.clean``).
Live build is like WYSIWYG. If you want to edit the documentation, its Live build is like WYSIWYG. If you want to edit the documentation, its
recommended to use. The Makefile target ``docs-live`` builds the docs, opens recommended to use. The Makefile target ``docs.live`` builds the docs, opens
URL in your favorite browser and rebuilds every time a reST file has been URL in your favorite browser and rebuilds every time a reST file has been
changed. changed.
.. code:: sh .. code:: sh
$ make docs-live $ make docs.live
... ...
The HTML pages are in dist/docs. The HTML pages are in dist/docs.
... Serving on http://0.0.0.0:8000 ... Serving on http://0.0.0.0:8000
@ -169,7 +169,7 @@ argument. E.g to find and use a free port, use:
.. code:: sh .. code:: sh
$ SPHINXOPTS="--port 0" make docs-live $ SPHINXOPTS="--port 0" make docs.live
... ...
... Serving on http://0.0.0.0:50593 ... Serving on http://0.0.0.0:50593
... ...
@ -180,21 +180,10 @@ argument. E.g to find and use a free port, use:
deploy on github.io deploy on github.io
------------------- -------------------
To deploy documentation at :docs:`github.io <.>` use Makefile target To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make
:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub docs.gh-pages`, which builds the documentation and runs all the needed git add,
folder ``gh-pages``, cleans it, copies the doc build into and runs all the commit and push:
needed git add, commit and push:
.. code:: sh .. code:: sh
$ make docs-clean gh-pages $ make docs.clean docs.gh-pages
...
SPHINX docs --> file://<...>/dist/docs
The HTML pages are in dist/docs.
...
Cloning into 'gh-pages' ...
...
cd gh-pages; git checkout gh-pages >/dev/null
Switched to a new branch 'gh-pages'
...
doc available at --> https://searx.github.io/searx

238
docs/dev/makefile.rst
View File

@ -1,33 +1,33 @@
.. _makefile: .. _makefile:
================ ========
Makefile Targets Makefile
================ ========
.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction .. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction
.. sidebar:: build environment .. sidebar:: build environment
Before looking deeper at the targets, first read about :ref:`make pyenv`. Before looking deeper at the targets, first read about :ref:`make
install`.
To install system requirements follow :ref:`buildhosts`. To install system requirements follow :ref:`buildhosts`.
With the aim to simplify development cycles, started with :pull:`1756` a All relevant build tasks are implemented in :origin:`manage.sh` and for CI or
``Makefile`` based boilerplate was added. If you are not familiar with IDE integration a small ``Makefile`` wrapper is available. If you are not
Makefiles, we recommend to read gnu-make_ introduction. familiar with Makefiles, we recommend to read gnu-make_ introduction.
The usage is simple, just type ``make {target-name}`` to *build* a target. The usage is simple, just type ``make {target-name}`` to *build* a target.
Calling the ``help`` target gives a first overview (``make help``): Calling the ``help`` target gives a first overview (``make help``):
.. program-output:: bash -c "cd ..; make --no-print-directory help" .. program-output:: bash -c "cd ..; make --no-print-directory help"
.. contents:: Contents .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
.. _make pyenv: .. _make install:
Python environment Python environment
================== ==================
@ -36,31 +36,42 @@ Python environment
``source ./local/py3/bin/activate`` ``source ./local/py3/bin/activate``
With Makefile we do no longer need to build up the virtualenv manually (as We do no longer need to build up the virtualenv manually. Jump into your git
described in the :ref:`devquickstart` guide). Jump into your git working tree working tree and release a ``make install`` to get a virtualenv with a
and release a ``make pyenv``: *developer install* of searx (:origin:`setup.py`). ::
.. code:: sh
$ cd ~/searx-clone $ cd ~/searx-clone
$ make pyenv $ make install
PYENV usage: source ./local/py3/bin/activate PYENV [virtualenv] installing ./requirements*.txt into local/py3
... ...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
With target ``pyenv`` a development environment (aka virtualenv) was build up in If you release ``make install`` multiple times the installation will only
``./local/py3/``. To make a *developer install* of searx (:origin:`setup.py`) rebuild if the sha256 sum of the *requirement files* fails. With other words:
into this environment, use make target ``install``: the check fails if you edit the requirements listed in
:origin:`requirements-dev.txt` and :origin:`requirements.txt`). ::
.. code:: sh
$ make install $ make install
PYENV usage: source ./local/py3/bin/activate PYENV OK
PYENV using virtualenv from ./local/py3 PYENV [virtualenv] requirements.sha256 failed
PYENV install . [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt
[virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt
You have never to think about intermediate targets like ``pyenv`` or ...
``install``, the ``Makefile`` chains them as requisites. Just run your main PYENV [virtualenv] installing ./requirements*.txt into local/py3
target. ...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
.. sidebar:: drop environment .. sidebar:: drop environment
@ -68,10 +79,7 @@ target.
<make clean>` first. <make clean>` first.
If you think, something goes wrong with your ./local environment or you change If you think, something goes wrong with your ./local environment or you change
the :origin:`setup.py` file (or the requirements listed in the :origin:`setup.py` file, you have to call :ref:`make clean`.
:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call
:ref:`make clean`.
.. _make run: .. _make run:
@ -81,77 +89,44 @@ the :origin:`setup.py` file (or the requirements listed in
To get up a running a developer instance simply call ``make run``. This enables To get up a running a developer instance simply call ``make run``. This enables
*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` *debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py``
instance, disables *debug* option again and opens the URL in your favorite WEB instance, disables *debug* option again and opens the URL in your favorite WEB
browser (:man:`xdg-open`): browser (:man:`xdg-open`)::
.. code:: sh $ make run
PYENV OK
$ make run SEARX_DEBUG=1 ./manage.sh pyenv.cmd python ./searx/webapp.py
PYENV usage: source ./local/py3/bin/activate ...
PYENV install . INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
./local/py3/bin/python ./searx/webapp.py
...
INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
...
.. _make clean: .. _make clean:
``make clean`` ``make clean``
============== ==============
Drop all intermediate files, all builds, but keep sources untouched. Includes Drop all intermediate files, all builds, but keep sources untouched. Before
target ``pyclean`` which drops ./local environment. Before calling ``make calling ``make clean`` stop all processes using :ref:`make install`. ::
clean`` stop all processes using :ref:`make pyenv`.
.. code:: sh
$ make clean $ make clean
CLEAN pyclean CLEAN pyenv
CLEAN clean PYENV [virtualenv] drop ./local/py3
CLEAN docs -- ./build/docs ./dist/docs
CLEAN locally installed npm dependencies
CLEAN test stuff
CLEAN common files
.. _make docs: .. _make docs:
``make docs docs-live docs-clean`` ``make docs docs.autobuild docs.clean``
================================== =======================================
We describe the usage of the ``doc*`` targets in the :ref:`How to contribute / We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
Documentation <contrib docs>` section. If you want to edit the documentation Documentation <contrib docs>` section. If you want to edit the documentation
read our :ref:`make docs-live` section. If you are working in your own brand, read our :ref:`make docs.live` section. If you are working in your own brand,
adjust your :ref:`settings global`. adjust your :ref:`settings global`.
.. _make books: .. _make docs.gh-pages:
``make books/{name}.html books/{name}.pdf`` ``make docs.gh-pages``
=========================================== ======================
.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html
.. _XeTeX: https://tug.org/xetex/
.. sidebar:: info
To build PDF a XeTeX_ is needed, see :ref:`buildhosts`.
The ``books/{name}.*`` targets are building *books*. A *book* is a
sub-directory containing a ``conf.py`` file. One example is the user handbook
which can deployed separately (:origin:`docs/user/conf.py`). Such ``conf.py``
do inherit from :origin:`docs/conf.py` and overwrite values to fit *book's*
needs.
With the help of Intersphinx_ (:ref:`reST smart ref`) the links to searx’s
documentation outside of the book will be bound by the object inventory of
``DOCS_URL``. Take into account that URLs will be picked from the inventary at
documentation's build time.
Use ``make docs-help`` to see which books available:
.. program-output:: bash -c "cd ..; make --no-print-directory docs-help"
:ellipsis: 0,-6
.. _make gh-pages:
``make gh-pages``
=================
To deploy on github.io first adjust your :ref:`settings global`. For any To deploy on github.io first adjust your :ref:`settings global`. For any
further read :ref:`deploy on github.io`. further read :ref:`deploy on github.io`.
@ -161,37 +136,66 @@ further read :ref:`deploy on github.io`.
``make test`` ``make test``
============= =============
Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit``
additional :ref:`pylint checks <make pylint>`. You can run tests selective, and ``test.robot``. You can run tests selective, e.g.::
e.g.:
.. code:: sh
$ make test.pep8 test.unit test.sh $ make test.pep8 test.unit test.sh
. ./local/py3/bin/activate; ./manage.sh pep8_check TEST test.pep8 OK
[!] Running pep8 check ...
. ./local/py3/bin/activate; ./manage.sh unit_tests TEST test.unit OK
[!] Running unit tests ...
TEST test.sh OK
.. _make pylint: .. _make test.sh:
``make pylint`` ``make test.sh``
=============== ================
:ref:`sh lint` / if you have changed some bash scripting run this test before
commit.
.. _make test.pylint:
``make test.pylint``
====================
.. _Pylint: https://www.pylint.org/ .. _Pylint: https://www.pylint.org/
Before commiting its recommend to do some (more) linting. Pylint_ is known as Pylint_ is known as one of the best source-code, bug and quality checker for the
one of the best source-code, bug and quality checker for the Python programming Python programming language. The pylint profile we use at searx project is
language. Pylint_ is not yet a quality gate within our searx project (like found in project's root folder :origin:`.pylintrc`.
:ref:`test.pep8 <make test>` it is), but Pylint_ can help to improve code
quality anyway. The pylint profile we use at searx project is found in
project's root folder :origin:`.pylintrc`.
Code quality is a ongoing process. Don't try to fix all messages from Pylint, .. _make search.checker:
run Pylint and check if your changed lines are bringing up new messages. If so,
fix it. By this, code quality gets incremental better and if there comes the ``search.checker.{engine name}``
day, the linting is balanced out, we might decide to add Pylint as a quality ================================
gate.
To check all engines::
make search.checker
To check a engine with whitespace in the name like *google news* replace space
by underline::
make search.checker.google_news
To see HTTP requests and more use SEARX_DEBUG::
make SEARX_DEBUG=1 search.checker.google_news
.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection
To filter out HTTP redirects (3xx_)::
make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
``make pybuild`` ``make pybuild``
@ -200,9 +204,7 @@ gate.
.. _PyPi: https://pypi.org/ .. _PyPi: https://pypi.org/
.. _twine: https://twine.readthedocs.io/en/latest/ .. _twine: https://twine.readthedocs.io/en/latest/
Build Python packages in ``./dist/py``. Build Python packages in ``./dist/py``::
.. code:: sh
$ make pybuild $ make pybuild
... ...
@ -210,9 +212,11 @@ Build Python packages in ``./dist/py``.
running sdist running sdist
running egg_info running egg_info
... ...
$ ls ./dist/py/ running bdist_wheel
searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz
To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs $ ls ./dist
twine_ to be installed. Since you are not the owner of :pypi:`searx` you will searx-0.18.0-py3-none-any.whl searx-0.18.0.tar.gz
never need the latter.
To upload packages to PyPi_, there is also a ``pypi.upload`` target (to test use
``pypi.upload.test``). Since you are not the owner of :pypi:`searx` you will
never need to upload.

13
docs/dev/reST.rst
View File

@ -15,8 +15,8 @@ generated and deployed at :docs:`github.io <.>`. For build prerequisites read
:ref:`docs build`. :ref:`docs build`.
The source files of Searx's documentation are located at :origin:`docs`. Sphinx The source files of Searx's documentation are located at :origin:`docs`. Sphinx
assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs-live assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs.live
<make docs-live>` to build HTML while editing. <make docs.live>` to build HTML while editing.
.. sidebar:: Further reading .. sidebar:: Further reading
@ -1276,13 +1276,12 @@ Templating
.. sidebar:: Build environment .. sidebar:: Build environment
All *generic-doc* tasks are running in the :ref:`build environment <make All *generic-doc* tasks are running in the :ref:`make install`.
pyenv>`.
Templating is suitable for documentation which is created generic at the build Templating is suitable for documentation which is created generic at the build
time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`build time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make
environment <make pyenv>` (with searx modules installed). We use this e.g. to install` (with searx modules installed). We use this e.g. to build chapter:
build chapter: :ref:`engines generic`. Below the jinja directive from the :ref:`engines generic`. Below the jinja directive from the
:origin:`docs/admin/engines.rst` is shown: :origin:`docs/admin/engines.rst` is shown:
.. literalinclude:: ../admin/engines.rst .. literalinclude:: ../admin/engines.rst

6
docs/utils/lxc.sh.rst
View File

@ -119,15 +119,15 @@ of coffee).::
To build (live) documentation inside a archlinux_ container:: To build (live) documentation inside a archlinux_ container::
sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live sudo -H ./utils/lxc.sh cmd searx-archlinux make docs.clean docs.live
... ...
[I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080 [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
To get IP of the container and the port number *live docs* is listening:: To get IP of the container and the port number *live docs* is listening::
$ sudo ./utils/lxc.sh show suite | grep docs-live $ sudo ./utils/lxc.sh show suite | grep docs.live
... ...
[searx-archlinux] INFO: (eth0) docs-live: http://n.n.n.12:8080/ [searx-archlinux] INFO: (eth0) docs.live: http://n.n.n.12:8080/
.. _lxc.sh help: .. _lxc.sh help: