searx/dev/makefile.html

355 lines
22 KiB
HTML
Raw Normal View History

2019-12-30 18:31:17 +01:00
<!DOCTYPE html>
<html lang="en">
2019-12-30 18:31:17 +01:00
<head>
<meta charset="utf-8" />
2020-09-30 10:35:05 +02:00
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
2019-12-30 18:31:17 +01:00
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Makefile &#8212; Searx Documentation (Searx-1.1.0.tex)</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/searx.css" />
<link rel="stylesheet" type="text/css" href="../_static/tabs.css" />
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
2020-02-15 10:08:58 +01:00
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
2020-02-15 10:08:58 +01:00
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
2019-12-30 18:31:17 +01:00
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="reST primer" href="reST.html" />
<link rel="prev" title="Translation" href="translation.html" />
2019-12-30 18:31:17 +01:00
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
2019-12-30 18:31:17 +01:00
<li class="right" >
<a href="reST.html" title="reST primer"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="translation.html" title="Translation"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">Searx Documentation (Searx-1.1.0.tex)</a> &#187;</li>
2020-09-30 10:35:05 +02:00
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Developer documentation</a> &#187;</li>
<li class="nav-item nav-item-this"><a href="">Makefile</a></li>
2019-12-30 18:31:17 +01:00
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="makefile">
<span id="id1"></span><h1>Makefile<a class="headerlink" href="#makefile" title="Permalink to this heading"></a></h1>
<aside class="sidebar">
2019-12-30 18:31:17 +01:00
<p class="sidebar-title">build environment</p>
<p>Before looking deeper at the targets, first read about <a class="reference internal" href="#make-install"><span class="std std-ref">Python environment</span></a>.</p>
2020-06-19 11:15:15 +02:00
<p>To install system requirements follow <a class="reference internal" href="../admin/buildhosts.html#buildhosts"><span class="std std-ref">Buildhosts</span></a>.</p>
</aside>
<p>All relevant build tasks are implemented in <a class="reference external" href="https://github.com/searx/searx/blob/master/manage.sh">Origin: manage.sh</a> and for CI or
IDE integration a small <code class="docutils literal notranslate"><span class="pre">Makefile</span></code> wrapper is available. If you are not
familiar with Makefiles, we recommend to read <a class="reference external" href="https://www.gnu.org/software/make/manual/make.html#Introduction">gnu-make</a> introduction.</p>
2019-12-30 18:31:17 +01:00
<p>The usage is simple, just type <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">{target-name}</span></code> to <em>build</em> a target.
2020-06-19 11:15:15 +02:00
Calling the <code class="docutils literal notranslate"><span class="pre">help</span></code> target gives a first overview (<code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">help</span></code>):</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>buildenv
rebuild ./utils/brand.env
babel.compile
pybabel compile ./searx/translations
data.*
all : update searx/languages.py and ./data/*
languages : update searx/data/engines_languages.json &amp; searx/languages.py
useragents: update searx/data/useragents.json with the most recent versions of Firefox.
docs.*
html : build HTML documentation
live : autobuild HTML documentation while editing
gh-pages : deploy on gh-pages branch
prebuild : build reST include files (./build/docs/includes)
clean : clean documentation build
docker
build : build docker image
push : build and push docker image
gecko.driver
download &amp; install geckodriver if not already installed (required for
robot_tests)
node.*
env : download &amp; install npm dependencies locally
clean : drop npm installations
py.*
build : Build python packages at ./dist
clean : delete virtualenv and intermediate py files
pyenv.* :
install : developer install of searx into virtualenv
uninstall : uninstall developer installation
cmd ... : run command ... in virtualenv
OK : test if virtualenv is OK
pypi.upload:
Upload python packages to PyPi (to test use pypi.upload.test)
test.* :
pylint : lint PYLINT_FILES, searx/engines, searx &amp; tests
pep8 : pycodestyle (pep8) for all files except PYLINT_FILES
unit : run unit tests
coverage : run unit tests with coverage
robot : run robot test
clean : clean intermediate test stuff
themes.* :
all : build all themes
oscar : build oscar theme
simple : build simple theme
----
run - run developer instance
install - developer install of searx into virtualenv
uninstall - uninstall developer installation
clean - clean up working tree
search.checker - check search engines
test - run shell &amp; CI tests
test.sh - test shell scripts
ci.test - run CI tests
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
<nav class="contents local" id="contents">
2020-02-15 10:08:58 +01:00
<p class="topic-title">Contents</p>
2019-12-30 18:31:17 +01:00
<ul class="simple">
<li><p><a class="reference internal" href="#python-environment" id="id8">Python environment</a></p></li>
<li><p><a class="reference internal" href="#make-run" id="id9"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">run</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-clean" id="id10"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">clean</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-docs-docs-autobuild-docs-clean" id="id11"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">docs</span> <span class="pre">docs.autobuild</span> <span class="pre">docs.clean</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-docs-gh-pages" id="id12"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">docs.gh-pages</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-test" id="id13"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">test</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-test-sh" id="id14"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">test.sh</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-test-pylint" id="id15"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">test.pylint</span></code></a></p></li>
<li><p><a class="reference internal" href="#search-checker-engine-name" id="id16"><code class="docutils literal notranslate"><span class="pre">search.checker.{engine</span> <span class="pre">name}</span></code></a></p></li>
<li><p><a class="reference internal" href="#make-pybuild" id="id17"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">pybuild</span></code></a></p></li>
2019-12-30 18:31:17 +01:00
</ul>
</nav>
<section id="python-environment">
<span id="make-install"></span><h2><a class="toc-backref" href="#id8" role="doc-backlink">Python environment</a><a class="headerlink" href="#python-environment" title="Permalink to this heading"></a></h2>
<aside class="sidebar">
2019-12-30 18:31:17 +01:00
<p class="sidebar-title">activate environment</p>
<p><code class="docutils literal notranslate"><span class="pre">source</span> <span class="pre">./local/py3/bin/activate</span></code></p>
</aside>
<p>We do no longer need to build up the virtualenv manually. Jump into your git
working tree and release a <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">install</span></code> to get a virtualenv with a
<em>developer install</em> of searx (<a class="reference external" href="https://github.com/searx/searx/blob/master/setup.py">Origin: setup.py</a>).</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ cd ~/searx-clone
$ make install
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e &#39;searx[test]&#39;
2019-12-30 18:31:17 +01:00
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
<p>If you release <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">install</span></code> multiple times the installation will only
rebuild if the sha256 sum of the <em>requirement files</em> fails. With other words:
the check fails if you edit the requirements listed in
<a class="reference external" href="https://github.com/searx/searx/blob/master/requirements-dev.txt">Origin: requirements-dev.txt</a> and <a class="reference external" href="https://github.com/searx/searx/blob/master/requirements.txt">Origin: requirements.txt</a>).</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ make install
PYENV OK
PYENV [virtualenv] requirements.sha256 failed
[virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt
[virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt
...
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e &#39;searx[test]&#39;
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
<aside class="sidebar">
2019-12-30 18:31:17 +01:00
<p class="sidebar-title">drop environment</p>
<p>To get rid of the existing environment before re-build use <a class="reference internal" href="#make-clean"><span class="std std-ref">clean target</span></a> first.</p>
</aside>
2019-12-30 18:31:17 +01:00
<p>If you think, something goes wrong with your ./local environment or you change
the <a class="reference external" href="https://github.com/searx/searx/blob/master/setup.py">Origin: setup.py</a> file, you have to call <a class="reference internal" href="#make-clean"><span class="std std-ref">make clean</span></a>.</p>
</section>
<section id="make-run">
<span id="id2"></span><h2><a class="toc-backref" href="#id9" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">run</span></code></a><a class="headerlink" href="#make-run" title="Permalink to this heading"></a></h2>
2019-12-30 18:31:17 +01:00
<p>To get up a running a developer instance simply call <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">run</span></code>. This enables
<em>debug</em> option in <a class="reference external" href="https://github.com/searx/searx/blob/master/searx/settings.yml">Origin: searx/settings.yml</a>, starts a <code class="docutils literal notranslate"><span class="pre">./searx/webapp.py</span></code>
2019-12-30 18:31:17 +01:00
instance, disables <em>debug</em> option again and opens the URL in your favorite WEB
browser (<a class="reference external" href="https://manpages.debian.org/jump?q=xdg-open">man: xdg-open</a>):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ make run
PYENV OK
SEARX_DEBUG=1 ./manage.sh pyenv.cmd python ./searx/webapp.py
2019-12-30 18:31:17 +01:00
...
INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
</section>
<section id="make-clean">
<span id="id3"></span><h2><a class="toc-backref" href="#id10" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">clean</span></code></a><a class="headerlink" href="#make-clean" title="Permalink to this heading"></a></h2>
<p>Drop all intermediate files, all builds, but keep sources untouched. Before
calling <code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">clean</span></code> stop all processes using <a class="reference internal" href="#make-install"><span class="std std-ref">Python environment</span></a>.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ make clean
CLEAN pyenv
PYENV [virtualenv] drop ./local/py3
CLEAN docs -- ./build/docs ./dist/docs
CLEAN locally installed npm dependencies
CLEAN test stuff
CLEAN common files
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
</section>
<section id="make-docs-docs-autobuild-docs-clean">
<span id="make-docs"></span><h2><a class="toc-backref" href="#id11" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">docs</span> <span class="pre">docs.autobuild</span> <span class="pre">docs.clean</span></code></a><a class="headerlink" href="#make-docs-docs-autobuild-docs-clean" title="Permalink to this heading"></a></h2>
<p>We describe the usage of the <code class="docutils literal notranslate"><span class="pre">doc.*</span></code> targets in the <a class="reference internal" href="contribution_guide.html#contrib-docs"><span class="std std-ref">How to contribute /
2019-12-30 18:31:17 +01:00
Documentation</span></a> section. If you want to edit the documentation
read our <a class="reference internal" href="contribution_guide.html#make-docs-live"><span class="std std-ref">live build</span></a> section. If you are working in your own brand,
adjust your <a class="reference internal" href="../admin/settings.html#settings-global"><span class="std std-ref">Global Settings</span></a>.</p>
</section>
<section id="make-docs-gh-pages">
<span id="id4"></span><h2><a class="toc-backref" href="#id12" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">docs.gh-pages</span></code></a><a class="headerlink" href="#make-docs-gh-pages" title="Permalink to this heading"></a></h2>
<p>To deploy on github.io first adjust your <a class="reference internal" href="../admin/settings.html#settings-global"><span class="std std-ref">Global Settings</span></a>. For any
further read <a class="reference internal" href="contribution_guide.html#deploy-on-github-io"><span class="std std-ref">deploy on github.io</span></a>.</p>
</section>
<section id="make-test">
<span id="id5"></span><h2><a class="toc-backref" href="#id13" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">test</span></code></a><a class="headerlink" href="#make-test" title="Permalink to this heading"></a></h2>
<p>Runs a series of tests: <a class="reference internal" href="#make-test-pylint"><span class="std std-ref">make test.pylint</span></a>, <code class="docutils literal notranslate"><span class="pre">test.pep8</span></code>, <code class="docutils literal notranslate"><span class="pre">test.unit</span></code>
and <code class="docutils literal notranslate"><span class="pre">test.robot</span></code>. You can run tests selective, e.g.:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ make test.pep8 test.unit test.sh
TEST test.pep8 OK
...
TEST test.unit OK
...
TEST test.sh OK
</pre></div>
</div>
</section>
<section id="make-test-sh">
<span id="id6"></span><h2><a class="toc-backref" href="#id14" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">test.sh</span></code></a><a class="headerlink" href="#make-test-sh" title="Permalink to this heading"></a></h2>
<p><a class="reference internal" href="../admin/buildhosts.html#sh-lint"><span class="std std-ref">Lint shell scripts</span></a> / if you have changed some bash scripting run this test before
commit.</p>
</section>
<section id="make-test-pylint">
<span id="id7"></span><h2><a class="toc-backref" href="#id15" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">test.pylint</span></code></a><a class="headerlink" href="#make-test-pylint" title="Permalink to this heading"></a></h2>
<p><a class="reference external" href="https://www.pylint.org/">Pylint</a> is known as one of the best source-code, bug and quality checker for the
Python programming language. The pylint profile we use at searx project is
found in projects root folder <a class="reference external" href="https://github.com/searx/searx/blob/master/.pylintrc">Origin: .pylintrc</a>.</p>
</section>
<section id="search-checker-engine-name">
<span id="make-search-checker"></span><h2><a class="toc-backref" href="#id16" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">search.checker.{engine</span> <span class="pre">name}</span></code></a><a class="headerlink" href="#search-checker-engine-name" title="Permalink to this heading"></a></h2>
<p>To check all engines:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>make search.checker
</pre></div>
</div>
<p>To check a engine with whitespace in the name like <em>google news</em> replace space
by underline:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>make search.checker.google_news
</pre></div>
</div>
<p>To see HTTP requests and more use SEARX_DEBUG:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>make SEARX_DEBUG=1 search.checker.google_news
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
<p>To filter out HTTP redirects (<a class="reference external" href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection">3xx</a>):</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>make SEARX_DEBUG=1 search.checker.google_news | grep -A1 &quot;HTTP/1.1\&quot; 3[0-9][0-9]&quot;
...
Engine google news Checking
https://news.google.com:443 &quot;GET /search?q=life&amp;hl=en&amp;lr=lang_en&amp;ie=utf8&amp;oe=utf8&amp;ceid=US%3Aen&amp;gl=US HTTP/1.1&quot; 302 0
https://news.google.com:443 &quot;GET /search?q=life&amp;hl=en-US&amp;lr=lang_en&amp;ie=utf8&amp;oe=utf8&amp;ceid=US:en&amp;gl=US HTTP/1.1&quot; 200 None
--
https://news.google.com:443 &quot;GET /search?q=computer&amp;hl=en&amp;lr=lang_en&amp;ie=utf8&amp;oe=utf8&amp;ceid=US%3Aen&amp;gl=US HTTP/1.1&quot; 302 0
https://news.google.com:443 &quot;GET /search?q=computer&amp;hl=en-US&amp;lr=lang_en&amp;ie=utf8&amp;oe=utf8&amp;ceid=US:en&amp;gl=US HTTP/1.1&quot; 200 None
--
</pre></div>
</div>
</section>
<section id="make-pybuild">
<h2><a class="toc-backref" href="#id17" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">make</span> <span class="pre">pybuild</span></code></a><a class="headerlink" href="#make-pybuild" title="Permalink to this heading"></a></h2>
<p>Build Python packages in <code class="docutils literal notranslate"><span class="pre">./dist/py</span></code>:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>$ make pybuild
2019-12-30 18:31:17 +01:00
...
BUILD pybuild
running sdist
running egg_info
...
running bdist_wheel
$ ls ./dist
searx-0.18.0-py3-none-any.whl searx-0.18.0.tar.gz
2019-12-30 18:31:17 +01:00
</pre></div>
</div>
<p>To upload packages to <a class="reference external" href="https://pypi.org/">PyPi</a>, there is also a <code class="docutils literal notranslate"><span class="pre">pypi.upload</span></code> target (to test use
<code class="docutils literal notranslate"><span class="pre">pypi.upload.test</span></code>). Since you are not the owner of <a class="reference external" href="https://pypi.org/project/searx">PyPi: searx</a> you will
never need to upload.</p>
</section>
</section>
2019-12-30 18:31:17 +01:00
2020-09-30 10:35:05 +02:00
<div class="clearer"></div>
2019-12-30 18:31:17 +01:00
</div>
</div>
</div>
<span id="sidebar-top"></span>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/searx_logo_small.png" alt="Logo"/>
</a></p>
<h3>Project Links</h3>
<ul>
<li><a href="https://searx.github.io/searx/blog/index.html">Blog</a>
2020-09-30 10:35:05 +02:00
<li><a href="https://github.com/searx/searx">Source</a>
2019-12-30 18:31:17 +01:00
2020-09-30 10:35:05 +02:00
<li><a href="https://github.com/searx/searx/wiki">Wiki</a>
2019-12-30 18:31:17 +01:00
<li><a href="https://twitter.com/Searx_engine">Twitter</a>
<li><a href="https://github.com/searx/searx/issues">Issue Tracker</a>
2019-12-30 18:31:17 +01:00
</ul><h3>Navigation</h3>
<ul>
<li><a href="../index.html">Overview</a>
<ul>
<li><a href="index.html">Developer documentation</a>
<ul>
<li>Previous: <a href="translation.html" title="previous chapter">Translation</a>
<li>Next: <a href="reST.html" title="next chapter">reST primer</a></ul>
</li>
</ul>
</li>
</ul>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
2019-12-30 18:31:17 +01:00
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>document.getElementById('searchbox').style.display = "block"</script>
2019-12-30 18:31:17 +01:00
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer" role="contentinfo">
&#169; Copyright 2015-2022, Adam Tauber, Noémi Ványi.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 5.3.0.
2019-12-30 18:31:17 +01:00
</div>
2020-02-15 10:08:58 +01:00
<script src="../_static/version_warning_offset.js"></script>
2019-12-30 18:31:17 +01:00
</body>
</html>