devol
/
searx
mirror of https://github.com/searx/searx
1
0
Fork 0
Code Issues Releases Wiki Activity
searx/admin/filtron.html

238 lines
15 KiB
HTML
Raw Normal View History

various mods: uwsgi setup and version update ++ py3 img src fix ++ new sphinx version - closes #939
2017-06-12 12:34:30 +02:00
gh-pages: updated
2019-12-30 18:31:17 +01:00
<!DOCTYPE html>
add generated files of parent
2016-10-30 01:02:58 +02:00
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1">
gh-pages: updated
2020-02-15 10:08:58 +01:00
<title>How to protect an instance &#8212; Searx Documentation (Searx-0.16.0.tex)</title>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<link rel="stylesheet" href="../_static/searx.css" type="text/css" />
add generated files of parent
2016-10-30 01:02:58 +02:00
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
gh-pages: updated
2020-02-15 10:08:58 +01:00
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/language_data.js"></script>
add generated files of parent
2016-10-30 01:02:58 +02:00
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
[enh] add morty documentation
2017-08-08 15:19:02 +02:00
<link rel="next" title="How to setup result proxy" href="morty.html" />
gh-pages: updated
2019-12-30 18:31:17 +01:00
<link rel="prev" title="Architecture" href="architecture.html" />
<script>DOCUMENTATION_OPTIONS.URL_ROOT = '../';</script>
add generated files of parent
2016-10-30 01:02:58 +02:00
gh-pages: updated
2019-12-30 18:31:17 +01:00
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="morty.html" title="How to setup result proxy"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="architecture.html" title="Architecture"
accesskey="P">previous</a> |</li>
gh-pages: updated
2020-02-15 10:08:58 +01:00
<li class="nav-item nav-item-0"><a href="../index.html">Searx Documentation (Searx-0.16.0.tex)</a> &#187;</li>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Administrator documentation</a> &#187;</li>
</ul>
</div>
add generated files of parent
2016-10-30 01:02:58 +02:00
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="how-to-protect-an-instance">
<h1>How to protect an instance<a class="headerlink" href="#how-to-protect-an-instance" title="Permalink to this headline"></a></h1>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<p>Searx depens on external search services. To avoid the abuse of these services
it is advised to limit the number of requests processed by searx.</p>
<p>An application firewall, <code class="docutils literal notranslate"><span class="pre">filtron</span></code> solves exactly this problem. Information
on how to install it can be found at the <a class="reference external" href="https://github.com/asciimoo/filtron">project page of filtron</a>.</p>
add generated files of parent
2016-10-30 01:02:58 +02:00
<div class="section" id="sample-configuration-of-filtron">
<h2>Sample configuration of filtron<a class="headerlink" href="#sample-configuration-of-filtron" title="Permalink to this headline"></a></h2>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<p>An example configuration can be find below. This configuration limits the access
of:</p>
<ul class="simple">
<li><p>scripts or applications (roboagent limit)</p></li>
<li><p>webcrawlers (botlimit)</p></li>
<li><p>IPs which send too many requests (IP limit)</p></li>
<li><p>too many json, csv, etc. requests (rss/json limit)</p></li>
<li><p>the same UserAgent of if too many requests (useragent limit)</p></li>
add generated files of parent
2016-10-30 01:02:58 +02:00
</ul>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<div class="highlight-json notranslate"><div class="highlight"><pre><span></span><span class="p">[{</span>
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;search request&quot;</span><span class="p">,</span>
<span class="nt">&quot;filters&quot;</span><span class="p">:[</span>
<span class="s2">&quot;Param:q&quot;</span><span class="p">,</span>
<span class="s2">&quot;Path=^(/|/search)$&quot;</span>
<span class="p">],</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;time-interval-in-sec (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;limit&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;max-request-number-in-interval (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;subrules&quot;</span><span class="p">:[</span>
<span class="p">{</span>
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;roboagent limit&quot;</span><span class="p">,</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;time-interval-in-sec (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;limit&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;max-request-number-in-interval (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;filters&quot;</span><span class="p">:[</span>
<span class="s2">&quot;Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)&quot;</span>
<span class="p">],</span>
<span class="nt">&quot;actions&quot;</span><span class="p">:[</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<span class="p">{</span>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;block&quot;</span><span class="p">,</span>
<span class="nt">&quot;params&quot;</span><span class="p">:{</span>
<span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="s2">&quot;Rate limit exceeded&quot;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;botlimit&quot;</span><span class="p">,</span>
<span class="nt">&quot;limit&quot;</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span>
<span class="nt">&quot;stop&quot;</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span>
<span class="nt">&quot;filters&quot;</span><span class="p">:[</span>
<span class="s2">&quot;Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)&quot;</span>
<span class="p">],</span>
<span class="nt">&quot;actions&quot;</span><span class="p">:[</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<span class="p">{</span>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;block&quot;</span><span class="p">,</span>
<span class="nt">&quot;params&quot;</span><span class="p">:{</span>
<span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="s2">&quot;Rate limit exceeded&quot;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;IP limit&quot;</span><span class="p">,</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;time-interval-in-sec (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;limit&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;max-request-number-in-interval (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;stop&quot;</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span>
<span class="nt">&quot;aggregations&quot;</span><span class="p">:[</span>
<span class="s2">&quot;Header:X-Forwarded-For&quot;</span>
<span class="p">],</span>
<span class="nt">&quot;actions&quot;</span><span class="p">:[</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<span class="p">{</span>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;block&quot;</span><span class="p">,</span>
<span class="nt">&quot;params&quot;</span><span class="p">:{</span>
<span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="s2">&quot;Rate limit exceeded&quot;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;rss/json limit&quot;</span><span class="p">,</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;time-interval-in-sec (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;limit&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;max-request-number-in-interval (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;stop&quot;</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span>
<span class="nt">&quot;filters&quot;</span><span class="p">:[</span>
<span class="s2">&quot;Param:format=(csv|json|rss)&quot;</span>
<span class="p">],</span>
<span class="nt">&quot;actions&quot;</span><span class="p">:[</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<span class="p">{</span>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;block&quot;</span><span class="p">,</span>
<span class="nt">&quot;params&quot;</span><span class="p">:{</span>
<span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="s2">&quot;Rate limit exceeded&quot;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;useragent limit&quot;</span><span class="p">,</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;time-interval-in-sec (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;limit&quot;</span><span class="p">:</span><span class="s2">&quot;&lt;max-request-number-in-interval (int)&gt;&quot;</span><span class="p">,</span>
<span class="nt">&quot;aggregations&quot;</span><span class="p">:[</span>
<span class="s2">&quot;Header:User-Agent&quot;</span>
<span class="p">],</span>
<span class="nt">&quot;actions&quot;</span><span class="p">:[</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<span class="p">{</span>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="s2">&quot;block&quot;</span><span class="p">,</span>
<span class="nt">&quot;params&quot;</span><span class="p">:{</span>
<span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="s2">&quot;Rate limit exceeded&quot;</span>
<span class="p">}</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<span class="p">}</span>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}]</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
</pre></div>
</div>
</div>
<div class="section" id="route-request-through-filtron">
<h2>Route request through filtron<a class="headerlink" href="#route-request-through-filtron" title="Permalink to this headline"></a></h2>
<p>Filtron can be started using the following command:</p>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$ filtron -rules rules.json
add generated files of parent
2016-10-30 01:02:58 +02:00
</pre></div>
</div>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<p>It listens on <code class="docutils literal notranslate"><span class="pre">127.0.0.1:4004</span></code> and forwards filtered requests to
<code class="docutils literal notranslate"><span class="pre">127.0.0.1:8888</span></code> by default.</p>
<p>Use it along with <code class="docutils literal notranslate"><span class="pre">nginx</span></code> with the following example configuration.</p>
<div class="highlight-nginx notranslate"><div class="highlight"><pre><span></span><span class="k">location</span> <span class="s">/</span> <span class="p">{</span>
<span class="kn">proxy_set_header</span> <span class="s">Host</span> <span class="nv">$http_host</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">X-Real-IP</span> <span class="nv">$remote_addr</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">X-Forwarded-For</span> <span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">X-Scheme</span> <span class="nv">$scheme</span><span class="p">;</span>
<span class="kn">proxy_pass</span> <span class="s">http://127.0.0.1:4004/</span><span class="p">;</span>
<span class="p">}</span>
add generated files of parent
2016-10-30 01:02:58 +02:00
</pre></div>
</div>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<p>Requests are coming from port 4004 going through filtron and then forwarded to
port 8888 where a searx is being run.</p>
add generated files of parent
2016-10-30 01:02:58 +02:00
</div>
</div>
</div>
</div>
</div>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<span id="sidebar-top"></span>
add generated files of parent
2016-10-30 01:02:58 +02:00
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
gh-pages: updated
2019-12-30 18:31:17 +01:00
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/searx_logo_small.png" alt="Logo"/>
</a></p>
<h3>Project Links</h3>
<ul>
<li><a href="https://github.com/asciimoo/searx">Source</a>
<li><a href="https://github.com/asciimoo/searx/wiki">Wiki</a>
gh-pages: updated
2020-02-15 10:08:58 +01:00
<li><a href="https://searx.space/">Public instances</a>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<li><a href="https://twitter.com/Searx_engine">Twitter</a>
</ul><h3>Navigation</h3>
add generated files of parent
2016-10-30 01:02:58 +02:00
<ul>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<li><a href="../index.html">Overview</a>
<ul>
<li><a href="index.html">Administrator documentation</a>
<ul>
<li>Previous: <a href="architecture.html" title="previous chapter">Architecture</a>
<li>Next: <a href="morty.html" title="next chapter">How to setup result proxy</a></ul>
</li>
</ul>
</li>
add generated files of parent
2016-10-30 01:02:58 +02:00
</ul>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" />
<input type="submit" value="Go" />
</form>
</div>
add generated files of parent
2016-10-30 01:02:58 +02:00
</div>
gh-pages: updated
2020-02-15 10:08:58 +01:00
<script>$('#searchbox').show(0);</script>
add generated files of parent
2016-10-30 01:02:58 +02:00
</div>
</div>
<div class="clearer"></div>
</div>
gh-pages: updated
2019-12-30 18:31:17 +01:00
<div class="footer" role="contentinfo">
&#169; Copyright 2015-2019, Adam Tauber, Noémi Ványi.
gh-pages: updated
2020-02-15 10:08:58 +01:00
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 2.4.1.
add generated files of parent
2016-10-30 01:02:58 +02:00
</div>
gh-pages: updated
2020-02-15 10:08:58 +01:00
<script src="../_static/version_warning_offset.js"></script>
gh-pages: updated
2019-12-30 18:31:17 +01:00
add generated files of parent
2016-10-30 01:02:58 +02:00
</body>
</html>