Exceptions in the execution of the callback must be caught / ignored and logged
on the ERROR log.
To test, apply this patch to provoke a ValueError exception::
diff --git a/searx/data/tracker_patterns.py b/searx/data/tracker_patterns.py
index ed4415bce..695ed05d2 100644
--- a/searx/data/tracker_patterns.py
+++ b/searx/data/tracker_patterns.py
@@ -114,6 +114,7 @@ class TrackerPatternsDB:
Returns bool ``True`` to use URL unchanged (``False`` to ignore URL).
If URL should be modified, the returned string is the new URL to use.
"""
+ raise ValueError("test callback exceptions")
new_url = url
parsed_new_url = urlparse(url=new_url)
Start a `make run` instance and query for example `amazon` .. have a look at the
ERROR log:
ERROR searx.result_types: filter_urls (field 'url'): ignore ValueError('test callback exceptions') from callback searx/data/tracker_patterns.py:117
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation is normalized, type annotations are applied, and the results
are freed from the HTML markup (which is partially present).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the result-list, the ``number_of_results`` indicate the number of hits in the
Index, they do not indicate how many results are in the answer.
In the past, search engines such as google or ddg had an indication on the first
page of a search term of how many hits there were for this term in total in
their index.
This info was added up in SearXNG and delivered under ``number_of_results``.
Nowadays the search engines no longer indicate how many hits there are in the
index and so this field in SearXNG is also superfluous.
- https://github.com/searxng/searxng/issues/2457#issuecomment-2566181574
- https://github.com/searxng/searxng/issues/2987
- https://github.com/searxng/searxng/issues/5034
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
[mod] normalize variable name for the max number of results per request
In the past, we have used different names for the variable that specifies the
maximum number of hits in the outgoing request.
- ``page_size``
- ``number_of_results``
- ``nb_per_page``
Since *page_size* is the most accurate term and is also used in the XPath
engines, all other engines are adjusted accordingly within this
patch .. documentation adjusted accordingly.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Previously, `update_engine_traits.py` would fetch traits for all engines, which
is very slow and by side-effect touches engine data that are unrelated to the
engine you're currently working on.
To be faster with developing `update_engine_traits.py` supports now engine
arguments.
To test, jump into the developer environment and run the script::
$ ./manage dev.env
(dev.env)$ ./searxng_extra/update/update_engine_traits.py --help
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The readability can be improve with the following two changes:
Slightly increase the content width to make the engine list more readable::
.content {
width: 52em; /* instead of 46em */
}
To avoid word wrap in bang::
p code.literal {
text-wrap: nowrap;
}
Sugested-by: https://github.com/searxng/searxng/pull/3408#issuecomment-2094082144
The type checker in my IDE shut down after over 500 errors / after this
patch there are still 125 criticisms, however its an improvement and a better
starting point.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The AOL engines deliver no results or too many incorrect results; in #5972 it is
reported that the AOL-images do not work, and on AOL-web the language selection
seems to depend more on the IP than on the selected language / Locally, the
language selection works for me, on the public server only English results are
displayed, which significantly worsens the result list.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
PR #5947 upgraded the pygments dependency, this commit updates
the static files ::
$ make themes.all
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Karmasearch seem to crash when searching with long queries >= 100 characters.
The returned JSON is exactly this `["",[]]`, which will crash when trying to
access `resp.json()["results"]`
Close: https://github.com/searxng/searxng/issues/5911
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The end-of-life (EOL) of a Python release is reached after 5 years, with the
last three years dedicated to security fixes. [1]
Unfortunately, this doesn't apply to common libraries (dependencies): bug fixes
are often only included in major releases, and minor releases with corresponding
security fixes are rarely offered.
To make matters worse, these dependencies often prematurely discontinue their
support for older Python releases (because, for example, they want/need to use
new Python features).
If we want to offer secure software, we are faced with the dilemma of either
supporting old Python releases and accepting that there are security bugs in the
dependencies, or ending support for older Python versions before the actual EOL
of the Python release.
[1] https://devguide.python.org/versions/
Closes: https://github.com/searxng/searxng/issues/5869
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Submitting an issue on GitHub isn't a end user task .. most issue reports coming
from this form are not filled out, since end users are not familiar with the
needs of a developer community.
Closes: https://github.com/searxng/searxng/issues/5820
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The online engines emulate a request as it would come from a web browser, which
is why the HTTP headers in the default settings should also be set the way a
standard web browser would set them.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The default settings for the suspend times were previously 24 hours and 3 hours,
respectively. Based on my experience, these defaults are too high; most engines
handle suspend times of 3 minutes or 1 hour (captcha) without any problems.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Package typer-slim does nothing other than depend on typer. The only reason this
package exists is as a migration path for old projects that used to depend on
typer-slim, so that they can get the latest version of typer.
Install instead:
pip install typer
Package typer-slimis deprecated and will stop receiving any updates and
published versions.
[1] https://pypi.org/project/typer-slim/
Closes: https://github.com/searxng/searxng/issues/5742
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Unit test fails::
Traceback (most recent call last):
File "/share/searxng/local/py3/lib/python3.10/site-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
File "/share/searxng/tests/unit/test_locales.py", line 121, in test_locale_optimized_territory
self.assertEqual(locales.match_locale(locale, locale_list), expected_locale)
AssertionError: 'fr-CH' != 'fr-BE'
- fr-CH
+ fr-BE
With the `babel` update from 2.17.0 to 2.18.0 the population DB has been
updated (the test was implemented for the old values).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
For unknown locales, the return value of::
locales.get_locale(params['searxng_locale'])
is None which cuase the following issue::
ERROR searx.engines.presearch : exception : 'NoneType' object has no attribute 'territory'
Traceback (most recent call last):
File "search/processors/online.py", line 256, in search
search_results = self._search_basic(query, params)
File "search/processors/online.py", line 231, in _search_basic
self.engine.request(query, params)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "engines/presearch.py", line 153, in request
request_id, cookies = _get_request_id(query, params)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "engines/presearch.py", line 140, in _get_request_id
if l.territory:
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'territory'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To test this patch build a python wheel::
$ make clean py.build
and llok out if you are missing any files in the wheel::
$ unzip -l dist/searxng-*-py3-none-any.whl
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The error only occurs in the CI action, which is why we need to commit the coded
debug. As soon as the bug is identified and fixed, this commit can be reverted
/ the ``set -x`` can be removed from the code.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
brave web:
xpath selectors needed to be justified
brave images & videos:
The JS code with the JS object was read incorrectly; not always, but quite
often, it led to exceptions when the Python data structure was created from it.
BTW: A complete review was conducted and corrections or additions were made to
the type definitions.
To test all brave engines in once::
!br !brimg !brvid !brnews weather
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch is based on PR #2792 (old PR from 2023)
- js_obj_str_to_python handle more cases
- bring tests from chompjs ..
- comment out tests do not pass
The tests from chompjs give some overview of what is not implemented.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- The three Yandex engines should use the same network context.
- There is no reason to set these engines inactive
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The code injection and monkey patching examine the names in the module of the
engine; if a variable there starts without an underscore and has the value None,
then this variable needs to be configured. This outdated concept does not fit
engines that may have multiple URLs. At least not as long as the value of the
base URL (list) is None.
The default is now an empty list instead of None
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- if engine load fails, set the engine to inactive
- dont' load a engine, when the config says its inactive
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The requests changed here all run outside of the network context timeout,
thereby preventing the engine's timeout from being applied (the engine's timeout
can become longer than it was configured).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The JS string, whose encoding will be corrupted if all single quotes (followed
by a comma) are replaced with double quotes. Bug was introduced in PR #4573.
Here is a simple example in which the list get corrupted::
>>> s = r"""[ 'foo\'', 'bar']"""
>>> print(s)
[ 'foo\'', 'bar']
>>> print(s.replace("',", "\","))
[ 'foo\'", 'bar']
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To complete a SearXNG developer environment, nvm needs to be
installed (ensured). Without this patch::
$ LANG=C ./manage dev.env
...
./utils/lib_nvm.sh: line 27: .nvm/nvm.sh: No such file or directory
./utils/lib_nvm.sh: line 28: .nvm/bash_completion: No such file or directory
...
(dev.env)$
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since #5280 has been merged, msgspec, like yaml, is a fixed part of the SearXNG
*settings framework* and therefore, like yaml, must be installed in the virtual
environment before installing SearXNG (``searx``).
The actual reason is that in SearXNG we store settings in the configuration that
are required for the installation of the ``searx`` package. This means that
these settings (from settings.yml) are read in during the installation, and all the
necessary tools for this (pyyaml, msgspec, setuptools, etc.) must be installed
beforehand (chicken or the egg dilemma).
Related:
- https://github.com/searxng/searxng/pull/5346
- https://github.com/searxng/searxng/pull/5280
- https://github.com/searxng/searxng/pull/5254
Closes: https://github.com/searxng/searxng/issues/5352
Since #5280 has been merged, msgspec, like yaml, is a fixed part of the SearXNG
*settings framework* and therefore, like yaml, must be installed in the virtual
environment before installing SearXNG (``searx``).
The actual reason is that in SearXNG we store settings in the configuration that
are required for the installation of the ``searx`` package. This means that
these settings (from settings.yml) are read in during the installation, and all the
necessary tools for this (pyyaml, msgspec, setuptools, etc.) must be installed
beforehand (chicken or the egg dilemma).
Related:
- https://github.com/searxng/searxng/pull/5280
- https://github.com/searxng/searxng/pull/5254
Closes: https://github.com/searxng/searxng/issues/5343
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This PR adds a new result type: File
Python class: searx/result_types/file.py
Jinja template: searx/templates/simple/result_templates/file.html
CSS (less) client/simple/src/less/result_types/file.less
Class 'File' (singular) replaces template 'files.html' (plural). The renaming
was carried out because there is only one file (singular) in a result. Not to be
confused with the category 'files' where in multiple results can exist.
As mentioned in issue [1], the class '.category-files' was removed from the CSS
and the stylesheet was adopted in result_types/file.less (there based on the
templates and no longer based on the category).
[1] https://github.com/searxng/searxng/issues/5198
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The size of the full-size images from ``thumbnail.url`` is usually several
MB. By reducing the full-size image to 80 pixels, the data size for a thumb is
reduced from MB to a few KB.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The settings are currently an untyped key/value structure, whose types are
dynamically built at runtime. The construction process of this structure
is *hand-crafted*.
In the long term, we want a static typing of this structure, based on a standard
tool. The ``msgspec.Struct`` structures are suitable as a standard tool.
This patch makes a first step towards static typing and implements the "brand"
section using ``msgspec.Struct`` structures.
BTW: searx/settings_defaults.py - ``git_url`` and ``git_branch`` had been
removed in aee613d256, this is a leftover.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Starting with Python 3.14 msgspec reports::
File "/share/searxng/searx/weather.py", line 261, in <module>
class Temperature(msgspec.Struct, kw_only=True):
...<60 lines>...
return template.format(value=val_str, unit=unit)
TypeError: Using a non-empty mutable collection (['°C', '°F', 'K']) \
as a default value is unsafe.\
Instead configure a `default_factory` for this field.
The problem is solved by the fact that there are now global constants for the
units (BTW singular/plural names of the type definitions are fixed):
- TEMPERATURE_UNITS
- PRESSURE_UNITS
- WIND_SPEED_UNITS
- RELATIVE_HUMIDITY_UNITS
- COMPASS_POINTS
- COMPASS_UNITS
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the case of .. response, for example, an HTTP 302 is returned by Google
Scholar::
Our systems have detected unusual traffic from your computer
network. Please try again later.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
When installing SearXNG (e.g.):
pip install --use-pep517 --no-build-isolation -e .
An import exception is raised:
ModuleNotFoundError: No module named 'lxml'
The ``setup.py`` file imports ``searx``, which in turn triggers various other
imports. However, the name XPath is only needed for type checking.
Closes: https://github.com/searxng/searxng/issues/5177
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
cppreference has replaced its search (``mwiki/index.php?title=``) with a DDG
search.
The engine was first introduced in SearXNG with PR-3274 [1], and even back then
the mediawiki proved to be incompatible, which is why the API could not be used
at the time. Now there isn't even a dedicated search function anymore.. I think
the cppreference project suffers from a lack of maintenance.
[1] https://github.com/searxng/searxng/pull/3247
Closes: https://github.com/searxng/searxng/issues/5271
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This is an old and grumpy hack / SearXNG is a Flask application with
client/server structure, which can't be turned into a command line tool the way
it was done here.
Maintaining this hack is becoming increasingly complex the more we try to
remodel the core code, which is why we should now remove the hack from SearXNG.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In some engines, under certain circumstances, the content field can also have
the value ``None``; in these cases, a length check results in an exception::
File "/usr/local/searxng/searx/results.py", line 360, in merge_two_main_results
if len(other.content) > len(origin.content):
^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()
[1] https://github.com/searxng/searxng/issues/5250#issuecomment-3352863488
Reported-by: @scross01 [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The ``JSONEncoder`` (``format="json"``) must perform a conversion to the
built-in types for the ``msgspec.Struct``::
if isinstance(o, msgspec.Struct):
return msgspec.to_builtins(o)
The result types are already of type ``msgspec.Struct``, so they can be
converted into built-in types.
The field types (in the result type) that were not yet of type ``msgspec.Struct``
have been converted to::
searx.weather.GeoLocation@dataclass -> msgspec.Struct
searx.weather.DateTime -> msgspec.Struct
searx.weather.Temperature -> msgspec.Struct
searx.weather.PressureUnits -> msgspec.Struct
searx.weather.WindSpeed -> msgspec.Struct
searx.weather.RelativeHumidity -> msgspec.Struct
searx.weather.Compass -> msgspec.Struct
BTW: Wherever it seemed sensible, the typing was also modernized in the modified
files.
Closes: https://github.com/searxng/searxng/issues/5250
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser