The downloader used to open a connection to the web server hosting our
content which would have been decompressed (if necessary) on the fly and
also been parsed on the fly so that it could have been fed into the
database easily.

Some webservers do not seem to be patient enough to keep the connection
open if things take a little bit longer than usual. That caused the
import to fail.

This patch changes the behaviour that we would download all content
first, store it locally, and then start processing it.

Fixes: #12852
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Cc: Peter Müller <peter.mueller@ipfire.org>

The check block was indented incorrectly and would have never been
executed.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

To comply with the Debian Python policy. Also add a transitional package
with the old name.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

These are all assigned to the US Department of Defense.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

The current way of truncating all override data straight away leaves us
with no data at all, should a source turn out to be unreachable or
returning bogus files (yes, Cloudflare, I _am_ looking at you).

It is therefore better to only delete data we know to have a valid
replacement for, rather than just dropping the source altogether.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

For the ipset format, we will rewrite the header after we know the total
number of entries that have been written in order to optimise the hash
table.

When the output stream isn't seekable, we cannot write the header again
which is being fixed in this patch.

Fixes: #12885
Reported-by: Jon Murphy <jon.murphy@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fixes: #12887
Reported-by: Jon Murphy <jon.murphy@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

In order to run the test suite, we need to make the Python module
loadable from the build directory so that we first of all test the right
code and that it just works without running "make install" first.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This reverts commit 643b8d06.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

When performing checks, it is useful to be able to rely on a flat
network plan so that any larger parent networks in some countries/ASes
won't match any subnets.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

We are using a couple of newer Python features which are only available
from 3.9 onwards.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Otherwise this test runs for forever
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

The prefix is being stored as a total number of bits now since it is
easier for any bitwise maths later. This is however causing an incorrect
prefix being computed when splitting a network into two subnets for
IPv4.

To get the correct prefix, loc_network_prefix must be called.

Introduced in 1fd09d0bReported-by: Valter Jansons <valter.jansons@gmail.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

If loc_network_subnets is being called with some invalid input, we just
fail without logging a useful message or setting errno accordingly.

This is being changed with this patch.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This is a performance improvement when exporting networks flattened. For
the subnet search, we allocate an empty list many times which is often
not required.

This patch changes this behaviour that (if needed) the lists will be
allocated and will stay around and cleared if necessary.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This patch moves the subqueries out of the large query, so that the
database will materialize them for faster lookup.

We also drop the "UNION ALL" and replace it with just "UNION" because we
do not want any duplicate networks. That will save us many iterations
later on.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

When we add a large number of ASes to the list, we will constantly
re-allocate the whole list because we are only growing it by 64 entries
at a time.

This patch changes that we will double the list in size every time we
run out of space and we will start with 1024 elements.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>