Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This is useful for tests
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This patch will allow mmap() to fail and will make the reader
fall back to read from the given file descriptor.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Since libloc is currently only using mmap() to map any databases into
memory, all mapped address range needs to be page aligned.

In the past, every database was aligned to a 4k page boundary. Some
architectures use 64k pages by default which could therefore not open
the database.

This patch changes the alignment to 64k which should hopefully be a
good common denominator. However, older database cannot be opened.

If we would ever have to support an architecture which's page size
is not evenly fitting into 64k, we will have to implement an
alternative to mmap.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

We do not have any changes that need to be propagated down to the file.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This will allow us to build this package when systemd is not available
(e.g. on Debian kfreebsd/hurd).
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Those functions have been very inconsistent in that way that some
happily accepted a NULL pointer and others didn't.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

errno is not either set properly, or left set properly by any function
that we have called.

Return codes should either be zero or non-zero for functions unless they
are some value.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This fixes:

Traceback (most recent call last):
  File "/usr/bin/location-importer", line 1607, in <module>
    main()
  File "/usr/bin/location-importer", line 1605, in main
    c.run()
  File "/usr/bin/location-importer", line 140, in run
    ret = args.func(args)
  File "/usr/bin/location-importer", line 1234, in handle_update_overrides
    self._update_overrides_for_spamhaus_drop()
  File "/usr/bin/location-importer", line 1504, in _update_overrides_for_spamhaus_drop
    for sline in t.readlines():
NameError: name 't' is not defined
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

To generate deterministic results
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Since we are breaking the import into smaller chunks now, it might be
that some data already exists in the database. This is now being
ignored and data won't be replaced.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

When importing source data fails for one source, we won't import
anything at all.

This patch changes that behaviour so that we will import all data one
after the other but continue if there was a problem with one source.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

This patch changes the order of some operations and drops a few
redundant ones (potentiall).

That should allow us to parse blocked data faster.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

All imports should have been conducted in one large transaction so that
we can remove any previous data.

This was not the case because of an indentation issue and could have
caused that the transaction was commited without all data being
successfully re-imported.

Fixes: #12852
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

The downloader used to open a connection to the web server hosting our
content which would have been decompressed (if necessary) on the fly and
also been parsed on the fly so that it could have been fed into the
database easily.

Some webservers do not seem to be patient enough to keep the connection
open if things take a little bit longer than usual. That caused the
import to fail.

This patch changes the behaviour that we would download all content
first, store it locally, and then start processing it.

Fixes: #12852
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Cc: Peter Müller <peter.mueller@ipfire.org>

The check block was indented incorrectly and would have never been
executed.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>