Commits · e646a8f35ec7eff009414b3fd107c9af5cf39a86 · nexedi / libloc

16 Nov, 2020 1 commit

Implement filtering for multiple countries in the enumerator · e646a8f3

Michael Tremer authored Nov 16, 2020

This will allow us to speed up the export of the database
if only a few countries should be returned.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

e646a8f3

15 Nov, 2020 1 commit
- Move network lists into an own file · e0b9ff5f
  Michael Tremer authored Nov 15, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  e0b9ff5f
13 Nov, 2020 6 commits
- python: Move tree flattening into C · c242f732
  Michael Tremer authored Nov 13, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  c242f732
- network: Unexport all tree functions · 7933f5bf
  Michael Tremer authored Nov 13, 2020
```
These should not be exported
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  7933f5bf
- network: Add function to pop first element from stack · 8d777f12
  Michael Tremer authored Nov 13, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  8d777f12
- network: Sort result of excluded lists · d3375368
  Michael Tremer authored Nov 13, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  d3375368
- database: Filter results coming from stack · 2113e71b
  Michael Tremer authored Nov 13, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  2113e71b
- database: Move network filtering into a separate function · 69248038
  Michael Tremer authored Nov 13, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  69248038
12 Nov, 2020 15 commits
- networks: Copy all attributes when splitting networks · 594ca328
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  594ca328
- test: Update API · d3ae93c2
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  d3ae93c2
- database: Add option to return networks flattened · d87fd7a3
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  d87fd7a3
- network: Add function to exclude multiple networks at once · add5bb65
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  add5bb65
- network: Add new subnet function · 33a051e0
  Michael Tremer authored Nov 12, 2020
```
The old one is too difficult to use in terms of order
of input parameters and return value.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  33a051e0
- network: Add more debugging output to stacks · 9a7732c8
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  9a7732c8
- python: Export networks exclude function · 037c65d3
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  037c65d3
- network: Reduce debugging output · f5e50a47
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  f5e50a47
- database: Pass flag to enumerator to flatten output · 681ff05c
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  681ff05c
- network: Make lists unique · 6d22a179
  Michael Tremer authored Nov 12, 2020
```
Networks that are in the list won't be added again
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  6d22a179
- networks: Add function to merge two lists · f802f3a4
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  f802f3a4
- networks: Add function to check if network is part of a list · e52ba217
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  e52ba217
- networks: Add function to check if two networks overlap · 6159d384
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  6159d384
- network: Add functions to break network into subnets · 850e7516
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  850e7516
- networks: Add function to dump lists · 8b220527
  Michael Tremer authored Nov 12, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  8b220527
11 Nov, 2020 2 commits
- networks: Add list to manage groups of networks · ecce288d
  Michael Tremer authored Nov 11, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  ecce288d
- test: Add tests for database enumerator · e96704f4
  Michael Tremer authored Nov 11, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  e96704f4
06 Nov, 2020 1 commit

location-importer.in: always convert organisation handles into upper cases · e7d612e5

Peter Müller authored Nov 03, 2020

Fixes: #12523
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

e7d612e5

29 Oct, 2020 2 commits

location.in: do not confuse UTC with local time zones · 60c1ac03
Peter Müller authored Oct 29, 2020
```
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
```
60c1ac03

location update: Remove double conversion of timestamps · 0c74f6b1

Peter Müller authored Oct 29, 2020

This caused that the timestamp in DNS was misinterpreted
as local time and often, databases could not be downloaded.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>

0c74f6b1

28 Oct, 2020 1 commit

location: Fix Python syntax error in verify() · e99a7226

Michael Tremer authored Oct 28, 2020

The database is now being opened before the requested
method is called and handle_verify() wasn't updated.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

e99a7226

27 Oct, 2020 8 commits

export: Flatten the tree before exporting it · bbed1fd2

Michael Tremer authored Oct 27, 2020

This patch removes the possibility that any IP address ranges
might show up in multiple exported files.

If this was content from the database:

  * 10.0.0.0/16 - DE
  * 10.0.1.0/24 - FR

Then the IP address 10.0.1.1 would match for both countries.

The algorithm will now break the larger /16 subnet apart into
smaller subnets so that 10.0.1.0/24 is no longer overlapped.

There was some time spent on this to make this as efficient
as possible.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

bbed1fd2

location-importer.in: avoid log spam for too small networks · ebb087cf

Peter Müller authored Oct 21, 2020

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

ebb087cf

location-importer.in: omit historic/orphaned RIR data · 84187ab5

Peter Müller authored Oct 21, 2020

Some RIRs include detailled information regarding networks not managed
by or allocated to themselves, particually APNIC. We need to filter
those networks (they usually have a characteristic network name) in
order to prevent operational quirks or returning wrong country codes.

Fixes: #12501
Partially fixes: #12499

Cc: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

84187ab5

importer.py: fetch LACNIC data via HTTPS · eee65490

Peter Müller authored Oct 21, 2020

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

eee65490

location-importer.in: filter bogus IP networks for both Whois and extended sources · bd341642

Peter Müller authored Oct 21, 2020

Sanity checks for parsed networks have been put into a separate function
to avoid boilerplate code for extended sources. This makes the location
database less vulnerable to garbage written into RIR databases on
purpose or by chance.

Fixes: #12500
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

bd341642

export.py: fix exporting IP networks for crappy xt_geoip module · 28c73fa3

Peter Müller authored Oct 21, 2020

In contrast to the location database itself, the xt_geoip module
consumes a list of IP networks for each country, and returns after the
first match.

We therefore need to...

(a) sort IP networks by their size, allow as precise matches as possible
(b) export _any_ IP networks - including inverted subnets - to prevent
    undefined overlaps
(c) do the entire thing as fast as possible, consuming as less disk
    space as possible, which is why we can't just iterate over all /24
    chunks

Partially fixes: #12499
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

28c73fa3

Revert "Revert "location-importer.in: only import relevant data from AFRINIC, APNIC and RIPE"" · 002deb6b

Peter Müller authored Oct 21, 2020

This reverts commit 13f67f28.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

002deb6b

Revert "Revert "Revert "Revert "importer: Import raw sources for inetnum's again"""" · aadac4c5

Peter Müller authored Oct 21, 2020

This reverts commit 44341478.
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

aadac4c5

22 Oct, 2020 1 commit
- network: Allow adding single IP addresses and automatically add the prefix · 26ab419b
  Michael Tremer authored Oct 22, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  26ab419b
21 Oct, 2020 2 commits
- importer: Restructure SQL query to be executed in parallel · bbea93a7
  Michael Tremer authored Oct 21, 2020
```
There are no functional changes, this just runs quicker now.
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  bbea93a7
- importer: Add search index to network_overrides table · 991baf53
  Michael Tremer authored Oct 21, 2020
```
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
```
  991baf53