importer: Purge any redundant entries
When importing inetnums, we might import various small networks which are not relevant for us as long as they do not have a different country code than their parent network. Therefore we delete all these entries to keep the database smaller without losing any information. The second version of this patch introduces a SQL statement parallelised across all CPUs available, while the DELETE-statement of the first version literally took ages to complete. However, cleaning up those data still takes about 26 hours (!) on our location02 testing machine, making daily updates of the location database impossible to the current knowledge. real 1521m30.620s user 38m45.521s sys 9m6.027s Special thanks goes to Michael for spending numerous hours on this, setting up a testing environment, doing PostgreSQL magic and providing helpful advice while debugging. Partially fixes: #12458 Cc: Michael Tremer <michael.tremer@ipfire.org> Signed-off-by:Peter Müller <peter.mueller@ipfire.org> Signed-off-by:
Michael Tremer <michael.tremer@ipfire.org>
Showing
Please register or sign in to comment