Location August 2022

location@lists.ipfire.org

4 participants
3 discussions

[PATCH] location-importer.in: Conduct sanity checks per DROP list
by Peter Müller 04 Oct '22

04 Oct '22

Previously, the lack of distinction between different DROP lists caused only the last one to be persisted. Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 58 +++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 8d47497..e4a9ab0 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1427,18 +1427,21 @@ class CLI(object): def _update_overrides_for_spamhaus_drop(self): downloader = location.importer.Downloader() - ip_urls = [ - "https://www.spamhaus.org/drop/drop.txt", - "https://www.spamhaus.org/drop/edrop.txt", - "https://www.spamhaus.org/drop/dropv6.txt" + ip_lists = [ + ("Spamhaus DROP list", "https://www.spamhaus.org/drop/drop.txt"), + ("Spamhaus EDROP list", "https://www.spamhaus.org/drop/edrop.txt"), + ("Spamhaus DROPv6 list", "https://www.spamhaus.org/drop/dropv6.txt") ] - asn_urls = [ - "https://www.spamhaus.org/drop/asndrop.txt" + asn_lists = [ + ("Spamhaus ASN-DROP list", "https://www.spamhaus.org/drop/asndrop.txt") ] - for url in ip_urls: - # Fetch IP list + for list in ip_lists: + name = list[0] + url = list[1] + + # Fetch IP list from given URL f = downloader.retrieve(url) # Split into lines @@ -1448,11 +1451,11 @@ class CLI(object): # downloads. if len(fcontent) > 10: self.db.execute(""" - DELETE FROM autnum_overrides WHERE source = 'Spamhaus ASN-DROP list'; - DELETE FROM network_overrides WHERE source = 'Spamhaus DROP lists'; - """) + DELETE FROM network_overrides WHERE source = '%s'; + """ % name, + ) else: - log.error("Spamhaus DROP URL %s returned likely bogus file, ignored" % url) + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) continue # Iterate through every line, filter comments and add remaining networks to @@ -1475,8 +1478,8 @@ class CLI(object): # Sanitize parsed networks... if not self._check_parsed_network(network): - log.warning("Skipping bogus network found in Spamhaus DROP URL %s: %s" % \ - (url, network)) + log.warning("Skipping bogus network found in %s (%s): %s" % \ + (name, url, network)) continue # Conduct SQL statement... @@ -1488,14 +1491,31 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (network) DO UPDATE SET is_drop = True""", "%s" % network, - "Spamhaus DROP lists", + name, True ) - for url in asn_urls: + for list in asn_lists: + name = list[0] + url = list[1] + # Fetch URL f = downloader.retrieve(url) + # Split into lines + fcontent = f.readlines() + + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM autnum_overrides WHERE source = '%s'; + """ % name, + ) + else: + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) + continue + # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... with self.db.transaction(): @@ -1518,8 +1538,8 @@ class CLI(object): # Filter invalid ASNs... if not self._check_parsed_asn(asn): - log.warning("Skipping bogus ASN found in Spamhaus DROP URL %s: %s" % \ - (url, asn)) + log.warning("Skipping bogus ASN found in %s (%s): %s" % \ + (name, url, asn)) continue # Conduct SQL statement... @@ -1531,7 +1551,7 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (number) DO UPDATE SET is_drop = True""", "%s" % asn, - "Spamhaus ASN-DROP list", + name, True ) -- 2.35.3

3 9

Re: Upload libloc to Debian
by Jochen Sprickerhof 17 Aug '22

17 Aug '22

[resent for the location mailing lits] Hi Michael, * Michael Tremer <michael.tremer(a)ipfire.org> [2022-07-01 14:19]: >Hello Jochen, > >Thank you for your email. > >The best place to get in touch with our team is our mailing list over here: > > https://lists.ipfire.org/mailman/listinfo/location > >However, I am not sure if you prefer to talk about this matter in private for the time being. Fine with me, added to Cc. >> On 1 Jul 2022, at 13:14, Jochen Sprickerhof <jochen(a)sprickerhof.de> wrote: >> >> Hi Stefan, >> >> I'm a Debian Developer and F-Droid team member. One part of the F-Droid tooling is called issuebot [1] a bot to automatically check new app submissions. We plan to integrate libloc into issuebot [2] and would like to upload it to Debian for that. I've found your packaging work and would use it as a basis if you agree and would have some questions: > >That sounds great! > >First of all, we would like to have more users to libloc. So far we have been trying to work together with some projects and that has been great cooperations, but our time is of course limited, so I am very happy to hear, that you found our little piece of software and went ahead and integrated it. > >> - What is the license of the database itself, i.e., what is hosted here? >> >> https://location.ipfire.org/databases/1/ > >The database itself is licensed under CC-BY-SA 4.0. > >We selected this, because it is the same license that Maxmind used to use, and there has been a debate on the Debian-legal list which concluded in that being the best license that we can use for this. That sounds great, could you add a licence file to it, maybe? >> - Would it make sense to package the database along with the library? > >First of all, it would be great to have some help to get the library into Debian. That would help us a lot to integrate it into other software. We already have Tor using it, and there are patches for Suricata (the latter might not be packaged for Debian yet either). Suricata seems to be there: https://packages.debian.org/sid/suricata I did a quick grep in the source but did not see references to libloc, do you know how it is integrated? >In IPFire we do this: We ship a recent-ish version of the database that we update every time we update libloc. That way, users have at least a base to start from, but the updater will update that file very soon. > >I believe that Debian is doing the same with clamav which has a meta package with a version of the database. I think following that would make sense from my point of view. Sounds great, I see that the updated puts the file in /var/lib/location/database.db But I don't see where it is used. The libloc-database package would probably install to /usr/share/libloc-database/database.db. Would it be easier if libloc uses that path as a backup if the one in /var is not there or should the package add a symlink to be overwritten by the updater? >> - According to the Debian Python policy the Python package needs to be renamed to python3-location. It would simplify the transition if you could do the same in your package repo and add a transition package [3]. > >Oh, I named it that way because that is how Python packages were usually named in Debian. We can of course absolutely change this. > >Would you be up for sending a patch for this? Sure, as a patch to the list or a pull request on Github? >That might be a lot easier since you are already an expert with the Debian packaging system. We have been fighting a couple of problems that we couldn’t really solve. Maybe you can help us with those, too: https://bugzilla.ipfire.org/show_bug.cgi?id=12468 I was not able to reproduce this, could you send me a log of when this happens? How do you build the package? With the debian/build.sh? >> - Similar I would propose to merge the location-importer package into the python package (for the library part) and the location package (for the binary). > >We tried to keep the main package as small as possible. The importer part is not very large, but it does have a dependency to PostgreSQL. Hence we split it, because regular users won’t need it. Oh, that does not seem to be modeled in the package. Is the location-importer by normal users of libloc? Otherwise I would propose to add a suggest to PostgreSQL only. >I don’t have any hard feelings on changing this. What is your motivation for the change? We try not to split the packages into too small parts in Debian. >> - Finally, are you interested in help maintaining the lib in Debian? > >Yes, absolutely. We have been calling for people to help us getting the library into distributions, because we currently do not have the man power to dig into the intricacies of the build systems of all the major distributions. Hence we need you! > >But we are of course here to help, test, fix bugs and what ever else needs to be done. > >What else does this entail that I didn’t cover, yet? This is primary a questions how is listed as a maintainer for the package. Packages in Debian need exactly one maintainer which could be a person or a team. In addition there can be multiple Uploaders listed who effectively maintain the package as well (this is due to history reasons). We could use use tracker.debian.org as the team, i.e.: Maintainer: libloc maintainers <libloc(a)tracker.debian.org> And add however of you want as Uploaders. Note that you would have to ask me or an other Debian Developer to actually upload the package. Also I would put the Debian packaging on https://salsa.debian.org Where most Debian packages are hosted. I'm happy to guide you if you want to help with it. >Thanks for getting in touch! > >-Michael > >> >> Cheers Jochen >> >> [1]: https://gitlab.com/fdroid/issuebot >> [2]: https://gitlab.com/fdroid/issuebot/-/issues/61 >> [3]: https://wiki.debian.org/RenamingPackages >

2 12

[PATCH] location-importer.in: Fix dangling variable
by Peter Müller 14 Aug '22

14 Aug '22

This fixes: Traceback (most recent call last): File "/usr/bin/location-importer", line 1607, in <module> main() File "/usr/bin/location-importer", line 1605, in main c.run() File "/usr/bin/location-importer", line 140, in run ret = args.func(args) File "/usr/bin/location-importer", line 1234, in handle_update_overrides self._update_overrides_for_spamhaus_drop() File "/usr/bin/location-importer", line 1504, in _update_overrides_for_spamhaus_drop for sline in t.readlines(): NameError: name 't' is not defined Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org> --- src/scripts/location-importer.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 1785791..8d47497 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1499,7 +1499,7 @@ class CLI(object): # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... with self.db.transaction(): - for sline in t.readlines(): + for sline in f.readlines(): # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") -- 2.35.3

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Location August 2022