[PATCH 1/6] perl-Net-CIDR-Lite: New package.

public inbox for development@lists.ipfire.org
 help / color / mirror / Atom feed

* [PATCH 1/6] perl-Net-CIDR-Lite: New package.
@ 2019-01-10 12:00 Stefan Schantl
  2019-01-10 12:00 ` [PATCH 2/6] xtables-addons: Update to 3.2 Stefan Schantl
                   ` (4 more replies)
  0 siblings, 5 replies; 7+ messages in thread
From: Stefan Schantl @ 2019-01-10 12:00 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 4950 bytes --]

This is a runtime dependency of the xt_geoip_build perl script
shipped by xtables-addons in version 3.2.

Reference #11960.

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
---
 config/rootfiles/common/perl-Net-CIDR-Lite |  6 ++
 lfs/perl-Net-CIDR-Lite                     | 79 ++++++++++++++++++++++
 make.sh                                    |  1 +
 3 files changed, 86 insertions(+)
 create mode 100644 config/rootfiles/common/perl-Net-CIDR-Lite
 create mode 100644 lfs/perl-Net-CIDR-Lite

diff --git a/config/rootfiles/common/perl-Net-CIDR-Lite b/config/rootfiles/common/perl-Net-CIDR-Lite
new file mode 100644
index 000000000..176afa3d2
--- /dev/null
+++ b/config/rootfiles/common/perl-Net-CIDR-Lite
@@ -0,0 +1,6 @@
+#usr/lib/perl5/site_perl/5.12.3/Net/CIDR
+usr/lib/perl5/site_perl/5.12.3/Net/CIDR/Lite.pm
+#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/Net/CIDR
+#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/Net/CIDR/Lite
+#usr/lib/perl5/site_perl/5.12.3/x86_64-linux-thread-multi/auto/Net/CIDR/Lite/.packlist
+#usr/share/man/man3/Net::CIDR::Lite.3
diff --git a/lfs/perl-Net-CIDR-Lite b/lfs/perl-Net-CIDR-Lite
new file mode 100644
index 000000000..66d989111
--- /dev/null
+++ b/lfs/perl-Net-CIDR-Lite
@@ -0,0 +1,79 @@
+###############################################################################
+#                                                                             #
+# IPFire.org - A linux based firewall                                         #
+# Copyright (C) 2007-2018  IPFire Team  <info(a)ipfire.org>                     #
+#                                                                             #
+# This program is free software: you can redistribute it and/or modify        #
+# it under the terms of the GNU General Public License as published by        #
+# the Free Software Foundation, either version 3 of the License, or           #
+# (at your option) any later version.                                         #
+#                                                                             #
+# This program is distributed in the hope that it will be useful,             #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
+# GNU General Public License for more details.                                #
+#                                                                             #
+# You should have received a copy of the GNU General Public License           #
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
+#                                                                             #
+###############################################################################
+
+
+###############################################################################
+# Definitions
+###############################################################################
+include Config
+VER        = 0.21
+
+THISAPP    = Net-CIDR-Lite-$(VER)
+DL_FILE    = ${THISAPP}.tar.gz
+DL_FROM    = $(URL_IPFIRE)
+DIR_APP    = $(DIR_SRC)/$(THISAPP)
+TARGET     = $(DIR_INFO)/$(THISAPP)
+
+###############################################################################
+# Top-level Rules
+###############################################################################
+
+objects = $(DL_FILE)
+
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+
+$(DL_FILE)_MD5 = 12280b3754886b876918f03f53aee4f5
+
+install : $(TARGET)
+
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+
+md5 : $(subst %,%_MD5,$(objects))
+
+dist:
+	@$(PAK)
+
+###############################################################################
+# Downloading, checking, md5sum
+###############################################################################
+
+$(patsubst %,$(DIR_CHK)/%,$(objects)) :
+	@$(CHECK)
+
+$(patsubst %,$(DIR_DL)/%,$(objects)) :
+	@$(LOAD)
+
+$(subst %,%_MD5,$(objects)) :
+	@$(MD5)
+
+###############################################################################
+# Installation Details
+###############################################################################
+
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
+	@$(PREBUILD)
+	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
+	cd $(DIR_APP) && perl Makefile.PL
+	cd $(DIR_APP) && make $(MAKETUNING) $(EXTRA_MAKE)
+	cd $(DIR_APP) && make install
+	@rm -rf $(DIR_APP)
+	@$(POSTBUILD)
diff --git a/make.sh b/make.sh
index fd626a999..601f1675b 100755
--- a/make.sh
+++ b/make.sh
@@ -1434,6 +1434,7 @@ buildipfire() {
   lfsmake2 mpd
   lfsmake2 libmpdclient
   lfsmake2 mpc
+  lfsmake2 perl-Net-CIDR-Lite
   lfsmake2 perl-Net-SMTP-SSL
   lfsmake2 perl-MIME-Base64
   lfsmake2 perl-Authen-SASL
-- 
2.19.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 2/6] xtables-addons: Update to 3.2
  2019-01-10 12:00 [PATCH 1/6] perl-Net-CIDR-Lite: New package Stefan Schantl
@ 2019-01-10 12:00 ` Stefan Schantl
  2019-01-10 12:00 ` [PATCH 3/6] xtables-addons: Use shipped xt_geoip_build Stefan Schantl
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Stefan Schantl @ 2019-01-10 12:00 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

Reference #11959

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
---
 lfs/xtables-addons | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lfs/xtables-addons b/lfs/xtables-addons
index af2784c1a..a8aba7455 100644
--- a/lfs/xtables-addons
+++ b/lfs/xtables-addons
@@ -27,7 +27,7 @@ include Config
 VERSUFIX = ipfire$(KCFG)
 MODPATH = /lib/modules/$(KVER)-$(VERSUFIX)/extra/
 
-VER        = 2.13
+VER        = 3.2
 
 THISAPP    = xtables-addons-$(VER)
 DL_FILE    = $(THISAPP).tar.xz
@@ -48,7 +48,7 @@ objects = $(DL_FILE)
 
 $(DL_FILE) = $(DL_FROM)/$(DL_FILE)
 
-$(DL_FILE)_MD5 = be20b0b9c4b001b364431a836e361d33
+$(DL_FILE)_MD5 = 80ea89ba8d5a001a8d71c7f05b2f0141
 
 install : $(TARGET)
 
-- 
2.19.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 3/6] xtables-addons: Use shipped xt_geoip_build
  2019-01-10 12:00 [PATCH 1/6] perl-Net-CIDR-Lite: New package Stefan Schantl
  2019-01-10 12:00 ` [PATCH 2/6] xtables-addons: Update to 3.2 Stefan Schantl
@ 2019-01-10 12:00 ` Stefan Schantl
  2019-01-10 12:00 ` [PATCH 4/6] xt_geoip_update: Adjust script to download and use the GeoLite2 database Stefan Schantl
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Stefan Schantl @ 2019-01-10 12:00 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 2792 bytes --]

Use the shipped xt_geoip_build directly instead of holding a copy in our GIT.

Reference #11959

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
---
 lfs/xtables-addons         |  4 ++
 src/scripts/xt_geoip_build | 89 --------------------------------------
 2 files changed, 4 insertions(+), 89 deletions(-)
 delete mode 100644 src/scripts/xt_geoip_build

diff --git a/lfs/xtables-addons b/lfs/xtables-addons
index a8aba7455..da67aa761 100644
--- a/lfs/xtables-addons
+++ b/lfs/xtables-addons
@@ -94,6 +94,10 @@ ifeq "$(USPACE)" "1"
 
 	cd $(DIR_APP) && make $(MAKETUNING)
 	cd $(DIR_APP) && make install
+
+	# Install xt_geoip_build.
+	cd $(DIR_APP) && install -m 755 GeoIP/xt_geoip_build \
+		/usr/local/bin/
 else
 	cd $(DIR_APP) && ./configure \
 		--with-kbuild=/usr/src/linux-$(KVER)/
diff --git a/src/scripts/xt_geoip_build b/src/scripts/xt_geoip_build
deleted file mode 100644
index 202156f13..000000000
--- a/src/scripts/xt_geoip_build
+++ /dev/null
@@ -1,89 +0,0 @@
-#!/usr/bin/perl
-#
-#	Converter for MaxMind CSV database to binary, for xt_geoip
-#	Copyright © Jan Engelhardt, 2008-2011
-#
-use Getopt::Long;
-use IO::Handle;
-use Text::CSV_XS; # or trade for Text::CSV
-use strict;
-
-my $csv = Text::CSV_XS->new({
-	allow_whitespace => 1,
-	binary => 1,
-	eol => $/,
-}); # or Text::CSV
-my $target_dir = ".";
-
-&Getopt::Long::Configure(qw(bundling));
-&GetOptions(
-	"D=s" => \$target_dir,
-);
-
-if (!-d $target_dir) {
-	print STDERR "Target directory $target_dir does not exist.\n";
-	exit 1;
-}
-
-my $dir = "$target_dir/LE";
-if (!-e $dir && !mkdir($dir)) {
-	print STDERR "Could not mkdir $dir: $!\n";
-	exit 1;
-}
-
-&dump(&collect());
-
-sub collect
-{
-	my %country;
-
-	while (my $row = $csv->getline(*ARGV)) {
-		if (!defined($country{$row->[4]})) {
-			$country{$row->[4]} = {
-				name => $row->[5],
-				pool_v4 => [],
-				pool_v6 => [],
-			};
-		}
-		my $c = $country{$row->[4]};
-
-		push(@{$c->{pool_v4}}, [$row->[2], $row->[3]]);
-
-		if ($. % 4096 == 0) {
-			print STDERR "\r\e[2K$. entries";
-		}
-	}
-
-	print STDERR "\r\e[2K$. entries total\n";
-	return \%country;
-}
-
-sub dump
-{
-	my $country = shift @_;
-
-	foreach my $iso_code (sort keys %$country) {
-		&dump_one($iso_code, $country->{$iso_code});
-	}
-}
-
-sub dump_one
-{
-	my($iso_code, $country) = @_;
-	my($file, $fh_le, $fh_be);
-
-	printf "%5u IPv4 ranges for %s %s\n",
-		scalar(@{$country->{pool_v4}}),
-		$iso_code, $country->{name};
-
-	$file = "$target_dir/LE/".uc($iso_code).".iv4";
-	if (!open($fh_le, "> $file")) {
-		print STDERR "Error opening $file: $!\n";
-		exit 1;
-	}
-	foreach my $range (@{$country->{pool_v4}}) {
-		print $fh_le pack("VV", $range->[0], $range->[1]);
-		#print $fh_be pack("NN", $range->[0], $range->[1]);
-	}
-	close $fh_le;
-}
-- 
2.19.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 4/6] xt_geoip_update: Adjust script to download and use the GeoLite2 database
  2019-01-10 12:00 [PATCH 1/6] perl-Net-CIDR-Lite: New package Stefan Schantl
  2019-01-10 12:00 ` [PATCH 2/6] xtables-addons: Update to 3.2 Stefan Schantl
  2019-01-10 12:00 ` [PATCH 3/6] xtables-addons: Use shipped xt_geoip_build Stefan Schantl
@ 2019-01-10 12:00 ` Stefan Schantl
  2019-01-10 12:00 ` [PATCH 5/6] GeoIP: Drop legacy GeoIP perl module Stefan Schantl
  2019-01-10 12:00 ` [PATCH 6/6] geoip-functions.pl: Re-write code to lookup the iso country code of a given IP-address Stefan Schantl
  4 siblings, 0 replies; 7+ messages in thread
From: Stefan Schantl @ 2019-01-10 12:00 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 2967 bytes --]

Fixes #11961.

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
---
 src/scripts/xt_geoip_update | 63 ++++++++++++++++---------------------
 1 file changed, 27 insertions(+), 36 deletions(-)

diff --git a/src/scripts/xt_geoip_update b/src/scripts/xt_geoip_update
index 0aea4d03e..73484c7a0 100644
--- a/src/scripts/xt_geoip_update
+++ b/src/scripts/xt_geoip_update
@@ -24,13 +24,10 @@ TMP_FILE=$(mktemp -p $TMP_PATH)
 
 SCRIPT_PATH=/usr/local/bin
 DEST_PATH=/usr/share/xt_geoip
+DB_PATH=/var/lib/GeoIP
 
-DL_URL=https://geolite.maxmind.com/download/geoip/database
-DL_FILE=GeoIPCountryCSV.zip
-
-CSV_FILE=GeoIPCountryWhois.csv
-
-ARCH=LE
+DL_URL=http://geolite.maxmind.com/download/geoip/database/
+DL_FILE=GeoLite2-Country-CSV.zip
 
 eval $(/usr/local/bin/readhash /var/ipfire/proxy/settings)
 
@@ -57,42 +54,41 @@ function download() {
 	# Get the latest GeoIP database from server.
 	wget $DL_URL/$DL_FILE $PROXYSETTINGS -O $TMP_FILE
 
-	# Extract files.
+	# Extract files to database path.
 	unzip $TMP_FILE -d $TMP_PATH
 
 	return 0
 }
 
-function build() {
-	echo "Convert database..."
+function install() {
+	echo "Install CSV database..."
 
-	# Check if the csv file exists.
-	if [ ! -e $TMP_PATH/$CSV_FILE ]; then
-		echo "$TMP_PATH/$CSV_FILE not found. Exiting."
-		return 1
+	# Check if the database dir exists.
+	if [ ! -e "$DB_PATH" ]; then
+		mkdir -p $DB_PATH &>/dev/null
 	fi
 
-	# Run script to convert the CSV file into several xtables
-	# compatible binary files.
-	if ! $SCRIPT_PATH/xt_geoip_build $TMP_PATH/$CSV_FILE -D $TMP_PATH; then
-		echo "Could not convert ruleset. Aborting." >&2
+	# Check if the directory for binary databases exists.
+        if [ ! -e "$DEST_PATH" ]; then
+                mkdir -p $DEST_PATH &>/dev/null
+        fi
+
+	# Install CSV databases.
+	if ! cp -af $TMP_PATH/*/* $DB_PATH &>/dev/null; then
+		echo "Could not copy files. Aborting." >&2
 		return 1
 	fi
 
 	return 0
 }
 
-function install() {
-	echo "Install databases..."
-
-	# Check if our destination exist.
-	if [ ! -e "$DEST_PATH" ]; then
-		mkdir -p $DEST_PATH &>/dev/null
-	fi
+function build() {
+	echo "Convert database..."
 
-	# Install databases.
-	if ! cp -af $TMP_PATH/$ARCH $DEST_PATH &>/dev/null; then
-		echo "Could not copy files. Aborting." >&2
+	# Run script to convert the CSV file into several xtables
+	# compatible binary files.
+	if ! $SCRIPT_PATH/xt_geoip_build -S $DB_PATH -D $DEST_PATH; then
+		echo "Could not convert ruleset. Aborting." >&2
 		return 1
 	fi
 
@@ -113,23 +109,18 @@ function main() {
 	# Download ruleset.
 	download || exit $?
 
-	# Convert the ruleset.
-	if ! build; then
-		# Do cleanup.
-		cleanup || exit $?
-		exit 1
-	fi
-
-	# Install the converted ruleset.
 	if ! install; then
 		# Do cleanup.
 		cleanup || exit $?
 		exit 1
 	fi
 
-	# Finaly remove temporary files.
+	# Remove temporary files.
 	cleanup || exit $?
 
+	# Convert the ruleset.
+	build || exit $?
+
 	return 0
 }
 
-- 
2.19.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 5/6] GeoIP: Drop legacy GeoIP perl module.
  2019-01-10 12:00 [PATCH 1/6] perl-Net-CIDR-Lite: New package Stefan Schantl
                   ` (2 preceding siblings ...)
  2019-01-10 12:00 ` [PATCH 4/6] xt_geoip_update: Adjust script to download and use the GeoLite2 database Stefan Schantl
@ 2019-01-10 12:00 ` Stefan Schantl
  2019-01-10 12:00 ` [PATCH 6/6] geoip-functions.pl: Re-write code to lookup the iso country code of a given IP-address Stefan Schantl
  4 siblings, 0 replies; 7+ messages in thread
From: Stefan Schantl @ 2019-01-10 12:00 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 5484 bytes --]

The legacy GeoIP perl module cannot handle the new GeoLite2 databases
provided from maxmind and therefore needs to be dropped.

Reference #11960

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
---
 config/rootfiles/common/GeoIP | 12 -----
 lfs/GeoIP                     | 84 -----------------------------------
 make.sh                       |  1 -
 3 files changed, 97 deletions(-)
 delete mode 100644 config/rootfiles/common/GeoIP
 delete mode 100644 lfs/GeoIP

diff --git a/config/rootfiles/common/GeoIP b/config/rootfiles/common/GeoIP
deleted file mode 100644
index d76ba645e..000000000
--- a/config/rootfiles/common/GeoIP
+++ /dev/null
@@ -1,12 +0,0 @@
-#usr/bin/geoip-lookup
-#usr/lib/perl5/site_perl/5.12.3/Geo
-#usr/lib/perl5/site_perl/5.12.3/Geo/IP
-usr/lib/perl5/site_perl/5.12.3/Geo/IP/PurePerl.pm
-#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Geo
-#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Geo/IP
-#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Geo/IP/PurePerl
-#usr/lib/perl5/site_perl/5.12.3/MACHINE-linux-thread-multi/auto/Geo/IP/PurePerl/.packlist
-#usr/share/GeoIP
-usr/share/GeoIP/GeoIP.dat
-#usr/share/man/man1/geoip-lookup.1
-#usr/share/man/man3/Geo::IP::PurePerl.3
diff --git a/lfs/GeoIP b/lfs/GeoIP
deleted file mode 100644
index ce758d8a5..000000000
--- a/lfs/GeoIP
+++ /dev/null
@@ -1,84 +0,0 @@
-###############################################################################
-#                                                                             #
-# IPFire.org - A linux based firewall                                         #
-# Copyright (C) 2007-2018  IPFire Team  <info(a)ipfire.org>                     #
-#                                                                             #
-# This program is free software: you can redistribute it and/or modify        #
-# it under the terms of the GNU General Public License as published by        #
-# the Free Software Foundation, either version 3 of the License, or           #
-# (at your option) any later version.                                         #
-#                                                                             #
-# This program is distributed in the hope that it will be useful,             #
-# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
-# GNU General Public License for more details.                                #
-#                                                                             #
-# You should have received a copy of the GNU General Public License           #
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
-#                                                                             #
-###############################################################################
-
-###############################################################################
-# Definitions
-###############################################################################
-
-include Config
-
-VER        = 1.25
-DATVER     = 30062018
-
-THISAPP    = Geo-IP-PurePerl-$(VER)
-DL_FILE    = $(THISAPP).tar.gz
-DL_FROM    = $(URL_IPFIRE)
-DIR_APP    = $(DIR_SRC)/$(THISAPP)
-TARGET     = $(DIR_INFO)/$(THISAPP)
-
-###############################################################################
-# Top-level Rules
-###############################################################################
-
-objects = $(DL_FILE) GeoIP.dat-$(DATVER).gz
-
-$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
-GeoIP.dat-$(DATVER).gz = $(DL_FROM)/GeoIP.dat-$(DATVER).gz
-
-$(DL_FILE)_MD5 = a47a1b71f7cd7c46cca9efcc448e0726
-GeoIP.dat-$(DATVER).gz_MD5 = d538e57ad9268fdc7955c6cf9a37c4a9
-
-install : $(TARGET)
-
-check : $(patsubst %,$(DIR_CHK)/%,$(objects))
-
-download :$(patsubst %,$(DIR_DL)/%,$(objects))
-
-md5 : $(subst %,%_MD5,$(objects))
-
-###############################################################################
-# Downloading, checking, md5sum
-###############################################################################
-
-$(patsubst %,$(DIR_CHK)/%,$(objects)) :
-	@$(CHECK)
-
-$(patsubst %,$(DIR_DL)/%,$(objects)) :
-	@$(LOAD)
-
-$(subst %,%_MD5,$(objects)) :
-	@$(MD5)
-
-###############################################################################
-# Installation Details
-###############################################################################
-
-$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
-	@$(PREBUILD)
-	@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
-	cd $(DIR_APP) && patch -Np0 -i $(DIR_SRC)/src/patches/geoip_1_25_change_database_path.patch
-	cd $(DIR_APP) && perl Makefile.PL
-	cd $(DIR_APP) && make $(MAKETUNING) $(EXTRA_MAKE)
-	cd $(DIR_APP) && make install
-	cd $(DIR_APP) && mkdir -p /usr/share/GeoIP && \
-		zcat $(DIR_DL)/GeoIP.dat-$(DATVER).gz > /usr/share/GeoIP/GeoIP.dat
-	cd $(DIR_APP) && chmod 777 /srv/web/ipfire/html/images/flags
-	@rm -rf $(DIR_APP)
-	@$(POSTBUILD)
diff --git a/make.sh b/make.sh
index 601f1675b..1d9163dce 100755
--- a/make.sh
+++ b/make.sh
@@ -1305,7 +1305,6 @@ buildipfire() {
   lfsmake2 python-daemon
   lfsmake2 python-ipaddress
   lfsmake2 glib
-  lfsmake2 GeoIP
   lfsmake2 ntp
   lfsmake2 openssh
   lfsmake2 fontconfig
-- 
2.19.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 6/6] geoip-functions.pl: Re-write code to lookup the iso country code of a given IP-address.
  2019-01-10 12:00 [PATCH 1/6] perl-Net-CIDR-Lite: New package Stefan Schantl
                   ` (3 preceding siblings ...)
  2019-01-10 12:00 ` [PATCH 5/6] GeoIP: Drop legacy GeoIP perl module Stefan Schantl
@ 2019-01-10 12:00 ` Stefan Schantl
  2019-01-10 15:01   ` Michael Tremer
  4 siblings, 1 reply; 7+ messages in thread
From: Stefan Schantl @ 2019-01-10 12:00 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 3635 bytes --]

Drop the usage of the old legacy GeoIP perl module which was not able to handle the
new GeoLite2 databases.

Write some code to directly access the databases and extract the required data.

Usage of the GeoIP2 perl module would provide a lot of more functionality which is not
used/needed. Unfortunately ir requires at lot of additional perl modules which are
not available on IPFire and would only be build and shipped for this module. Buildig all
of them will slow down the entire build process, mess up the system and requires a lot
more space on disk.

Fixes #11962.

Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
---
 config/cfgroot/geoip-functions.pl | 75 ++++++++++++++++++++++++++++---
 1 file changed, 68 insertions(+), 7 deletions(-)

diff --git a/config/cfgroot/geoip-functions.pl b/config/cfgroot/geoip-functions.pl
index be50d5e14..e8ce8377f 100644
--- a/config/cfgroot/geoip-functions.pl
+++ b/config/cfgroot/geoip-functions.pl
@@ -23,21 +23,82 @@
 
 package GeoIP;
 
-use Geo::IP::PurePerl;
+require '/var/ipfire/network-functions.pl';
+
 use Locale::Codes::Country;
 
-my $database;
+# Path where all the GeoIP related databases are stored.
+my $geoip_database_dir = "/var/lib/GeoIP";
+
+# Database which contains all IPv4 networks.
+my $address_ipv4_database = "GeoLite2-Country-Blocks-IPv4.csv";
+
+# Database wich contains the locations data.
+my $location_database = "GeoLite2-Country-Locations-en.csv";
 
 sub lookup($) {
 	my $address = shift;
+	my $location_id;
+	my $country_code;
+
+	# Check if the given address is valid.
+	unless(&Network::check_ip_address($address)) {
+		return;
+	}
+
+	# Open the address database.
+	open(ADDRESS, "$geoip_database_dir/$address_ipv4_database") or die "Could not open $geoip_database_dir/$address_ipv4_database. $!\n";
+
+	# Loop through the file.
+	while(my $line = <ADDRESS>) {
+		# Remove newlines.
+		chomp($line);
+
+		# Split the line content.
+		my ($network, $geoname_id, $registered_country_geoname_id, $represented_country_geoname_id, $is_anonymous_proxy, $is_satellite_provider) = split(/\,/, $line);
+
+		# Check if the given address is part of the current processed network.
+		if (&Network::ip_address_in_network($address, $network)) {
+			# Store the geoname_id for this address.
+			$location_id = $geoname_id;
+
+			# Break loop.
+			last;
+		}
+	}
+
+	# Return nothing if no location_id could be found.
+	return unless($location_id);
+
+	# Close filehandle.
+	close(ADDRESS);
+
+	# Open the location database.
+	open(LOCATION, "$geoip_database_dir/$location_database") or die "Could not open $geoip_database_dir/$location_database. $!\n";
 
-	# Load the database into memory if not already done
-	if (!$database) {
-		$database = Geo::IP::PurePerl->new(GEOIP_MEMORY_CACHE);
+	# Loop through the file.
+	while(my $line = <LOCATION>) {
+		# Remove newlines.
+		chomp($line);
+
+		# Split the line content.
+		my ($geoname_id, $locale_code, $continent_code, $continent_name, $country_iso_code, $country_name, $is_in_european_union) = split(/\,/, $line);
+
+		# Check if the correct location_id has been found.
+		if ($geoname_id eq $location_id) {
+			# Store the county code.
+			$country_code = $country_iso_code;
+
+			# Break loop.
+			last;
+		}
 	}
 
-	# Return the name of the country
-	return $database->country_code_by_name($address);
+	# Close filehandle.
+	close(LOCATION);
+
+	# Return the obtained country code.
+	return $country_code;
 }
 
 # Function to get the flag icon for a specified country code.
-- 
2.19.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH 6/6] geoip-functions.pl: Re-write code to lookup the iso country code of a given IP-address.
  2019-01-10 12:00 ` [PATCH 6/6] geoip-functions.pl: Re-write code to lookup the iso country code of a given IP-address Stefan Schantl
@ 2019-01-10 15:01   ` Michael Tremer
  0 siblings, 0 replies; 7+ messages in thread
From: Michael Tremer @ 2019-01-10 15:01 UTC (permalink / raw)
  To: development

[-- Attachment #1: Type: text/plain, Size: 4717 bytes --]

Hey Stefan,

I merged all patches from above. The first one needed some smaller modifications but I did that for you to save some time here.

Read below for a problem in this patch:

> On 10 Jan 2019, at 12:00, Stefan Schantl <stefan.schantl(a)ipfire.org> wrote:
> 
> Drop the usage of the old legacy GeoIP perl module which was not able to handle the
> new GeoLite2 databases.
> 
> Write some code to directly access the databases and extract the required data.
> 
> Usage of the GeoIP2 perl module would provide a lot of more functionality which is not
> used/needed. Unfortunately ir requires at lot of additional perl modules which are
> not available on IPFire and would only be build and shipped for this module. Buildig all
> of them will slow down the entire build process, mess up the system and requires a lot
> more space on disk.
> 
> Fixes #11962.
> 
> Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org>
> ---
> config/cfgroot/geoip-functions.pl | 75 ++++++++++++++++++++++++++++---
> 1 file changed, 68 insertions(+), 7 deletions(-)
> 
> diff --git a/config/cfgroot/geoip-functions.pl b/config/cfgroot/geoip-functions.pl
> index be50d5e14..e8ce8377f 100644
> --- a/config/cfgroot/geoip-functions.pl
> +++ b/config/cfgroot/geoip-functions.pl
> @@ -23,21 +23,82 @@
> 
> package GeoIP;
> 
> -use Geo::IP::PurePerl;
> +require '/var/ipfire/network-functions.pl';
> +
> use Locale::Codes::Country;
> 
> -my $database;
> +# Path where all the GeoIP related databases are stored.
> +my $geoip_database_dir = "/var/lib/GeoIP";
> +
> +# Database which contains all IPv4 networks.
> +my $address_ipv4_database = "GeoLite2-Country-Blocks-IPv4.csv";
> +
> +# Database wich contains the locations data.
> +my $location_database = "GeoLite2-Country-Locations-en.csv";
> 
> sub lookup($) {
> 	my $address = shift;
> +	my $location_id;
> +	my $country_code;
> +
> +	# Check if the given address is valid.
> +	unless(&Network::check_ip_address($address)) {
> +		return;
> +	}
> +
> +	# Open the address database.
> +	open(ADDRESS, "$geoip_database_dir/$address_ipv4_database") or die "Could not open $geoip_database_dir/$address_ipv4_database. $!\n";
> +
> +	# Loop through the file.
> +	while(my $line = <ADDRESS>) {
> +		# Remove newlines.
> +		chomp($line);
> +
> +		# Split the line content.
> +		my ($network, $geoname_id, $registered_country_geoname_id, $represented_country_geoname_id, $is_anonymous_proxy, $is_satellite_provider) = split(/\,/, $line);
> +
> +		# Check if the given address is part of the current processed network.
> +		if (&Network::ip_address_in_network($address, $network)) {
> +			# Store the geoname_id for this address.
> +			$location_id = $geoname_id;
> +
> +			# Break loop.
> +			last;

You cannot simply break here. You are returning the first match which might not be the smallest subnet in the whole database.

What you need to do is to collect all matches in a hash (subnet and ID) and then search for the smallest subnet which will be the correct match.

I think that that can be implemented in a hand full of lines, so hopefully you can work on this before the end of today. That would be great.

I merged this patch now anyways so that we have a chance to test this all and for that a slightly inaccurate result might not be a big problem.

> +		}
> +	}
> +
> +	# Return nothing if no location_id could be found.
> +	return unless($location_id);
> +
> +	# Close filehandle.
> +	close(ADDRESS);
> +
> +	# Open the location database.
> +	open(LOCATION, "$geoip_database_dir/$location_database") or die "Could not open $geoip_database_dir/$location_database. $!\n";
> 
> -	# Load the database into memory if not already done
> -	if (!$database) {
> -		$database = Geo::IP::PurePerl->new(GEOIP_MEMORY_CACHE);
> +	# Loop through the file.
> +	while(my $line = <LOCATION>) {
> +		# Remove newlines.
> +		chomp($line);
> +
> +		# Split the line content.
> +		my ($geoname_id, $locale_code, $continent_code, $continent_name, $country_iso_code, $country_name, $is_in_european_union) = split(/\,/, $line);
> +
> +		# Check if the correct location_id has been found.
> +		if ($geoname_id eq $location_id) {
> +			# Store the county code.
> +			$country_code = $country_iso_code;
> +
> +			# Break loop.
> +			last;
> +		}
> 	}
> 
> -	# Return the name of the country
> -	return $database->country_code_by_name($address);
> +	# Close filehandle.
> +	close(LOCATION);
> +
> +	# Return the obtained country code.
> +	return $country_code;
> }
> 
> # Function to get the flag icon for a specified country code.
> -- 
> 2.19.1

Best,
-Michael

> 


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2019-01-10 15:01 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-10 12:00 [PATCH 1/6] perl-Net-CIDR-Lite: New package Stefan Schantl
2019-01-10 12:00 ` [PATCH 2/6] xtables-addons: Update to 3.2 Stefan Schantl
2019-01-10 12:00 ` [PATCH 3/6] xtables-addons: Use shipped xt_geoip_build Stefan Schantl
2019-01-10 12:00 ` [PATCH 4/6] xt_geoip_update: Adjust script to download and use the GeoLite2 database Stefan Schantl
2019-01-10 12:00 ` [PATCH 5/6] GeoIP: Drop legacy GeoIP perl module Stefan Schantl
2019-01-10 12:00 ` [PATCH 6/6] geoip-functions.pl: Re-write code to lookup the iso country code of a given IP-address Stefan Schantl
2019-01-10 15:01   ` Michael Tremer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox