[PATCHv2 0/7] Introduce perl LWP-based flexible downloader function

[PATCHv2 0/7] Introduce perl LWP-based flexible downloader function

[Stefan Schantl]

This reworked patchset contains a libwwwperl based flexible downloader
function which easily can be used to download files or grab and display any
web content.

The function now has been moved into an own perl library file
(http-client-functions.pl) in order to cleanup the general functions
library file. This should help to speed up the WUI by avoid to load to
much and big perl modules.

The FetchPublicIp function has been reworked to use the new downloader
and therefore moved to the http-client-functions file. As a result of
this, the GetDyndnsRedIP function, which is using that function also
has to be moved to the new library file and some existing CGI files needed
to be adjusted to work properly again.

The following changes from the feedback of the previously sent patchset
are also part of this patchset:

* Move the downloader into a seperate file.
* Die in case no URL has been passed to the downloader.
* Allow to specify the timeout time (default to 60sec if not set) insted of hardcode it.

-Stefan

Stefan Schantl (7):
  http-client-functions.pl: Introduce LWP-based flexible downloader
    function.
  http-client-functions.pl: Add FetchPublicIP function.
  general-functions.pl: Drop FetchPublicIp function.
  Move GetDyndnsRedIP from general-functions.pl to
    http-client-functions.pl
  ddns.cgi, wio.cgi: Use GetDyndnsRedIP from http-client-functions.pl
    file
  ids-functions.pl: Use new downloader function from
    http-client-functions.pl
  http-client-functions.pl: Allow to user define the timeout value.

 config/cfgroot/general-functions.pl     |  46 ----
 config/cfgroot/http-client-functions.pl | 345 ++++++++++++++++++++++++
 config/cfgroot/ids-functions.pl         | 206 ++------------
 config/rootfiles/common/configroot      |   1 +
 html/cgi-bin/ddns.cgi                   |   3 +-
 html/cgi-bin/wio.cgi                    |   3 +-
 lfs/configroot                          |   1 +
 7 files changed, 369 insertions(+), 236 deletions(-)
 create mode 100644 config/cfgroot/http-client-functions.pl

-- 
2.47.2

[PATCHv2 1/7] http-client-functions.pl: Introduce LWP-based flexible downloader function.

[Stefan Schantl]

This perl library contains a function which can be used to grab content
and/or store it into files.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 config/cfgroot/http-client-functions.pl | 290 ++++++++++++++++++++++++
 config/rootfiles/common/configroot      |   1 +
 lfs/configroot                          |   1 +
 3 files changed, 292 insertions(+)
 create mode 100644 config/cfgroot/http-client-functions.pl

diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl
new file mode 100644
index 000000000..26ead6908
--- /dev/null
+++ b/config/cfgroot/http-client-functions.pl
@@ -0,0 +1,290 @@
+#!/usr/bin/perl -w
+############################################################################
+#                                                                          #
+# This file is part of the IPFire Firewall.                                #
+#                                                                          #
+# IPFire is free software; you can redistribute it and/or modify           #
+# it under the terms of the GNU General Public License as published by     #
+# the Free Software Foundation; either version 2 of the License, or        #
+# (at your option) any later version.                                      #
+#                                                                          #
+# IPFire is distributed in the hope that it will be useful,                #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of           #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the            #
+# GNU General Public License for more details.                             #
+#                                                                          #
+# You should have received a copy of the GNU General Public License        #
+# along with IPFire; if not, write to the Free Software                    #
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA #
+#                                                                          #
+# Copyright (C) 2025 IPFire Team <info@ipfire.org>                         #
+#                                                                          #
+############################################################################
+
+package HTTPClient;
+
+require '/var/ipfire/general-functions.pl';
+
+use strict;
+
+# Load module to move files.
+use File::Copy;
+
+# Load module to get file stats.
+use File::stat;
+
+# Load module to deal with temporary files.
+use File::Temp;
+
+# Load module to deal with the date formats used by the HTTP protocol.
+use HTTP::Date;
+
+# Load the libwwwperl User Agent module.
+use LWP::UserAgent;
+
+# Function to grab a given URL content or to download and store it on disk.
+#
+# The function requires a configuration hash to be passed.
+#
+# The following options (hash keys) are supported:
+#
+# URL -> The URL to the content or file. REQUIRED!
+# FILE -> The filename as fullpath where the content/file should be stored on disk.
+# ETAGSFILE -> A filename again as fullpath where Etags should be stored and read.
+# ETAGPREFIX -> In case a custom etag name should be used, otherwise it defaults to the given URL.
+# MAXSIZE -> In bytes until the downloader will abort downloading. (example: 10_485_760 for 10MB)
+#
+# If a file is given an If-Modified-Since header will be generated from the last modified timestamp
+# of an already stored file. In case an Etag file is specified an If-None-Match header will be added to
+# the request - Both can be used at the same time.
+#
+# In case no FILE option has been passed to the function, the content of the requested URL will be returned.
+#
+# Return codes (if FILE is used):
+#
+# nothing - On success
+# no url - If no URL has been specified.
+# not_modified - In case the servers responds with "Not modified" (304)
+# dl_error - If the requested URL cannot be accessed.
+# incomplete download - In case the size of the local file does not match the remote content_lenght.
+#
+sub downloader (%) {
+	my (%args) = @_;
+
+	# Remap args hash and convert all keys into upper case format.
+	%args = map { uc $_ => $args{$_} } keys %args;
+
+	# The amount of download attempts before giving up and
+	# logging an error.
+	my $max_dl_attempts = 3;
+
+	# Temporary directory to download the files.
+	my $tmp_dl_directory = "/var/tmp";
+
+	# Assign hash values.
+	my $url = $args{"URL"} if (exists($args{"URL"}));
+	my $file = $args{"FILE"} if (exists($args{"FILE"}));
+	my $etags_file = $args{"ETAGSFILE"} if (exists($args{"ETAGSFILE"}));
+	my $etagprefix = $url;
+	$etagprefix = $args{"ETAGPREFIX"} if (exists($args{"ETAGPREFIX"}));
+	my $max_size = $args{"MAXSIZE"} if (exists($args{"MAXSIZE"}));
+
+	# Abort with error "no url", if no URL has been given.
+	die "downloader: No URL has been given." unless ($url);
+
+	my %etags = ();
+	my $tmpfile;
+
+	# Read-in proxysettings.
+	my %proxysettings=();
+	&General::readhash("${General::swroot}/proxy/settings", \%proxysettings);
+
+	# Create a user agent instance.
+	#
+	# Request SSL hostname verification and specify path
+	# to the CA file.
+	my $ua = LWP::UserAgent->new(
+		ssl_opts => {
+			SSL_ca_file     => '/etc/ssl/cert.pem',
+			verify_hostname => 1,
+		},
+	);
+
+	# Set timeout to 10 seconds.
+	$ua->timeout(10);
+
+	# Assign maximum download size if set.
+	$ua->max_size($max_size) if ($max_size);
+
+	# Generate UserAgent.
+	my $agent = &General::MakeUserAgent();
+
+	# Set the generated UserAgent.
+	$ua->agent($agent);
+
+	# Check if an upstream proxy is configured.
+	if ($proxysettings{'UPSTREAM_PROXY'}) {
+		# Start generating proxy url.
+		my $proxy_url = "http://";
+
+		# Check if the proxy requires authentication.
+		if (($proxysettings{'UPSTREAM_USER'}) && ($proxysettings{'UPSTREAM_PASSWORD'})) {
+			# Add proxy auth details.
+			$proxy_url .= "$proxysettings{'UPSTREAM_USER'}\:$proxysettings{'UPSTREAM_PASSWORD'}\@";
+		}
+
+		# Add proxy server address and port.
+		$proxy_url .= $proxysettings{'UPSTREAM_PROXY'};
+
+		# Append proxy settings.
+		$ua->proxy(['http', 'https'], $proxy_url);
+	}
+
+	# Create a HTTP request element and pass the given URL to it.
+	my $request = HTTP::Request->new(GET => $url);
+
+	# Check if a file to store the output has been provided.
+	if ($file) {
+		# Check if the given file already exits, because it has been downloaded in the past.
+		#
+		# In this case we are requesting the server if the remote file has been changed or not.
+		# This will be done by sending the modification time in a special HTTP header.
+		if (-f $file) {
+			# Call stat on the file.
+			my $stat = stat($file);
+
+			# Omit the mtime of the existing file.
+			my $mtime = $stat->mtime;
+
+			# Convert the timestamp into right format.
+			my $http_date = time2str($mtime);
+
+			# Add the If-Modified-Since header to the request to ask the server if the
+			# file has been modified.
+			$request->header( 'If-Modified-Since' => "$http_date" );
+		}
+
+		# Generate a temporary file name, located in the tempoary download directory and with a suffix of ".tmp".
+		# The downloaded file will be stored there until some sanity checks are performed.
+		my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "$tmp_dl_directory/", UNLINK => 0 );
+		$tmpfile = $tmp->filename();
+	}
+
+	# Check if an file for etags has been given.
+	if ($etags_file) {
+		# Read-in Etags file for known Etags if the file is present.
+		&readhash("$etags_file", \%etags) if (-f $etags_file);
+
+		# Check if an Etag for the requested file is stored.
+		if ($etags{$etagprefix}) {
+			# Grab the stored tag.
+			my $etag = $etags{$etagprefix};
+
+			# Add an "If-None-Match header to the request to ask the server if the
+			# file has been modified.
+			$request->header( 'If-None-Match' => $etag );
+		}
+	}
+
+	my $dl_attempt = 1;
+	my $response;
+
+	# Download and retry on failure.
+	while ($dl_attempt <= $max_dl_attempts) {
+		# Perform the request and save the output into the tmpfile if requested.
+		$response = $ua->request($request, $tmpfile);
+
+		# Check if the download was successfull.
+		if($response->is_success) {
+			# Break loop.
+			last;
+
+		# Check if the server responds with 304 (Not Modified).
+		} elsif ($response->code == 304) {
+			# Remove temporary file, if one exists.
+			unlink("$tmpfile") if (-e "$tmpfile");
+
+			# Return "not modified".
+			return "not modified";
+
+		# Check if we ran out of download re-tries.
+		} elsif ($dl_attempt eq $max_dl_attempts) {
+			# Obtain error.
+			my $error = $response->content;
+
+			# Remove temporary file, if one exists.
+			unlink("$tmpfile") if (-e "$tmpfile");
+
+			# Return the error message from response..
+			return "$error";
+		}
+
+		# Remove temporary file, if one exists.
+		unlink("$tmpfile") if (-e "$tmpfile");
+
+		# Increase download attempt counter.
+		$dl_attempt++;
+	}
+
+	# Obtain the connection headers.
+	my $headers = $response->headers;
+
+	# Check if an Etag file has been provided.
+	if ($etags_file) {
+		# Grab the Etag from the response if the server provides one.
+		if ($response->header('Etag')) {
+			# Add the provided Etag to the hash of tags.
+			$etags{$etagprefix} = $response->header('Etag');
+
+			# Write the etags file.
+			&General::writehash($etags_file, \%etags);
+		}
+	}
+
+	# Check if the response should be stored on disk.
+	if ($file) {
+		# Get the remote size of the content.
+		my $remote_size = $response->header('Content-Length');
+
+		# Perform a stat on the temporary file.
+		my $stat = stat($tmpfile);
+
+		# Grab the size of the stored temporary file.
+		my $local_size = $stat->size;
+
+		# Check if both sizes are equal.
+		if(($remote_size) && ($remote_size ne $local_size)) {
+			# Delete the temporary file.
+			unlink("$tmpfile");
+
+			# Abort and return "incomplete download" as error.
+			return "incomplete download";
+		}
+
+		# Move the temporaray file to the desired file by overwriting a may
+		# existing one.
+		move("$tmpfile", "$file");
+
+		# Omit the timestamp from response header, when the file has been modified the
+		# last time.
+		my $last_modified = $headers->last_modified;
+
+		# Check if we got a last-modified value from the server.
+		if ($last_modified) {
+			# Assign the last-modified timestamp as mtime to the
+			# stored file.
+			utime(time(), "$last_modified", "$file");
+		}
+
+		# Delete temporary file.
+		unlink("$tmpfile");
+
+		# If we got here, everything worked fine. Return nothing.
+		return;
+	} else {
+		# Decode the response content and return it.
+		return $response->decoded_content;
+	}
+}
+
+1;
diff --git a/config/rootfiles/common/configroot b/config/rootfiles/common/configroot
index 9839eee45..51472e7c5 100644
--- a/config/rootfiles/common/configroot
+++ b/config/rootfiles/common/configroot
@@ -79,6 +79,7 @@ var/ipfire/fwlogs
 var/ipfire/general-functions.pl
 var/ipfire/graphs.pl
 var/ipfire/header.pl
+var/ipfire/http-client-functions.pl
 var/ipfire/location-functions.pl
 var/ipfire/ids-functions.pl
 var/ipfire/ipblocklist-functions.pl
diff --git a/lfs/configroot b/lfs/configroot
index 9f6c1ff8c..1f752ddb6 100644
--- a/lfs/configroot
+++ b/lfs/configroot
@@ -76,6 +76,7 @@ $(TARGET) :
 
 	# Copy initial configfiles
 	cp $(DIR_SRC)/config/cfgroot/header.pl			$(CONFIG_ROOT)/
+	cp $(DIR_SRC)/config/cfgroot/http-client-functions.pl   $(CONFIG_ROOT)/
 	cp $(DIR_SRC)/config/cfgroot/general-functions.pl	$(CONFIG_ROOT)/
 	cp $(DIR_SRC)/config/cfgroot/network-functions.pl	$(CONFIG_ROOT)/
 	cp $(DIR_SRC)/config/cfgroot/location-functions.pl	$(CONFIG_ROOT)/
-- 
2.47.2

[PATCHv2 2/7] http-client-functions.pl: Add FetchPublicIP function.

[Stefan Schantl]

This functions uses the newly introduced downloader to fetch
the pulic IPv4 address on red and will replace the current used one
from the general-functions.pl library.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 config/cfgroot/http-client-functions.pl | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl
index 26ead6908..bfb9fdd20 100644
--- a/config/cfgroot/http-client-functions.pl
+++ b/config/cfgroot/http-client-functions.pl
@@ -287,4 +287,24 @@ sub downloader (%) {
 	}
 }
 
+#
+# Tiny function to grab the public red IPv4 address using LWL.
+#
+sub FetchPublicIp {
+	# URL to grab the public IP.
+	my $url = "https://checkip4.dns.lightningwirelabs.com";
+
+	# Call downloader to fetch the public IP.
+	my $response = &downloader("URL" => $url);
+
+	# Omit the address from the resonse message.
+	if ($response =~ /Your IP address is: (\d+.\d+.\d+.\d+)/) {
+		# Return the address.
+		return $1;
+	}
+
+	# Unable to grab the address - Return nothing.
+	return;
+}
+
 1;
-- 
2.47.2

[PATCHv2 3/7] general-functions.pl: Drop FetchPublicIp function.

[Stefan Schantl]

This function has been reworked and moved into the
http-client-functions library.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 config/cfgroot/general-functions.pl | 21 ---------------------
 1 file changed, 21 deletions(-)

diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-functions.pl
index 8ba6e3f79..861f95dec 100644
--- a/config/cfgroot/general-functions.pl
+++ b/config/cfgroot/general-functions.pl
@@ -17,7 +17,6 @@ package General;
 use strict;
 use Socket;
 use IO::Socket;
-use Net::SSLeay;
 use Net::IPv4Addr qw(:all);
 
 $General::version = 'VERSION';
@@ -961,26 +960,6 @@ sub findhasharraykey {
     }
 }
 
-sub FetchPublicIp {
-    my %proxysettings;
-    &General::readhash("${General::swroot}/proxy/settings", \%proxysettings);
-    if ($_=$proxysettings{'UPSTREAM_PROXY'}) {
-        my ($peer, $peerport) = (/^(?:[a-zA-Z ]+\:\/\/)?(?:[A-Za-z0-9\_\.\-]*?(?:\:[A-Za-z0-9\_\.\-]*?)?\@)?([a-zA-Z0-9\.\_\-]*?)(?:\:([0-9]{1,5}))?(?:\/.*?)?$/);
-        Net::SSLeay::set_proxy($peer,$peerport,$proxysettings{'UPSTREAM_USER'},$proxysettings{'UPSTREAM_PASSWORD'} );
-    }
-    my $user_agent = &MakeUserAgent();
-    my ($out, $response) = Net::SSLeay::get_http(  'checkip4.dns.lightningwirelabs.com',
-                				    80,
-        					    "/",
-						    Net::SSLeay::make_headers('User-Agent' => $user_agent )
-						);
-    if ($response =~ m%HTTP/1\.. 200 OK%) {
-	$out =~ /Your IP address is: (\d+.\d+.\d+.\d+)/;
-	return $1;
-    }
-    return '';
-}
-
 #
 # Check if hostname.domain provided have IP provided
 # use gethostbyname to verify that
-- 
2.47.2

[PATCHv2 4/7] Move GetDyndnsRedIP from general-functions.pl to http-client-functions.pl

[Stefan Schantl]

This function depends on the previously FetchPublicIp function
and so also has to be moved.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 config/cfgroot/general-functions.pl     | 25 ---------------------
 config/cfgroot/http-client-functions.pl | 30 +++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 25 deletions(-)

diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-functions.pl
index 861f95dec..bbd0f9839 100644
--- a/config/cfgroot/general-functions.pl
+++ b/config/cfgroot/general-functions.pl
@@ -998,31 +998,6 @@ sub DyndnsServiceSync ($;$;$) {
     }
     return 0;
 }
-#
-# This sub returns the red IP used to compare in DyndnsServiceSync
-#
-sub GetDyndnsRedIP {
-    my %settings;
-    &General::readhash("${General::swroot}/ddns/settings", \%settings);
-
-    open(IP, "${General::swroot}/red/local-ipaddress") or return 'unavailable';
-    my $ip = <IP>;
-    close(IP);
-    chomp $ip;
-
-    # 100.64.0.0/10 is reserved for dual-stack lite (http://tools.ietf.org/html/rfc6598).
-    if (&General::IpInSubnet ($ip,'10.0.0.0','255.0.0.0') ||
-        &General::IpInSubnet ($ip,'172.16.0.0.','255.240.0.0') ||
-        &General::IpInSubnet ($ip,'192.168.0.0','255.255.0.0') ||
-        &General::IpInSubnet ($ip,'100.64.0.0', '255.192.0.0'))
-    {
-	if ($settings{'BEHINDROUTER'} eq 'FETCH_IP') {
-    	    my $RealIP = &General::FetchPublicIp;
-	    $ip = (&General::validip ($RealIP) ?  $RealIP : 'unavailable');
-	}
-    }
-    return $ip;
-}
 
 # Translate ICMP code to text
 # ref: http://www.iana.org/assignments/icmp-parameters
diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl
index bfb9fdd20..f1f7de309 100644
--- a/config/cfgroot/http-client-functions.pl
+++ b/config/cfgroot/http-client-functions.pl
@@ -307,4 +307,34 @@ sub FetchPublicIp {
 	return;
 }
 
+#
+# This sub returns the red IP used to compare in DyndnsServiceSync
+#
+sub GetDyndnsRedIP {
+	my %settings;
+
+	# Read-in ddns settings.
+	&General::readhash("${General::swroot}/ddns/settings", \%settings);
+
+	# Try to grab the address from the local-ipaddress file.
+	my $ip = &General::grab_address_from_file("${General::swroot}/red/local-ipaddress") or return 'unavailable';
+
+	# 100.64.0.0/10 is reserved for dual-stack lite (http://tools.ietf.org/html/rfc6598).
+	if (&General::IpInSubnet ($ip,'10.0.0.0','255.0.0.0') ||
+		&General::IpInSubnet ($ip,'172.16.0.0.','255.240.0.0') ||
+		&General::IpInSubnet ($ip,'192.168.0.0','255.255.0.0') ||
+		&General::IpInSubnet ($ip,'100.64.0.0', '255.192.0.0'))
+	{
+		if ($settings{'BEHINDROUTER'} eq 'FETCH_IP') {
+			# Call function to omit the real address.
+			my $RealIP = &FetchPublicIp;
+
+			# Check if the grabbed address is valid.
+			$ip = (&General::validip ($RealIP) ?  $RealIP : 'unavailable');
+		}
+	}
+
+	return $ip;
+}
+
 1;
-- 
2.47.2

[PATCHv2 5/7] ddns.cgi, wio.cgi: Use GetDyndnsRedIP from http-client-functions.pl file

[Stefan Schantl]

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 html/cgi-bin/ddns.cgi | 3 ++-
 html/cgi-bin/wio.cgi  | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/html/cgi-bin/ddns.cgi b/html/cgi-bin/ddns.cgi
index 0e3ccbe45..34475b75c 100644
--- a/html/cgi-bin/ddns.cgi
+++ b/html/cgi-bin/ddns.cgi
@@ -29,6 +29,7 @@ use experimental 'smartmatch';
 require '/var/ipfire/general-functions.pl';
 require "${General::swroot}/lang.pl";
 require "${General::swroot}/header.pl";
+require "${General::swroot}/http-client-functions.pl";
 
 #workaround to suppress a warning when a variable is used only once
 my @dummy = ( ${Header::table2colour}, ${Header::colouryellow} );
@@ -559,7 +560,7 @@ open(FILE, $datafile) or die "Unable to open $datafile.";
 close(FILE);
 
 # Get IP address of the red interface.
-my $ip = &General::GetDyndnsRedIP();
+my $ip = &HTTPClient::GetDyndnsRedIP();
 my $id = 0;
 my $toggle_enabled;
 
diff --git a/html/cgi-bin/wio.cgi b/html/cgi-bin/wio.cgi
index f31f5d565..30a51104c 100644
--- a/html/cgi-bin/wio.cgi
+++ b/html/cgi-bin/wio.cgi
@@ -50,6 +50,7 @@ require '/var/ipfire/general-functions.pl';
 require '/var/ipfire/network-functions.pl';
 require '/var/ipfire/lang.pl';
 require '/var/ipfire/header.pl';
+require '/var/ipfire/http-client-functions.pl';
 require '/usr/lib/wio/wio-lib.pl';
 require '/usr/lib/wio/wio-graphs.pl';
 
@@ -1163,7 +1164,7 @@ close (FILE);
 		@temp = split (/\,/, $_);
 
 		if ( $temp[7] eq "on" ) {
-			$bgcolor = ( &General::DyndnsServiceSync (&General::GetDyndnsRedIP,$temp[1],$temp[2]) ? "$Header::colourgreen" : "$Header::colourred" );
+			$bgcolor = ( &General::DyndnsServiceSync (&HTTPClient::GetDyndnsRedIP,$temp[1],$temp[2]) ? "$Header::colourgreen" : "$Header::colourred" );
 		}
 		else {
 			$bgcolor = "blue";
-- 
2.47.2

[PATCHv2 6/7] ids-functions.pl: Use new downloader function from http-client-functions.pl

[Stefan Schantl]

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 config/cfgroot/ids-functions.pl | 206 +++-----------------------------
 1 file changed, 18 insertions(+), 188 deletions(-)

diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl
index 399f5cbf8..1a72e4c3e 100644
--- a/config/cfgroot/ids-functions.pl
+++ b/config/cfgroot/ids-functions.pl
@@ -27,6 +27,7 @@ package IDS;
 
 require '/var/ipfire/general-functions.pl';
 require "${General::swroot}/network-functions.pl";
+require "${General::swroot}/http-client-functions.pl";
 require "${General::swroot}/suricata/ruleset-sources";
 
 # Load perl module to deal with Archives.
@@ -44,15 +45,6 @@ use File::Path qw(rmtree);
 # Load module to get file stats.
 use File::stat;
 
-# Load module to deal with temporary files.
-use File::Temp;
-
-# Load module to deal with the date formats used by the HTTP protocol.
-use HTTP::Date;
-
-# Load the libwwwperl User Agent module.
-use LWP::UserAgent;
-
 # Load function from posix module to format time strings.
 use POSIX qw (strftime);
 
@@ -130,9 +122,6 @@ my $suricatactrl = "/usr/local/bin/suricatactrl";
 # Prefix for each downloaded ruleset.
 my $dl_rulesfile_prefix = "idsrules";
 
-# Temporary directory to download the rules files.
-my $tmp_dl_directory = "/var/tmp";
-
 # Temporary directory where the rulesets will be extracted.
 my $tmp_directory = "/tmp/ids_tmp";
 
@@ -299,61 +288,13 @@ sub checkdiskspace () {
 #
 ## This function is responsible for downloading the ruleset for a given provider.
 ##
-## * At first it initialize the downloader and sets an upstream proxy if configured.
-## * The next step will be to generate the final download url, by obtaining the URL for the desired
-##   ruleset and add the settings for the upstream proxy.
-## * Finally the function will grab the rule file or tarball from the server.
-##   It tries to reduce the amount of download by using the "If-Modified-Since" HTTP header.
-#
-## Return codes:
-##
-## * "no url" - If no download URL could be gathered for the provider.
-## * "not modified" - In case the already stored rules file is up to date.
-## * "incomplete download" - When the remote file size differs from the downloaded file size.
-## * "$error" - The error message generated from the LWP::User Agent module.
+## It uses the LWP-based downloader function from the general-functions.pl to
+## download the ruleset for a requested provider.
 #
 sub downloadruleset ($) {
 	my ($provider) = @_;
 
-	# The amount of download attempts before giving up and
-	# logging an error.
-	my $max_dl_attempts = 3;
-
-	# Read proxysettings.
-	my %proxysettings=();
-	&General::readhash("${General::swroot}/proxy/settings", \%proxysettings);
-
-	# Init the download module.
-	#
-	# Request SSL hostname verification and specify path
-	# to the CA file.
-	my $downloader = LWP::UserAgent->new(
-		ssl_opts => {
-			SSL_ca_file     => '/etc/ssl/cert.pem',
-			verify_hostname => 1,
-		}
-	);
-
-	# Set timeout to 10 seconds.
-	$downloader->timeout(10);
-
-	# Check if an upstream proxy is configured.
-	if ($proxysettings{'UPSTREAM_PROXY'}) {
-		my $proxy_url;
-
-		$proxy_url = "http://";
-
-		# Check if the proxy requires authentication.
-		if (($proxysettings{'UPSTREAM_USER'}) && ($proxysettings{'UPSTREAM_PASSWORD'})) {
-			$proxy_url .= "$proxysettings{'UPSTREAM_USER'}\:$proxysettings{'UPSTREAM_PASSWORD'}\@";
-		}
-
-		# Add proxy server address and port.
-		$proxy_url .= $proxysettings{'UPSTREAM_PROXY'};
-
-		# Setup proxy settings.
-		$downloader->proxy(['http', 'https'], $proxy_url);
-	}
+	my %settings = ();
 
 	# Grab the download url for the provider.
 	my $url = $IDS::Ruleset::Providers{$provider}{'dl_url'};
@@ -371,141 +312,30 @@ sub downloadruleset ($) {
 	# Abort and return "no url", if no url could be determined for the provider.
 	return "no url" unless ($url);
 
-	# Pass the requested URL to the downloader.
-	my $request = HTTP::Request->new(GET => $url);
-
-	# Generate temporary file name, located in the tempoary download directory and with a suffix of ".tmp".
-	# The downloaded file will be stored there until some sanity checks are performed.
-	my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "$tmp_dl_directory/", UNLINK => 0 );
-	my $tmpfile = $tmp->filename();
+	# Pass the requested URL to the settings hash.
+	$settings{'URL'} = $url;
 
 	# Call function to get the final path and filename for the downloaded file.
 	my $dl_rulesfile = &_get_dl_rulesfile($provider);
 
-	# Check if the rulesfile already exits, because it has been downloaded in the past.
-	#
-	# In this case we are requesting the server if the remote file has been changed or not.
-	# This will be done by sending the modification time in a special HTTP header.
-	if (-f $dl_rulesfile) {
-		# Call stat on the file.
-		my $stat = stat($dl_rulesfile);
-
-		# Omit the mtime of the existing file.
-		my $mtime = $stat->mtime;
-
-		# Convert the timestamp into right format.
-		my $http_date = time2str($mtime);
-
-		# Add the If-Modified-Since header to the request to ask the server if the
-		# file has been modified.
-		$request->header( 'If-Modified-Since' => "$http_date" );
-	}
-
-	# Read-in Etags file for known Etags if the file is present.
-	my %etags = ();
-	&General::readhash("$etags_file", \%etags) if (-f $etags_file);
-
-	# Check if an Etag for the current provider is stored.
-	if ($etags{$provider}) {
-		# Grab the stored tag.
-		my $etag = $etags{$provider};
-
-		# Add an "If-None-Match header to the request to ask the server if the
-		# file has been modified.
-		$request->header( 'If-None-Match' => $etag );
-	}
-
-	my $dl_attempt = 1;
-	my $response;
-
-	# Download and retry on failure.
-	while ($dl_attempt <= $max_dl_attempts) {
-		# Perform the request and save the output into the tmpfile.
-		$response = $downloader->request($request, $tmpfile);
-
-		# Check if the download was successfull.
-		if($response->is_success) {
-			# Break loop.
-			last;
-
-		# Check if the server responds with 304 (Not Modified).
-		} elsif ($response->code == 304) {
-			# Remove temporary file, if one exists.
-			unlink("$tmpfile") if (-e "$tmpfile");
-
-			# Return "not modified".
-			return "not modified";
-
-		# Check if we ran out of download re-tries.
-		} elsif ($dl_attempt eq $max_dl_attempts) {
-			# Obtain error.
-			my $error = $response->content;
-
-			# Remove temporary file, if one exists.
-			unlink("$tmpfile") if (-e "$tmpfile");
-
-			# Return the error message from response..
-			return "$error";
-		}
-
-		# Remove temporary file, if one exists.
-		unlink("$tmpfile") if (-e "$tmpfile");
+	# Add the file information to the settings hash.
+	$settings{'FILE'} = $dl_rulesfile;
 
-		# Increase download attempt counter.
-		$dl_attempt++;
-	}
-
-	# Obtain the connection headers.
-	my $headers = $response->headers;
-
-	# Get the timestamp from header, when the file has been modified the
-	# last time.
-	my $last_modified = $headers->last_modified;
-
-	# Get the remote size of the downloaded file.
-	my $remote_filesize = $headers->content_length;
+	# Add Etag details to the settings hash.
+	$settings{'ETAGSFILE'} = $etags_file;
+	$settings{'ETAGPREFIX'} = $provider;
 
-	# Grab the Etag from response it the server provides one.
-	if ($response->header('Etag')) {
-		# Add the Etag to the etags hash.
-		$etags{$provider} = $response->header('Etag');
+	# Call the downloader and pass the settings hash.
+	my $response = &HTTPClient::downloader(%settings);
 
-		# Write the etags file.
-		&General::writehash($etags_file, \%etags);
+	# Return the response message if the downloader provided one.
+	if ($response) {
+		return $response;
 	}
 
-	# Perform stat on the tmpfile.
-	my $stat = stat($tmpfile);
-
-	# Grab the local filesize of the downloaded tarball.
-	my $local_filesize = $stat->size;
-
-	# Check if both file sizes match.
-	if (($remote_filesize) && ($remote_filesize ne $local_filesize)) {
-		# Delete temporary file.
-		unlink("$tmpfile");
-
-		# Return "1" - false.
-		return "incomplete download";
-	}
-
-	# Overwrite the may existing rulefile or tarball with the downloaded one.
-	move("$tmpfile", "$dl_rulesfile");
-
-	# Check if we got a last-modified value from the server.
-	if ($last_modified) {
-		# Assign the last-modified timestamp as mtime to the
-		# rules file.
-		utime(time(), "$last_modified", "$dl_rulesfile");
-	}
-
-	# Delete temporary file.
-	unlink("$tmpfile");
-
-	# Set correct ownership for the tarball.
-	set_ownership("$dl_rulesfile");
+	# Set correct ownership for the downloaded rules file.
+	&set_ownership("$dl_rulesfile");
 
-	# If we got here, everything worked fine. Return nothing.
 	return;
 }
 
-- 
2.47.2

[PATCHv2 7/7] http-client-functions.pl: Allow to user define the timeout value.

[Stefan Schantl]

This allows to specify the the timeout value.
Defaults to to 60 seconds if not set.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
---
 config/cfgroot/http-client-functions.pl | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl
index f1f7de309..c9484c575 100644
--- a/config/cfgroot/http-client-functions.pl
+++ b/config/cfgroot/http-client-functions.pl
@@ -89,6 +89,10 @@ sub downloader (%) {
 	$etagprefix = $args{"ETAGPREFIX"} if (exists($args{"ETAGPREFIX"}));
 	my $max_size = $args{"MAXSIZE"} if (exists($args{"MAXSIZE"}));
 
+	# Timeout defaults to 60 Seconds if not set.
+	my $timeout = 60;
+	$timeout = $args{"TIMEOUT"} if (exists($args{"TIMEOUT"}));
+
 	# Abort with error "no url", if no URL has been given.
 	die "downloader: No URL has been given." unless ($url);
 
@@ -110,8 +114,9 @@ sub downloader (%) {
 		},
 	);
 
-	# Set timeout to 10 seconds.
-	$ua->timeout(10);
+	# Set the timeout to the configured value.
+	# Defaults to 60 seconds if not set.
+	$ua->timeout($timeout);
 
 	# Assign maximum download size if set.
 	$ua->max_size($max_size) if ($max_size);
-- 
2.47.2

Re: [PATCHv2 0/7] Introduce perl LWP-based flexible downloader function

[Michael Tremer]

Hello Stefan,

Thank you for working on this. I like this a lot.

This makes it all very independent from the other code and we can pull it all in wherever it is needed.

I didn’t test this, but the changes look good to me!

-Michael

> On 18 Apr 2025, at 11:54, Stefan Schantl <stefan.schantl@ipfire.org> wrote:
> 
> This reworked patchset contains a libwwwperl based flexible downloader
> function which easily can be used to download files or grab and display any
> web content.
> 
> The function now has been moved into an own perl library file
> (http-client-functions.pl) in order to cleanup the general functions
> library file. This should help to speed up the WUI by avoid to load to
> much and big perl modules.
> 
> The FetchPublicIp function has been reworked to use the new downloader
> and therefore moved to the http-client-functions file. As a result of
> this, the GetDyndnsRedIP function, which is using that function also
> has to be moved to the new library file and some existing CGI files needed
> to be adjusted to work properly again.
> 
> The following changes from the feedback of the previously sent patchset
> are also part of this patchset:
> 
> * Move the downloader into a seperate file.
> * Die in case no URL has been passed to the downloader.
> * Allow to specify the timeout time (default to 60sec if not set) insted of hardcode it.
> 
> -Stefan
> 
> Stefan Schantl (7):
>  http-client-functions.pl: Introduce LWP-based flexible downloader
>    function.
>  http-client-functions.pl: Add FetchPublicIP function.
>  general-functions.pl: Drop FetchPublicIp function.
>  Move GetDyndnsRedIP from general-functions.pl to
>    http-client-functions.pl
>  ddns.cgi, wio.cgi: Use GetDyndnsRedIP from http-client-functions.pl
>    file
>  ids-functions.pl: Use new downloader function from
>    http-client-functions.pl
>  http-client-functions.pl: Allow to user define the timeout value.
> 
> config/cfgroot/general-functions.pl     |  46 ----
> config/cfgroot/http-client-functions.pl | 345 ++++++++++++++++++++++++
> config/cfgroot/ids-functions.pl         | 206 ++------------
> config/rootfiles/common/configroot      |   1 +
> html/cgi-bin/ddns.cgi                   |   3 +-
> html/cgi-bin/wio.cgi                    |   3 +-
> lfs/configroot                          |   1 +
> 7 files changed, 369 insertions(+), 236 deletions(-)
> create mode 100644 config/cfgroot/http-client-functions.pl
> 
> -- 
> 2.47.2
> 
>