[PATCH] use protocol defined in server-list.db for mirror communication

[PATCH] use protocol defined in server-list.db for mirror communication

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

For each mirror server, a protocol can be specified in the
server-list.db database. However, it was not used for the
actual URL query to a mirror before.

This might be useful for deploy HTTPS pinning for Pakfire.
If a mirror is known to support HTTPS, all queries to it
will be made with this protocol.

This saves some overhead if HTTPS is enforced on a mirror
via 301 redirects. To enable this, the server-list.db
needs to be adjusted.

Partially fixes #11661.

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu>
Cc: Michael Tremer <michael.tremer(a)ipfire.org>
---
 src/pakfire/lib/functions.pl | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl
index c97d4254d..94f9f1826 100644
--- a/src/pakfire/lib/functions.pl
+++ b/src/pakfire/lib/functions.pl
@@ -171,8 +171,11 @@ sub fetchfile {
 			}
 		}
 
+		# Use specified protocol for mirror communication (allows HTTPS pinning)
+		my $urlproto = lc $proto;
+
 		$final_data = undef;
-	 	my $url = "http://$host/$file";
+	 	my $url = "$urlproto://$host/$file";
 		my $response;
 		
 		unless ($bfile =~ /^counter.py\?.*/) {
-- 
2.13.6

Re: [PATCH] use protocol defined in server-list.db for mirror communication

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

Hello,

would it not be a good idea to check if $proto is either HTTP or HTTPS?

-Michael

On Sat, 2018-03-24 at 16:22 +0100, Peter Müller wrote:
> For each mirror server, a protocol can be specified in the
> server-list.db database. However, it was not used for the
> actual URL query to a mirror before.
> 
> This might be useful for deploy HTTPS pinning for Pakfire.
> If a mirror is known to support HTTPS, all queries to it
> will be made with this protocol.
> 
> This saves some overhead if HTTPS is enforced on a mirror
> via 301 redirects. To enable this, the server-list.db
> needs to be adjusted.
> 
> Partially fixes #11661.
> 
> Signed-off-by: Peter Müller <peter.mueller(a)link38.eu>
> Cc: Michael Tremer <michael.tremer(a)ipfire.org>
> ---
>  src/pakfire/lib/functions.pl | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl
> index c97d4254d..94f9f1826 100644
> --- a/src/pakfire/lib/functions.pl
> +++ b/src/pakfire/lib/functions.pl
> @@ -171,8 +171,11 @@ sub fetchfile {
>  			}
>  		}
>  
> +		# Use specified protocol for mirror communication (allows
> HTTPS pinning)
> +		my $urlproto = lc $proto;
> +
>  		$final_data = undef;
> -	 	my $url = "http://$host/$file";
> +	 	my $url = "$urlproto://$host/$file";
>  		my $response;
>  		
>  		unless ($bfile =~ /^counter.py\?.*/) {

Re: [PATCH] use protocol defined in server-list.db for mirror communication

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

Hello Michael,

I do not see the benefit in doing so. In functions.pl, just a few lines
above, it says:

	$proto = "HTTP" unless $proto;

Of course, we will mostly see HTTP and a few HTTPS mirrors here, but
that leaves other protocols (FTP???) possible, thereof, I did not
strictly checked if only one of these protocols is set.

Does that make sense to you?

Best regards,
Peter Müller

> Hello,
> 
> would it not be a good idea to check if $proto is either HTTP or HTTPS?
> 
> -Michael
> 
> On Sat, 2018-03-24 at 16:22 +0100, Peter Müller wrote:
> > For each mirror server, a protocol can be specified in the
> > server-list.db database. However, it was not used for the
> > actual URL query to a mirror before.
> > 
> > This might be useful for deploy HTTPS pinning for Pakfire.
> > If a mirror is known to support HTTPS, all queries to it
> > will be made with this protocol.
> > 
> > This saves some overhead if HTTPS is enforced on a mirror
> > via 301 redirects. To enable this, the server-list.db
> > needs to be adjusted.
> > 
> > Partially fixes #11661.
> > 
> > Signed-off-by: Peter Müller <peter.mueller(a)link38.eu>
> > Cc: Michael Tremer <michael.tremer(a)ipfire.org>
> > ---
> >  src/pakfire/lib/functions.pl | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl
> > index c97d4254d..94f9f1826 100644
> > --- a/src/pakfire/lib/functions.pl
> > +++ b/src/pakfire/lib/functions.pl
> > @@ -171,8 +171,11 @@ sub fetchfile {
> >  			}
> >  		}
> >  
> > +		# Use specified protocol for mirror communication (allows
> > HTTPS pinning)
> > +		my $urlproto = lc $proto;
> > +
> >  		$final_data = undef;
> > -	 	my $url = "http://$host/$file";
> > +	 	my $url = "$urlproto://$host/$file";
> >  		my $response;
> >  		
> >  		unless ($bfile =~ /^counter.py\?.*/) {  

Re: [PATCH] use protocol defined in server-list.db for mirror communication

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2173 bytes --]

What if someone is getting a malformed list? Can this not be abused? FTP would
not work with the Perl module of course...

On Mon, 2018-03-26 at 20:50 +0200, Peter Müller wrote:
> Hello Michael,
> 
> I do not see the benefit in doing so. In functions.pl, just a few lines
> above, it says:
> 
> 	$proto = "HTTP" unless $proto;
> 
> Of course, we will mostly see HTTP and a few HTTPS mirrors here, but
> that leaves other protocols (FTP???) possible, thereof, I did not
> strictly checked if only one of these protocols is set.
> 
> Does that make sense to you?
> 
> Best regards,
> Peter Müller
> 
> > Hello,
> > 
> > would it not be a good idea to check if $proto is either HTTP or HTTPS?
> > 
> > -Michael
> > 
> > On Sat, 2018-03-24 at 16:22 +0100, Peter Müller wrote:
> > > For each mirror server, a protocol can be specified in the
> > > server-list.db database. However, it was not used for the
> > > actual URL query to a mirror before.
> > > 
> > > This might be useful for deploy HTTPS pinning for Pakfire.
> > > If a mirror is known to support HTTPS, all queries to it
> > > will be made with this protocol.
> > > 
> > > This saves some overhead if HTTPS is enforced on a mirror
> > > via 301 redirects. To enable this, the server-list.db
> > > needs to be adjusted.
> > > 
> > > Partially fixes #11661.
> > > 
> > > Signed-off-by: Peter Müller <peter.mueller(a)link38.eu>
> > > Cc: Michael Tremer <michael.tremer(a)ipfire.org>
> > > ---
> > >  src/pakfire/lib/functions.pl | 5 ++++-
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl
> > > index c97d4254d..94f9f1826 100644
> > > --- a/src/pakfire/lib/functions.pl
> > > +++ b/src/pakfire/lib/functions.pl
> > > @@ -171,8 +171,11 @@ sub fetchfile {
> > >  			}
> > >  		}
> > >  
> > > +		# Use specified protocol for mirror communication (allows
> > > HTTPS pinning)
> > > +		my $urlproto = lc $proto;
> > > +
> > >  		$final_data = undef;
> > > -	 	my $url = "http://$host/$file";
> > > +	 	my $url = "$urlproto://$host/$file";
> > >  		my $response;
> > >  		
> > >  		unless ($bfile =~ /^counter.py\?.*/) {  
> 
> 

[PATCH v2] use protocol defined in server-list.db for mirror communication

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1681 bytes --]

For each mirror server, a protocol can be specified in the
server-list.db database. However, it was not used for the
actual URL query to a mirror before.

This might be useful for deploy HTTPS pinning for Pakfire.
If a mirror is known to support HTTPS, all queries to it
will be made with this protocol.

This saves some overhead if HTTPS is enforced on a mirror
via 301 redirects. To enable this, the server-list.db
needs to be adjusted.

The second version of this patch only handles protocols
HTTP and HTTPS, since we do not expect anything else here
at the moment.

Partially fixes #11661.

Signed-off-by: Peter Müller <peter.mueller(a)link38.eu>
Cc: Michael Tremer <michael.tremer(a)ipfire.org>
---
 src/pakfire/lib/functions.pl | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl
index c97d4254d..6cc177128 100644
--- a/src/pakfire/lib/functions.pl
+++ b/src/pakfire/lib/functions.pl
@@ -31,6 +31,8 @@ use HTTP::Message;
 use HTTP::Request;
 use Net::Ping;
 
+use Switch;
+
 package Pakfire;
 
 # A small color-hash :D
@@ -172,7 +174,18 @@ sub fetchfile {
 		}
 
 		$final_data = undef;
-	 	my $url = "http://$host/$file";
+
+		my $url;
+		switch ($proto) {
+			case "HTTP" { $url = "http://$host/$file"; }
+			case "HTTPS" { $url = "https://$host/$file"; }
+			else {
+				# skip all lines with unknown protocols
+				logger("DOWNLOAD WARNING: Skipping Host: $host due to unknown protocol ($proto) in mirror database");
+				next;
+			}
+		}
+
 		my $response;
 		
 		unless ($bfile =~ /^counter.py\?.*/) {
-- 
2.13.6