Re: connections.cgi: iptables connection tracking.

7 Apr 2013

I finished progress on sorting with connections.cgi and i want to share
with you guys.
Green arrows for sorting on source ip, source port, destination ip and
destination port, protocol, connections status, upload/download and ttl
will now appear on iptables connections tracking WUI.
It would be a pleasure to hear from you guys if you have further
suggestions.
To keep the sorting less time consuming and with a minimum memory
footprint, i added a bash script what is doing all the sorting and removed
the sort command from the piped command line.
The bash script "consort.sh" goes to /usr/local/bin.
The two diffs diff against the "next" git repository:

diff --git a/src/scripts/consort.sh b/src/scripts/consort.sh
new file mode 100755
index 0000000..1682f7a
--- /dev/null
+++ b/src/scripts/consort.sh
@@ -0,0 +1,158 @@
+#/bin/bash
+###############################################################################
+#
    #
+# IPFire.org - A linux based firewall
    #
+# Copyright (C) 2007-2013  IPFire Team  info@ipfire.org
    #
+#
    #
+# This program is free software: you can redistribute it and/or modify
   #
+# it under the terms of the GNU General Public License as published by
   #
+# the Free Software Foundation, either version 3 of the License, or
    #
+# (at your option) any later version.
    #
+#
    #
+# This program is distributed in the hope that it will be useful,
    #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
   #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    #
+# GNU General Public License for more details.
   #
+#
    #
+# You should have received a copy of the GNU General Public License
    #
+# along with this program.  If not, see http://www.gnu.org/licenses/.
    #
+#
    #
+###############################################################################
+
+# sort conntrack table entries based on ip addresses
+# @parm sort field
+do_ip_sort() {
+ sed \
+ -r \
+ 's/.*src=([0-9.]+).*dst=([0-9.]+).*src=.*/'$1'#\0/' $FILE_NAME \
+ | sort \
+  -t. \
+  -k 1,1n$SORT_ORDER -k 2,2n$SORT_ORDER -k 3,3n$SORT_ORDER -k
4,4n$SORT_ORDER \
+ | sed \
+  -r \
+  's/.*#(.*)/\1/'
+}
+
+# sort conntrack table entries based on port addresses
+# @parm sort field
+do_port_sort() {
+ sed \
+ -r \
+ 's/.*sport=([0-9]+).*dport=([0-9]+).*src=.*/'$1'#\0/' $FILE_NAME \
+ | sort \
+ -t# \
+ -k 1,1n$SORT_ORDER \
+ | sed \
+ -r \
+ 's/.*#(.*)/\1/'
+}
+
+# sort conntrack table entries based on protocol
+do_protocol_sort() {
+ sed \
+ -r \
+ 's/^[0-9a-zA-Z]+[ ]+[0-9]+[ ]+([a-zA-Z0-9]+)/\1#\0/' $FILE_NAME \
+ | sort \
+ -t# \
+ -k 1,1$SORT_ORDER  \
+ | sed \
+ -r \
+ 's/.*#(.*)/\1/'
+}
+
+# sort conntrack table entries based on connection status
+do_status_sort() {
+ sed \
+ -r \
+ 's/^[0-9a-zA-Z]+[ ]+[0-9]+[ ]+[a-zA-Z0-9]+[ ]+[0-9]+[ ]+[0-9]+[
]+([a-zA-Z_0-9]+)[
]+|^[0-9a-zA-Z]+[ ]+[0-9]+[ ]+[a-zA-Z0-9]+[ ]+[0-9]+[ ]+[0-9]+([ ]+)/\1#\0/'
$FILE_NAME \
+ | sort \
+ -t# \
+ -k 1,1$SORT_ORDER  \
+ | sed \
+ -r \
+ 's/.*#(.*)/\1/'
+}
+
+# sort conntrack table entries based on connection time to life
+do_ttl_sort() {
+ sed \
+ -r \
+ 's/^[0-9a-zA-Z]+[ ]+[0-9]+[ ]+[a-zA-Z0-9]+[ ]+[0-9]+[ ]+([0-9]+)[ ]+/\1#\0/'
$FILE_NAME \
+ | sort \
+ -t# \
+ -k 1,1n$SORT_ORDER  \
+ | sed \
+ -r \
+ 's/.*#(.*)/\1/'
+}
+
+# sort conntrack table entries based on downloaded bytes
+do_downloaded_bytes_sort() {
+ sed \
+ -r \
+ 's/.*src=.*bytes=([0-9]+).*src=/\1#\0/' $FILE_NAME \
+ | sort \
+ -t# \
+ -k 1,1n$SORT_ORDER  \
+ | sed \
+ -r \
+ 's/.*#(.*)/\1/'
+}
+
+# sort conntrack table entries based on uploaded bytes
+do_uploaded_bytes_sort() {
+ sed \
+ -r \
+ 's/.*src=.*bytes=([0-9]+).*/\1#\0/' $FILE_NAME \
+ | sort \
+ -t# \
+ -k 1,1n$SORT_ORDER  \
+ | sed \
+ -r \
+ 's/.*#(.*)/\1/'
+}
+
+SORT_ORDER=
+FILE_NAME=
+
+if [ $# -lt 2 ]; then
+ echo "Usage: consort <sort criteria
1=srcIp,2=dstIp,3=srcPort,4=dstPort,5=protocol,6=connection status>
<a=ascending,d=descending> [input file]"
+ echo " consort.sh 1 a a.txt"
+ echo " cat a.txt | consort 1 d"
+ exit;
+fi
+
+if [[ 'a d A D' =~ $2 ]]; then
+ if [[ 'd D' =~ $2 ]]; then
+ SORT_ORDER=r
+ fi
+else
+ echo "Unknown sort order "$2""
+ exit;
+fi
+
+if [ $# == 3 ]; then
+ if [ ! -f $3 ]; then
+ echo "File not found."
+ exit;
+ fi
+ FILE_NAME=$3
+fi
+
+if [[ '1 2' =~ $1 ]]; then
+ do_ip_sort $1
+elif [[ '3 4' =~ $1 ]]; then
+ do_port_sort $(($1-2))
+elif [[ '5' =~ $1 ]]; then
+ do_protocol_sort
+elif [[ '6' =~ $1 ]]; then
+ do_status_sort
+elif [[ '7' =~ $1 ]]; then
+ do_ttl_sort
+elif [[ '8' =~ $1 ]]; then
+ do_downloaded_bytes_sort
+elif [[ '9' =~ $1 ]]; then
+ do_uploaded_bytes_sort
+else
+ echo "Unknown sort criteria "$1""
+fi
diff --git a/html/cgi-bin/connections.cgi b/html/cgi-bin/connections.cgi
index 1edf3e5..d566cf7 100644
--- a/html/cgi-bin/connections.cgi
+++ b/html/cgi-bin/connections.cgi
@@ -34,6 +34,31 @@
my $colour_multicast = "#A0A0A0";
+# sort arguments for connection tracking table
+# the sort field. eg. 1=src IP, 2=dst IP, 3=src port, 4=dst port
+my $SORT_FIELD = 0;
+# the sort order. (a)scending orr (d)escending
+my $SORT_ORDER = 0;
+# cgi query arguments
+my %cgiin;
+# debug mode
+my $debug = 0;
+
+# retrieve query arguments
+# note: let a-z A-Z and 0-9 pass as value only
+if (length ($ENV{'QUERY_STRING'}) > 0){
+ my $name;
+ my $value;
+ my $buffer = $ENV{'QUERY_STRING'};
+ my @pairs = split(/&/, $buffer);
+ foreach my $pair (@pairs){
+ ($name, $value) = split(/=/, $pair);
+ $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg; # e.g. "%20"
=> " "
+ $value =~ s/[^a-zA-Z0-9]*//g; # a-Z 0-9 will pass
+ $cgiin{$name} = $value;
+ }
+}
+
 &Header::showhttpheaders();
my @network=();
@@ -43,12 +68,40 @@
 my %netsettings=();
 &General::readhash("${General::swroot}/ethernet/settings", %netsettings);
+# output cgi query arrguments to browser on debug
+if ( $debug ){
+ &Header::openbox('100%', 'center', 'DEBUG');
+ my $debugCount = 0;
+ foreach my $line (sort keys %cgiin) {
+ print "$line = '$cgiin{$line}'<br />\n";
+ $debugCount++;
+ }
+ print "&nbsp;Count: $debugCount\n";
+ &Header::closebox();
+}
+
 #workaround to suppress a warning when a variable is used only once
 my @dummy = ( ${Header::table1colour} );
 undef (@dummy);
-# Read the connection tracking table.
-open(CONNTRACK, "/usr/local/bin/getconntracktable | sort -k 5,5
--numeric-sort --reverse |") or die "Unable to read conntrack table";
+# check sorting arguments
+if ( $cgiin{'sort_field'} ~~ [ '1','2','3','4','5','6','7','8','9' ] ) {
+ $SORT_FIELD = $cgiin{'sort_field'};
+
+ if ( $cgiin{'sort_order'} ~~ [ 'a','d','A','D' ] ) {
+ $SORT_ORDER = lc($cgiin{'sort_order'});
+ }
+}
+
+# Read and sort the connection tracking table
+# do sorting
+if ($SORT_FIELD and $SORT_ORDER) {
+ # field sorting when sorting arguments are sane
+ open(CONNTRACK, "/usr/local/bin/getconntracktable |
/usr/local/bin/consort.sh $SORT_FIELD $SORT_ORDER |") or die "Unable to
read conntrack table";
+} else {
+ # default sorting with no query arguments
+ open(CONNTRACK, "/usr/local/bin/getconntracktable | sort -k 5,5
--numeric-sort --reverse |") or die "Unable to read conntrack table";
+}
 my @conntrack = <CONNTRACK>;
 close(CONNTRACK);
@@ -263,21 +316,81 @@
  <br>
 END
+if ($SORT_FIELD and $SORT_ORDER) {
+ my @sort_field_name = (
+ $Lang::tr{'source ip'},
+ $Lang::tr{'destination ip'},
+ $Lang::tr{'source port'},
+ $Lang::tr{'destination port'},
+ $Lang::tr{'protocol'},
+ $Lang::tr{'connection'}.' '.$Lang::tr{'status'},
+ $Lang::tr{'expires'}.' ('.$Lang::tr{'seconds'}.')',
+ $Lang::tr{'download'},
+ $Lang::tr{'upload'}
+ );
+ my $sort_order_name;
+ if (lc($SORT_ORDER) eq "a") {
+ $sort_order_name = $Lang::tr{'sort ascending'};
+ } else {
+ $sort_order_name = $Lang::tr{'sort descending'};
+ }
+
+print <<END
+ <div style="font-weight:bold;margin:10px;font-size: 70%">
+ $sort_order_name: $sort_field_name[$SORT_FIELD-1]
+ </div>
+END
+;
+}
+
 # Print table header.
 print <<END;
  <table width='100%'>
- <tr>
+ <tr valign="top"">
+ <th align='center'>
+ <a href="?sort_field=5&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=5&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ </th>
+ <th align='center' colspan="2">
+ <a href="?sort_field=1&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=1&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+ <a href="?sort_field=3&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=3&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ </th>
+ <th align='center' colspan="2">
+ <a href="?sort_field=2&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=2&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+ <a href="?sort_field=4&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=4&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ </th>
+ <th align='center'>
+ <a href="?sort_field=8&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=8&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ &nbsp;&nbsp;&nbsp;&nbsp;
+ <a href="?sort_field=9&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=9&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ </th>
+ <th align='center'>
+ <a href="?sort_field=6&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=6&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ </th>
+ <th align='center'>
+ <a href="?sort_field=7&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=7&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
+ </th>
+ </tr>
+ <tr valign="top"">
  <th align='center'>
  $Lang::tr{'protocol'}
  </th>
- <th align='center'>
+ <th align='center' colspan="2">
  $Lang::tr{'source ip and port'}
  </th>
- <th>&nbsp;</th>
- <th align='center'>
+ <th align='center' colspan="2">
  $Lang::tr{'dest ip and port'}
  </th>
- <th>&nbsp;</th>
  <th align='center'>
  $Lang::tr{'download'} /
  <br>$Lang::tr{'upload'}
2013/3/25 Michael Tremer michael.tremer@ipfire.org
...
Hey,
a shell script is not the best way how to sort things, but I think we
can accept this because before the current version of the CGI file,
there was a lot of sorting done with shell commands as well.
I did not try to run your code, because I am waiting for the other "two
more sorting options".
Michael
On Sat, 2013-03-23 at 09:38 +0100, Kay-Michael Köhler wrote:
...
I did some progress on sorting with connections.cgi and i want to
share with you guys.
To keep the sorting less time consuming and with a minimum memory
footprint, i added a bash script what is doing all the sorting and
removed the sort command from the piped command line.
Eight green arrows for sorting on source ip, source port, destination
ip and destination port will now appear on iptables connections
tracking WUI.
I will add two more sorting option later after finished some other
work for ipfire.
The bash script "consort.sh" goes to /usr/local/bin.
Ok, now the two diffs to the actual git
diff --git a/src/scripts/consort.sh b/src/scripts/consort.sh
new file mode 100755
index 0000000..1633beb
--- /dev/null
+++ b/src/scripts/consort.sh
@@ -0,0 +1,83 @@
+#/bin/bash
+###############################################################################
...
+#
#
+# IPFire.org - A linux based firewall
#
+# Copyright (C) 2007-2013  IPFire Team  info@ipfire.org
#
+#
#
+# This program is free software: you can redistribute it and/or
modify        #
+# it under the terms of the GNU General Public License as published
by        #
+# the Free Software Foundation, either version 3 of the License, or
#
+# (at your option) any later version.
#
+#
#
+# This program is distributed in the hope that it will be useful,
#
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
         #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#
+# GNU General Public License for more details.
         #
+#
#
+# You should have received a copy of the GNU General Public License
#
+# along with this program.  If not, see
http://www.gnu.org/licenses/.       #
+#
#
+###############################################################################
...



+# sort conntrack table entries based on ip addresses
+# @parm sort field
+do_ip_sort() {

sed \
-r \
's/.*src=([0-9.]+).*dst=([0-9.]+).*src=.*/'$1'#\0/' $FILE_NAME \
| sort \
-t. \
-k 1,1n$SORT_ORDER -k 2,2n$SORT_ORDER -k 3,3n$SORT_ORDER -k 4,4n

$SORT_ORDER \

| sed \
-r \
's/.*#(.*)/\1/'

+}



+# sort conntrack table entries based on port addresses
+# @parm sort field
+do_port_sort() {

sed \
-r \
's/.*sport=([0-9]+).*dport=([0-9]+).*src=.*/'$1'#\0/' $FILE_NAME \
| sort \
-t# \
-k 1,1n$SORT_ORDER \
| sed \
-r \
's/.*#(.*)/\1/'

+}



+SORT_ORDER=
+FILE_NAME=



+if [ $# -lt 2 ]; then

echo "Usage: consort <sort criteria

1=srcIp,2=dstIp,3=srcPort,4=dstPort> <a=ascending,d=descending> [input
file]"

echo " consort.sh 1 a.txt"
echo " cat a.txt | consort 1"
exit;

+fi



+if [[ 'a d A D' =~ $2 ]]; then

if [[ 'd D' =~ $2 ]]; then
SORT_ORDER=r
fi

+else

echo "Unknown sort order "$2""
exit;

+fi



+if [ $# == 3 ]; then

if [ ! -f $3 ]; then
echo "File not found."
exit;
fi
FILE_NAME=$3

+fi



+if [[ '1 2' =~ $1 ]]; then

do_ip_sort $1

+elif [[ '3 4' =~ $1 ]]; then

do_port_sort $(($1-2))

+else

echo "Unknown sort criteria "$1""

+fi
diff --git a/html/cgi-bin/connections.cgi
b/html/cgi-bin/connections.cgi
index 1edf3e5..0c20957 100644
--- a/html/cgi-bin/connections.cgi
+++ b/html/cgi-bin/connections.cgi
@@ -34,6 +34,31 @@
my $colour_multicast = "#A0A0A0";
+# sort arguments for connection tracking table
+# the sort field. eg. 1=src IP, 2=dst IP, 3=src port, 4=dst port
+my $SORT_FIELD = 0;
+# the sort order. (a)scending orr (d)escending
+my $SORT_ORDER = 0;
+# cgi query arguments
+my %cgiin;
+# debug mode
+my $debug = 0;



+# retrieve query arguments
+# note: let a-z A-Z and 0-9 pass as value only
+if (length ($ENV{'QUERY_STRING'}) > 0){

my $name;
my $value;
my $buffer = $ENV{'QUERY_STRING'};
my @pairs = split(/&/, $buffer);
foreach my $pair (@pairs){
($name, $value) = split(/=/, $pair);
$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg; # e.g.

"%20" => " "

$value =~ s/[^a-zA-Z0-9]*//g; # a-Z 0-9 will pass
$cgiin{$name} = $value;
}

+}



&Header::showhttpheaders();
my @network=();
@@ -43,12 +68,43 @@
 my %netsettings=();
 &General::readhash("${General::swroot}/ethernet/settings", %
netsettings);
+# output cgi query arrguments to browser on debug
+if ( $debug ){

&Header::openbox('100%', 'center', 'DEBUG');
my $debugCount = 0;
foreach my $line (sort keys %cgiin) {
print "$line = '$cgiin{$line}'<br />\n";
$debugCount++;
}
print "&nbsp;Count: $debugCount\n";
&Header::closebox();

+}



#workaround to suppress a warning when a variable is used only once
 my @dummy = ( ${Header::table1colour} );
 undef (@dummy);
-# Read the connection tracking table.
-open(CONNTRACK, "/usr/local/bin/getconntracktable | sort -k 5,5
--numeric-sort --reverse |") or die "Unable to read conntrack table";




+# check sorting arguments
+if ( $cgiin{'sort_field'} ~~ [ '1','2','3','4' ] ) {

$SORT_FIELD = $cgiin{'sort_field'};

if ( $cgiin{'sort_order'} ~~ [ 'a','d','A','D' ] ) {
$SORT_ORDER = lc($cgiin{'sort_order'});
}

+}



+# Read and sort the connection tracking table
+# do sorting
+if ($SORT_FIELD and $SORT_ORDER) {

# field sorting when sorting arguments are sane
open(CONNTRACK, "/usr/local/bin/getconntracktable

| /usr/local/bin/consort.sh $SORT_FIELD $SORT_ORDER |") or die "Unable
to read conntrack table";
+} else {

# default sorting with no query arguments
open(CONNTRACK, "/usr/local/bin/getconntracktable | sort -k 5,5

--numeric-sort --reverse |") or die "Unable to read conntrack table";
+}



my @conntrack = <CONNTRACK>;
 close(CONNTRACK);
@@ -263,21 +319,49 @@
  <br>
 END
+if ($SORT_FIELD and $SORT_ORDER) {

my @sort_field_name = (
$Lang::tr{'source ip'},
$Lang::tr{'destination ip'},
$Lang::tr{'source port'},
$Lang::tr{'destination port'}
);
my $sort_order_name;
if (lc($SORT_ORDER) eq "a") {
$sort_order_name = $Lang::tr{'sort ascending'};
} else {
$sort_order_name = $Lang::tr{'sort descending'};
}


+print <<END

<div style="font-weight:bold;margin:10px;font-size: 80%">

$sort_order_name: $sort_field_name[$SORT_FIELD-1]
</div>


+END
+;
+}



# Print table header.
 print <<END;
  <table width='100%'>
- <tr>
+ <tr valign="top"">
  <th align='center'>
  $Lang::tr{'protocol'}
  </th>
- <th align='center'>
+ <th align='center' colspan="2">
+ <a href="?sort_field=1&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=1&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
  $Lang::tr{'source ip and port'}
+ <a href="?sort_field=3&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=3&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
  </th>
- <th>&nbsp;</th>
- <th align='center'>
+ <th align='center' colspan="2">
+ <a href="?sort_field=2&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=2&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
  $Lang::tr{'dest ip and port'}
+ <a href="?sort_field=4&sort_order=d"><img style="width:10px"
src="/images/up.gif"></a>
+ <a href="?sort_field=4&sort_order=a"><img style="width:10px"
src="/images/down.gif"></a>
  </th>
- <th>&nbsp;</th>
  <th align='center'>
  $Lang::tr{'download'} /
  <br>$Lang::tr{'upload'}
2013/3/10 Michael Tremer michael.tremer@ipfire.org
        Hi,
    sure, this is fine with me. Just try to make the sorting
    process
    efficient so that even ten thousands of connections are
    properly
    displayed.

    -Michael

    On Sun, 2013-03-10 at 17:01 +0100, Kay-Michael Köhler wrote:
    > Hello everyone
    >
    >
    > i'm going to start development on connections.cgi to have
    some kind of
    > sorting at "iptables connection
    tracking" (status->connections)
    >
    >
    > I think it is a good idea to have the following (asc/desc)
    sort
    > options:
    >
    >
    > "Protocol"
    > "Source IP:Port"
    > "Destination IP Port"
    > "Connection status"
    >
    >
    > If you guys agree it would be a please for me to share and
    post the
    > patch here when i'm done.
    >
    >
    > Regards
    >
    >
    > Kay-Michael

    > _______________________________________________
    > Development mailing list
    > Development@lists.ipfire.org
    > http://lists.ipfire.org/mailman/listinfo/development

    _______________________________________________
    Development mailing list
    Development@lists.ipfire.org
    http://lists.ipfire.org/mailman/listinfo/development




    

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Re: connections.cgi: iptables connection tracking.