From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: [PATCH] squid-graph 3.2: Proxy Graphs rebuilt (V2) Date: Mon, 09 Oct 2017 20:19:51 +0200 Message-ID: <0723214a-0c65-b86c-8a19-a9a8270e12e0@ipfire.org> In-Reply-To: <1507559329.4045.18.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0468311850951263771==" List-Id: --===============0468311850951263771== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On 09.10.2017 16:28, Michael Tremer wrote: > On Wed, 2017-10-04 at 21:01 +0200, Matthias Fischer wrote: >> Hi, >>=20 >> On 04.10.2017 18:36, Michael Tremer wrote: >> > Any screenshots? >>=20 >> See attachment. Hope this helps. >=20 > Yes looks good. >=20 > But why is it half German and half English? Any chance we can localise the = squid > graph sections?! This is done through 'squid-graph.cgi'. I'll take a look... >> Please note: >> I'd like to handle this addon as "under review", because during my >> testing of the last week I found that analysing rather LARGE squid >> access logs can last a while (here: ~15sec) and produce system loads of >> about ~15% while working. >=20 > Well, that is probably okay. Well...then... I wasn't sure. ;-) >> Because of this I moved this job from 'makegraphs' (which runs every 5 >> minutes) to a hourly cron job. This fits my needs and the load is ok. >=20 > How do you handle it then that the access log file is being rotated? You wi= ll > miss log lines on a busy system which isn't really what we want. Yep. As I wrote, this was only for testing. At the time, I had about 1.5 million lines in 'access.log' and wasn't sure how this would be handled. See original comment below. I'll try a translation and will report. Best, Matthias >=20 > -Michael >=20 >>=20 >> Original comment about this from squid-graphs-README: >>=20 >> "RUNNING SQUID-GRAPH IN REAL TIME VIA THE WEB >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>=20 >> As of v3, Squid Graph does not support this feature anymore. If you >> would still like to do this, simply create a shell script to execute >> Squid Graph. >>=20 >> We do not recommend this unless your log files are really small. Also, >> we did not audit Squid Graph for any possible security issues that might >> arise in an online CGI enviroment. YOU HAVE BEEN WARNED." >>=20 >> Best, >> Matthias >>=20 >> > Tester's feedback wanted! >> >=20 >> > On Fri, 2017-09-29 at 08:50 +0200, Matthias Fischer wrote: >> > > This is a rather old fellow. >> > >=20 >> > > For details see: >> > > https://sourceforge.net/projects/squid-graph/ >> > >=20 >> > > "Squid Graph is a free, simple, yet powerful Squid v2 native logfile >> > > analysis >> > > tool >> > > that generates reports with graphical representation of the proxy serv= er's >> > > traffic, >> > > somewhat like the popular MRTG tool." >> > >=20 >> > > I found by chance that the corresponding translation strings were still >> > > present, >> > > so I thought "Why not?". ;-) >> > >=20 >> > > Here, it works without problems and 'squid 3.5.27'. >> > >=20 >> > > The original 'sgraph.cgi' was slightly tuned - like in IPCop - and >> > > polished >> > > for better readability. >> > >=20 >> > > If someone likes it, here it is - again... >> > >=20 >> > > Best, >> > > Matthias >> > >=20 >> > > Signed-off-by: Matthias Fischer >> > > --- >> > > config/menu/20-status.menu | 10 +++- >> > > config/rootfiles/common/squid-graph | 3 ++ >> > > config/rootfiles/common/web-user-interface | 1 + >> > > html/cgi-bin/sgraph.cgi | 72 >> > > ++++++++++++++++++++++++++ >> > > lfs/squid-graph | 81 >> > > ++++++++++++++++++++++++++++++ >> > > make.sh | 1 + >> > > src/scripts/makegraphs | 9 +++- >> > > 7 files changed, 175 insertions(+), 2 deletions(-) >> > > create mode 100644 config/rootfiles/common/squid-graph >> > > create mode 100644 html/cgi-bin/sgraph.cgi >> > > create mode 100644 lfs/squid-graph >> > >=20 >> > > diff --git a/config/menu/20-status.menu b/config/menu/20-status.menu >> > > index 2bcf0d5e8..dfc6dae9b 100644 >> > > --- a/config/menu/20-status.menu >> > > +++ b/config/menu/20-status.menu >> > > @@ -58,7 +58,15 @@ >> > > 'title' =3D> "$Lang::tr{'hardware >> > > graphs'}", >> > > 'enabled' =3D> 1, >> > > }; >> > > - $substatus->{'61.entropy'} =3D { >> > > + >> > > + $substatus->{'61.proxygraphs'} =3D { >> > > + 'caption' =3D> "$Lang::tr{'ssproxy >> > > graphs'}", >> > > + 'uri' =3D> '/cgi-bin/sgraph.cgi', >> > > + 'title' =3D> "$Lang::tr{'ssproxy graphs'}", >> > > + 'enabled' =3D> 1, >> > > + }; >> > > + >> > > + $substatus->{'62.entropy'} =3D { >> > > 'caption' =3D> "$Lang::tr{'entropy'}", >> > > 'uri' =3D> '/cgi-bin/entropy.cgi', >> > > 'title' =3D> "$Lang::tr{'entropy graphs'}", >> > > diff --git a/config/rootfiles/common/squid-graph >> > > b/config/rootfiles/common/squid-graph >> > > new file mode 100644 >> > > index 000000000..0b558d87a >> > > --- /dev/null >> > > +++ b/config/rootfiles/common/squid-graph >> > > @@ -0,0 +1,3 @@ >> > > +srv/web/ipfire/html/sgraph >> > > +srv/web/ipfire/html/sgraph/logo.png >> > > +usr/bin/squid-graph >> > > diff --git a/config/rootfiles/common/web-user-interface >> > > b/config/rootfiles/common/web-user-interface >> > > index f204f1657..b14efb402 100644 >> > > --- a/config/rootfiles/common/web-user-interface >> > > +++ b/config/rootfiles/common/web-user-interface >> > > @@ -72,6 +72,7 @@ srv/web/ipfire/cgi-bin/routing.cgi >> > > #srv/web/ipfire/cgi-bin/samba.cgi >> > > #srv/web/ipfire/cgi-bin/sambahlp.cgi >> > > srv/web/ipfire/cgi-bin/services.cgi >> > > +srv/web/ipfire/cgi-bin/sgraph.cgi >> > > srv/web/ipfire/cgi-bin/shutdown.cgi >> > > srv/web/ipfire/cgi-bin/speed.cgi >> > > srv/web/ipfire/cgi-bin/system.cgi >> > > diff --git a/html/cgi-bin/sgraph.cgi b/html/cgi-bin/sgraph.cgi >> > > new file mode 100644 >> > > index 000000000..dea4c068a >> > > --- /dev/null >> > > +++ b/html/cgi-bin/sgraph.cgi >> > > @@ -0,0 +1,72 @@ >> > > +#!/usr/bin/perl >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > +# = =20 >> > > =20 >> > > # >> > > +# IPFire.org - A linux based >> > > firewall # >> > > +# Copyright (C) 2016 IPFire Team = =20 >> > > =20 >> > > # >> > > +# = =20 >> > > =20 >> > > # >> > > +# This program is free software: you can redistribute it and/or >> > > modify # >> > > +# it under the terms of the GNU General Public License as published >> > > by # >> > > +# the Free Software Foundation, either version 3 of the License, >> > > or # >> > > +# (at your option) any later >> > > version. # >> > > +# = =20 >> > > =20 >> > > # >> > > +# This program is distributed in the hope that it will be >> > > useful, # >> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty >> > > of # >> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See >> > > the # >> > > +# GNU General Public License for more >> > > details. # >> > > +# = =20 >> > > =20 >> > > # >> > > +# You should have received a copy of the GNU General Public >> > > License # >> > > +# along with this program. If not, see . =20 >> > > =20 >> > > # >> > > +# = =20 >> > > =20 >> > > # >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > + >> > > +use strict; >> > > + >> > > +# enable only the following on debugging purpose >> > > +#use warnings; >> > > +#use CGI::Carp 'fatalsToBrowser'; >> > > + >> > > +require '/var/ipfire/general-functions.pl'; >> > > +require "${General::swroot}/lang.pl"; >> > > +require "${General::swroot}/header.pl"; >> > > + >> > > +my @graphs =3D (); >> > > + >> > > +&Header::showhttpheaders(); >> > > + >> > > +my $dir =3D "/srv/web/ipfire/html/sgraph"; >> > > +my $sgraphdir =3D "/srv/web/ipfire/html/sgraph"; >> > > + >> > > +&Header::openpage($Lang::tr{'ssproxy graphs'}, 1, ''); >> > > + >> > > +&Header::openbigbox('100%', 'left'); >> > > + >> > > +&Header::openbox('100%', 'left', $Lang::tr{'proxy access graphs'} . "= :" >> > > ); >> > > + >> > > +if (open(IPACHTML, "$sgraphdir/index.html")) >> > > +{ >> > > + my $skip =3D 1; >> > > + while () >> > > + { >> > > + $skip =3D 1 if /^
$/; >> > > + if ($skip) >> > > + { >> > > + $skip =3D 0 if /

/; >> > > + next; >> > > + } >> > > + s/]+)>/3D'Graph'> > > \/>/; >> > > + s/
/
/g; >> > > + s/
/
/g; >> > > + s/<([^>]*)>/\L<$1>\E/g; >> > > + s/(size|align|border|color)=3D([^'"> ]+)/$1=3D'$2'/g; >> > > + print; >> > > + } >> > > + close(IPACHTML); >> > > +} >> > > +else { >> > > + print $Lang::tr{'no information available'}; } >> > > + >> > > +&Header::closebox(); >> > > + >> > > +&Header::closebigbox(); >> > > + >> > > +&Header::closepage(); >> > > diff --git a/lfs/squid-graph b/lfs/squid-graph >> > > new file mode 100644 >> > > index 000000000..e94c0c130 >> > > --- /dev/null >> > > +++ b/lfs/squid-graph >> > > @@ -0,0 +1,81 @@ >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > +# = =20 >> > > =20 >> > > # >> > > +# IPFire.org - A linux based >> > > firewall # >> > > +# Copyright (C) 2007-2015 IPFire Team = =20 >> > > =20 >> > > # >> > > +# = =20 >> > > =20 >> > > # >> > > +# This program is free software: you can redistribute it and/or >> > > modify # >> > > +# it under the terms of the GNU General Public License as published >> > > by # >> > > +# the Free Software Foundation, either version 3 of the License, >> > > or # >> > > +# (at your option) any later >> > > version. # >> > > +# = =20 >> > > =20 >> > > # >> > > +# This program is distributed in the hope that it will be >> > > useful, # >> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty >> > > of # >> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See >> > > the # >> > > +# GNU General Public License for more >> > > details. # >> > > +# = =20 >> > > =20 >> > > # >> > > +# You should have received a copy of the GNU General Public >> > > License # >> > > +# along with this program. If not, see . =20 >> > > =20 >> > > # >> > > +# = =20 >> > > =20 >> > > # >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > + >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > +# Definitions >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > + >> > > +include Config >> > > + >> > > +VER =3D 3.2 >> > > + >> > > +THISAPP =3D squid-graph-$(VER) >> > > +DL_FILE =3D $(THISAPP).tar.gz >> > > +DL_FROM =3D $(URL_IPFIRE) >> > > +DIR_APP =3D $(DIR_SRC)/$(THISAPP) >> > > +TARGET =3D $(DIR_INFO)/$(THISAPP) >> > > + >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > +# Top-level Rules >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > + >> > > +objects =3D $(DL_FILE) >> > > + >> > > +$(DL_FILE) =3D $(DL_FROM)/$(DL_FILE) >> > > + >> > > +$(DL_FILE)_MD5 =3D 89ac122a99dc2db67efbdcd2467b9939 >> > > + >> > > +install : $(TARGET) >> > > + >> > > +check : $(patsubst %,$(DIR_CHK)/%,$(objects)) >> > > + >> > > +download :$(patsubst %,$(DIR_DL)/%,$(objects)) >> > > + >> > > +md5 : $(subst %,%_MD5,$(objects)) >> > > + >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > +# Downloading, checking, md5sum >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > + >> > > +$(patsubst %,$(DIR_CHK)/%,$(objects)) : >> > > + @$(CHECK) >> > > + >> > > +$(patsubst %,$(DIR_DL)/%,$(objects)) : >> > > + @$(LOAD) >> > > + >> > > +$(subst %,%_MD5,$(objects)) : >> > > + @$(MD5) >> > > + >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > +# Installation Details >> > > +#####################################################################= #### >> > > #### >> > > ## >> > > + >> > > +$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) >> > > + @$(PREBUILD) >> > > + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf >> > > $(DIR_DL)/$(DL_FILE) >> > > + >> > > + cp -f $(DIR_SRC)/squid-graph/squid-graph /usr/bin/squid-graph >> > > + chmod 755 /usr/bin/squid-graph >> > > + >> > > + -mkdir -p /srv/web/ipfire/html/sgraph >> > > + cp -f $(DIR_SRC)/squid-graph/logo.png /srv/web/ipfire/html/sgraph >> > > + >> > > + @rm -rf $(DIR_APP) >> > > + @$(POSTBUILD) >> > > diff --git a/make.sh b/make.sh >> > > index 815b25aca..eac4b229f 100755 >> > > --- a/make.sh >> > > +++ b/make.sh >> > > @@ -638,6 +638,7 @@ buildipfire() { >> > > lfsmake2 snort >> > > lfsmake2 oinkmaster >> > > lfsmake2 squid >> > > + lfsmake2 squid-graph >> > > lfsmake2 squidguard >> > > lfsmake2 calamaris >> > > lfsmake2 tcpdump >> > > diff --git a/src/scripts/makegraphs b/src/scripts/makegraphs >> > > index 0b56b9325..0191ef70b 100644 >> > > --- a/src/scripts/makegraphs >> > > +++ b/src/scripts/makegraphs >> > > @@ -79,7 +79,7 @@ sub updatehdddata{ >> > > print "Error in RRD::create for hdd-".$array[$#array].": >> > > $ERROR\n" if $ERROR; >> > > } >> > > =20 >> > > - # Temperaturlesen w=EF=BF=BDrde die Platte aufwecken!!! >> > > + # Temperaturlesen w=C3=BCrde die Platte aufwecken!!! >> > > if (!$standby){ >> > > $temp =3D 0; >> > > my $smart_output =3D ''; >> > > @@ -134,3 +134,10 @@ foreach (@disks){ >> > > =20 >> > > updatehdddata($disk); >> > > } >> > > + >> > > +### >> > > +### Squid Graphs >> > > +### >> > > +if ( -e "/var/log/squid/access.log" ) { >> > > + system ("/usr/bin/squid-graph --no-console-log --tcp-only -- >> > > output- >> > > dir=3D/srv/web/ipfire/html/sgraph < /var/log/squid/access.log >/dev/nu= ll >> > > 2>&1"); >> > > +} >>=20 >>=20 >=20 --===============0468311850951263771==--