From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: [PATCH] update-ipblocklists: remove " Skipping" log entries Date: Mon, 24 Jun 2024 17:21:39 +0200 Message-ID: <9ea76fdc-431d-4979-803e-cc95addebba9@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8120377644176013569==" List-Id: --===============8120377644176013569== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Jon, On 18/06/2024 22:36, jon wrote: > Michael / Adolf, >=20 > Comments below . . . >=20 >=20 >> On Jun 7, 2024, at 5:40 AM, Michael Tremer > wrote: >> >> Hello everyone, >> >> I understand that this is all spamming the logs, but it is also valuable f= or debugging. >> >> So I suggest we change the log level to DEBUG instead of removing the mess= age entirely. >> >> We should also consider to reword a few of them, because they sound a bit = aggressive with all those exclamation marks and sometimes don=E2=80=99t conve= y enough information... >> >>> On 7 Jun 2024, at 03:49, jon > wrote: >>> >>> Since I love metrics! . . . >>> >>> These are the counts of ipblocklist messages for 1 week. >>> You=E2=80=99ll see " Skipping .* blocklist - It has not been modifi= ed!" is the clear winner!! >>> >>> 1925 Skipping .* blocklist - Too frequent update attempts! >> >> WHY IS THIS SO SHOUTY? The message should rather say that the update is be= ing skipped because of the hold-off timer not having expired, yet. >=20 > This is one of the two messages I removed in my patch. =C2=A0Is it OK to re= move since it falls under the =C2=A0=E2=80=9CI have done nothing=E2=80=9D cat= egory? >=20 >> >>> 4383 Skipping .* blocklist - It has not been modified! >> >> This should be a DEBUG message. >=20 > From my guess at looking at the code, this is stating the XYZ blocklist ha= s not changed since we downloaded it last and there is no reason to update. = =C2=A0To me it is not an error. =C2=A0And to me it falls under the =C2=A0=E2= =80=9CI have done nothing=E2=80=9D category. >=20 > I can do a simple patch to change this from INFO to DEBUG. =C2=A0But since = there is no working filter for log messages, it really wont do much good. =C2= =A0(The Logs > Log Settings > Detail Level of Low, Medium, High does not work) If the Low, Medium, High filter for log messages is not doing anything then i= t would seem that now is the time to fix that so that we can have the differe= nt levels of log messages and users can select the level of logs to prevent t= oo much detail, if not required. The level filter should then be able to apply to all the logs so that all inf= o is available but the filter ensures that the debug and similar messages are= only shown if the log level is set at High. Regards, Adolf. >=20 > Jon >=20 >> >>> 0 Could not update .* blocklist - Download error! >> >> What kind of download error? At least we should have a HTTP error code her= e. Running into rate limiting is different than getting 404 or even 500. >> >>> 4 Could not update .* blocklist - Unexpected error! >> >> What errors are unexpected? I think we should add more detail here. >> >>> 1069 Successfully updated .* blocklist. >> >> I don=E2=80=99t mind logging things. It is a good thing. Papertrails allow= us to find bugs a lot faster and also qualify how bug a problem is. >> >> However we should not log too much stuff that simply says =E2=80=9CI have = done nothing=E2=80=9D. INFO should log major events like a successful update. >> >> -Michael >> >>>> On Jun 6, 2024, at 4:30 PM, jon > wrote: >>>> >>>> Wow! =C2=A0Some lists don=E2=80=99t need an update too often. >>>> >>>> ``` >>>> [root(a)ipfire ~] # while IFS=3D'=3D' read -r theList theEpoch ; do prin= tf "%-40s" "${theList}=3D${theEpoch}" ; printf "%(%F)T\n" "${theEpoch}" ; don= e < /var/ipfire/ipblocklist/modified | sort -k2,2 -k1,1 >>>> BOGON=3D1424305106 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A02015-02-18 >>>> ALIENVAULT=3D1636726250 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02021-11-12 >>>> FEODO_IP=3D1663973704 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02022-= 09-23 >>>> TOR_EXIT=3D1663971223 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02022-= 09-23 >>>> FEODO_RECOMMENDED=3D1663973404 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A02022-09-23 >>>> BLOCKLIST_DE=3D1667772005 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02022-11-06 >>>> DOH_SERVERS=3D1690684412 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02023-07-29 >>>> TOR_ALL=3D1710361882 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A020= 24-03-13 >>>> EMERGING_FWRULE=3D1717561802 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02024-06-04 >>>> SHODAN=3D1717634749 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A02024-06-05 >>>> EMERGING_COMPROMISED=3D1717621199 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A02024-06-05 >>>> CIARMY=3D1717707841 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A02024-06-06 >>>> DSHIELD=3D1717706701 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A020= 24-06-06 >>>> BOGON_FULL=3D1717707302 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02024-06-06 >>>> SPAMHAUS_DROP=3D1717696303 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02024-06-06 >>>> SPAMHAUS_EDROP=3D1717705720 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02024-06-06 >>>> FEODO_AGGRESSIVE=3D1717708203 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A02024-06-06 >>>> [root(a)ipfire ~] # >>>> >>>> ``` >>>> >>>>> On Jun 6, 2024, at 9:55 AM, Adolf Belka > wrote: >>>>> >>>>> Hi All, >>>>> >>>>> On 05/06/2024 18:47, jon wrote: >>>>>> Comments below... >>>>>> Jon >>>>>>> On Jun 5, 2024, at 4:55 AM, Adolf Belka >> wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> On 05/06/2024 11:28, Michael Tremer wrote: >>>>>>>> Hello Jon, >>>>>>>> >>>>>>>> Why should this not be logged? >>>>>>>> >>>>>> Michael - To me Line 89 ` Skipping $blocklist blocklist - Too fr= equent update attempts!` has little to no value since it is time based (i.e.,= it is not time to update). >>>>>> See: https://github.com/ipfire/ipfire-2.x/blob/master/src/scripts/upda= te-ipblocklists#L89 > >>>>>> And to me the Line 103 ` Skipping $blocklist blocklist - It has = not been modified!` has little value. >>>>>> See: https://github.com/ipfire/ipfire-2.x/blob/master/src/scripts/upda= te-ipblocklists#L103 > >>>>>> If it is to be used for troubleshooting maybe the date of last modific= ation be added to the log message (e.g., $last_modified): >>>>>> See: https://github.com/ipfire/ipfire-2.x/blob/master/config/cfgroot/i= pblocklist-functions.pl#L167 > >>>>> >>>>> I will look at doing something like that. >>>>> >>>>> Regards, >>>>> Adolf. >>>>> >>>>>> Otherwise I would remove. >>>>>> Just my 2c, >>>>>>>> -Michael >>>>>>>> >>>>>>>>> On 4 Jun 2024, at 21:22, Jon Murphy >> wrote: >>>>>>>>> >>>>>>>>> - Remove two log entries from message log. >>>>>>>>> >>>>>>>>> Signed-off-by: Jon Murphy >> >>>>>>>>> --- >>>>>>>>> src/scripts/update-ipblocklists | 4 ++-- >>>>>>>>> 1 file changed, 2 insertions(+), 2 deletions(-) >>>>>>>>> >>>>>>>>> diff --git a/src/scripts/update-ipblocklists b/src/scripts/update-i= pblocklists >>>>>>>>> index a17b47999..dddde8d27 100644 >>>>>>>>> --- a/src/scripts/update-ipblocklists >>>>>>>>> +++ b/src/scripts/update-ipblocklists >>>>>>>>> @@ -86,7 +86,7 @@ foreach my $blocklist (@blocklists) { >>>>>>>>> # Check if enough time has passed since the last download of the li= st. >>>>>>>>> if ($time <=3D $holdoff_time) { >>>>>>>>> # To frequent updates, log to syslog. >>>>>>>>> - &_log_to_syslog(" Skipping $blocklist blocklist - Too frequ= ent update attempts!"); >>>>>>>>> + # &_log_to_syslog(" Skipping $blocklist blocklist - Too fre= quent update attempts!"); >>>>>>>>> >>>>>>>>> # Skip this provider. >>>>>>>>> next; >>>>>>>>> @@ -100,7 +100,7 @@ foreach my $blocklist (@blocklists) { >>>>>>>>> # Handle different return codes. >>>>>>>>> if ($return eq "not_modified") { >>>>>>>>> # Log notice to syslog. >>>>>>>>> - &_log_to_syslog(" Skipping $blocklist blocklist - It has no= t been modified!"); >>>>>>>>> + # &_log_to_syslog(" Skipping $blocklist blocklist - It has = not been modified!"); >>>>>>>>> } elsif ($return eq "dl_error") { >>>>>>>>> # Log error to the syslog. >>>>>>>>> &_log_to_syslog(" Could not update $blocklist blocklist - Do= wnload error\!"); >>>>>>> The log message about not being modified was what a forum user was ab= le to use to identify that the Alien Vault list had not been updated for at l= east 17 months. >>>>>>> That information could not be found from the Alien Vault site as ther= e is no timestamp on the file being downloaded to be able to be processed. >>>>>>> >>>>>> Adolf - I did not change the ` Successfully updated ...` so a us= er should be able make a determination something stopped. >>>>>>> I would not want to lose this information otherwise when another prov= ider silently closes their list because they have been taken over or decide t= o concentrate on funded lists it will prove very hard to figure out if the li= sts are still active, even more so as more lists get added. >>>>>>> >>>>>> See my "troubleshooting" comment above. >>>>>>> Regards, >>>>>>> Adolf. >>>>>>> >>>>>>> >>>>>>>>> --=20 >>>>>>>>> 2.30.2 >>>>>>>>> >>>>>>> >>>>>>> --=20 >>>>>>> Sent from my laptop >>>>>>> >>>> >>> >> >=20 --===============8120377644176013569==--