From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH] update-ipblocklists: remove " Skipping" log entries Date: Fri, 07 Jun 2024 11:40:19 +0100 Message-ID: <4A00625F-8B6D-46F3-A09E-584F87A3EB47@ipfire.org> In-Reply-To: <9E099FC3-0DD0-4FDC-94D4-3E44CB093DC0@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6909088517383466585==" List-Id: --===============6909088517383466585== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello everyone, I understand that this is all spamming the logs, but it is also valuable for = debugging. So I suggest we change the log level to DEBUG instead of removing the message= entirely. We should also consider to reword a few of them, because they sound a bit agg= ressive with all those exclamation marks and sometimes don=E2=80=99t convey e= nough information... > On 7 Jun 2024, at 03:49, jon wrote: >=20 > Since I love metrics! . . . >=20 > These are the counts of ipblocklist messages for 1 week. =20 > You=E2=80=99ll see " Skipping .* blocklist - It has not been modified= !" is the clear winner!! >=20 > 1925 Skipping .* blocklist - Too frequent update attempts! WHY IS THIS SO SHOUTY? The message should rather say that the update is being= skipped because of the hold-off timer not having expired, yet. > 4383 Skipping .* blocklist - It has not been modified! This should be a DEBUG message. > 0 Could not update .* blocklist - Download error! What kind of download error? At least we should have a HTTP error code here. = Running into rate limiting is different than getting 404 or even 500. > 4 Could not update .* blocklist - Unexpected error! What errors are unexpected? I think we should add more detail here. > 1069 Successfully updated .* blocklist. I don=E2=80=99t mind logging things. It is a good thing. Papertrails allow us= to find bugs a lot faster and also qualify how bug a problem is. However we should not log too much stuff that simply says =E2=80=9CI have don= e nothing=E2=80=9D. INFO should log major events like a successful update. -Michael >> On Jun 6, 2024, at 4:30 PM, jon wrote: >>=20 >> Wow! Some lists don=E2=80=99t need an update too often. >>=20 >> ``` >> [root(a)ipfire ~] # while IFS=3D'=3D' read -r theList theEpoch ; do printf= "%-40s" "${theList}=3D${theEpoch}" ; printf "%(%F)T\n" "${theEpoch}" ; done = < /var/ipfire/ipblocklist/modified | sort -k2,2 -k1,1 >> BOGON=3D1424305106 2015-02-18 >> ALIENVAULT=3D1636726250 2021-11-12 >> FEODO_IP=3D1663973704 2022-09-23 >> TOR_EXIT=3D1663971223 2022-09-23 >> FEODO_RECOMMENDED=3D1663973404 2022-09-23 >> BLOCKLIST_DE=3D1667772005 2022-11-06 >> DOH_SERVERS=3D1690684412 2023-07-29 >> TOR_ALL=3D1710361882 2024-03-13 >> EMERGING_FWRULE=3D1717561802 2024-06-04 >> SHODAN=3D1717634749 2024-06-05 >> EMERGING_COMPROMISED=3D1717621199 2024-06-05 >> CIARMY=3D1717707841 2024-06-06 >> DSHIELD=3D1717706701 2024-06-06 >> BOGON_FULL=3D1717707302 2024-06-06 >> SPAMHAUS_DROP=3D1717696303 2024-06-06 >> SPAMHAUS_EDROP=3D1717705720 2024-06-06 >> FEODO_AGGRESSIVE=3D1717708203 2024-06-06 >> [root(a)ipfire ~] #=20 >>=20 >> ``` >>=20 >>> On Jun 6, 2024, at 9:55 AM, Adolf Belka wrote: >>>=20 >>> Hi All, >>>=20 >>> On 05/06/2024 18:47, jon wrote: >>>> Comments below... >>>> Jon >>>>> On Jun 5, 2024, at 4:55 AM, Adolf Belka > wrote: >>>>>=20 >>>>> Hi All, >>>>>=20 >>>>> On 05/06/2024 11:28, Michael Tremer wrote: >>>>>> Hello Jon, >>>>>>=20 >>>>>> Why should this not be logged? >>>>>>=20 >>>> Michael - To me Line 89 ` Skipping $blocklist blocklist - Too freq= uent update attempts!` has little to no value since it is time based (i.e., i= t is not time to update). >>>> See: https://github.com/ipfire/ipfire-2.x/blob/master/src/scripts/update= -ipblocklists#L89 >>>> And to me the Line 103 ` Skipping $blocklist blocklist - It has no= t been modified!` has little value. >>>> See: https://github.com/ipfire/ipfire-2.x/blob/master/src/scripts/update= -ipblocklists#L103 >>>> If it is to be used for troubleshooting maybe the date of last modificat= ion be added to the log message (e.g., $last_modified): >>>> See: https://github.com/ipfire/ipfire-2.x/blob/master/config/cfgroot/ipb= locklist-functions.pl#L167 >>>=20 >>> I will look at doing something like that. >>>=20 >>> Regards, >>> Adolf. >>>=20 >>>> Otherwise I would remove. >>>> Just my 2c, >>>>>> -Michael >>>>>>=20 >>>>>>> On 4 Jun 2024, at 21:22, Jon Murphy > wrote: >>>>>>>=20 >>>>>>> - Remove two log entries from message log. >>>>>>>=20 >>>>>>> Signed-off-by: Jon Murphy > >>>>>>> --- >>>>>>> src/scripts/update-ipblocklists | 4 ++-- >>>>>>> 1 file changed, 2 insertions(+), 2 deletions(-) >>>>>>>=20 >>>>>>> diff --git a/src/scripts/update-ipblocklists b/src/scripts/update-ipb= locklists >>>>>>> index a17b47999..dddde8d27 100644 >>>>>>> --- a/src/scripts/update-ipblocklists >>>>>>> +++ b/src/scripts/update-ipblocklists >>>>>>> @@ -86,7 +86,7 @@ foreach my $blocklist (@blocklists) { >>>>>>> # Check if enough time has passed since the last download of the list. >>>>>>> if ($time <=3D $holdoff_time) { >>>>>>> # To frequent updates, log to syslog. >>>>>>> - &_log_to_syslog(" Skipping $blocklist blocklist - Too frequen= t update attempts!"); >>>>>>> + # &_log_to_syslog(" Skipping $blocklist blocklist - Too frequ= ent update attempts!"); >>>>>>>=20 >>>>>>> # Skip this provider. >>>>>>> next; >>>>>>> @@ -100,7 +100,7 @@ foreach my $blocklist (@blocklists) { >>>>>>> # Handle different return codes. >>>>>>> if ($return eq "not_modified") { >>>>>>> # Log notice to syslog. >>>>>>> - &_log_to_syslog(" Skipping $blocklist blocklist - It has not = been modified!"); >>>>>>> + # &_log_to_syslog(" Skipping $blocklist blocklist - It has no= t been modified!"); >>>>>>> } elsif ($return eq "dl_error") { >>>>>>> # Log error to the syslog. >>>>>>> &_log_to_syslog(" Could not update $blocklist blocklist - Down= load error\!"); >>>>> The log message about not being modified was what a forum user was able= to use to identify that the Alien Vault list had not been updated for at lea= st 17 months. >>>>> That information could not be found from the Alien Vault site as there = is no timestamp on the file being downloaded to be able to be processed. >>>>>=20 >>>> Adolf - I did not change the ` Successfully updated ...` so a user= should be able make a determination something stopped. >>>>> I would not want to lose this information otherwise when another provid= er silently closes their list because they have been taken over or decide to = concentrate on funded lists it will prove very hard to figure out if the list= s are still active, even more so as more lists get added. >>>>>=20 >>>> See my "troubleshooting" comment above. >>>>> Regards, >>>>> Adolf. >>>>>=20 >>>>>=20 >>>>>>> --=20 >>>>>>> 2.30.2 >>>>>>>=20 >>>>>=20 >>>>> --=20 >>>>> Sent from my laptop >>>>>=20 >>=20 >=20 --===============6909088517383466585==--