From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: [PATCH] update-ipblocklists: remove " Skipping" log entries Date: Fri, 07 Jun 2024 09:40:51 +0200 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4602041776206147746==" List-Id: --===============4602041776206147746== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Jon, On 06/06/2024 23:30, jon wrote: > Wow! =C2=A0Some lists don=E2=80=99t need an update too often. The code needs to provide the actual last date only for the lists that are en= abled, not all the ones that have an entry in the modified file. That file ke= eps all the history for lists that were enabled but then rapidly disable and = not used again. The Bogon file is one of those. It was enabled and for the first entry the co= de puts in place a reference unix epoch value from 2015, before the blocklist= code was released. Then that value is updated when the list is next updated = but the update is only done on entries that are enabled. I think it still shows the benefit of the data. If there really were ip block= lists enabled that had last been updated in September 2022, I would want to k= now that because it would bring into question the benefit of that list and ra= ise the question of removing that list from the sources file. Regards, Adolf. >=20 > ``` > [*root(a)ipfire*~] # while IFS=3D'=3D' read -r theList theEpoch ; do printf= "%-40s" "${theList}=3D${theEpoch}" ; printf "%(%F)T\n" "${theEpoch}" ; done = < /var/ipfire/ipblocklist/modified | sort -k2,2 -k1,1 > BOGON=3D1424305106=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 2015-02-18 > ALIENVAULT=3D1636726250 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 2021-11-12 > FEODO_IP=3D1663973704 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 2022-09-23 > TOR_EXIT=3D1663971223 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 2022-09-23 > FEODO_RECOMMENDED=3D1663973404=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 202= 2-09-23 > BLOCKLIST_DE=3D1667772005 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 2022-11-06 > DOH_SERVERS=3D1690684412=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 2023-07-29 > TOR_ALL=3D1710361882=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 2024-03-13 > EMERGING_FWRULE=3D1717561802=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 2024-06-04 > SHODAN=3D1717634749 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 2024-06-05 > EMERGING_COMPROMISED=3D1717621199 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2024-06-05 > CIARMY=3D1717707841 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 2024-06-06 > DSHIELD=3D1717706701=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 2024-06-06 > BOGON_FULL=3D1717707302 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 2024-06-06 > SPAMHAUS_DROP=3D1717696303=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 2024-06-06 > SPAMHAUS_EDROP=3D1717705720 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 2024-06-06 > FEODO_AGGRESSIVE=3D1717708203 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 202= 4-06-06 > [*root(a)ipfire*~] # >=20 > ``` >=20 >> On Jun 6, 2024, at 9:55 AM, Adolf Belka > wrote: >> >> Hi All, >> >> On 05/06/2024 18:47, jon wrote: >>> Comments below... >>> Jon >>>> On Jun 5, 2024, at 4:55 AM, Adolf Belka >> wrote: >>>> >>>> Hi All, >>>> >>>> On 05/06/2024 11:28, Michael Tremer wrote: >>>>> Hello Jon, >>>>> >>>>> Why should this not be logged? >>>>> >>> Michael - To me Line 89 ` Skipping=C2=A0$blocklist=C2=A0blocklist -= Too frequent update attempts!` has little to no value since it is time based= (i.e., it is not time to update). >>> See: https://github.com/ipfire/ipfire-2.x/blob/master/src/scripts/update-= ipblocklists#L89 > >>> And to me the Line 103 ` Skipping=C2=A0$blocklist blocklist - It ha= s not been modified!` has little value. >>> See: https://github.com/ipfire/ipfire-2.x/blob/master/src/scripts/update-= ipblocklists#L103 > >>> If it is to be used for troubleshooting maybe the date of last modificati= on be added to the log message (e.g., $last_modified): >>> See: https://github.com/ipfire/ipfire-2.x/blob/master/config/cfgroot/ipbl= ocklist-functions.pl#L167 > >> >> I will look at doing something like that. >> >> Regards, >> Adolf. >> >>> Otherwise I would remove. >>> Just my 2c, >>>>> -Michael >>>>> >>>>>> On 4 Jun 2024, at 21:22, Jon Murphy >> wrote: >>>>>> >>>>>> - Remove two log entries from message log. >>>>>> >>>>>> Signed-off-by: Jon Murphy >> >>>>>> --- >>>>>> src/scripts/update-ipblocklists | 4 ++-- >>>>>> 1 file changed, 2 insertions(+), 2 deletions(-) >>>>>> >>>>>> diff --git a/src/scripts/update-ipblocklists b/src/scripts/update-ipbl= ocklists >>>>>> index a17b47999..dddde8d27 100644 >>>>>> --- a/src/scripts/update-ipblocklists >>>>>> +++ b/src/scripts/update-ipblocklists >>>>>> @@ -86,7 +86,7 @@ foreach my $blocklist (@blocklists) { >>>>>> # Check if enough time has passed since the last download of the list. >>>>>> if ($time <=3D $holdoff_time) { >>>>>> # To frequent updates, log to syslog. >>>>>> - &_log_to_syslog(" Skipping $blocklist blocklist - Too frequent= update attempts!"); >>>>>> + # &_log_to_syslog(" Skipping $blocklist blocklist - Too freque= nt update attempts!"); >>>>>> >>>>>> # Skip this provider. >>>>>> next; >>>>>> @@ -100,7 +100,7 @@ foreach my $blocklist (@blocklists) { >>>>>> # Handle different return codes. >>>>>> if ($return eq "not_modified") { >>>>>> # Log notice to syslog. >>>>>> - &_log_to_syslog(" Skipping $blocklist blocklist - It has not b= een modified!"); >>>>>> + # &_log_to_syslog(" Skipping $blocklist blocklist - It has not= been modified!"); >>>>>> } elsif ($return eq "dl_error") { >>>>>> # Log error to the syslog. >>>>>> &_log_to_syslog(" Could not update $blocklist blocklist - Downl= oad error\!"); >>>> The log message about not being modified was what a forum user was able = to use to identify that the Alien Vault list had not been updated for at leas= t 17 months. >>>> That information could not be found from the Alien Vault site as there i= s no timestamp on the file being downloaded to be able to be processed. >>>> >>> Adolf - I did not change the ` Successfully updated ...` so a user = should be able make a determination something stopped. >>>> I would not want to lose this information otherwise when another provide= r silently closes their list because they have been taken over or decide to c= oncentrate on funded lists it will prove very hard to figure out if the lists= are still active, even more so as more lists get added. >>>> >>> See my "troubleshooting" comment above. >>>> Regards, >>>> Adolf. >>>> >>>> >>>>>> --=20 >>>>>> 2.30.2 >>>>>> >>>> >>>> --=20 >>>> Sent from my laptop >>>> >=20 --===============4602041776206147746==--