Hi Michael,
On 03/12/2024 18:44, Michael Tremer wrote:
Okay, this sounds really good then and seems to be a lot easier than expected.
Are there any changes we would like to make to the graphs in this step? Remove? Add new things?
It does seem much easier than expected. Based on that, my preference would be to just do the collectd version change and make sure that everything is really as easy as I have thought. Then we can look at any further changes or additions to graphs at that point.
Regards,
Adolf.
On 29 Nov 2024, at 19:24, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 25/11/2024 19:25, Adolf Belka wrote:
Hi Michael,
On 23/11/2024 13:09, Adolf Belka wrote:
Hi Michael,
On 23/11/2024 10:12, Michael Tremer wrote:
Hello Adolf,
Hmm, then let us maybe look at migrating collectd. It will be a pain, but it seems that fixing the autoconf problem won’t be much easier.
There is a migration script here:
https://github.com/collectd/collectd/blob/db4a13bdf37b1103c2014261458323e26b...
Should we maybe start with building a new branch with the latest version of collectd 5 and simply check if a newly installed system is still working? I don’t think there will be that many changes required. Would you be up for giving that a go?
I will give that a go and see what happens and we can work from there.
I gave it a go and have been very successful.
Building the collectd-5.12.0 did result in various build failures. I think I had to do around 7 build attempts but I progressively moved forward and eventually succeeded in a successful build of the new collectd and then the rest of the IPFire build.
I then installed the CU190 iso with collectd-5.12.0 into a vm system. The fresh install already updates all the rrd files with the updated changes from 4.x to 5.x
There were a range of graphs that failed due to, for example, a DS named ping not being found in the ping rrd file.
Checking the ping rrd file with rrdtool info I found that in 5.x the ping rrd file DS is now "value" in place of "ping".
I then found the DEF statement in graphs.pl for the ping graph and changed the rrd:ping section to rrd:value and the graph started operating and collecting data. I also did it with the conntrack entry which changed from "entropy" to "value" and again the graph started working.
So I know now what I need to do - find all the places where the DS has changed in an rrd file and correct those in the IPFire code.
I tested out the migrate-4-5.px file and in its first phase it creates a script which will update the DS values to their new versions. I can use the output of that script to identify which files have got a change in them, find where in the code that file is accessed and modify it.
We will also need to use the migrate script in the restore portion of the backup process as the rrd files are in the backup and restoring them will give files with a DS of "ping" for the ping rrd one, which will not be recognised with the modified DS values in graphs.pl etc.
Running the migrate script after doing a restore will convert all the DS values to their 5.x values. If the entries in the backup are already updated then the rrdtool tune command run by the migrate script will give the error message
ERROR: unknown data source name 'ping'
in the example of the ping rrd file but in that case the entry is left as it is so the migrate script should be able to work on restores with the 4.x and 5.x rrd values.
It looks like both DS names and DS types have been changed for some of the plugins.
Anyway, things are looking very promising. Should be able to look at testing this out in unstable early in the new year.
So it turned out to be much easier than any of us thought. The changes needed in graphs.pl are only 4 lines to be changed. The interface directory becomes the interface-$interface directory and there are two lines for that and then the rrd:ping (gateway graph) and rrd:entropy (conntrack graph) entries need to both be changed to rrd:value.
I also added the migrate-4-5 script into backup.pl
Tested all above out on my vm testbed. Fresh install of IPFire gave the new names and directories as required and everything worked. I also then restored an older backup that had the old names etc and the code in backup.pl also worked without any hitch. The graphs all worked.
The only thing I haven't done yet is adding the migrate code I put into backup.pl into the update.sh script. I will do that when next is updated to become CU191.
I will also just confirm that with the new collectd is happy to build with autoconf-2.72. Everything I have done so far is with the existing autoconf-2.71.
I think that the collectd update should be good to go into Core Update 192 early in the new year.
Regards,
Adolf.
Regards,
Adolf.
Regards,
Adolf
If we can then run the migration script on existing systems we might almost be there…
-Michael
On 22 Nov 2024, at 18:01, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 20/11/2024 19:25, Michael Tremer wrote: > Let’s maybe try to avoid running autoconf altogether. > There might be a small chance because only two patches change configure.ac. There are others that change Makefile.in, so this might not work at all. Let’s see… > Could you please remove the patches in src/patches/collectd with number 12 and 13? Then remove the autoreconf commands from lfs/collectd. > Things should hopefully still apply and after the first run of configure, make should not try to run autoconf again. Not sure but worth a try.
I removed the two patches and also removed the two autoreconf commands, leaving the autoupdate command in place.
Unfortunately it failed to build and had the following error message
aclocal.m4:16: warning: this file was generated for autoconf 2.67. You have another version of autoconf. It may work, but is not guaranteed to. If you have problems, you may need to regenerate the build system entirely. To do so, use the procedure documented by the package, typically `autoreconf'. configure.in:533: error: AC_PROG_CC cannot be called after AM_PROG_CC_C_O configure.in:533: the top level autom4te: error: /tools_x86_64/bin/m4 failed with exit status: 1 make[1]: *** [Makefile:375: configure] Error 1 make[1]: Leaving directory '/usr/src/collectd-4.10.9' make: *** [collectd:117: /usr/src/log/collectd-4.10.9] Error 2
> I can’t remember what has stopped us from moving with upstream apart from the RRD file format. If this all doesn’t work, maybe we should look at this all again and see if we can’t migrate to the latest upstream version again.
As far as I can remember it was mentioned that the issue was the changed format and RRD being used in so many places making the update require changes in so many places in IPFire-2.x
Regards,
Adolf.
> Best, > -Michael >> On 19 Nov 2024, at 13:55, Adolf Belka adolf.belka@ipfire.org wrote: >> Hi Michael, >> >> Also reran the collectd autoreconf with the autoconf-2.71 and in that case the line has yes:) so something about the autoconf-2.72 is causing that one yes: in the configure file to be missing the ) . There are something like 20 of those yes: entries and with the autoconf-2.72 only the one I show below is the one with a missing ), all the rest have a yes:) entry. >> >> Regards, >> >> Adolf. >> >>> On 19/11/2024 14:43, Adolf Belka wrote: >>> Hi Michael, >>> >>> I obtained the configure created by the autoreconf etc and I have found the error. Here is the affected line 18710 >>> >>> >>> 18706 if test "x$ac_ct_CC" = x; then >>> 18707 CC="" >>> 18708 else >>> 18709 case $cross_compiling:$ac_tool_warned in >>> 18710 yes: >>> 18711 >>> 18712 { printf "%s\n" "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 >>> 18713 printf "%s\n" "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} >>> 18714 ac_tool_warned=yes ;; >>> 18715 esac >>> 18716 CC=$ac_ct_CC >>> 18717 fi >>> 18718 else >>> 18719 CC="$ac_cv_prog_CC" >>> 18720 fi >>> >>> That line should be yes:) so it is missing a right bracket. Not sure if you can figure where and why that is occurring but presumably a patch can be written to be applied after the autoreconf etc and before the ./configure command >>> >>> Regards, >>> Adolf. >>> >>> >>>> On 19/11/2024 12:41, Michael Tremer wrote: >>>> Hello Adolf, >>>> In the build script we are running autoreconf which will regenerate the configure script: >>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/collectd;h=d1d4ea72138... >>>> Is there any output of that command? There should be some warnings which might help us to find out what we need to change. >>>>> On 13 Nov 2024, at 10:30, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> Hi All, >>>>> I did an update run with autoconf taking it from 2.71 to 2.72 >>>>> autoconf built without any problems, however collectd then failed for a syntax error in the configure file. >>>>> Confirmed that this was due to autoconf by changing the versions back and forward and the problem went away and then came back again. >>>>> The collectd build error message is >>>>> checking for size_t... yes >>>>> checking for uid_t... yes >>>>> checking for gid_t... yes >>>>> ./configure: line 18710: syntax error near unexpected token `newline' >>>>> ./configure: line 18710: `yes:' >>>>> make: *** [collectd:120: /usr/src/log/collectd-4.10.9] Error 2 >>>>> The section around line 18710 in the configure file is >>>>> 18707 do : >>>>> 18708 as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh` >>>>> 18709 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default" >>>>> 18710 if eval test "x$"$as_ac_Header"" = x"yes"; then : >>>> There is an extra : here which is probably what causes the syntax error. >>>>> 18711 cat >>confdefs.h <<_ACEOF >>>>> 18712 #define `$as_echo "HAVE_$ac_header" | $as_tr_cpp` 1 >>>>> 18713 _ACEOF >>>>> 18714 >>>>> 18715 else >>>>> 18716 with_libiptc="no (header file missing)" >>>>> 18717 fi >>>>> However there are 15 occurrences of the exact same text as line 18710 in the configure file so I am not convinced if that line is the root cause for the syntax error but some other earlier error that causes a knock-on effect. >>>>> However, my knowledge of the coding syntax is definitely not enough to figure out what needs to be fixed/changed. >>>>> As collectd is a very old version it is likely that some structural coding or syntax was fine in the past but now with the change from autoconf-2.71 to 2.72 it is no longer allowed, or is flagged up when in the past it was just ignored. >>>>> There are quite a few changes in autoconf-2.72 with some being marked as backwards compatibilities. All of them except one say that existing configure scripts will continue working. The one that doesn't mention that is the following change:- >>>>> Configure scripts no longer support pre-1989 C compilers. >>>>> Specifically, compilers that *only* implement the original “K&R” >>>>> function definition syntax, and not the newer “prototyped” syntax, >>>>> will not be able to parse the test programs now emitted by >>>>> AC_CHECK_FUNC, AC_LANG_CALL, and similar macros. AC_PROG_CC still >>>>> accepts such compilers, but this may change in the near future. >>>>> This change was necessary in order to support the upcoming 2024 >>>>> edition of the C standard (often referred to as “C23”), which will >>>>> officially remove the function declaration syntax used by >>>>> AC_CHECK_FUNC in Autoconf 2.71 and earlier. We feel that support >>>>> for compilers that support only C 2024 is more useful, nowadays, >>>>> than support for compilers that don’t implement a core feature of >>>>> C 1989. >>>>> However I am unable to figure out from this if the problem I am experiencing is related to this or not. I would not have thought so as I don't believe we are using a pre-1989 C compiler. >>>>> Any ideas from anyone on how to fix this issue? >>>>> There is nothing critical from a security or other vulnerability aspect in autoconf-2.72 but it would be nice to figure this out before we get to a stage where it has to be made to work. >>>>> Regards, >>>>> Adolf.
-- Sent from my laptop