Re: Packaging different Zabbix agent versions

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 12589 bytes --]

Hello,

> On 2 Nov 2024, at 14:30, Robin Roevens <robin.roevens(a)disroot.org> wrote:
> 
> Hi Michael
> 
> Michael Tremer schreef op wo 30-10-2024 om 09:54 [+0000]:
>> Hello Robin,
>> 
>> This is an interesting question to which I don’t have a very good
>> answer.
>> 
>>> On 26 Oct 2024, at 23:08, Robin Roevens <robin.roevens(a)disroot.org>
>>> wrote:
>>> 
>>> Hi all
>>> 
>>> I would like to start releasing packages for Zabbix Agent 7.0 (new
>>> LTS)
>>> as it offers some added value when used with Zabbix server 7.0
>>> (heartbeat, item-level custom timeouts, ...).
>>> However, the agent is not backwards compatible. Hence, if I would
>>> update the zabbix_agentd package to contain v7.0, people still
>>> running
>>> Zabbix 6.0 LTS (still supported until feb. 2027) or even 6.2 or 6.4
>>> would then no longer be able to monitor IPFire.
>> 
>> I consider this quite bad software design since it is causing some
>> issues with their user base and creates a lot more work for
>> distributions to package two of the same software. We also ready have
>> a lot of different monitoring tools and occasionally people request
>> more, so I assume that each of them have already only a small user
>> base as they divide the market - and now having two versions of
>> Zabbix is even further splitting the Zabbix user base so that I am
>> sure that for each version we will only have very few users, but
>> still a lot of work to package it.
>> 
>> This is just for the record. It would be nicer if upstream would give
>> us better migration paths.
>> 
> 
> I understand the standpoint from distro-side but Zabbix on the other
> hand is a commercial entity that needs to make a living out of  the
> product, but (luckily) is and wants to remain a true open-source
> company. So their sole income comes from selling support (and services)
> on the product; so from their standpoint it is beneficial to keep
> supporting a few versions from the past.

Oh yeah that is absolutely true and we are doing exactly the same here, too.

> They release a LTS version every 1.5 years and support that version for
> 5 years, so in theory currently also v5.0 is still supported and there
> will always be 3 LTS versions that are supported at the same time.
> (https://www.zabbix.com/life_cycle_and_release_policy)

I don’t know if this is reasonable or not because I am not a big Zabbix user, but why not?

> To make things even more complicated, there is also a zabbix_agent2,
> which is also a Zabbix Agent with the same capabilities but written in
> Go instead of in C. They live next to each other (and Zabbix claims
> this will remain as such) with the Go version having a bit more
> features due to the availability of additional Go extensions for it. I
> already decided in past not to make packages of agent2 for IPFire for
> now, as it has a larger footprint both in size and in resources when
> running. And in my opinion, it provides too little of added value for
> IPFire. Despite the fact that I have already received the question in
> the past (https://github.com/RobinR1/zbx-template-ipfire/issues/3)

Interesting. I personally don’t believe that Go is making it very far for low-level services in operating systems. I think we ship one piece of Go software and that requires us to ship the entire compiler which cannot easily be bootstrapped. So I am also happier with the C version.

> I have meanwhile checked a few other distro's and it seems they
> currently ship only 6.0.x (LTS) versions of the agent and agent2 by
> default. But Zabbix provides it's own repositories for all major
> distro's with all supported versions. Additionally there are a few
> other community builds of different Zabbix versions for different
> distro's out there.. So in the end the user has many choices.
> 
> For IPFire there are no additional package sources nor Zabbix repo's to
> obtain different versions from. So if we want our users to have a
> choice, we need to provide it ourselves.

Indeed.

> 
>>> So it would be nice if we could provide both zabbix_agentd 6.0 and
>>> 7.0
>>> packages so users can choose depending on their Zabbix server
>>> version.
>>> I think offering different versions of the same software is not yet
>>> done before?
>> 
>> Luckily this was not necessary, or in case of Icinga/Nagios there was
>> a fork that renamed the package so that there were no file conflicts.
>> 
>> Pakfire in IPFire 2 was never supposed to be a full-blown package
>> management system and therefore it does not support conflicts like
>> Pakfire in IPFire 3 does. There, we would simply configure that those
>> packages cannot be installed at the same time and the package
>> management system will take care of it.
>> 
>>> How should I do this? Just add a new package zabbix_agentd7 next to
>>> the
>>> current zabbix_agentd package?
>> 
>> There is a way… but before, maybe it is worth to conduct a little
>> survey on community.ipfire.org <http://community.ipfire.org/> to see
>> if there are actually any users who have a preference either way? I
>> don’t know much about Zabbix so I am not sure how fast people are
>> with upgrading (I assume a single device in your infrastructure that
>> does not support version 7 will block you from moving everything
>> forward?).
> 
> Luckily that is not the fact. The server keeps supporting older agents.
> Currently Server 7.0 still supports agents v1.4 up to 7.0. Downside of-
> course is that older agents will miss newer features and one of those
> newer features currently is rather visible: Agent 7 sends heartbeats to
> the server which gives a green availability icon for that agent on the
> Server 7 interface. The agent we currently ship doesn't do that yet and
> shows up with an 'unknown' status on a Server 7, for which I also
> already had a user question:
> https://community.ipfire.org/t/zabbix-template-for-ipfire/3522/8?u=robinr1
> 
> The main reason however I myself am so eager to have an Agent 7 package
> is that agent 7 supports different timeout settings for each monitored
> metric instead of only a global timeout setting. As I'm running IPFire
> on a Lightning Wire Mini appliance, I bump on timeouts for some checks
> that on a full blown pc won't have problems, but I wouldn't want to
> increase the global timeout.
> 
> Not providing a package for agent 7 on IPFire would not keep anyone
> from upgrading it's monitoring infrastructure. The other way around
> however, if we would only supply a v7 of the agent, people with an
> older Zabbix server will no longer be able to monitor their IPFire
> instance. So we have to be careful stopping support of older versions
> of the agent.
> 
> I do assume home users probably upgrade fairly quick, but enterprises
> tend to be slower in their upgrade paths. The institution I work for is
> currently still on 6.4 and I know of (big) companies even still running
> 4.0. But I assume that kind of companies are probably not IPFire users
> (and v4 would already be done for anyways..). And as we never had
> complaint about v5 not being available for IPFire, I think we can
> currently assume all our users that also use Zabbix are at least on v6.

Okay, then let’s ship both.

>> 
>>> But if so:
>>> - Can we then rename the zabbix_agentd pak to zabbix_agentd6 (I
>>> don't
>>> immediately see how that should be handeled during upgrade?).
>>> If not possible or feasable, that's not that big of a deal as in
>>> time
>>> (feb 2027) that package can be removed and that "problem" then has
>>> solved itself. So this is maybe just some unneeded hassle.
>> 
>> We could rename it, that might make the whole thing a little bit
>> cleaner.
>> 
>> Either you can add the major release number or simply call it zabbix-
>> agent-legacy? Whatever you fell is most appropriate.
> 
> I would go with a major version number then, as 'legacy' is within the
> Zabbix community sometimes also used to refer to the C version of the
> Zabbix agent opposed to the Go Zabbix agent2 version (which we don't
> provide).

Okay. I didn’t know that.

> 
>> 
>> This change would have to be done in the core update though.
>> 
>>> - Can we prevent users from installing both packages as they will
>>> conflict? (both file-wise and used port-wise)
>> 
>> Hmm, that will be a problem. Pakfire does not support conflicts. We
>> can either add this (because to simply conflict with a name this
>> might not be too hard to do), or another version would be to actually
>> build everything in such a way that there are no real conflicts.
>> Rename /usr/sbin/zabbix_agentd to /usr/sbin/zabbix_agentd-7 and so
>> on.
> 
> Building the package so that there is no real conflict is doable, but
> as the package would automatically (re)start the agent, it would still
> try to map the same tcp port. We could also start using a different
> port by default for each package but then we are deviating from a well
> known default that can only lead to confusion.

I would keep the default port. If people try to run both at the same time they are on their own.

> 
>> 
>> There are install scripts that we have used before to stop the
>> installation process, but those are being called *after* the files
>> have been extracted so the formerly installed version will already
>> have been overwritten.
> 
> That would indeed not really help. I was thinking about detecting if
> the other version is installed, if so, removing the old version in
> favor of the latest installed version. But also here, an uninstall of
> the old version would then probably occur *after* the install of the
> new version, removing files from the new version.
> 
>> 
>>> - And what should I do with the IPFire specific customizations as
>>> they
>>> will be identical for both packages.:
>>>  - Just duplicate them to both packages, (probably the easiest)
>>>  - or move them to yet another package
>> 
>> Good question. It might be easiest to just copy things. Because then
>> we can just drop the older version whenever time is right and don’t
>> have to merge packages together again. I hope this break is a one-off
>> and version 8 won’t do this again.
> 
> As I said earlier, there will probably always be 3 LTS versions that
> are supported at the same time. Currently we already don't provide a
> v5.0 package which is technically also still supported. But we didn't
> have complaints :-) But in theory this break could be permanent thing.

Okay.

>> 
>>> "zabbix_agentd_ipfire_extension".. And then let both agent packages
>>> depend on that one. But that may be confusing to the user as that
>>> package would require one of the then 2 zabbix_agentd packages to
>>> be of
>>> any use, but we can't do a zabbix_agentd OR zabbix_agentd7
>>> dependency,
>>> I think?
>> 
>> I would not be against it if you prefer this option. Technically it
>> will work.
>> 
>>> ...
>>> 
>>> Or should we just refrain from adding v7.0 for now.. and then
>>> probably
>>> until 2027 or until ipfire3, (if pakfire there supports such
>>> packages
>>> conflict/dependency mechanisms?).. but by then there probably is
>>> already an 8.0 and possibly a 9.0..
>> 
>> Good question, maybe this is worth to investigate with the community
>> if there is demand for version 7 already. I don’t know how fast
>> people will upgrade. Having said my thoughts from earlier, IPFire
>> should not be the blocker why people cannot upgrade their entire
>> infrastructure to version 7.
>> 
>> Is there any reason that justifies this big break?
> 
> As outlined above, my main reason currently is the item-level
> configurable timeouts in v7 that can improve monitoring on embedded
> devices with slower resources, but I will launch a community survey to
> check if there is a broader need for a version 7 already.
> I don't think Fireinfo currently has information about how many addon
> installs there are in our userbase ? 

No. We don’t track that and I am not sure what we would gain from having exact numbers. We already kind of know what the most popular add-ons are.

> Anyway, lets wait for community feedback, and then we can start
> thinking about the practical side again.

Okay, I will wait for you to take the lead on this :)

-Michael

> 
> Robin 
> 
>> 
>> -Michael
>> 
>>> So...
>>> 
>>> I would like your thoughts about this ?
>>> 
>>> Robin