Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 2414 bytes --]

Hello *,

a quick heads-up on reports on the oss-security mailing list that indicate the upstream
tarball of xz containing a backdoor since version 5.6.0, with the target objective appearing
to constitute in backdooring SSH: https://openwall.com/lists/oss-security/2024/03/29/4

Please note that this is a developing situation, so take the assessments below with a
pinch of salt.

- The latest stable version of IPFire, IPFire 2.29 - Core Update 184, is NOT affected by
  the backdoor discussed in the oss-security post linked above. This is because it includes
  xz 5.4.6 (as mentioned in https://www.ipfire.org/blog/ipfire-2-29-core-update-184-released).
  Further, since IPFire does NOT patch OpenSSH in order to include lzma compression (which
  is a requirement for the unveiled backdoor to work), my understanding at this time is that
  OpenSSH on stable IPFire installations is not affected.

  This is further corroborated by the backdoor known so far only becoming active under
  certain build environment conditions that are not met by IPFire 2.x's build environment.

  However, it currently appears as if the xz developer has actively worked towards including
  a backdoor, rather than their account having been compromised. Therefore, it may be that
  there are other backdoors in the xz upstream tarball, and that they have been included in
  earlier versions.

- Forthcoming Core Update 185 includes two patches that update xz to 5.6.0 and 5.6.1,
  respectively. These versions are known to include the aforementioned OpenSSH backdoor.
  The IPFire development team will discuss reversion of xz to a version not known to be
  affected thus far in the next few days. Currently, both Debian and Fedora opted to
  revert back to version 5.4.5, rather than 5.4.6 (which is what IPFire currently ships
  in stable Core Update 184, but is not known to include any malicious code, which only
  commenced in version 5.6.0).

  Again, since no custom patching of OpenSSH is in place, the unveiled SSH backdoor would
  not have been functional on IPFire installations.

IPFire is currently unaware of the unveiled backdoor impacting any other service that is
usually directly exposed on IPFire installations to the internet, such as OpenVPN or IPsec.

For reference, CVE-2024-3094 has been assigned by Red Hat for this issue.

Thanks, and best regards,
Peter Müller

Re: Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3621 bytes --]

Hello,

> On 29 Mar 2024, at 21:53, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello *,
> 
> a quick heads-up on reports on the oss-security mailing list that indicate the upstream
> tarball of xz containing a backdoor since version 5.6.0, with the target objective appearing
> to constitute in backdooring SSH: https://openwall.com/lists/oss-security/2024/03/29/4
> 
> Please note that this is a developing situation, so take the assessments below with a
> pinch of salt.
> 
> - The latest stable version of IPFire, IPFire 2.29 - Core Update 184, is NOT affected by
>  the backdoor discussed in the oss-security post linked above. This is because it includes
>  xz 5.4.6 (as mentioned in https://www.ipfire.org/blog/ipfire-2-29-core-update-184-released).
>  Further, since IPFire does NOT patch OpenSSH in order to include lzma compression (which
>  is a requirement for the unveiled backdoor to work), my understanding at this time is that
>  OpenSSH on stable IPFire installations is not affected.

I agree with this assessment.

I believe that IPFire does not meat a number of criteria that are required for this backdoor to be usage:

* The build is checking whether a Debian/RPM package is being built and only then injects the backdoor code. This should not be the case on IPFire and therefore I believe that we don’t even include the malicious code paths.

* OpenSSH is not directly linked against liblzma and we do not have anything else (e.g. that pulls it in). So the library is not loaded into sshd at runtime, so even if liblzma was affected on IPFire, we should not have a compromised SSH service.

>  This is further corroborated by the backdoor known so far only becoming active under
>  certain build environment conditions that are not met by IPFire 2.x's build environment.
> 
>  However, it currently appears as if the xz developer has actively worked towards including
>  a backdoor, rather than their account having been compromised. Therefore, it may be that
>  there are other backdoors in the xz upstream tarball, and that they have been included in
>  earlier versions.
> 
> - Forthcoming Core Update 185 includes two patches that update xz to 5.6.0 and 5.6.1,
>  respectively. These versions are known to include the aforementioned OpenSSH backdoor.
>  The IPFire development team will discuss reversion of xz to a version not known to be
>  affected thus far in the next few days. Currently, both Debian and Fedora opted to
>  revert back to version 5.4.5, rather than 5.4.6 (which is what IPFire currently ships
>  in stable Core Update 184, but is not known to include any malicious code, which only
>  commenced in version 5.6.0).

Adolf has been really quick in providing a patch to downgrade back to 5.4.5. As far as I can see this is what Debian is doing and believing that they have some more information about everything I would like to follow their decisions. I have currently no reason to believe that 5.4.6 is compromised.

The patch has been merged and I have included everything that is linked against XZ in the updater as a precaution.

>  Again, since no custom patching of OpenSSH is in place, the unveiled SSH backdoor would
>  not have been functional on IPFire installations.
> 
> IPFire is currently unaware of the unveiled backdoor impacting any other service that is
> usually directly exposed on IPFire installations to the internet, such as OpenVPN or IPsec.
> 
> For reference, CVE-2024-3094 has been assigned by Red Hat for this issue.
> 
> Thanks, and best regards,
> Peter Müller

-Michael

Re: Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4717 bytes --]

Hi All,

The xz github repository has been disabled by github.

Therefore effectively xz will now stay at version 5.4.x

Likely that any new xz repository that gets set up to do future bug and 
security fixes is going to be looked at suspiciously from now on, 
assuming they can get a repo accepted.

Therefore I am wondering if we need to put into process a plan to change 
all packages that currently use xz archiving in the lfs etc to use gz or 
bz2.

When that is completed, if xz is not used for anything else within 
IPFire other than archiving/de-archiving files, then we could remove xz 
completely.

I am just thinking that if we don't do this then if a security bug is 
discovered in xz (not from deliberate implementation but just a 
historical bug) there won't be anyone updating xz to fix it and even if 
they did would people use it?

Is this a reasonable thought or am I being to paranoid.

Regards,

Adolf.

On 30/03/2024 13:37, Michael Tremer wrote:
> Hello,
>
>> On 29 Mar 2024, at 21:53, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello *,
>>
>> a quick heads-up on reports on the oss-security mailing list that indicate the upstream
>> tarball of xz containing a backdoor since version 5.6.0, with the target objective appearing
>> to constitute in backdooring SSH: https://openwall.com/lists/oss-security/2024/03/29/4
>>
>> Please note that this is a developing situation, so take the assessments below with a
>> pinch of salt.
>>
>> - The latest stable version of IPFire, IPFire 2.29 - Core Update 184, is NOT affected by
>>   the backdoor discussed in the oss-security post linked above. This is because it includes
>>   xz 5.4.6 (as mentioned in https://www.ipfire.org/blog/ipfire-2-29-core-update-184-released).
>>   Further, since IPFire does NOT patch OpenSSH in order to include lzma compression (which
>>   is a requirement for the unveiled backdoor to work), my understanding at this time is that
>>   OpenSSH on stable IPFire installations is not affected.
> I agree with this assessment.
>
> I believe that IPFire does not meat a number of criteria that are required for this backdoor to be usage:
>
> * The build is checking whether a Debian/RPM package is being built and only then injects the backdoor code. This should not be the case on IPFire and therefore I believe that we don’t even include the malicious code paths.
>
> * OpenSSH is not directly linked against liblzma and we do not have anything else (e.g. that pulls it in). So the library is not loaded into sshd at runtime, so even if liblzma was affected on IPFire, we should not have a compromised SSH service.
>
>>   This is further corroborated by the backdoor known so far only becoming active under
>>   certain build environment conditions that are not met by IPFire 2.x's build environment.
>>
>>   However, it currently appears as if the xz developer has actively worked towards including
>>   a backdoor, rather than their account having been compromised. Therefore, it may be that
>>   there are other backdoors in the xz upstream tarball, and that they have been included in
>>   earlier versions.
>>
>> - Forthcoming Core Update 185 includes two patches that update xz to 5.6.0 and 5.6.1,
>>   respectively. These versions are known to include the aforementioned OpenSSH backdoor.
>>   The IPFire development team will discuss reversion of xz to a version not known to be
>>   affected thus far in the next few days. Currently, both Debian and Fedora opted to
>>   revert back to version 5.4.5, rather than 5.4.6 (which is what IPFire currently ships
>>   in stable Core Update 184, but is not known to include any malicious code, which only
>>   commenced in version 5.6.0).
> Adolf has been really quick in providing a patch to downgrade back to 5.4.5. As far as I can see this is what Debian is doing and believing that they have some more information about everything I would like to follow their decisions. I have currently no reason to believe that 5.4.6 is compromised.
>
> The patch has been merged and I have included everything that is linked against XZ in the updater as a precaution.
>
>>   Again, since no custom patching of OpenSSH is in place, the unveiled SSH backdoor would
>>   not have been functional on IPFire installations.
>>
>> IPFire is currently unaware of the unveiled backdoor impacting any other service that is
>> usually directly exposed on IPFire installations to the internet, such as OpenVPN or IPsec.
>>
>> For reference, CVE-2024-3094 has been assigned by Red Hat for this issue.
>>
>> Thanks, and best regards,
>> Peter Müller
> -Michael

-- 
Sent from my laptop

Re: Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7012 bytes --]

Hello,

> On 30 Mar 2024, at 13:30, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> The xz github repository has been disabled by github.
> 
> Therefore effectively xz will now stay at version 5.4.x
> 
> Likely that any new xz repository that gets set up to do future bug and security fixes is going to be looked at suspiciously from now on, assuming they can get a repo accepted.
> 
> Therefore I am wondering if we need to put into process a plan to change all packages that currently use xz archiving in the lfs etc to use gz or bz2.

I had the same thought. But only a thought. I am not entirely sure what would actually change if we would do this step. I would also like to (re-)consider if us publishing any archives compressed using XZ is the right way to go now.

On the list of pro (XZ):

* XZ is widely used
* It has a great compression rate
* Technically the format has not been compromised, just one implementation of a compressor/decompressor. The backdoor did not even target the generated or extracted files which it could have done as well compromising any archived data, deliberately corrupting it… I don’t know what bad people like doing all day.

On the cons are:

* https://lzip.nongnu.org/xz_inadequate.html - I find these arguments quite convincing, however, other formats have their weaknesses, too.
* If we would go back to another well-known format (of which there is only Gzip), we would ship way larger files.
* If we would go to a format that has similar properties (LZIP or Zstandard), we face issues with usability because decoders might not be so easily available. No idea what Windows supports these days.

> When that is completed, if xz is not used for anything else within IPFire other than archiving/de-archiving files, then we could remove xz completely.
> 
> I am just thinking that if we don't do this then if a security bug is discovered in xz (not from deliberate implementation but just a historical bug) there won't be anyone updating xz to fix it and even if they did would people use it?
> 
> Is this a reasonable thought or am I being to paranoid.

What is wrong with a bit of healthy paranoia? I think it might be better to be more careful when there is no significant overhead in labour for us.

So I was going to do the following when updating a package or packaging something new:

* For projects that publish their tarballs compressed using other formats, I would prefer those over XZ. We don’t *really* care how large the downloaded source is. A couple of extra bytes don’t hurt.
* The order would be still to prefer what compresses better: Zstandard, LZIP, Gzip.

But generally we cannot influence what people use to publish their tarballs as. So we will have a large portion of files compressed using XZ and there is no reason to currently believe that they are compromised through this particular backdoor. We might in the end actually have a lot of people taking the XZ code apart which should bring us very close to certainty that there is nothing else hidden. Let’s hope we will get this for other formats, too.

-Michael

> Regards,
> 
> Adolf.
> 
> On 30/03/2024 13:37, Michael Tremer wrote:
>> Hello,
>> 
>>> On 29 Mar 2024, at 21:53, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>> 
>>> Hello *,
>>> 
>>> a quick heads-up on reports on the oss-security mailing list that indicate the upstream
>>> tarball of xz containing a backdoor since version 5.6.0, with the target objective appearing
>>> to constitute in backdooring SSH: https://openwall.com/lists/oss-security/2024/03/29/4
>>> 
>>> Please note that this is a developing situation, so take the assessments below with a
>>> pinch of salt.
>>> 
>>> - The latest stable version of IPFire, IPFire 2.29 - Core Update 184, is NOT affected by
>>>  the backdoor discussed in the oss-security post linked above. This is because it includes
>>>  xz 5.4.6 (as mentioned in https://www.ipfire.org/blog/ipfire-2-29-core-update-184-released).
>>>  Further, since IPFire does NOT patch OpenSSH in order to include lzma compression (which
>>>  is a requirement for the unveiled backdoor to work), my understanding at this time is that
>>>  OpenSSH on stable IPFire installations is not affected.
>> I agree with this assessment.
>> 
>> I believe that IPFire does not meat a number of criteria that are required for this backdoor to be usage:
>> 
>> * The build is checking whether a Debian/RPM package is being built and only then injects the backdoor code. This should not be the case on IPFire and therefore I believe that we don’t even include the malicious code paths.
>> 
>> * OpenSSH is not directly linked against liblzma and we do not have anything else (e.g. that pulls it in). So the library is not loaded into sshd at runtime, so even if liblzma was affected on IPFire, we should not have a compromised SSH service.
>> 
>>>  This is further corroborated by the backdoor known so far only becoming active under
>>>  certain build environment conditions that are not met by IPFire 2.x's build environment.
>>> 
>>>  However, it currently appears as if the xz developer has actively worked towards including
>>>  a backdoor, rather than their account having been compromised. Therefore, it may be that
>>>  there are other backdoors in the xz upstream tarball, and that they have been included in
>>>  earlier versions.
>>> 
>>> - Forthcoming Core Update 185 includes two patches that update xz to 5.6.0 and 5.6.1,
>>>  respectively. These versions are known to include the aforementioned OpenSSH backdoor.
>>>  The IPFire development team will discuss reversion of xz to a version not known to be
>>>  affected thus far in the next few days. Currently, both Debian and Fedora opted to
>>>  revert back to version 5.4.5, rather than 5.4.6 (which is what IPFire currently ships
>>>  in stable Core Update 184, but is not known to include any malicious code, which only
>>>  commenced in version 5.6.0).
>> Adolf has been really quick in providing a patch to downgrade back to 5.4.5. As far as I can see this is what Debian is doing and believing that they have some more information about everything I would like to follow their decisions. I have currently no reason to believe that 5.4.6 is compromised.
>> 
>> The patch has been merged and I have included everything that is linked against XZ in the updater as a precaution.
>> 
>>>  Again, since no custom patching of OpenSSH is in place, the unveiled SSH backdoor would
>>>  not have been functional on IPFire installations.
>>> 
>>> IPFire is currently unaware of the unveiled backdoor impacting any other service that is
>>> usually directly exposed on IPFire installations to the internet, such as OpenVPN or IPsec.
>>> 
>>> For reference, CVE-2024-3094 has been assigned by Red Hat for this issue.
>>> 
>>> Thanks, and best regards,
>>> Peter Müller
>> -Michael
> 
> -- 
> Sent from my laptop
> 

Re: Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 2858 bytes --]

Hi,

Jm2c:

I'm working on an update (httpd 2.5.59) and just saw that the "backdoor
versions" of 'xz 5.6.0/5.6.1' are still available on ipfire.org
(/pub/sources/source-2.x).

Would it not be advisable to delete these versions so that no mischief
can be done with them?

Just m2c...

Best
Matthias

On 29.03.2024 22:53, Peter Müller wrote:
> Hello *,
> 
> a quick heads-up on reports on the oss-security mailing list that indicate the upstream
> tarball of xz containing a backdoor since version 5.6.0, with the target objective appearing
> to constitute in backdooring SSH: https://openwall.com/lists/oss-security/2024/03/29/4
> 
> Please note that this is a developing situation, so take the assessments below with a
> pinch of salt.
> 
> - The latest stable version of IPFire, IPFire 2.29 - Core Update 184, is NOT affected by
>   the backdoor discussed in the oss-security post linked above. This is because it includes
>   xz 5.4.6 (as mentioned in https://www.ipfire.org/blog/ipfire-2-29-core-update-184-released).
>   Further, since IPFire does NOT patch OpenSSH in order to include lzma compression (which
>   is a requirement for the unveiled backdoor to work), my understanding at this time is that
>   OpenSSH on stable IPFire installations is not affected.
> 
>   This is further corroborated by the backdoor known so far only becoming active under
>   certain build environment conditions that are not met by IPFire 2.x's build environment.
> 
>   However, it currently appears as if the xz developer has actively worked towards including
>   a backdoor, rather than their account having been compromised. Therefore, it may be that
>   there are other backdoors in the xz upstream tarball, and that they have been included in
>   earlier versions.
> 
> - Forthcoming Core Update 185 includes two patches that update xz to 5.6.0 and 5.6.1,
>   respectively. These versions are known to include the aforementioned OpenSSH backdoor.
>   The IPFire development team will discuss reversion of xz to a version not known to be
>   affected thus far in the next few days. Currently, both Debian and Fedora opted to
>   revert back to version 5.4.5, rather than 5.4.6 (which is what IPFire currently ships
>   in stable Core Update 184, but is not known to include any malicious code, which only
>   commenced in version 5.6.0).
> 
>   Again, since no custom patching of OpenSSH is in place, the unveiled SSH backdoor would
>   not have been functional on IPFire installations.
> 
> IPFire is currently unaware of the unveiled backdoor impacting any other service that is
> usually directly exposed on IPFire installations to the internet, such as OpenVPN or IPsec.
> 
> For reference, CVE-2024-3094 has been assigned by Red Hat for this issue.
> 
> Thanks, and best regards,
> Peter Müller

Re: Heads up: Backdoor in upstream xz tarball, stable version of IPFire likely unaffected, testing version somewhat affected

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]

Hello Matthias,

I think there is very little point in removing those as they will be mirrored all over the internet.

We are also mirroring the Git repository which contains some bad eggs, too.

Let’s keep this for history :)

-Michael

> On 5 Apr 2024, at 17:51, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> Jm2c:
> 
> I'm working on an update (httpd 2.5.59) and just saw that the "backdoor
> versions" of 'xz 5.6.0/5.6.1' are still available on ipfire.org
> (/pub/sources/source-2.x).
> 
> Would it not be advisable to delete these versions so that no mischief
> can be done with them?
> 
> Just m2c...
> 
> Best
> Matthias
> 
> On 29.03.2024 22:53, Peter Müller wrote:
>> Hello *,
>> 
>> a quick heads-up on reports on the oss-security mailing list that indicate the upstream
>> tarball of xz containing a backdoor since version 5.6.0, with the target objective appearing
>> to constitute in backdooring SSH: https://openwall.com/lists/oss-security/2024/03/29/4
>> 
>> Please note that this is a developing situation, so take the assessments below with a
>> pinch of salt.
>> 
>> - The latest stable version of IPFire, IPFire 2.29 - Core Update 184, is NOT affected by
>>  the backdoor discussed in the oss-security post linked above. This is because it includes
>>  xz 5.4.6 (as mentioned in https://www.ipfire.org/blog/ipfire-2-29-core-update-184-released).
>>  Further, since IPFire does NOT patch OpenSSH in order to include lzma compression (which
>>  is a requirement for the unveiled backdoor to work), my understanding at this time is that
>>  OpenSSH on stable IPFire installations is not affected.
>> 
>>  This is further corroborated by the backdoor known so far only becoming active under
>>  certain build environment conditions that are not met by IPFire 2.x's build environment.
>> 
>>  However, it currently appears as if the xz developer has actively worked towards including
>>  a backdoor, rather than their account having been compromised. Therefore, it may be that
>>  there are other backdoors in the xz upstream tarball, and that they have been included in
>>  earlier versions.
>> 
>> - Forthcoming Core Update 185 includes two patches that update xz to 5.6.0 and 5.6.1,
>>  respectively. These versions are known to include the aforementioned OpenSSH backdoor.
>>  The IPFire development team will discuss reversion of xz to a version not known to be
>>  affected thus far in the next few days. Currently, both Debian and Fedora opted to
>>  revert back to version 5.4.5, rather than 5.4.6 (which is what IPFire currently ships
>>  in stable Core Update 184, but is not known to include any malicious code, which only
>>  commenced in version 5.6.0).
>> 
>>  Again, since no custom patching of OpenSSH is in place, the unveiled SSH backdoor would
>>  not have been functional on IPFire installations.
>> 
>> IPFire is currently unaware of the unveiled backdoor impacting any other service that is
>> usually directly exposed on IPFire installations to the internet, such as OpenVPN or IPsec.
>> 
>> For reference, CVE-2024-3094 has been assigned by Red Hat for this issue.
>> 
>> Thanks, and best regards,
>> Peter Müller
>