Re: ARM 64?

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 8543 bytes --]

Hello,

On Fri, 2018-05-25 at 12:45 +1000, Mathew McBride wrote:
> Hello,
> 
> On 25/5/18, 12:44 am, "Michael Tremer" <michael.tremer(a)ipfire.org> wrote:
> 
>     Hey,
>     
>     IPFire is not based on any other distribution like Ubuntu is (was?) based
> on
>     Debian. So we have to port the entire userspace as well as the kernel. Our
>     kernel is also highly customised because the needs of a firewall and
> secure and
>     hardened systems vary a lot from other server systems.
> Understood. I've had a look through the repositories - I take it only 3.x is
> planned to have armv8/arm64 support. Does the userspace build on arm64 at the
> moment?

Yes, IPFire 3 builds on ARM64. If it is running outside of a VM is completely
unknown. I was really passionate about this, but since we had no hardware that
was good enough to develop IPFire towards, this is more of a proof-of-concept
than anything else.

IPFire 3 is really hard to port because everything is a package and there is no
easy way how to bootstrap the distribution starting with no package.

IPFire 2 can only be built in one go. Nothing is a package. Every change
requires us to recompile the *entire* distribution. That takes a long time (~9
hours without ccache and about 4-5 with ccache).

>     We also put a huge emphasis on performance. Networks need to be fast.
>     
>     There isn't too much great build hardware out there for us right now. We
> have
>     some smaller system that I can use to play around a bit for a start in a
> data
>     center. The Mustang system I have doesn't run too well because there is a
> huge
>     hardware bug in the SATA controller.
>     
> There are a couple of options out there, NXP has a Cortex-A72 based SoC's (and
> devboards) that would be better suited to build tasks. I'm not sure how it
> stacks up with X-Gene/Mustang but the marketing claims are that A72 (when
> given a good amount of L2/L3 cache) can hold up against Xeon D's in single
> thread.

Oh yeah, they are awesome. So fast and power consumption is quite okay. I got
the first generation XGene and read that any newer generations are more
efficient.

> I'm hoping ThunderX2 (Cavium) hosts start appearing on cloud providers soon as
> well. There are ThunderX (v1) instances available, but that core had very poor
> single thread performance for a server.

With IPFire 3 we can distribute the build process really well. So it wouldn't
matter to have a few build instances more as long as that is financially
feasible.

> I do kernel and other builds (Funtoo/Gentoo) all the time on our hardware; but
> this is more for 'dogfooding'/load testing purposes! 

I think you hardware is good enough for a builder. But I still am not sure what
to expect from the CPU. It will be faster than a Raspberry Pi, but not a
Mustang.

> 
>     >     > If you build an image that uses UEFI boot it will work on both our
>     > hardware
>     >     > (bare metal), as a VM and on the ARM64 server platforms
> (Ampere/XGene,
>     >     > Centriq, ThunderX), or even our competitors (MacchiatoBin etc.)
>     >     
>     >     and the OS is on SSD or SD card?
>     > Yes, u-boot is on the NAND. It will boot EFI distributions on any
> supported
>     > block device (SD, USB, SSD).
>     > There are some limitations - no EFI persistent variables etc. (yet) or
> RTC
>     > service, but I think that will eventually be solved.
>     
>     In this hardware or in a future one?
>  
> On this hardware. For example, to implement the EFI variable service, we will
> probably have to make the NAND inaccessible to Linux in EFI-mode. 

That would be fine with me. We have no use for the NAND for the OS.

> (If you package your images to work on 'removable media', as required for ARM
> standard VM's, you probably don't need efivars anyway).
> With the real time clock - we have a discrete RTC chip on our board, whereas
> the major distributions expect a 'platform' RTC (like x86). Not an issue for
> embedded distributions but having the EFI RTC will solve system clock issues
> on the major distros.
> 
>    
>     > 'Real' ARM servers (these days) also use ACPI instead of passing a
> device
>     > tree, but this doesn't really make much of a difference in user space.
>     
>     Not to userspace but to our kernel.
>     
>     We don't support EFI yet on x86 because there was never any demand for it.
>     Servers are still happy with the legacy mode and people don't seem to
> trust the
>     bigger EFI blobs that are running on modern systems.
>     
>     I suppose this hardware comes without a management engine and all of that
> stuff?
>     
> Definitely. No hidden blobs here :)

*thumbs up*

So we would have a huge advantage in the security of the hardware. It looks like
the system isn't even vulnerable for (most of?) Spectre.

> There is a TrustZone firmware running in the ring/EL above the OS, for the NXP
> Layerscape/QorIQ SoC's this firmware is open source, and not strictly required
> to run the system (it gets loaded by u-boot after power on).

What does the firmware do?

> It is, however, required to do certain functions that the ARM server standard
> expects (such as moving between ring/ELs) - EFI boot does not work without it.

-Michael

> 
>     Best,
>     -Michael
> 
> 
> Cheers,
> Matt
>     > Matt
>     >     > 
>     >     > -------- Forwarded Message -------- 
>     >     > Subject: 
>     >     > Re: ARM 64?
>     >     > Date: 
>     >     > Wed, 23 May 2018 10:58:00 +0100
>     >     > From: 
>     >     > Michael Tremer mailto:michael.tremer(a)ipfire.org
>     >     > To: 
>     >     > mailto:guy(a)traverse.com.au, mailto:development(a)lists.ipfire.org
>     >     > 
>     >     > Hello Guy,
>     >     > 
>     >     > thank you very much for getting in touch.
>     >     > 
>     >     > Yes, we are working on an ARM 64 port, but so far we have not seen
> any
>     >     > hardware
>     >     > that was worth doing the port for. All those small and cheap
> single-
>     > board
>     >     > computers lack power, the bigger systems are basically unavailable
> and
>     > way too
>     >     > expensive.
>     >     > 
>     >     > This board is way different though. CPU, Memory and especially the
> NICs
>     > are
>     >     > something that are way better sized and make a nice small
> appliance for
>     > bigger
>     >     > SOHOs or small to medium-sized offices.
>     >     > 
>     >     > Not entirely sure why there is only one 10G port. Usually where
> 10G goes
>     > in,
>     >     > it
>     >     > has to go out somewhere else again...
>     >     > 
>     >     > The big question is what software support is like in the Linux
> kernel
>     > for
>     >     > this.
>     >     > I have seen the patches linked on the product page and can only
> defer to
>     > Arne
>     >     > to
>     >     > have a look at it.
>     >     > 
>     >     > My question would now be: What is the desired RRP for this or
> wholesale?
>     > I am
>     >     > just curious to find out if it is competitive or if people would
> find it
>     > too
>     >     > expensive and buy Intel again. Then we shouldn't really bother
> with
>     > putting
>     >     > the
>     >     > time into a port. If you are not comfortable sharing prices on the
> list,
>     >     > please
>     >     > feel free to email me in private about this.
>     >     > 
>     >     > About the sponsorship. Thank you very much for considering us. I
> would
>     > be
>     >     > happy
>     >     > to have a closer look. Would you be able to ship this into the UK
> or
>     > Germany?
>     >     > 
>     >     > Best,
>     >     > -Michael
>     >     > 
>     >     > On Wed, 2018-05-23 at 12:46 +1000, Guy Ellis wrote:
>     >     > > Dear list,
>     >     > > 
>     >     > > Just wondering if there is any interest in supporting ARM 64
> hardware 
>     >     > > moving forward?
>     >     > > 
>     >     > > We can assist with donated hardware and support if there is
> interest.
>     >     > > https://traverse.com.au/products/ls1043s-router-board/
>     >     > > 
>     >     > > Regards,
>     >     > >   - Guy.
>     >     > > 
>     >     > > 
>     >     > 
>     >     > 
>     >     > 
>     >     
>     > 
>     > 
>     
> 
>