Overfull Filesystems how to solve?

Overfull Filesystems how to solve?

[Jonatan Schlag]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

Hi,

Daniel encountered the problem that his root partition was written full
because of the qemu addon being too big (200 MB). See bug #12268 for
details.

So he suggested moving the path /usr/share/qemu to /var. I do not
really like this solution as there are standards how the filesystem
hierarchy works under Linux. 

Furthermore, as the systems become bigger (eg. moving python3 to the
core) this problem will become more relevant to us. So I would like to
start a discussion on how to solve this. Moving things to /var can only
be a temporary solution. Should we force a reinstallation, which solves
the root of the problem?

Jonatan

Re: Overfull Filesystems how to solve?

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]

On Sat, Jan 18, 2020 at 10:57:08AM +0100, Jonatan Schlag (jonatan.schlag(a)ipfire.org) wrote:
> 
> Hi,
> 
> Daniel encountered the problem that his root partition was written full
> because of the qemu addon being too big (200 MB). See bug #12268 for
> details.
> 
> So he suggested moving the path /usr/share/qemu to /var. I do not
> really like this solution as there are standards how the filesystem
> hierarchy works under Linux. 
> 
> Furthermore, as the systems become bigger (eg. moving python3 to the
> core) this problem will become more relevant to us. So I would like to
> start a discussion on how to solve this. Moving things to /var can only
> be a temporary solution. Should we force a reinstallation, which solves
> the root of the problem?

One obvious alternative would be using LVM. Combined with a suitable
filesystem like ext4, it makes resizing filesystems pretty trivial.
It does add a bit of overhead but not enough to matter here, IMHO.

Another would be using ZFS, which solves the problem a bit differently
(and arguably better), but it's not included in mainline kernels due
to licensing issues so it would add maintenance work.

FWIW, I like LVM and tend to use it in all systems where disk space
and allocation is likely to change.

-- 
Tapani Tarvainen

Re: Overfull Filesystems how to solve?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

Hey Tapani,

> On 18 Jan 2020, at 10:30, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
> 
> On Sat, Jan 18, 2020 at 10:57:08AM +0100, Jonatan Schlag (jonatan.schlag(a)ipfire.org) wrote:
>> 
>> Hi,
>> 
>> Daniel encountered the problem that his root partition was written full
>> because of the qemu addon being too big (200 MB). See bug #12268 for
>> details.
>> 
>> So he suggested moving the path /usr/share/qemu to /var. I do not
>> really like this solution as there are standards how the filesystem
>> hierarchy works under Linux. 
>> 
>> Furthermore, as the systems become bigger (eg. moving python3 to the
>> core) this problem will become more relevant to us. So I would like to
>> start a discussion on how to solve this. Moving things to /var can only
>> be a temporary solution. Should we force a reinstallation, which solves
>> the root of the problem?
> 
> One obvious alternative would be using LVM. Combined with a suitable
> filesystem like ext4, it makes resizing filesystems pretty trivial.
> It does add a bit of overhead but not enough to matter here, IMHO.
> 
> Another would be using ZFS, which solves the problem a bit differently
> (and arguably better), but it's not included in mainline kernels due
> to licensing issues so it would add maintenance work.
> 
> FWIW, I like LVM and tend to use it in all systems where disk space
> and allocation is likely to change.

LVM is generally a good idea. However it is not implementable on systems that are already running out of space.

Back in the day it seemed to be a good idea to have different partitions for / and /var. New systems will now just have one partition for /. Simply put, you will have all the space your hardware allows you and if you are running out, you are running out.

ZFS is not available on Linux.

-Michael

> 
> -- 
> Tapani Tarvainen

Re: Overfull Filesystems how to solve?

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 3525 bytes --]

On Sun, Jan 19, 2020 at 06:16:25PM +0000, Michael Tremer (michael.tremer(a)ipfire.org) wrote:

> > On 18 Jan 2020, at 10:30, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
> > 
> > On Sat, Jan 18, 2020 at 10:57:08AM +0100, Jonatan Schlag (jonatan.schlag(a)ipfire.org) wrote:
[...]
> >> Furthermore, as the systems become bigger (eg. moving python3 to the
> >> core) this problem will become more relevant to us. So I would like to
> >> start a discussion on how to solve this. Moving things to /var can only
> >> be a temporary solution. Should we force a reinstallation, which solves
> >> the root of the problem?

> > One obvious alternative would be using LVM. Combined with a suitable
> > filesystem like ext4, it makes resizing filesystems pretty trivial.
> > It does add a bit of overhead but not enough to matter here, IMHO.
> > 
> > Another would be using ZFS, which solves the problem a bit differently
> > (and arguably better), but it's not included in mainline kernels due
> > to licensing issues so it would add maintenance work.
> > 
> > FWIW, I like LVM and tend to use it in all systems where disk space
> > and allocation is likely to change.
 
> LVM is generally a good idea. However it is not implementable on
> systems that are already running out of space.

True, unless you are doing a reinstall anyway. And it would help
in the long run.

> Back in the day it seemed to be a good idea to have different
> partitions for / and /var. New systems will now just have one
> partition for /. Simply put, you will have all the space your
> hardware allows you and if you are running out, you are running out.

Although I see little use in having separate /var, under some
circumstances having one filesystem fill up might not be as disastrous
as filling / up. E.g., it might well make sense to have /var/log or
/tmp as separate filesystems. (I've seen firewalls DOSsed by
overflowing logs and all kinds of systems filling up /tmp because of
some process writing there went haywire.) It would require some
thinking about which directories would be likely to fill up
temporarily in a fixable way.

LVM also makes it easy to add another disk quickly without
reinstalling everything. Or even replacing the system disk with
a bigger one, having two disks connected only temporarily.

On the other hand, having a single / filesystem has the advantage of
simplicity, and in case of disk filling up in a permanent kind of way
(as opposed to temporary, fixable situations) it is the best one can
do, as you say.

It would also be possible to get the best of both worlds by using LVM
and still having just / filesystem initially or as an option, maybe
depending on disk size or user preference.

> ZFS is not available on Linux.

It's not in mainline kernels as I noted but it is certainly available
in the sense that OpenZFS can be installed (e.g., Ubuntu and Proxmox
build their default kernels with ZFS). I would still prefer LVM though.

Anyway, I may have misunderstood the context. If we are talking only
about what to do with a current installations that run out of space in
root (/) but have plenty left in /var, reinstallation seems like the
obvious thing to do. (Merging separate / and /var into one filesystem
without reinstalling would be technically possible but probably not
worth the trouble.)

LVM/ZFS/whatever was only an idea for the long term, worth considering
in the next major version perhaps.

-- 
Tapani Tarvainen

Re: Overfull Filesystems how to solve?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7679 bytes --]

Hi,

> On 20 Jan 2020, at 04:17, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
> 
> On Sun, Jan 19, 2020 at 06:16:25PM +0000, Michael Tremer (michael.tremer(a)ipfire.org) wrote:
> 
>>> On 18 Jan 2020, at 10:30, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
>>> 
>>> On Sat, Jan 18, 2020 at 10:57:08AM +0100, Jonatan Schlag (jonatan.schlag(a)ipfire.org) wrote:
> [...]
>>>> Furthermore, as the systems become bigger (eg. moving python3 to the
>>>> core) this problem will become more relevant to us. So I would like to
>>>> start a discussion on how to solve this. Moving things to /var can only
>>>> be a temporary solution. Should we force a reinstallation, which solves
>>>> the root of the problem?
> 
>>> One obvious alternative would be using LVM. Combined with a suitable
>>> filesystem like ext4, it makes resizing filesystems pretty trivial.
>>> It does add a bit of overhead but not enough to matter here, IMHO.
>>> 
>>> Another would be using ZFS, which solves the problem a bit differently
>>> (and arguably better), but it's not included in mainline kernels due
>>> to licensing issues so it would add maintenance work.
>>> 
>>> FWIW, I like LVM and tend to use it in all systems where disk space
>>> and allocation is likely to change.
> 
>> LVM is generally a good idea. However it is not implementable on
>> systems that are already running out of space.
> 
> True, unless you are doing a reinstall anyway. And it would help
> in the long run.
> 
>> Back in the day it seemed to be a good idea to have different
>> partitions for / and /var. New systems will now just have one
>> partition for /. Simply put, you will have all the space your
>> hardware allows you and if you are running out, you are running out.
> 
> Although I see little use in having separate /var, under some
> circumstances having one filesystem fill up might not be as disastrous
> as filling / up. E.g., it might well make sense to have /var/log or
> /tmp as separate filesystems. (I've seen firewalls DOSsed by
> overflowing logs and all kinds of systems filling up /tmp because of
> some process writing there went haywire.) It would require some
> thinking about which directories would be likely to fill up
> temporarily in a fixable way.

I would generally be in favour of that, but…

I have no idea how we can slice the disk into correctly sized slices. No matter how long we thought about that, we always had a unsatisfactory solution for some users. If we would have simply increased the size of the / partition, then we would have made more than 2GB of space a requirement. But is (let’s say) 4GB enough? Do we need 8GB in the future? The /boot partition used to be 20MB because a kernel was less than 1MB including the initramdisk. Right now, we are allocating 128MB. That is x5 the space. The initramdisk alone is now over 20MB. This is an example that should illustrate how we cannot use figures from right now because they might be totally wrong in a couple of years.

> LVM also makes it easy to add another disk quickly without
> reinstalling everything. Or even replacing the system disk with
> a bigger one, having two disks connected only temporarily.
> 
> On the other hand, having a single / filesystem has the advantage of
> simplicity, and in case of disk filling up in a permanent kind of way
> (as opposed to temporary, fixable situations) it is the best one can
> do, as you say.

We always had LVM and RAID on the roadmap for IPFire 3. Actually nobody has asked for this ever. So although it would be nice, there is no demand for it.

I guess that is mainly because IPFire’s add-ons are becoming less and less popular. Not because IPFire itself is becoming less popular - quite the opposite actually, but because of those services are easier to install in a virtual machine somewhere. Either in the cloud, a Raspberry Pi or simply as a VM on the IPFire box itself. So that means people who need more space just add an extra hard drive and they are done.

> It would also be possible to get the best of both worlds by using LVM
> and still having just / filesystem initially or as an option, maybe
> depending on disk size or user preference.

The code in the installer is actually quite nice and clean. I refactored it a couple of years ago and added support for RAID1.

I guess that could be extended to support LVM, but again, I am not sure if that will ever have any users.

>> ZFS is not available on Linux.
> 
> It's not in mainline kernels as I noted but it is certainly available
> in the sense that OpenZFS can be installed (e.g., Ubuntu and Proxmox
> build their default kernels with ZFS). I would still prefer LVM though.

So would I.

We got badly burned a couple of times with including out-of-tree kernel features. Reiser4 was a great file system and we felt that it was important to have it back in the day when hardisks were very slow. Reiser4 was great, but never made it into the kernel. Since there was then a lawsuit against Hans Reiser and (if I remember if correctly) the company fell apart, we could not longer update our kernel because patches for more recent kernels were not available. We were stuck. Removing support for it was not really an option because all people using Reiser4 would have needed to reinstall.

In the end we had to bite the bullet and deprecate all systems with Reiser4, but that was a huge pain that kept us busy over years.

Therefore I am very very very careful to include anything into our kernel that we cannot maintain ourselves (and we simply do not have the expertise or time to maintain a file system).

We currently have the same problem with the layer 7 module which had many bugs that we had to fix. If I could get rid of it, I would in a whim. Currently it works well though. No idea what will be with the next big kernel update.

On ZFS vs LVM: Linux was supposed to have an alternative to ZFS: btrfs. But that seemed to have become so complex and so fragile because it reimplemented LVM and RAID in form of sub volumes, etc. I think that should better be in the block layer instead of the file system layer to keep things more simple.

I suppose btrfs is properly dead now. ZFS does not look like it is ever going to be in mainline Linux ever.

Ext4 and XFS have caught up now, too. The advantages of things like btrfs have become almost nihilated for most users. They are both fast and robust and very very well tested. They just work and we do not have to worry about them I suppose. It literally does not matter what FS you are using for IPFire. They all perform exactly the same.

> Anyway, I may have misunderstood the context. If we are talking only
> about what to do with a current installations that run out of space in
> root (/) but have plenty left in /var, reinstallation seems like the
> obvious thing to do. (Merging separate / and /var into one filesystem
> without reinstalling would be technically possible but probably not
> worth the trouble.)

We thought about that, but that would indeed be dangerous and not worth our development time.

We simply need to clean up some files first of all and then we will see what problems are left.

After all I think we need to make sure not to bloat the system and 2GB sound plenty for me for a firewall.

The bloat is somewhere else actually. We are shipping almost half a gigabyte of firmware for various hardware. That all just adds up.

> LVM/ZFS/whatever was only an idea for the long term, worth considering
> in the next major version perhaps.

Agreed.

-Michael

> 
> -- 
> Tapani Tarvainen

Re: Overfull Filesystems how to solve?

[Tapani Tarvainen]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

On Jan 20 10:38, Michael Tremer (michael.tremer(a)ipfire.org) wrote:

[...]

> I have no idea how we can slice the disk into correctly sized
> slices. No matter how long we thought about that, we always had a
> unsatisfactory solution for some users. If we would have simply
> increased the size of the / partition, then we would have made more
> than 2GB of space a requirement. But is (let’s say) 4GB enough? Do
> we need 8GB in the future? The /boot partition used to be 20MB
> because a kernel was less than 1MB including the initramdisk. Right
> now, we are allocating 128MB. That is x5 the space. The initramdisk
> alone is now over 20MB. This is an example that should illustrate
> how we cannot use figures from right now because they might be
> totally wrong in a couple of years.

Yes. But of course that's exactly where LVM helps: partition sizes can
be adjusted as needed. Even /boot isn't a problem anymore now that
Grub understands LVM.

Even so, multiple partitions do add complexity and overhead, and
especially with a small disk most users probably would be better off
with a single filesystem. Though even then having LVM under it would
not cost much and it might save the day in some situation.

> We always had LVM and RAID on the roadmap for IPFire 3.

Good.

> Actually nobody has asked for this ever. So although it would be
> nice, there is no demand for it.

I would use it if it was available. (I'd probably make at least
/var/tmp and /var/log their own filesystems.)

> I guess that is mainly because IPFire’s add-ons are becoming less
> and less popular. Not because IPFire itself is becoming less popular
> - quite the opposite actually, but because of those services are
> easier to install in a virtual machine somewhere. Either in the
> cloud, a Raspberry Pi or simply as a VM on the IPFire box itself.

That may well be true.

> The code in the installer is actually quite nice and clean. I
> refactored it a couple of years ago and added support for RAID1.
>
> I guess that could be extended to support LVM, but again, I am not
> sure if that will ever have any users.

I predict it would have at least one user. :-)

> [...] I am very very very careful to include anything into our
> kernel that we cannot maintain ourselves (and we simply do not have
> the expertise or time to maintain a file system).

Agreed.

> I suppose btrfs is properly dead now.

That's perhaps a bit premature, but it's not looking too healthy
either.

> ZFS does not look like it is ever going to be in mainline Linux
> ever.

Yes. More's the pity.

> Ext4 and XFS have caught up now, too. The advantages of things like
> btrfs have become almost nihilated for most users. They are both
> fast and robust and very very well tested. They just work and we do
> not have to worry about them I suppose. It literally does not matter
> what FS you are using for IPFire. They all perform exactly the same.

Yes. In some situations there are noticeable speed differences, but
disk speed is rarely a problem in a firewall.

> The bloat is somewhere else actually. We are shipping almost half a
> gigabyte of firmware for various hardware. That all just adds up.

Right. Just dropping stuff for hardware that isn't supported anyway
should be fairly easy and probably go a long way in solving the issue.

-- 
Tapani Tarvainen

Re: Overfull Filesystems how to solve?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4501 bytes --]

Hi,

> On 20 Jan 2020, at 11:39, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
> 
> On Jan 20 10:38, Michael Tremer (michael.tremer(a)ipfire.org) wrote:
> 
> [...]
> 
>> I have no idea how we can slice the disk into correctly sized
>> slices. No matter how long we thought about that, we always had a
>> unsatisfactory solution for some users. If we would have simply
>> increased the size of the / partition, then we would have made more
>> than 2GB of space a requirement. But is (let’s say) 4GB enough? Do
>> we need 8GB in the future? The /boot partition used to be 20MB
>> because a kernel was less than 1MB including the initramdisk. Right
>> now, we are allocating 128MB. That is x5 the space. The initramdisk
>> alone is now over 20MB. This is an example that should illustrate
>> how we cannot use figures from right now because they might be
>> totally wrong in a couple of years.
> 
> Yes. But of course that's exactly where LVM helps: partition sizes can
> be adjusted as needed. Even /boot isn't a problem anymore now that
> Grub understands LVM.
> 
> Even so, multiple partitions do add complexity and overhead, and
> especially with a small disk most users probably would be better off
> with a single filesystem. Though even then having LVM under it would
> not cost much and it might save the day in some situation.
> 
>> We always had LVM and RAID on the roadmap for IPFire 3.
> 
> Good.
> 
>> Actually nobody has asked for this ever. So although it would be
>> nice, there is no demand for it.
> 
> I would use it if it was available. (I'd probably make at least
> /var/tmp and /var/log their own filesystems.)
> 
>> I guess that is mainly because IPFire’s add-ons are becoming less
>> and less popular. Not because IPFire itself is becoming less popular
>> - quite the opposite actually, but because of those services are
>> easier to install in a virtual machine somewhere. Either in the
>> cloud, a Raspberry Pi or simply as a VM on the IPFire box itself.
> 
> That may well be true.
> 
>> The code in the installer is actually quite nice and clean. I
>> refactored it a couple of years ago and added support for RAID1.
>> 
>> I guess that could be extended to support LVM, but again, I am not
>> sure if that will ever have any users.
> 
> I predict it would have at least one user. :-)

True, but we could put the work into IPFire 3 which would have more users.

With limited resources we need to make stupid decisions like this.

> 
>> [...] I am very very very careful to include anything into our
>> kernel that we cannot maintain ourselves (and we simply do not have
>> the expertise or time to maintain a file system).
> 
> Agreed.
> 
>> I suppose btrfs is properly dead now.
> 
> That's perhaps a bit premature, but it's not looking too healthy
> either.

I didn’t mean to be too harsh on the developers, but the last news I heard was that many of them left the project.

Fedora planned to make it the default FS many many years ago and it still did not happen. I suppose nobody is waiting for it any more.

>> ZFS does not look like it is ever going to be in mainline Linux
>> ever.
> 
> Yes. More's the pity.
> 
>> Ext4 and XFS have caught up now, too. The advantages of things like
>> btrfs have become almost nihilated for most users. They are both
>> fast and robust and very very well tested. They just work and we do
>> not have to worry about them I suppose. It literally does not matter
>> what FS you are using for IPFire. They all perform exactly the same.
> 
> Yes. In some situations there are noticeable speed differences, but
> disk speed is rarely a problem in a firewall.
> 
>> The bloat is somewhere else actually. We are shipping almost half a
>> gigabyte of firmware for various hardware. That all just adds up.
> 
> Right. Just dropping stuff for hardware that isn't supported anyway
> should be fairly easy and probably go a long way in solving the issue.

We have walked through /lib/firmware before. Old hardware does not need to load firmware. It either does not need it at all or there is a ROM on the hardware itself. It is the modern stuff that is still in use where they removed the ROM to save cost. We have estimated that we might be able to save 5% of disk space, but it is simply not worth it to put the work in every time we update the firmware package to save a couple of megabytes.

-Michael

> -- 
> Tapani Tarvainen

Re: Overfull Filesystems how to solve?

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Hello,

> On 18 Jan 2020, at 09:57, Jonatan Schlag <jonatan.schlag(a)ipfire.org> wrote:
> 
> Hi,
> 
> Daniel encountered the problem that his root partition was written full
> because of the qemu addon being too big (200 MB). See bug #12268 for
> details.
> 
> So he suggested moving the path /usr/share/qemu to /var. I do not
> really like this solution as there are standards how the filesystem
> hierarchy works under Linux. 

That was my suggestion for a quick fix.

However, looking at the files in /usr/share/qemu, I am not sure what we actually need. Can we not just delete some and we are fine again?

  https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/packages/qemu;h=e5c0cd18953c3de3121b43f40af277f29016b7f7;hb=HEAD

The current root file lists things like EFI ROMs for ARM which we probably do not support or need. BIOS ROMs for SPARC which we 100% do not support. If we just do not ship those files, this problem can be solved without moving the files.

> Furthermore, as the systems become bigger (eg. moving python3 to the
> core) this problem will become more relevant to us. So I would like to
> start a discussion on how to solve this. Moving things to /var can only
> be a temporary solution. Should we force a reinstallation, which solves
> the root of the problem?

Yes, we have a script to cleanup what we can. We still need to make sure that the distribution is as small as possible and we have not been spending too much time on this recently. Everything becomes bigger and bigger, but I would suggest that we try to support the systems with the 2GB root partition for as long as we can.

Best,
-Michael

> 
> Jonatan
>