Re: Overfull Filesystems how to solve?

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 7679 bytes --]

Hi,

> On 20 Jan 2020, at 04:17, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
> 
> On Sun, Jan 19, 2020 at 06:16:25PM +0000, Michael Tremer (michael.tremer(a)ipfire.org) wrote:
> 
>>> On 18 Jan 2020, at 10:30, Tapani Tarvainen <ipfire(a)tapanitarvainen.fi> wrote:
>>> 
>>> On Sat, Jan 18, 2020 at 10:57:08AM +0100, Jonatan Schlag (jonatan.schlag(a)ipfire.org) wrote:
> [...]
>>>> Furthermore, as the systems become bigger (eg. moving python3 to the
>>>> core) this problem will become more relevant to us. So I would like to
>>>> start a discussion on how to solve this. Moving things to /var can only
>>>> be a temporary solution. Should we force a reinstallation, which solves
>>>> the root of the problem?
> 
>>> One obvious alternative would be using LVM. Combined with a suitable
>>> filesystem like ext4, it makes resizing filesystems pretty trivial.
>>> It does add a bit of overhead but not enough to matter here, IMHO.
>>> 
>>> Another would be using ZFS, which solves the problem a bit differently
>>> (and arguably better), but it's not included in mainline kernels due
>>> to licensing issues so it would add maintenance work.
>>> 
>>> FWIW, I like LVM and tend to use it in all systems where disk space
>>> and allocation is likely to change.
> 
>> LVM is generally a good idea. However it is not implementable on
>> systems that are already running out of space.
> 
> True, unless you are doing a reinstall anyway. And it would help
> in the long run.
> 
>> Back in the day it seemed to be a good idea to have different
>> partitions for / and /var. New systems will now just have one
>> partition for /. Simply put, you will have all the space your
>> hardware allows you and if you are running out, you are running out.
> 
> Although I see little use in having separate /var, under some
> circumstances having one filesystem fill up might not be as disastrous
> as filling / up. E.g., it might well make sense to have /var/log or
> /tmp as separate filesystems. (I've seen firewalls DOSsed by
> overflowing logs and all kinds of systems filling up /tmp because of
> some process writing there went haywire.) It would require some
> thinking about which directories would be likely to fill up
> temporarily in a fixable way.

I would generally be in favour of that, but…

I have no idea how we can slice the disk into correctly sized slices. No matter how long we thought about that, we always had a unsatisfactory solution for some users. If we would have simply increased the size of the / partition, then we would have made more than 2GB of space a requirement. But is (let’s say) 4GB enough? Do we need 8GB in the future? The /boot partition used to be 20MB because a kernel was less than 1MB including the initramdisk. Right now, we are allocating 128MB. That is x5 the space. The initramdisk alone is now over 20MB. This is an example that should illustrate how we cannot use figures from right now because they might be totally wrong in a couple of years.

> LVM also makes it easy to add another disk quickly without
> reinstalling everything. Or even replacing the system disk with
> a bigger one, having two disks connected only temporarily.
> 
> On the other hand, having a single / filesystem has the advantage of
> simplicity, and in case of disk filling up in a permanent kind of way
> (as opposed to temporary, fixable situations) it is the best one can
> do, as you say.

We always had LVM and RAID on the roadmap for IPFire 3. Actually nobody has asked for this ever. So although it would be nice, there is no demand for it.

I guess that is mainly because IPFire’s add-ons are becoming less and less popular. Not because IPFire itself is becoming less popular - quite the opposite actually, but because of those services are easier to install in a virtual machine somewhere. Either in the cloud, a Raspberry Pi or simply as a VM on the IPFire box itself. So that means people who need more space just add an extra hard drive and they are done.

> It would also be possible to get the best of both worlds by using LVM
> and still having just / filesystem initially or as an option, maybe
> depending on disk size or user preference.

The code in the installer is actually quite nice and clean. I refactored it a couple of years ago and added support for RAID1.

I guess that could be extended to support LVM, but again, I am not sure if that will ever have any users.

>> ZFS is not available on Linux.
> 
> It's not in mainline kernels as I noted but it is certainly available
> in the sense that OpenZFS can be installed (e.g., Ubuntu and Proxmox
> build their default kernels with ZFS). I would still prefer LVM though.

So would I.

We got badly burned a couple of times with including out-of-tree kernel features. Reiser4 was a great file system and we felt that it was important to have it back in the day when hardisks were very slow. Reiser4 was great, but never made it into the kernel. Since there was then a lawsuit against Hans Reiser and (if I remember if correctly) the company fell apart, we could not longer update our kernel because patches for more recent kernels were not available. We were stuck. Removing support for it was not really an option because all people using Reiser4 would have needed to reinstall.

In the end we had to bite the bullet and deprecate all systems with Reiser4, but that was a huge pain that kept us busy over years.

Therefore I am very very very careful to include anything into our kernel that we cannot maintain ourselves (and we simply do not have the expertise or time to maintain a file system).

We currently have the same problem with the layer 7 module which had many bugs that we had to fix. If I could get rid of it, I would in a whim. Currently it works well though. No idea what will be with the next big kernel update.

On ZFS vs LVM: Linux was supposed to have an alternative to ZFS: btrfs. But that seemed to have become so complex and so fragile because it reimplemented LVM and RAID in form of sub volumes, etc. I think that should better be in the block layer instead of the file system layer to keep things more simple.

I suppose btrfs is properly dead now. ZFS does not look like it is ever going to be in mainline Linux ever.

Ext4 and XFS have caught up now, too. The advantages of things like btrfs have become almost nihilated for most users. They are both fast and robust and very very well tested. They just work and we do not have to worry about them I suppose. It literally does not matter what FS you are using for IPFire. They all perform exactly the same.

> Anyway, I may have misunderstood the context. If we are talking only
> about what to do with a current installations that run out of space in
> root (/) but have plenty left in /var, reinstallation seems like the
> obvious thing to do. (Merging separate / and /var into one filesystem
> without reinstalling would be technically possible but probably not
> worth the trouble.)

We thought about that, but that would indeed be dangerous and not worth our development time.

We simply need to clean up some files first of all and then we will see what problems are left.

After all I think we need to make sure not to bloat the system and 2GB sound plenty for me for a firewall.

The bloat is somewhere else actually. We are shipping almost half a gigabyte of firmware for various hardware. That all just adds up.

> LVM/ZFS/whatever was only an idea for the long term, worth considering
> in the next major version perhaps.

Agreed.

-Michael

> 
> -- 
> Tapani Tarvainen