Allow for custom image partition scheme

[BrendanSimon]

Recoverable Partition Scheme

Currently the image creation is fixed in the brickstrap.sh script. It would be better if this was factored out user/target to configuration, so that each build type can specify it's own partition scheme, etc.

My use case is for devices deployed in remote locations, that need to be remotely upgraded, and cannot be bricked if something goes wrong in the upgrade process. e.g. loss of network, power drop out, etc. In general this is solved by have multiple rootfs partitions.

My use case

Partition	Purpose	File System	Size
1	BOOT	VFAT	32MB, 64MB
2	ROOTFS1	ext4	1GB
3	ROOTFS2	ext4	1GB
4	DATA	VFAT, ext4	14GB - 30GB

More general use case

Partition	Purpose	File System	Size
1	BOOT	VFAT, FAT32, FAT16, FAT12	32MB, 48MB, 64MB
2	ROOTFS1	ext2, ext3, ext4	512MB, 1GB, 2GB
3	ROOTFS2	ext2, ext3, ext4	512MB, 1GB, 2GB
4	DATA	VFAT, ext2, ext3, ext4	1GB - 30GB

Other partitioning schemes can be considered, including those with logical volumes, extended volumes, lvm, etc.

Ping-Pong Rootfs

---

When a reboot occurs, the boot loader should have the capability to fallback to the last working rootfs if the latest rootfs fails integrity checks, does not load properly or does not run to a certain state where it would set something to notify the bootloader of it's success (e.g. a u-boot env var). This is sometimes referred to as a ping-pong rootfs system.

Golden Rootfs

---

A similar approach is to have the first partition the golden partition that never gets touched once programmed in the factory, and the second rootfs is the only one that gets upgraded. This can have issues of upgrading an upgraded system. i.e. upgrading a running system. It can be solved by booting into the golden partition before doing the upgrade of the 2nd partition. This can mean the system is down while doing the upgrade, which may not be acceptable.

An extension of the Golden Rootfs is to have a 3 rootfs partions -- one golden that never ever gets touched, and two other rootfs partitions for ping-pong. The Golden (or Recovery/Fallback) partition is usually small and simple whose main purpose is to be able to upgrade the main rootfs partitions with valid information.

[dlech]

I'm all for this. The Ping-Pong Rootfs and Golden Rootfs are outside of the scope of brickstrap (other than being implemented in a board definition directory), but we can certainly create images that can be used by these.

I'm thinking that we modify the create-image function to check for a file named create-image in the board definition. If the file exists, then it is used - if not, the existing function is called.

I also have a use case for BeagleBone Black where I need to offset the initial partition a bit to leave room for dding the bootloader at a specific address. I started working on it here.

[BrendanSimon]

Sure. The upgrade process is user specific, but it relies on an underlying partition scheme having been setup. I wouldn't want to repartition on the fly. That's just asking for trouble ;-)

System can use their own update scripts or other software.

Currently I'm trialling swupdate. It seems to be authored by at least one person at DENX Software Engineering so it has reasonable U-Boot support :) It's not fully mature yet, but has a lot of flexible infrastructure.

https://github.com/sbabic/swupdate

[BrendanSimon]

The guestfish utility does not seem to have prepared disk images for anything other than two partitions. e.g. -N bootroot

I think for my usecase, -N bootroot, followed by part-add would probably work, but the hard part is determining the start and stop sectors (probably not that hard). Would be nice to specify order of partitions, size and let the utility work it out. Similar to parted

But for total generality, one can not assume the number of partitions or filesystem types, so I'm guessing that we need to use:

part-init
part-add x N
mkfs x N

[dlech]

You can see a more general form of using guestfish that I started for the BBB here. You are on the right track.

[dlech]

I have hard-coded the sectors, but you could make bash do the math.

[BrendanSimon]

Also the partition to resize to max available space needs to be configurable. In my use case this would be partition 4, the DATA partition, used for user data such as application log files, etc.

[BrendanSimon]

Actually, I think resizing is done on the running system. It should be possible to do this on an actual SD card mounted in the CHROOT environment, yes? Though I guess that wont work for devices using onboard flash (eMMC, etc).

[dlech]

Actually, I think resizing is done on the running system.

Yes, that is how it is currently done and is the easiest way in my opinion.

It should be possible to do this on an actual SD card mounted in the CHROOT environment, yes?

If you want to resize the partitions on an SD card "offline", the best way would be to use a partitioning tool like parted or gparted. It would be dangerous to do this in a chroot because /dev is going to be bind mounted from your host computer and it would be easy to think /dev/sda is your SD card when really it is your hard drive.

[BrendanSimon]

I notice that wikiepedia says the partition type for FAT32 should be 0Ch for capacity > 7.8GB, and 0Bh for capacity < 7.8GB

For SDHC cards:
    Capacity of less than 16,450,560 logical sectors (smaller than 7.8 GB): FAT32 with partition type 0Bh and EBPB 7.1
    Capacity of at least 16,450,560 logical sectors (larger than 7.8 GB): FAT32 with partition type 0Ch and EBPB 7.1

My linux kernel isn't booting properly (it hangs) when I boot off Sandisk Ultra 16GB & 32GB Micro SD cards, but works perfectly well with Sandisk Ultra 8GB card. I'm wondering if the MBR type of 0Bh could be an issue? Or I could have just got lucky with the 8GB card and/or unlucky with the 16GB/32GB card?

[dlech]

Your FAT partition is < 7.8GB in all cases though, right? So partition type is always 0x0b.

[BrendanSimon]

Yes (64MB) but even if there was a need to increase it, it would definitely be < 7.8GB.
The wikipedia page wasn't clear if it was total disk capacity or the partition capacity, but you are probably right that it is partition capacity.

[cmacq2]

The Wikipedia article isn't very clear on the 'why' of it, and what the issue actually is. It's nothing to do with FAT16 versus FAT32 per se because that distinction is about limits on file systems and not disk partitions (which is what this limit is about).

The specific limit here is the size of the partition in number of sectors. (Which is still different from capacity limits in bytes, as sectors can in theory vary in size as well.) Specifically the limit is due to the fact only 24 bits are used to describe the size of the partition (leaving the first/last 8 out of 32 for a tag, i.e. the boot flag or partition type).

See: http://www.win.tue.nl/~aeb/partitions/partition_tables-2.html

[cmacq2]

Once PR #28 is merged, the new images output directory means it becomes feasible to address this issue thoroughly. My idea is as follows:

1. Have the create-image function source any create-image.sh scripts from the project dir. These scripts may register driver functions for creating custom image types. (basically setting a variable with the name of the function).
2. Have create-image dispatch to a driver script based on the selected image type(s). More than one type could be selected at the same time, all create-image would have to is loop over the list, similar to br_iterate_components. (Basically a case block inside a for loop.)
3. Implement default driver(s), which is basically copying the guts of current create-image to another function. Default drivers would take precedence in create-image over any drivers registered by the project with the same name. (Basically 'named' cases, and only the * case actually considers variables set in step 1).
4. Wrap this in a nice interface for selecting image types using a commandline switch, e.g. -F, and an environment variable (for use by project config, to set defaults).

[cmacq2]

PR #29 implements steps 1 thru 3 (more boilerplate comments than code). Step 4 should be similarly easy to do once PR #28 is in.

[BrendanSimon]

Will the new scheme do the following?

• create an filesystem archive for each partition (e.g. project-boot.tar, project-rootfs1.tar, project-rootfs2.tar, project-data.tar) ?
  • Could encode the partition number and/or the partition label in the name.
  • Can we optionally also compress the archive?
• Create an empty partition (e.g. rootfs2)
• Create a disk image with the boot partition with boot.tar contents, rootfs1 partition with rootfs1.tar contents, _rootfs2 partition with _rootfs1.tar* contents*, data partition with data.tar contents.

In my use case, the contents of rootfs2.tar is exactly the same as rootfs1.tar (or empty). The two partitions are used for active software and inactive software (for upgrades). The other scenario is rootfs1 and rootfs2 are different contents.

[dlech]

create an filesystem archive for each partition (e.g. project-boot.tar, project-rootfs1.tar, project-rootfs2.tar, project-data.tar) ?

I don't think so. The way guestfish works is that you can create mount points, so you make a guestfish script that mounts each partition to the appropriate place in the file system. Then you just use a single tar to populate everything. In your case where you have 2 identical partitions where one is probably not mounted, you can just dd the contents of one partition to another in the image that is created.

Can we optionally also compress the archive?

It's an option, but... this would be a waste of cpu since the tar would be compressed, only to be uncompressed when creating the image file. The tar is only meant to be a temporary intermediate file.

The rest of your comments seem to be just a repeat of the first item, so to sum up, sure you can do this, you just need to write a guestfish script to do it.

[BrendanSimon]

There is the use case for burning an SD card with all partitions and contents in one easy step :)

The other use case is for remotely upgrading units in the field with the same or similar partition scheme. In that case I would be transferring the tar.gz files (over a slowish cellular network), then uncompressing/unarchiving into the inactive/upgrade partition. Generally this will just be one rootfs image, however there is another case where the the bootloader (BOOT.BIN, which includes u-boot and an FPGA firmware blob) may require updating (usually due to FPGA firmware updates) and possibly uEnv.txt also.

So it would be nice to be able to generate an archive of each filesystem, as well as be able to have a combo of the filesystems as disk image. If brickstrap or guestfish can't support this for some reason, then the I will have to do it as a post-processing task.

If I understand what you are saying about guestfish, it requires a single tar image of the entire filesystem, with known directories (mount points) which it will use to copy information to the disk image partitions. A post processing step could (optionally) do the same thing to generate the individual rootfs archives. Not sure if guestfish can be used to extract the files from the image into the archives? Hmmm, interesting ....

[dlech]

OK. For this sort of thing, I would run brickstrap all to create the initial image to get the device up and going. Then for updates just run brickstrap create-rootfs and add a hook that creates the tar files that you need. The hook would basically be copied and pasted from the current create-tar function and modified to only get the files you need (e.g. add an exclude for the boot and data partition mount points) and add the z or J flag to compress it. With the new component command line option, you can make this hook a separate component so that it does not run when you build the full image file.

[BrendanSimon]

OK, a hook can be used, though I'm not sure what a component actually is at this stage. I believe the component stuff is still a work in progress, yes?

[dlech]

Components are implement, just no proper docs yet. This should give you an idea of how it works though.

[dlech]

In the context of my earlier comment, component just means what we used to call a "board file" or "board definition directory"

[cmacq2]

Will the new scheme do the following?

The scheme permits the project/configuration author to execute custom bash functions during the create-image step. As a result the capability of a bash function (i.e. sky) is the limit, provided you are willing to put in the work. Additionally suitable drivers may be provided via the brickstrap script itself (depending on your needs).

create an filesystem archive for each partition (e.g. project-boot.tar, project-rootfs1.tar, project-rootfs2.tar, project-data.tar) ?

Yes: you could do this by creating a corresponding driver for each type of rootfs/image you wish to output.

Could encode the partition number and/or the partition label in the name.

Yes, though it is more advisable to use the $1 argument to your driver function (which is the file name brickstrap expects). The filename brickstrap expects you to create in your driver function is derived as follows: <basename>.<driver name>.<file type suffix>

Where:

• basename is an arbitrary base/template name specified using the new -I option at build time.
• driver name is the name under which you registered your driver function as part of this new driver scheme.
• file type suffix is the corresponding file name suffix you registered with the driver name.

The advantage of sticking to the name specified in $1 to your driver function is that brickstrap does the sanity checking to make sure that you don't accidentally overwrite the file (unless -f was specified).

But if you really do not care about that and you really care about the exact file names you can either:

• Write a deployment script that you execute after brickstrap and sets up suitably named symlinks somewhere (or just cp files).
• Ignore it and simply write to arbitrary file names (a bit awkward for the end user).

Can we optionally also compress the archive?

Sure: custom drivers amount to a bash function so you can do pretty much anything you like in there.

Create an empty partition (e.g. rootfs2)

Ditto.

Create a disk image with the boot partition with boot.tar contents, rootfs1 partition with rootfs1.tar contents, rootfs2 partition with rootfs1.tar* contents*, data partition with data.tar contents.

Ditto.

The caveat is of course: with a custom driver you have to figure out the right commands or right guestfish script to set up your image the way you want from just the rootfs.tar tarball.

[dlech]

@BrendanSimon, have a look at https://github.com/ev3dev/brickstrap/blob/3bd88298f6405503a4535c259bb3e31770b6a4b6/image-drivers/reduntant-rootfs-plus-data it does just about everything you are asking, I think.

guestfish is really amazing and can do just about anything. Read all about it: http://libguestfs.org/guestfish.1.html

[cmacq2]

After some revisions PR #29 is now merged. This issue can probably be closed. @BrendanSimon ?

[BrendanSimon]

I will take a look and will let you know :)