[FOTA] Local-OTA for Embedded Linux System

The Local-OTA, as the name implies, is a scheme for upgrading the platform locally. This scheme is the basis of FOTA and can be integrated into FOTA.

OSTree provides a number of very significant technological advantages over other full-filesystem updating schemes. For embedded systems that need a solution for safe, atomic, full-filesystem updates, the usual approach is to have some kind of dual-bank scheme. Here, we’re going to take a look at the difference between OSTree and dual-bank systems, and the advantages OSTree can provide.

The work progress is shown in the following table:

Steps	Content	Status
1	Basic Knowledge	Done
2	Create remote repository	Done
3	Setup HTTP server for remote repository	Done
4	Transplant OSTree tool to embedded device	Done
5	Deloy the ostree ROOTFS metadata on embedded device	Done
6	Enabling the ostree boot metadata on embedded device	Done
7	Developement the RAMDISK for embedded device	Done

1. Comparing full-filesystem update strategies

In general, there are 2 methods for performing updates to an end node follows:

Method 1: Dual-bank system:

Note, The firmware contains the Linux Kernel, ramdisk, Linux Device Tree, and Linux RootFS.

Method 2: Update is performed on top of the existing version

1.1 Dual-bank

In a dual-bank system, the read-only root filesystem is kept on a different partition from the writable user space, so that when an update is needed the whole partition can be overwritten. For atomicity and safety, this read-only partition is duplicated: there are two complete copies of the filesystem, kept on different partitions, and the active partition can be selected at boot time.

When the system needs to be updated, the new filesystem image is written to the inactive partition, and the next time the system reboots, that partition becomes the active one.

The main advantage of this updated model is its safety. Updates are always strictly atomic, and there is always a known good image that can be rolled back to. However, there are significant trade-offs in flexibility and materials costs that must be made: the size of the root partition must be chosen when the system is flashed for the very first time, and the duplication of the root partition doubles the space required. When choosing how big to make the root partition, a device manufacturer has to consider not just how big their filesystem image currently is, but also must estimate and plan for the size of all future updates. If the size chosen is too small, it may restrict the ability to add new features. Making it larger, of course, adds to the bill of goods for the product—​and since it’s duplicated, every extra megabyte of future capacity actually costs two megabytes to accommodate.

1.2 Single on top of the existing version (OSTree)

OSTree checksums individual files and stores them as content-addressed objects, much like git. The read-only filesystem is built by "checking out" a particular revision and hard-linking the content-addressed objects into the actual Linux directory structure. Multiple filesystem versions can be stored, and any content that is duplicated across versions is only stored once. A complete history of all versions is stored in TreeHub, but it is not required to store that complete revision history on the device. Only one partition is needed—​writable user space can be on the same partition as the OSTree content store.

When an embedded system needs to be updated, the Treehub (remote repository) sends a small metadata file with a particular commit identifier. The device pulls that commit from Treehub (remote repository), only downloading the new files, and only downloading binary diffs of changed files. Once the pull is complete and verified, the system is instructed to boot into the new version the next time it starts up.

With OSTree, you no longer need to guess how much room you might need in the future to expand your system; the OSTree content store expands and contracts as needed. You also save a significant amount of space, since only diffs between versions need to be stored. OSTree also allows you to garbage-collect old images: if you upgrade 1.0 → 1.1 → 1.2, for example, by default the OTA Connect client will garbage-collect all local objects unique to 1.0. If you decided later on that you in fact did want to go back to v1.0, you still could: if you pushed v1.0 from the remote repository, the device would download only the diff from the remote repository, repopulate the local object store, and then reboot into that version. Of course, it’s also possible to configure OSTree to keep more than two revisions on the local disk.

A demo of the NXP-IMX6ULL platform with HTTP downloading the firmware is shown as the link: https://youtu.be/3i48NbAS2jU?t=1659

2. OSTree Method

The updating process is shown in the following figure:

OSTree checksums individual files and stores them as content-addressed objects, much like git. The OSTree Tool and OSTree library are in the link https://github.com/ostreedev/ostree.

The meta-updater is in the Linux ramdisk and it is an open-source tool, please refer to the link: https://github.com/advancedtelematic/meta-updater

2.1 OSTree Steps

The following steps illustrate the deploy the OSTree for embedded devices:

• Setup Remote Repository.

• Integrate the Repository.

• Embedded Device Deployment.

• Remote repository usage.

2.1.1 Setup Remote Repository

To create an OSTree repository, you can follow these steps:

• Install the ostree package on your system if it is not already installed. This package contains the necessary tools to manage an OSTree repository.

• Create an empty directory to serve as the root of your repository. This directory will contain all the objects and metadata needed for the repository.

• mkdir my-ostree-repo

• Initialize the repository using the ostree init command. This will create the necessary metadata files and directories.

• ostree init --mode=archive-z2 --repo=my-ostree-repo

  The --mode=archive-z2 option specifies the compression format to use for the repository. You can choose a different format if you prefer.

• Add content to the repository using the ostree commit command. This command takes a directory or a file as input and creates a new commit in the repository.

• ostree commit --branch=my-branch --repo=my-ostree-repo /path/to/content

  The --branch option specifies the name of the branch to create, and the /path/to/content parameter specifies the directory or file to include in the commit.

• Repeat step 4 for each piece of content you want to add to the repository.

• Publish the repository using the ostree summary and ostree summary-to-filename commands. These commands create a summary file that describes the contents of the repository and allows clients to quickly check for updates. $ ostree summary --repo=my-ostree-repo $ ostree summary-to-filename --repo=my-ostree-repo > my-ostree-repo/refs/heads/my-branch.summary

• Make the repository available to clients by serving it over HTTP or HTTPS. You can use any web server to serve the repository directory.

• sudo python3 -m http.server 80 --bind 0.0.0.0 --directory my-ostree-repo/ This command serves the repository directory on port 80 and allows any client to access it. You should use a different port and restrict access to trusted clients in a production environment.

2.1.2 Repository Integration

In the main storage partition, we have basically two directories that are mounted at /sysroot:

• boot directory (/boot)

• OSTree repository (/ostree)

The embedded device filesystem trees (also called deployments) are checked out at /sysroot/ostree/deploy/<os>/deploy/<commit> (files there are just hard links to objects in the repository)

A deployment is bind-mounted as a read-write rootfs at /, and the /usr directory from the deployment is bind-mounted read-only at /usr. A deployment is shown in the following figure:

• The storage device is the deployed repository.

• The runtime is the embedded device filesystem.

To integrate an OSTree repository, you can follow these steps:

• Generate the sysroot partition with the boot directory (/boot) and the OSTree repository (/ostree).

• Prepare the default deployment in /sysroot/ostree/deploy/deploy/.

• Make sure U-Boot will be able to load and boot the kernel artifacts (kernel image, device tree, ramdisk).

• Boot a ramdisk image that will mount the OSTree deployment and switch to it. The step is already (mostly) implemented in meta-updater, please refer to https://docs.ota.here.com/ota-client/latest/bsp-integration.html

2.1.3 Deployment

A device’s SD card or eMMC partition is shown in the following figure:

• Boot Directory: The boot partition stores the data of the last three boot images.

• OSTree Repository: The ostree stores the OSTree repository metadata.

• System Root: the sysroot is the real root partition used. The ramdisk will chroot() to this path.

In the ramdisk, we can backup the sysroot to sysroot.old, and checkout the newest rootfs from the repository by ostree checkout --repo=./ostree/repo master sysroot. If there is no issue in the newest rootfs, the sysroot.old can be dropped from the filesystem. Finally, the ramdisk boots the sysroot as the rootfs.

2.1.4 Usage Remote Repository

The ostree remote command is used to manage OSTree repositories. OSTree is a tool for managing bootable, immutable file systems, and the remote command specifically deals with adding, removing, and listing remote repositories.

Here are some examples of how to use the ostree remote command:

To add a remote repository:

ostree remote add --set=gpg-verify=false --no-gpg-verify myrepo https://example.com/repo

This command adds a remote repository named myrepo located at https://example.com/repo and sets gpg-verify to false.

To remove a remote repository:ostree remote delete myrepo

This command removes the remote repository named myrepo.

To list remote repositories:ostree remote list

This command lists all the remote repositories configured on the system.

To show details of a remote repository:ostree remote show myrepo

This command shows the details of the remote repository named myrepo, such as its URL, GPG verification settings, and any proxies or authentication configured.

ostree remote add --no-gpg-verify origin http://127.0.0.1:8000/
ostree remote show-url origin --repo=repo
ostree remote refs origin
ostree remote pull origin:master
ostree log origin:master

3. OSTree Transplanting

It is difficult to cross-compile the OSTree in the ARM platform (ARMv7/v8) because the ostree is based on various libraries. But there are multiple friendly build systems out there, such as Buildroot and Yocto for example. There are also ostree recipes for the latter at least. I'd recommend looking into one of those instead of a hand-rolling one.

The buildroot (https://buildroot.org/) is a simple, efficient, and easy-to-use tool to generate embedded Linux systems through cross-compilation. We can utilize the buildroot to help us to generate the ostree binary and libraries on the embedded Linux system.

• For the ZYNQ platform, you can refer to the link https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842369/Build+Linux+for+Zynq-7000+AP+SoC+using+Buildroot.

• For the S32g platform, you can refer to the link S32G Linux Boot Time Optimization

• For the Layerscape platform, you can refer to the Layerscape Software Development Kit User Guide.

3.1 Config

You should configure the platform by the corresponding BSP user guide above as step 1. The next step should add the ostree and its net libraries using the menuconfig.

3.1.1 OSTree package

3.1.2 libsoup and libcurl

In order to the HTTP function, the libsoup and libcurl libraries shall be also enabled.

The libcurl set by:

The libsoup set by:

The ostree can be run on the embedded device.

4. Demo

4.1 Preparing remote rootfs repository

In this demo, we make use of the output of the buildroot directly. The ~/temp mocks the remote root repository, and we create a repo directory as the remote ostree repository.

mkdir -p repo && ostree --repo=repo init --mode=archive-z2 && mkdir -p rootfs

Add an empty rootfs to the repository and commit it as the V1 version.

ostree --repo=repo commit --branch=master --subject="image v1 (empty)" rootfs/

Copy the rootfs of buildroot output to the rootfs path, the rootfs layout is shown in the following figure:

We commit the new rootfs as the “image v2“

ostree --repo=repo commit --branch=master --subject="image v2 (zynq rootfs)" rootfs/

We can list the commit records by ostree log master --repo=repo

Publish the repository using the ostree summary and ostree summary-to-filename commands. These commands create a summary file that describes the contents of the repository and allows clients to quickly check for updates.

ostree summary --repo=repo ./repo/branch.summary -u

In this demo, we use the python http server to mock the remote server by:

python3 -m http.server 8000 --bind 192.168.32.2 --directory repo

4.2 Creating the device rootfs

The SD rootfs (/dev/mmcblk0p2) file structure is shown in the following figure:

We should init the repository for the ostree repo in the SD card.

ostree --repo=repo init --mode=archive-z2

Add a remote address for the ostree repo

ostree remote add --no-gpg-verify origin http://192.168.32.2:8000/

Pull the metadata for ostree repo

ostree remote pull origin:master

client log

server log

Check out the newest rootfs by

ostree checkout --repo=repo master sysroot

4.3 meta-updater

The meta-updater is in the ramdisk, its function:

• Detect if an upgrade is required

• Update the boot.1 files to boot partition (/dev/mmcblk0p1)

• Pull and Checkout the newest rootfs from the ostree repository.

• Boot the new rootfs

In the ramdisk init script, we add

echo "[FOTA-meta-updater] mount -o rw /dev/mmcblk0p2 /mnt"
mount -o rw /dev/mmcblk0p2 /mnt
if [ $? -eq 0 ]; then
    echo "[INFO] mount done."
else
    echo "[ERR] mount failed."
    exit -1
fi

echo "[FOTA-meta-updater] delete old rootfs"
rm -rf /mnt/sysroot

sync

echo "[FOTA-meta-updater] ostree update the newest rootfs from the repo"
ostree checkout master /mnt/sysroot --repo=/mnt/repo
if [ $? -eq 0 ]; then
    echo "[INFO] ostree checkout master /mnt/sysroot done."
else
    echo "[ERR] ostree checkout master /mnt/sysroot failed."
    exit -1
fi
echo "[FOTA-meta-updater] ostree updated!!!!!!!!!!!"

echo "[FOTA-meta-updater] remount the new rootfs"
mount --bind /mnt/sysroot $ROOTFS_DIR
if [ $? -eq 0 ]; then
    echo "[INFO] mount bind done."
else
    echo "[ERR] mount bind failed."
    exit -1
fi

echo "[FOTA-meta-updater] mounted new rootfs"
echo "[FOTA-meta-updater] booting from the new rootfs: "
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."
echo "[FOTA-meta-updater] ..........................................."

The new rootfs can recover from the repository.

Ref

• https://witekio.com/blog/ostree-tutorial-system-updates/

• https://developer.toradex.com/torizon/in-depth/ostree/

• https://ostree.readthedocs.io/en/stable/manual/introduction/

• https://docs.ota.here.com/ota-client/latest/ostree-and-treehub.html

• https://www.youtube.com/watch?v=skDBX9Rywv8&ab_channel=SergioPrado

• https://www.youtube.com/watch?v=3i48NbAS2jU&ab_channel=SergioPrado

• https://github.com/buildroot/buildroot

• https://www.mankier.com/1/ostree-checkout

• https://ostree.readthedocs.io/en/stable/manual/deployment/

• https://manpages.debian.org/testing/ostree/ostree-admin-status.1.en.html

• https://github.com/ostreedev/ostree/issues/2223

• https://www.youtube.com/watch?v=jYscnrpyB9Y&ab_channel=Toradex

• https://github.com/ostreedev/ostree/tree/main/src/switchroot

• https://github.com/ostreedev/ostree/issues/2826

• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842369/Build+Linux+for+Zynq-7000+AP+SoC+using+Buildroot

• Preparing the Host and Platform Project for OSTree Support