OSTree is a library - actually libostree - that handles updates for filesystem trees, that is, the entire Linux root filesystem. It applies updates atomically, therefore guaranteeing the system integrity in case of a power-cut or dropped internet connectivity for instance. Different from the A/B partition mechanism of updates, OSTree works in a git-like model, providing file updates and thus saving storage space as well as connection bandwidth and allowing more than 2 software revisions on the device at the same time. In addition, some directories such as /var and /etc and handled as special cases, providing persistent configuration and user data to be preserved between updates.
OSTree is a key technology used by Torizon OTA, which is based on Aktualizr. If Aktualizr is used, OSTree is managed by it and should not be manipulated directly. This article shows how to use OSTree directly from command-line, useful for development or when the Torizon OTA/Aktualizr is not being used.
If you just want to use the Torizon OTA, it will abstract the OSTree for you, therefore you can skip this article. If you want to learn more, or if you plan to only use the OSTree capabilities from TorizonCore, but you don't plan to use Torizon OTA, then this article may be useful for you.
This article complies to the Typographic Conventions for Torizon Documentation.
This article shows how-to perform a TorizonCore update using the ostree command directly in a shell on the device.
Whenever TorizonCore is built using Yocto/OpenEmbedded's bitbake, a OSTree repository is automatically initialized and a new OSTree commit is created. For every following build, a new commit is created where only changed files are added to the repository. Inside the OSTree repository, the builds are organized in branches (or REFs). REFs follow a naming convention consisting of the TorizonCore major version number, machine name, distribution name, image name and release type, e.g.:
4/colibri-imx6/torizon/torizon-core-docker/nightly
REFs with a major version 0 are builds from sources taken from the master branch, typically meant for TorizonCore development only.
Toradex uploads the OSTree repository of TorizonCore builds produced by the Toradex CI infrastructure to make them available publicly https://feeds.toradex.com/ostree/.
This article shows how to update from both sources:
Note: OSTree allows to switch between different sources transparently. E.g. it is possible to deploy a OSTree from a local build on a TorizonCore installation made from the official builds.
Warning: All device commands must be executed as root, so you have to prefix the following command lines with "sudo" to execute them as root.
This chapter shows how to update from a TorizonCore installation to a newer build obtained from the Toradex provided TorizonCore OSTree.
On the board, add the Toradex OSTree repository as a new remote using ostree remote, similar to how one would add a remote in a git repo:
# ostree remote add --no-gpg-verify torizon https://feeds.toradex.com/ostree/
Note: Currently Toradex does not sign the vanilla TorizonCore OSTree commits (hence the --no-gpg-verify is required). In the Torizon OTA the TorizonCore commits are cryptographic signed by Uptane/aktualizr.
The ostree admin commands allow to manage the OSTree on the host system. The command ostree admin status shows details about the currently active deployments (root filesystems).
# ostree admin status
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
This shows an installation of a monthly build on a Colibri iMX6. At this point, there is only a single deployment active, the one from the original installation.
The ostree admin upgrade command can be used to upgrade to the latest monthly release. This command downloads the latest commit of the origin branch, and deploys that in one step. At the time of writing (July 16th) the monthly branch is not available on the OSTree repository. Future monthly releases are expected to be available.
To switch to a different branch, e.g. to nightly build or a new major release, the branch name needs to be explicitly stated. The ostree remote refs command allows to get a list of branches available on a remote OSTree repository:
# ostree remote refs torizon
...
torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
...
Use OSTree pull to download the lastest version of a particular branch:
# ostree pull torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
32 metadata, 53 content objects fetched; 51978 KiB transferred in 21 seconds
To see more details about the fetched OSTree commits, ostree log can be used:
# ostree log torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
commit 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e
ContentChecksum: fc8b18d816f877671d52193221f888d9b651732f1644b08827bdc9bdec61d996
Date: 2020-07-15 19:43:31 +0000
Version: 4.0.0-devel-20200715+build.166
4.0.0-devel-20200715+build.166
meta-toradex-torizon HEAD:75acc4a3acfd2104e47769b2a7e038f2503d57dc
meta-toradex-distro HEAD:276c966a04bbe4d19866becb3b8fc147d1079049
meta-toradex-bsp-common HEAD:419eba81a51fb8f8e5a4eef9f23f1734c7044c78
meta-oe HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-networking HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-filesystems HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-python HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-perl HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b
meta-virtualization HEAD:514ce5f6b240600f06b9956737be1e33900bacac
meta-updater HEAD:12e0cd9e18afadfdd90d81fb947950b4620901fc
meta-toradex-nxp HEAD:acbfe83716e6a045d9290d932eec29a9afb468f1
meta-freescale HEAD:f535ddcd5cd3da846a34b1721ceda3c8b18b37a0
meta-freescale-3rdparty HEAD:dbcc686f52c3c84db8cb86aa8973a4e373651b98
meta-yocto-bsp HEAD:2d2dc20359ca75203f2194415b53731ad8ff0066
meta-poky HEAD:2d2dc20359ca75203f2194415b53731ad8ff0066
meta-security HEAD:440c37f0b623ccc0aa0328613908608d6362adda
meta-lmp-base HEAD:067487d1d4a823237eb6e7bbdebf9b023b58affa
meta HEAD:ee95a399285abbde84e0148ca957b59d65bcad0a
<< History beyond this commit not fetched >>
To see which files have been changed, ostree diff can be used. To show the complete diff of the current running commit and the target REF, you have to specify the commit from the ostree admin status command explicitly (leave away the part after the dot).
# ostree diff 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
M /usr/package.manifest
M /usr/bin/chardetect3
...
The command ostree admin deploy allows to create a new deployment using the given branch which will be used upon next boot:
# ostree admin deploy torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
Copying /etc changes: 4 modified, 1 removed, 7 added
Transaction complete; bootconfig swap: yes; deployment count change: 1
OSTree shows that there is a switch to a new root filesystem tree pending:
# ostree admin status
torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0 (pending)
Version: 4.0.0-devel-20200715+build.166
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
On the next boot, the system will boot using the new filesystem tree.
# reboot
After reboot you can view your current and previous deployment (still available for rollback):
# ostree admin status
* torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0
Version: 4.0.0-devel-20200715+build.166
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0 (rollback)
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
If you are building TorizonCore image locally you may provide an ostree repo directly from your PC. If you have Python installed, just move to the ostree_repo folder of your build directory (located in the OpenEmbedded build directory under deploy/images/$MACHINE/ostree_repo) and bring-up the Python built-in simple HTTP server:
$ python -m SimpleHTTPServer 8081
In this way your PC will be sharing the image you just built on port 8081.
On the board, add the local OSTree repository using ostree remote, similar to how one would add a remote in a git repo:
# ostree remote add --no-gpg-verify local-build \
http://<your PC host name/ip address>:8081/
You then can list the available branches and deploy a new tree using the same commands documented above (using the remote name local-build).
If you don't have network connectivity on target board an update can also be deployed using a USB flash drive or SD card.
If you are building TorizonCore image locally the OSTree repository will be in deploy/images/$MACHINE/ostree_repo. The repository is in archive mode and hence can be copied with a regular copy command:
$ cp -r deploy/images/$MACHINE/ostree_repo path/to/flash-drive
You can also create a new OSTree repository and fetch an update from the Toradex TorizonCore OSTree repository:
$ cd path/to/flash-drive
$ ostree init --repo=ostree_repo --mode=archive
$ ostree remote add --repo=ostree_repo --no-gpg-verify torizon https://feeds.toradex.com/ostree/
$ ostree pull --repo=ostree_repo torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
$ ostree refs --repo=ostree_repo --create=4/colibri-imx6/torizon/torizon-core-docker/nightly torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
Note: It seems that the ostree pull-local command used on the target cannot handle REFs with a colon, hence a new remote independent reference needs to be created.
On the target, list the refs available on the given repository:
# ostree refs --repo=path/to/flash-drive
torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
4/colibri-imx6/torizon/torizon-core-docker/nightly
Pull the commit from the given branch name from the USB flash drive to the target board:
# ostree pull-local /home/root/pendrive/ostree_repo/ 4/colibri-imx6/torizon/torizon-core-docker/nightly
Create a new deployment from the given branch name:
# ostree admin deploy 4/colibri-imx6/torizon/torizon-core-docker/nightly
OSTree shows that there is a switch to a new root filesystem tree pending:
# ostree admin status
torizon 1d98e25015a390556e7ab93c9b94e01e8632d30d800b36bff1ddd54fa9fc7cf4.0 (pending)
origin refspec: apalis-imx6/torizon/torizon-core-docker
* torizon d0d53487d305358ad50c37b1b2a94fc78eaea5e770749a33f5607573e2447660.0
origin refspec: d0d53487d305358ad50c37b1b2a94fc78eaea5e770749a33f5607573e2447660
On the next boot, the system will boot using the new filesystem tree.
# reboot
OSTree allows to rollback to the last root filesystem in case of a bad update. A rollback can be manually triggered by setting the U-Boot environment variable rollback:
> env set rollback 1
The boot script will load the kernel/device tree and initramfs from the rollback deployment. This is shown in the boot message as well as visible when looking at ostree admin status
...
Rollback enabled. Booting previously deployed version.
...
# ostree admin status
torizon 8ad8f2f63be22b271abfeec7a9fbb4d105b882ac08b33aeacfbdce412f79383e.0 (pending)
Version: 4.0.0-devel-20200715+build.166
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/nightly
* torizon 0b8e2f573de3d9c913abbb064674b04e2ce86da25ba7ea226b08a23db5cbd31c.0
Version: 4.0.0-devel-202007+build.17
origin refspec: torizon:4/colibri-imx6/torizon/torizon-core-docker/monthly
In this state a new (hopefully working) OSTree can be deployed.
TorizonCore relies on U-Boot boot count support to provide automatic rollback in case a bad update has been deployed. Any update that causes issues is expected to reboot. For that TorizonCore configures the kernel to automatically reboot in case of a kernel panic, and instructs systemd to reboot in case a user defined critical application (by default docker.service) does not start. If the system detects more reboots than what is defined in U-Boot's bootlimit environment variable, the rollback mechanism will be triggered. With aktualizr the U-Boot boot count is automatically enabled after an upgrade. When manually deploying an update using OSTree (ostree admin deploy), the boot count support needs to be enabled manually to enable automatic rollback support:
# boot_partition=$(readlink /dev/emmc-boot0)
# echo 0 > /sys/block/${boot_partition}/force_ro
# fw_setenv upgrade_available 1
# fw_setenv bootcount 0
# echo 1 > /sys/block/${boot_partition}/force_ro
# fw_setenv upgrade_available 1
# fw_setenv bootcount 0
After the reboot, the output of the ostree admin status can be used to see if the upgrade succeeded.
After an unsuccessful update, the system stays in "rollback" mode. A new update need to be deployed and the rollback variable needs to be cleared:
# fw_setenv rollback 0
After a successful update, the boot count should be disabled again (since regular flash is used as boot counter, this prevents unnecessary wear on the flash used for the U-Boot environment)
# fw_setenv upgrade_available 0
By default, OSTree will retain only the last two deployments, the current one and the last one to roolback in case of a bad update. If you want to retain more than two deployments, the --retain parameter can be used when creating a new deployment:
# ostree admin deploy --retain torizon:5/$MACHINE/torizon-rt/torizon-core-podman/nightly
Copying /etc changes: 4 modified, 2 removed, 9 added
Transaction complete; bootconfig swap: yes; deployment count change: 1
Now OSTree will show three deployments:
# ostree admin status
torizon eb07b368ff1e41350835af66252b736e3f73f4f063d70d599208b6776a0e4a1e.0 (pending)
Version: 5.1.0-devel-20201122+build.128
origin refspec: torizon:5/apalis-imx8/torizon-rt/torizon-core-podman/nightly
* torizon 4ebb32723cdd786c172ec7fd2be213bf49faf13bbec4b885b248978c822325e9.0
Version: 5.1.0-devel-20201122+build.128
origin refspec: torizon:5/apalis-imx8/torizon/torizon-core-podman/nightly
torizon 5bda7e34fcbda221e728e9b7ad79349ba88671e166b683bb7f3399d43fd2aff0.0 (rollback)
Version: 5.1.0-devel-20201122+build.128
origin refspec: torizon:5/apalis-imx8/torizon-rt/torizon-core-docker/nightly
When deploying a new filesystem with OSTree, some directories will be handled in a special way:
/etc: when OSTree creates a deployment, it performs a 3-way merge using the old default configuration, the active system's /etc, and the new default configuration. In the final filesystem tree for a deployment, /etc is a regular writable directory that can be used to store system configuration./var: this directory is not touched by OSTree, and can be used to store logs, databases and any other data consumed or generated by applications./home: this directory is also not touched by OSTree and can be used to store any user-related files. It is actually a link to /var/rootdirs/home.
