Search by Tags

Aktualizr - Modifying the Settings of Torizon OTA Client

 

Article updated at 25 Apr 2022
Compare with Revision




Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.



Remember that you can always refer to the Torizon Documentation, there you can find a lot of relevant articles that might help you in the application development.

Torizon 5.0.0

Introduction

Torizon OTA is the recommended method for Over-the-Air updates using Torizon, our easy-to-use embedded Linux platform.

TorizonCore is built with OSTree and Aktualizr. OSTree and Aktualizr are complementary and together they form the foundation for OTA (over-the-air) update capabilities on the device. To learn more about the technical aspects of Torizon OTA, see the Torizon OTA (Over the Air Updates) article.

If you just want to use the Torizon OTA with default settings, the pre-installed software on a TorizonCore image abstracts the settings of Aktualizr for you, therefore you can skip this article.

However, if you need to modify some common configurations of the client-side of Torizon OTA, such as how to block updates, update the server polling time, disable the automatic reboot, among other, or if you just want to learn more about Aktualizr, then this article may be useful for you.

This article will be also useful if you plan to only use the Aktualizr capabilities from TorizonCore, but you don't plan to use the Torizon OTA service.

Prepare for Production

After changing the setting in the SoM, we recommend that you use the TorizonCore Builder Tool to apply customization options to a TorizonCore image for production.

This article complies to the Typographic Conventions for Torizon Documentation.

Prerequisites

Before Starting: Stopping and Restarting the Torizon OTA Client Service

Attention: You will need to execute the commands as root. You can log in as root or (better) use the sudo command when logged in with a regular user to do it.

Stopping Aktualizr

Before to start changing the Torizon OTA client settings, make sure that Aktualizr is stopped on the board so it won't interfere while creating configuration files:

# systemctl stop aktualizr

Starting Aktualizr

After applying the changes described below, you need to re-start Aktualizr with the commands below:

# systemctl start aktualizr

You can use journald to view the logs and monitor the operation of the update client:

# journalctl -f -u aktualizr*

Disabling and Enabling Aktualizr

If you don't want Aktualizr to start on every reboot, disable it. If you want to revert it, enable it again:

# systemctl disable aktualizr
# systemctl enable aktualizr

Modifying the Settings of Torizon OTA Client

In this section, we will show how to make common adjustments to Aktualizr settings.

Aktualizr is configured via .toml configuration files. It searches the following directories for files with a .toml extension:

  • /usr/lib/sota/conf.d/
  • /etc/sota/conf.d/

It then processes all the .toml files it finds in alphabetical order. If a config option is specified in multiple files, the last entry overrules any previous entries. TorizonCore includes the following config files by default in /usr/lib/sota/conf.d/:

  • 20-sota-device-cred.toml
  • 30-rollback.toml
  • 40-hardware-id.toml
  • 50-docker-compose.toml
  • 60-polling-interval.toml

If you want to override config options in those files, you should create a config file in /etc/sota/conf.d/ to add or modify config options.

Once you have changed the configuration options to the way that suits you, you can apply the changes to a custom TorizonCore image that you can use for production programming, following the instructions in Capture Changes in the Configuration of a Board on TorizonCore.

Configuring for Offline Updates

Aktualizr's standard configuration is to monitor update requests coming through the Remote Updates method, because of this an Offline Update request will not work out-of-the-box. To reconfigure a device for using Secure Offline updates, you must create or modify a configuration file at /etc/sota/conf.d/. If the folder doesn't exist, you can create it with the following command:

# sudo mkdir -p /etc/sota/conf.d/

Use the following configuration in the file:

# cat /etc/sota/conf.d/100-offline-updates.toml
[uptane]
enable_offline_updates = true
enable_online_updates = false
offline_updates_source = /var/rootdirs/media/USB/update

Note that offline_updates_source is the folder that aktualizr will monitor for Offline Updates. In this case, it’s the folder where our USB drive auto-mounts, being update the name you have on your update medium volume.

Note: You also have to restart the aktualizr-torizon client for the changes to take place.

Configuring the Polling Frequency

In TorizonCore, Aktualizr is configured to poll the server for new updates every 5 minutes. To modify this behavior, we can change the uptane.polling_sec option. For example, to change the polling frequency to 1 hour, you could create a configuration file in /etc/sota/conf.d/ called 61-custom-polling-interval.toml:

# cat <<EOF > /etc/sota/conf.d/61-custom-polling-interval.toml 
[uptane]
polling_sec = 3600
EOF

After applying the changes, don't forget to Restart Aktualizr.

Configuring the Hardware ID

When Toradex publishes our TorizonCore images to the OTA server, their metadata includes a list of which hardware IDs they are compatible with. This ensures that you don't accidentally send an image built for a colibri-imx7 to an apalis-imx8. This hardware compatibility is enforced by the update client: if your device reports that it is an apalis-imx8, it will refuse any update images that don't list apalis-imx8 as a compatible hardware ID.

Normally, it's best not to modify the hardware ID your device reports. If you give it a custom hardware ID, it will no longer accept stock TorizonCore images. But sometimes, that's exactly what you want: for example, if you have several different SKUs that need slightly different OS images, or if you have products with the same SoM but a different carrier board or peripherals. In those cases, you might want to make sure that each SKU or variant has its own hardware ID, and then publish a customized TorizonCore build for each one; that way you can make sure that nobody can accidentally create an OTA update that sends incompatible OS images.

To achieve that, you need to do two things:

  • Modify the Aktualizr config on your device image to report a custom hardware ID
  • Upload the customized image to Torizon OTA along with metadata indicating it's compatible with your custom hardware ID

Configuring the Hardware ID on TorizonCore

To configure the hardware ID in the Aktualizr config, you need to set the provision.primary_ecu_hardware_id config value. For example, if you wanted your device to be registered with the hardware ID apalis-imx8-sku001, you could create a configuration file in /etc/sota/conf.d/ called 41-custom-hardware-id.toml:

# cat <<EOF > /etc/sota/conf.d/41-custom-hardware-id.toml 
[provision]
primary_ecu_hardware_id = apalis-imx8-sku001
EOF

Note: Because this is a change to provisioning configuration, you will need to re-provision the device if you want to change it on a live device. It's highly recommended to apply the changes to a custom TorizonCore image that you can use for production programming, following the instructions in Capture Changes in the Configuration of a Board on TorizonCore. That way, your devices will use their custom hardware ID as soon as they are programmed.

Configuring Custom Hardware IDs for Torizon OTA Images

When sending a custom image to the OTA server, you can specify a list of hardware IDs it is compatible with. Using the push command of the TorizonCore Builder Tool, you can specify one or more --hardwareid arguments to override the image's compatible hardware IDs.

$ torizoncore-builder push my-branch --credentials credentials.zip --package-name my-custom-image --hardwareid apalis-imx8-sku0001 

Each image can have several hardware IDs assigned to them. To add multiple hardware IDs to a custom image, use the --hardwareid argument once for each hardware ID.

$ torizoncore-builder push my-branch --credentials credentials.zip --package-name my-custom-image --hardwareid apalis-imx8-sku0001 --hardwareid apalis-imx8-sku0003 --hardwareid apalis-imx8-sku007-beta

Remember, Torizon OTA will only permit you to install an image on a device if the hardware IDs match.

Configuring the loglevel

Aktualizr produces logs at different levels of detail depending on the loglevel setting:

  • trace (0) logs everything, including the content of every HTTP request
  • debug (1) has extra information useful for debug purposes, but is not as verbose as trace
  • info (2) is the default, and logs basic information about normal and exceptional Aktualizr operations
  • warning (3) only logs warnings and other exceptional information
  • error (4) only logs errors
  • fatal (5) only logs fatal errors

If you are troubleshooting an issue with the OTA client, it may be helpful to increase the loglevel. To do this, we can change the logger.loglevel option. For example, to enable trace logging, you could create a configuration file in /etc/sota/conf.d/ called 99-custom-loglevel.toml:

# cat <<EOF > /etc/sota/conf.d/99-custom-loglevel.toml 
[logger]
loglevel = 0
EOF

Note: More detailed loglevels, especially trace, can write a large volume of log data if left on for a long time. We recommend leaving the loglevel at the default except when specifically needed for troubleshooting.

After applying the changes, don't forget to Restart Aktualizr.

Allowing and Blocking Updates

You can lock Aktualizr to avoid receiving new updates on the client.

Every time before applying an update, Aktualizr attempts to acquire a lock on /run/lock/aktualizr-lock using flock.

To help control when updates are applied, you can have a custom code in your application(s) that acquires and releases this lock (see flock (2) man page for more details).

If you are using the command line, it is possible to apply an advisory lock on a open file using the command flock. For example, the command below will apply an advisory lock on /run/lock/aktualizr-lock for 30 seconds:

# sudo flock --verbose -x /run/lock/aktualizr-lock -c "sleep 30"

Enabling and Disabling Automatic Reboot

By default, TorizonCore is configured to automatically reboot the device after a successful update of the operating system. If you want to disable this feature, run the commands below:

# systemctl stop ostree-pending-reboot.path
# systemctl disable ostree-pending-reboot.path

In case you disabled the automatic reboot for system updates and want to re-enable it, run the commands below:

# systemctl enable ostree-pending-reboot.path
# systemctl start ostree-pending-reboot.path

After applying the changes, don't forget to Restart Aktualizr.

Changing the Default Reboot Command

By default, Aktualizr will use /sbin/reboot to reboot the device after a successful update (if enabled). To modify the command used to reboot the device, we can change the bootloader.reboot_command option. For example, you could create a configuration file in /etc/sota/conf.d/ called 21-custom-reboot.toml pointing to a custom reboot script:

# cat <<EOF > /etc/sota/conf.d/21-custom-reboot.toml 
[bootloader]
reboot_command = "/my-custom-reboot-command"
EOF

Advanced Aktualizr Config Options

Aktualizr has many more config options available. For more details, please consult the project's official documentation.

Setting the Device ID and Name During Provisioning

When you provision a device with Torizon OTA, it will be assigned a Device ID and a Device Name. Both Device ID and Device Name must be unique in your account. By default, Device ID is derived from the SoM model and serial number, and Device Name is generated from a random wordlist.

If you want to provide those values yourself, you can add them to the provisioning command by adding the -d option for Device ID and the -n option for Device Name. Your provisioning command will normally be of the form docker run -v /:/som_sysroot --network=host --privileged -it torizon/torizon-provisioner:0.0.11 provision-device -t ${token} && sudo systemctl restart aktualizr fluent-bit. To set the custom Device ID and Device Name, you would change that to docker run -v /:/som_sysroot --network=host --privileged -it torizon/torizon-provisioner:0.0.11 provision-device -t ${token} -d "my-device-id" -n "My Product 0053" && sudo systemctl restart aktualizr fluent-bit. Note that the added parameters need to come before the &&.

Configuring Secondaries (Application Container Updates)

Note: regular users of Torizon OTA don't need to configure secondaries, they are already configured by default.

Secondary is a concept in Uptane-compatible OTA systems that make it possible to update not only the main operating system but also other firmware and devices connected to it. TorizonCore uses secondaries to update containers via docker-compose files.

If you want to override the default configuration, take it as an example and create custom configuration files under /etc/sota/. To get the content from the default files, execute the commands below:

# cat /usr/lib/sota/conf.d/50-docker-compose.toml 
[uptane]
secondary_config_file = "/usr/lib/sota/docker-compose.json"

# cat /usr/lib/sota/docker-compose.json
{
    "docker-compose": [
        {
            "partial_verifying": false,
            "ecu_hardware_id": "docker-compose",
            "full_client_dir": "/var/sota/storage/docker-compose",
            "ecu_private_key": "sec.private",
            "ecu_public_key": "sec.public",
            "firmware_path": "/var/sota/storage/docker-compose/docker-compose.yml",
            "target_name_path": "/var/sota/storage/docker-compose/target_name",
            "metadata_path": "/var/sota/storage/docker-compose/metadata"
        }
    ]
}

After applying changes, don't forget to Restart Aktualizr.

Update Checks and Rollbacks

Greenboot (Generic Health Check Framework) is a Fedora project that helps manage systemd services health. TorizonCore uses Greenboot as a framework to make update checks and rollbacks more flexible and manageable by the user.

By default, TorizonCore will consider a successful boot if the boot-complete systemd target is successfully executed. This is because the main operating system services required for proper operation, including the Docker daemon, are inside boot-complete.target. And if boot-complete.target fails during an update, TorizonCore will automatically reboot, and after three tries, it will rollback to the previous operating system version.

In case you want to add additional checks to confirm a successful update, you can add shell scripts to /etc/greenboot/check/required.d/. As a convention, the script name should start with two numbers and finish with .sh (Example: 01_check_system.sh). Scripts in /etc/greenboot/check/required.d/ will be executed as part of the boot health checks.

If the scripts in /etc/greenboot/check/required.d/ are successfully executed, and the boot-complete.target is successfully started, the system will enter in the GREEN state. In this case, the greenboot-task-runner service will be triggered, and user-defined scripts inside /etc/greenboot/green.d/ will be executed. These scripts can be used to execute post-install operations during a successful update, but be aware that they will run every time the operating system boots.

Now, if one of the scripts in /etc/greenboot/check/required.d/ fail (exit code is not 0), the boot-complete.target will also fail, and the system will enter in the RED state. In this case, the redboot-task-runner service will be triggered, and user-defined scripts inside /etc/greenboot/red.d/ will be executed. These scripts can be used to execute post-install operations during a failed update, but be aware that they will run every time the operating system boots (when boot-complete.target fails). After redboot-task-runner service finishes execution, the redboot-auto-reboot service is triggered, and this service will run a script that will reboot the system (in case an update is in progress). After 3 reboots, the system will rollback to the previous operating system version.

To confirm if a boot was successful (GREEN status) or if it failed (RED status), you can check the status of the greenboot-status systemd service:

$ systemctl status greenboot-status
* greenboot-status.service - greenboot MotD Generator
     Loaded: loaded (/usr/lib/systemd/system/greenboot-status.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2021-04-26 12:50:04 UTC; 16min ago
    Process: 809 ExecStart=/usr/libexec/greenboot/greenboot-status (code=exited, status=0/SUCCESS)
   Main PID: 809 (code=exited, status=0/SUCCESS)

Apr 26 12:50:04 apalis-imx6-05039068 systemd[1]: Starting greenboot MotD Generator...
Apr 26 12:50:04 apalis-imx6-05039068 greenboot-status[814]: Boot Status is GREEN - Health Check SUCCESS
Apr 26 12:50:04 apalis-imx6-05039068 systemd[1]: Started greenboot MotD Generator.

Alternatively, you can list the content of the /run/boot-status file:

$ cat /run/boot-status
Boot Status is GREEN - Health Check SUCCESS

For more information about the Greenboot framework, please see the project's website.

Troubleshooting

To see in more detail what Aktualizr is doing, you can stop the systemd service and start Aktualizr manually, e.g. with an increased loglevel for debugging:

# systemctl stop aktualizr
# aktualizr-torizon --loglevel 1
# systemctl stop aktualizr
# aktualizr --loglevel 1

To provision the device again, make sure to stop Aktualizr, remove the device and start Aktualizr again:

# rm /var/sota/sql.db
# rm -rf /var/sota/storage/

Common Issues

If you see the following message:

response http code: 400
response: "An error occurred: Missing entity: Ecu"
could not put manifest

Make sure to properly delete the storage of the secondary (see above).