TorizonCore Builder Issue Tracker

Description: As reported on Hello-mod kernel module doesn’t build Building kernel modules for Verdin AM62 fails with error "make[1]: *** No rule to make target 'arch/arm64/tools/gen-sysreg.awk', needed by 'arch/arm64/include/generated/asm/sysreg-defs.h'. Stop."

Workaround: This has been fixed and you must use the Torizon OS monthly 202403 or newer.

Description: As reported on the TorizonCore Builder issue tracker on GitHub, the command "platform lockbox" fails on non-host networks.

Description: The error "subprocess.CalledProcessError: Command 'PATH=$PATH:/storage/toolchain/arm-gnu-toolchain-11.3.rel1-x86_64-arm-none-linux-gnueabihf/bin make -C /storage/linux ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- modules_prepare' returned non-zero exit status 2." happens when trying to build an out-of-tree module for a TorizonCore 6.3.0 image.

Workaround: Use the early access release of TorizonCore Builder until the stable release 3.8.0 is available.

Description: When taking a Toradex Easy Installer image as input, running it through the build command, and then serving it (by running "images unpack" followed by "ostree serve"), you should be able (on a device) to fetch the refs and commits from the served repository. However, what one gets on a device when trying to list the remote refs is "error: Remote refs not available; server has no summary file".

This is an advanced use case, and most users of TCB should not be affected.

Description: If you try to create a lockbox with a container image that contains a dot in the name, for example "my.remote.registry:4567/user.name/image-name:image-tag", TorizonCore Builder fails with "AssertionError: Could not fetch manifest of"

Workaround: Use the Early Access Version of TorizonCore Builder, which has the bug fixed, while the 3.8.0 version or later is not released.

Description: When using a command that canonicalizes a Docker Compose file, such as the "platform push", the error "Error: Can't determine digest for image 'image-tag'" will happen if the Docker Compose references at least one container image that is OCI compatible instead of Docker compatible.

Description: As posted in the community thread Torizoncore-builder maximum version for docker-compose.yml, TorizonCore Builder fails when trying to bundle some Docker Compose files with the 3.x specification, due to missing properties such as "device_cgroup_rules".

Description: TorizonCore Builder supports using private secure registries with their own certificates, as documented on remarks about using private registries. A limitation is that this support is enabled through the standalone TorizonCore Builder commands, but not extended to the "build" command.

Support has been added through an external contribution on the project's GitHub, reviewed, and approved. Documentation is available in the TorizonCore Builder "build" command article, under the output --> easy-installer --> bundle --> ca-certificate.

Description: The TorizonCore Builder command "dt checkout" is not yet supported for TorizonCore 6.
As we re-evaluate the device tree and overlays workflow with TorizonCore Builder, and the potential deprecation of the "dt checkout" command, we document a workaround instructing you to directly pull the Git repositories with the device trees and overlays source-code.
Please consider that, from TorizonCore 6 onward, the "dt checkout" may not become available. If it happens, a new workflow will be available instead. If it doesn't happen, at some point "dt checkout" support will be reintroduced.

Update: the workaround is documented on TorizonCore Builder Tool - Customizing TorizonCore Images - Device Tree.

Description: TorizonCore Builder is not yet updated to handle TorizonCore 6 images as input. This leads to failure in two commands: (1) in the "build" command when using the "toradex-feed" option, and (2) in the "images download" command.

The resulting error is "Error: Don't know how to handle a major version of 6".

Workaround: For the "build" command, either manually download the TorizonCore 6 image and use the "local" property or use the image URL and the "remote" property. For the "images download" command, manually download and unpack the image.

Download links are available on Toradex Download Links (Torizon, Linux BSP, WinCE and Partner Demos).

Description: In the Torizon Platform Services web UI, it is possible to set compatibility on application packages, describing a list of compatible OS packages and versions. We plan to make it possible on the TorizonCore Builder command line, allowing you to integrate it into your CI scripts.

Update: adding compatibility based on the package version isn't trivial. To overcome it, it was decided to support initially setting up the compatibility based on the hash instead of the version. The hash is available in the web UI, making it easy for you to find out its value.

Description: Currently, the "tcb platform push" does not accept advanced registry access, for example access to private registries. This is an issue when doing canonicalization, which requires access to the registry.

Workaround: Here is a suggested workaround which hasn't been fully validated:

• Use the bundle command on the docker-compose file to be used in the lockbox.
• The bundle command will fetch the Docker images (unnecessarily) but it will also create a docker-compose in canonical form, by default in bundle/docker-compose.yml; even though it doesn't have the .lock.yml extension this file is actually in canonical form.
• Copy the file to a name having the .lock.yml extension.
• Push the renamed/copied file to OTA with platform push; if setting the package name (with the --package-name switch) make sure to end it with the .lock.yml extension).
• The pushed compose file should then be suitable for use with the lockbox.

Description: At the moment, we only support using insecure private registries on TCB, with the "--dind-param" flag of the "tcb bundle" command, as documented in the example in the commands manual. It has been requested by customers to support secure private registries as well.

Update: this has been implemented. Learn more on the remarks about using Docker registries.

Description: The documentation that lists files ignored by the "isolate" command is outdated. For example, the files "/etc/passwd" and "/etc/shadow" are indeed captured, since you are allowed to customize the default password.

Description: In the Torizon Platform Services web UI, it is possible to set a description for packages. We plan to make it possible on the TorizonCore Builder command line, allowing you to integrate it into your CI scripts.

Learn how to use the "--description" flag in the platform push documentation.

Description: As of TorizonCore 3.5.0, the setup script does not get automatically updated. This is an issue because sometimes changes in the setup script are required for using newer versions of TorizonCore Builder.

Update: after some investigation and customer feedback, it was decided that the script will print a message asking you to update it. This message will provide the exact command you must run, as to make the update easy to execute.

Description: Toradex Easy Installer images may contain an End User License Agreement (EULA). The TorizonCore images have NXP’s EULA included. Therefore, a mechanism for accepting the EULA with TorizonCore Builder must be added, as to ensure that you have actively accepted such a license even in a production programming environment, where it isn't required to accept the EULA on every installation.

Description: As a final step to releasing the secure offline updates MVP, public documentation on how to use it will be written.

Update: documentation is now available on How to use secure offline updates with TorizonCore.

Description: The "platform push" (formerly "images push") command allows you to set the package name and version. At the moment, the default package name is set to "docker-compose_file.yml", which is not a sane default. We'll fix it by using the base input file name instead. For example if the input file is "my-docker-compose.yml", the base name will be "my-docker-compose".

Description: The push command has the ability to canonicalize Docker Compose files, creating their corresponding lockfiles with hashes instead of tags. This is a good practice and particularly important for the upcoming secure offline updates. At the moment, canonicalization is not always happening.

Description: An OSTree deployment can be compatible with multiple Hardware IDs.

At the moment, the "push" command has an undocumented parameter named "--hardwareid" that only sets the Hardware ID if the value is missing in the OSTree metadata.

We will change the default behavior so that "--hardwareid" can be used to override the default Hardware ID by the user, and can be passed multiple times to support more than one Hardware ID.

Description: Toradex Easy Installer supports two features that are very useful for production programming: autoinstall and auto-reboot.

To make it possible to easily set these properties in a custom TorizonCore image, TorizonCore Builder must provide an interface for these capabilities.

Description: The TorizonCore Builder "ostree serve" command is not working on Windows machines, due to a limitation of the Docker host network driver.

When sourcing the setup script, you must pass an optional flag that will expose the given port to the host.

Description: When passing the switch --ostree-repo-directory to the ostree serve command, for example "torizoncore-builder ostree serve --ostree-repo-directory ostree_repo/", the tool throws an error:
"UnboundLocalError: local variable 'storage_dir' referenced before assignment"

Description: The TorizonCore Builder "deploy" command uses the hardcoded port 8080 to deploy customization to a board. Sometimes this port is not free for some reason and the command fails.
We will improve the logic and let TorizonCore Builder find and use a random free port for the deployment.

Description: The "dt checkout" command conveniently clones the device-trees repository in the correct branch for a given input image. It does not provide an option to update the repository given that some updates were made upstream. This task adds the option "--update" that will attempt to pull the latest.

Workaround: Use Git commands as "git pull" to update the repository.

Description: TorizonCore Builder is capable of setting OSTree name and version when pushing commits to the Torizon Platform Services.
This task documents how you can use this feature. Learn how to use it in the article Signing and Pushing TorizonCore Images to Torizon OTA.

Description: TorizonCore has two splash screens: one that shows when booting, and another that shows on reboot or shutdown.
It was reported that the reboot/shutdown splash was not being updated by TorizonCore Builder. This ticket address it and makes sure that both splash screens are updated.

Description: Create a --remote-port parameter to make it possible to set the SSH port TorizonCore Builder will use to connect to devices running TorizonCore.

Description: In TorizonCore Builder, the deploy and combine commands are responsible to generate Easy Installer images.

To be able to install them using the LAN in an easy way, we have introduced the `images serve` command, that brings up a local Easy Installer image server.

Description: When using "platform push" to upload a generic package to the Cloud servers, the command sometimes fails with the following error message:
com.advancedtelematic.libtuf.http.TufServerHttpClient$RoleChecksumNotValid$: could not overwrite targets, trying to update an older version of role. Did you run `targets pull` ?
Error (2) running uptane command "uptane-sign" with arguments "['targets', 'push', '--repo', '/deploy/tuf-repo']"

Workaround: This has been fixed in early access releases of TorizonCore Builder.

Description: The error "TypeError: 'levels' is an invalid keyword argument for print()" is thrown when you set a log level that is not valid.

Workaround: Use a valid log level.

Description: As of TCB 3.11, TorizonCore 5 will not be supported anymore. It may still work, but issues with TorizonCore 5 introduced in 3.11 and newer will not be fixed anymore.

For as long as TorizonCore 5 is still supported in maintenance mode, patch releases of TCB 3.10 may be released with hotfixes for critical bugs that only affect TorizonCore 5 customization with TCB.

Description: As reporeted on the Toradex Community, the parsing of tcbuild.yml silently ignored a syntax error, leading to a hard-to-debug situation.

Workaround: Use TCB early access, where the issue is fixed, or wait for TCB 3.11.0 release.

Description: After recent changes in the backend, TorizonCore Builder is failing when we try to push docker-compose files to OTA.

Description: There is an issue when using the isolate command when there are symlinks to be isolated and using the --changes-directory argument.

Description: Improve the error messages on TorizonCore Builder when "unpack" hasn't been run yet.

Description: The `images` command has an option to clear the storage directories. It does not clear the /splash directory, though. This ticket aims to fix it.

Description: TorizonCore Builder runs inside a container, and by default it creates files owned by root. This is a bit annoying since we sometimes want to edit or erase some files with our user in the development PC. We investigate and implement a way to make files belong to the user and not root.

Description: For the commands that require device connection, make --remote-username and --remote-password parameters optional.

Description: Torizoncore builder union command fails with enabled systemd services.

Workaround: Disable the systemd service before running the "isolate" command. This will require you to manually enable it after deployment.

Description: If you have a symlink in the changes directory that points to a file using the full path, TorizonCore Builder’s union command fails.

Description: The interface of the union command was simplified to make it easier to use:

• the --extra-changes-directory switch was removed and its functionality moved to --changes-directory;
• the switch --union-branch no longer exists either - the branch name is now a positional parameter;

Description: When uncompressing a local image, TorizonCore Builder leaves behind the uncompressed directory, even though it isn't used anymore. In this ticket we clean up the uncompressed directory.

Description: TorizonCore Builder should be able to save and restore credentials when isolating files from the device and union-ing files to another TorizonCore image.

Description: There's an issue with the bundle sub-command where the the tar file output that gets created is inconsistent. The proper expected name that should be created is docker-storage.tar.xz, yet sometimes a docker-storage.tar is created instead.

Description: Upon the first login, it is mandatory to change the password. This modification is not captured by TorizonCore Builder's isolate command.

Description: The isolate command fails because of spaces in the name of files.

Description: TorizonCore Builder cannot fetch the new NXP BSP branch (toradex_5.4-2.3.x-imx) from device trees repository.

Description: If you attempt to bundle a container with TorizonCore Builder on Windows from WSL 2, it fails.

Description: With the introduction of the `kernel build_module` command, the size of TorizonCore Builder increased by a factor of ~10x, due to the download of a duplicated toolchain. This commit addresses this issue and also removes the toolchain from TorizonCore Builder, leaving the download only to those users who need to execute the `kernel build_module` command. This approach increases the time for the first run of the command, while leaving TorizonCore Builder much smaller by default.

Description: TorizonCore Builder seems to not be listing all compatible overlays for some machines, like Apalis iMX8.

Description: Make it possible to set custom kernel arguments using TorizonCore Builder.

Description: Add 'images download/unpack' commands in TorizonCore Builder to automatically download Tezi images from Artifactory (based on the version running on a device) and unpack for further customizations.

Description: Create a shell script to automate environment configuration and warn the user when a new TorizonCore Builder version is available.

Description: To increase awareness about the TorizonCore Builder documentation, we add a hyperlink to it at the end of the `--help`.

Description: If you execute `torizoncore-builder deploy dt_changes` and pass the board hostname as parameter for `--remote-host` instead of the board IP, for example `apalis-imx8-06543292.local`, it does not work.

Workaround: It has been fixed in TorizonCore Builder 2.2.0. Use this version of the tool.

Description: Refactor device tree commands and adapt them to use overlays.txt.

Description: After using TorizonCore Builder to customize an Easy Installer image with a device tree overlay applied, the generated Easy Installer image has incorrect permissions in the /home/torizon directory.

Description: Update the TorizonCore Builder container to Debian Bullseye.

Description: Torizoncore Builder generates an invalid compose file after bundle command.

Description: When TorizonCore Builder replaces the tags in docker-compose file with concrete sha256 hashes, it drops the MACHINE variable.

Description: We want to create the "dt list-overlays" (or similar name) sub-command to list available (and compatible with the module) device tree overlays.

Description: Create documentation on how to use TorizonCore Builder to apply device trees

Description: Add additional metadata to OTA targets.json.

Description: Add feature in TorizonCore Builder to set a splash screen image.

Description: Add the push command to torizoncore-builder tool to upload changes in an OSTree repository to Torizon OTA.

Description: TorizonCore Builder is a tool that helps to integrate Docker containers and user's customization in a production image.

Description: Some containers such as our base containers with Vivante binary drivers need the customer to accept an EULA. We should support adding an EULA to the Tezi image as part of the pre-provisioning process (in torizoncore-builder).

Description: TorizonCore Builder does not support the ZFS filesystem. If you setup a computer with a Linux distro that uses ZFS - for example, Ubuntu 22.04 LTS with the experimental ZFS support enabled - you will see errors such as "ConnectionRefusedError: [Errno 111] Connection refused" when running the bundle command.

On TorizonCore Builder we run Docker in Docker (DinD) and use the OverlayFS driver. As stated on Docker storage drivers - Supported storage drivers per Linux distribution, it is the recommended storage driver and requires XFS or ext4 as the filesystem where "/var/lib/docker" (from the DinD) is located. In practice, this means that you have to use a filesystem compatible with OverlayFS on your computer for TorizonCore Builder to work, even if you don't run the Docker daemon itself with OverlayFS.

At the moment, our plans are to make the error message more intuitive, stating that ZFS is not supported. We will keep monitoring the adoption of ZFS and possibly other filesystems and, as they gain traction and become the default on major Linux distros, we might consider adding support for them in upcoming TorizonCore versions.

Workaround: Use a Linux distro with either ext4 or XFS (with `d_type=true` enabled) as the filesystem

Description: As reported in our community, with TorizonCore Builder, If you bundle a container that sets extended attributes on files inside the container (for example, using the "setcap" command), the container will fail upon execution with the error "SIOCSIFFLAGS: Operation not permitted" or another related error.

If you remove the bundled container and manually pull it from Docker Hub, then try to run it again, the same error will not happen.

While this bug was investigated, we came to docker build does not preserve xattrs in the generated image, and after several tests such as setting capabilities and building the container with the classic Docker vs Buildkit, we concluded that:

• Docker does not keep all extended attributes that are set in a Dockerfile, regardless of using Buildkit or the Classic Docker builder, and using Docker directly (without Compose).
• Even some capabilities that are kept by Docker get lost when bundling a container with TorizonCore Builder

Workaround: For this specific bug: set the suid for executable files rather than using setcap. For example use "chmod u+s" instead of "setcap" seems to work as well and may carry through the bundle process.

In general, do not rely on xattrs or any feature based on them inside a container. Capabilities and ACLs are features that may be set via xattrs and can also be set via docker-compose properties, which is the recommended way to set them in a portable and reliable way.

Description: Activating device tree overlay fails on Apalis i.MX8

Description: It is possible to run TorizonCore Builder without using a Docker volume, by passing some extra flags to the setup script: "source tcb-env-setup.sh -d -s /srv/builder-tool/storage".

When using the "tcb build" command, though, it does not create the output directory as expected.

It has been reported on the TCB GitHub issues.

Workaround: Use docker volumes (the default option) when using the build command.

Description: The TorizonCore Builder "images serve" command was added to TorizonCore Builder on version 3.2.0. It does not work on Windows due to two limitations: (1) the Docker host network driver is not supported on Windows and (2) the multicast DNS (mDNS) used by Avahi does not work.

Workaround: Take the output image from TorizonCore Builder and use an SD Card/USB stick to install it to a Toradex SoM.

Alternatively, use a Linux machine.