TorizonCore Builder Issue Tracker

Description: On TCB 3.8.0, when you use the "platform lockbox" to create a lockbox that has been defined in the Platform Services, it may fail with the error "TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'" in case your lockbox contains an OS update with a custom OS image that you pushed to the Platform Services.

Description: As reported in the Toradex community, downloading lockboxes with the early access version of TCB (c.a. July 15th, 2023) fails with the error "Could not fetch file".

Workaround: Use the latest early access TCB release, or a stable TCB release.

Description: When you try to push an OCI image to the Torizon Platform, or create a lockbox with one, TorizonCore Builder fails due to lack of support for OCI images.

Description: TorizonCore Builder and the Torizon Platform are limited to only accepting files that end in ".lock.yml" for creating lockboxes, used in offline updates.

Description: When using the "platform push" command to push docker-compose files with canonicalization (i.e. when the --canonicalize or --canonicalize-only switches are passed), TorizonCore Builder has to access the registries where the referenced images are stored.
For container images available on container registries whose certificate is signed by a known CA (except Docker Hub) - for example, GCR - the command fails.

Description: Until this feature is implemented, only OS and application updates are supported in TorizonCore. As of this feature, any other subsystem can be updated, such as but not limited to: Cortex-M, system configuration, application metadata, firmware for external HW.

Update: learn how to push any firmware to the Platform Services on Signing and Pushing TorizonCore Packages to Torizon Platform Services.

Description: Due to a mismatch between how the client and the server calculate the hashes of packages names and versions, updates of packages that contain multi-character bytes in their name or version fail.

If you try to use unsupported characters, TorizonCore Builder will fail and let you know.

Description: The tip of the TorizonCore Builder branch is named "early access". Toradex builds this version every weekend, and you can leverage it for critical bug fixes or feature previews. There is no guarantee that what is on the early access has any documentation available yet or that it even works at all. Partially finished work may be included in early access before a release.

Learn about when you should use it, and how to use it, in the tcb-env-setup project's README.

Description: Until the release of TorizonCore 6, the GCC toolchain used by TorizonCore Builder matched the version used to build TorizonCore 5 images. It must be updated to match TorizonCore 6 images so the likelihood of compiler issues due to version mismatch is reduced.

Update: this version bump is being implemented in a retro compatible way. If you are using TorizonCore 5 as input image, TorizonCore Builder will still use the older toolchain version that matches it, and if you are using TorizonCore 6 as input image, the newer toolchain will be automatically selected.

Description: Until TorizonCore 3.6.0, abbreviated commands were accepted by the TorizonCore Builder command-line argument parser.

This behavior is undocumented and may lead you to write scripts that may get broken in the future if we add new options that may conflict with existing ones (taking a current example: --login vs --login-to). We decided that being more strict leads to a higher quality of any scripts you may integrate TorizonCore Builder on.

Workaround: This is an undocumented feature and must not provide any breaking changes. If you see any issues, it may be due to a typo. Please double-check your scripts.

Description: Currently, the "tcb platform lockbox" does not accept advanced registry access, for example access to private registries. This is an issue when doing canonicalization, which requires access to the registry.

Description: For an unknown reason, the "--login" parameter of "tcb bundle" is being ignored in a CI environment.

Update: this bug happened once but was not reproducible. We have improved automated tests to validate the "--login" parameter further. If you are able to reproduce the bug, please let us know.

Description: The TorizonCore Builder Setup Script relies on the Docker API to fetch the latest available version of the tool. The version of the API used has been deprecated, thus breaking the script.

Workaround: This has been fixed. Please re-download the setup script.

Description: When trying to use autocompletion feature of TCB with Zsh (by pressing Tab), it does not work and the following error is printed "_alternative:shift:14: argument to shift must be non-negative".

Keep in mind that Zsh is not fully supported. We fix bugs when reported and reasonable to maintain, but otherwise all our tests focus on Bash.

Workaround: Until we fix it, either use Bash, or don't use autocompletion on Zsh.

Description: Sometimes TorizonCore Builder fetches wrong metadata from the Platform Services server and offline updates fail. While this seems to be server-related, we plan to investigate it further and confirm the root cause, improving TorizonCore Builder if possible.

Update: it was confirmed as an issue on the Platform Services, and fixed there. No updates were required on TorizonCore Builder.

Description: Autocompletion for "tcb bundle" shows old flags such as "--directory" instead of "--bundle-directory", and "tcb deploy" shows "--sysroot-directory" instead of "--deploy-sysroot-directory".

Description: When running "tcb push", you have two options: 1) push an OSTree reference, also known as OS update and; 2) push a docker-compose.yml file, also known as application update.

Currently, the autocompletion only takes into consideration 1), and therefore it does not suggest YML files as possible autocomplete options.

Description: When bundling a Docker Compose file with TorizonCore Builder 3.5.0, a stack trace is printed and the error "KeyError: 'dir'" is displayed

Description: To create a custom TorizonCore image with provisioning data, it is first required that you fetch such data from the Platform Services.

The command "platform provisioning-data" fetches this data. Learn how to use it on The Platform Command.

Description: There seems to be a discrepancy between the "build" command documentation and what actually happens. Specifically in regards to adding device tree overlays. In the overlays section of the "tcbuild.yaml" there’s a "clear" property that does the following according to the documentation:

clear (boolean, optional): Boolean indicating whether or not to ignore all the binary overlays (dtbo files) that may already be present in the base image. This property is useful only if no custom device tree was set. Default: false.

If "clear" is not specified in "tcbuild.yaml" then pre-existing overlays will not be cleared by default. However, the actual behavior is that if "clear" is not specified in "tcbuild.yaml" then pre-existing overlays will be cleared by default.

Description: When running the bundle command on Windows, an error happens and it says "ConnectionRefusedError: [Errno 111] Connection refused".

Workaround: Run the setup script like so: "source tcb-env-setup.sh -- --network=host".

Description: At the moment, if you (1) use TorizonCore Builder to pre-provision (bundle) containers into a Toradex Easy Installer image, and (2) upload the same version of the containers to the Torizon Platform Services, your device will not report to the Platoform that it has that SW version installed.

We are working on a fix on TorizonCore Builder that will allow existing TorizonCore versions to provide this version information to the server. We are also working on a fix in Aktualizr that will allow devices that are already provisioned to be updated to newer versions of TorizonCore Builder and then properly report the version to the server.

Description: When using variable substitution with the "build" command, the YAML parser do validation before doing the substitution. This results in the error "Parsing errors found".

Description: To allow provisioning devices at scale to the Torizon Platform Services, we will make it possible to include provisioning credentials into a custom TorizonCore image.

To learn more, check out the "provision" properties in the "output section" of the build command manual.

Description: If the Linux distro installed in your development PC has already moved to using cgroup rules v2, you will get the following error when running the TorizonCore Builder "bundle" command: "docker.errors.NullResource: Resource ID was not provided".

Description: Currently, in TorizonCore, default kernel arguments are not updateable. That means when we deploy a new OSTree commit, the kargs defined in this new commit will not be used during deployment.

Description: For the offline-updates MVP - and also as a best practice - a canonical Docker Compose file is required. It is a Docker Compose file that does not have comments or blank lines, and instead of tags, it uses SHA hashes for the versions of all containers. The purpose is to ensure the version of all packages for your application will be ensured, or in other words, locked.
Due to this, we are introducing the concept of a docker-compose.lock.yaml file, that is the canonical version of a regular docker-compose.yaml file.
In this task, we have introduced this feature along with documentation explaining how to use it, and why it is important.

Description: Make it possible to set the storage directory in the setup script, and change the --storage-directory parameter to be hidden (it will be deprecated later).

Description: When running the `deploy` command, an image compatible with Toradex Easy Installer is created. Some metadata contained in image.json, such as the image size, are not updated by TorizonCore builder, and it uses the same values as the input image.

Description: When using "platform push" to upload a generic package to the Cloud servers, the command sometimes fails with the following error message:
com.advancedtelematic.libtuf.http.TufServerHttpClient$RoleChecksumNotValid$: could not overwrite targets, trying to update an older version of role. Did you run `targets pull` ?
Error (2) running uptane command "uptane-sign" with arguments "['targets', 'push', '--repo', '/deploy/tuf-repo']"

Workaround: This has been fixed in early access releases of TorizonCore Builder.

Description: The error "TypeError: 'levels' is an invalid keyword argument for print()" is thrown when you set a log level that is not valid.

Workaround: Use a valid log level.

Description: As of TCB 3.11, TorizonCore 5 will not be supported anymore. It may still work, but issues with TorizonCore 5 introduced in 3.11 and newer will not be fixed anymore.

For as long as TorizonCore 5 is still supported in maintenance mode, patch releases of TCB 3.10 may be released with hotfixes for critical bugs that only affect TorizonCore 5 customization with TCB.

Description: As reporeted on the Toradex Community, the parsing of tcbuild.yml silently ignored a syntax error, leading to a hard-to-debug situation.

Workaround: Use TCB early access, where the issue is fixed, or wait for TCB 3.11.0 release.

Description: After recent changes in the backend, TorizonCore Builder is failing when we try to push docker-compose files to OTA.

Description: There is an issue when using the isolate command when there are symlinks to be isolated and using the --changes-directory argument.

Description: Improve the error messages on TorizonCore Builder when "unpack" hasn't been run yet.

Description: The `images` command has an option to clear the storage directories. It does not clear the /splash directory, though. This ticket aims to fix it.

Description: TorizonCore Builder runs inside a container, and by default it creates files owned by root. This is a bit annoying since we sometimes want to edit or erase some files with our user in the development PC. We investigate and implement a way to make files belong to the user and not root.

Description: For the commands that require device connection, make --remote-username and --remote-password parameters optional.

Description: Torizoncore builder union command fails with enabled systemd services.

Workaround: Disable the systemd service before running the "isolate" command. This will require you to manually enable it after deployment.

Description: If you have a symlink in the changes directory that points to a file using the full path, TorizonCore Builder’s union command fails.

Description: The interface of the union command was simplified to make it easier to use:

• the --extra-changes-directory switch was removed and its functionality moved to --changes-directory;
• the switch --union-branch no longer exists either - the branch name is now a positional parameter;

Description: When uncompressing a local image, TorizonCore Builder leaves behind the uncompressed directory, even though it isn't used anymore. In this ticket we clean up the uncompressed directory.

Description: TorizonCore Builder should be able to save and restore credentials when isolating files from the device and union-ing files to another TorizonCore image.

Description: There's an issue with the bundle sub-command where the the tar file output that gets created is inconsistent. The proper expected name that should be created is docker-storage.tar.xz, yet sometimes a docker-storage.tar is created instead.

Description: Upon the first login, it is mandatory to change the password. This modification is not captured by TorizonCore Builder's isolate command.

Description: The isolate command fails because of spaces in the name of files.

Description: TorizonCore Builder cannot fetch the new NXP BSP branch (toradex_5.4-2.3.x-imx) from device trees repository.

Description: If you attempt to bundle a container with TorizonCore Builder on Windows from WSL 2, it fails.

Description: With the introduction of the `kernel build_module` command, the size of TorizonCore Builder increased by a factor of ~10x, due to the download of a duplicated toolchain. This commit addresses this issue and also removes the toolchain from TorizonCore Builder, leaving the download only to those users who need to execute the `kernel build_module` command. This approach increases the time for the first run of the command, while leaving TorizonCore Builder much smaller by default.

Description: TorizonCore Builder seems to not be listing all compatible overlays for some machines, like Apalis iMX8.

Description: Make it possible to set custom kernel arguments using TorizonCore Builder.

Description: Add 'images download/unpack' commands in TorizonCore Builder to automatically download Tezi images from Artifactory (based on the version running on a device) and unpack for further customizations.

Description: Create a shell script to automate environment configuration and warn the user when a new TorizonCore Builder version is available.

Description: To increase awareness about the TorizonCore Builder documentation, we add a hyperlink to it at the end of the `--help`.

Description: If you execute `torizoncore-builder deploy dt_changes` and pass the board hostname as parameter for `--remote-host` instead of the board IP, for example `apalis-imx8-06543292.local`, it does not work.

Workaround: It has been fixed in TorizonCore Builder 2.2.0. Use this version of the tool.

Description: Refactor device tree commands and adapt them to use overlays.txt.

Description: After using TorizonCore Builder to customize an Easy Installer image with a device tree overlay applied, the generated Easy Installer image has incorrect permissions in the /home/torizon directory.

Description: Update the TorizonCore Builder container to Debian Bullseye.

Description: Torizoncore Builder generates an invalid compose file after bundle command.

Description: When TorizonCore Builder replaces the tags in docker-compose file with concrete sha256 hashes, it drops the MACHINE variable.

Description: We want to create the "dt list-overlays" (or similar name) sub-command to list available (and compatible with the module) device tree overlays.

Description: Create documentation on how to use TorizonCore Builder to apply device trees

Description: Add additional metadata to OTA targets.json.

Description: Add feature in TorizonCore Builder to set a splash screen image.

Description: Add the push command to torizoncore-builder tool to upload changes in an OSTree repository to Torizon OTA.

Description: TorizonCore Builder is a tool that helps to integrate Docker containers and user's customization in a production image.

Description: Some containers such as our base containers with Vivante binary drivers need the customer to accept an EULA. We should support adding an EULA to the Tezi image as part of the pre-provisioning process (in torizoncore-builder).

Description: TorizonCore Builder does not support the ZFS filesystem. If you setup a computer with a Linux distro that uses ZFS - for example, Ubuntu 22.04 LTS with the experimental ZFS support enabled - you will see errors such as "ConnectionRefusedError: [Errno 111] Connection refused" when running the bundle command.

On TorizonCore Builder we run Docker in Docker (DinD) and use the OverlayFS driver. As stated on Docker storage drivers - Supported storage drivers per Linux distribution, it is the recommended storage driver and requires XFS or ext4 as the filesystem where "/var/lib/docker" (from the DinD) is located. In practice, this means that you have to use a filesystem compatible with OverlayFS on your computer for TorizonCore Builder to work, even if you don't run the Docker daemon itself with OverlayFS.

At the moment, our plans are to make the error message more intuitive, stating that ZFS is not supported. We will keep monitoring the adoption of ZFS and possibly other filesystems and, as they gain traction and become the default on major Linux distros, we might consider adding support for them in upcoming TorizonCore versions.

Workaround: Use a Linux distro with either ext4 or XFS (with `d_type=true` enabled) as the filesystem

Description: As reported in our community, with TorizonCore Builder, If you bundle a container that sets extended attributes on files inside the container (for example, using the "setcap" command), the container will fail upon execution with the error "SIOCSIFFLAGS: Operation not permitted" or another related error.

If you remove the bundled container and manually pull it from Docker Hub, then try to run it again, the same error will not happen.

While this bug was investigated, we came to docker build does not preserve xattrs in the generated image, and after several tests such as setting capabilities and building the container with the classic Docker vs Buildkit, we concluded that:

• Docker does not keep all extended attributes that are set in a Dockerfile, regardless of using Buildkit or the Classic Docker builder, and using Docker directly (without Compose).
• Even some capabilities that are kept by Docker get lost when bundling a container with TorizonCore Builder

Workaround: For this specific bug: set the suid for executable files rather than using setcap. For example use "chmod u+s" instead of "setcap" seems to work as well and may carry through the bundle process.

In general, do not rely on xattrs or any feature based on them inside a container. Capabilities and ACLs are features that may be set via xattrs and can also be set via docker-compose properties, which is the recommended way to set them in a portable and reliable way.

Description: Activating device tree overlay fails on Apalis i.MX8

Description: It is possible to run TorizonCore Builder without using a Docker volume, by passing some extra flags to the setup script: "source tcb-env-setup.sh -d -s /srv/builder-tool/storage".

When using the "tcb build" command, though, it does not create the output directory as expected.

It has been reported on the TCB GitHub issues.

Workaround: Use docker volumes (the default option) when using the build command.

Description: The TorizonCore Builder "images serve" command was added to TorizonCore Builder on version 3.2.0. It does not work on Windows due to two limitations: (1) the Docker host network driver is not supported on Windows and (2) the multicast DNS (mDNS) used by Avahi does not work.

Workaround: Take the output image from TorizonCore Builder and use an SD Card/USB stick to install it to a Toradex SoM.

Alternatively, use a Linux machine.