This article aims to provide helpful information for the first steps with Cameras on TorizonCore. So, it means that this article is going to guide you with the specific details and considerations that surround the usage of a video capture device using containers.
The main goal of using Torizon and containers is to simplify the developer's life and keep the development model close to one of desktop or server applications. However, some requirements inherent to embedded systems development may be considered in the process. Therefore, running a video capture application inside a container requires some aspects like hardware access, running multiple containers, and others. You can read more about these requirements in the section Container Requirements
If you want to learn more about the Torizon best practices and container development workflow for embedded systems applications, you can refer to Torizon Best Practices Guide. Developing applications for Torizon can be done using command-line tools or, as we recommend, the Visual Studio Code Extension for Torizon.
This article workflow is valid for USB Cameras with USB Video Class (UVC) standard driver and some CSI cameras. To find more about USB cameras you can refer to Webcam (Linux). We recommend using one of our supported CSI cameras which you can find at Camera on Toradex Computer on Modules. The process described here regarding the use of a MIPI-CSI camera was done with the CSI Camera Set 5MP AR0521 Color. To use the AR0521 camera you have to enable a Device-Tree Overlay, see more at First Steps with CSI Camera Set 5MP AR0521 Color (Torizon).
Note: This article does not cover the usage of IP cameras, you can find information about this topic at Audio/Video over RTP With GStreamer (Linux).
This article complies with the Typographic Conventions for the Toradex Documentation.
Running your application in a container means that it will run in a "sandboxed" environment. This fact may limit its interaction with other components in the system. Hardware access and the use of multiple containers, mainly focused on the usage of a Graphical User Interface (GUI), are the most important topics that you must know to have a good understanding of how to proceed with camera usage. You can read more about multiple containers at Using Multiple Containers with TorizonCore.
Docker provides ways to access specific hardware devices from inside a container, and often one needs to grant access from both a permission and a namespace perspective.
c 199:* rmw.--device) is better for security since you avoid exposing a storage device that may be erased by an attacker.To use camera devices, it's required to bind mount specific directories of the host machine. In other words, you have to guarantee access to certain directories. So, you have to use four bind mounts:
/dev: Mount /dev allows access to the devices that are attached to the local system./tmp: Mounting /tmp as a volume in the container allows your application to interact with Weston (GUI)./sys: Grant access to kernel subsystems./var/run/dbus: Provides access to system services.Another useful resource is displaying the camera output, especially during the setup and debugging phases of the application development, even if not necessary for the final application. To perform this, you can use a display by configuring a Weston-based container, or VNC. To read more about GUI in TorizonCore, refer to the section Graphical User Interface (GUI) in Torizon Best Practices Guide.
The implementation details will be explained in this session. See the Quickstart Guide with the instructions about how to compile the image on a host pc and pull the image onto the board. You can also scp this file to the board and build it locally.
Attention: Make sure you have configured your Build Environment for Torizon Containers
Now it's a good time to use torizon-samples repository:
$ cd ~/torizon-samples/gstreamer/bash/simple-pipeline
docker build --build-arg BASE_NAME=wayland-base-vivante --build-arg IMAGE_ARCH=linux/arm64/v8 -t <your-dockerhub-username>/gst_example .
$ docker build -t <your-dockerhub-username>/gst_example .
After the build, push the image to your Dockerhub account:
$ docker push <your-dockerhub-username>/gst_example
For more details about the dockerfile, refer to How to use Gstreamer on TorizonCore.
To run a GStreamer pipeline inside a container, in order to avoid possible problems in launching the containers that are required to start the video, make sure to stop all the other containers that might be running on your device.
# docker stop $(docker run -a -q)
Considering that a streaming video application requires a GUI, you have to pull a Debian Bullseye container featuring the Weston Wayland compositor and start it on the module.
(Optional) pull the torizon/weston container image:
# docker pull torizon/weston:$CT_TAG_WESTON
Start the weston compositor:
# docker run -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG \
-v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ \
--device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' \
--device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' \
torizon/weston:$CT_TAG_WESTON --developer weston-launch --tty=/dev/tty7 --user=torizon
(Optional) pull the torizon/weston container image:
# docker pull torizon/weston:$CT_TAG_WESTON
Start the weston compositor:
# docker run -d --rm --ipc=host --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG \
-v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ \
--device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' \
--device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' \
torizon/weston:$CT_TAG_WESTON --developer weston-launch \
--tty=/dev/tty7 --user=torizon -- --use-pixman
(Optional) pull the torizon/weston-vivante container image:
# docker pull torizon/weston-vivante:$CT_TAG_WESTON_VIVANTE
Start the weston compositor:
Attention: Please, note that by executing the following line you are accepting the terms and conditions of the NXP's End-User License Agreement (EULA)
# docker run -e ACCEPT_FSL_EULA=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG \
-v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ \
--device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' \
--device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' \
torizon/weston-vivante:$CT_TAG_WESTON_VIVANTE --developer weston-launch \
--tty=/dev/tty7 --user=torizon
If it is the first time you are dealing with a specific video device and you're not sure which device is the capture one, you may follow the next steps to discover what is the right video capture device:
ls /dev/video*.# ls /dev/video*
/dev/video0 /dev/video1 /dev/video2 /dev/video3 /dev/video12 /dev/video13
--devices.# docker run --rm -it -v /tmp:/tmp -v /var/run/dbus:/var/run/dbus -v /dev:/dev -v /sys:/sys \
--device /dev/video0 --device /dev/video1 --device /dev/video2 --device /dev/video3 --device /dev/video12 --device /dev/video13 \
--device-cgroup-rule='c 199:* rmw' \
<your-dockerhub-username>/<Dockerfile-name>
v4l2-ctl --list-devices. Now you are able to see which devices are video capture ones.## v4l2-ctl --list-devices
vpu B0 (platform:):
/dev/video12
/dev/video13
mxc-jpeg decoder (platform:58400000.jpegdec):
/dev/video0
mxc-jpeg decoder (platform:58450000.jpegenc):
/dev/video1
Video Capture 5 (usb-xhci-cdns3-1.2):
/dev/video2
/dev/video3
/dev/media0
v4l2-ctl -D to find information about the Video Capture devices listed before. In this case, just the /dev/video2 and /dev/video3. As you are able to see, the camera is represented by /dev/video3.## v4l2-ctl --device /dev/video3 -D
Driver Info:
Driver name : uvcvideo
Card type : Metadata 5
Bus info : usb-xhci-cdns3-1.2
Driver version : 5.4.161
Capabilities : 0x84a00001
Video Capture
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04a00000
Metadata Capture
Streaming
Extended Pix Format
Media Driver Info:
Driver name : uvcvideo
Model : USB 2.0 Camera: USB Camera
Serial : 01.00.00
Bus info : usb-xhci-cdns3-1.2
Media version : 5.4.161
Hardware revision: 0x00000003 (3)
Driver version : 5.4.161
Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:
ID : 0x00000004 (4)
Name : Metadata 5
Function : V4L2 I/O
/dev/video3).The next step is to launch the container with Gstreamer and Video4Linux2, making sure to pass the right video device /dev/video* under the parameter --device.
# docker run --rm -it -v /tmp:/tmp -v /var/run/dbus:/var/run/dbus -v /dev:/dev -v /sys:/sys \
--device /dev/<video-device> \
--device-cgroup-rule='c 199:* rmw' \
<your-dockerhub-username>/<Dockerfile-name>
Once inside the container, you are able to use Video4Linux2 and Gstreamer resources. The basic structure of the pipeline relies on the usage of a data source (in this case a Video4Linux2 source), a filter, and a data sink, in this case, the video sink Wayland.
Gstreamer pipeline diagram
One of the simplest pipelines for showing video can be done the following way:
## gst-launch-1.0 <videosrc> ! <capsfilter> ! <videosink>
Or more specifically
## gst-launch-1.0 v4l2src device=</dev/video*> ! <capsfilter> ! fpsdisplaysink video-sink=waylandsink
If you want to learn more about how to elaborate a more complex Gstreamer pipeline, refer to How to use Gstreamer on TorizonCore.
To discover more properties of v4l2src to configure the pads, you can use gst-inspect:
As you are dealing with a raw video, you will have to use, in this case, the video/x-raw and configure its properties: format, width, height, and framerate.
## gst-launch-1.0 v4l2src device='/dev/<video-device>' ! "video/x-raw, format=<video-format>, framerate=<framerate>, width=<supported-width>, height=<supported-height>" ! fpsdisplaysink video-sink=waylandsink
You also can check information about the sink using the gst-inspect:
Note: You have to keep in mind that different cameras may support different video formats, resolutions (height and width), etc. Then, even the simplest pipeline can vary
So, the final structure of the pipeline should be similar to the following one:
## gst-launch-1.0 v4l2src device='/dev/<video-device>' ! "video/x-raw, format=<video-format>, framerate=<framerate>, width=<supported-width>, height=<supported-height>" ! fpsdisplaysink video-sink=waylandsink text-overlay=<true-or-false> sync=<true-or-false>
The following subsections cover the specific considerations of the described process to use MIPI CSI-2 cameras and Webcams.
The USB 2.0 Host interface is available on all the modules to connect USB Cameras. However, USB 3.0 is only available on specific modules from the Apalis SoM Family and the Verdin SoM Family, check the datasheets for details. Our article Webcam (Linux) is a starting point on how to set up a USB camera in Linux, as mentioned in Camera on Toradex Computer on Modules.
The following pipeline was used to show video from a webcam:
## gst-launch-1.0 v4l2src device='/dev/video3' ! "video/x-raw, format=YUY2, framerate=5/1, width=640, height=480" ! fpsdisplaysink video-sink=waylandsink text-overlay=false sync=false
To use a MIPI-CSI Camera you have to make sure that your module and carrier board have a MIPI-CSI connection. It’s important to highlight the following considerations from Camera on Toradex Computer on Modules:
Depending on the MIPI CSI-2 Camera you are using, you may enable a specific device tree overlay. As in this example, it was used the Camera Set 5MP AR0521 Color, the overlay process using TorizonCore Builder was described at First Steps with CSI Camera Set 5MP AR0521 Color (Torizon).
With the overlay enabled, the next step is to launch a video stream using a pipeline. To achieve this, you should follow a process similar to the one described in the article First Steps with CSI Camera Set 5MP AR0521 Color (Torizon).
The following pipeline was used to show video with Camera Set 5MP AR0521 Color:
## gst-launch-1.0 v4l2src device='/dev/video0' ! "video/x-raw, format=RGB16, framerate=30/1, width=1920, height=1080" ! fpsdisplaysink video-sink=waylandsink text-overlay=false sync=false
This hello-world application, like the usage of Gstreamer directly on the terminal, is just a video stream. For this application you are going to use OpenCV together with V4L2 and GStreamer, using Visual Studio Code Extension for Torizon. If you want to learn more about OpenCV, refer to Torizon Sample: Using OpenCV for Computer Vision.
So, the first step is to create a python application with QML, as described at Python development on TorizonCore. We’re not going to use QML, we just need the GUI application launching Weston.
Then, go to Torizon Extension and proceed with the described modifications. Add the following volumes under the parameter volumes:
/dev, value=/dev/tmp, value=/tmp/sys, value=/sys/var/run/dbus, value=/var/run/dbusAdd access to the camera device (/dev/video2) under the parameter devices. Add a a cgroup rule under the extraparms parameter as follows: key=device_cgroup_rules and value=[ "c 199:* rmw" ]
Finally, modify the field extrapackages including all the packages and plugins explained before, and add the OpenCV package for python3, which is python3-opencv.
python3-opencv v4l-utils gstreamer1.0-qt5 libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-pulseaudio
The code must be written inside the main.py and begin with the import of the OpenCV library, called cv2
import cv2
OpenCV provides the VideoCapture class used to capture the video. A pipeline similar to the one created before is going to be the input parameter of the class. The difference between the pipelines is the use of appsink instead of waylandsink, as the pipeline will send data to your program, not to a display.
cap = cv2.VideoCapture("v4l2src device=/dev/video2 ! video/x-raw, width=640, height=480, format=YUY2 ! videoconvert ! video/x-raw, format=BGR ! appsink")
To capture and read the frames we’re going to use read:
ret, frame = cap.read() # Capture frame
And to display the frames we can use imshow:
cv2.imshow('frame', frame) # Display the frames
Then, release the capture and destroy the window:
cap.release() # Release the capture object cv2.destroyAllWindows() # Destroy all the windows
Tip: To read frames and display them continuously, you can put the capture and display lines inside a loop.
