Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release
or cat /etc/issue
on the board.
DLR is a compact runtime for Machine Learning Models. This runtime is used to run models compiled and tuned by Amazon SageMaker Neo.
In this article, we will show how to run Object Detection algorithms optimized by SageMaker Neo in an Apalis iMX8 board. We will also show how you can obtain frame data coming from a camera from a GStreamer pipeline, modify it with OpenCV and output to a display using a second GStreamer pipeline.
Hardware
Software and Knowledge
Toradex provides a simple demo for Object Detection using DLR runtime executing an SSD Object Detection algorithm based on ImageNet.
To execute this demo in your Apalis iMX8, with Torizon installed, first start the Weston container. Execute the following command in your board terminal:
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
# docker run -e ACCEPT_FSL_EULA=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG \
-v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ \
--device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' \
torizon/weston-vivante:$CT_TAG_WESTON_VIVANTE --developer weston-launch --tty=/dev/tty7 --user=torizon
After Weston start, run the demo image from dockerhub on the board:
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
docker run -e ACCEPT_FSL_EULA=1 --rm -d --name=dlr-example --device-cgroup-rule='c 81:* rmw' -v /dev/galcore:/dev/galcore -v /run/udev/:/run/udev/ -v /sys:/sys -v /dev:/dev -v /tmp:/tmp --network host torizonextras/arm64v8-sample-dlr-gstreamer-vivante:${CT_TAG_WAYLAND_BASE_VIVANTE}
Tip: Learn more about the tag CT_TAG_WAYLAND_BASE_VIVANTE on TorizonCore Containers Tags and Versioning. Check the available tags for this sample on https://hub.docker.com/r/torizonextras/arm64v8-sample-dlr-gstreamer-vivante/tags.
This demo will get frames of a camera in /dev/video0
, execute the object detection algorithm inference and output to a monitor using the Wayland protocol. Update inference.py
file accordingly if necessary.
The example files contains a generic model for object detection, trained with ImageNet dataset , with the following object tags categories: 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor'.
To stop the containers:
# docker stop weston;docker stop dlr-example
First, clone the torizon sample files from the repository. These files are the base for this article.
$ git clone https://github.com/toradex/torizon-samples.git
Change to the project's directory.
$ cd torizon-samples/dlr-gstreamer
This is the structure of the project's directory:
$ tree
.
├── Dockerfile
├── inference.py
├── model
│ ├── model.json
│ ├── model.params
│ └── model.so
└── README.txt
The example's Dockerfile uses torizon/wayland-base-vivante
image as starting point. The torizon/wayland-base-vivante
image contains components used for image processing: The Vivante's GPU drivers and Wayland. For this image, we install OpenCV, GStreamer, NumPy, and also the DLR runtime.
The project's Python script executes the application. It uses GStreamer's Python API to collect frames in RGB format, process it, and output the video.
We will use appsink GStreamer's element to collect frames in RGB format coming from the camera. It's counterpart - appsrc will put the processed frames into a second bus responsible for the image output.
The frames are then processed and used as input of the neural network. The following code snippet shows how the pipeline is created. In this example, we set the function on_new_frame
to handle data when a new frame is available on appsink.
# GStreamer Init Gst.init(None) pipeline1_cmd="v4l2src device=/dev/video2 do-timestamp=True ! queue leaky=downstream ! videoconvert ! \ videoscale n-threads=4 method=nearest-neighbour ! \ video/x-raw,format=RGB,width="+str(width_out)+",height="+str(height_out)+" ! \ queue leaky=downstream ! appsink name=sink \ drop=True max-buffers=1 emit-signals=True max-lateness=8000000000" pipeline2_cmd = "appsrc name=appsource1 is-live=True block=True ! \ video/x-raw,format=RGB,width="+str(width_out)+",height="+ \ str(height_out)+",framerate=10/1,interlace-mode=(string)progressive ! \ videoconvert ! waylandsink" #v4l2sink max-lateness=8000000000 device=/dev/video14" pipeline1 = Gst.parse_launch(pipeline1_cmd) appsink = pipeline1.get_by_name('sink') appsink.connect("new-sample", on_new_frame, appsink) pipeline2 = Gst.parse_launch(pipeline2_cmd) appsource = pipeline2.get_by_name('appsource1') pipeline1.set_state(Gst.State.PLAYING) bus1 = pipeline1.get_bus() pipeline2.set_state(Gst.State.PLAYING) bus2 = pipeline2.get_bus()
To use DLR using python, the first step is to import the DLR runtime API. Look at the example's Dockerfile if you need to see how it is installed.
To import the DLR python API:
from dlr import DLRModel
On the script initialization, we need to set a DLR runtime model by indicating where our model files are located.
model = DLRModel('./model', 'cpu')
The run
function of DLR will process the neural network.
outputs = model.run({'data': nn_input}) objects=outputs[0][0] scores=outputs[1][0] bounding_boxes=outputs[2][0]
In this example, nn_input
is the neural network input - A NumPy array with the shape 240x240 containing the frame in RGB format.
The return of the run
function is the output of the neural network with arrays containing the object categories detected, it's score and the corresponding bounding boxes.
The output data is processed by the script and OpenCV functions are used to draw the bounding boxes in the frame. In this demo, only the objects with score > 0.5 will be displayed.
OpenCV is used to draw bounding boxes:
#Draw bounding boxes i = 0 while (scores[i]>0.5): y1=int((bounding_boxes[i][1]-nn_input_size/8)*width_out/nn_input_size) x1=int((bounding_boxes[i][0])*height_out/(nn_input_size*3/4)) y2=int((bounding_boxes[i][3]-nn_input_size/8)*width_out/nn_input_size) x2=int((bounding_boxes[i][2])*height_out/(nn_input_size*3/4)) object_id=int(objects[i]) cv2.rectangle(img,(x2,y2),(x1,y1),colors[object_id%len(colors)],2) cv2.rectangle(img,(x1+70,y2+15),(x1,y2),colors[object_id%len(colors)],cv2.FILLED) cv2.putText(img,class_names[object_id],(x1,y2+10), cv2.FONT_HERSHEY_SIMPLEX, 0.4,(255,255,255),1,cv2.LINE_AA) i=i+1 cv2.rectangle(img,(110,17),(0,0),(0,0,0),cv2.FILLED) cv2.putText(img,"inf. time: %.3fs"%last_inference_time,(3,12), cv2.FONT_HERSHEY_SIMPLEX, 0.4,(255,255,255),1,cv2.LINE_AA)
As explained earlier, the example contains model files trained with the ImageNet dataset.
You can replace the existing content of this folder by models trained with your own datasets, by replacing the files in this directory. For more information about how to train your own model, visit the Training your own model page
On the host PC,cd
to the project's directory and build the image:
$ cd <Dockerfile-directory>
$ docker build -t <your-dockerhub-username>/dlr-example --cache-from torizonextras/dlr-example .
After the build, push the image to your Dockerhub account:
$ docker push <your-dockerhub-username>/dlr-example
This example will run a pipeline that uses waylandsink GStreamer's plugin. This plugin runs on top of Weston. It will be necessary to start 2 containers: One with the Weston image, and one with the application image with Wayland support. Both will communicate through shared directories.
First, start the container with the Weston. In the terminal of your board:
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
# docker run -e ACCEPT_FSL_EULA=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG -v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ --device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' torizon/weston-vivante:$CT_TAG_WESTON_VIVANTE --developer weston-launch --tty=/dev/tty7 --user=torizon
You will see the Weston screen in your HDMI monitor.
After Weston start, pull your application from your dockerhub account to the board.
# docker pull <your-dockerhub-username>/dlr-example
After the pull, run a container based on the image.
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
docker run -e ACCEPT_FSL_EULA=1 --rm -d --name=dlr-example --device-cgroup-rule='c 81:* rmw' -v /dev/galcore:/dev/galcore -v /run/udev/:/run/udev/ -v /sys:/sys -v /dev:/dev -v /tmp:/tmp --network host <your-dockerhub-username>/dlr-example
The inference should be displayed on the screen.
# docker stop dlr-example
# docker stop weston
DLR is a compact runtime for Machine Learning Models. This runtime is used to run models compiled and tuned by SageMaker Neo. In this article, we will show how to run Object Detection algorithms optimized by SageMaker Neo in an Apalis iMX8 board. We will also show how you can obtain frame data coming from a camera from a GStreamer pipeline, modify it with OpenCV and output to a display using a second GStreamer pipeline.
/dev/video2
. Change the inference.py file accordingly if necessary.Toradex provides a simple demo for Object Detection using DLR runtime executing an SSD Object Detection algorithm based on ImageNet.
To execute this demo in your Apalis iMX8, with Torizon installed, execute the following command in your board terminal:
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
# docker run -e ACCEPT_FSL_EULA=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG \
-v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ \
--device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' \
torizon/arm64v8-debian-weston-vivante:buster --developer weston-launch --tty=/dev/tty7 --user=torizon
The command above automatically pulls the demo image from dockerhub and runs it on the board.
This demo will get frames of a camera in \dev\video0
, execute the object detection algorithm inference and output to a monitor using Wayland protocol.
The example files contains a generic model for object detection, trained with ImageNet dataset , with the following object tags categories: 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor'.
To stop the containers:
# docker stop weston;docker stop dlr-example
First, clone the torizon sample files from the repository. These files are the base for this article.
$ git clone https://github.com/toradex/torizon-samples.git
Change to the project's directory.
$ cd torizon-samples/dlr-gstreamer
This is the structure of the project's directory:
$ tree
.
├── Dockerfile
├── inference.py
├── model
│ ├── model.json
│ ├── model.params
│ └── model.so
└── README.txt
The example's Dockerfile uses torizon/arm64v8-debian-wayland-base-vivante
image as starting point. The torizon/arm64v8-debian-wayland-base-vivante
image contains components used for image processing: The Vivante's GPU drivers and Wayland. For this image, we install OpenCV, GStreamer, NumPy, and also the DLR runtime.
The project's Python script executes the application. It uses GStreamer's Python API to collect frames in RGB format, process it, and output the video.
We will use appsink GStreamer's element to collect frames in RGB format coming from the camera. It's counterpart - appsrc will put the processed frames into a second bus responsible for the image output.
The frames are then processed and used as input of the neural network. The following code snippet shows how the pipeline is created. In this example, we set the function on_new_frame
to handle data when a new frame is available on appsink.
# GStreamer Init Gst.init(None) pipeline1_cmd="v4l2src device=/dev/video2 do-timestamp=True ! queue leaky=downstream ! videoconvert ! \ videoscale n-threads=4 method=nearest-neighbour ! \ video/x-raw,format=RGB,width="+str(width_out)+",height="+str(height_out)+" ! \ queue leaky=downstream ! appsink name=sink \ drop=True max-buffers=1 emit-signals=True max-lateness=8000000000" pipeline2_cmd = "appsrc name=appsource1 is-live=True block=True ! \ video/x-raw,format=RGB,width="+str(width_out)+",height="+ \ str(height_out)+",framerate=10/1,interlace-mode=(string)progressive ! \ videoconvert ! waylandsink" #v4l2sink max-lateness=8000000000 device=/dev/video14" pipeline1 = Gst.parse_launch(pipeline1_cmd) appsink = pipeline1.get_by_name('sink') appsink.connect("new-sample", on_new_frame, appsink) pipeline2 = Gst.parse_launch(pipeline2_cmd) appsource = pipeline2.get_by_name('appsource1') pipeline1.set_state(Gst.State.PLAYING) bus1 = pipeline1.get_bus() pipeline2.set_state(Gst.State.PLAYING) bus2 = pipeline2.get_bus()
To use DLR using python, the first step is to import the DLR runtime API. Look at the example's Dockerfile if you need to see how it is installed.
To import the DLR python API:
from dlr import DLRModel
On the script initialization, we need to set a DLR runtime model by indicating where our model files are located.
model = DLRModel('./model', 'cpu')
The run
function of DLR will process the neural network.
outputs = model.run({'data': nn_input}) objects=outputs[0][0] scores=outputs[1][0] bounding_boxes=outputs[2][0]
In this example, nn_input
is the neural network input - A NumPy array with the shape 240x240 containing the frame in RGB format.
The return of the run
function is the output of the neural network with arrays containing the object categories detected, it's score and the corresponding bounding boxes.
The output data is processed by the script and OpenCV functions are used to draw the bounding boxes in the frame. In this demo, only the objects with score > 0.5 will be displayed.
OpenCV is used to draw bounding boxes:
#Draw bounding boxes i = 0 while (scores[i]>0.5): y1=int((bounding_boxes[i][1]-nn_input_size/8)*width_out/nn_input_size) x1=int((bounding_boxes[i][0])*height_out/(nn_input_size*3/4)) y2=int((bounding_boxes[i][3]-nn_input_size/8)*width_out/nn_input_size) x2=int((bounding_boxes[i][2])*height_out/(nn_input_size*3/4)) object_id=int(objects[i]) cv2.rectangle(img,(x2,y2),(x1,y1),colors[object_id%len(colors)],2) cv2.rectangle(img,(x1+70,y2+15),(x1,y2),colors[object_id%len(colors)],cv2.FILLED) cv2.putText(img,class_names[object_id],(x1,y2+10), cv2.FONT_HERSHEY_SIMPLEX, 0.4,(255,255,255),1,cv2.LINE_AA) i=i+1 cv2.rectangle(img,(110,17),(0,0),(0,0,0),cv2.FILLED) cv2.putText(img,"inf. time: %.3fs"%last_inference_time,(3,12), cv2.FONT_HERSHEY_SIMPLEX, 0.4,(255,255,255),1,cv2.LINE_AA)
As explained earlier, the example contains model files trained with the ImageNet dataset.
You can replace the existing content of this folder by models trained with your own datasets, by replacing the files in this directory. For more information about how to train your own model, visit the Training your own model page
On the host PC,cd
to the project's directory and build the image:
$ cd <Dockerfile-directory>
$ docker build -t <your-dockerhub-username>/dlr-example --cache-from torizonextras/dlr-example .
After the build, push the image to your Dockerhub account:
$ docker push <your-dockerhub-username>/dlr-example
This example will run a pipeline that uses waylandsink GStreamer's plugin. This plugin runs on top of Weston. It will be necessary to start 2 containers: One with the Weston image, and one with the application image with Wayland support. Both will communicate through shared directories.
First, start the container with the Weston. In the terminal of your board:
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
# docker run -e ACCEPT_FSL_EULA=1 -d --rm --name=weston --net=host --cap-add CAP_SYS_TTY_CONFIG -v /dev:/dev -v /tmp:/tmp -v /run/udev/:/run/udev/ --device-cgroup-rule='c 4:* rmw' --device-cgroup-rule='c 13:* rmw' --device-cgroup-rule='c 199:* rmw' --device-cgroup-rule='c 226:* rmw' torizon/arm64v8-debian-weston-vivante --developer weston-launch --tty=/dev/tty7 --user=torizon
You will see the Weston screen in your HDMI monitor.
After Weston start, pull your application from your dockerhub account to the board.
# docker pull <your-dockerhub-username>/dlr-example
After the pull, run a container based on the image.
Attention: Please, note that by executing the following line you are accepting the NXP's terms and conditions of the End-User License Agreement (EULA)
docker run -e ACCEPT_FSL_EULA=1 --rm -d --name=dlr-example -v /dev/galcore:/dev/galcore -v /run/udev/:/run/udev/ -v /sys:/sys -v /dev:/dev -v /tmp:/tmp --network host <your-dockerhub-username>/dlr-example
The inference should be displayed on the screen.
# docker stop dlr-example
# docker stop weston