I met a business person at a business meeting and exchanged business cards with a handshake. He was younger than I thought, and he had the latest laptop and a brand new cell phone. He put down my business card and took pictures of the front and back. On the other hand, I wrote the date and the issue on my business card, and when I looked up... he handed me back my business card and said... "I saved all of them on my phone with a business card app. It's more convenient for me. It's hard to find business cards, so I save them every time I receive them."

From the text on business cards to recognizing human faces, most forms of images can now be recognized with a mobile phone. This is done using a computer vision library called OpenCV. Developed by Intel and currently used on various platforms such as Windows and Linux, OpenCV is a library that focuses on real-time image processing. It currently consists of over 2,500 algorithms and can be used in interfaces such as C, C++, Python, and MatLab.

Thanks to the advancement of camera performance, which acts like the human eye in devices, and the improvement of chip performance that processes tens of millions of pixels in real time, object recognition functions that were previously only available in existing CPUs have become possible in the mobile phones we always carry with us. These developments are being used in a variety of ways in industrial and medical devices, including face detection and recognition, object recognition, 3D model extraction of objects, 3D coordinate generation from stereo cameras, image stitching for high-resolution image generation, image search, red-eye removal, and eye movement tracking.

Xilinx, which provides various support in the vision field, has decided to support the development of OpenCV, following Open VX and OpenCL. Jeong Woong, DSP Specialist in the Embedded Business Unit at Xilinx, explained the background of the support, saying, "The continuous increase in the number of users and the expansion of deep learning functions supported from version 3.0 are creating a larger user base for OpenCV." Additionally, it is expected that Xilinx's Zynq SoC will gain a market advantage with higher performance than existing processor-based vision platforms.

As evidence for this, when executing Harris Corner Detect, Stereo Depth Map, and Dense Optical Flow Function of OpenCV through ARM cortex-A53, it showed 2.4 FPS, 2.1 FPS, and 0.1 FPS, which are significantly lower than the general image processing requirement of 30 FPS. Also, considering the continuous increase in image resolution in the market, it was said that if the same function is supported in high definition, a performance shortage will occur, and this will require acceleration of the OpenCV function.
17x the performance of eGPU, half the power usage
When accelerating the OpenCV function using Xilinx's Zynq SoC, it shows a performance of up to 700 FPS, which is more than 17 times higher than the 43 FPS of Tegra X1 from NVIDIA, a representative graphic chip vendor, when comparing the Stereo LBM of 1080P. Moreover, the power consumption is 2.2W to 4.8W, which is nearly half of the 7.9W of Tegra X1. LK Dense Optical Flow based on 720P also showed 170 FPS, which is up to 24 times higher than 7 FPS compared to the same NVIDIA Tegra X1, and the power consumption showed the same amount as the Stereo LBM case.
Reduce processing latency through programmable logic
Regarding the part where such high-efficiency performance is possible, Xilinx's Vice President Jeong-ung mentioned Xilinx's unique DDR operation method. The processing data flow of a typical processor or GPU is structured as a sequence block processing in which image sensor data such as a camera is first stored in DDR, each processing block uploads DDR data, processes it, and then stores the processed results back in DDR. In contrast, Xilinx's Zynq SoC implements programmable logic and processes in a streaming manner, where processing results for each pixel are available before the processing of one block is completed, and processing begins in the next processing block.
Streaming is performed at the line buffer level rather than the pixel level, and can be stored using BRAM, an internal block RAM within the programmable logic, or ULTRARAM, a large-capacity BRAM. This can provide higher performance because processing can be done within the programmable logic without accessing external memory.

Vice President Jeong Woong said, "If we look at the latency when image processing blocks are processed sequentially using Xilinx programmable logic, it takes approximately 15.73 msec, or 15.73 million clocks, when only the Bilateral Filter is processed." He also said, "If the Wrap Transform is processed after the Bilateral Filter, it takes approximately 15.85 msec, and even if the Customer ISP block is processed after that, it takes approximately 15.86 msec."

Typically, for sequential block processing, the approximate latency is the sum of the latencies of each processing block, excluding DDR access times. However, in a pipeline structure, there is a slight additional latency added to the block latency, which has the highest latency.
With these strengths, Xilinx's xfOpenCV library can avoid frame buffer or external memory access by automatically configuring the pipeline structure between each block in the same structure as the existing Open CV code, thereby providing high performance in terms of processing speed. In addition, it has the advantage of improving power efficiency by not using external memory access.


Provides over 50 different usable libraries
Currently, Xilinx provides more than 50 libraries for OpenCV. It is broadly divided into six main functions: basic function library, Geometric Transforms, image processing and filters, object recognition, 3D Reconstruction, motion analysis, and location tracking. All sources can be downloaded from GitHub.
Vice President Jeong Woong explained that Xilinx's leading position in the computer vision field is due to the SoC structure that only their ZynQ has. In the case of Zynq US+, it has a Cortex A53 APU, a dual core for CG devices, and a quad core for EG/EV devices. In addition, the R5 Real-Time Processor is a dual core, enabling real-time application processing.

And it has a built-in ARM Mali GPU, so it can perform tasks like rendering images on the display, and in the case of EV devices, it has a built-in H.265 video codec. In addition, it also has cutting-edge power management and security and safety features that are critical to all embedded applications.
Vice President Jeong Woong said, "Zynq SoC has various processors, but the part that is most different from other SoCs is the programmable logic that implements HW to enable parallel processing of algorithms." He added, "If you design an embedded vision system using Zynq SoC, you will be able to experience more flexible and efficient performance than existing processor-based platforms."

Xilinx will be hosting an Open CV public webinar on March 13th utilizing its ZynQ SoC, where it will answer any questions engineers may have about OpenCV in real time.