Rapid AI innovation is hindering AI algorithm proof-of-concept (PoC).
Adaptive hardware is ideal for AI productization and commercialization.
Easy to use adaptive hardware without specialized knowledge


As the pace of innovation in the AI field accelerates, change is also occurring rapidly. While the software industry has successfully commercialized AI, hardware industries, including automotive, industrial, and smart retail, are still in the early stages of AI productization.

This is because obstacles still exist between proof-of-concept (PoC) for AI algorithms and actual hardware deployment. These include small data sizes, imperfect inputs, and constantly evolving models.

Software developers and AI scientists can overcome this challenge with adaptable hardware.

◇ Overcoming small-scale data challenges through parallel AI inference and training on edge devices.

Giant internet companies like Google and Facebook collect and analyze massive amounts of data every day. In contrast, the hardware industry has much more limited access to big data, which results in less-than-perfect AI models. Therefore, continuous efforts are needed to improve accuracy, including collecting more data and performing training and inference on identically constructed hardware in a fully online environment.

Adaptive computing, such as edge-proven FPGAs and adaptive SoCs, can address this by continuously updating with newly collected data to perform both inference and training.

Traditional AI training requires cloud or large-scale on-premise data centers and can take days or weeks to complete. In contrast, most real-world data is generated at the edge. Running both AI inference and training on the same edge device can improve total cost of ownership (TCO) and reduce latency and security concerns.

◇ Imperfect inputs must be compensated and accelerated through pre- and post-processing

While publishing AI model proof-of-concepts (PoCs), such as those for improving coronavirus detection accuracy using X-ray images, is becoming increasingly easy, most PoCs rely on well-calibrated input images. In reality, input from cameras and sensors in medical devices, robots, and moving vehicles often contain arbitrary distortions, such as dark images and objects at various angles.

Therefore, these inputs must be calibrated and reformatted through sophisticated preprocessing before being fed to AI models. Furthermore, postprocessing is also crucial for understanding the AI model's output and making appropriate decisions. While some chips are exceptionally capable of accelerating AI inference, most can only accelerate a portion of the overall application.

For smart retail, the preprocessing involves decoding multiple streams of video, followed by traditional computer vision algorithms for resizing and reconstructing the video and converting its format. Post-processing includes object tracking and database lookups.

End customers are not sensitive to AI inference execution speed, but they are concerned about meeting video stream performance and real-time responsiveness across the entire application pipeline.

FPGAs and adaptive SoCs have proven their ability to accelerate pre- and post-processing through domain-specific architectures (DSAs). Adding AI inference DSAs allows the entire system to be optimized to meet overall product requirements.

◇ The latest models are constantly changing, so existing devices must be reprogrammed to cope.

In the AI research community, leading AI researchers around the world are developing new AI models every day. These innovative models can improve accuracy, reduce computing requirements, and handle new types of AI applications.

Innovation continues to put pressure on existing semiconductor hardware devices, driving the need for new architectures that efficiently support advanced algorithms.

Standard benchmarks like MLPerf show that modern CPUs, GPUs, and AI ASIC chips often underperform their advertised performance by 30% or more when running real-world AI workloads. This creates a growing need for new DSAs that can keep pace with the pace of innovation.

Today's trends that further accelerate the need for a new DSA include:

Depthwise convolution is a layer that requires efficient, high-capacity memory bandwidth and specialized internal memory caching. AI chips typically have fixed L1/L2/L3 cache architectures, limiting internal memory bandwidth and resulting in low efficiency.

Researchers are continually developing new "custom layers" that most current AI chips don't natively support. Running these on a host CPU without acceleration often leads to performance bottlenecks.

Sparse Neural Networks (SNNs) are a key optimization area, capable of reducing network size by up to 99% by trimming network edges and removing fine-grained matrix values from convolutions. Efficient hardware implementation requires specialized SNN architectures and dedicated encoders and decoders, which most AI chips do not support.

"Binary/Ternary" is an extreme optimization method that performs all mathematical operations using bit manipulation. Most AI ASICs and GPUs only have 8-bit, 16-bit, or floating-point calculation units, resulting in extremely low precision, making it difficult to improve performance and power efficiency. FPGAs and adaptable SoCs are the best solutions for developers to develop complete DSAs and reprogram existing devices according to the application's workload.

This is confirmed by the latest MLPerf data, where Xilinx and Mipsology achieved 100% of the hardware datasheet performance based on the ResNet-50 standard benchmark.

◇ DSA implementation is now possible without hardware expertise.

Until now, the biggest challenge with FPGAs and adaptive SoCs has been the hardware expertise required to implement and build DSAs. However, the emergence of tools such as the Vitis™ unified software platform, which supports widely used AI frameworks like TensorFlow and Pytorch, as well as C++ and Python, has significantly narrowed the gap between software and AI developers.

In addition to software abstraction tools, open-source libraries such as the Vitis hardware acceleration library are gaining increasing adoption among the developer community.

Xilinx's recent design competition attracted over 1,000 developers, resulting in a plethora of innovative projects ranging from hand-controlled drones to reinforcement learning using binary neural networks. Notably, most of the submitted projects were developed by software and AI developers with no prior FPGA experience.

This is evidence that the FPGA industry is taking the right steps to empower software and AI developers to solve real-world AI productization challenges.

Until recently, general software developers and AI scientists were unable to take advantage of adaptable hardware. In the past, this required specific hardware expertise, but new open-source tools have emerged that allow them to use adaptable hardware.

FPGAs and adaptable SoCs have become accessible to countless software developers and AI scientists due to their newfound programmability, and these devices will become the hardware solutions for next-generation applications.

Indeed, DSA will be the future of AI inference as software developers and AI scientists leverage adaptable hardware for next-generation applications.



This article is a summary of the article titled “How adaptive computing solves AI productization challenge” by Nick Ni, director of product marketing for AI, software, and ecosystem at Xilinx.