Apply eFPGA to an embedded 360-degree vision vision system

On April 11, 2018, the Ministry of Industry and Information Technology, the Ministry of Public Security and the Ministry of Transport jointly issued the "Notice on Printing and Distributing the "Intelligent Network Linked Vehicle Road Test Management Regulations (Trial)", which provides for the intelligent road network test of China's intelligent network. Relevant legal basis. While the three ministries have given qualifications to the intelligent networked vehicles, they have also proposed a number of strict conditions.

Among them, in the seventh chapter (four) of the second chapter "test subject, test driver and test vehicle", the three ministries require: with vehicle status record, storage and online monitoring function, can return the following first in real time 2, 3 items of information, and automatically record and store the following information at least 90 seconds before the occurrence of a vehicle accident or failure, the data storage time is not less than 3 years:

1. Vehicle control mode;

2. Vehicle location;

3. Vehicle speed, acceleration and other motion states;

4. Environmental awareness and response status;

5. Vehicle lighting, signal real-time status;

6. 360 degree video surveillance outside the vehicle;

7. Reflecting in-car video and voice monitoring of test driver and human-computer interaction status;

8. Remote control commands received by the vehicle (if any);

9. Vehicle failure conditions (if any).

It can be seen that the above conditions, in addition to several functional requirements for intelligent networked vehicles for road testing, will also promote the development of several new generations of communication, monitoring, control and storage technologies in related fields. The development of automotive-specific SoCs offers new market opportunities.

Embedded FPGAs (eFPGAs) will play an important role in such chips. For the purpose of satisfying the acquisition and processing of 360 degree video surveillance data of the vehicle mentioned in Article 6, it is obvious that using eFPGA to design related function chips. As a company that provides both stand-alone FPGA chips and eFPGA IP products, Achronix can help smart car SoC designers develop and debug related functions on the FPGA chip first, and then migrate the design to the market without having to significantly modify the design after entering the batch application. On top of the SoC equipped with eFPGA.

Apply eFPGA to an embedded 360-degree vision vision system

The embedded 360° vision vision system with multiple high resolution cameras has entered various applications such as automotive sensor fusion, video surveillance, target detection, motion analysis and more. In such systems, multiple real-time camera video streams (up to six) are aggregated and processed frame by frame for distortion and other image artifact correction, exposure and white balance adjustment, and then dynamically stitched into a 360° panoramic view Output at 4K resolution and 60 fps frame rate, and finally project onto a spherical coordinate space.

High resolution fisheye camera lenses currently used for such applications typically have a wide angle of view (FOV). One of the biggest bottlenecks in the surround camera system is: storing/reading and accessing multiple camera input data in real time or from external memory, and then processing it as a single frame. The hardware needs to complete the processing operation between the original sensor data and the spliced ​​output video input from the input camera within one frame delay.

High-performance computing platforms have been moving toward the use of FPGAs in conjunction with CPUs to provide specialized hardware acceleration for real-time image processing tasks. This configuration allows the CPU to focus on particularly complex algorithms where they can quickly switch threads and contexts and assign repetitive tasks to an FPGA to act as a configurable hardware accelerator/coprocessor/unload engine. Even with FPGAs and CPUs as discrete devices, the system can increase overall efficiency because they don't conflict, but they fit together like a glove on your hand.

For example, images obtained from fisheye lenses suffer from severe distortion, and splicing operations based on multiple camera video generations are highly computationally intensive tasks because of the point pixel operation. This stitching requires a lot of real-time image processing and a highly parallel architecture. However, this next-generation application exceeds the performance that FPGAs can continue to implement, mainly due to the latency of the chip's throughput data. This in turn affects the overall latency, throughput speed, and performance of the entire system.

Add eFPGA semiconductor intellectual property (IP) that can be embedded with the CPU in an SoC. Compared with a stand-alone FPGA chip plus CPU solution, the embedded FPGA array structure has unique advantages, the main advantage is that the performance is stronger. An eFPGA can be directly connected to an ASIC (no I/O buffer) through a wide parallel interface, providing significantly improved throughput and latency counted in single-digit clock cycles. Low latency is the key to complex image real-time processing, such as correcting the distortion of the fisheye lens.

With Speedcore eFPGA IP, customers can define their logic, memory and DSP resource requirements, and Achronix can then configure its IP to meet their needs. Lookup tables (LUTs), RAM cell blocks, and DSP64 cell blocks can be combined like building blocks to create the best programmable structure for any given application.

In addition to standard logic, embedded memory and DSP unit modules, customers can define their own function blocks in the Speedcore eFPGA structure. By integrating these custom function blocks with the traditional building blocks into the logic array structure, optimized functions can be added to reduce the area and improve the performance of the target application, which can greatly improve the performance of eFPGA, especially for embedded Visual and image processing algorithms are very effective.

Successfully solving high-performance image processing with custom cell blocks is a good example. You can use YOLO, a state-of-the-art, real-time object detection algorithm that uses neural networks. Early methods greatly improved performance. The algorithm relies on a large number of matrix multipliers, which are built using FPGAs and RAM blocks when implemented in an FPGA; the optimal configuration between the DSP and RAM blocks required by YOLO, and a typical FPGA There is a problem with the mismatch found in the array structure. For example, an FPGA array structure might provide 18×27 multiply/accumulate block blocks and 32×128 RAM of DSP block blocks, while the best solution at this point might be a 16×8 DSP block with 48×1024 RAM. By creating custom cell blocks that implement optimal DSP and RAM module configurations, the resulting Speedcore array structure uses 40% less chip area to achieve the same functionality and achieve higher levels of system performance.

Embedding an FPGA array structure in an SoC provides two additional system-level benefits:

Lower power consumption - Programmable I/O circuitry accounts for half of the total power consumption of a stand-alone FPGA chip, while an eFPGA can be connected directly to other modules in the host SoC, eliminating the need for large programmable I/O buffers .

Lower system cost - Since eFPGA only needs to implement specific functions, the eFPGA's die size is much smaller than the equivalent independent FPGA chip, because eFPGA no longer requires programmable I/O buffers and unnecessary interface logic.

Thanks to ultra-low latency and real-time processing, 360° view-based vision systems can be effectively implemented. Speedcore eFPGA with custom cell blocks is used in conjunction with a CPU in the same master SoC, making it ideal for dedicated functions such as target detection. And image recognition, distortion and distortion correction, and finally stitching the final image together. Embedding an FPGA array structure in an SoC is a natural development process for system integration in the ultra-deep sub-micron era.