AP005 » Active Pedestrian Hazard Detection
This project intends to create a working prototype of an add-on solution for motor vehicles (including motor cycles) to actively detect pedestrian movements and warn of potential collision hazards.
The solution will project information onto a heads-up display to minimise distraction to the driver.
Recently, a friend of ours was hit by a car. The driver admitted to being distracted while at the wheel. I have developed a number of machine learning and image-based projects in recent years, so it is a logical next step to bring these fields of research and experimentation together in order to solve a worthwhile challenge.
My proposal is a small, self-contained apparatus which can detect potential hazards before they enter the path of the vehicle, and alert the driver. By being predictive in nature, the solution could provide a few seconds of extra warning, and this might just be enough to prevent injury, or worse.
Modern cars are frequently equipped with a number of cameras for such tasks as parking assistance and lane detection. Ideally, I would use these existing video sources, since it would simplify the installation; but if these sources are unavailable or unsuitable, a pair of stand-alone cameras could be used.
The software must be able to cope with extremes of lighting and weather conditions. In order to work effectively at night, the cameras will require illumination using Infra-Red, Ultra-Violet or a combination of these systems. The final choice will depend on the nature of the cameras in use.
The apparatus needs to require no intervention on the part of the user, and the output will project simple coloured icons onto the windscreen of the vehicle. This is done to reduce the amount of information to an absolute minimum, yet still be effective.
I investigated using an add-on compute device such as the Movidius NNC, and it would most likely work well, but it is not feasible for an individual to embed these processor chips into a design, and connecting via USB can add long-term reliability issues.
The use of an embedded FPGA device is attractive in the project because, by it's mobile nature, the apparatus must be self-contained and operate disconnected from other computing resources. Using the Intel OpenVINO toolkit allows me to extend the code I have already used and take advantage of the FPGA device.
The binary components provided by Terasic as part of the OpenVINO Starter Kit (OSK) mean that the board can be used as an accelerator without HETERO interface in the OpenVINO tools. This considerably reduced the time to create a workable prototype system, and also allowed testing the code on a notebook machine (albeit at lower levels of performance)
It is hope that eventually technology such at this could be integrated by vehicle manufacturers, or at least made available as a third-party product.
The first diagram shows the basic image processing components.
The second diagram describes the software components. Blue represents Open-Source Library Sets (Such as OpenCV), while yellow represents custom code:
For this project to be viable, there needs to be a high-performance computation platform, storage and image processing. It also needs to be as compact and low-cost as possible.
The FPGA (and, indeed the OpenVINO Starter Kit) allows existing code platforms to be enhanced performance-wise with minimal changes to code. The codebase has been developed continually since 2017 as part of an autonomous vehicle project, but the computational requirements have thus far been prohibitive to offer this functionality as a stand-alone device.
With the Cyclone-V FPGA and OpenCL/OpenVINO, it is possible to produce a system which meets the performance criteria. The device also addresses the need for digital outputs to signal the driver, control illumination and related activities.
While the OpenVINO Starter Kit is unwieldy for production, this package however serves as a proof-of-concept, and a production version would integrate the FPGA into the finished product.
Purpose and Mission
This project is the result of an idea to help save pedestrians from injury. While it makes no attempt (and in fact such attempts would be very difficult) to control the vehicle, it acts as a silent passenger, calling "watch out!" should a pedestrian suddenly stray from the safety of the footpath.
Because there are already so many distractions in a modern car, it is essential that:
This project is targeted mostly at automotive manufacturers, who would be best able to integrate the solution with their vehicles as they are being produced. However, it is also possible that these devices could be fitted to existing vehicles.
In either case, the project needs to be able to be installed with a minimum of understanding of it's inner functions.
One of the most significant challenges is in the time required to recognise a potential pedestrian. Most smaller embedded processor platforms offer limited performance, and in this case using an external processing aid, such as an FPGA is a definite advantage.
The plan is to first produce a proof-of-concept design, most likely using conventional PC components, such as the Terasic OpenVINO Starter Kit to demonstrate that this is viable for further development.
On the assumption that this PoC is successful, the next logical step is to produce a viable prototype perhaps using Mini-PCIe based devices to produce a compact, robust solution. From there, further customisation could take the device to production.
There are four main functional areas in this solution, Image acquisition, detection, and path detection.
A pair of 4K resolution cameras face forward, but offset to the left or right. Each camera is fitted with a 60-degree FOV lens (determined experimentally), and the cameras are aligned so that the images do not overlap.
The images are imported at a 10Hz rate using standard Video For Linux (V4L) functions, and de-blurred using OpenCV libraries. The images are also reduced to grayscale and contrast enhanced to simplify the processing.
Image recognition is implemented using YOLO ("You Only Look Once"), which leverages OpenCV (and by inference OpenVINO). The core functions are offloaded to the FPGA to improve performance.
A match table lookup is performed for objects which are predicted to be likely to move in such a way as they would cross paths with our vehicle. As a vehicle might be moving at 60Km/h (17 m/s), these operations must be performed very quickly. Using the FPGA to accelerate object management is critical to this task.
Once a probable subject has been detected, it is essential to determine the current path, as well as predict the likelihood of a collision. This is based on an assumption that the vehicle's path will not vary in the next second, so we have 10 frames of image data to process. If the vehicle is moving at a more-or-less constant speed and in a more-or-less constant heading for these frames, then the motion of the subject is easier to calculate.
Image and path detection are run in parallel streams for each image source.
A decision was made early in the design process to make the alerting mechanism as simple as possible, and not to impose additional workload or distractions on the driver. To this end, the prototype uses an LED matrix (8x8) to draw an arrow which is reflected onto the windscreen (windshield). In the prototype, this display is controlled by an Arduino Nano controller because of issues with the GPIO interface via the linux driver. In time, this issue can be resolved and the matrix would be controlled via the FPGA's GPIO ports.
The code for the project is under continual development and may be reviewed on GitHub:
This is essentially a real-time application. The cameras each feed the device at 10 frames per second, and all predictions must be complete within 1 second.
Since there are two cameras involved, there are two computational threads, and this doubles the workload of the processor.
Field testing of the code with and without FPGA acceleration gave a 90x improvement in throughput for a single camera scenario, which was well above the expected outcome.
The following diagram details the main processing section of the device. Because there are two cameras which may detect images independently, the detection routine runs in parallel threads within the process. The flowchart show duplicate steps for clarity of purpose. The code itself is not duplicated.
Motion processing is an independent thread which monitors a queue structure. Images are placed on the queue by the core routines. When an image is processed, it must be determined firstly if the object is moving since the last frame by comparing a coordinate set. This is valid since the frame rate is relatively high compared to the speed of the vehicle.
If the object has moved, the direction is then checked against the position of the camera to determine if the object is approaching the camera laterally. If so, an alert is raised.