Annual: 2019

AP035 »
📁Machine Learning
📅Oct 11, 2019
Regional Final

👀 5960   💬 15



For the past few years, there is an increasing trend in the road accidents across the world, there is no exemption even for the developed countries like US for this problem, which reported 40,000 accidents deaths in the year 2018. This shows the gross severity of the problem which should be addressed immediately.
Road accidents lead to lots of unpredictable consequences such as abnormal deaths, perpetual injuries, loss of earnings, etc. The primary causes of these unexpected accidents are distracted and drowsy driving, over speeding, violation of traffic rules, drunken driving, etc. To overcome these constraints, we aim to design an efficient and reliable autonomous car.
Autonomous car is capable of sensing its environment and takes decisions in accordance with the rules without the intervention of human beings. Our design works on the concepts of Convolution Neural Networks (CNN), Machine Learning, Computer vision, and Image/Video Processing. In our project, the autonomous system will detect the lane and obstacles, calculate the distance from the obstacle to the car, blow the horn when a pedestrian is detected and controls the car’s acceleration based on the information given to the system.
Simultaneously, the provision will be made for live audio streaming (especially for the visually impaired) and displaying video streaming to the persons inside the vehicle.
Parallel architecture features of FPGA will be utilized for arriving quick decisions using CNN.
If time permits, we may plan to implement destination arrival using the GPS.

Demo Video

  • URL:

  • Project Proposal

    1. High-level Project Description


                                Figure 1: Hierarchical view of the design   

    Initially, the system interfaces with various cameras and sensors installed at various parts of the car. Our system contains mainly three parts. Those are Information, Processing & Decision, and controlling.

    In the Information part, we collect the data from car surroundings with the help of cameras and sensors. After that, we give the data to the processing & Decision block. It analyzes the data and performs lane detection, object detection, and distance measurement using DE-10 Nano. It extracts some features and passes the control signals to the controlling block. Based on those signals, the car's movement controlled. The perfectness of obtained information makes processing decisions efficiently and gives accurate control signals for the safe journey of a car.          



                            Figure 2: Real time Implementation of autonomous car on DE-10 nano board

    Lane Detection

    Lane detection is the first and foremost step in the self-driving car. We have used image processing algorithms for lane detection, which consists of Canny Detector (a multi-stage algorithm optimized for fast real-time edge detection), Hough Transform techniques to identify the lanes.

    Object/Pedestrian Detection

    Object detection is a key component in the self-driving cars by combining with the sensors, which helps in decision making and motion control while ensuring the stability of the vehicle. In our design, we have implemented the object/human detection algorithm with pre-trained HOG (Histogram of Oriented Gradients ) and Linear Support Vector Machine Model using Opencv Computer Vision library with python.

    Distance Measurement

    According to object/Pedestrian detection, the system measures the distance from the object with the help of the sensor. Based on the measured distance, the system will take the decisions to control the car's acceleration, steering, and horn. Here we are using the LIDAR technology for measuring the distance.



    Personal vehicles are the target of our design. The purpose is to avoid uncontrolled car collisions, careless driving on highways, bad decision making by the driver, inexperienced driver's driving and to improve smooth traffic flow, and so on by using machine learning algorithms and the enormous power of FPGA.


    This idea has vast applications in various fields.

    • Visually impaired people are free to access this car.

    • On a military basis, by making some modifications in the system design, it may avoid itself to become a target for the opponent.

    • The machine learning concept used in this project could also be applied to agricultural vehicles like a tractor with some modification of the design.

    • By using this algorithm, we can develop robotic vehicles and send them to a new planet.


    Not only normal people, both visually impaired and physically disabled people, are able to access this car easily.



    2. Block Diagram

    System flow diagram:-



    3. Intel FPGA Virtues in Your Project

    In our project, we mainly deal with real-time video processing and data processing.

    • Real-time video processing is a relatively complex and resource-intensive task why because to get real-time responses in the output based on the input video stream. We need instantaneous processing for all the inputs. So we require a high computing power device. That can be done with the DE10 Nano kit. It consists of 800MHZ Dual-core – Arm cortex-A9 processor, which is feasible to complete all the required tasks.

    • FPGA has the ability to parallelize the tasks, while the size consumption of such a system will be less than the consumption of CPU and GPU.

    • FPGA boost up the performance of Neural Networks. Especially for the image processing technique, FPGA takes less time as compared with a microcontroller.

    • Therefore, FPGA is the best choice for the developing of the devices that are designed for the processing of video streams.

    • Expansion of I/O

    4. Design Introduction

    Purpose of design

    This design revolves around the concept of self-driving cars with the latest technology of FPGA. The central idea behind this design is to reduce the death rate of humans while traveling on roads. We may avoid the uncontrollable car collisions, careless driving, lousy decision taken by drivers, violation of traffic rules and over speeding, etc. On the other hand, in the upcoming years, it is going to play a crucial role in the automobile industry.

    This design has the capability and functionality with a wide range of applications for any vehicle. Design of our project can be used in different areas mainly on-road vehicles(i.e., car, bus, truck, transportation vehicle). 

                                            Figure 3: Different views of the autonomous car

                                                 Figure 4: LIDAR position in the car

             Figure 5 :  The placement of DE-10 Nano board, Cytron motor shields, arduino in the design 




                                                  Figure 6: Position of the Mobile_camera and monitor 


    Design components :

    TF-MINI LIDAR - LIDAR(Light Detection and Ranging) is a distance measuring sensor that can detect obstacles and gives the information of distance between obstacle and car. 

    Mobile_camera - it is embedded with the DE10 Nano board. Also, it streams the live video from the environment when the car is running.

    DE10 Nano board - This board plays a central role in our entire design.

    Arduino Uno - It controls the car's acceleration by commanding the motor shields.

    Motor shields(cytron) - It can control the car speed, and it will give the perfect directions to move by receiving the commands from Arduino.

    Monitor - Live display of all the tasks performed by the car.

    12v Battery - To energize the car.


    Hardware Implementation :

    In the above figure 3, we can see our design implementation. It has three motors - two are at the backside for thrusting, and another one is placed in the front for steering. The steering decision is available after road lane detection and serially communicates the DE-10 nano board. The car is controlled by the live video taken from Mobile_camera which sends video frames to DE-10 Nano takes the frames as an input. This forms a closed-loop feedback system with Mobile_camera as a sensor, DE-10 Nano board as a controller, and motors as actuators. On the other hand, TF MINI LIDAR controls the car's acceleration by measuring the distance of any object/pedestrian detected.

    5. Function Description

    Lane Detection:

    Lane detection is one of the preliminary steps in the autonomous vehicle. Watching lane lines highlighted by the computer vision algorithm as the vehicle moves give a great joy of accomplishment. Detecting lane lines is indeed a very crucial task. It provides lateral bounds on the movement of the vehicle and gives an idea about how much deviated, the car is from the center of the lane. Lane detection relies only on camera images. We apply computer vision techniques to process the image and give lane markings as the output. We have used the OpenCV library to implement the Lane Detection Algorithm.

    Implementation of the Lane Detection is as follows:

                          Figure 7: Algorithmic flow of lane detection

    1. Getting Input Image:
     In our algorithm, we have taken an input frame (Live Video stream) from the USB tethered Mobile camera, which connected to DE-10 Nano through OTG port (See Fig. 8 (a) ).

    2. Image resizing:
    Resize the input image for optimized processing and convenient display. In our algorithm, we have resized the image into a 640 x 480 resolution.

    3. BGR to HSV Color space conversion:
    BGR color space describes the colors of the image in terms of the amount of blue, red, and green present. HSV color space describes colors in terms of Hue, Saturation, and Value of the image. We require a better representative model of color, and hence, we have the HSV space.

    4. Color Extraction & Bitwise Operation:
    Extract desired colors from HSV converted  image with BGR upper and lower threshold values. Make a mask with the above-extracted output. Now, perform bitwise operations to delete the unnecessary part in the orginal image using a mask. It results the image with our desired color ranges. In our case, we have extracted the yellow color on the road dividers and deleted the remaining unnecessary colors (See Fig. 8 (b) ).

    5. Grayscale conversion:
    It Converts the extracted image into grayscale. Color increases the complexity. The luminance of a pixel value of a grayscale image ranges from 0 to 255. It reduces the ambiguity and complexity of multiple pixel values of colors: from a 3D pixel value (R, G, B) to a 1D value. It is very useful when dealing with image segmentation, thresholding, and edge detection (See Fig. 8 (c) ).

    6. Gaussian Blur:
    In image processing, a Gaussian blur (also known as Gaussian smoothing) is the result of blurring an image and removing noise by a Gaussian function. In Gaussian Blur operation, the image is convolved with a Gaussian filter. The Gaussian filter is a low-pass filter that removes the high-frequency components. It is nothing but using different-weight-kernel, in both the x and y directions. A kernel is nothing more than a (square) array of pixels (a small image so to speak).

    The standard deviation of the Gaussian determines the degree of smoothing. The Gaussian outputs a `weighted average' of each pixel's neighborhood, with the average weighted more towards the value of the central pixels. Gaussian provides gentler smoothing and preserves edges better than a similarly sized mean filter. In our case, we have applied Gaussian smoothing for better edge detection (See Fig. 8 (d) ).

    7. Canny edge detection:

    The Canny edge detector is an edge detection operator that uses a multi-stage algorithm to detect a wide range of edges in images. The Canny edge detection algorithm is composed of 4 steps.

    •  Noise reduction: First step is to remove the noise with a 5x5 gaussian filter
    •  Gradient Calculation: Smoothened image is then filtered with a Sobel kernel in both     horizontal and vertical directions to get the first derivative in the horizontal direction (Gx) and      vertical direction (Gy). From these two derivatives, we have calculated the edge gradient and direction for each pixel.               
    • Non-Maximum Suppression: After getting gradient magnitude and direction, a full scan of an image is done to remove unwanted pixels that may not constitute the edge. For this, at every pixel, a pixel is checked it is a local minimum in its neighborhood, in the direction of the gradient.
    • Hysteresis Thresholding: This stage decides which are all edges and which are not. For this, we need two threshold values, minimum and maximum values. Any edges with intensity gradients more than maximum value are sure to edge and those below minimum values are sure to be non-edges, so discarded. Those who lie between these two values are classified as edges or non-edges based on their connectivity (See Fig. 8 (e) ).

    8.  Region of Interest Selection (ROI): 
    We need to avoid the unnecessary edges to plot the Hough line transforms. By performing Arithmetic operations on the image, we can get the interested and optimized selected area to plot the lane (See Fig. 8 (f) ).

    9. Dilation with Adaptive Thresholding:
    Dilation is one of the Morphological operations, that apply a structuring element to an input image and generates an output image. Here, the binarized ROI image is given as an input to be dilated with a structuring element of 5 × 5 kernels to fill the gaps and to remove small random variations. Dilation adds pixels to the boundaries of detected objects in an image. To make the image binarization, we have used the Adaptive Thresholding method, which calculates the threshold for smaller regions of images. It gives better results of images with varying illumination. The process of binarization is starting with image division into small strips and then applying the global threshold to each strip (See Fig. 8 (g) ).

    10. Probabilistic Hough Transform:
    Hough transform is a feature extraction method for detecting simple shapes such as circles, lines, etc in an image. To Increase the computing speed without losing much accuracy, we have decided to work with the Probabilistic Hough transform. 

    The algorithmic flow of the Hough transform is as follows:

    • Edge detection, e.g. using canny edge detector.
    • Mapping of edge points to the hough space and storage in Accumulator.
    • Interpretation of the accumulator to yield infinite lines of length. The interpretation is done by thresholding and possibly other constraints.
    • Conversion of infinite lines to finite lines.

     We have considered some parameters and passed to this hough transform are dilated image, distance resolution of the accumulator in pixels as a rho, angle resolution of the accumulator in radians as a theta, threshold for the accumulator, the minimum length of the line and the maximum gap between the lines.

    11. Drawing Lane Lines:
    From the points of detected lines in Hough Transform, we have drawn the lane lines with geometric shapes on the image (See Fig. 8 (h) ).

    12. Path Deviation Calculation from Lane:
    It’s a very crucial and final step in the lane detection algorithm. We have to decide and stabilize the position of the vehicle based on the lane on the road. Here we have measured the distance from horizon to top on both sides left and right. We have measured the distance from the horizon of the image to lane’s lowest coordinate point Let left distance be DL and right distance be DR and calculated the difference from DL and DR. After many trials using trial and error method we have set one threshold of difference for decision making to steer and to balance the stable position of the vehicle on the road. As the extracted road region is approximate, consider a tolerance limit of Delta
    . The pseudo code for deciding the steering direction is given below.

    If  DL-DR  =  Delta → Steer forward, no turns

    If  DL-DR  <  Delta  → Steer Left 

    If  DL-DR  > Delta  → Steer Right 

                                                                         Figure 8: Implementation of lane detection

    Object/Pedestrian Detection:

    Object detection is a key component in the self-driving cars by combining with the sensors, which helps in decision making and motion control while ensuring the stability of the vehicle. In our design, we have implemented the object/human detection algorithm with pre-trained HOG (Histogram of Oriented Gradients ) and Linear Support Vector Machine Model using Opencv Computer Vision library with python.

    Histogram of oriented gradients (HOG) is a feature descriptor used to detect objects in computer vision and image processing. The HOG descriptor technique counts occurrences of gradient orientation in localized portions of an image - detection window, or region of interest (ROI). Gradients ( x and y derivatives ) of an image are  useful because the magnitude of gradients is large around edges and corners ( regions of abrupt intensity changes ) and we know that edges and corners pack in a lot more information about object shape than flat regions. According to Dalal and Triggs, we have taken the recommended values for the HOG parameters.

    Implementation of the HOG descriptor algorithm is as follows:

    • Divide the image into small connected regions called cells, and each cell compute a histogram of gradient directions or edge orientations for the pixels within the cell.
    • Discretize each cell into angular bins according to the gradient orientation.
    • Each cell's pixel contributes weighted gradient to its corresponding angular bin.
    • Groups of adjacent cells are considered as spatial regions called blocks. The grouping of cells into a block is the basis for grouping and normalization of histograms.
    • Normalized group of histograms represents the block histogram. The set of these block histograms represents the descriptor, the distribution ( histograms ) of directions of gradients ( oriented gradients ) are used as features.

    The Support Vector Machine is a supervised Machine Learning algorithm, which is highly preferred by many as it produces significant accuracy with less computation power. Support Vector Machine can be used for both regression and classification tasks. However, it is widely used in classification algorithms like object detection and image classification. In our object detection case, the HOG descriptor provides extracted features to SVM to train the model. Then we can take the trained SVM model to test the input images for detection results.

              Figure 9: Algorithmic flow of object detection.

    Working procedure: 

    We have a pre-trained model integrated with the OpenCV library. So that we can inference these pre-trained models in real-world scenarios. A real-time video stream is given to the system which processes each frame. Initially, we loaded the pedestrian detector “Histogram of Oriented Gradients descriptor,” then we set a pre-trained model “Support Vector Machine” with OpenCV built-in functions. We need to remove some noises and resize the image/frame.

    Resizing the image ensures that less sliding windows in the image pyramid analysis that need to be evaluated (i.e., we extract features from HOG descriptor and then passed on to the Linear SVM), thus reducing detection time. It improves the accuracy of the detection result. 

    Now by instantiating the multiscale people detection function, which constructs image pyramid analysis and sliding window from HOG descriptor by passing some arguments such as image, scaling factor for image pyramid, and step size of x,y directions for sliding window.

    The multiscale detection function returns the output of bounding box coordinates of detected person/object and weight of production.   

    While detecting the multiple persons in a single image/frame and also detecting the person, there is a case of getting unwanted bounding boxes, and overlapping will occur. We need to suppress them to get a good result. In this case, we used the Non-Maxima Suppression technique to resolve this problem. Non-Maxima Suppression (NMS) works on the principle Mean Shift Algorithm ( helps to find the maximum points of the bounding box ) to remove the extra bounding boxes, and bounding boxes overlap by applying some threshold to NMS function.

    We can get an accurate object/people detection results (Fig 10).  

                                Figure 10: implementation of object detection

    Distance Measurement :

    In our project, Measuring of distance from an object to car can be considered as a prominent role. This measured distance can control the car’s acceleration which can avoid accidents.

     For this purpose, we have embedded a Distance Measuring Sensor named TF MINI LIDAR, which can sense objects or obstacles. It is mainly capable of real-time and contact-less distance measurement and is featured by accurate, stable and high-speed distance measurement.


    TFmini based on TOF, namely, “Time of Flight principle.” To be specific, the sensor transmits a modulation wave of a near-infrared ray periodically, which wave will reflect after contacting the object. It will obtain the time of flight by measuring the round-trip phase difference and then calculates the relative range between the sensor and the detection object.


                                           Figure 11: Schematics of TOF Principle

    Data Communication Protocol:-

    TFmini adopts the serial port data communication protocol.

    Data Output Format and Code:-

    Data structure: Each data package is 9 Byte, including the distance information (Dist), signal strength information (Strength), distance mode (Mode), and data check byte (CheckSum), etc. The data format is hexadecimal (HEX). 

    Data Format:


    Verilog Implementation:

    The universal asynchronous receiver- transmitter (UART) takes bytes of data and transmits the individual bits in a sequential fashion. At the destination, a second UART re-assembles the bits into complete bytes. Each UART contains a shift register, which is the fundamental method of conversion between serial and parallel forms.

                                                                            Figure 12: Data frame format

    The above figure12 shows, in UART data will flow in a serial fashion. Here the time period for every bit is inverse of baud rate (1/115200). With the help of this basic idea to retrieve the serial data from TF-MINI LIDAR we made a block level design in verilog.

      TF-MINI LIDAR block contains mainly three sub blocks:               1) Baud rate generator

                                                                                                                  2) Receiver (Rx)

                                                                                                                 3) Transmitter (Tx) 

                                              Figure 13: Top module of TF- MINI LIDAR implemented in verilog

    Baud rate generator:

             This module will create the ticks that are 16 times faster than the baud rate i.e. 115200 our system clk_frequency is 50 MHz. So the width of main clk is 20ns. And the baud rate of TF-MINI LIDAR is 115200, so we have generated TICKS with a frequency 16 times of UART signal. The width of the UART signal is 1/115200 equal to 8.6us.

    To generate one TICK we required 27 clk pulses of width 20ns.     {((8.6us/16)/20ns) = 27}.

    By following this pattern of TICK generation, we generated a TICKS train.

    Receiver (Rx):

             In the receiver, the receiving module receives the 9bytes of serial data. Initially, we access the one byte of serial data. In that one byte of data to retrieve one bit of data, we need 16 TICKS. Every time after obtaining the 8bits of data (BYTE), the stop bit will warn about the arrival of 8bit of data into shift registers. By taking the 9 bytes of data serially into shift registers, we measured the distance from the sensor to the object. 

                      Figure 14: Timing diagram for collection serial data

    Figure 14 shows how the data from UART was accessed.

    Transmitter (Tx):

    In the same manner, it transmits the data serially.


    Car’s Motion Control:

    Car’s Motion control entirely depends on the output of the decision block. It executes the commands by receiving control signals from processing and decision block. The Processing block consists of a Lane detection and Object detection algorithm. They generate some output, sent to the decision block. Parallelly, Lidar sends detected distance from object to Decision block. Decision block collects all the data from the execution of different algorithms, process it, and it will send appropriate signals to the control block.

    The control block receives the signals, and it maintains the stable position of the vehicle by controlling the car’s actuators like steering, accelerator, brake, and horn. The vehicle’s (CAR) motion can be considered in four ways “Thrusting, Steering, Deceleration, and Stop.”

    Steering wheel can be managed by signal came from the lane detection algorithm. It says whether the car should turn left or turn right or go straight forward. (Our car has Ackermann steering geometry, and it  controlled with motor shield named Cytron MD20amp). To decelerate or blow the horn or stop the vehicle, the decision will come from the Lidar.

                                                     Figure 15: Block diagram of car's motion controller


    6. Performance Parameters

                                                                Figure 16: TF MINI LIdar Main module(RTL Schematic) 

                                                             Figure 17: Baud rate generator module (RTL Schematic)

                                                            Figure 18 : Receiving module (RTL Schematic)

                                                         Figure 19 : Transmitting module (RTL Schematic)

                                                Figure 20 : Logic utilization summary

                                                                             Figure 21: Output of Lane Detection.

    (a) Input frame.

    (b) Colour Extraction.

    (c) Gray scale Conversion.

    (d) Gassian Blur.

    (e) Canny edge detection.

    (f) Region of Interest.

    (g) Dilation.

    (h) Lane detected Frame.

                                                               Figure 22: Output of object Detection


                                                        Figure 23: Commands for motion control


    7. Design Architecture

                                                                         Figure 24: Design Architecture


    Nazeer Shaik
    All the best
    🕒 Jul 08, 2019 01:55 AM
    Keep hard working... All d bst...
    🕒 Jul 07, 2019 06:10 PM
    All The Besttt
    🕒 Jul 06, 2019 09:30 PM
    An interesting project. ..All the best
    🕒 Jul 05, 2019 02:10 PM
    Aleksandr Amerikanov
    An interesting project.
    I would like to receive explanations from you:
    How will work your system with simultaneously temporary and permanent road markings? Marking lines can intersect and contradict one another altogether. How does it determine the priority of markup?
    The same story with a partially erased road markings, especially at crossroads.
    In general, the markup usually complements the traffic signs. What if the markings contradicts signs, or even temporary signs?
    The same story with the crossing guard.
    🕒 Jul 05, 2019 07:29 AM
    Dear Aleksandr Amerikanov,
    Thank you for showing interest in our project. Its our pleasure to answer you.
    Our system will treat both temporary and permanent road markings as same. In some scenarios like ‘work is in progress’ on the roads, we meet simultaneous temporary and permanent road markings, and then we will go with temporary road markings.
    As you have mentioned that the marking lines can merge and contradiction may arise. It is absurd that depending on marking lines completely as they can be manipulated unconditionally. So the solution for the problem is that creation of virtual marking lines artificially while considering the original marking lines simultaneously. In the cases of which you mentioned, we prefer virtual lane creation using canny detection algorithm and Hough transform.
    When partially erased marking lines at the places like cross roads also, virtual lane creation aids us to tackle the problem. The system traces the virtual line rather than completely relying on the markup lines on the road.
    In general markings reflect the traffic signs. In very rare cases we see traffic signs contradiction. If we meet such situations, we choose lane markings.
    More over in the case of signs contradiction at crossing guards, we prefer crossing guard signs and update the routing map based on the destined path.
    We hope our answer clears all your queries.
    🕒 Jul 07, 2019 02:00 AM
    Dr. Jason Thoreou
    An interesting concept.

    In my experience, canny edge detector and Hough transform face difficulties over range of lighting, dust conditions. For example, sharp shadows could be easily picked as virtual lines and in countries like India, many roads do not have well-defined lanes. Such errors could cost lives in a real world implementation. Which is why the autonomous navigation community has moved to deep learning.

    It is good to see you are not relying on a well-defined map, but inferring from the environment while navigating. Basic image processing algorithms that work on tightly controlled demonstration setups may fail drastically in real world situations. It would be a good idea to use lidars (or time of flight sensors). I have seen some research papers which have implemented a lightweight lidar-based SLAM (Simultaneous Localization and Mapping) on low-end FPGAs. It would be fairly straightforward to directly use their implementation in making your project applicable to real-world practical situations.

    Good project idea and best of luck!
    🕒 Jul 07, 2019 04:26 AM
    nice work yaar . can encourage more students for this type of works.
    🕒 Jul 04, 2019 03:00 AM
    b v r adiseshu
    very innovative work. we need more works like this
    🕒 Jul 04, 2019 02:56 AM
    Ashok Varma Mallagunta
    All the very best
    🕒 Jul 03, 2019 10:26 PM
    Naga Surya Sai Eswari . Galidevara
    All the best for your project
    🕒 Jul 03, 2019 06:14 PM
    Naga Ramya saladi
    All the best for ur project
    🕒 Jul 03, 2019 02:34 PM
    Naga Ramya saladi
    All the best for ur project
    🕒 Jul 03, 2019 02:33 PM
    All the best for the project.
    🕒 Jul 01, 2019 12:25 PM
    Pavan krishna
    Very thoughtful of you. Nice work.
    🕒 Jul 01, 2019 12:25 PM

    Please login to post a comment.