As per the UNCTAD calculations, the total cargo unloaded through maritime trade in the year 2000 is about 5984 million tons and the same has become 11076 million tons in the year 2019. But this marine ecosystem is quite challenging when it comes to transportation such as accidents, loss of crew, oil spills, Suez canal blockage, effect on marine life etc.,
To bring in efficiency in the above, autonomous vehicles “Ships” may help the maritime ecosystem by minimizing human errors and handling the situations of weather uncertainty conditions. Currently the work on the “Autonomous ship” is in the development of remote-operated vessels in Rolls-Royce. The MV Yara Birkeland is an autonomous 120 TEU container ship that is under construction and is due to be launched in 2021.
Now, if the above autonomous ships use the FPGA platform, There may be better performance in the operational speed of decision making and response mechanisms due to the special features of the FPGA platforms.
We are hereby proposing the design of a prototype of autonomous ships using the FPGA module (along with Azure Cloud) which will have the following features:
1.Ship movement with Global Positioning System.
2.Real-time monitoring of ship (using Azure Cloud).
3.Object detection and lane detection.
4.Route planning without collision.
By automating every task and resolving every challenge we can expect that AUTONOMOUS SHIP can revolutionize the maritime industry. It will allow the ship owners to manage their fleet to optimize operations and maximize profit. Implicitly, as the accidents get reduced (such as oil spilling etc) the pollution in the marine system can also be reduced.

Demo Video

Project Proposal

1. High-level project introduction and performance expectation

The pace of the progress in the latest technical advancement is increasing in an unpredictable way. In this era, we are creating a system which possesses a capability of taking the decision on its own according to the problem. It’s high time we have to look into the problems of marine transport.

The better solution for these problems is “Autonomous ships”, by automating every task for transport and in the shipping process we can expect a better productive output. 

Our prototype is interfaced with the “Camera&Sensors” to observe and extract the information from the surroundings and also interfaced with a “GPS module” for navigation purposes and takes the decision according to the surroundings.The “cloud” connected to the FPGA such that the real time location of the prototype(Ship) can be monitored by the user and security also maintained.

Tasks/Functions of autonomous ship:

1.Ship Positioning is a function which can locate ship position by using the GPS data.

2.Route planning algorithm generates a collision avoidance path by considering the outputs from the “Ship positioning”,“Object detection frame” and “Lane detection frame”    bycamera accurately. 

3.Object detection is an algorithm which can detect and can track objects by using camera    frames.

4.Lane detection is an algorithm which can detect lanes/edges of canals.

5 .Cloud is an online computing platform which includes servers, storage and database."The cloud is the better solution for the security threats in autonomous ships” .

These FPGA outputs are given to the Control unit , which controls all the movement functions to drive the prototype(Ship) in the desired and safest path by overcoming the challenges in the marine system.The path can be changed by user while on running by using cloud computing. Thus, we ensure that our prototype(Ship) travels in the safest path that has no collisions.

2. Block Diagram

3. Expected sustainability results, projected resource savings

As we all know that particularly the marine ecosystem has witnessed a lot of accidents. Human errors are the reason behind most accidents. Human error caused Economical loss, crew loss, damage to the marine ecosystem _[1]. 

Since we are using pre-trained and accurate models, we can expect that the smart ship will not make any estimation, precision and decision-making errors as humans do.

Generally, there are hundreds of Crewmen in container ships along with their requirements. Smart ships open an exciting possibilities to redefine the way a ship is desined at  present. When there are less people on board, many constraints on the ship layout are removed. We can remove or resize the entire deckhouse. This will save cost, as well as enabling the ship to carry more cargo. A ship contains systems only to serve the crew. Thier removal will simplify the entire ship, improve reliabillity and productivity while reducing build and operating costs.

 SMART SHIP can set new standards and layout for the container-ships.

SMART SHIP  helps the shipping industry owners to optimise the ship and maximise their profit.

4. Design Introduction

The design of our prototype is completly based upon DE10 NANO FPGA  board .Firstly we have interfaced camera module to DE10 NANO FPGA board for intake of camera images and these images will process in DE10 NANO FPGA board for object detection ,canal edge detection. DE10 NANO is interfaced with motor driver , ultrasonic sensors and servo motor.

Motor drivers is used to contol the speed and direction of the motors.Ultrasonic sensor is attached to ship's front side for distance measurement. Another ultrasonic sensor combined with servo is attached to back side of ship to measure distance with in three angles .DE10 NANO interfaced with mobile phone to utilize GPS hardware on it. 

This is hardware design of the autonomous ship prototype.

5. Functional description and implementation

The development of autonomous vehicles has seen great progress during the last 10 years. This has been enabled by advances in technologies, which enable perception of the surrounding environment, path planning and vehicle control and vehicle monitoring in real time.

5.1 Situational Awareness:

Situational awareness is defined as the perception and comprehension of information that allows a machine to project future courses of action needed to respond to a dynamic environment. Situational awareness in the context of an automated vehicle can refer to the degree to which the machine is aware of the current and future driving conditions facing the vehicle.

Situational awareness should be particularly important when a driver’s secondary task requires them to disengage from the driving. Performance of the driver is hindered when driver has to perform many tasks simultaneously. Since the machine is capable to do multitasking without degradation of performance, we can increase the level of situational awareness, sustainability and reliability.

One of the key technologies for any reliable autonomous vehicle navigation is sensor fusion. When creating Situational Awareness for an autonomous vehicle, no single sensor technology can provide sufficient performance under all possible conditions. Therefore, in order to guarantee that the information on the vehicle’s surroundings is sufficiently accurate at all times, the input from multiple sensors has to be combined and analyzed.

In this prototype, we have used camera and distance measurement sensors for Situational Awareness.

5.1.1 Camera:

Camera is used for capturing continuous images for image processing tasks.we have interfaced usb based camera with DE10 nano using  micro usb (OTG) cable.

5.1.1.1 Object Detection:

Object detection is the one the main part of our project.The main moto for this is to detect object ahead of the ship so that it can travel without any accidents. So we selected another ship as object .For detection of objects we have used haar cascade classifer method for that we trained ship haar cascade model using viola jones algorithm.

In the case of decision making we have divided image frame(camera) into 4 blocks .Ship will be deviated based on which block does ship is detected .So ship will take left direction for detection on right part of the frame and vice-versa and also we have increased motor speed in turning for middle 2 blocks using PWM signals.Because if ship detected in middle of the frame we have to take deviation very quickly.In other hand if ship detected in any another blocks ship will took turning slowly with respect to block.

5.1.1.2 Canal Edge Detection:

Canal edge detection is mainly used to move ship safely from one to another end of the canal.So to distiguish edges of the canals with water ,we have apllied canny edge detection alogithm with region of Interest mask.Here canny edge algorithm convert camera's input frame not detectable edges after that we applied mask (consider only ROI) for some part of frame where only canal edges are visible clearly.

But resulatant frame has some other noises with canal edges so by using gaussian blur algorithm we have removed those noises.To steering on canal we applied Dialation with adaptive thresholding and probabilistic hough transform methods.Here dialation will improve edge points of canal frame and probabilistic hough transform gives points of desired lines on canals.

we have measured the distance from horizon of the image to both canal's edge lowest coordinate points.The difference between that two distances(Let Dist) gives the streering decision to ship.In detail we compared Dist with Threshold value (Based on canal).

5.1.2 Distance Measurement Sensors:

An ultrasonic sensor measures the distance of a target object by emitting ultrasonic sound waves at a frequency above the range of human hearing and converts the reflected sound into an electrical signal.

In order to calculate the distance between the sensor and the object, the sensor measures the time it takes between the emission of the sound by the transmitter to its contact with the receiver. The formula for the calculation  D=1/2 * T ×C  (Where D is the distance from target object, T is the time stamp difference between received signal and transmitted signal, C is the speed of sound ~ 343 meters/second in air, 432 meters/second in water).

In this prototype, we have used HC-SR04 ultrasonic sensor for front distance, back distance with wide angle coverage, water depth, under water obstacles.

5.1.2.1 Front distance:

An ultrasonic sensor is placed at the bow (front side of the ship) to detect the obstacles which involves in the vehicle path.

5.1.2.2 Back distance with wide angle coverage:

An ultrasonic sensor is placed at the stern (back side of the ship) is interfaced with Servo Motor SG90. Servo Motor sweeps the ultrasonic sensor 0⁰ to 180⁰, which covers the obstacle detection for the remaining three sides of the ship.

5.1.2.3 Water depth:

Water depth is one of the crucial parameter that we have to consider in path planning. Water depth is directly proportional to the height of the tidal waves, which affects the stability of the ship. So we placed an ultrasonic sensor to bottom surface of the ship which is facing downwards to measure the water depth.

5.1.2.4 Under water obstacles:

The ship is partially submerged in the water. So there are some possible obstacles like seamounts and aquatic animals that we have to consider during path planning. To prevent accidents and preserve life loss of aquatic animals, we placed an ultrasonic senor at the bottom part of the bow.

The data collected from sensor fusion carried forward to path planning module.

5.2 Path planning:

Path planning or routing of ship is the interconnection of the start location, end location and some way points in the path. To know the current GPS location of the ship we have used the GPS sensor in a Mobile phone. A Mobile is interfaced with DE-10 Nano FPGA, the current location data and the Magnetometer data in the Mobile is sent to DE-10 Nano FPGA.

This module compares current GPS location with next way point location and generates the control signals for vehicle control. If any obstacles blocks the path, the ship will deviate from the path and then compares with all the next ways points and decides the shortest and safest point to move. If the ship enters into the canals, based on the canal edges distance the ship direction is changed by lane detection module. If any obstacle is detected very close to ship by ultrasonic sensors the ship deviates from its path accordingly.

This is GPS specified path planned for the ship.

5.3 Vehicle control:

In this prototype, we used a motor driver to control two DC motors, which are connected to propellers to drive the ship. To control the driving movement of the ship we have to maintain the direction of the ship and speed control.

5.3.1 Ship Direction:

‘Turn left the ship’ and ‘Turn right the ship’ are the two ways we are using to maintain the ship in desired path and direction. Rotating one propeller in clockwise and the other propeller in anti-clockwise in water makes the ship to rotate left or right. Inward Twin screw mechanism is for right turn. Outward twin screw is mechanism is for left turn.

5.3.2 Speed controller:

We have generated a Pulse Width Modulation (PWM) signal to drive the ship in various speeds. By changing the duty cycle of a PWM signal, the ship can travel in various speeds.

5.4 Vehicle Monitoring:

We have used the below azure web services, azure maps, azure IoT edge, azure device provisioning service for real-time ship monitoring. A webapp named shafan is deployed in azure cloud for data visualization. The data is sent through IoT edge, Device Provisioning Service and azure blob storage.

We have created azure maps and enabled get current conditions service, then we have to use a RESTful API key provided to get the weather details. We can fetch the response of API key from the webapp.

This is the main page which redirects the every owner to his own portal for ship monitoring. This can handle a network of ships data. The route planning for all the ships without interference is maintained.  

In this page, we can fetch the speed of the ship, navigation of the ship with GPS (Global Positioning System), front view of the ship, distance measurement, weather details at a geo location.

A “Safe Point” interrupt can be given by the user to navigate the ship to a safe point if user found any suspicious external conditions. In bad weather conditions the ship’s path is altered by base station.

A black box is created to note all the crucial events faced by ship and decisions taken by the base station. Black box is also the communication tool to inform any crucial events status to all the ships in the network.

6. Performance metrics, performance to expectation

6.1 Ultra sonic sensor:

The reciever signal in ultrasonic sensor travels at high frequency. We have to read recieved signal in 20 to 40 microseconds. We found that DE10 Nano FPGA board is capable of reading ultrasonic sensor data accurately.

6.2 Speed Variations:

When the ship enters into suez canal and panama canal ,the ship has to be restricted below 4 nautical miles per hour. In the case of object detection also the speed has to be altered. It is very difficult to control a ship, since the ship has more intertia compared to inland vehicles. Here the speed is controlled by DE10 Nano FPGA board.

6.3 Hardware Minimization:

Instead of using four ultrasonic sensors for four sides of the ship. we have attached rare ultrasonic sensor with servo motor which rotates 0⁰ to 180⁰. Thus we have reduced the power consumption and minimized the hardware.

6.4  GPS Location:

We got accurate GPS values with an accuracy of 2 meters and with high refresh rate.

7. Sustainability results, resource savings achieved

7.1 Preventing Accidents:

We have designed the prototype such that there is no chance of ship accident by measuring obstacles accurately around the ship.

Ship does not enters into bad weather condition location, it changes its path based on bad weather.

Path planning of the ship includes the wind speed and direction, so that ship can’t flip around at any condition.

7.2 Preserving Marine Life:

With the help of underwater ultrasonic we detect the animals in the water at surface level and take decisions accordingly to save lives of aquatic animals.

7.3 Lights:

We have addressed Night time problems regarding Object detection and Lane detection by enabling sufficient lights in front of the ship with clear light.

7.4 Other ships can detect shafan at night:

We cannot know the ship direction at night time. For this problem we have followed the COLREG (Convention on the International Regulations for Preventing) rules at night navigation. The idea is to put Green lights to right side(STARBOARD) and Red lights to left side(PORT). So that others can know the position and direction of shafan at night.

7.5 Black box in Azure webapplication:

If any changes occur in external environment like weather issues, azure cloud is takes a decision and changes the path of ship and other activities running in the ships. We have created a Black box which stores all the decisions took by cloud. Further we will analyse the data if any wrong decisions are taken.

7.7 Collision Avoidance:

The ship will deviate when an obstacle present in the path, and joins back into the same path.

7.8 Path Plannig:

If If the ship deviates from the GPS route path , after the obstacle has passed the ship finds the shortest and allowed path to join back into the desired path.

8. Conclusion

There is no automation technology involved in ships for their travelling and other activities. With our design, we tried to automate the ship to travel from source point to destination point in a safer way. Our system works efficiently in all the cases related to ship journey. Our design deals with lane detection and object detection algorithms which helps to reach our destination in safe mode. Using GPS co-ordinates, our system will fix the destination and with the developed algorithms it will travel in safer path.

Futher our steps are towards environmental sustainability in marine ecosystem. We want develop an assistance system for smart ship which can clear oil spills and plastic on the surface of water bodies. We want to consider all other factors for safe and stable autonomous ship.