In this project, we propose the first underwater and deep-learning-enabled intelligent microbial delivery system for coral reef habitat recovery. The system will be able to deliver coral probiotics and monitor its efficacy. The delivery is precisely regulated by a deep learning network that monitors the color change of corals.

Demo Video

Project Proposal

1. High-level project introduction and performance expectation

Coral reefs ecosystem is a vital habitat of the oceans, about 25% of the ocean's fish depend on healthy coral reefs. However, this habitat is endangered by climate change. Climate change causes the temperature of water to increase, and the slightest change in temperature causes the corals to bleach. The bleaching process occurs when the coral starts to expel algae that are living in its tissues, causing it to lose its color and turn white. Without algae, the corals will lose energy and will not be able to undergo the photosynthesis process. If the coral is whitened for a long period, it will eventually die. Different types of microorganism have been found to provide benefits to the coral reefs. These beneficial microorganism for corals (BMC) can retard or even stop the bleaching process. 

Several laboratory studies demonstrated the efficacy of BMCs. Long-term experimental strategies in-situ must be developed to understand the specific role of BMCs in coral recovery. FPGAs provide a low-cost, flexible, and reconfigurable platform with great stability and efficient embedded memory, which makes it very suitable for prototyping and testing at edge in real-time applications. In recent years, deep learning has achieved great success in various aspects of our life, continuously improving the intelligence of machines. Edge-enabled systems can be completely self-regulated and deliver super-human accuracy in certain applications, such as image classification. Hence, we propose a smart computer-vision-enabled underwater microbial delivery system regulated by a FPGA processing core for coral reef habitat recovery. The proposed system makes it possible for marine researchers to conduct precise and reliable monitoring experiments on the efficacy of BMCs in reducing bleaching effect, thus contributing to a significant impact on restoring coral reef ecosystem.

2. Block Diagram

2. System design and real-time deployment

The proposed system is illustrated in Fig. 1. The whole system consists of a FPGA processing core, a BMC delivery subsystem, solar panel, and Microsoft Azure cloud. The DE10-Nano Cyclone V SoC FPGA Board is used as the main control and computing engine. The BMC delivery subsystem is responsible for pumping BMC solutions to corals. The solar panel provides power. The Microsoft Azure cloud remotely controls the delivery system and monitors the coral conditions. 

Fig. 1 The proposed system design.

The details of the system components are elaborated as follows.

BMC delivery subsystem:

The essential parts of this subsystem are a pump and a BMC reservoir. The pump used to dispense the BMCs will be directly controlled through PWM from the FPGA core. A MOSFET driver will be responsible for the power control. This subsystem is also responsible for sampling temperature data and monitoring the condition in the BMC reservoir. Temperature sensors, level sensor, and EVAL-CN0397-ARDZ are installed. Temperature sensors monitor the surrounding temperature of both the corals and the reservoir since BMCs are also affected by the temperature. Level sensors detect the amount of BMC solutions currently available in the reservoir. Since the sea luminosity affects the corals, the underwater luminosity will be continuously monitored using EVAL-CN0397-ARDZ. The top of the BMC subsystem is designed to be transparent.

FPGA processing core:

The FPGA processing core consist of the DE10 FPGA equipped with cloud connectivity kit, the DC1012A-A, DC1338B, and solar battery. The FPGA processing core is a deep-learning-enabled intelligent processing platform deployed at edge, mainly used for the following purposes: temperature monitoring, reservoir level monitoring, sea luminosity monitoring, coral status classification, and BMC delivery control. We use DC1012A-A to interface the sensors in the BMC delivery subsystems and provide digital data for the FPGA core. The processing core uses the temperature data and level data from the reservoir to monitor the health status and availability of BMCs, respectively. It then transmits the information to the cloud for researchers to be aware of the conditions of the BMC reservoir. It also receives the underwater coral temperature data and luminosity data and analyzes the surrounding conditions of corals. Temperature data can reflect the bleaching process of corals to some extent, but can not convey precise conditions of corals. So, we install an underwater camera to capture the images of corals periodically. The main computing work in the core is done to classify the coral images captured by the camera and determine the bleaching stage of the corals. For the classification, we implement a convolutional neural network (CNN) on the FPGA. CNNs are by far the most powerful image classification method, which have surpassed human in many applications. They utilize high-dimensional convolutional filters to extract essential input features and correctly classify the input. The CNN can be trained offline with high accuracy before being transferred onto the FPGA. Thus, the CNN-equiped FPGA core can determine the bleaching stage very precisely and make prompt decision as to the proper amount of BMC solution and the best delivery time for any bleaching stage. The classification process can be seen in Fig. 2. This processing core is deployed at edge and can make decision by itself without researchers’ interaction. Additionally, the DC1338B is used to monitor the power harvested and used from the core. This way, the user will know when the solar panel gets damaged or covered. 

Fig. 2 NN processing and command action.

Microsoft Azure cloud:

All the data received by the FPGA processing core are transmitted to the cloud through the connectivity kit. The cloud is used to store the data for experimental analysis and visualize the change of corals under the influence of the BMCs. The cloud also allows for the researchers to interact with the system.

Based on the proposed design, we have conceived the detailed plan for real-time deployment at edge, as reflected in Fig. 3. A buoy will be used as a supporting platform for the FPGA processing core and solar panel, which will be located near the target corals. A sealed and water-proof customized delivery system will be deployed underwater closely to the corals and hardwired to the buoy. A small mobile internet station will be installed on the buoy, creating a Wi-Fi network for data transmission between the FPGA and cloud.

Fig. 3 The proposed plan for real-time deployment.

3. Expected sustainability results, projected resource savings

The system will log the temperature, measure the intensity of light that reaches the corals,  and be able to automate the release of the BMCs into coral reefs habitat. The system also needs to take pictures of the corals periodically, process it and send all the result data to Microsoft Azure cloud. Furthermore, the motion sensors of the cloud connectivity kit can be used to monitor the underwater sea current and patterns. 

As better stated in section 7, the creation of a smart Coral Reef BMCs delivery system that can aid the recovery of the corals and provide fundamental data on bleaching factors and agents.

4. Design Introduction

Because of the harsh environment, it is essential to minimize the number of components in the system. For instance, a single microcontroller would not be sufficient to process data related to multiple BMC reservoirs, such as image acquisitions, temperature readouts and reservoir control. Yet, the Intel FPGA offers the best solution to this project, with the FPGA interfacing all cameras plus the reservoir parallelly. The processing of all images using hardware accelerated filters takes only a fraction of a second. By using a single controller for the whole system, we can deploy a single battery and the FPGA can be encapsulated near to it. The scaled up of the system can be acheived by simply connecting additional reservoirs/cameras to the FPGA/battery.

The figure below shows the battery used for this project. The enclosure was custom made, as well as, the holders for the battery and the connection to it. The need of a centralize control was a key point for this project. 

5. Functional description and implementation

Below it is the functional diagram of the process implemented by the FPGA. The stiring and the release are executed by the secondary system. The FPGA only sends the command to do it so.

The final system block diagram is given below:

The secondary system can be composed of the BMCs delivery subsystem with or without sensors. For real-life deployment, only one secondary system is composed with sensors, as the temperature and light condition will not vary within the region of interest. In this work, the secondary system is controlled using a microcontroller interfaced with the FPGA by UART protocol. Below is a picture of an enclosure with a microcontroller interfacing EVAL-CN0397-ARDZ. The code for it was modified from the original code provided here.

6. Performance metrics, performance to expectation

Coral bleaching identification

With the goal of improving the outcome of the project, we tested different approaches. Classical computer vision approaches were verified to deliver faster and reliable results for coral bleaching detection when compared to CNN-related techniques.. Thus, we are continuously working to further improve the outcome of the project.

An original image from a coral undergoing the bleaching process is shown in Figure 6.1. We artificially modified the original image to show two examples: Fig 6.2, with the coral being revitalized, and Fig 6.3, with the coral in decay. To know if the BMCs are recovering the corals, the colored pixels of the segmented images are counted. The total number of pixels is fed to the system's user, who can then identify if the coral is bleaching or reverting the bleaching process. Figure 6.4 shows a bar graph of the pixel count after segmenting.

Figure 6.1 Original photo from an underwater coral at the red sea.

Figure 6.2 Artificially revitilized coral.

 Figure 6.3 Artificially decayed coral.

 Figure 6.4 Pixel counting for each cenario.

BMC release system

The design proposed in Fig. 1 and 3 was modified to accommodate the environmental constraints of corals and the technical aspects of the project. Throughout the tests, the beneficial microbials started agglutinating, creating the need for a stirring mechanism/device. Instrumentation wise, the modules DC1012A-A and DC1338B have not been implemented in the final design due to not arriving in time. 

Figure below shows the current setup for BMC delivery.

Data monitoring on Azure cloud

The Azure cloud is used for data visualization and monitoring of the rehabilitation process. The cloud solution was deployed following the template provided by Microsoft for the competition. The code used to upload the data was written in python and followed the original code from Intel Github (source). 

Here we show different data streams (temperature, level, light and image results[pixel count]) as a function of time. Based on the size of the transmitted data and the quantity of messages needed to be transmitted daily, the selected Azure plan was the F1. 

The Microsoft Azure template was modified to allow a better visualization of the data. The dashboard is given in figure 6.7. The main enclosure is the system controller, where it is located the FPGA.

 Figure 6.7 Dashboard created in Microsoft Azure cloud.

The user can also check further information, such as the "health status", from the raw data. The health status would simply be implemented by acquiring the amount of segmented pixels in the image and subtracting it by the pixel count of the day before. If the pixel count is higher, means that the coral is recovering. Fig 6.8 shows the screenshot of the raw telemetry data.

 Figure 6.8 Screenshot of telemetry.

Open problem: real life deployment

As part of performance metrics and performance expectation, the real life deployment of the system is a key factor for our project. We tested the corals in lab conditions at the Red Sea Research Center (RSRC)’s corals tank setup. As can be seen below, the segmentation process is still challenging, with the pixel count prone to errors because of the movement of marine animals that pass in front of the corals and the regular sea movement.  One way to minimize such error is to acquire multiple images from the coral, and if the comparison between two sequential images displays a big difference between in pixel count, the measurement/conclusion can be said as faulty. 

 Figure 6.9 Coral tank setup at the Red Sea Research Center (Saudi Arabia).

Figure 6.10 Output from Fig 6.9.

7. Sustainability results, resource savings achieved

The proposed intelligent system will help the assessment of corals by automating field experiments. This will require less manual control and is less prone to human errors. The work is also beneficial cost-wise, because each diving expedition is costly and unfeasible to be made on a daily basis. As the first open-sea probiotic experiment in the world, this project will provide meaningful data and help researchers understand key factors about the use of BMCs for coral reefs recovery from bleaching. We believe that the proposed system will make a significant contribution to the restoration of coral reef ecosystem. Figure 7.1 and 7.2 shows a manual BMC application process and the world's first coral probiotics village in the Red Sea, located around 20 km off the shore. 

Figure 7.1 Manual application of BMCs at the Coral village

Figure 7.2 Red Sea Coral research village 

By automating the coral bleaching identification process, we decrease the need of costly expeditions and trained marine researchers. The project can be scaled up for fit the different coral reefs environments around the world, creating a cloud-based world-wide coral reef recovery system.

We highlight the following research papers that explore the uses and potential of BMCs for coral reef recovery. Note that they use manual process, as showing in Fig 7.1, to delivery BMCs for their project. Our smart solution would bring a direct impact for such kind of research. 

[1]SANTORO, Erika P. et al. Coral microbiome manipulation elicits metabolic and genetic restructuring to mitigate heat stress and evade mortality. Science Advances, v. 7, n. 33, p. eabg3088, 2021.

[2]VOOLSTRA, Christian R. et al. Extending the natural adaptive capacity of coral holobionts. Nature Reviews Earth & Environment, v. 2, n. 11, p. 747-762, 2021.

[3]SILVA, Denise P. et al. Multi-domain probiotic consortium as an alternative to chemical remediation of oil spills at coral reefs and adjacent sites. Microbiome, v. 9, n. 1, p. 1-19, 2021.

[4]VILELA, Caren Leite Spindola et al. Exploring the Diversity and Biotechnological Potential of Cultured and Uncultured Coral-Associated Bacteria. Microorganisms, v. 9, n. 11, p. 2235, 2021.

8. Conclusion

Coral reefs recovery is a long process that can be facilitated by using Intel FPGA and monitored by using Azure Cloud. A preliminary result of the project is demonstrated here, with the project being expanded and used in real life in the near future. So far, the project has been tested under laboratory condition, and performs satisfactorily. As an outlook for improving performance, hardware accelerated filters will be applied, with the goal of processing images even faster than the current state. Also, additional sensors will be used for data gathering on the coral reefs habitat with the goal of enabling an ocean 'internet of things'.

Code avaliability

The code of this project is shared at the following repository.