The high level of river pollution in industrial and metropolitan environments negatively impacts the ecosystem and raises the government's cost of maintenance and cleaning of those areas. A significant number of rivers, though, are never or rarely cleaned since there is not enough data about their pollution level, nor where the garbage foci are. Hence, a significant portion of all the river waste worldwide is never discovered. It remains unattended, increasing environmental degradation and furthering the impunity of bad actors that pollute rivers without having their actions put to judgment.

We propose a river waste monitoring system composed of an UAV equipped with an image capturing and processing device based on a FPGA. The proposed system also includes support stations with solar panels that will send the collected data to a cloud application while also sending data back to the UAV about its energy status.

The UAV will be capable of flying over the river waters and the riverbank in search of waste. It will also have a small compartment onboard, that will be used to collect water samples, which will then be sent back to a support station for testing.

The support stations will aid the UAVs by collecting solar energy to charge them and analyzing the water collected by the UAVs.
The cloud application will have a dashboard that will display the garbage accumulation spots on a map alongside pictures of the trash and historical data.

The FPGA is an essential part of the system because it will provide the computing power and precision needed for the obstacle and waste recognition algorithm to work in real-time while also being energy-efficient.

The project's expected outcome is to have a relatively low-cost, self-sustainable autonomous system that can be easily deployed on various rivers and efficiently map the river and riverbank area for garbage accumulation spots while also assessing the water quality. This system will provide the local government and agencies with real-time and detailed data about the river's health and waste accumulation spots. Furthermore, the data gathered will be valuable in policies and efforts to restore the river's condition and educate the local community about correct waste disposal practices and other ways of ensuring the nearby river's well-being.

Project Proposal

1. High-level project introduction and performance expectation

Purpose of the design:
Deliver an autonomous UAV-centered, FPGA-powered system capable of periodically gathering actionable data about a river’s water quality and garbage spots. It will provide the captured information in a visualization platform, empowering the local society and government with precise insights indispensable to plan and execute projects to restore and maintain the river’s health.

High-Level project introduction:
An UAV will fly along a river taking pictures and processing them to look for garbage. This data will be sent to a support station when the UAV is docked to it, that then will be sent to a cloud database, where it will be displayed on a dashboard.

• The system will autonomously detect garbage accumulation spots and send data about them (such as pictures, GPS location and timestamp) to the cloud.

• The UAV will capture water at strategic points on the river and transport it to the support stations to test for pH levels that will be logged to the cloud.

• All the data captured will be sent to a cloud server that will store and analyze the data sent by the UAV and the docking stations, garbage accumulation spots, water quality, and displayed it in a dashboard and a map.

• The system will have docking stations where the UAV will be able to recharge and take shelter from the weather. Those stations will have water testing capabilities and will communicate to the system’s servers sending data about its location, docking status and battery level.

• The UAV will follow a previously planned path and decide, based on its current battery status, whether to continue going further down the river or to go back to the nearest support station for recharging. The support stations will directly communicate with the UAV when it’s docking or docked, so they can cooperate without relying on the system’s servers.

Application scope:
Periodically monitor a river’s health and conservation, log pollution level and map garbage spots. Show the collected data with a time dimension in order to see the history on how the pollution of a river increases or decreases as time passes. 

Targeted users:
Government, Researchers, River cleaning & related business, Local population.

Why use Intel FPGA?
Intel FPGA devices are an integral part of the system’s design, since the UAV will rely on high performance edge computing to execute image recognition, manage navigation and communicate wirelessly, while having to be carried and powered on the air.

• In order to detect and classify obstacles and garbage, the FPGA will run an algorithm based on Convolutional Neural Networks (CNN), which is normally a very computationally intensive and power hungry task designed to run on specialized hardware such as ASICs and GPUs, though an FPGA is very capable for this workload because of its high parallel processing power and portability, thus being the ideal solution for such a setting. The FPGA will feed the ARM microprocessor in real time with the output of the algorithm.

• The navigation of the UAV will be orchestrated by the ARM microprocessor on the FPGA on board the UAV and will follow a predefined path but make small adjustments and take decisions in real time based on its battery life and obstacles, to avoid, respectively, running out of energy and crashing. Also, when garbage is found, the navigation system makes the UAV stop and collect a water sample,after this, go back to the nearest support station to provide the sample for testing, and then get back on the predefined path along the river.

• Wireless communication with the support stations is indispensable given that it is the way the UAVs will know where the closest support stations are and what’s their status, which is crucial to ensure that they can safely get shelter and recharge their energy when needed.

All of those tasks need to be run in parallel, thus highlighting the parallel processing power needed. They also need to be run in real time in order to take part in the decision making of the UAV, which could have a prohibitively high latency if the computing were to be done on the cloud, while also being less reliable for deployment on remote environments. Hence, accentuating the need for an edge-computing system.

2. Block Diagram

High-Level Functional Block Diagram :

Low-Level Functional Diagram : 

3. Expected sustainability results, projected resource savings

We expect the project to facilitate rivers' surveillance and cleaning efforts, which will avoid the degradation of these water bodies by processes such as eutrophication, which can have a devastating effect on the river's well-being and biodiversity. We also believe that it can help recognize abnormalities that may or may not necessarily be related to pollution, such as high levels of fish mortality. Furthermore, such actions can also have a tremendous positive impact on the local communities since the change in the water quality would help them with the extractivism of the river fauna.

Since the system will be able to handle multiple concurrent users accessing the map and dashboard, it will be possible for the riverside community to have access to all the collected data, which can help raise awareness about the importance of taking care of the rivers.

We also foresee the potential for marine conservation since research has shown that most of the plastic that ends up in the oceans comes from the rivers. Thus, by promoting rivers' restoration and conservation, we are also helping to improve seawater quality.  

We project both economic and environmental resource savings, respectively, by providing a low-cost and more efficient alternative to human surveillance and by deploying a self-sustaining system that can rely only on renewable energy to be fully functional.

4. Design Introduction

At first, we extensively researched ways we could do object recognition with an FPGA, and we found different methods. We decided to go with an implementation made with HOG (Histogram of oriented gradients) and SVM (Support Vector Machine) since we found a good amount of materials that could serve as a basis for our project. During the semifinal qualification period, we started doing our own research and implementation of the mentioned technologies.

We experimented with different datasets and decided to go with the FloW Dataset, since it contains images of various types of garbage floating in river waters and had already been annotated with bounding boxes for the position of each object of interest. After data augmenting the dataset, we trained multiple models with the techniques mentioned for different slices of the dataset.

We decided to use the FPGA as a server that would receive the images for processing, process them, and make the results (bounding boxes) available on the internet.

5. Functional description and implementation

Our SVM implementation differentiates between two classes of objects: positive and negative, being positive the class of objects we’re trying to detect and negative everything else that may be in the images but are not part of the object we want to  detect. We’ve tried two ways of training, garbage being the positive class and the environment being the negative, and the opposite, using the garbage as the negative class and using the environment as the positive one. We experimented with this approach because of the low flexibility present in the HOG SVM method: it is good for classes that contain instances of objects with very similar borders, but it is harder to get good results when the positive images have a high level of border format variability, such as garbage.

In order to train the model with a bigger dataset, we coded some data augmentation algorithms to mainly rotate and distort the images so the model would be able to recognize the same object from different perspectives.

Once the model is trained, we apply the sliding window technique that slides a window across the image: at each window we compute our HOG descriptors and apply our classifier in order to decide if there is garbage inside that window. After this, we will have finished scanning the image, so we apply a non-maximum suppression algorithm to remove redundant and overlapping bounding boxes.

One of the biggest challenges we faced on the model implementation was the presence of reflections on the surface of the water, both on the dataset images and when running the detection in real time. A solution would be to use a special polarizing filter camera lens and create a new dataset from scratch, while using the same kind of lens when running the system in real time, but due to time constraints we were unable to experiment with this approach.

Another great challenge of using the HOG SVM technique was the constraint of having to use only one size of image (and hence, aspect ratio) for the objects, which led to images of objects with high amounts of water in it, which in practice brought more noise to the training.

In order to implement the http server on the FPGA, we focused on finding a way of exchanging data between the FPGA’s arm processor and the available FPGA logic gates, so we could implement the server on the arm processor and send the images to a digital module on the FPGA that would be responsible for outputting the results of the sliding window technique running on a pre-trained HOG SVM model. We experimented with ways that the wifi connection via RFS could be used to implement the communication between the FPGA and the source of images.

6. Performance metrics, performance to expectation

The garbage detection model presented good performance by the automatic metrics our software outputted, however, in practice, the results weren’t really accurate due to the noise introduced by the large amount of reflections seen in the water sections of the images.

7. Sustainability results, resource savings achieved

As we did not finish the implementation of the system, no sustainability results or resource savings could be achieved nor measured

8. Conclusion

The contest gave us, undergraduate computer engineering students, an incredible experience for having given the opportunity to work with the De10-Nano and with other technologies such as the images classification methods HOG SVM and datasets such as TACO, FloW and UAVVaste.

We had some problems during the development of our garbage detection method. Given that we used some datasets that had high levels of reflection on the water, that ended up being noise for our method of training. This led the model to have a high efficiency, but, not very accurate, for example, the model was classifying some points correctly and other points wrongly. Because of this, we searched for other datasets to improve our model

In addition, we had some problems, such as delays in the delivery of the DE10-Nano board and the quick evals boards that were not sent to us. Because of this we were not able to develop the aspects of water evaluation and its PH. Another problem faced during the proposal submission/Selection to the development Design period was the presence of the covid 19 pandemic, which made it impossible for the group to meet in person to work together and focus on the boards development, leaving one person with the board and the others helping from distance.

Finally, our project is not 100% functional and with some points in development and others halted. Although we haven’t achieved our project’s idea to its fullest potential, we hope to have contributed with the contest and be able to contribute further on future events realized by the InnovateFPGA organization and the partner companies that gave us the opportunity to participate in the competition.

############# Reflections on the achieved project and the design's potential #############
Since our project aims to help the efforts of categorizing and mapping the water quality and pollution level in river waters, it helps the mission of achieving clean water and sanitation with cleaner rivers. However, this is such an important problem, that, by tackling it, we are directly or indirectly encompassing a lot of other sustainability goals, that are closely related to the quality of the waters close to communities: by ensuring that the water is clean, we raise the general well-being of the population nearby, not only by preventing diseases caused by bad quality water but also by protecting life below water and consequently on land, which a lot of times are both resources of food that help decrease the hunger factor, along with the levels of poverty. With the knowledge of which rivers have the most pollution or in which part of the river most garbage accumulates, the government and ONGs can try to understand the source of pollution and work together with the local people to help turn their cities into a more sustainable place. All of this is very significant, since it influences the lives of millions of people, both close to the rivers, by its direct influence, and across the globe, by the effects of climate change.

Our project is one of simple but reliable design, and its complete implementation could realistically be deployed in any area, be it close to the city, or located in remote places (since it would be able to run without constant internet connection), and thus, be able to make a real-world impact on our understanding and collection of data about pollution in rivers all over the world.

By categorizing and locating hotspots of waste in rivers, and utilizing this information to create actions that aim to clean them, we could very deeply reduce the amount of garbage that ends up in the ocean, since most of the ocean waste comes from rivers. By doing this we would certainly be able to lessen the environmental impact of garbage on water bodies both local (rivers) and global (across the oceans), and help all countries achieve a cleaner future for their citizens.

There is a great deal of research and publicly available papers on the impact of garbage and waste on water bodies, both international and located in our home country, Brazil.

Our project is still in the works but is close to coming to fruition and real-world testing. We have already understood all that needs to be done and to achieve a functional state, we just need to finish implementing the following points:
- A visual garbage detection model with higher accuracy (we already have one in python but is still of bad accuracy. In these last days, we started experimenting with a model trainer programmed in C, but we still need to run tests to accurately judge its results ).
- The integration of the visual detection module with the FPGA (the module is external and already exists, we just need to insert it in the FPGA and ensure it runs smoothly).
- The communication between the FPGA and the ARM processor (these last days we configured the necessary interface: we made the memory mapping and we are now able to read and write from and to memory from both the FPGA and the ARM processor. This is already implemented, we just need to run tests to ensure its reliability)
- The communication of the FPGA via network to be able to send navigation commands to the drone, and also receive the images (the means of communication with the drone already exist, and these last days we configured the USART port which gives the ARM's operating system access to the WIFI chip in the FPGA's kit. Going forward, we just need to perform the connection itself of the FPGA kit with the drone, utilizing a smartphone app that already exists as an intermediary).  

In order to measure the actual success of the project once it is fully completed, we will be careful to use the following SMART metrics:
1. The FPGA must be able to function as a server, receiving images uploaded by the drone's extended software to a local network
2. The ARM processor and the FPGA must be able to function together smoothly, without major bottlenecks, and be able to achieve a processing rate of at least 1 frame per discrete drone position.
3. The garbage detection algorithm must present a low amount of false negatives, but a medium amount of false positives is doable. It is more important to not fail to detect pollution than to be 100% accurate all the time.
4. The drone must be able to perform more than one discrete step per minute, and the system must be able to handle this workload without lag, in order to maximize energy efficiency.
5. The system must be able to function without internet access or, at least, be modular enough to enable a future implementation to function without the internet: only on a local network.

All the points are specific, measurable, relevant, time-based, and realistically attainable if we have the chance to keep developing them further.