InnovateFPGA

Food Saver aims to reduce food waste in resorts and hotels by preventing overproduction and improving storage and service conditions.

Demo Video

[URL: https://youtu.be/0UtsnCb1Gi4]

Project Proposal

1. High-level project introduction and performance expectation

Hotels and resorts give exaggerated services to satisfy the customers and overlook enormous waste, especially for food. Food Saver aims to reduce food waste in resorts and hotels by preventing overproduction and improving storage and service conditions in excess of 10 billion US dollars annually.

Five significant areas contribute to food waste:

· Storage Losses: The food wasted due to improper and unhygienic storage conditions.

· Preparation Losses: The food mainly consists of seeds, peels, etc., from fruits and vegetables.

· Serving Losses: The food left on serving dishes, in canteens and bowls.

· Overprepareds: The food that is never served.

· Leftovers: The food that the customer leaves on the plate.

Food Saver focuses on the following three points:

Overprepareds

Overprepareds occur when the production of the food is higher than its consumption. Estimating the required amount of food supply is a crucial part of the problem. To overcome this problem, Food Saver aims to utilize the basic correlation between the number of people and the food consumed. Food Saver uses the motion sensors mounted on the doors to count those entering and leaving every meal. Then it sends the acquired statistics to the cloud. Then, the cloud estimates the required food. Over time, accumulated data will be helpful to train an AI-based algorithm to predict the required amount of food even before months.

Serving Losses

The served food must preserve its temperature and moisture for customers' utmost satisfaction, while the food waiting to be served requires the opposite to prevent the growth of mold and other bacteria. Also, the food that is served must be replaced regularly to meet food health standards. So, the problem is the food has to be prepared in large quantities due to timing constraints, but it also should not wait a long time in serving conditions. Food Saver aims to solve this problem by utilizing a serving mechanism that consists of temperature-controlled divisions. Conditions of these divisions are controlled such that the short-term serving foods are contained in optimal temperature divisions. In contrast, longer-term serving foods are held in lower temperature divisions. Food Saver detects when the top division is emptied and ascends the lower division for the serving by using weight sensors. Hence, Food Saver facilitates the food serving process while keeping a higher shelf life.

Storage Losses

Improper storage conditions such as high temperature and moisture lead to mold, and mold releases CO2 (Hesseltine, 1976). Food Saver aims to prevent storage losses by monitoring the temperature, moisture, and CO2 conditions of the food prepared in large quantities. AI algorithm will check the dependency of these parameters in real-time to alert and activate cooling and ventilation systems that will be integrated with UPS power systems (Dagnas & Membre, 2013).

Food Saver sends all the acquired data to the cloud. Hence, it can be a part of a more intelligent, complex, and interconnected network by connecting multiple nodes.

2. Block Diagram

3. Expected sustainability results, projected resource savings

The USA wastes 63 million tons of food worth 218 billion dollars in a year (Briggs, 2016). In this design, our focus is the waste in consumer-serving businesses like hotels and restaurants, which sums up to 40 percent of whole food waste (Himelstein, 2017). Our project will significantly reduce food waste and eliminate unnecessary transportation, workforce, storage area, and decrease carbon dioxide emission. Total value proposition of our solution is well above 8 billion USD and saver of green pastures and part of ecosystem preservation act called Green Deal compatible.

REFERENCES:

Dagnas, S., & Membre, J.-M. (2013). Predicting and Preventing Mold Spoilage of Food Products. In Journal of Food Protection (Vol. 76, Issue 3, pp. 538–551). International Association for Food Protection. https://doi.org/10.4315/0362-028x.jfp-12-349

FSEC Energy Research Center. (2020, August 12). Mold Growth. Retrieved November 4, 2021, from https://energyresearch.ucf.edu/consumer/buildings/building-science-basics/mold-growth/

H. Briggs, J.D. Lindeberg, A. Rein, B. Chorn and K. Tanger. (2016, June 1). The ReFED Roadmap To Reducing Food Waste. Refed. Retrieved October 30, 2021, from https://refed.com/articles/the-refed-roadmap-to-reducing-food-waste/

Himelstein, L. (2017, October 4). Hotel Buffets, a Culprit of Food Waste, Get Downsized. The New York Times. https://www.nytimes.com/2017/09/08/dining/hotel-buffet-food-waste.html

Hesseltine, C. W. (1976). Conditions Leading to Mycotoxin Contamination of Foods and Feeds. In Advances in Chemistry (pp. 1–22). AMERICAN CHEMICAL SOCIETY. https://doi.org/10.1021/ba-1976-0149.ch001

Okumus, B. (2019). How do hotels manage food waste? evidence from hotels in Orlando, Florida. In Journal of Hospitality Marketing & Management (Vol. 29, Issue 3, pp. 291–309). Informa UK Limited. https://doi.org/10.1080/19368623.2019.1618775

Reynolds, C., Goucher, L., Quested, T., Bromley, S., Gillick, S., Wells, V. K., Evans, D., Koh, L., Carlsson Kanyama, A., Katzeff, C., Svenfelt, Å., & Jackson, P. (2019). Review: Consumption-stage food waste reduction interventions – What works and how to design better interventions. In Food Policy (Vol. 83, pp. 7–27). Elsevier BV. https://doi.org/10.1016/j.foodpol.2019.01.009

Youngs, A., Nobis, G., & Town, P. (1983). Food waste from hotels and restaurants in the U.K. In Waste Management & Research (Vol. 1, Issue 4, pp. 295–308). Elsevier BV. https://doi.org/10.1016/0734-242x(83)90034-4

4. Design Introduction

The initial design goals were monitoring the food condition, controlling the food temperature, pre-processing the sensor data, and communicating with the cloud server. FPGA was the perfect choice for such tasks since even the sheer number of sensors and diversified compute requirements were enough to justify an FPGA. The design was aimed toward big hotels and resorts that accommodate a large number of customers that waste huge amounts of food due to the reasons mentioned above. However, during the development phase, we have come to the conclusion that some parts of the design are not feasible.

The temperature-controlled automatic food serving mechanism was aimed to mitigate serving food losses. The system consists of temperature-controlled divisions and an automatic shelf system to serve and replace the food. The major flaw of this design is the movement of the divisions. Each division either has to have its own cooling/heating solution or they have to connect to a center thermal system via moving thermal bridges. It is not practical to implement a thermal system that has moving connections. So, giving each division its own cooling/heating unit was the best option. But, such an approach not only increases the overall system price drastically but also reduces the efficiency of the whole system.

The food condition monitoring system was aimed toward short and long-term food storage. It consists of a gas sensor array and a WiFi card. It monitors the condition of the food and sends the sensor data to the FPGA. Such a system can detect and warn if any bacteria or fungi exist. Before implementing the design to FPGA we have collected data samples via STM32 Discovery Board and TGS2620, MQ2, MQ3, and MQ5 gas sensor array. Our goal was to detect the early development of the bacteria growth since humans may not be able to notice such a situation if there is no visible indication or a distinct smell. Unfortunately, the system was not reliable for early detection. We have tested the system with apples and it sometimes took a whole day before bacteria detection kicks in after visible marks start to appear. Moreover, if the food is ventilated detection delay drastically increases. On top of that, if the sensor array is used to monitor hot or wet food sensors get dirty and their precision is broken. MQ3 has not recovered after monitoring hot mushroom soup.

5. Functional description and implementation

Since the main portion of the design has proven to be unfeasible we have decided to improve the remaining parts. The final design monitors the human traffic via deep neural network based object detection and object tracking. Such information is useful as extra data to feed to a purpose build AI or as a local statistic to decide the amount of food to be cooked.

System Top Level Diagram

Object detection and tracking are sequential operations by nature. First, it has to be continuous in time. Webcam thread acquires frames. Object detection thread detects people from frames. The tracker thread tracks the detected people. One part of the operation can't be started without the previous one being finished. So, the simplest algorithm for object tracking is:

forever:

           read frame

           detect objects every couple of frames

           track objects

           display

This is indeed a viable solution, assuming one turn with the object detection takes less than the frame period. Unfortunately for DE10-Nano HPS, one turn takes 1.2 seconds, which is 24 times longer than the 20FPS video period. There are two viable solutions to speed up this process: FPGA OpenCL object detection accelerator or a multicore algorithm.

The first option is the best choice from a performance perspective. We are guaranteed to solve the FPS issue. But, It is the most challenging and time-consuming choice. Porting a state of art object detection algorithm to OpenCL and debugging it sounds like a nightmare. And as someone who has known about deep neural networks for a couple of months, it adds another layer of doubt.

The second option is the easiest one. We can directly use a pre-trained and tested Deep Neural Network. Also, one of the threads can be easily replaced by an FPGA accelerator down the line and combine it with the first method. But, there is a chance that HPS alone will not be enough to solve the FPS problem.

I obviously went for the second option. Object detection takes 1 second for a single ARM core without tracking and webcam. If I can offload tracking and detection to the second core, we can achieve real-time tracking and a detection update every second. Sounds plausible.

New algorithm:

There are three threads. The object detection thread runs non-stop and detects objects. During this heavy and long computation, the webcam thread stores the frames. As soon as object detection ends, recorded frames and detected object results are sent to the Tracker thread, and the most recent frame is fed into the detector.

The new algorithm introduces a couple of new challenges; Consumer-Producer problem and data duplication problem:

The webcam thread produces an unknown number of frames between each object detection. The size of the frame packet depends on object detection speed. The longer the detection takes, the more frames are produced.
The object detection thread must read the most recent frame.
All threads have to synchronize at the end of object detection.
Threads must not eat away CPU resources when they are waiting for synchronization.
There can't be a large number of duplicate frames.

Solution:

Conditional variables solve the waiting problem. They also mitigate the CPU usage problem when idling. Pointers solve the duplication problem. Preallocate three frame packets and swap the pointers in each thread, starting from tracker to webcam. But, they introduce another problem. Who will swap them?

Swapping requires coordination among all threads. Swapping while one of them is operating on it will generate a core dump. So, the swapper thread either has to be dedicated to its job or wait for three different conditional variables from each thread. Or it has to be the fastest one among the tracker, detector, and webcam. I choose the second option since there are two physical cores on DE10-Nano, and adding more threads will not translate to more performance, at least not directly. So, the fastest thread that will swap frames is the webcam thread. In the worst-case tracker and object detector will be ready just before the webcam starts to read a frame. In this case, wasted compute time will be equal to the frame time.

6. Performance metrics, performance to expectation

An ideal design has to reach near-perfect object detection and tracking performance, otherwise, any other statistical method, for example, estimating the number of customers from plates used, is a more easy and feasible solution. Our design has 70% accuracy according to our tests. It confuses if one people blocks another one from the camera or if there is strong directional lighting and shadows. Also, since the design use frame to frame absolute displacement without any other trajectory predicting methods any objects that are lost from tracking are unaccounted for.

7. Sustainability results, resource savings achieved

Our design's detection capabilities are already behind statistical predictions. However, even if it was operating with a %100 detection and tracking rate resource-saving would still be minimal. Overprepareds are only a small portion of the wasted food. In addition to that predicting number of people is not enough to accurately estimate the amount of food that will be consumed. Which further degrades the significance of the system. Still, collecting data can and probably will be useful for future studies aiming to tackle the wasted food problem.

8. Conclusion

This study has several limitations. First, even though the usage of the gas sensor array is eliminated by deeming it unfeasible, such a system can be implemented by using high-quality sensors, particle filtering and sensor ventilation, and a large dataset. Second, object detection and tracking performance are measured with custom recordings and can not be used to ensure reliable operation in all conditions.

In summary, this paper has investigated different methods to prevent food waste in hotels and resorts. And concluded that food waste prevention requires cheap and easy to enforce solutions. Even reducing one source of food waste requires an overhaul of the current food serving norms. Hence, such methods are not only economically unviable but also unreliable.

Food Saver