China is the world's largest tobacco producing country. In 2016, the tobacco industry obtained 10795 billion yuan for industrial and commercial tax, in which 10006 billion yuan had been paid to state. In china, about 16 million mu of tobacco is grown in the whole country, and the total output is about 41.5% of the world's total output. Thus, tobacco becomes an important cash crop for our country, and plays an important role in Chinese agricultural economy.

In recent years, with the improvement of people's living standards, more and more people are aware of the importance of good tobacco quality to health. Based on this concept, people desire to improve tobacco quality. Because the only way to improve the quality tobacco products is to improve the quality of tobacco, thus, the work of tobacco grading is getting more and more important, and the classification of tobacco leaves has become the basic work for tobacco industry. However, at present the classification of tobacco leaves in China is mainly done by manual which refers to grading tobacco leaves by workers’ subjective feelings of the visual, tactile, smell and so on. This method has the shortcomings of low accuracy, low efficiency and great subjective factors. In the field of tobacco automation classification, most of the foreign and aboard research results are still restrained in the theoretical or laboratory stage, and there is a big gap with the goal of marketization, industrialization and commercialization. Therefore, this paper proposes a novel tobacco automatic grading system with Intel FPGA.

In this paper, an automatic tobacco leaf recognition and grading system is designed and made. The system is composed of six functional modules, such as DE10 - Nano development board suite, high precision pump, two-dimensional slide rail, conveyor belt, D5M camera kit, indexing disk and so on。The system can carry tobacco leaves on the conveyor belt after image collection, pattern recognition and classification, and use a sucker to automatically load different grades of tobacco into different containers, so as to realize the classification. The whole system has the advantages of high automation, easy to use, safe and hygienic, and low cost. The design realizes a miniaturization and high efficiency tobacco grading system, and solves the current situation of tobacco grading mainly depending on manual completion.

A、Hardware

This design has the advantages of high automation, convenient use, safety and health and low cost, and divides the structure into "grab - transmission - image acquisition - Data Processing - control - grading" parts. The system is mainly composed of six components as follows: (1) DE10 - Nano development board Suite (2) high precision pump (3) two-dimensional slide rail (4) conveyor belt (5) D5M Camera Kit (6) indexing disk. The physical block diagram can be shown in Figure 1.

Figure 1. Block diagram

The functions of each function module are as follows:

1. DE10-nano development board：Terasic DE10-Nano is a development kit based on Inter SOC, which integrates the capabilities of a Cyclone V FPGA and a dual core ARM Cortex-A9 processor. In this system, DE10-Nano is the core of the work. Some image data processing and the controlling work are processed by it.
2. High precision pump：High precision pump can absorb the tobacco leaves so as to automatically granting tobacco leaves. Because of its adjustable suction ability, it can ensure that the tobacco leaves are not damaged and stable during the process of absorbing tobacco leaves.
3. Two-dimensional slide rail：Two dimensional slide rail can be moved in two directions, horizontal and vertical. In this system, after the high precision pump grabs the tobacco, the two-dimensional slide rail controls the displacements of the high precision pump, and finally the tobacco leaves are moved and placed on the conveyor belt.
4. Conveyer belt：The conveyor belt is responsible for transporting tobacco leaves and providing an image acquisition platform. There is an image acquisition area in the middle of the conveyor belt. When the tobacco leaves are transmitted to this area, the images can be collected. When the indexing plate rotates to the right direction, the conveyor belt can transfer the tobacco leaves to the container of the corresponding indexing plate.
5. D5M camera：D5M is a 5 million pixel high-speed camera launched by Terasic Company, which is responsible for the collection of tobacco leaf images in this system. When the tobacco leaves are in the image acquisition area, the camera takes the image of the tobacco leaf and sends the collected image data to DE10-nano, and then a specific IP core written in the FPGA side of the DE10-nano development board will analyze the data.
6. Indexing plate：A number of containers are installed on the panel to recover the tobacco. When the image data is analyzed by FPGA, the result is passed to the ARM, then Linux is used to control the GPIO, and make the indexing plate to rotate based on the image analyzing results. Finally, the tobacco can eventually flow into the right container from the conveyor belt.

B、Software data

The software part mainly analyzes the image data collected by the camera and the control of the two dimensional slide, indexing plate and the display of the grading results. On the image data processing, we use the OpenCL technology to design a characteristic IP verification image data processing and format conversion to realize the analysis of the tobacco image data. In the control of the mechanical peripherals, we use the Linux to control the GPIO to control the peripherals, while Linux opens the VGA driver to display the hierarchical results on the LCD screen. The user can make the display grading results and the number of tobacco leaves by the control of the dial switch. The whole system takes full advantage of FPGA's high speed parallel processing capability and ARM Linux's flexible computing and system control advantages. The data flow of software is as follows:

1. Linux uses the fd=open () function to open the USB driver to read the image data collected by the camera and store it in the DDR3 memory physical address of the HPS through the virtual address mapping.
7. FPGA uses OpenCL to realize an image processing IP core, and DDR3 passes the data to the IP core by h2f bridge. The IP kernel can quickly transform the format of the image and analyze the image features. Finally, the image feature results of the tobacco leaf can be obtained.
8. The image data processing results of FPGA will be transmitted to the HPS terminal through h2f bridge. According to the results obtained, Linux uses the GPIO core to make the dial to rotate. At the same time, Linux opens the driver of VGA and displays the results on the LCD screen. The data flows through system block diagram are shown in Figure 2.

Figure 2 data flow chart

C、image processing

Using programmable logic units in FPGA, we have written a specific IP core through OpenCL to perform image format conversion, pattern recognition and machine learning. According to the extraction of image features, the establishment of standard library, the formulation of grade judgment rules, the theory of tobacco leaf classification solution, and the color characteristics of tobacco leaf drawing as the reference factors, the tobacco classification algorithm is designed. The design of these algorithms is accomplished through OpenCL, which has the advantages of strong real-time and fast processing speed. The algorithm flow is shown in Figure 3.

Figure 3. Algorithm flow

First, the tobacco leaf area is segmented according to the image obtained, and the content of tobacco leaves concerned is extracted. Secondly, the color components of the tobacco leaves are extracted according to the content of the divided tobacco leaf area, the color components of each grade sample are counted, the value of RGB channels of each grade tobacco leaves is recorded, and the color components are converted to Lab values so that the parameters are suitable for tobacco leaf classification. Finally, according to the obtained Lab value, the decision algorithm is divided into 3 steps: knowledge sample learning, new sample analysis, and new sample decision. The main idea of the decision algorithm is to learn the spatial distribution area of the color of this grade by learning a certain number of known tobacco samples, and to represent the color distribution area in a circular area. Then calculate the Lab value of the unknown grade tobacco leaf sample, and determine which grade it will fall into the circular distribution area of which grade.

To study the image characteristics of tobacco leaves, the first thing to do is to segment the target area from the image, that is, the content of tobacco leaves in the image. Tobacco has a special color feature. If a suitable belt is used as the background, there is a significant difference in color between the leaves and the background in the image. In order to study the implementation of segmentation, 30 RGB color sample images were collected using the white belt as background. The gray value of the RGB color channels in the 30 images of the tobacco leaf, the background part and the shadow part (the shadow formed by the lighting of the tobacco leaves) is measured and counted. The following conclusions can be drawn according to the statistical results:

1. The B (blue) channel value in tobacco leaf area is always the lowest, indicating that the blue light absorption effect of tobacco leaves is the most obvious, which is very different from the background.
9. The brightness of the tobacco R (red) channel is generally the highest, and the R channel curve of tobacco leaves shows a crossover with the brightness curve in the background. This indicates that low frequency visible reflectance of tobacco leaves is relatively adequate, which is consistent with the transcendental perception of tobacco color.
10. The value of the RGB three channel in the shadow area is low, because the problem of the shape of the tobacco leaves is unavoidable to form a shadow, the light intensity of the shadow is low, and the low gray level of the channels on the image is of course, in which the value of the shadow B channel is even lower than the value of the B channel in the tobacco leaf area.

This design uses two channel segmentation algorithm to filter white background and black shadow. The B channel value of tobacco leaves is relatively low while the R channel value is relatively high. The brightness of the white background is good in the image, so the values of R channel and B channel are very high, while the R channel and the B channel value in the shadow area are both low. A B channel is not enough to complete segmentation. Pixels with low B channel value and high R channel value are pixels of tobacco leafThe tobacco leaf pixels in the image which are satisfied with the above conditions are retained, and other pixels that are not conformed to the conditions are removed, and the remaining area is a complete image of the single leaf tobacco after the image segmentation is completed. RGB is the most commonly used color system, and the image captured by the general camera is also represented under this system.

Because RGB system expresses the color information of objects according to the chromatic mechanism of human vision, it is easy to express and is also very convenient in image processing. But the RGB color system has its lack of uniformity in the uniformity of the color space, RGB color space is a non-uniform chromaticity space, and non uniform color space is not too suitable for machine vision to judge, a uniform color model should be more suitable for color judgment in computer vision. The Lab system has uniformity, so it is applied to the color judgment of tobacco leaves. But the image acquisition system can not get the Lab value directly. In order to measure the color of the tobacco in the Lab space, the RGB value of the captured image should be converted to the Lab value first. The specific method is to convert the RGB value of the tobacco leaf area into XYZ value, and then get the Lab value through the XYZ value transformation. CIEl976Lab is a homogeneous color model (the Lab color model described in this article refers to the CIEl976Lab color model), and its uniformity overcomes the above color difference problem . The Lab model is a special color model developed by the international lighting Committee. Its characteristics are represented by its uniformity. In addition, the color gamut of the Lab color model is wide, and it contains not only the RGB model, all the color gamut in the CMYK model, but also the wider color range that shows the colors they can't perform. Lab color model can be transformed from RGB color model to [10]. The conversion formula is shown in formula 3.1.

         LAB(a、b)=K*(R/255、G/255、B/255)              （3.1）

The concrete algorithm is as follows, where the coefficient matrix normalization matrix is:

     |X|   |0.433910  0.376220  0.189860| |R/255|

     |Y| = |0.212649  0.715169  0.072182|*|G/255|

     |Z|   |0.017756  0.109478  0.872915| |B/255|

     L = 116*Y1/3      for Y>0.008856

     L = 903.3*Y       for Y<=0.008856

     a = 500*(f(X)-f(Y))

     b = 200*(f(Y)-f(Z))

among f(t)=t1/3              for t>0.008856

      f(t)=7.787*t+16/116    for t<=0.008856

After converting the RGB model of the collected tobacco to the Lab model, the decision of the tobacco leaf classification is carried out, including three steps: sample learning, feature value extraction and new sample decision.

1、Known sample learning

Sample learning is to determine the actual distribution of tobacco color in this level by studying the color of a series of known grades of tobacco leaf samples. [8] The grade of tobacco leaves is L, and L is taken as the 50 sample image to complete the learning process. In the Lab system, only AB involves color, so the design of the algorithm is a reference to the A and B components, which does not involve the brightness, so that the three-dimensional color space is compressed to the two-dimensional plane. The mean values of a and B components of each pixel in each image area were calculated by A and B respectively. The corresponding A and B values of the 30 sample images are marked by ordinal numbers respectively, and the mean values of the N charts are An and Bn.

The 50 images of the L level have 50 corresponding sequence points (A, B), which are distributed on a plane with the B component as the transverse ordinate. The center point of the distribution region is recorded as a point O, and two times the mean of the distance from each point to the point O at the sequence point (A and B) is dL, and the O is the center and the radius of the dL is a distribution circle. Use this circle to denote the color distribution of tobacco grade L in the ab plane The color distribution in the ab plane of the first grade tobacco in the ab plane corresponds to the circular area, which is recorded as Class1; the color distribution of the two grade tobacco in the ab plane corresponds to the circular area 2, which is recorded as class2, and the color distribution of the three grade tobacco in the ab plane corresponds to the circular area 3, which is recorded as class3, and the color distribution of the four grade tobacco in the ab plane corresponds to the round area 4. It's class4.

Figure 4. Image processing

2、Extraction of characteristic value of unknown grade tobacco

The new sample image of the unknown level is also segmented to get the effective area of the tobacco leaves, and the mean value A and B of the A and B components in the Lab color space of the new sample tobacco leaf area are calculated.

3、New sample verdict

If the characteristic values (A, B) fall into the circular distribution area and do not fall into the overlapping area, only one decision will be needed to get the result. If the value of the tobacco leaf characteristic value (A, B) falls into the circular distribution overlapping area, the determination rule is based on the distance of the two phase circle center of the tobacco leaf characteristic value (A, B). It is as follows: the grade of the tested tobacco leaves is located in the circular area where the center of the distance is closer, and other conditions are identified as the problem leaf.

At this point, the tobacco leaf classification algorithm has realized the task in the solution through four steps: tobacco leaf segmentation, standard sample learning, and feature extraction of samples to be measured, and color classification of tobacco leaves.

The core advantage of this system is the image acquisition based on SOC and the analysis and processing of the specific IP check image data by using OpenCL technology. The platform selects Altera DE10-nano development platform produced by the friend crystal technology, which uses a system level chip (SOC) integrating the large capacity FPGA and the Cortex-A9 architecture dual core ARM. In addition, two interconnected high-speed bridges and a lightweight control bridge were designed between FPGA and ARM. The two bridges of HPS to FPGA (H2F) and FPGA to HPS are the 128-bit bits of each bridge at high speed, and each bridge has a bandwidth of 128*2*200MHz=51200Mbps. The main features are as follows:

1、Integration of system

As an industrial equipment, the tobacco automatic classification system needs to be installed in a fixed place and a complete system, which needs to ensure considerable stability. If the traditional scheme is used, it is bound to need multiple modules to work together, so the structure and stability of the system are not as good as the single system.

2、The scalability of the system

In order to ensure that the system can be upgraded flexibly, we must adopt a good upgrade plan. The traditional scheme is implemented with ARM or DSP, and its scalability is not good. There are bottlenecks in communication between ARM+FPGA and DSP+FPGA.

3、system performance

The traditional scheme is limited to communication bandwidth or operation speed, and cannot realize high speed image analysis and processing, while SOC based tobacco leaf classification can achieve high speed image acquisition and image processing. At the same time, it precisely controls motor movement, so it has the potential to realize automatic classification of high speed tobacco leaves.

In addition, the system uses the OpenCL technology to design a specific IP core to complete the functions of image format conversion, pattern recognition and machine learning. It has the advantages of strong real-time and fast processing speed.

China is the world's largest tobacco producing country. In 2016, the tobacco industry obtained 10795 billion yuan for industrial and commercial tax, in which 10006 billion yuan had been paid to state. In china, about 16 million mu of tobacco is grown in the whole country, and the total output is about 41.5% of the world's total output. Thus, tobacco becomes an important cash crop for our country, and plays an important role in Chinese agricultural economy.

In recent years, with the improvement of people's living standards, more and more people are aware of the importance of good tobacco quality to health. Based on this concept, people desire to improve tobacco quality. Because the only way to improve the quality tobacco products is to improve the quality of tobacco, thus, the work of tobacco grading is getting more and more important, and the classification of tobacco leaves has become the basic work for tobacco industry. However, at present the classification of tobacco leaves in China is mainly done by manual which refers to grading tobacco leaves by workers’ subjective feelings of the visual, tactile, smell and so on. This method has the shortcomings of low accuracy, low efficiency and great subjective factors. In the field of tobacco automation classification, most of the foreign and aboard research results are still restrained in the theoretical or laboratory stage, and there is a big gap with the goal of marketization, industrialization and commercialization. Therefore, this paper proposes a novel tobacco automatic grading system with Intel FPGA.

In this paper, an automatic tobacco leaf recognition and grading system is designed and made. The system is composed of six functional modules, such as DE10 - Nano development board suite, high precision pump, two-dimensional slide rail, conveyor belt, D5M camera kit, indexing disk and so on。The system can carry tobacco leaves on the conveyor belt after image collection, pattern recognition and classification, and use a sucker to automatically load different grades of tobacco into different containers, so as to realize the classification. The whole system has the advantages of high automation, easy to use, safe and hygienic, and low cost. The design realizes a miniaturization and high efficiency tobacco grading system, and solves the current situation of tobacco grading mainly depending on manual completion.

The system function mainly contains five parts which can be described as follows:

1. Image acquisition and display: using D5M camera to collect images of tobacco leaves.
1. Mechanical arm automatically pick up tobacco leaves: mechanical arm and high precision pump are used to grab tobacco leaves to achieve automatic picking of tobacco leaves.
2. Automatic classification of indexing dial: DE10-nano determines the grade of tobacco leaves, and uses Linux to turn GPIO and control indexing dial.
3. LAB algorithm classification: using LAB algorithm to determine Tobacco Grades.
4. Remote display and control: the system can be controlled remotely by mobile phone APP or computers.

We design and make an automatic tobacco leaf recognition and grading system. The system can automatically load different grades of tobacco into different containers after image collection, pattern recognition and classifications. This system makes full use of the advantages of high-speed parallel computing between FPGA and ARM in DE10-nano, and has the performances as follows.

1. Fast picture procession: Because the image processing is mainly by OpenCL of FPGA hardware, including format conversion, pattern recognition and machine learning, the picture data processing is very fast. Real time test shows that one picture of tobacco is less than one tenth of a second.
2. Fast tobacco grading: The proposed system can grade at list ten tobacco leaves per second. This is because the system image processing is very fast, the mechanical part can respond quickly and keep pace with the signal processing.
3. Low cost of system: The system is mainly composed of six parts, the Cyclone V FPGA, a pump, a slide rail, a conveyor belt, a camera and an indexing disk. The overall cost of this system is less than 2000 yuan RMB, so the cost is very low.

In the whole system, DE10-Nano is the core of control. It controls the high precision pump and the two-dimensional slide rail to capture and move the tobacco and analyze the pattern recognition and depth study of the data collected by the camera. First, a high-precision pump is used to grasp the tobacco leaves through suction, and then the two-dimensional slide rail controls the suction disc of the high-precision pump for displacement, and finally is placed on the conveyor belt. After that, the tobacco leaves are transmitted on the conveyor belt. In the course of transmission, DE10-Nano uses the ARM hard core that runs Linux to drive the USB camera.

The image data of the tobacco leaves are collected to the specific physical address of the DDR3, and then the image data is transmitted to the IP kernel of the FPGA image processing through the h2f bus. The IP kernel is a custom IP core and can complete the image format conversion. With the functions of pattern recognition and machine learning, the design of these algorithms is accomplished by OpenCL, which has the advantages of real-time and fast processing speed. Then, the image processing results of OpenCL output are transmitted to DDR3 through the f2h bus, and ARM in SOC reads the data, and then the C program is used to further analyze the data to achieve the feature extraction of the tobacco and get the classification results. Finally, according to the characteristics of the tobacco leaf and the classification information, the ARM commands the motor of the conveyor belt and the dividing plate to complete the appropriate action, and put the different grades of tobacco in different containers to realize the automatic classification of the tobacco. Its workflow is shown in Figure 5.

Figure 5. Working flow

The functions of the system can be divided into the following steps:

Step 1

The high precision pump is used to absorb tobacco leaves by suction. The two dimensional slide rail controls the movement of tobacco leaves by controlling the sucker of the high-precision pump. In the process of grabbing tobacco leaves, two dimensional slide rail and high precision pump are used to grab and move tobacco leaves onto the conveyor belt. When the first piece of tobacco is transported on the conveyor belt, the two-dimensional slide rail immediately controls the high precision pump to grab the displacement of the next leaf.

Step 2

The tobacco leaves are placed on the conveyor belt for transmission. When the images are detected in the middle of the conveyor belt, the D5M camera collects the image of the tobacco leaf and passes the image data to the DE10-nano development board to analyze the data.

Step 3

After receiving the receipt, the DE10-nano development board transmits the data to FPGA, and then stores it to a specific IP core written in OpenCL. Then the IP kernel analyzes the data, and then passes the result through the h2f high-speed bridge to the ARM end, and the Linux is controlled according to the results obtained by the GPIO check disk to carry out the corresponding control, so that the tobacco can eventually be transferred to the corresponding grade disks for the final automatic classification of the tobacco leaves.