本論文旨在建立一個自動識別蔬菜和水果種類並自動計價的無人電子秤系統。本系統採用深度學習中目標檢測(object detection neural network)與圖像分類(image classification neural network)兩種網路相結合的方式來達成目標。當顧客把欲購買之蔬果，無論是否放置於塑膠袋中，置於電子秤上，則該電子秤可自動快速準確地辨識出蔬果類別。根據類別結果對應單價自動計算總價並將單價、重量、總價等資訊顯示在本系統開發之結帳網站中。顧客可使用結帳網站之列印明細功能列印賬單。如此免去顧客在賣場等待過磅時間且節省賣場人工成本，同時達到省時、省錢、無人接觸的目的。
本系統採用Raspberry Pi 4為移動設備端並利用Evdev監測電子秤的重量變化。一個Raspberry Pi 4可以支持多個電子秤同時使用。當重量不為0kg且重量保持不變時，移動設備端控制相機拍照並傳輸圖像與重量到主機辨識端。主機辨識端採用桌上型電腦並利用深度學習Efficient Det的目標檢測網路濾除複雜的背景與塑膠袋之影響，首先取得蔬果位置資訊並截取出多張乾淨的蔬果圖。再使用Efficient Net，ResNet，Mobile Net和Denese Net四個深度學習圖像分類網路分別對多張蔬果圖進行辨識。系統會以各個網路為單位對多張蔬果圖的辨識結果做第一次多數投票法，選取出現次數最多的結果作為該網路的辨識結果。以各個網路在測試資料集的準確率高低對其進行加權多數投票法，得到最終蔬果的結果。為提升辨識速度，本系統採取了平行運算與多線程(multi-threading)的方式，改善因投票機制導致運算時間成本增加的問題。此外為提高辨識準確率，採取Focal Loss Function和SAM(Sharpness-Aware Minimization)提升網路的性能。Focal Loss Function用來解決訓練資料量不平衡，SAM用來提升模型的泛化性。在包含多元變化情況的測試資料集中，本系統總共可辨識40種蔬果，總辨識準確率達96.9%。

The main purpose of this thesis is to develop a self-service electronic weighing system for recognizing and pricing fruits and vegetables automatically. The object detection neural network and image classification neural network with deep learning are combined to build the system in order to achieve the desired goal. When customers put the fruits and vegetables, no matter inside plastic bags or not, on the electronic scale, their categories can be recognized quickly and accurately as well. The unit price, weight and the total price of fruits or vegetables will be shown on the designed screen and the total price will be calculated. Then customers can print the bill. In this way, the electronic scale can reduce the customer’s waiting time for weighting and save the labor cost for the store, so that the purpose of saving time, saving money and self-service can be reached simultaneously.
This system uses Raspberry Pi 4 as a mobile device to monitor the weight change of the electronic scale. A Raspberry Pi 4 can support multiple electronic scales. The device takes a photo and sends it to the computer when the weight is fixed and not zero. Firstly, the computer uses an object detection neural network Efficient Det to filter out background and plastic bags based on the location information and crop several clean images. Then, we use four neural network models Efficient Net, ResNet, Mobile Net and Denese Net, respectively, to recognize the category of the fruits and vegetables. At last, we make the majority voting for the results from four models to get the final category result. Furthermore, the system adopts parallel and multi-threading to speed up the recognition process. On the other hand, to improve the recognition accuracy, the Focal Loss Function and SAM (Sharpness-Aware Minimization) are used to improve the system performance. The Focal Loss Function is used to solve the imbalance of the training data set and SAM is used to improve the robustness of the system. Based on performing lots of experiments, the proposed system can recognize 40 kinds of fruits and vegetables with 96.9% recognition accuracy.

[1] C. Huang, Y. Wang, X. Li, L. Ren, J. Zhao, Y. Hu, et al, “Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China,” Lancet, 2020, vol. 395, no. 10223, pp. 497–506.
[2] "COVID-19-Taiwan,"[Online]. Available: https://sites.google.com/cdc.gov.tw/2019ncov/taiwan. [Accessed: June, 2021].
[3] "農產品批發市場交易行情站," [Online]. Available: https://amis.afa.gov.tw/m_fruit/FruitChartMarketTransVolumeScale.aspx. [Accessed: June, 2021].
[4] G. Zeng, "Fruit and vegetables classification system using image saliency and convolutional neural network," in Proceedings of 2017 IEEE 3rd Information Technology and Mechatronics Engineering Conference (ITOEC), 3-5 Oct.,2017: IEEE, pp. 613-617.
[5] S. Jana and R. Parekh, "Shape-based fruit recognition and classification," in Proceedings of International Conference on Computational Intelligence, Communications, and Business Analytics, 26 September 2017: Springer, pp. 184-196.
[6] H. Mureşan and M. Oltean, "Fruit recognition from images using deep learning," Acta Universitatis Sapientiae, Informatica,, 2018, vol. 10, no. 1, pp. 26-42, 4 Jan..
[7] "Fruit and Vegetable Detection and Feature Extraction Using Instance Segmentation-Part-1,"[Online].Available: https://medium.com/codalyze/fruit-and-vegetable-detection-and-feature-extraction-using-instance-segmentation-part-1-178cc73122cb. [Accessed: June, 2021].
[8] "水果識別-測試, " [Online]. Available: https://zhuanlan.zhihu.com/p/49591788. [Accessed: June, 2021].
[9] 邹谜，伍世虔與王欣, "一種用於機器人水果採摘的快速識別方法," 碩士, 冶金裝備及其控制, 武漢科技大學, 2019.
[10] 廉小亲, 成开元與安飒, 吴叶兰, 关文洋, "基於深度學習和遷移學習的水果圖像分類," 計算機與信息工程學院, 北京工商大學, 食品安全大數據技術, 2019.
[11] I. B. Mustaffa and S. F. B. M. Khairul, "Identification of fruit size and maturity through fruit images using OpenCV-Python and Rasberry Pi," 2017 International Conference on Robotics, Automation and Sciences (ICORAS), Melaka, Malaysia, 2017, pp.1-3.
[12] E. Rachmawati, I. Supriana and M. L. Khodra, "Toward a new approach in fruit recognition using hybrid RGBD features and fruit hierarchy property," 2017 4th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), Yogyakarta, Indonesia, 2017, pp. 1-6.
[13] Y. He, F. Pan, B. Wang, Z. Teng and J. Wu, "Transfer Learning Based Fruits Image Segmentation for Fruit-Picking Robots," 2020 IEEE 3rd International Conference on Computer and Communication Engineering Technology(CCET), Beijing, China, 2020, pp.71-75.
[14] J. Redmon and A. Farhadi, "Yolov3: An incremental improvement," arXiv preprint arXiv:1804.02767, 2018.
[15] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, "YOLOv4: Optimal Speed and Accuracy of Object Detection," arXiv preprint arXiv:2004.10934, 2020.
[16] S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, 6 Jan.,2016, vol. 39, no. 6, pp. 1137-1149.
[17] M. Tan, R. Pang and Q. V. Le, " EfficientDet: Scalable and Efficient Object Detection," arXiv preprint arXiv:1911.09070, 2019.
[18] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 10 Dec., 2015, pp. 770-778.
[19] A. G, Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications, " arXiv preprint arXiv:1704.04861, 2017.
[20] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, "Densely Connected Convolutional Networks," arXiv preprint arXiv:1608.06993, 2018.
[21] S. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. H. Torr, "Res2net: A new multi-scale backbone architecture," In Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, 1 Sep.,2019.
[22] M. Tan and Q. V. Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," arXiv preprint arXiv:1905.11946, 2019.
[23] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, "Sharpness-Aware Minimization for Efficiently Improving Generalization," arXiv preprint arXiv:2010.01412, 2020.
[24] T. Y. Lin, P. Goyal, R. Girshick, K. He, & P. Dollár, "Focal loss for dense object detection," In Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980-2988.
[25] "Woolworths‘s mind-blowing artificial intelligence speeding up check-outs," [Online]. Available: https://www.news.com.au/lifestyle/food/eat/woolworths-hidden-camera-at-checkouts-makes-shopping-faster/news-story/62b2f85fc468b59bc75a326708a96aea. [Accessed: June, 2021].
[26] "Intelligent scales provide comfortable self-service experience," [Online]. Available: https://www.digisystem.com/solutions/SOL00065/. [Accessed: June, 2021].
[27] "python 實時檢測鍵盤輸入函數," [Online]. Available: https://blog.csdn.net/qq_40930675/article/details/84667762. [Accessed: June, 2021].
[28] "Linux中使用evdev捕獲鍵盤(遙控器)輸入," [Online]. Available: https://blog.csdn.net/ManWZD/article/details/103376521. [Accessed: June, 2021].
[29] "鈺恆JWM-30K 外貿出口電子秤110V寬電壓," [Online]. Available: https://world.taobao.com/item/595919691836.htm. [Accessed: June, 2021].
[30] "Logitech C270 HD 網絡攝影機," [Online]. Available: https://www.logitech.com/zh-tw/products/webcams/c270-hd-webcam.960-000626.html. [Accessed: June, 2021].
[31] "GP3120 203DPI 熱敏桌面USB 接口條形碼物流單標籤打印機," [Online]. Available: https://chinese.alibaba.com/product-detail/gp3120-203dpi-thermal-desktop-usb-interface-barcode-logistics-single-label-printer-62501757650.html. [Accessed: June, 2021].
[32] "Raspberry Pi 4:處理器大升級，性能直逼入門電腦," [Online]. Available: https://www.techapple.com/archives/30761. [Accessed: June, 2021].
[33] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature Pyramid Networks for Object Detection," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936-944.
[34] S. Liu, L. Qi, H. F. Qin, J. P. Shi, and J. Y. Jia, "Path Aggregation Network for Instance Segmentation," arXiv preprint arXiv:1803.01534, 2018.
[35] "LabelImg," [Online]. Available: https://github.com/tzutalin/labelImg. [Accessed: June, 2021].
[36] "Data_Augmentation_Zoo_for_Object_Detection," [Online]. Available: https://github.com/zzlpointcloud/Data_Augmentation_Zoo_for_Object_Detection. [Accessed: June, 2021].
[37] B. Zoph, and Q. V. Le, "Neural Architecture Search with Reinforcement Learning," arXiv preprint arXiv:1611.01578, 2017.
[38] F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions," arXiv preprint arXiv:1610 02357, 2017.
[39] J. Hu, L. Shen, S. Albanie, G. Sun, and E. H. Wu, "Squeeze-and-Excitation Networks," arXiv preprint arXiv:1610.02357, 2019.
[40] H. Chockler, E. Farchi, B. Godlin and S. Novikov, "Cross-Entropy Based Testing," Formal Methods in Computer Aided Design (FMCAD'07), 2017, pp. 101-108.