Key technologies for machine vision for picking robots: Review and benchmarking

In the rapid development of modern agriculture, picking robots, as a key technology to improve production efficiency, reduce labor costs, and improve operation quality are gradually becoming the focus of research. By simulating human perception, decision making and action processes, these intelligent systems are able to identify and pick ripe fruits from a variety of crops while avoiding damage to crops or missed picking. As the core component of the picking robot, the vision system undertakes the important task of perceiving and understanding the environment. In addition to accurately identifying target fruits in changing natural environments, large amounts of data must be processed in real time to enable efficient and precise farming operations.

However, due to the complexity of the agricultural environment, the vision system of picking robots faces many challenges. The changes in lighting conditions, the shading between fruits, and the differences in the appearance of fruits with different maturities all necessitate greater requirements for the recognition ability of the visual system. In addition, real-time requirements also challenge the processing speed and algorithm efficiency of the system. To address these challenges, researchers are constantly exploring new image processing techniques and computer vision algorithms, such as using machine learning and deep learning to improve recognition accuracy, using multisensor data fusion to enhance environmental awareness, and developing algorithms with greater real-time performance to meet the needs of dynamic picking.

In this paper published in Machine Intelligence Research, the research team of Prof. Wang Yaonan from Hunan University review the composition and function of the vision system of picking robots, discuss the current progress of image processing and object recognition technology, analyze the challenges of existing technology, and look forward to the future direction of technology development. Through this review, researchers aim to provide researchers and engineers in the field with a comprehensive analysis of the current state of research and future trends to promote the further development and application of picking robot technology.

Section 2 is about target detection and recognition. Recognizing the target fruit is not only necessary for the picking robot but also a crucial factor that affects its performance. When operating in an unstructured environment, target fruit recognition is influenced by various interference conditions. Therefore, automatic target fruit recognition has become one of the key problems in visual control. To address this issue, both domestic and international scholars have proposed numerous recognition methods, some of which have been applied to picking robots. Based on a summary of these methods, they can be categorized into the following three types: single feature analysis method, multifeature fusion analysis method, and machine learning-based method.

Section 3 is the introduction of multisensor information fusion. In the field of picking robots, vision sensors play a crucial role in capturing rich image information about files and fruits. However, because these sensors usually do not contain depth information, they are limited in obtaining accurate three-dimensional spatial data and are susceptible to interference from environmental factors such as changes in lighting. LiDAR, meanwhile, is known for its excellent robustness and accurate three-dimensional space measurement capabilities despite its shortcomings in identifying specific categories of targets. In the actual operation of agricultural robots, due to the variability of the operating environment, it is often difficult for a single sensor to provide consistently reliable detection results in all cases. To overcome this challenge, researchers are beginning to explore multisensor fusion techniques to enhance robots’ environmental awareness. The multisensor obstacle detection system consists of four key units: data acquisition, data preprocessing, information fusion and decision making. According to the different stages of fusion, the information fusion unit can be divided into three levels: the original data level, the feature level and the decision level. As technology continues to evolve and innovate, researchers believe that this technique will play an increasingly important role in the field of smart agriculture.

With the rapid advancement in the fields of computer vision and machine learning, particularly the exceptional performance of deep learning across various applications, there has been a significant enhancement in the environmental and mental perception capabilities of harvesting robots. Currently, intelligent agricultural machinery equipped with sophisticated navigation and environmental sensing systems is becoming increasingly common in agricultural production. However, despite these advancements, the development of environmental perception technology for harvesting robots still faces a number of challenges that require resolution. Section 4 is an in-depth discussion of four key aspects concerning the challenges and future directions of autonomous harvesting robots operational environmental perception technology: Environmental sensing performance during picking robot, multisource sensor data fusion, machine vision and mechanical fault tolerance and outlook.

The perception of the environment is essential for the functionality of harvesting robots. This discussion encompasses a range of sensor technologies and image processing strategies vital for the detection and positioning of horticultural produce, evaluating its precision, velocity, and resilience. The efficacy of these visual systems in identifying and locating fruits and vegetables depends on the specific crop type and is subject to a variety of influences, including light conditions and obstructions. The discourse synthesizes the current state of research on the environmental perception technologies utilized by developing robots, probing the challenges inherent in the field and the prospective trajectories of its evolution. Given the intricate and ever-shifting landscape of agricultural settings, which are inherently unstructured, these systems require heightened reliability and swift response times. There is a pressing need for significant research advancements in sensor capabilities for environmental purposes, the amalgamation of data from diverse sensors, and the enhancement of machine vision alongside mechanical fault tolerance. Progress in these technical and methodological domains is crucial for the ongoing innovation and improvement of environmental perception technologies in harvesting robots, thereby reinforcing the drive towards smart agriculture.

See the article:

Key Technologies for Machine Vision for Picking Robots: Review and Benchmarking

http://doi.org/10.1007/s11633-024-1517-1