RoboCrop: Teaching robots how to pick tomatoes

In the agricultural sector, labor shortages are increasing the need for automated harvesting using robots. However, some fruits, like tomatoes, are tricky to harvest. Tomatoes typically bear fruit in clusters, requiring robots to pick the ripe ones while leaving the rest on the vine, demanding advanced decision-making and control capabilities.

To teach robots how to become tomato pickers, Osaka Metropolitan University Assistant Professor Takuya Fujinaga, Graduate School of Engineering, programmed them to evaluate the ease of harvesting for each tomato before attempting to pick it.

Fujinaga’s new model uses image recognition paired with statistical analysis to evaluate the optimal approach direction for each fruit. The system involves image processing/vision of the fruit, its stems, and whether it is concealed behind another part of the plant. These factors inform robot control decisions and help it choose the best approach.

The model represents a shift in focus from the traditional ‘detection/recognition’ model to what Fujinaga calls a ‘harvest‑ease estimation’. “This moves beyond simply asking ‘can a robot pick a tomato?’ to thinking about ‘how likely is a successful pick?’, which is more meaningful for real‑world farming,” he explained.

When tested, Fujinaga’s new model demonstrated an 81% success rate, far above predictions. Notably, about a quarter of the successes were tomatoes that were successfully harvested from the right or left side that had previously failed to be harvested by a front approach. This suggested that the robot changed its approach direction when it initially struggled to pick the fruit.

Ultimately, Fujinaga’s research highlights the nuance involved in fruit-picking for robots with factors including fruit clustering, stem geometry, background leaves, and occlusion all being important. “This research establishes ‘ease of harvesting’ as a quantitatively evaluable metric, bringing us one step closer to the realization of agricultural robots that can make informed decisions and act intelligently,” he said.

Fujinaga sees a future where robots will be able to independently determine whether crops are ready for harvest. “This is expected to usher in a new form of agriculture where robots and humans collaborate,” he explained. “Robots will automatically harvest tomatoes that are easy to pick, while humans will handle the more challenging fruits.”

The findings were published in Smart Agricultural Technology.

###

About OMU 

Established in Osaka as one of the largest public universities in Japan, Osaka Metropolitan University is committed to shaping the future of society through “Convergence of Knowledge” and the promotion of world-class research. For more research news, visit https://www.omu.ac.jp/en/ and follow us on social media: X, Facebook, Instagram, LinkedIn.