Abstract

Deploying autonomous robots in precision agriculture requires systems capable of navigating and interact with partially structured, dynamic environments. Large Language Models (LLMs) are emerging as a powerful interface enabling field operators to repurpose mobile robots through natural language (NL). However, they lack the ability to robustly operate in environments such as commercial orchards. This requires onboard systems to manage perceptive feedback as missions unfold. Furthermore, relying on predetermined large static waypoint databases limits scalability, can exceed effective LLM context windows, and decreases LLM performance. This paper presents an end-to-end distributed software architecture that bridges high-level LLM mission planning with feedback driven robot mission control. The system operates by translating NL requests into structured XML mission specifications, which are transferred to the robot and parsed into fault-tolerant Behavior Trees (BTs). To optimize and scale mission generation, we introduce an orchard management (OM) abstraction layer that interpolates navigation targets from user-defined GPS polygons, thus improving token efficiency. Built on the Robot Operation System (ROS2), the architecture decouples mission orchestration from hardware execution, leveraging Nav2 for mobility, MoveIt2 for manipulation, and dedicated nodes for 3D LiDAR and RGB-D perception. The BT execution engine dynamically manages runtime failures through integrated retry logic, ensuring a robust transition between waypoint navigation and perception-driven canopy approaches. Field experiments conducted in commercial pistachio and citrus orchards validate the architecture’s effectiveness, demonstrating that integrating deliberative LLM planning with reactive BT execution significantly improves the reliability, fault tolerance, and scalability of autonomous data collection systems operating in the wild.

https://ieeexplore.ieee.org/document/11703742

Figure 7. High level view of how the robot determines its relative
position to the target tree. Highlighted points are closest-point
candidates.