AgriLoop - Learning Expert-Guided Crop Intervention through Human-in-the-Loop LLM Robotics

AgriLoop - Learning Expert-Guided Crop Intervention through Human-in-the-Loop LLM Robotics

BMFTR

September 1, 2026

August 31, 2029

2026-2029

Modern greenhouse production faces increasing challenges from skilled-labor shortages, complex crop-management requirements, and the need for sustainable cultivation. While agricultural robots can perform monitoring and intervention tasks, existing systems are typically task-specific and lack mechanisms for incorporating expert knowledge during operation.

AgriLoop addresses this gap by developing a human-in-the-loop framework that integrates 3D crop-state perception, multimodal artificial intelligence, and physically grounded robotic execution. The framework establishes editable, temporally updated 3D scene representations and semantic scene graphs that provide a shared basis for visual understanding, language grounding, and interaction. Through a multilingual interface, agricultural experts can inspect and correct scene representations, ask embodied questions about crop conditions, and provide natural-language instructions for robotic interventions.

These instructions are grounded in scene entities and translated into executable sequences of robot skills, such as targeted inspection, leaf manipulation, and selective harvesting, subject to feasibility and safety constraints. Conversely, uncertainty-aware decision mechanisms enable robots to request expert clarification or assistance when observations are ambiguous or proposed interventions require additional verification.

To support operation on resource-constrained robotic platforms, the project investigates lightweight language, vision-language, and vision-language-action models with hardware-aware inference and runtime optimization. Structured interaction data capturing scene states, expert feedback, robot actions, and observed outcomes will support learning preference-aware decision models and progressively more autonomous task execution under human oversight.

The German-Taiwanese collaboration combines Bonn's expertise in greenhouse robotics, 3D perception, and physical execution with Taiwan's expertise in multimodal reasoning, efficient AI models, and learning from human feedback. The project aims to deliver an integrated greenhouse demonstrator for evaluating grounded human-robot interaction, safe robotic intervention, and uncertainty-aware autonomous assistance under realistic operating conditions.