agentAI

A research-data platform for physical & embodied agents — upload any modality (images, video, GPS/IMU sensor logs), get an automatic diagnosis, then collect → review → train, all from one dashboard.

🔗 Live platform: https://lab-b660m-c.tailfa6424.ts.net — sign in with your MTSU Google account. Runs on the lab server; reachable from the lab network / Tailscale. GPU jobs queue on PSC Bridges-2.

Built at the MTSU Great Robotics Lab. Create a project for any research field, add data of any kind, then point AI agents at it to harvest, filter, label, and train — any number, any mix, or none. The dashboard runs on the lab server; GPU jobs queue on the PSC Bridges-2 cluster.

_{Every research domain is a project. Open one, or start a new one for any data type — images, video, sensor, …}

Jump to: 5-minute demo · Every modality · Autonomous agents · Run it locally · Research & benchmarks

---

🚀 5-minute demo — diagnose a robot's sensor logs

A hands-on walkthrough you can reproduce on the live platform. We take one patrol robot's raw sensor logs — a GPS track and an IMU (accelerometer + gyro) — and let the platform turn them into a diagnosis: how clean each signal is, what went wrong and exactly when, and which sensors caught the same physical event.

The payoff (Step 4③): the platform finds that at t ≈ 60 s the GPS and the IMU flagged trouble at the same instant — the fingerprint of a real pothole — while a bad GPS fix at 30 s and a dead IMU channel at 75 s each show up on one sensor alone. That cross-sensor agreement is something no single log can tell you.

Step 1 — Grab the demo data

Download docs/demo/patrol_multisensor.zip (24 KB): two CSVs — gps.csv (timestamp, lat, lon, speed_mps, heading_deg) and imu.csv (timestamp, ax, ay, az, gyro_z), 1,000 rows each at 10 Hz, sharing one clock. It's realistically-shaped driving data with three faults deliberately injected so every detector has something true to find.

Step 2 — Create a project

On the home page, click New Project, name it Patrol Multisensor, and tick Sensor as the data type — or just describe your goal in plain English and let the AI propose the setup. No config files, no code.

Step 3 — Upload the logs

Open the project and drop the .zip onto the upload box. The platform detects both CSVs as sensor data and registers the dataset. Set a goal — "Find cross-sensor events where GPS and IMU agree" — so the AI review knows what you actually care about.

Step 4 — Read the diagnosis

Open Analyze. The page is modality-aware; for sensor data it builds a three-layer diagnosis, top to bottom.

① The route — and how clean each signal is. The GPS track is drawn as its actual shape (colored by speed), with a red ✕ everywhere the position jumps implausibly. Below it, every signal gets a noise score: residual after smoothing as a % of its range, plus SNR. GPS lat/lon come back clean (0.5–0.8 %); the IMU axes read noisier because they carry a constant gravity component with little real variation — the page says so instead of crying "bad data".

② What went wrong — and exactly when. Four grounded detectors — robust rapid-change (median + MAD z-score), GPS teleports (physically implausible implied speed), sampling gaps, and stuck-sensor flatlines — each report a precise timestamp. Here: a 118 m GPS jump at 30 s, a stuck IMU az channel for 61 samples at 75 s, and a burst of activity at 60 s.

③ Where the sensors agree — the real events. This is the part a single sensor can't give you. Every sensor's events go onto one shared time axis; a red dashed line marks any instant where two or more sensors flagged at once. The 30 s GPS glitch and the 75 s IMU flatline each stand alone — likely sensor faults. But at 60.03 s the GPS jump and speed-drop line up with the IMU's vertical-acceleration spike: that agreement is the fingerprint of a real pothole, not a glitch.

Honest by design. Alignment uses the files' absolute timestamps (a true shared clock) when present, and says so on the card; otherwise it falls back to each file's own start and tells you it's assuming synchronized logging. A one-click AI review then narrates all of this in plain English and rates training-readiness — a small model runs locally on the server, a stronger cluster model is selectable per project.

---

🧩 One upload box, every modality

That sensor pipeline isn't a special case — it's the same upload box every project has. Drop a .zip, a folder, images, a raw .mp4, or CSV logs, and the platform auto-detects what the data is (by content, not file extension) and runs the analysis that fits it. Verified live with real uploads:

You upload	Auto-detected as	The analysis page shows
Field photos + YOLO labels	image	class distribution, boxed sample previews, near-duplicate check
GPS / IMU logs (CSV)	sensor	the 3-layer diagnosis above — route, noise, anomalies, cross-sensor timeline
Robot camera clip (raw .mp4)	video	duration / fps / frames / resolution + 8 auto-extracted frame previews

For image datasets you get class distribution, image-dimension stats, near-duplicates, boxed samples, and the same one-click AI review. No data.yaml? Click ✎ Edit class names and name each YOLO class right on the page — saving writes data.yaml and the real names appear immediately.

---

🤖 It's also an autonomous agent platform

Upload-and-analyze is the ground floor. A project can also run agents — any number, any mix, or none:

• Collector — harvests data by your queries · Filter — DINOv2 quality-scores it · Labeler — pushes to Roboflow for human labeling · Trainer — trains on the cluster GPU · Evaluator — runs model.val on a held-out split.
• Compounding rounds — each round is one collect → filter → label → train → evaluate pass, recorded with who ran it and when; evaluation metrics feed the next round's collection.
• Fill in a research field and the project auto-generates harvest queries and an accept-vocabulary from it — a new domain is config, not code. There's a built-in 2-minute guide at /guide.

---

⚡ Run it locally

cd weed_llm_benchmark              # the package lives here
pip install -r requirements-dev.txt   # light: boots the dashboard without torch
echo mypassword > ~/.dashpass      # auth fails closed without a configured password
DASH_USER=me uvicorn weed_optimizer_framework.tools.dashboard_server:app --port 8000
# → open http://localhost:8000  (log in: me / mypassword)

Running tests, CI details, and contributing conventions are in docs/DEVELOPMENT.md.

---

What's in this repo

Path	What it is
weed_llm_benchmark/	The live platform — dashboard, autonomous harvest → label → train pipeline, MongoDB, Roboflow sync. This is where active work happens.
multagent/	EMACF robotics agent framework (Brain / Perception / Targeting / Navigation) — the earlier embodied-robot direction, kept for reference.
docs/	Screenshots + platform roadmap.

Latest: v3.2.0 — sensor-analysis pipeline: signal noise/SNR, timestamped anomaly detection (GPS teleports, stuck sensors, sampling gaps), and cross-modal temporal alignment (finds the instants where multiple sensors flag the same physical event). See weed_llm_benchmark/CHANGELOG.md and RESEARCH_LOG.md.

---

Research & benchmarks

The platform grew out of two research threads. Full detail is collapsed below to keep this page usable — expand it for the architecture, the 19-model VLM benchmark, every phase, and the results tables.

📄 Click to expand — EMACF framework, the VLM-vs-YOLO benchmark, all phases & results, tech stack, papers

Origin

This repository implements EMACF (Embodied Multi-Agent Cognitive Framework) — a domain-agnostic agent architecture where an LLM serves as a robot's "brain" and specialized real-time agents serve as its "body". The same core infrastructure is designed to handle weed detection, robot navigation, human-robot interaction, and future applications. The dataset platform above is the weed-detection domain, generalized.

Repository structure (research view)

Directory	Description	Status
multagent/	EMACF core framework — event-driven multi-agent system extending MetaGPT for real-time robot control. BrainAgent (LLM), PerceptionAgent (YOLO), TargetingAgent (laser), NavigationAgent, cloud-edge comms, real-time dashboard.	Reference
weed_llm_benchmark/	Vision-LLM benchmark + dataset platform — evaluates 19 open-source vision LLMs against YOLO on CottonWeedDet12 (5,648 images, 12 species), and hosts the live harvest/label/train platform.	Active
robot_navigation/	Robot navigation (planned) — autonomous navigation sharing the same EMACF framework.	Planned

Architecture overview

┌──────────────────────────────────────────────────────────────────┐
│                    EMACF (multagent/)                             │
│                                                                  │
│  ┌──────────┐  ┌──────────────┐  ┌───────────┐  ┌────────────┐  │
│  │  Brain    │  │  Perception  │  │ Targeting  │  │ Navigation │  │
│  │  Agent    │  │  Agent       │  │ Agent      │  │ Agent      │  │
│  │  (LLM)   │  │  (YOLO)      │  │ (Laser)    │  │ (Movement) │  │
│  └────┬─────┘  └──────┬───────┘  └─────┬─────┘  └──────┬─────┘  │
│       │               │                │               │         │
│       └───────────────┴────────────────┴───────────────┘         │
│                           EventBus                               │
│                              │                                   │
│  ┌───────────────────────────┴────────────────────────────────┐  │
│  │                    Edge Bridge (WebSocket)                  │  │
│  └────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘
                               │
                    ┌──────────┴──────────┐
                    │   Edge Device       │
                    │   (LaserCar)        │
                    │   Camera + Laser    │
                    │   + Safety Monitor  │
                    └─────────────────────┘

Key design principles

1. Universal agent framework — domain logic lives in agent implementations, not the core. The same EmbodiedTeam, EventBus, and AgentRegistry handle any task.
2. Event-driven — agents react to events in real time, not turn-based. The BrainAgent (LLM) only intervenes on significant events.
3. Hot-pluggable agents — add/remove agents at runtime via AgentRegistry.
4. LLM-agnostic — switch between vLLM, Ollama, OpenAI, or any backend via config.
5. Cloud-edge separation — heavy compute (YOLO, LLM) on cloud GPU; edge device handles only hardware I/O with independent safety fallback.

Progress — EMACF framework (multagent/)

Component	Status	Description
Core Framework	Done	EmbodiedTeam, EventBus, AgentRegistry, EdgeBridge
BrainAgent	Done	LLM-based cognitive center with memory and self-optimization
PerceptionAgent	Done	YOLO11n detection + tracking + noise filtering + trajectory prediction
TargetingAgent	Done	Coordinate transform + firing control + laser patterns
NavigationAgent	Done	Mode management + vehicle commands
Dashboard	Done	Vue 3 real-time visualization (live feed, agent status, metrics)
Edge Client	Done	Camera streaming, command execution, safety monitor

Progress — Weed LLM benchmark (weed_llm_benchmark/)

Phase	Status	Description
Phase 0: Evaluation Module	Done	evaluate.py (mAP, precision, recall), datasets.py, format converters
Phase 1: YOLO Baseline	Done	YOLO11n fine-tuned on CottonWeedDet12 — mAP@0.5=0.929, P=0.930, R=0.850
Phase 2: Full LLM Benchmark	Done	15 models evaluated on CottonWeedDet12 (9 with mAP > 0)
Phase 3: YOLO+LLM Fusion	Done	Only OWLv2 filter improves YOLO (+0.018 F1); LLMs cannot rescue YOLO misses
Phase 3B: Cross-Species Generalization	Done	YOLO drops 27% on unseen species; Florence-2 precision exceeds YOLO; LLM augmentation +0.009 F1
Phase 3C: Anti-Forgetting Methods	Done	All simple methods fail; label quality (SAM+Depth) is the bottleneck
Phase 3D: SAM+Depth Enhanced Labeling	Done	SAM+Florence-2 caption: worse (-6.8% old, -11% new); caption classification too noisy
Phase 3E: Agent Optimizer	Done	First improvement! Florence+OWLv2 consensus: +0.016 F1 on unseen species, -0.020 forgetting
Phase 3F: Florence-2 Fine-tune	Done	Negative: fine-tuning degraded both old (-11%) and new species
Phase 4: HyperAgent Closed-Loop	Done	Qwen2.5-7B Brain: 3 rounds executed, system works but Brain needs stronger reasoning
Phase 4B: Weed Optimizer Framework	Done	14 files, 3,522 lines. Ollama function calling, job chain, plant.id API, HuggingFace model discovery.
Phase 4C: Clone + Train External Models	Done	YOLOv8s trained from COCO→CottonWeed: F1=0.888; DETR zero-shot: F1=0; YOLO11n baseline: F1=0.917
Phase 4D: plant.id Integration	Done	API key configured, local test OK (Status 201). Cluster needs pre-cache. 49 credits left.
Phase 4E: DeepSeek-R1 Brain	Done	7 action types (vs Qwen's 1). Autonomously searched HuggingFace + downloaded models.
Phase 4F: Extended Run (6h48m)	Done	7 rounds autonomous. Filter removed 16.3% label noise. Brain reasoning loop validated.
Phase 4G: Anti-Forgetting (LoRA + freeze + distill)	Done	Hybrid LoRA: 37 Conv2d layers, 38.15% params. Near-zero mAP forgetting.
Phase 4H: Gemma 4 Brain + Evaluator Fix	Done	Gemma 4 31B (MoE). Corrected evaluator (dual-conf). New mAP50: +9.7%, old mAP50: -0.6%.
Phase 5: Dataset platform (dashboard, provenance, multi-domain)	Active	The platform shown at the top of this page.
Paper writing	Planned	Figures, tables, manuscript

Benchmark results (CottonWeedDet12, 848 test images)

Model	Type	mAP@0.5	mAP@0.5:0.95	Precision	Recall	F1	Time
YOLO11n (fine-tuned)	Detector	0.929	0.865	0.930	0.850	0.888	—
Florence-2-base (0.23B)	VLM	0.434	0.392	0.789	0.519	0.626	558s
Florence-2-large (0.77B)	VLM	0.329	0.302	0.692	0.431	0.531	662s
InternVL2-8B	VLM	0.208	0.091	0.545	0.354	0.429	3838s
Qwen2.5-VL-3B	VLM	0.196	0.068	0.333	0.249	0.285	5898s
MiniCPM-V-4.5	VLM	0.192	0.043	0.407	0.340	0.371	6595s
OWLv2-large	Detector	0.184	0.117	0.194	0.943	0.322	2519s
Qwen2.5-VL-7B	VLM	0.176	0.059	0.334	0.214	0.261	6047s
InternVL2-2B	VLM	0.002	0.001	0.038	0.025	0.031	2094s
InternVL2.5-8B	VLM	0.000	0.000	0.016	0.001	0.001	6238s
Grounding-DINO-base	Detector	0.000	0.000	—	—	—	843s
Llama 3.2 Vision 11B	VLM	0.000	0.000	0.005	0.007	0.006	11370s
Moondream / Molmo / LLaVA	VLM	0.000	0.000	—	—	—	—

Technology stack

Component	Technology
Agent Framework	MetaGPT (extended for embodied AI)
Backend	FastAPI + asyncio + WebSocket
Object Detection	Ultralytics YOLO11n (fine-tuned on CottonWeedDet12)
LLM Integration	vLLM / Ollama / OpenAI-compatible APIs
Dashboard	Server-rendered FastAPI (platform) + Vue 3 + ECharts (robot demo)
Storage	MongoDB + append-only JSON registry + Roboflow (labeling surface)
Hardware	HeliosDAC (laser) + ESP32 (motor/laser control)
Compute Cluster	PSC Bridges-2 (V100 GPUs)

Papers

1. "Universal Embodied Multi-Agent Cognitive Framework for Agricultural Robotics" — EMACF architecture (targeting Scientific Reports)
2. "Can Vision LLMs Detect Weeds? A Benchmark of Open-Source Multimodal Models for Agricultural Object Detection" — VLM benchmark (targeting Computers and Electronics in Agriculture)

Per-project docs

• multagent/README.md — EMACF setup and usage
• weed_llm_benchmark/README.md — platform + benchmark framework documentation

---

License

Proprietary — research use only. © 2026 MTSU Great Robotics Lab, all rights reserved. See LICENSE.