Voice-controlled autonomous navigation robot with LLM intent understanding. BSAI301 course project (Team 2).
"Go to the front desk" → ASR → LLM intent parsing → Nav2 autonomous navigation → TTS "Arrived at front desk"
┌─────────────────────────────┐ ┌──────────────────────────────────┐
│ PC (Windows 11) │ │ Raspberry Pi 5 (Debian 12) │
│ │ │ Docker: MentorPi (ROS2 Humble) │
│ agent_gui.py ◄──Tkinter──┤ │ │
│ voice_nav.py ◄──Google──┤ │ ┌─ Hardware ──────────────────┐ │
│ keyboard_control.py │ │ │ MS200 LiDAR (360°) │ │
│ ~110 diagnostic scripts │ │ │ ascamera depth camera │ │
│ │ │ │ │ WonderEchoPro mic array │ │
│ │ rosbridge (9090) │ │ │ STM32 motor controller │ │
│ │ SSH (paramiko) │ │ └────────────────────────────┘ │
│ ├───────────────────┼────►│ │
│ │ │ │ ┌─ Perception ───────────────┐ │
│ │ │ │ │ EKF + IMU odometry │ │
│ │ │ │ │ RTAB-VSLAM (3D mapping) │ │
│ │ │ │ │ scan_to_scan_filter_chain │ │
│ │ │ │ └────────────────────────────┘ │
│ │ │ │ │
│ │ │ │ ┌─ Navigation (Nav2) ─────────┐ │
│ │ │ │ │ AMCL localization │ │
│ │ │ │ │ ThetaStar / Navfn planner │ │
│ │ │ │ │ TEB local controller │ │
│ │ │ │ │ BT navigator + recovery │ │
│ │ │ │ │ Velocity smoother │ │
│ │ │ │ └──────────────────────────────┘ │
│ │ │ │ │
│ │ │ │ ┌─ AI Agent ──────────────────┐ │
│ │ │ │ │ LLM: Llama-3.1-8B (OpenRouter)│
│ │ │ │ │ VLLM: Qwen2.5-VL-72B │ │
│ │ │ │ │ ASR: Whisper-1 │ │
│ │ │ │ │ TTS: tts-1 (onyx voice) │ │
│ │ │ │ └──────────────────────────────┘ │
└─────────────────────────────┘ └──────────────────────────────────┘
User speaks → Whisper ASR → text
→ LLM intent parsing → {"action": "move(front_desk)"}
→ coordinate lookup (locations.yaml) → NavigateToPose Action
→ AMCL localization → global path (ThetaStar) → local plan (TEB)
→ cmd_vel → STM32 motors → robot moves
→ arrival detection → TTS "Arrived at front desk"
| Component | Model |
|---|---|
| Computer | Raspberry Pi 5, 8GB |
| Chassis | LanderPi Mecanum (3-wheel omni-directional) |
| LiDAR | MS200 (LD19), 360°, 10Hz |
| Depth camera | Angstrong ascamera |
| Microphone | WonderEchoPro (xf_mic) |
| Motor controller | STM32 |
cp config_template.py config_final.py
# Edit config_final.py — add your OpenRouter API keypython agent_gui.pyThe GUI lets you:
- Launch the AI agent on the robot via SSH
- Select map (map_01 / map_02)
- Send text commands or use PC microphone
- See LLM responses and navigation status
python scripts/voice_nav.pySpeak location names: 衣柜, 大门, 厨房, 客厅, 卧室, 充电桩.
python scripts/keyboard_control.pyWASD to drive, Q to quit.
LanderPi/
├── agent_gui.py PC-side Tkinter GUI (rosbridge + SSH)
├── speech_full_stub.py LLM/TTS/ASR adapter (runs on robot)
├── speech_stub.py Minimal stub variant
├── awake_stub.py Wake-word detection stub
├── config_template.py Configuration template (API keys, models)
│
├── config/
│ ├── localization.launch.py.fixed AMCL + map_server launcher
│ ├── navigation_base.launch.py.fixed Nav2 stack launcher
│ └── nav2_full.yaml AMCL, costmaps, planner, controller params
│
├── docs/
│ ├── ROBOT_NAVIGATION_README.md Full project documentation
│ ├── NAVIGATION_PLAN.md Development plan
│ ├── FEATURE_DETAILS.md Component deep-dive
│ └── TODO_NAVIGATION.md Task list & diagnosis notes
│
├── maps/
│ ├── map_01.yaml / map_01.pgm Map 1 (12.6m × 8.1m)
│ ├── map_02.yaml / map_02.pgm Map 2
│ ├── locations.yaml Named locations → coordinates
│ └── *.png Map visualizations
│
└── scripts/ (~110 utility scripts)
├── voice_nav.py PC voice navigation client
├── keyboard_control.py Manual WASD driving
├── start_all.py One-click deploy + launch
├── switch_to_nav.py Switch 3D mapping → 2D navigation
├── location_marker.py TF-based location marking (runs on robot)
├── location_navigator.py Location → NavigateToPose (runs on robot)
├── check_*.py Diagnostic scripts (15+)
├── fix_*.py Configuration fix scripts (10+)
├── test_*.py Navigation/motor test scripts (10+)
├── verify_*.py Verification scripts
└── diag_*.py / debug_*.py Deep diagnostic scripts
The scripts/ directory contains ~110 scripts organized by purpose:
| Category | Count | Purpose |
|---|---|---|
| Diagnostics | ~25 | SSH to robot, check ROS2 state, topics, nodes, TF, configs |
| TF Fixes | ~15 | Fix transform timestamp/chain issues blocking AMCL |
| Nav2 Fixes | ~10 | Fix planner, scan topics, costmaps, restart Nav2 |
| Restart/Launch | ~10 | Docker restart, clean launch, stop boot service |
| Navigation Tests | ~8 | Send goals, verify movement, test full pipeline |
| Motor Tests | ~12 | Direct motor control, cmd_vel pipeline diagnostics |
| Map Tools | ~12 | RTAB-Map DB export, map visualization, data extraction |
| Core Operations | ~8 | Voice nav, keyboard control, location marking, deploy |
- Mecanum drive — AMCL uses
OmniMotionModel, TEB usesholonomic_robot: true - Scan topic — LD19 publishes to
/scan, but Nav2 uses/scan_raw(after filter chain). Required remapping - FastDDS SHM disabled —
RMW_FASTRTPS_USE_SHM=0prevents ARM shared-memory issues - EKF odometry — fuses IMU + wheel odometry into
/odomTF - Location system —
location_marker.pysaves poses via TF2,location_navigator.pyreads and publishesNavigateToPoseactions - Stub pattern — HiWonder SDK interfaces (
CircleMic,RealTimeASR) are replaced with custom stubs that use OpenAI/OpenRouter APIs
- TF timestamp mismatch — robot_state_publisher/JSP race caused empty
/tf_static, AMCL message filter dropped all scans. Fixed viatransform_toleranceincrease, EKF time offset, RSP publish frequency - scan_to_scan_filter_chain topic remapping — hardcoded topic names required ROS remapping
- AMCL lifecycle — manual lifecycle management needed due to Docker/launch timing
- Nav2 planner crash — ThetaStar config incompatibility → fallback to Navfn
- cmd_vel pipeline — velocity smoother sitting between controller and robot driver, required topic remapping
- FastDDS on ARM — shared memory mode caused node discovery failures
PC: Python 3, Tkinter, paramiko, websocket-client, speech_recognition, pyaudio, openai
Robot: Raspberry Pi 5, Docker, ROS2 Humble, Nav2, RTAB-Map, rosbridge
MIT