Integrating advances in artificial intelligence has a long tradition in development of cockpit assistance system. Recently, Large Language Models (LLMs)

• The use of artificial intelligence (AI) in the flight deck: Enhancing human-AI teamwork in aviation: https://www.sciencedirect.com/science/article/pii/S2941198X25000430
• AviationGPT: A Large Language Model for the Aviation Domain: https://arxiv.org/pdf/2311.17686
• LeRAAT: LLM-Enabled Real-Time Aviation Advisory Tool: https://arxiv.org/pdf/2503.16477

Isaac Sim coordinate conversion

This workspace consumes Isaac Sim terrain-local x/y/z values in the renderer, but drone GPS arrives over MAVLink and must be translated back into that same local frame in the backend.

The converter logic is derived from the same Jsonnet mission config used by the simulator:

• Source config: /home/qnc/Desktop/pip_isaacsim/env_isaacsim/lib/python3.11/site-packages/isaacsim/extsUser/auspex.platform_interface/auspex_platform_interface_python/config/config.jsonnet
• Mission manifest endpoint: GET /api/mission-config
• Backend config loader: server/mission-config.ts
• Shared converter: shared/isaacsim_coordinates.ts
• Frontend mission wrapper: src/isaacsim_mission.ts
• Desktop shell: electron/main.ts

What Isaac Sim actually does

There are two related coordinate paths in the platform code:

1. Jsonnet helper path in config.jsonnet

   • gps_to_local() and offset_to_gps() are used while building config data.
   • In the current config, terrain_config.lat/lon are the AOI center.

2. Runtime spawn path in spawner.py and core/geo_ref.py

   • SpawnContext.resolve_position() calls TerrainGeoConverter.gps_to_terrain().
   • TerrainGeoConverter applies:
     • GPS origin from terrain_config.lat/lon/origin_alt
     • optional UTM conversion when use_utm = true
     • yaw rotation via yaw_deg
     • a terrain XY offset passed in from spawner.py

The app mirrors the runtime spawn path for simulator-published local poses, but drones need one extra distinction:

• the backend converts drone MAVLink GPS into terrain-local positions before broadcasting /ws
• the renderer converts dropped GPS targets back into terrain-local coordinates for ROS-targeted ground actors

Drone GPS mismatch: Isaac terrain frame vs PX4 world frame

There is an important split in the current Isaac Sim and Pegasus stack:

• SpawnContext and TerrainGeoConverter spawn actors in a terrain-local frame that uses the half-AOI XY offset from spawner.py
• Pegasus GPS sensors and PX4 MAVLink telemetry use the simulated world origin (0, 0, 0) as their geodetic anchor

That means PX4 GPS does not come back in the same geodetic frame that the terrain-local gps_to_terrain() helper uses for people/carters.

In practice:

• simulator-local people/carter poses use the terrain converter with the half-AOI offset
• drone MAVLink GPS must be translated with a zero-XY-offset converter that matches Pegasus world coordinates
• when the operator drops a drone target on the map, the backend first resolves the clicked map location into Isaac local coordinates, then reprojects that local point into PX4/Pegasus GPS before sending DO_REPOSITION

If this translation is skipped, drones get shifted by roughly half the AOI size twice, which is why they can appear hundreds of meters away from the expected map position.

Current Isaac Sim terrain convention

The current platform runtime always passes:

• terrain_xy_offset_m = (aoi_length_m / 2, aoi_width_m / 2, 0)

from spawner.py into TerrainGeoConverter.

That means published ROS poses behave like:

• (0, 0) = south-west terrain corner
• (aoi_length_m / 2, aoi_width_m / 2) = mission GPS anchor / AOI center

This now matches the current config.jsonnet comments directly, so the half-AOI offset is not a workaround for an old mismatch. It is the intended runtime convention.

Current mission assumptions

From the active Jsonnet mission:

• terrain_config.lat = 47.836262
• terrain_config.lon = 11.614310
• aoi_length_m = 1000
• aoi_width_m = 1000
• yaw_deg = 0
• use_utm = true

So the effective terrain center in local coordinates is:

• (500, 500)

and the browser converts local x/y back to GPS using the same:

• origin lat/lon
• half-AOI offset
• yaw rotation
• UTM preference

as the runtime Isaac Sim spawner.

Desktop runtime

This workspace now runs as an Electron desktop app instead of a browser-only Vite page.

• Electron starts the ROS backend with scripts/run-backend.sh.
• scripts/run-backend.sh builds and sources the local ROS workspace in ros2_ws/, starts the control adapter, and then runs server/index.ts.
• There is no rosbridge or roslib in the data path anymore.
• The renderer loads mission config from /api/mission-config and live telemetry from /ws.

Generated workspace outputs

The local ROS interface package is built into generated workspace folders under ros2_ws/:

• ros2_ws/build/
• ros2_ws/install/
• ros2_ws/log/

The app also writes runtime/build artifacts into:

• dist/
• log/

These directories are ignored in .gitignore because they are generated outputs, not source.

Control architecture

The app no longer sends backend-specific control commands directly.

• Renderer -> POST /api/agents/:agentName/target
• HTTP backend -> ROS action client/service client
• ROS control adapter -> concrete backend implementation

The generic ROS contract is defined in ros2_ws/src/tactical_map_interfaces:

• Action: /tactical_map/command_agent_target
• Service: /tactical_map/list_agent_command_capabilities

The action accepts mission-level commands, not transport-specific ones:

• coordinate_frame = isaacsim_local with x/y/z
• coordinate_frame = wgs84 with lat/lon/relative_alt_m

This keeps the Electron app and the HTTP backend independent from whether the real control path is PX4 ROS, Pegasus services, MAVSDK, or raw MAVLink.

Current concrete backends

Today, the ROS control adapter in server/control_adapter.ts translates the generic ROS command contract into the currently available backends:

• People and carters:
  • publish std_msgs/msg/Float32MultiArray to /<agent>/set_target
• Drones:
  • use the direct PX4 MAVLink guided controller in server/mavlink.ts

So the current implementation is still mixed under the hood, but that detail is now hidden behind one ROS action/service boundary.

Drone telemetry and status

The control adapter also owns the drone-side telemetry bridge so the HTTP backend does not open its own competing PX4 control link.

• Each drone still uses its direct PX4 MAVLink socket pair internally:
  • Drone_1: receive 14541, send 14581
  • Drone_2: receive 14542, send 14582
• server/mavlink.ts reads:
  • HEARTBEAT
  • SYS_STATUS
  • EXTENDED_SYS_STATE
  • STATUSTEXT
  • GLOBAL_POSITION_INT
• The control adapter converts drone GPS into Isaac Sim terrain-local coordinates with shared/isaacsim_coordinates.ts.
• It republishes drones into ROS as:
  • /<drone>/pose as geometry_msgs/msg/PoseStamped
  • /<drone>/status as tactical_map_interfaces/msg/DroneStatus
• server/index.ts now consumes those ROS topics just like any other telemetry source and forwards them to the renderer on /ws.

This means the renderer now sees one ROS-only command/telemetry contract even though the current drone implementation still talks MAVLink behind the adapter.

For drones specifically, server/control_adapter.ts now uses two coordinate converters:

• mission/map converter: Isaac terrain-local with the half-AOI offset
• PX4/Pegasus converter: same lat/lon/yaw/UTM settings, but zero XY offset

That split matches how Pegasus computes GPS from state.position and fixes the previous +500 m / +500 m style drift on the tactical map.

The current MAVLink guided implementation still sends the same PX4 guided commands QGroundControl uses:

• MAV_CMD_NAV_TAKEOFF
• MAV_CMD_COMPONENT_ARM_DISARM
• MAV_CMD_DO_REPOSITION

On first control-adapter connection, if a drone is still on the ground, the controller automatically arms and sends takeoff before the operator drags any new target.

• Default auto-takeoff height: 5 m
• Override with DRONE_AUTO_TAKEOFF_ALT_METERS
• Disable with DRONE_AUTO_TAKEOFF_ON_CONNECT=0

Startup commands

• npm run dev: starts Vite plus Electron for local development.
• npm run start: builds the renderer and launches the Electron app.
• npm run preview: serves the built renderer through the backend without Electron.
• npm run build:ros: builds the local ROS interface workspace manually.

npm run build:ros builds the tactical_map_interfaces package that defines the generic ROS action/service contract used by the app backend and control adapter.

Agent dragging

• Right-click Person_1 or Person_2 on the map to enter drag mode.
• Move the mouse to the new target location and left-click to drop.
• People and carters convert the dropped GPS point back into Isaac Sim local x/y/z coordinates and send frame=isaacsim_local to the backend.
• Drones prompt for a target height above ground and send frame=wgs84 with lat/lon/relativeAltMeters.
• The HTTP backend forwards those commands to /tactical_map/command_agent_target.
• The ROS control adapter decides how to execute them with the currently configured backend.
• The people controller is patched so externally published targets clear the existing patrol queue instead of being appended after it.
• A trail is shown once an agent has been drag-commanded and starts moving.