Automatic Audio Panning in Dome Theatre Environments

MSc Thesis · Media Technology · Linköping University

Infrared tracking and audio panning system that places a presenter's voice at their physical position in a 7.1 surround dome.

---

Overview

Pre-rendered dome content is mixed so audio matches the visuals on screen. In live presentations, a presenter's voice stays fixed on the centre channel no matter where they stand. When the audience sees the presenter to their left but hears their voice from the centre, audio and visuals conflict.

This project addresses the problem with two technologies:

1. IR Camera Tracking - An IR-sensitive camera detects a bright IR LED worn by the presenter. A homography (perspective transform calibrated on-site) maps pixel positions to room coordinates. An alpha-beta filter smooths the output and handles noise and occlusion. An optional moving-average filter adds further smoothing.

2. Hybrid VBAP/DBAP Spatial Panner - A SuperCollider engine receives tracker coordinates via OSC and pans the voice through the 7.1 array by blending Vector Base Amplitude Panning (VBAP) and Distance-Based Amplitude Panning (DBAP). A slider crossfades between the two algorithms in real time.

A user study with N=17 participants compared three conditions: VBAP, DBAP, and a static mono-to-all Standard baseline. VBAP was preferred on 8 out of 9 evaluation questions including directionality, spatial quality, engagement, and overall preference.

┌───────────┐     ┌──────────────┐     OSC /source/position/xyz     ┌────────────┐     ┌──────────────┐
│ IR Camera │ ──► │ ir_tracker.py │ ───────────────────────────────► │ HVDBAP.scd │ ──► │ 7.1 Speakers │
└───────────┘     └──────┬───────┘                                  └────────────┘     └──────────────┘
                         │
                         ▼ (optional --gui)
                  ┌────────────────┐
                  │ Virtual Panner │
                  │  (matplotlib)  │
                  └────────────────┘

Research Questions

1. Tracking Viability: How effectively can a single-camera infrared tracking system using homography mapping provide reliable, low-latency positional data in a darkened dome?

2. Perceptual Differences: What are the perceptual differences in source localization accuracy and spatial spread between VBAP and DBAP for dynamic speech panning in a 7.1 setup?

3. User Experience: To what extent does real-time, spatially synchronized speech panning enhance the audience's perceived immersion and presence compared to static reproduction?

Key Findings (N=17)

Metric	Result
First-choice preference	VBAP: 8, DBAP: 5, Standard: 3
Questions VBAP led on	8 of 9 (directionality, AV alignment, spatial quality, spatial spread, smoothness, engagement, subjective assessment, elevation)
Largest gap	Spatial spread: VBAP +0.64 above DBAP
Tracking accuracy	Homography + alpha-beta filter produced stable, low-latency tracking; cross-track error via line-walking tests showed the system followed the performer reliably
Edge-seat performance	VBAP preserved directional cues at all seating positions; DBAP lost localization near edge seats due to nearest-speaker precedence

Hardware Requirements

• IR-sensitive camera (e.g. modified webcam with IR-pass filter, or IP camera like Axis). The camera index is set as the CAMERA_INDEX constant at the top of ir_tracker.py.
• IR LED emitter worn by the presenter (shoulder height for calibration)
• Darkened room - the system uses intensity thresholding. Keep ambient IR sources (stage lights, projectors) to a minimum. Projector light in the tested dome was invisible in IR.
• 7.1 speaker system with an audio interface capable of 8-channel output (tested with RME MADIface USB)
• Computer running both Python (tracking) and SuperCollider (audio engine)

Software Requirements

Python

pip install -r requirements.txt

Dependencies: numpy, matplotlib, python-osc, opencv-python

SuperCollider

Download from supercollider.github.io. The system was tested with SuperCollider 3.x on Windows.

Setup

1. SuperCollider Audio Engine

Open HVDBAP.scd in the SuperCollider IDE and configure the top-of-file constants:

~audioDevice = " WASAPI : Speakers (RME MADIface USB)";  // your output device
~microphone = "WASAPI : Desktop Microphone (Wireless GO II RX)";  // your mic input
~soundSourcePath = "music.mp3";  // or use live mic (line 78: SoundIn.ar(0))
~roomRadiusMeters = 7.4;  // radius of your speaker dome

Select all and run (Ctrl+Enter). The server boots, loads the buffer, creates the hybrid panner SynthDef, and starts listening for OSC on port 57120.

2. Camera Calibration

Place the IR LED on the presenter's shoulder at performance height. Run:

python ir_tracker.py --calibrate

• Move the LED to a known room position
• Press SPACE to capture it
• Type the real-world room coordinates (metres, space-separated: x y)
• Repeat for at least 4 points (more improves accuracy)
• Press Q when done
• Enter the room radius when prompted

This generates calibration.json containing the homography matrix, room radius, and calibration point pairs.

3. ROI Setup (Optional but Recommended)

Draw an exclusion region to ignore hot spots outside the performance area:

python ir_tracker.py --setup-roi

Drag a rectangle around the performance area and press SPACE or ENTER. This masks out bright IR sources (exit signs, equipment LEDs) outside the tracked zone.

Usage

Basic Tracking (headless, OSC only)

python ir_tracker.py

The tracker opens the camera, detects the IR blob, applies the homography, runs the alpha-beta filter, and sends /source/position/xyz to SuperCollider over OSC.

With GUI (tracker + Virtual Panner)

python ir_tracker.py --gui

Launches the matplotlib Virtual Panner window alongside the camera feed. The red dot follows the tracked position. Sliders control DBAP blur, VBAP spread (side-pull), hybrid blend weight, and master gain in real time.

Command-Line Options

Flag	Description
--calibrate	Run interactive homography calibration
--setup-roi	Draw ROI rectangle on camera feed
--gui	Also launch VirtualPanner matplotlib GUI
--threshold 200	Blob detection intensity threshold (0–255, default 200)
--fps 45	Target camera framerate (default 45)
--osc-ip 127.0.0.1	OSC destination IP
--osc-port 57120	OSC destination port
--record tracking.csv	Save position + error data to CSV for jitter/accuracy tests
--debug-vision	Show binary blob mask window for threshold tuning
--limit-fps	Enable frame rate limiting (caps rendering to target FPS)

Keyboard Controls (during tracking)

Key	Action
Q	Quit
M	Toggle Moving Average filter on/off
B	Toggle camera preview window on/off
V	Toggle Virtual Panner GUI on/off (if --gui not used)
S	Open camera settings dialog (exposure, gain, etc.)
SPACE	Start a point jitter test (with --record)
L	Start/stop a line accuracy test (with --record)

Tracking Tests

When running with --record tracking.csv, two measurement modes are available:

• Point test (SPACE): Measures jitter - how much the tracked position deviates from a known static point over 10 seconds.
• Line test (L): Measures dynamic accuracy - cross-track error as you walk a straight taped line on the floor.

Results are saved as CSV with columns: timestamp, test type, target info, raw position, filtered position, raw error, filtered error.

Architecture

ir_tracker.py - IR Blob Tracker

Frame-by-frame pipeline:

1. Capture: Frame from camera at TARGET_FPS (45 fps default)
2. Blob detection: Threshold grayscale image, apply ROI mask, morphological close, extract largest contour centroid
3. Coordinate transform: Map pixel (u, v) → room (x, y) via homography (calibration.json), then normalize to unit circle by room radius
4. Alpha-beta filter: Smooth measurement-to-estimate convergence (alpha=0.3), coast through occlusion with velocity decay (velocity_decay=0.90), clamp to max speed (0.5 units/s)
5. Moving Average (optional): 2-frame sliding window for additional jitter reduction
6. OSC output: Send (x, y, 0) to SuperCollider on /source/position/xyz

The debug overlay on the camera feed shows: room boundary polygon, ROI rectangle, brightest pixel + intensity value, green blob centroid, orange ghost dot (filter prediction), tracking status, FPS counter, and test progress.

HVDBAP.scd - SuperCollider Hybrid Panner

Receives OSC and processes audio:

• Input: Live microphone (SoundIn.ar(0)) or buffer playback
• VBAP: Vector-based panning between speaker pairs. Spread control biases position toward side speakers by azimuth (side-pull).
• DBAP: Distance-based panning with configurable blur radius (dbapBlur).
• Hybrid blend: Linear crossfade between VBAP and DBAP gains. Controlled by hybridWeight (0 = full DBAP, 1 = full VBAP).
• Output routing: 8-channel hardware output mapped to the 7.1 dome layout with LFE channel low-pass filtered at 120 Hz.

OSC addresses:

• /source/position/xyz - target position (x, y, z)
• /source/control/dbapBlur - DBAP spatial blur (0.2-2.0)
• /source/control/vbapSpread - VBAP side-pull (0-1)
• /source/control/hybridWeight - VBAP/DBAP blend (0-1)
• /source/control/gain - Master gain (0-2)

Includes monitorHeadphones SynthDef for stereo headphone monitoring.

virtual_panner.py - Matplotlib Control GUI

Interactive 2D visualizer and remote control:

• Click & drag the red dot to manually position the source
• Sliders for radius, azimuth, elevation, hybrid weight, DBAP blur, VBAP spread, and master gain
• Sends OSC directly to SuperCollider for manual experimentation
• When launched by ir_tracker.py --gui, the red dot follows the tracked position

Files

File	Role
ir_tracker.py	Camera capture, blob detection, calibration, homography, alpha-beta filter, moving average filter, OSC output, ROI setup, main tracking loop, jitter/accuracy testing
HVDBAP.scd	SuperCollider engine: hybrid VBAP/DBAP SynthDef, OSC receivers, speaker layout configuration, headphone monitoring
virtual_panner.py	Matplotlib GUI for manual source positioning with real-time parameter sliders
calibration.json	Generated at runtime: homography matrix (3×3), room radius, pixel/room point pairs, ROI rectangle
requirements.txt	Python dependencies

The User Study

A within-subjects experiment with 17 participants across 5 seats. Each heard three 5-minute segments (VBAP, DBAP, Standard/mono-to-all) in randomized Latin-square order and rated them on a 7-point Likert scale across 9 questions: directionality, AV alignment, ventriloquism, spatial quality, smoothness, spatial spread, engagement, subjective preference, and elevation.

Statistical method: Friedman tests (non-parametric repeated-measures ANOVA) for overall differences and Wilcoxon signed-rank post-hoc tests with Bonferroni correction for pairwise comparisons.

License

This project is part of an MSc thesis at Linköping University. See LICENSE for details.

Citation

Kling, M. (2026). Automatic Audio Panning in Dome Theatre Environment: Evaluation of Real-time Immersive Source Panning Methods in Sparse Surround Dome Theatre. MSc Thesis, Linköping University.

URN: urn:nbn:se:liu:diva-225906 ISRN: LiU-ITN-TEK-A--26/032--SE OAI: oai:DiVA.org:liu-225906 DiVA, id: diva2:2081505

http://www.diva-portal.org/smash/record.jsf?pid=diva2:2081505