YeahDaddy

Sprite-based animated companion app with real-time lip sync.

What It Does

• Plays pre-rendered image sequences at configurable framerates (24/30/60/144 FPS)
• Real-time audio-driven lip sync using a lightweight ONNX phoneme recognition model
• Multiple character actions: idle, talking, dancing, laughing
• Local HTTP API endpoint to receive audio and trigger lip sync externally
• Target size: 200-300MB

Tech Stack

• Backend: Rust (Tauri v2 + axum HTTP server + tract ONNX inference)
• Frontend: TypeScript + Canvas/WebGL sprite renderer
• Lip Sync: Real-time phoneme-to-viseme mapping via small ONNX model (~5-10MB)
• Build: Cargo + npm + Tauri CLI

Getting Started

Prerequisites

• Rust toolchain (rustup)
• Node.js + npm
• Tauri CLI (cargo install tauri-cli)

Development

npm install
cargo tauri dev

Build

cargo tauri build

Project Structure

yeahdaddy/
├── src/                    # Frontend (TypeScript)
│   ├── renderer/           # Canvas/WebGL sprite rendering
│   ├── audio/              # Web Audio API playback
│   ├── animation/          # State machine, frame sequencing
│   ├── api/                # Client for local audio API
│   └── ui/                 # Settings, controls
├── src-tauri/              # Rust backend
│   ├── src/commands/       # Tauri IPC commands
│   ├── src/audio/          # Phoneme recognition (tract ONNX)
│   └── src/server/         # Local HTTP API (axum)
├── assets/                 # Spritesheets, audio files, ONNX model
└── package.json

API

Send audio to the companion for lip sync:

POST http://localhost:7890/api/audio
Content-Type: audio/wav

<raw audio bytes>

The character will lip sync to the audio in real-time.

License

Private