Proposal: evaluate PersonaPlex for Maya's voice surface

[ArangoGutierrez]

Proposal: evaluate PersonaPlex for Maya's voice surface

A proposal for review, not a roadmap. Drafted from an outside read of the repo and of NVIDIA/personaplex on 2026-04-21.

---

What this document is

An evaluation of whether NVIDIA/personaplex is a good fit for Maya. Proposed as a new capability surface — voice — that doesn't exist in the repo today. Pushback on any part of this is welcome; I've read the repo from the outside and haven't run anything.

---

What PersonaPlex is

PersonaPlex is NVIDIA's full-duplex speech-to-speech conversational model. It's based on the Moshi architecture and its weights are a fine-tune of Helium, trained on synthetic and real conversations with persona control and voice conditioning. 7B parameters. Paper. Weights. Code is MIT; weights are NVIDIA Open Model license.

Two things make it different from the usual Whisper → LLM → TTS stack:

1. Full duplex. It listens and speaks on overlapping streams, so turn-taking, backchannels ("mhm"), and interruptions feel natural. A cascade stack can't do this without engineering acrobatics.
2. Persona + voice conditioning. A text prompt sets the role ("You work for Maya's Bar, your name is Maya. Information: ..."), and a short audio embedding sets the voice. The same model serves a bar-owner persona and a vendor-service persona by swapping prompts.

Why it's relevant to Maya

Maya's identity is already written the way PersonaPlex expects. Compare the README's customer-service prompt format:

You work for CitySan Services which is a waste management and your name is Ayelen Lucero.
Information: Verify customer name Omar Torres. Current schedule: every other week...

...with a five-minute translation of SOUL.md and AGENTS.md:

You work for Maya's Bar in San Francisco and your name is Maya. You run the bar.
You are warm but no-nonsense. Your sentences are short. You don't use corporate filler.
You never suggest loyalty programs, QR menus, scaling, or franchising. You flag
contractual or financial questions to a human. Information: the current cocktail
menu, current tap list, today's 86 list, and current vendor contacts.

The persona format is load-bearing for Maya in a way it isn't for most projects. Her documented identity is specific enough to fit the prompt shape natively. That's not a reason by itself, but it's a signal that this model's contract was designed for the thing Maya is trying to be.

---

Three places it could fit

1. In-bar spoken interaction

Customers or staff ask Maya things out loud. "What's on tap?" "Do we have anything like a Negroni but lower ABV?" "What's the pour spec on the house Old Fashioned?" Maya answers in her voice.

Full-duplex matters here. A behind-the-bar conversation is interruption-heavy. A cascade stack that waits for end-of-speech will feel like a kiosk.

Privacy is the live question. Regulars' conversations are on the no-export list in AGENTS.md. A microphone that's always open in a bar is a different threat surface than a terminal. This is why the privacy work in issue #1 Phase 3 has to land first, not alongside.

2. Phone line / vendor calls

Inbound: someone calls the bar; Maya answers, handles hours, bookings that fit the house rules, routes anything real to a human via flag-to-human. Outbound: Maya places a vendor order call, or follows up on a missed delivery.

This is the strongest use case on paper. The README's customer-service prompts read like specs for exactly this:

You work for AeroRentals Pro which is a drone rental company and your name is Tomaz Novak.
Information: PhoenixDrone X ($65/4 hours, $110/8 hours)...

Swap the business and the info block, keep the shape. That's Maya on a vendor call.

It also cleanly folds in the vendor-order, vendor-contact, and music-book skills from issue #1's Phase 6. Those skills currently assume email. A voice channel is a third option for the ones where a call is more natural (booking a musician, chasing a late delivery) and email is a better fit for the others (placing a standing order).

3. Staff voice notes

Bartender says "86 the mezcal, we're out" into a mic while hands are wet; Maya updates state. A pinned iPad in the well runs a small recorder app.

PersonaPlex is the wrong tool for this. Staff voice notes are monologue — speech-to-text, not a conversation. The right tool is whisper.cpp or MLX-Whisper, running locally on Apple Silicon. It's smaller, faster, cheaper, and better at transcription. Calling PersonaPlex for this would be using a full-duplex model for what a purpose-built transcriber does cleanly.

This use case is mentioned in the proposal because it's real and worth implementing — but it's a separate skill with a separate runtime, not a PersonaPlex feature.

---

Where PersonaPlex does not fit

To be explicit about the line, since it matters for the privacy story:

• No voice prints of regulars stored. Ever. Even "anonymized." A voice print is PII.
• No voice-based authentication or recognition. If Maya could identify a regular by voice, she'd be profiling them by presence, which is on the no-list in AGENTS.md in spirit if not in words.
• No customer voices sent to remote inference. Whether model inference happens locally (see hardware section) is a hard gate on this entire proposal, not a nice-to-have.
• No "Maya remembers our last conversation" feature based on audio. Her memory layer stays text, sanitized, and governed by Phase 3.

---

Hardware: why DGX Spark is the honest target

Deployment is the load-bearing question. Four options, ranked by fit for Maya's constraints:

Mac Mini only, local inference (including via MLX-Moshi). Tempting because it's already the deployment target, and community MLX ports of Moshi do exist and do run on M-series Macs. But three problems stack: (1) PersonaPlex's reference install is CUDA-first and the README calls out a Blackwell-specific PyTorch index — the MLX ports target the base Moshi model, not the PersonaPlex fine-tune, so replicating the persona/voice behavior on MLX is itself an engineering project; (2) full-duplex at the <200ms turn-taking that makes full-duplex worth the word is an open question on Apple Silicon, and a cascade stack at 800ms feels worse than an MLX-native stack at 300ms would; (3) the Mac Mini is already the orchestrator and telephony bridge in the deployed system — adding model inference to it makes every workload compete for the same memory. Plausible as a follow-on spike if DGX Spark doesn't pan out; not the first move.

Discrete RTX 50-series workstation. Works, proven by NVIDIA's own documentation. But it's a tower full of fans; nobody wants that behind a bar or in a back office where patrons can hear it. Eliminated on ergonomics, not technology.

Cloud-hosted inference (HF Inference Endpoint, NVIDIA NIM, Modal). Best setup story, highest inference quality, but audio leaves the building. This directly contradicts the privacy stance in issue #1 Phase 3. It's a non-starter for in-bar use. It's arguably acceptable for vendor calls — a vendor's voice is not Maya's to protect — but splitting the deployment creates a second set of ops for a marginal convenience gain.

NVIDIA DGX Spark. This is the fit. GB10 Grace-Blackwell superchip, 128 GB unified memory, Ubuntu-based DGX OS. The PersonaPlex README already documents the Blackwell install path (pip install torch... --index-url https://download.pytorch.org/whl/cu130). The Spark sits on a shelf in the back office, on the bar's LAN. Audio never leaves the building. Rough memory-budget math for co-locating three models: PersonaPlex 7B at FP16 ≈ 14 GB + KV-cache and audio I/O buffers (~4-8 GB under full-duplex load); Whisper large-v3 ≈ 3 GB weights + ~2 GB runtime; a small local LLM for hard-stops enforcement (Llama-3.1-8B or Phi-4 class) ≈ 16 GB FP16 + 4 GB context — call it ≈ 50 GB steady state on a 128 GB unified pool, with headroom for spikes and the ConnectX-7 networking stack. Fits comfortably, but the actual number belongs to the spike, not the proposal. One box, local LAN, three capabilities, and the privacy story stays intact.

Honest caveats on Spark — to be resolved before committing, not after:

• ARM64 + CUDA. Spark is aarch64. PyTorch has ARM64+CUDA wheels now, but Moshi's Rust audio dependencies (sphn, Opus bindings) are the historical weak point on non-x86 hosts. Likely works; not proven by this document.
• GB10 specifically. The README's Blackwell note appears to target discrete workstation GPUs (RTX 50-series). I haven't seen public confirmation someone ran PersonaPlex on GB10. The install path is the same, but "should work" is not "someone did it."
• Full-duplex latency on Spark. Spark is designed for local inference, not ultra-low-latency streaming. A 7B full-duplex workload is well inside its compute budget, but the actual 95th-percentile turn-around time needs a test, not a forecast.

---

How this intersects with issue #1

This proposal does not replace issue #1; it attaches to it.

• Phase 3 is a hard gate. The sanitizer, hard-stops function, and voice check must exist before any mic or phone line goes live. Audio is harder to sanitize than logs (nothing to regex; you're editing waveforms). The voice check from Phase 3 gets a second job here: confirming the model didn't drift out of Maya's written voice rules even when generating audio. The cheapest implementation is to transcribe the audio at output time and run the text through the same voice check the text skills use.
• Legal gates, not open questions. Two statutory gates apply before any voice surface goes live, and they're gates, not preferences:
  • California SB 1001 (Bus. & Prof. Code §17940–43). Requires disclosure when a bot is used to "incentivize a purchase or sale of goods or services in a commercial transaction" in online interaction. Inbound phone calls for vendor orders or bookings fall inside that. The spoken disclosure at call start is not optional for use case Add open source community files and templates #2.
  • Cal. Penal Code §632 (two-party consent). Recording or transcribing confidential phone communications without all parties' consent is a criminal matter. Any Whisper transcription of a vendor call, or any logging of call audio for Phase 3 voice-check purposes, requires the opening consent prompt to explicitly cover recording — not just bot disclosure. These constraints apply to use case Add open source community files and templates #2 (phone) unambiguously. Use case Proposal: improvement plan from initial repo audit #1 (in-bar mic) is not a recorded phone call under §632, but signage at the bar is still warranted as a matter of trust — that part is a preference, not a statute.
• Phase 4 grows a node. The DGX Spark is a second unattended machine on the bar's LAN. Phase 4's Tailscale-only access, FileVault (or the Spark's equivalent), and immutable backup story all have to cover it too. Firewall rules on the Mac Mini and the Spark: neither exposes anything to the internet; the Spark only accepts traffic from the Mac Mini. PSTN note: use case Add open source community files and templates #2 introduces a SIP attack surface (inbound registrations, RTP floods, toll fraud) that Phase 4's Tailscale-only stance does not cover. Either terminate SIP at a hosted Twilio endpoint and let them absorb the public-internet side, or stand up a dedicated SBC (Kamailio / OpenSIPS) in front of the Mac Mini and add it to the Phase 4 threat model. This is new hardening work, not a re-labeling of Phase 4.
• Phase 6 stays unchanged. Skills implemented there stay text-first. This proposal adds a voice frontend that routes into the same skills. A skill that doesn't care where its input came from — text or transcribed voice — is the cleanest pattern.
• This is Track B, not a new phase. Numbering it "Phase 8 (or 9)" would pretend this proposal is on the critical path. It isn't. Call it Track B: parallel research that runs independently of Phases 0-7 and activates only after Phase 6 has proven at least two skills end-to-end and Phase 5 has proven the system can run unattended. If Phases 0-7 stall, Track B stays on the shelf without reordering anything.

---

Proposed shape: three-week spike, then decide

Committing to PersonaPlex sight-unseen on novel hardware is the wrong call. Running a bounded spike and deciding from data is the right one. If the spike fails, nothing's lost except three weeks of time on a research track that was never on the critical path.

Week 1 — Stand up. Provision the Spark. Install DGX OS, Tailscale, the CUDA toolkit matching the README. Install libopus-dev, pip install moshi/., accept the HF license, pull the personaplex-7b-v1 weights. Run the README's offline evaluation script on the provided test wav. Goal: the baseline assistant example produces audible output. Document every ARM64 gotcha encountered.

Week 2 — Make it Maya. Author Maya's persona prompt from SOUL.md and AGENTS.md. Record or select a voice prompt (see open questions). Run the offline evaluation on three scripted vendor-call-style scenarios (a standing order, a late delivery follow-up, a musician booking), 10 runs per scenario. Blind-compare to a cascade stack (Whisper + local LLM + a TTS voice) on the same scripts. Week 2 pass criteria (all must hold):

• Persona fidelity. At least 4 of 5 blind raters prefer PersonaPlex over the cascade stack on "sounds like Maya" across a combined 30-run sample.
• Hard-stops. 0 violations of the hard-stops list across 3 scenarios × 10 runs (30 runs). One violation fails Week 2.
• Voice-rule drift. When Maya's audio output is transcribed and passed through the Phase 3 voice check, ≥95% of turns pass (caps on em-dashes, exclamations, corporate filler, sentence length).
• Transcription fidelity. WER on Maya's output (transcribed via Whisper large-v3) ≤ 10% against the source script, so downstream voice-check and logging have a clean signal.

Week 3 — Wire one skill end-to-end. Pick menu-spec (read-only, trivial, from issue #1 Phase 6). A customer asks "what's the Old Fashioned spec?" into a mic; Maya looks it up and answers in voice. Exercise the Phase 3 voice check by transcribing Maya's output and running it through the text voice rules. Week 3 pass criteria:

• Latency. p95 end-of-user-speech → first-audio-out ≤ 500ms; p95 full turn (first-audio-out → end-of-Maya-speech ready to accept barge-in) ≤ 1.5s. p95 full turn > 2.0s is automatic no-go — at that point, it's not full-duplex, it's a cascade stack wearing a different hat.
• Barge-in. Barge-in (user interrupts mid-answer) succeeds on ≥ 90% of attempts across 20 trials.
• Sustained load. 60 minutes of continuous mixed-scenario conversation on the Spark without OOM, thermal throttling, or p95 latency regressing by more than 20% from the first minute.

Decision at the end of the spike. Go if Week 2 and Week 3 both pass all criteria above. No-go if any single Week 3 latency criterion fails, or two or more Week 2 criteria fail. Ambiguous middle (one Week 2 miss, Week 3 clean): write up and escalate to Christian.

---

Open questions for Christian

1. Whose voice, and under what contract? This is the biggest question and the one with the longest lead time. Four options: (a) hire a voice actor to record the conditioning clip (most natural, costs money); (b) use Christian's voice (free, raises different concerns about personal brand leaking into a commercial surface); (c) a friend who consents (charming); (d) one of PersonaPlex's pre-packaged NAT voices (instant, generic). Whichever is picked, a written release is needed before the voice prompt is committed. The release must cover: consent to voice-print use in a commercial context, scope of use (this bar only, or transferable?), persistence (the embedding is effectively a biometric derivative — where does it live, who can read it, does the Phase 3 sanitizer treat it as sensitive memory?), and revocation (if the person later objects, what gets destroyed and how — the committed embedding, downstream audio that contains it, model outputs that may have been logged). "What if they later object" is not answerable without that contract up front, and under California civil code §3344 unauthorized commercial use of someone's voice is actionable. Worth deciding before Week 2.

2. Telephony. Twilio SIP is the obvious choice and integrates cleanly with a local PersonaPlex server over RTP. A POTS-to-SIP bridge (an ATA) is the move if the bar has an existing phone number worth keeping. An inbound ring goes to Maya; how does she know to answer vs. hand off to voicemail? Answer: a simple router skill on the Mac Mini — business hours + keyword trigger on first few seconds.

3. Does "regulars' info never gets exported" extend to LLM API calls? This is a question issue Proposal: improvement plan from initial repo audit #1 Phase 3 already asked and deferred. PersonaPlex makes the question urgent, because the answer directly determines whether cloud inference is ever acceptable. My read is "yes, never exported," but this is your call.

4. Customer disclosure wording and placement. The statutory shape of the disclosure (SB 1001 for phone, §632 for recording) is covered in the "Legal gates" section above and is not up for discussion. What is up for discussion: the wording of the spoken disclosure that opens a vendor or customer call, and the text of the bar-side signage. Both should sound like Maya — not a corporate legal notice. Ideally Christian drafts these in his own voice, since they're the part of Maya that speaks for the house before Maya herself does.

5. When this proposal activates. Not before issue Proposal: improvement plan from initial repo audit #1 Phase 3 is complete and Phase 6 has shipped at least inventory-check and menu-spec. If Phase 0-3 stalls, this proposal stays on the shelf.

---

What this proposal intentionally leaves out

• No voice-based customer identification of any kind, even if technically possible.
• No voice prints of regulars stored. Customer audio is transient; nothing persists to disk.
• No voice-based "recommend me a drink" features. Maya serves what's on the menu; she doesn't recommend based on inferred mood or regulars' history.
• No PersonaPlex for the voice-notes use case. Whisper or MLX-Whisper, separate skill.
• No cloud inference. Spark on LAN or nothing.
• No fine-tuning of PersonaPlex on bar-specific data. The pre-trained model plus persona prompting is enough. Fine-tuning introduces data-handling risks with no proven upside.
• No integration with a POS. Same reason as issue Proposal: improvement plan from initial repo audit #1: out of scope until the base is solid.

---

References

• NVIDIA/personaplex repo
• PersonaPlex paper (arXiv 2602.06053) — arXiv ID sourced directly from the upstream README's paper badge and BibTeX entry; valid YYMM ID for Feb 2026
• Moshi paper (arXiv 2410.00037)
• Helium (Kyutai)
• NVIDIA DGX Spark
• California SB 1001 (Bolstering Online Transparency Act) — bot disclosure in commercial online interactions
• Cal. Penal Code §632 — two-party consent for recording confidential communications
• Cal. Civil Code §3344 — unauthorized commercial use of voice/likeness
• Issue Proposal: improvement plan from initial repo audit #1 in this repo — phase context this proposal attaches to

---

How to file

• Open as a single issue. The decision this document drives (spike yes/no, then go/no-go) is coherent; splitting into sub-issues before the spike runs creates ceremony without signal.
• Reference #1 with Refs: #1 (not Closes: #1) — this is Track B of the roadmap in issue Proposal: improvement plan from initial repo audit #1, not a replacement for any of its phases.
• Suggested labels: proposal, voice, hardware, research-spike. Add blocked-by:#1 if repo labels support that convention; otherwise note the Phase-3-and-Phase-5-precondition inline.
• No milestone until Phase 5 of issue Proposal: improvement plan from initial repo audit #1 is within one sprint of shipping. Attaching a milestone earlier inflates the critical path.