Oura integration - WIP#143
Draft
pipiche38 wants to merge 18 commits into
Draft
Conversation
The 0x43 debug spill wasn't documented, so I added it in three linked spots:
- §2.4 (reassembly) — a dated "byte-alignment desync → phantom tags" note: how a run of 0x43 debug records surfaced as a phantom 0x4C "sleep_summary" because 0x4C=L/0x49=I/0x53=S are legal ASCII chars that alias real tags, plus the defensive resync rule (reject len<4, resynchronise on implausible type/len, never let a mid-stream byte mint a Tier-B summary).
- §6.15 (0x43) — expanded with the actual decoded strings ("Sw to App", "in_bed=0", "…in_info=6", "bc 0x43"), the TLV layout, and the aliasing caution.
- §6.12 (sleep summaries) — a "beware phantom sightings" warning so nobody chases an ASCII 0x4C as a real fixture.
Gave .debugText its own ingest case (it previously fell into default and was dropped). It now logs each non-empty, trimmed diagnostic string with its ring counter: Oura: debug (0x43) rt=60470 "in_bed=0" Diagnostics only — never persisted or scored. As a bonus, decoding these here also means a debug string can no longer silently alias a real event tag on that code path.
…wo mechanisms: 1. Drop-prefix list for the known high-rate firmware chatter: DHR_mode, DHR data sub, DHR_state, DHR_info, ← ~150ms HR state-machine spam PPG_cont, S:, E:, AFs, blestda, nr, BQ ← per-beat PPG windows / AGC / BLE state 1. Case-sensitive on purpose: the uppercase S:/E: PPG markers are dropped, while the lowercase s:/e: sleep boundaries survive. 2. Consecutive-duplicate suppression — collapses runs of identical lines (e.g. the DHR_info:neg t ×20 burst) via a lastDebugText guard.
The .streaming block now hands the ring the current UTC (OuraCommands.syncTime) BEFORE draining history, so it emits a usable 0x42 time-sync event (§5.5). Without this every fetched record stayed "[no anchor yet]" and last-night sleep / skin-temp could not be placed on a calendar day (Sleep screen read matched=0). SyncTime writes the ring clock, exactly as the official Oura app does on every connect. Sent once per session, before the first fetch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OuraReassembler.feed only validated len >= 4, never the type byte. Once byte-misaligned inside a 0x43 debug-text payload, ASCII letters that alias real tags (p=0x70, I=0x49, L=0x4C, Q=0x51) minted a PHANTOM "summary" whose bogus len swallowed the following real records (e.g. the sleep-phase timeline). Now a record only starts on a plausible type - a known OuraEventTag or the 0x11/0x0D outer responses that round-trip - else drop one byte and re-scan. Adds isPlausibleRecordStart + 3 FramingTests; realigns instead of stalling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A reassembler-misaligned phantom landing on a known spo2 tag byte decoded to impossible values and got persisted (integration build: 159 spo2Sample rows, -320..12,856,474), violating the honest-data invariant on a ring whose SpO2 feature 0x04 is gated OFF. OuraStreamMapping now persists an .spo2 sample only when its value is in the plausible oxygen-saturation band [85,100], aligned with open_oura tools/run_spo2.py (r-ratio calibration clamped to [85,100]) and Oura's own reporting floor. Values outside are DROPPED, never clamped - clamping garbage to "100%" would fabricate an oxygen reading. Only 0x6F yields direct percentages (~95-96, s6.5); 0x7B (unpinned uint16) and 0x77 dc_raw (PPG waveform) are not percentages, and nothing downstream reads spo2 red (AnalyticsEngine nulls spo2Pct), so this loses no real signal and yields zero rows on the gated-off ring. Mirrored in the Kotlin twin for parity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…§6.12.1) Investigated open_oura docs/algorithms/sleepnet.md. The Oura app's 4-stage hypnogram is NOT on the BLE wire: it is SleepNet, an on-PHONE PyTorch model (per-30s epochs) shipped AES-256-GCM-encrypted with a server-delivered, cloud-cached key — the one Oura metric not reproducible without Oura's cloud. Documents the consequence for NOOP's honest-data invariant: we do not chase SleepNet. A NOOP sleep session is built from signals that cross the BLE boundary — the ring's Tier-A 2-bit 0x4E/0x5A phase codes (OURA_SLEEP_PHASE), its 0x49/0x76 boundaries and 0x43 debug in_bed/s:/e: narration, plus NOOP's own staging over raw HR/HRV/temp/motion — never relabelled as "Oura sleep stages". Adds a [sleepnet] citation key.
A framing desync mints a record with a garbage ring timestamp that, at 100 ms/tick, converts to a date days-to-years from the anchor yet still inside the loose 2020-2035 epoch gate (rt~1.9e9 -> +6yr/2033; rt~16.7M -> +19 days). On live Gen 3 this scattered ~25% of a night's skin-temp rows across 2020-2034, so they never landed on the correct calendar day - and the same phantoms carried the garbage SpO2 values. OuraDriver.unixSeconds now rejects any conversion more than +1 day after or -90 days before the session anchor (history is always recent-past relative to the 0x42 anchor). Returns nil so the caller drops/parks the sample instead of banking a mis-dated row (honest-data invariant) - the single chokepoint every history sample passes. Mirrored in the Kotlin twin; +4 tests (Swift+Kotlin), OURA_PROTOCOL.md 5.5 documents the guard. 83 OuraProtocol tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AnalyticsEngine derives sleep from the gravity-driven SleepStager, and an Oura ring streams no accelerometer - so Sleep + Skin Temp stayed blank even with a night of correctly-dated OURA_SLEEP_PHASE events. NOOP now builds its OWN session from the ring's coarse phase codes (the polished hypnogram is SleepNet, cloud-gated, not on the wire - OURA_PROTOCOL.md 6.12.1). - OuraSleepSessionBuilder (pure): (ts, 2-bit stage) -> [SleepSession] with merged StageSegments + efficiency; splits on gaps, drops sub-minimum/all-wake runs. - analyzeDay gains providedSleepSessions: when set it REPLACES the gravity detector, so the ring's night flows through the same funnels (sleep totals + skin-temp window). nil default keeps every WHOOP caller byte-identical. - 11 tests incl. an end-to-end analyzeDay-with-no-gravity proof. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deep-dived Th0rgal/open_oura (Rust workspace RE'd from libringeventparser.so + the Android app) and folded the findings in as section 9 (leads to fixture-validate, per the doc's trust rule): - canonical connect->sync sequence + the "no RData 0x03 on a normal pull" rule; - authoritative opcode / 0x2F sub-op set; cursor = deciseconds; legacy vs extended; - feature-gating mechanism (server flags default false client-side) explaining the SpO2 / real_steps OFF defaults; - undocumented event tags (0x62 respiration spot, 0x8B spo2_r_pi, 0x71/0x6E IBI, 0x59/0x56/0x6C/0x74/0x84/0x86, 0x87/0x88 backend-gated, 0x80 gen conflict); - RData raw sampler (0x03) + the raw-accelerometer lead; batched timestamping; - reference SpO2 quadratic + skin-temp algorithm. Flags two actions in section 8: the SyncTime (0x12) send-layout mismatch and the skinTempFunnel 300-sample floor vs open_oura's ~120. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ded a live body-weight provider from the profile down to the estimator, matching the codebase's existing closure-injection style:
AppModel.profile.weightKg
→ SourceCoordinator(bodyweightKg:) [new param]
→ OuraLiveSource(bodyweightKg:) [new param]
→ OuraActivityEstimator.summarize(bodyweightKg: bodyweightKg())
- Read live (a closure, not a captured value), so if you edit your weight in Settings it takes effect on the next flush.
- Removed the activityBodyweightKg = 75.0 constant entirely.
- Default fallback is 70 kg (only used if profile weight is unset/≤0) — deliberately matching the production calorie path's own 70 kg fallback, so there's one consistent default across the app rather than introducing a second (75).
The old syncTime built a 24-bit floor(unix/256) coarse time (~4.3-min resolution) with a bogus 0xF6 trailer copied from the 0x85 RTC-beacon. This did not match the native client's req_sync_time(secs, 0) and could set the ring clock wrongly / be silently ignored (masked only by the ring's own RTC still yielding a ~correct 0x42 anchor). Now emit `12 09 <unix_secs: u64 LE (8 B)> <tz: i8 half-hours>`, tz=0 (UTC) as the reference client sends; NOOP buckets local days downstream. Mirrored byte-for-byte in the Kotlin twin. Tests pin the byte layout and the signed tz byte on both sides. Doc §5.4/§8/§9.2 updated to FIXED.
An Oura ring streams no accelerometer, so the gravity-driven SleepStager returns nothing for its nights: the Sleep card read blank AND the skin-temp funnel window (gated to a sleepSession) never opened. IntelligenceEngine now extracts the OURA_SLEEP_PHASE events from the night window's events (already read for the off-wrist backstop, no extra store read), builds sessions via OuraSleepSessionBuilder, and passes them to analyzeDay as providedSleepSessions. When non-nil these replace the gravity detector's output, so the ring's night flows through the SAME funnels the WHOOP path uses (sleep totals, skin-temp window, rest). Any non-Oura owner emits no such events, so the gravity stager stays byte-identical for every existing device. The built sessions land in res.cachedSleep, which the existing pipeline already persists. Kotlin twin: added providedSleepSessions to AnalyticsEngine.analyzeDay, the new OuraSleepSessionBuilder, and the IntelligenceEngine wiring (parsing `phase` from EventRow.payloadJSON). StrandAnalytics 964 tests pass, incl. the builder suite.
…lock The phantom guard (unixSeconds anchor-relative window) already rejected a misframed record's garbage ring timestamp on the ingest path. But the pending-anchor DRAIN collapsed the two reasons unixSeconds returns nil -- "no anchor yet" and "anchor present but rt is phantom" -- into a single '?? now` fallback. So a misframed skin-temp/SpO2/phase parked before the anchor landed was stamped at wall-clock and BANKED, slipping past the guard and polluting "today" with a bogus sample (honest-data violation). Expose driver.hasAnchor so the drain can tell them apart: anchor present + rt rejected -> DROP (and log the count); no anchor this session -> keep the honest wall-clock fallback so a real last-night record is placed at ~arrival rather than silently lost.
# Conflicts: # Strand/BLE/OuraLiveSource.swift
…p source)
The decoded OURA_SLEEP_PHASE (0x4E) events turn out to be sparse bursts
the ring emits only at connection time, not a continuous overnight
timeline -- a real 8 h night persisted ~30 phase events clustered at the
sync moments, none during sleep, so a session built from them
under-counts (9.2 h read as ~5 h).
The ring's firmware instead narrates its OWN sleep decision as 0x43
debug text: `check_sleep`, then `s: <rt>` (bedtime) / `e: <rt>` (wake),
where <rt> is a ring timestamp in the 0x42 anchor domain -- so
unixSeconds(rt) converts each straight to UTC. OuraCheckSleepParser
(pure, both platforms) extracts the window and OuraLiveSource logs the
anchored bedtime->wake span. Validated on device: last night read
23:46 -> 07:43 (7h 56m) vs the wearer's actual 23:49 -> 07:41.
A lone `e:` can arrive with no fresh `s:`, pairing the new wake against
the previous night's stale start -- observed live as a phantom 33 h
window. An 18 h max-window guard rejects it (honest-data).
Also confirmed and documented (OURA_PROTOCOL.md 6.15): the Tier-B
sleep_summary tags (0x49/4B/4C/57/58 = ASCII I/K/L/W/X) captured so far
are framing-desync aliases of debug text ("DHR_state:2", "batt: 100"),
not real summaries -- do not decode until a clean binary fixture.
INVESTIGATION prototype: the window is LOGGED, not yet persisted as a
sleepSession.
Promote the §6.15 check_sleep prototype to a real sleepSession. The
ring's own bedtime->wake decision (validated on device against imported
WHOOP reference: 476 min / 23:46->07:43 vs WHOOP 461-472 min /
23:49->07:41 -- within ~4 min) is the reliable duration source where the
sparse OURA_SLEEP_PHASE bursts under-count.
- OuraLiveSource persists an OURA_SLEEP_WINDOW event (ts=bedtime,
payload {start,end}) on a stable window; DO NOTHING upsert => one row
per night.
- OuraSleepSessionBuilder.session(fromWindowStart:end:) builds one
session: a single stage-UNKNOWN "asleep" segment, efficiency 1.0.
- hypnogramMetrics counts TST as any non-wake time -- byte-identical for
every gravity/phase session (light|deep|rem are the only non-wake
labels those stagers emit), and it lets the "asleep" window contribute
its full duration to total sleep WITHOUT inventing deep/REM/light
(those stay 0). Honest: real duration, blank stages.
- IntelligenceEngine reads OURA_SLEEP_WINDOW and PREFERS it over the
phase-event builder (gravity stager for non-Oura owners).
Kotlin twin throughout. StrandAnalytics 973 / WhoopStore 234 tests pass.
NOTE: surfacing is still gated -- the ring streams no overnight HR so
IntelligenceEngine's hr>=200 night gate skips Oura nights; addressed
next.
The persisted OURA_SLEEP_WINDOW never reached a card: an Oura ring banks no overnight HR (disconnected while you sleep), and two HR-only gates blocked it. Fix both — but rank a bare window BELOW any full record, so the ring surfaces only nights nothing richer owns. 1. Day ownership. resolveDayOwner marked hasData from HR presence alone, so the active ring (priority 0) lost every night to an imported WHOOP strap (priority 2). It now also counts a persisted OURA_SLEEP_WINDOW — BUT as new DayOwnerResolver.Candidate.richData=false. A richer record (richData=true: HR-derived stages/recovery/HRV) wins before device priority, so a WHOOP import keeps a day it fully recorded (same duration, but stages+recovery preserved) and the ring's bare window only surfaces a day nothing richer owns. richData defaults true, so every legacy candidate collapses to the old priority-only order — existing days are byte-identical. 2. Scoring gate. The loop skipped any night with < 200 HR samples. It now also scores a night with an OURA_SLEEP_WINDOW (duration honest from the window; HR-derived fields stay null) — reached only on days the ring actually owns per (1). New WhoopStore.hasEvent(deviceId:kind:from:to:) — a LIMIT-1 EXISTS probe (no payload decode) drives both. Kotlin twin throughout (DayOwnerResolver richData, WhoopDao.hasEvent + repo, IntelligenceEngine). WhoopStore 235 / StrandAnalytics 976 tests pass (+3 resolver richness cases). Validation: last connection logged the ring's Jul 7->8 check_sleep window at 9h12m = 552 min, matching the imported WHOOP night's totalSleepMin=552 to the minute. That night WHOOP (richer) keeps the day; Jul 8->9 (7h56m, no WHOOP import) the ring owns and surfaces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
PS/ Documenting the findings in docs/OURA_PROTOCOL.md
Type of change
How it was tested
Test environment is MacOs. I'm rebuilding the all application and running it. Comparing Oura ring Gen 3 collected data against Whoop/MG (connected to Whoop App)
Checklist
swift testinPackages/<name>)android/(./gradlew testFullDebugUnitTest)StrandDesigntokens — no hardcoded colors, fonts, or spacingdocs/CONTRIBUTING.mdStrand.xcodeproj/) or any secrets/keystoresRelated issues