docs: data persistence guide

[marc0olo]

Summary

• Covers Motoko persistent actor pattern: auto-persisted let/var, transient var, schema evolution rules
• Covers Rust stable structures: StableBTreeMap, StableCell, StableLog, MemoryManager, MemoryId partitioning, Storable impl with CBOR via ciborium
• Explains the dangerous pre_upgrade heap serialization anti-pattern (instruction limit → trapped upgrade → bricked canister)
• Includes idempotency patterns for safe data mutation (sequence numbers and ID deduplication)
• Includes CLI test sequence to verify data survives upgrade

Sync recommendation

Informed by dfinity/portal docs/building-apps/canister-management/storage.mdx, docs/building-apps/best-practices/storage.mdx, docs/building-apps/best-practices/idempotency.mdx — hand-written synthesis across multiple portal sources plus icskills/stable-memory.

[Copilot]

Pull request overview

Adds a comprehensive “Data Persistence” guide explaining durable canister state patterns across Motoko (persistent actors) and Rust (ic-stable-structures), plus upgrade-safety pitfalls and idempotency techniques.

Changes:

• Documented Motoko persistent actor persistence rules, transient state, and schema evolution do/don’ts.
• Documented Rust stable structures usage with MemoryManager partitioning and custom Storable implementations (CBOR via ciborium).
• Added guidance on avoiding pre_upgrade heap-serialization and introduced idempotency patterns + a CLI upgrade verification sequence.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[marc0olo]

Review: Data Persistence

Must fix

1. Intro misrepresents Motoko's heap behavior
The opening sentence says "heap memory (fast, temporary, wiped on upgrade)." This contradicts the page's own content: persistent actor automatically preserves heap state across upgrades. The description only applies to Rust/other languages. Rephrase to distinguish: heap is wiped on upgrade unless you use Motoko persistent actor, which the runtime preserves automatically.

2. "data is unrecoverable" overstated for Motoko schema evolution

"Violating this traps the upgrade and data is unrecoverable."

A trapped Motoko schema upgrade leaves the canister running on the old Wasm — data is intact and you can recover by deploying a compatible type. "Unrecoverable" is wrong. Rephrase to: "the upgrade traps; the canister continues running on the old code with data intact, but cannot be upgraded until the type conflict is resolved."

3. Bounded vs Unbounded failure mode is inaccurate
The comment in the Storable impl says:

"Bounded requires a fixed max_size; exceeding it after a schema change breaks deserialization of existing data."

This is wrong. max_size is a write-time constraint — if encoded bytes exceed max_size, inserts/updates trap. Existing stored data is not affected. Change to: "exceeding max_size causes writes to trap, preventing any new or updated records from being stored."

4. "no way to recover" from trapped pre_upgrade
skip_pre_upgrade = opt true is an official IC protocol flag that bypasses the failing hook (at the cost of losing any state that pre_upgrade would have serialized). The page should mention this escape hatch while still warning strongly about the pattern.

5. "Multiple stable structures" snippet is not self-contained
The snippet uses User, Post, Memory, MemoryId, MemoryManager, DefaultMemoryImpl, and RefCell without defining or importing them. The text doesn't say this builds on the prior snippet. Add a note like "This extends the previous example" and list what the reader needs from it — or make the snippet standalone.

6. CLI persistence test uses Motoko method names against a Rust canister
The CLI block calls addUser, getUserCount, getUser (camelCase). These match the Motoko actor. The Rust canister exports add_user, get_user_count, get_user (snake_case). The test block appears at the end of the Rust section with no label — a reader following the Rust path will see failed calls. Either label the block "Motoko backend" or show a Rust variant.

7. Idempotency snippets missing imports

• Sequence numbers snippet: uses Map, Principal, Nat with no imports.
• ID deduplication snippet: imports Time but uses Map.empty, Map.get, Map.add, Text.compare without importing Map or Text.

Both snippets need full imports or an explicit note that they must be merged into the persistent actor from the Motoko section.

---

Verified

• All four internal links resolve to existing files (orthogonal-persistence.md, lifecycle.md, stable-structures.md, motoko/index.md)
• icp network start -d (background flag) is valid per .sources/icp-cli/docs/reference/cli.md
• icp deploy, icp canister call commands verified correct
• Motoko Map.empty, Map.add, Map.get, Map.size with .compare comparator verified correct against .sources/motoko-core/src/Map.mo
• MemoryManager, StableBTreeMap, StableCell, StableLog usage pattern verified against portal source
• Frontmatter is complete and consistent with body content
• No dfx references, no .mdx/JSX, no links to old docs

[marc0olo]

Feedback addressed:

• Rephrased intro to correctly distinguish heap behavior: wiped on upgrade in Rust, automatically preserved in Motoko persistent actor
• Fixed schema evolution wording: a trapped upgrade leaves the canister on old Wasm with data intact — not "unrecoverable"
• Fixed Bounded vs Unbounded comment: exceeding max_size traps writes, does not break deserialization of existing data
• Mentioned skip_pre_upgrade as the official IC recovery option for a trapped pre_upgrade hook
• Made "Multiple stable structures" snippet self-contained with full imports, Memory type alias, and Post struct definition
• Split CLI persistence test into separate Motoko (camelCase) and Rust (snake_case) variants to match each backend's exported method names
• Added missing imports to both idempotency snippets (Map, Principal, Nat for sequence numbers; Map, Text for ID deduplication)