linkedListOnSteem

A Python library for implementing a doubly-linked list data structure using custom_json transactions on the Steem blockchain.

⚠️ CAUTION: The primary intended use case for this library is single-process / single-user applications. While Optimistic Concurrency Control (OCC) methods (safe_append, safe_delete) are provided, there is inherent uncertainty and potential risk of list forks when lists are accessed and modified concurrently by multiple users or threads. Use multi-author or concurrent features with extreme caution.

Overview

Because Steem transactions are immutable once confirmed, traditional "next" pointers cannot be back-patched when new nodes are added. This library overcomes this constraint by maintaining a continuous chain of prev pointers (using precise block_num and trx_id references) and building a lightweight off-chain index.

The index is quickly rebuilt on-demand by locating the list's tail in the account history and walking the pointers backwards to the head. This requires only O(n) targeted API fetches rather than a broad, expensive blockchain scan, typically completing in seconds.

Features

• Append-Only Immutability: Store arbitrary JSON payloads securely on the Steem blockchain.
• Soft Deletion: Logical deletion via tombstone nodes appended to the list, preserving the integrity of the immutable prev chain while seamlessly hiding deleted items from normal traversal.
• Efficient Syncing:
  • rebuild_index(): Reconstructs the list in seconds, regardless of how long ago it was created.
  • sync(): Incrementally fetches only new nodes added since the last check.
• Multi-Author Concurrency Safe: Use safe_append() and safe_delete() to enable Optimistic Concurrency Control (OCC). If multiple accounts write to the same list simultaneously, the library uses direct block scanning to detect and resolve forks, and applies exponential backoff with jitter to retry operations reliably under high contention, ensuring data consistency.
• Local Caching: Export and import the list index (export_index() / import_index()) to avoid re-querying the blockchain on startup.
• Rich Traversal: Forward and reverse iteration, absolute and active index access (get(), get_active()), and predicate-based searching (find(), find_all()).

Requirements

• Python 3.7+
• steem-python (pip install steem)

Installation

From GitHub

pip install git+https://github.com/remlaps/linkedListOnSteem.git

Development Installation

For development purposes, you can clone the repository and install in editable mode:

git clone https://github.com/remlaps/linkedListOnSteem.git
cd linkedListOnSteem
pip install -e .

Quick Start

Make sure to set your Steem posting key securely, for example via an environment variable.

import os
from steem import Steem
from linkedListOnSteem import SteemLinkedList

# 1. Connect to Steem
POSTING_KEY = os.getenv("STEEM_POSTING_KEY")
steem = Steem(keys=[POSTING_KEY])

# 2. Initialize the List
ll = SteemLinkedList(
    steem_instance=steem,
    account="your-steem-account",
    ll_id="my_unique_list_id",
    custom_json_id="linkedListOnSteem"
)

# 3. Append Nodes
# Use safe_append if multiple app instances might write at the same time
ll.safe_append({"title": "First Entry", "value": 100})
ll.safe_append({"title": "Second Entry", "value": 200})

# 4. Read & Traverse
# (In a fresh session, rebuild the index first)
ll.rebuild_index() 

for node in ll:
    print(f"Sequence: {node.seq}, Data: {node.payload}")

Examples and Testing

• Examples: Check out examples/steem_linked_list_examples.py for a comprehensive walkthrough of all features, including soft-deletion, reverse traversal, and caching.
• Tests: Run tests/steem_linked_list_post_test.py to execute a full integration test suite. This script optionally broadcasts its execution report directly to the Steem blockchain as a top-level post.

Multi-Author Lists & Architectural Limitations

The library's concurrency control (safe_append, safe_delete) is designed to work across multiple accounts, allowing for collaborative lists.

Important Limitation: Due to how data is indexed on the Steem blockchain, discovering list updates from other authors is only reliable for recent activity (within the last ~10-15 minutes). The rebuild_index() and sync() methods are optimized to bridge short-term API node lag but cannot efficiently scan months of blockchain history to find an append from a different author.

For applications requiring robust multi-author collaboration over long timeframes, a dedicated off-chain indexer service (a "TailTracker") is the recommended architecture to discover list tails across the entire blockchain history. The library can then use that information to verify the data on-chain.

License

MIT