Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ UI: REPL (prompt_toolkit) │ one-shot CLI (argparse)
Dispatch: parse → validate → run → render
Commands: commands/{target,profile,media,network,content,interactions,batch,watch,operational,dossier}.py
Service: facade · history · analytics · exporter · watch
Backends: OSINTBackend ABC · HikerBackend (v0.1) · AiograpiBackend (v0.2)
Backends: OSINTBackend ABC · HikerBackend · AiograpiBackend
Models: @dataclass(slots=True) DTOs — Profile, Post, Story, User, Comment, Quota, ...
```

## Conventions

- **Async everywhere.** `httpx` (transitive via `hikerapi`), `asyncio` for fan-out, `asyncio.to_thread` for sqlite calls.
- **Backend boundary is a hard wall.** Raw HikerAPI / aiograpi dicts never leave `backends/`. Mappers in `_hiker_map.py` (and the future `_aiograpi_map.py`) are the only converters.
- **Lazy backend imports.** `import hikerapi` happens only inside `make_backend("hiker")`. v0.2's `import aiograpi` will be the same. Import errors stay localized.
- **Backend boundary is a hard wall.** Raw HikerAPI / aiograpi dicts never leave `backends/`. Mappers in `_hiker_map.py` and `_aiograpi_map.py` are the only converters.
- **Lazy backend imports.** `import hikerapi` and `import aiograpi` happen only inside `make_backend(...)`. Import errors stay localized.
- **Retry / backoff lives in one place.** `backends/_retry.py` decorates SDK-method calls inside `HikerBackend`; commands never know retries exist.
- **CDN streaming through a single helper.** `backends/_cdn.py` is the only code that pulls untrusted bytes off the network. Host allowlist, MIME sniff, byte budget, atomic write — every download passes through it.
- **Pagination as `AsyncIterator[T]` + `limit: int | None`.** Every collection method is an async generator. Cursor management lives inside the backend; commands consume one item at a time and stop on `limit`.
Expand Down Expand Up @@ -79,7 +79,7 @@ JSON exports are versioned: every file has `{"_schema": "insto.v1", "command": .

## Watch

Session-only in v0.1 (daemon mode is v0.2). `/watch <user> <interval>` registers an `asyncio.Task` on the same loop that runs `PromptSession.prompt_async()`. Each tick is wrapped in `asyncio.shield(...)` and a single retry; two consecutive failures mark the watch `paused`. Notifications go through `prompt_toolkit.patch_stdout` so the user's in-progress input line is not corrupted.
Session-only today; daemon mode is tracked as deferred work. `/watch <user> <interval>` registers an `asyncio.Task` on the same loop that runs `PromptSession.prompt_async()`. Each tick is wrapped in `asyncio.shield(...)` and a single retry; two consecutive failures mark the watch `paused`. Notifications go through `prompt_toolkit.patch_stdout` so the user's in-progress input line is not corrupted.

Session limits: max 3 active watches, 5-minute floor on the interval, all watches cancelled cleanly on REPL exit.

Expand Down
12 changes: 6 additions & 6 deletions docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

`insto` is split between command surface and backend. The backend is the only layer that touches a third-party API; everything above it consumes DTOs (`Profile`, `Post`, `User`, `Comment`, ...) and never sees a raw HikerAPI / aiograpi dict.

The contract lives in `insto/backends/_base.py:OSINTBackend`. Two implementations ship as of 0.2.0:
The contract lives in `insto/backends/_base.py:OSINTBackend`. Two implementations ship:

| | **hiker** (default install) | **aiograpi** (`insto[aiograpi]`) |
|---|---|---|
Expand All @@ -11,13 +11,13 @@ The contract lives in `insto/backends/_base.py:OSINTBackend`. Two implementation
| Account ban risk | None | Real |
| Stability | High | Brittle (Instagram churn) |
| Sees private accounts you follow | No | Yes |
| Sees DMs / saved feed | No | Yes (planned) |
| Sees DMs / saved feed | No | aiograpi has SDK methods; insto does not expose CLI commands yet |
| Quota visibility | Yes (`/sys/balance`) | No |
| Install footprint | base | `pip install 'insto[aiograpi]'` |

## Pick a backend

Default is **hiker**. Switch to aiograpi when you need data behind Instagram's login wall — private profiles you follow, saved feed, posts on accounts that 403 from logged-out HTTP. For OSINT on public profiles, hiker is the right choice nine times out of ten and carries no account-ban risk.
Default is **hiker**. Switch to aiograpi when you need data behind Instagram's login wall — private profiles you follow or posts on accounts that 403 from logged-out HTTP. For OSINT on public profiles, hiker is the right choice nine times out of ten and carries no account-ban risk.

You can flip backends mid-session at any time by editing `~/.insto/config.toml` (or running `insto setup` again):

Expand Down Expand Up @@ -84,13 +84,13 @@ pip install 'insto[aiograpi]' # in a venv only — see installation.md

Then run `insto setup`, pick `aiograpi`, paste your Instagram username + password, and (optionally) the TOTP seed for 2FA.

What works on aiograpi ( 0.8.0):
What works on aiograpi (>= 0.9.6):

- Every command — `/info`, `/posts`, `/reels`, `/stories`, `/highlights`, `/followers`, `/followings`, `/mutuals`, `/comments`, `/captions`, `/likes`, `/wcommented`, `/hashtags`, `/mentions`, `/locations`, `/tagged`, `/similar`, `/dossier`.
- Reads private profiles you follow.
- Login is **lazy** — the constructor stores credentials, the actual `client.login()` fires on the first network call. The session is then dumped to `~/.insto/aiograpi.session.json` (mode `0600`); subsequent runs reuse it without re-authenticating.

aiograpi 0.7 was missing `chaining` / `fetch_suggestion_details` (used by `/similar`); insto's `[aiograpi]` extra now requires aiograpi ≥ 0.8.0 so this is settled at install time.
aiograpi 0.9.x also exposes a much larger Direct, private GraphQL, music, archive, and collection surface. Insto intentionally exposes only read-oriented OSINT commands today. Read-only Direct inbox and saved-collection support are tracked as follow-up work in the [Roadmap](roadmap.md).

### Account-ban risk

Expand Down Expand Up @@ -120,4 +120,4 @@ Don't have the seed? Either re-enable 2FA in Instagram's settings to capture it,
| `blocked` | n/a | ✓ (only via aiograpi error response) |
| `deleted` | ✓ | ✓ |

Commands that strictly need a logged-in account (DMs, saved feed, posts of a private profile you follow) carry a `requires=("followed",)` annotation. They run cleanly on aiograpi; on hiker they exit with a typed message.
Commands that strictly need a logged-in account carry a `requires=("followed",)` annotation. They run cleanly on aiograpi; on hiker they exit with a typed message.
2 changes: 1 addition & 1 deletion docs/basic-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ insto @nasa -c hashtags --csv - | awk -F, '$2=="space"{print $3}'
/unwatch nasa
```

Watches are session-local — they die when you exit. Persistent watches (daemon mode) are deferred to v0.2.
Watches are session-local — they die when you exit. Persistent watches (daemon mode) are deferred.

## Privacy

Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Two surfaces over the same command grammar:

## What it is NOT

- Not a scraper. v0.1 talks to HikerAPI exclusively (paid, no account-ban risk). v0.2 will add `aiograpi` for private-account access.
- Not a scraper. The default backend is HikerAPI (paid, no account-ban risk). The optional `aiograpi` backend uses a real logged-in Instagram account for private-account access and carries account-ban risk.
- Not an account hijacker / DM-flooder / ban-evader.
- No AI / LLM features. No web UI.

Expand All @@ -42,7 +42,7 @@ Two surfaces over the same command grammar:

## Pick a backend

| | **hiker** (default in v0.1) | **aiograpi** (v0.2) |
| | **hiker** (default) | **aiograpi** (`insto[aiograpi]`) |
|---|---|---|
| Auth | API token | Instagram login + 2FA |
| Cost | Pay-per-call, [100 free requests](https://hikerapi.com/p/6k1q1388) at signup (no card) | Free |
Expand Down
55 changes: 55 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Roadmap

Deferred work that is still relevant after the aiograpi 0.9.x update.

## aiograpi follow-ups

### Read-only Direct inbox

Add read-only commands for Direct: list threads, show recent messages in a thread, and search threads by participant. Do not add send, reaction, or unsend commands to the core CLI.

Why: aiograpi 0.9.x synced the current Direct API surface, including message requests, single-message lookup, reactions, title updates, unsend, and voice/video attachments. For insto's OSINT surface, only the read-only subset fits the product boundary.

Priority: P2

### Saved collections / personal feed read-only

Audit the current aiograpi collection and feed surfaces, then add read-only commands only if the API is stable on a live-smoke burner account.

Why: the SDK has collection/feed capability, but insto does not yet have a CLI contract for saved media or personal feed data.

Priority: P3

### Private GraphQL pagination audit

Compare the current `user_followers`, `user_following`, and media methods with the newer private GraphQL helpers in aiograpi 0.9.x.

Why: private GraphQL may improve stability for followers, following, clips, inbox, and search, but it may also increase ban risk. Switch only after fixture diffs and small-limit live-smoke checks.

Priority: P2

## Existing deferred work

### Persistent watch daemon

Turn session-local `/watch` registrations into a persistent daemon with restart recovery and a control surface from REPL / one-shot CLI.

Priority: P2

### At-rest store encryption

Encrypt `~/.insto/store.db` and snapshot backups with SQLCipher, GPG, or age when a real multi-operator or compliance use case appears.

Priority: P2

### `/replay <N>`

Replay a previous command from `cli_history`, with an explicit whitelist for safe commands and an option to redirect the target.

Priority: P3

### Plugin API

Expose entry points for third-party commands and backends after there is a real external extension to design against.

Priority: P3
11 changes: 5 additions & 6 deletions insto/backends/_base.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
"""Abstract OSINT backend interface.

`OSINTBackend` is the contract every backend (HikerAPI v0.1, aiograpi v0.2,
future TikTok / Bluesky / Threads providers) must implement. The command and
service layers depend on this ABC, never on a concrete backend — that is what
keeps v0.2 a pure addition.
`OSINTBackend` is the contract every backend (HikerAPI, aiograpi, future
TikTok / Bluesky / Threads providers) must implement. The command and service
layers depend on this ABC, never on a concrete backend.

All collection-returning methods are async generators (`AsyncIterator[T]`)
with an optional `limit: int | None` parameter. Cursors / page tokens are an
Expand Down Expand Up @@ -45,8 +44,8 @@ class OSINTBackend(ABC):
# Capability tokens this backend exposes. Commands declare what they need
# via `@command(..., requires=("followed",))`; the dispatcher rejects the
# call when the active backend does not advertise the required tokens.
# HikerAPI exposes only public OSINT, so the default is empty; an
# `aiograpi` backend would extend this with `{"followed", ...}`.
# HikerAPI exposes only public OSINT, so the default is empty.
# AiograpiBackend extends this with `{"followed", ...}`.
capabilities: frozenset[str] = frozenset()

@abstractmethod
Expand Down
4 changes: 2 additions & 2 deletions insto/backends/_hiker_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,8 @@ def map_profile(d: dict[str, Any]) -> Profile:
"""Map a HikerAPI `user` payload to a `Profile` DTO.

Required: `pk`, `username`. Avatar URL prefers `_hd` then base.
`access` is derived from `is_private` (v0.1: HikerAPI cannot represent
`followed` / `blocked` — those land with the aiograpi backend in v0.2).
`access` is derived from `is_private`; HikerAPI cannot represent
`followed` / `blocked`, which are logged-in aiograpi-only states.
"""
pk = _require(d, "pk", "user")
username = _require(d, "username", "user")
Expand Down
8 changes: 4 additions & 4 deletions insto/backends/aiograpi.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@
layer never sees a raw aiograpi exception. New aiograpi exception
classes are caught by the `ClientError` fallback as `Transient`.

All `OSINTBackend` methods are implemented. `get_suggested(pk)` and
`iter_user_tagged(pk)` need aiograpi 0.8.0 (the release that added
`chaining` / `fetch_suggestion_details` and exposed `usertag_medias_v1`).
The `[aiograpi]` extra in `pyproject.toml` enforces this minimum.
All `OSINTBackend` methods are implemented. The `[aiograpi]` extra pins to
aiograpi >= 0.9.6 so this backend gets the current Direct, private GraphQL,
music, archive, and media pagination fixes while keeping insto's own command
surface read-oriented.
"""

from __future__ import annotations
Expand Down
6 changes: 3 additions & 3 deletions insto/backends/hiker.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,15 +150,15 @@ def _translate_http_status(exc: httpx.HTTPStatusError) -> BackendError:
# Instagram itself returns, not HikerAPI's plan / scope error.
# Typical causes:
# 1. The endpoint is login-walled (Instagram demands a session
# cookie). v0.1 hiker has no cookie; v0.2 aiograpi will.
# cookie). Hiker has no cookie; aiograpi does.
# 2. The target's profile is region-restricted, age-gated, or
# throttling third-party introspection.
# 401 stays the only "your HikerAPI access is wrong" signal.
# /info / /quota for other targets will keep working.
return Banned(
"Instagram returned 403 for this lookup (login-walled or "
"target-restricted). v0.1 hiker can't log in; this endpoint "
"will likely need the v0.2 aiograpi backend. Other commands "
"target-restricted). The hiker backend can't log in; this "
"endpoint will likely need the aiograpi backend. Other commands "
"and other targets should still work."
)
if status == 404:
Expand Down
6 changes: 3 additions & 3 deletions insto/commands/discovery.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""Discovery / utility commands powered by aiograpi 0.8.x ports.
"""Discovery / utility commands powered by aiograpi ports.

Three commands grouped here because they share a single source of new
capability (aiograpi >= 0.8.x) and don't fit neatly into the existing
``profile`` / ``network`` / ``media`` modules:
capability and don't fit neatly into the existing ``profile`` / ``network`` /
``media`` modules:

- ``/resolve <url>`` — expand an Instagram short-link
(``instagram.com/share/...``) to the canonical URL via a HEAD
Expand Down
2 changes: 1 addition & 1 deletion insto/commands/operational.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
async def quota_cmd(ctx: CommandContext) -> dict[str, Any]:
# Fresh fetch so /quota always reflects the live balance, not a stale
# header captured from the previous command's response. Backends that
# do not implement refresh_quota (e.g. FakeBackend, aiograpi v0.2)
# do not implement refresh_quota (e.g. FakeBackend, aiograpi)
# silently fall back to the cached value.
refresh = getattr(ctx.facade.backend, "refresh_quota", None)
if refresh is not None:
Expand Down
10 changes: 5 additions & 5 deletions insto/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
- Memory footprint stays small even for long iterators (followers / posts).
- `dataclasses.asdict(...)` produces a stable dict for JSON export.

Backend mappers (`backends/_hiker_map.py` and future aiograpi mapper) are
Backend mappers (`backends/_hiker_map.py` and `_aiograpi_map.py`) are
the *only* code allowed to construct these DTOs from raw provider payloads;
above the backend layer, code consumes DTOs and never sees raw dicts.
"""
Expand All @@ -29,9 +29,9 @@ class Profile:
is mutable metadata: when a rename is detected during snapshot diff, the
old value is appended to `previous_usernames`.

`access` reports visibility from the backend's vantage point. v0.1
(hiker) only ever returns `public | private | deleted`; v0.2 (aiograpi)
adds `followed | blocked`. `requires_followed=True` means the command
`access` reports visibility from the backend's vantage point. HikerAPI
returns `public | private | deleted`; aiograpi can also return
`followed | blocked`. `requires_followed=True` means the command
that produced this DTO needs a logged-in / following session — used by
commands that gate on follower-only content.
"""
Expand Down Expand Up @@ -216,7 +216,7 @@ def with_remaining(

@classmethod
def unknown(cls) -> Quota:
"""Backend that does not expose quota (aiograpi v0.2)."""
"""Backend that does not expose quota (aiograpi)."""
return cls(remaining=None, limit=None, reset_at=None)


Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ nav:
- Basic usage: basic-usage.md
- CLI reference: cli-reference.md
- Backends: backends.md
- Roadmap: roadmap.md
- Architecture: architecture.md
- Troubleshooting: troubleshooting.md
- Contributing: contributing.md
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Changelog = "https://github.com/subzeroid/insto/blob/main/CHANGELOG.md"

[project.optional-dependencies]
completion = ["shtab>=1.7"]
aiograpi = ["aiograpi>=0.8.5"]
aiograpi = ["aiograpi>=0.9.6"]
docs = [
"mkdocs>=1.6",
"mkdocs-material>=9.5",
Expand Down
Loading
Loading