IntentForge v2

Privacy-first, intent-driven search engine with zero-trust architecture.

IntentForge is a high-performance discovery platform that combines BM25 keyword matching with ONNX-powered semantic search to deliver intent-aligned results. Built with privacy at its core — no tracking, no data harvesting, no corporate dependencies.

---

Current Capabilities

🔍 Core Search

• Hybrid Search: Combines BM25 keyword matching with semantic vector search (ONNX embeddings)
• Intent Classification: Detects Navigational, Informational, Transactional, and Exploratory queries with adaptive semantic ratios
• Multi-tier Caching: L1 in-memory cache + Redis for sub-millisecond repeated queries
• Meta-search Aggregation: Simultaneously queries 9+ providers (Brave, Google, Wikipedia, Arxiv, Reddit, GitHub, Hacker News, Medium, DuckDuckGo)

🖼️ Image Search

• Zero-bandwidth indexing: Extracts context from HTML metadata (alt, title, captions, surrounding text) without downloading images
• Perceptual hashing: dHash (64-bit) for deduplication
• Visual fingerprints: ThumbHash for compact (~20-30 byte) thumbnail representation
• Quality gates: Filters tracking pixels, SVGs, and 1x1 placeholders

📹 Video Search

• Multi-source discovery: YouTube Unified, Piped, Invidious, Dailymotion, Vimeo, Internet Archive
• Intent-first scoring: Ranks by intent match, relevance, and quality signals
• Privacy-friendly frontends: Prefers Piped/Invidious over direct YouTube API

🛡️ Privacy & Security

• Anti-detection: TLS fingerprinting, randomized timing, viewport spoofing to bypass AI-generated content detection
• Tor integration: Optional route through Tor for anonymity
• No tracking: No cookies, no user profiling, no data retention
• Cloudflare bypass: Automated CAPTCHA solving for accessibility

📚 Content Extraction

• Trafilatura integration: Boilerplate removal, readable text extraction
• Multi-format support: Articles, documentation, technical content
• RSS firehose: 80+ curated sources for continuous content discovery

🚀 Performance

• Cross-encoder reranking: ms-marco-MiniLM-L6-v2 for precision reordering
• Adaptive semantic ratio: Query-type-aware blend of keyword vs semantic search
• Self-improvement: Automatic gap analysis triggers background crawling for weak results
• Common Crawl integration: Massive URL discovery from web archives

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        IntentForge Core (Rust)                  │
├──────────────┬──────────────┬───────────────┬───────────────────┤
│ Intent      │   Ranking    │  Discovery    │  Anti-Detection   │
│ Classifier  │   Engine     │  (Firehose)   │  + Cloudflare     │
├──────────────┴──────────────┴───────────────┴───────────────────┤
│                    Meilisearch Index (Hybrid)                   │
├─────────────────────────────────────────────────────────────────┤
│                 Query Layer (Python + FastAPI)                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐ │
│  │  ONNX Embed  │  │ Cross-Encoder│  │  Multi-tier Cache     │ │
│  │  (all-MiniLM)│  │  Reranker    │  │  L1 + Redis          │ │
│  └──────────────┘  └──────────────┘  └──────────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│                      Meta-Search API (Rust)                     │
│  Brave · Google · Wikipedia · Arxiv · Reddit · GitHub · HN     │
├─────────────────────────────────────────────────────────────────┤
│                    Video Services (Node.js)                     │
│           YouTube Unified · Piped · Invidious                  │
└─────────────────────────────────────────────────────────────────┘

---

API Endpoints

Endpoint	Method	Description
/search	GET	Hybrid intent-first search
/news	GET	News aggregation with geo-targeting
/images	GET	Semantic image discovery
/videos	GET	Intent-weighted video search
/meta	GET	Meta-search across all providers
/crawl	GET	Single URL content extraction
/crawl/batch	POST	Batch URL extraction
/health	GET	Service health check
/metrics	GET	Prometheus metrics

---

Stack

Component	Technology
Core Engine	Rust (Edition 2021)
Semantic Search	ONNX Runtime (all-MiniLM-L6-v2)
Cross-Encoder	ms-marco-MiniLM-L6-v2
Search Index	Meilisearch
Cache	Redis (Tier-2) + In-memory LRU (Tier-1)
Query Layer	Python + FastAPI
Meta-Search	Rust (9+ providers)
Video Discovery	Node.js (YouTube Unified)
Content Extraction	Trafilatura (Python)
Orchestration	Docker Compose

---

Privacy Principles

1. Zero data retention — No search logs, no analytics, no cookies
2. No corporate dependencies — All sources are open or privacy-respecting
3. Encrypted transport — HTTPS everywhere, Tor optional
4. No AI content detection fingerprinting — Built-in bypass
5. Open source — Full transparency on all components

---

Getting Started

# Full release build
cargo build --release

# Run API server (port 9100)
cargo run --release

# Docker full stack
docker-compose -f docker-compose.dev.yml up -d --build

---

Project Status

IntentForge v2 is operational and actively developed. The core search, image indexing, and video discovery systems are functional. Ongoing work focuses on latency optimization, expanded source coverage, and enhanced personalization.

See docs/ROADMAP.md for planned improvements.