Skip to content

Ishaanred/TextStruct

BranchesTags

Repository files navigation

TextStruct

TextStruct is a fully local, open-source pipeline for converting scanned textbooks into structured, LLM-ready data.

No APIs. No cloud services. Runs entirely offline after model download.

What TextStruct Does

  • Converts scanned PDFs into images
  • Runs layout-aware OCR locally (DocTR)
  • Intelligent text cleanup (headers/footers, OCR artifacts)
  • Section and subsection detection
  • Semantic chunking with embeddings
  • NEW: Pedagogical role classification (definitions, examples, procedures, etc.)
  • Outputs structured, RAG-ready JSON
  • Designed as a foundation for:
    • Educational knowledge extraction
    • Intent-aware RAG systems
    • Test generation
    • Study material creation

What It Does NOT Do (by design)

  • ❌ Chat UI
  • ❌ Cloud OCR
  • ❌ External APIs

Requirements

  • Docker
  • NVIDIA GPU + drivers (optional for acceleration)
  • nvidia-container-toolkit (for GPU support)
  • Python 3.9+ (if running without Docker)

Running with Docker (CPU by default)

TextStruct runs on CPU out of the box — no GPU required.

docker compose build
docker compose run --rm textstruct \
  python src/main.py --batch data/input --infer-chapters

GPU Support (Optional)

GPU acceleration is opt-in via an override file (requires the nvidia-container-toolkit on the host). Layer docker-compose.gpu.yml on top of the base compose file:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml run --rm textstruct \
  python src/main.py --batch data/input --infer-chapters --fast

Running Tests

Tests run from the repo root (pyproject.toml puts both the root and src/ on the path):

pip install -r requirements.txt -r requirements-dev.txt
pytest

Or inside Docker:

docker compose run --rm textstruct pytest

⚡ Fast Mode (GPU Recommended)

TextStruct supports GPU-batched OCR for significantly faster processing.

Requirements

  • NVIDIA GPU with ≥16 GB VRAM
  • CUDA-enabled Docker runtime

Usage

python src/main.py --batch data/input --infer-chapters --fast

Hardware Requirements & Performance

Minimum Requirements (CPU-Only Mode)

  • CPU: 4+ cores recommended
  • RAM: 4-8 GB minimum, 16 GB recommended for large PDFs
  • Disk: 2-5 GB for models + cache
  • Network: Optional (only for LLM features)

Hardware Stack Table

Pipeline Stage CPU GPU RAM Network CPU-Only? Optimization Available
PDF → Images ✓ ✗ 500MB-2GB ✗ ✅ Yes DPI reduction
OCR (default) ✓ ✗ 2-4GB ✗ ✅ Yes Model selection
OCR (--fast) partial ✓ 4-8GB ✗ ✅ Yes (fallback) GPU acceleration
Embeddings (auto) ✓ auto 2-4GB ✗ ✅ Yes GPU if available
Semantic Merge ✓ ✗ 500MB ✗ ✅ Yes Batch processing
Role Classification ✓ ✗ 500MB ✗ ✅ Yes 3-tier cascade
Role LLM Fallback ✗ ✗ 500MB ✓ ⚠️ Optional Requires Ollama
Math Cleanup ✓ ✗ 300MB ✗ ✅ Yes Regex patterns
Math LLM Cleanup ✗ ✗ 300MB ✓ ⚠️ Optional Requires Ollama
Vector Store ✓ ✗ 1-3GB ✗ ✅ Yes NumPy operations

Legend:

  • ✓ = Used
  • ✗ = Not used
  • auto = Auto-detected
  • partial = Light usage
  • ✅ Yes = Fully functional without GPU
  • ⚠️ Optional = Feature can be disabled

Performance Modes

Default Mode (CPU-Only, No Network)

python src/main.py input.pdf --refine --semantic-chunk
  • Speed: ~2-5 min per 100-page PDF
  • Hardware: CPU + 4GB RAM
  • Network: Not required
  • Accuracy: Good (heuristic role classification, regex math cleanup)

Fast Mode (GPU + CPU, No Network)

python src/main.py input.pdf --fast --refine --semantic-chunk
  • Speed: ~30-60 sec per 100-page PDF (3-5× faster OCR)
  • Hardware: GPU + 8GB RAM
  • Network: Not required
  • Accuracy: Same as default

Precision Mode (CPU + Network)

python src/main.py input.pdf --refine --semantic-chunk --classify-roles-llm --fix-math-ollama
  • Speed: ~5-15 min per 100-page PDF (LLM overhead)
  • Hardware: CPU + 4GB RAM
  • Network: Ollama required
  • Accuracy: Highest (LLM-based role classification and math cleanup)

Maximum Performance Mode (GPU + Network)

python src/main.py input.pdf --fast --refine --semantic-chunk --classify-roles-llm --fix-math-ollama
  • Speed: ~1-3 min per 100-page PDF
  • Hardware: GPU + 8GB RAM + Ollama
  • Network: Ollama required
  • Accuracy: Highest

Caching & Optimization

TextStruct implements aggressive caching to speed up repeated operations:

  1. Embedding Cache (.doctr_cache/embeddings/)

    • Stores all computed embeddings as hash-keyed .npy files
    • Cache hit = instant retrieval (no recomputation)
    • Shared across all pipeline runs

  2. Model Cache (~/.cache/huggingface/)

    • Sentence-transformers model (~420MB)
    • DocTR OCR models (~100-300MB)
    • Downloaded once, cached forever

  3. Ollama Model Cache

    • LLM model stays warm in Ollama container
    • First request: ~1-2s (model load)
    • Subsequent requests: ~50-200ms

  4. OCR Results (Future improvement)

    • Currently NOT cached
    • Re-OCR on every run (even if PDF unchanged)

Hardware Control Flags

Flag Effect Default Hardware Impact
--fast Enable GPU for OCR + faster models OFF GPU used if available, CPU fallback
--force-cpu Force CPU-only execution OFF Disables GPU even if available
--classify-roles-llm Use LLM for uncertain role classification OFF Network required (Ollama)
--fix-math-ollama Use LLM for complex math cleanup OFF Network required (Ollama)
--quiet Suppress info logs OFF No hardware impact
--verbose Show debug logs OFF No hardware impact

Known Performance Bottlenecks

Bottleneck Impact Workaround Future Fix
Sequential OCR 5-10 min for 100 pages Use --fast flag Batch OCR processing
Sequential LLM requests 1-3s per uncertain chunk Disable --classify-roles-llm Batch LLM API
Embedding recomputation 30-60s for 500 chunks Cache hits automatic Semantic deduplication
Linear vector search Slow for >10k chunks Use smaller PDFs ANN indexing (FAISS)

Recommended Configurations

For Development/Testing:

python src/main.py sample.pdf --fast --refine --semantic-chunk --quiet

For Production (Accuracy Priority):

python src/main.py input.pdf --refine --semantic-chunk --classify-roles-llm --build-index

For Production (Speed Priority):

python src/main.py input.pdf --fast --refine --semantic-chunk --build-index

For Batch Processing:

python src/main.py --batch data/input/ --fast --refine --semantic-chunk --quiet

📚 Pedagogical Role Classification (Phase 3.y)

TextStruct can classify chunks into 10 pedagogical roles for intent-aware RAG:

  • definition - Formal definitions
  • explanation - Why/how something works
  • procedure - Step-by-step instructions
  • example - Worked examples
  • question - Practice questions/exercises
  • result - Theorems, conclusions
  • reference - Tables, formulas, lookup data
  • evidence - Citations, data-backed claims
  • visual_proxy - Figure/diagram descriptions
  • context - Background, history, motivation

Fast Mode (Heuristics + Embeddings)

docker compose run --rm textstruct \
  python src/main.py data/input/textbook.pdf \
  --refine \
  --semantic-chunk \
  --classify-roles
  • ~10-15s for 100 chunks on CPU
  • 75-85% accuracy
  • Uses pattern matching + embedding similarity

High-Precision Mode (With LLM)

docker compose run --rm textstruct \
  python src/main.py data/input/textbook.pdf \
  --refine \
  --semantic-chunk \
  --classify-roles-llm
  • ~2-5 minutes for 100 chunks
  • 85-95% accuracy
  • Requires Ollama running on llm-net network
  • Uses LLM for uncertain cases only

Output Schema

{
  "id": "chunk_sem_00042",
  "text": "A prime number is...",
  "metadata": {
    "pedagogical_role": "definition",
    "role_confidence": 0.92,
    "semantic_coherence_score": 0.87,
    ...
  }
}

Pipeline Flags

Core Processing

  • --refine - Clean OCR artifacts and separate pedagogical callouts
  • --semantic-chunk - Merge similar chunks using embeddings (requires --refine)
  • --classify-roles - Fast role classification (requires --semantic-chunk)
  • --classify-roles-llm - High-precision LLM classification (requires Ollama)

Performance

  • --fast - GPU-accelerated OCR (requires ≥16GB VRAM)
  • --layout-aware - Handle multi-column layouts

Debugging

  • --debug-clean - Save before/after cleanup text
  • --debug-layout - Save annotated images showing bboxes, reading order, column boundaries (requires --layout-aware)

Example: Full Pipeline

docker compose run --rm textstruct \
  python src/main.py data/input/textbook.pdf \
  --layout-aware \
  --refine \
  --semantic-chunk \
  --classify-roles \
  --fast

🚀 LanceDB Vector Store (Phase 5.x)

TextStruct now supports LanceDB for high-performance vector storage and retrieval, replacing the legacy pickle-based in-memory store.

Why LanceDB?

  • 10-100x faster search on large collections (ANN indexing with IVF-PQ)
  • SQL-like filtering with WHERE clauses, range queries, OR logic
  • Parquet-backed persistence (no serialization overhead)
  • Scalable to millions of vectors with efficient disk-based storage
  • Backward compatible with existing pickle indexes

Installation

LanceDB support requires additional dependencies:

pip install lancedb>=0.3.0 pyarrow>=14.0.0

Or using Docker (already included in Dockerfile):

docker compose build

Building a LanceDB Index

python src/main.py input.pdf \
  --semantic-chunk \
  --build-index \
  --classify-roles

This creates data/output/<pdf_name>/vector_store.lancedb/ directory.

Querying with SQL Filters

LanceDB enables advanced filtering beyond simple key-value matching:

Range Queries

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "triangle theorems" \
  --sql-filter "page_start >= 10 AND page_start <= 20" \
  --top-k 5

Role + Confidence Filtering

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "area formulas" \
  --sql-filter "pedagogical_role = 'definition' AND role_confidence > 0.8"

OR Logic

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "worked examples" \
  --sql-filter "pedagogical_role = 'example' OR pedagogical_role = 'procedure'"

Interactive Mode with SQL

python src/query.py data/output/textbook/vector_store.lancedb --interactive

> triangle area sql:page_start >= 5 AND page_start <= 15
> examples sql:role_confidence > 0.85

Performance Comparison

Dataset Size Pickle (Linear) LanceDB (No Index) LanceDB (IVF-PQ)
40 chunks 1-2 ms 2-3 ms 3-5 ms
400 chunks 25-50 ms 30-60 ms 5-10 ms
4,000 chunks 250-500 ms 300-600 ms 15-30 ms
40,000 chunks 2.5-5 s 3-6 s 50-100 ms

Indexing time: ~5-15s for 1,000 chunks (one-time cost)

Migration from Pickle to LanceDB

Migrate existing pickle indexes:

python src/migrate_to_lancedb.py \
  --pickle data/output/textbook/vector_store.pkl \
  --lancedb data/output/textbook/vector_store.lancedb \
  --create-index \
  --verify

Batch migration:

python src/migrate_to_lancedb.py \
  --batch-dir data/output/ \
  --create-index

The migration tool:

  • Converts all metadata to LanceDB schema
  • Creates IVF-PQ index automatically
  • Verifies data integrity with random sampling
  • Preserves all semantic chunk metadata

Backward Compatibility

TextStruct maintains full backward compatibility:

  • Automatic fallback: If LanceDB not installed, uses pickle backend
  • Dual format support: Can read both .pkl and .lancedb stores
  • Query interface: All existing query code works unchanged
  • Migration path: Easy upgrade with verification

LanceDB Schema

LanceDB stores flattened metadata for SQL queries:

{
  "id": "chunk_sem_00001",
  "text": "A prime number is...",
  "vector": [0.123, -0.456, ...],  # 768-dim text embedding

  # Pedagogical metadata (flat for SQL)
  "pedagogical_role": "definition",
  "role_confidence": 0.92,
  "section": "1.2 Prime Numbers",
  "page_start": 15,
  "page_end": 15,

  # Vision metadata (Phase 5.v)
  "has_visual_content": true,
  "image_embedding": [0.789, -0.234, ...],  # 512-dim CLIP
  "image_path": "data/output/.../rois/roi_page0015_block0003_figure.png",
  "visual_type": "figure",

  # Full metadata as JSON (for complex nested structures)
  "metadata_json": "{...}"
}

CLI Flags

Flag Description Default
--build-index Build vector store after chunking OFF
--sql-filter <WHERE> SQL filtering (query mode only) None
--vector-backend lancedb|inmemory Choose backend explicitly lancedb

🎨 Vision-RAG Integration (Phase 5.v)

TextStruct can now extract visual content (figures, tables, diagrams) and enable hybrid text + vision retrieval using CLIP embeddings.

Why Vision-RAG?

  • Semantic image search: Find diagrams by text description
  • Hybrid retrieval: Combine text and visual similarity
  • Figure-aware chunking: Automatically link text to related visuals
  • Cross-modal queries: Text→image and image→image search

Installation

Vision-RAG requires additional dependencies:

pip install transformers>=4.30.0 open-clip-torch>=2.20.0

Or using Docker (already included):

docker compose build

Extracting Visual Content

Enable vision extraction during pipeline:

python src/main.py input.pdf \
  --layout-aware \
  --detect-tables \
  --detect-figures \
  --semantic-chunk \
  --build-index \
  --extract-visuals

What it does:

  1. Detects figure/table regions using Phase 4.x layout analysis
  2. Crops ROIs with 10px padding and saves as separate images
  3. Generates CLIP embeddings (512-dim) for each ROI
  4. Links ROIs to semantic chunks based on page proximity
  5. Stores both text (768-dim) and vision (512-dim) embeddings in LanceDB

Output structure:

data/output/<pdf_name>/
├── rois/
│   ├── roi_page0012_block0005_figure.png
│   ├── roi_page0015_block0002_table.png
│   └── ...
├── chunks_semantic.json  # Includes visual metadata
└── vector_store.lancedb/  # Hybrid text+vision index

Hybrid Search

Text-Only Query (Default)

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "triangle area formula"

Vision-Only Query

python src/query.py data/output/textbook/vector_store.lancedb \
  --query-image path/to/diagram.png

Hybrid Text + Vision Query

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "geometric proof" \
  --query-image sample_diagram.png \
  --text-weight 0.6 \
  --vision-weight 0.4

Fusion formula:

score = (text_weight × text_similarity) + (vision_weight × vision_similarity)

Chunk Schema with Visual Metadata

Chunks with linked visual content include:

{
  "id": "chunk_sem_00042",
  "text": "Figure 3.2 shows the relationship between...",
  "metadata": {
    "pedagogical_role": "visual_proxy",
    "has_visual_content": true,
    "image_path": "data/output/.../rois/roi_page0015_block0003_figure.png",
    "visual_type": "figure",
    "bbox": [120, 450, 480, 720],
    "page_start": 15,
    ...
  }
}

Vision Encoders

TextStruct supports two vision encoders:

CLIP (Default)

  • Model: OpenAI CLIP ViT-B/32
  • Embedding dim: 512
  • Use case: General figures, diagrams, photos
  • Speed: ~50-100ms per image (GPU), ~200-400ms (CPU)
  • Quality: Excellent for semantic visual search
python src/main.py input.pdf \
  --extract-visuals \
  --vision-encoder clip

ColPali (Experimental)

  • Specialized for: Dense documents, tables, multi-column layouts
  • Fallback: Uses CLIP if ColPali not available
  • Use case: Complex tables, multi-column text regions
python src/main.py input.pdf \
  --extract-visuals \
  --vision-encoder colpali

Performance

Vision Extraction:

  • ~50-100ms per ROI (GPU)
  • ~200-400ms per ROI (CPU)
  • Cached embeddings reused on subsequent runs

Hybrid Search:

  • Text-only: ~5-50ms (depends on index size)
  • Vision-only: ~10-80ms
  • Hybrid: ~15-100ms

Storage Overhead:

  • ~100-500 KB per extracted ROI image (PNG)
  • ~2 KB per vision embedding (.npy)
  • Example: 100-page textbook with 50 figures ≈ 25-50 MB

Vision Cache

Vision embeddings are cached to disk (same as text embeddings):

data/output/.doctr_cache/vision_embeddings/
├── cache_manifest.json
├── <hash1>.npy  # CLIP embedding for image 1
├── <hash2>.npy
└── ...

Cache hits: Instant retrieval (no recomputation)

CLI Flags

Flag Description Default
--extract-visuals Extract figure/table ROIs and generate embeddings OFF
--vision-encoder clip|colpali Choose vision encoder clip
--query-image <path> Image query for retrieval None
--text-weight <0-1> Text similarity weight in hybrid search 0.7
--vision-weight <0-1> Vision similarity weight in hybrid search 0.3

Requirements

  • Minimum: --layout-aware --semantic-chunk must be enabled
  • Recommended: --detect-tables --detect-figures for better ROI detection
  • GPU: Optional but recommended for faster vision encoding

Example: Full Vision-RAG Pipeline

python src/main.py data/input/geometry_textbook.pdf \
  --layout-aware \
  --detect-tables \
  --detect-figures \
  --refine \
  --semantic-chunk \
  --classify-roles \
  --build-index \
  --extract-visuals \
  --fast

Output:

  • Structured semantic chunks with pedagogical roles
  • LanceDB vector store with text + vision embeddings
  • Extracted ROI images in rois/ directory
  • Hybrid search support for text and image queries

🔍 Advanced Querying Examples

Example 1: Find High-Confidence Definitions on Specific Pages

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "prime factorization" \
  --sql-filter "pedagogical_role = 'definition' AND role_confidence > 0.85 AND page_start >= 20 AND page_start <= 40" \
  --top-k 3

Example 2: Find Examples or Procedures with Visual Content

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "area calculation" \
  --sql-filter "(pedagogical_role = 'example' OR pedagogical_role = 'procedure') AND has_visual_content = true"

Example 3: Hybrid Search for Geometric Proofs

python src/query.py data/output/textbook/vector_store.lancedb \
  --query "triangle congruence proof" \
  --query-image reference_diagram.png \
  --text-weight 0.5 \
  --vision-weight 0.5 \
  --top-k 5

Example 4: Interactive Exploration

python src/query.py data/output/textbook/vector_store.lancedb --interactive

> prime numbers
> definitions sql:role_confidence > 0.9
> examples sql:page_start >= 10 AND page_start <= 20
> exit

📊 Output Structure

Complete output directory structure with all features enabled:

data/output/<pdf_name>/
├── images/
│   ├── page_0001.png
│   ├── page_0002.png
│   └── ...
├── rois/  # Phase 5.v visual content
│   ├── roi_page0012_block0005_figure.png
│   ├── roi_page0015_block0002_table.png
│   └── ...
├── chunks_raw.json  # Raw OCR blocks
├── chunks_clean.json  # After cleanup
├── chunks_semantic.json  # After semantic merging + visual linking
├── vector_store.lancedb/  # LanceDB index
│   ├── data.lance
│   ├── _versions/
│   └── ...
└── .doctr_cache/
    ├── embeddings/  # Text embeddings
    │   ├── cache_manifest.json
    │   └── *.npy
    └── vision_embeddings/  # Vision embeddings
        ├── cache_manifest.json
        └── *.npy

🚦 Complete Pipeline Flags Reference

Core Processing

  • --refine - Clean OCR artifacts, separate pedagogical callouts
  • --semantic-chunk - Merge similar chunks using embeddings
  • --classify-roles - Fast role classification (heuristics + embeddings)
  • --classify-roles-llm - High-precision LLM classification (requires Ollama)
  • --build-index - Build LanceDB vector store (enables retrieval)

Layout & Vision

  • --layout-aware - Handle multi-column layouts, reading order
  • --detect-tables - Detect table regions
  • --detect-figures - Detect figure/diagram regions
  • --extract-visuals - Extract ROIs and generate vision embeddings (Phase 5.v)
  • --vision-encoder clip|colpali - Choose vision encoder (default: clip)

Performance

  • --fast - GPU-accelerated OCR (requires ≥16GB VRAM)
  • --force-cpu - Force CPU-only execution

Querying

  • --query <text> - Text query
  • --query-image <path> - Image query (vision-only or hybrid)
  • --sql-filter <WHERE> - SQL filtering (LanceDB only)
  • --text-weight <0-1> - Text weight in hybrid search (default: 0.7)
  • --vision-weight <0-1> - Vision weight in hybrid search (default: 0.3)
  • --top-k <N> - Number of results (default: 5)
  • --interactive - Interactive query mode

Debugging

  • --debug-clean - Save before/after cleanup text
  • --debug-layout - Save annotated layout images
  • --quiet - Suppress info logs
  • --verbose - Show debug logs

🎯 Recommended Workflows

Workflow 1: Maximum Accuracy (Research/Study Material Creation)

python src/main.py input.pdf \
  --layout-aware \
  --detect-tables \
  --detect-figures \
  --refine \
  --semantic-chunk \
  --classify-roles-llm \
  --build-index \
  --extract-visuals

Use case: Extracting high-quality structured data from textbooks for RAG systems, test generation, study guides

Workflow 2: Maximum Speed (Batch Processing)

python src/main.py --batch data/input/ \
  --fast \
  --refine \
  --semantic-chunk \
  --classify-roles \
  --build-index \
  --quiet

Use case: Processing large document collections quickly

Workflow 3: Vision-Focused (Diagram-Heavy Documents)

python src/main.py input.pdf \
  --layout-aware \
  --detect-figures \
  --semantic-chunk \
  --build-index \
  --extract-visuals \
  --vision-encoder clip

Use case: Math, physics, engineering textbooks with heavy visual content

Workflow 4: Lightweight (CPU-Only, No LLM)

python src/main.py input.pdf \
  --refine \
  --semantic-chunk \
  --force-cpu \
  --quiet

Use case: Quick local processing without GPU or network dependencies

📖 Documentation

  • Phase 3.y: Pedagogical role classification with LLM fallback
  • Phase 4.x: Layout-aware OCR, table/figure detection
  • Phase 5.x: LanceDB vector store with SQL filtering
  • Phase 5.v: Vision-RAG integration with CLIP/ColPali
  • Future (Phase 5.x+): Pedagogical bundling (definitions + examples)

See ToDo.md for detailed implementation notes and roadmap

About

A local-first engine for turning scanned textbooks into structured RAG databases. Features layout-aware OCR, deterministic math cleanup, and pedagogical role-indexing using LanceDB and CLIP vision.

Topics

python pdf ocr embeddings clip semantic-search layout-analysis rag doctr local-first vector-database document-ai retrieval-augmented-generation lancedb vision-rag

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages