Merge Feature/v0.7.4p2 into main

CHANGELOG.md

@@ -1,3 +1,162 @@
+# **v0.7.4 — Advanced Directory Parsing & Metadata Expansion**
+
+IOCX v0.7.4 significantly expands static PE coverage with advanced directory parsing, extended metadata extraction, and deterministic structural validation. This release improves correctness across modern compiler outputs while preserving IOCX’s static‑only, zero execution design.
+
+---
+
+## **Added**
+
+### **New RVA‑Graph Invariants**
+- **DATA_DIRECTORY_ZERO_SIZE_NONZERO_RVA**
+  Detects directories that simultaneously signal presence (non‑zero RVA) and absence (zero size).
+  Implemented with primary‑error semantics to suppress downstream mapping noise.
+
+- **DATA_DIRECTORY_RAW_MISMATCH**
+  Flags directories whose RVA maps into a section’s virtual range but whose computed raw offset lies outside the section’s raw data.
+  Includes a dedicated reason code and validator‑level consistency check.
+
+- **Raw‑mapping safety guard**
+  Prevents invalid raw‑offset calculations when sections contain no raw data.
+
+### **New Adversarial Fixtures for Directory Invariants**
+- `directory_zero_size_nonzero_rva.full.exe`
+- `directory_raw_mismatch.full.exe`
+
+### **Full Load Config Directory Parsing**
+- GuardCF metadata
+- Security cookie
+- SEH table
+- Compiler‑specific layout hints
+- Deterministic error handling for malformed structures
+
+### **Load Config Adversarial Fixtures**
+- `load_config_cookie_too_small.full.exe`
+- `load_config_malformed_size_too_small.full.exe`
+- `load_config_malformed_truncated.full.exe`
+- `load_config_malformed_cookie_in_overlay.full.exe`
+- `load_config_malformed_cookie_invalid.full.exe`
+- `load_config_malformed_guard_cf_inconsistent.full.exe`
+- `load_config_malformed_seh_invalid.full.exe`
+- `load_config_malformed_size_exceeds_section.full.exe`
+
+---
+
+## **99 Adversarial PE Fixtures for Structural & Parser‑Behaviour Testing**
+
+### **Entrypoint Fixtures (000–009)**
+Covers malformed `AddressOfEntryPoint` conditions:
+- Zero/negative EP
+- EP inside headers
+- EP outside `SizeOfImage`
+- EP unmapped to any section
+- EP in non‑executable section
+- EP spanning boundaries
+- EP in overlay
+
+**Outcome:** Entrypoint validator stable and deterministic across all malformed cases.
+
+---
+
+### **Section Table Fixtures (010–021)**
+Covers structural correctness of section headers and RVA/raw mappings:
+- Out‑of‑bounds RVA
+- Out‑of‑bounds raw offset
+- Overlapping sections
+- Unsorted sections
+- `VirtualSize < RawSize`
+- Misaligned boundaries
+- Section extends past `SizeOfImage`
+- Section mapped inside headers
+
+**Outcome:** All anomalies correctly identified; no false positives on valid baselines.
+
+---
+
+### **Optional Header Fixtures (022–033)**
+Covers correctness of Optional Header fields:
+- Invalid `SizeOfImage` / `SizeOfHeaders`
+- Invalid `FileAlignment` / `SectionAlignment`
+- Magic mismatch (PE32 vs PE32+)
+- Invalid subsystem / version fields
+- ImageBase misalignment
+- `NumberOfRvaAndSizes` too small
+
+**Outcome:** Optional‑header validator behaves consistently; malformed fields reliably detected.
+
+---
+
+### **Data Directory Fixtures (034–045)**
+Covers adversarial manipulations of the Data Directory Table:
+- Negative RVA / size
+- Zero/zero directory (valid)
+- Zero RVA with non‑zero size
+- Zero size with non‑zero RVA
+- Directory inside headers
+- Directory out of `SizeOfImage`
+- Directory in overlay
+- Unmapped directory
+- Directory spanning sections
+- Overlapping directories
+
+**Outcome:**
+All malformed cases correctly trigger the **primary structural anomaly**
+`optional_header_invalid_number_of_rva_and_sizes`.
+Fixture 036 (zero/zero) produces no anomalies, confirming non‑aggressive behaviour.
+
+---
+
+### **Overall Result for Fixtures 000–045**
+- **All 46 fixtures validated**
+- **No crashes or inconsistent behaviour**
+- **All anomalies match intended design**
+- Entrypoint, section, optional‑header, and directory validators confirmed stable
+
+---
+
+## **Comprehensive Layer‑2 Load Config Fixtures**
+
+A full suite of Load Config edge‑case binaries validating compiler differences, malformed structures, and ambiguous layouts:
+
+- **Minimal MinGW Load Config** (undersized structure detection)
+- **Cookie‑Only (Valid)** (minimum‑size compliance, RVA mapping, section writability)
+- **Cookie‑Only (Too Small)** (strict minimum‑size enforcement)
+- **Full MSVC Load Config** (SEH, GuardCF, cookie, full‑path validation)
+- **Full Clang/LLVM Load Config** (GuardCF without SEH)
+- **Large Padded Load Config** (oversized, schema‑unknown layouts)
+- **SEH‑Only Load Config** (partial‑structure handling)
+
+**Outcome:**
+Validates RVA/VA correctness, section‑mapping rules, minimum‑size enforcement, GuardCF consistency, SEH bounds checking, and compiler‑specific structural differences.
+
+---
+
+## **Changed**
+
+- Load Config validator surfaced new anomalies in contract tests:
+  - Crypto Entropy Payload
+  - Franken URL Domain IP
+  - Malformed Domain / IP / URL
+  - String Obfuscation Tricks
+  - Invalid Optional Header (PE32 / PE32+)
+
+- Internal schema now includes:
+  - `number_of_rva_and_sizes`
+  - `data_directories_raw`
+  Supporting adversarial optional‑header edge cases.
+
+- Optional‑header validator:
+  - Now checks declared vs raw directory counts
+  - FixtureSpec and emitter updated to support adversarial `NumberOfRvaAndSizes` mismatches
+  - Raw vs declared count logic now fully enforced
+
+---
+
+## **Documentation**
+- Updated RVA / Directory Anomalies table with new reason codes and behavioural notes
+- Added **Design Decision: Why Only the Optional‑Header Validator Uses Raw Data Directories**
+
+---
+
 # v0.7.3 — Structural Correctness & Deterministic Heuristics
 **Released: 2026‑05‑11**
 

README-pypi.md

@@ -1,8 +1,10 @@
 # **IOCX — Deterministic, Zero‑Risk IOC Extraction for Modern Security Pipelines**
 ### Official IOCX Project
 
-**IOCX** is a high‑performance, deterministic static analysis engine for extracting Indicators of Compromise (IOCs) from binaries and text.
-It exists for one reason: **to provide a fast, safe, predictable IOC extractor that DFIR teams and automation pipelines can trust.**
+**IOCX** is a deterministic, high‑performance static analysis engine for extracting high-signal Indicators of Compromise (IOCs) from binaries, text, and logs.
+It’s built for DFIR teams, SOC automation, CI/CD pipelines, and large‑scale threat‑intel ingestion.
+
+**Why it matters:** IOCX guarantees snapshot‑stable output, zero‑risk static analysis, and predictable performance even under adversarial input — something regex‑only extractors simply can’t provide.
 
 - **PyPI:** [https://pypi.org/project/iocx/](https://pypi.org/project/iocx/)
 - **GitHub:** [https://github.com/iocx-dev/iocx](https://github.com/iocx-dev/iocx)
@@ -38,15 +40,14 @@ If you need predictable, automatable IOC extraction — IOCX is built for you.
 
 ---
 
-## Version highlights (v0.7.3)
+## Version highlights (v0.7.4)
 
-- Major hardening of all PE structural validators
-- Deterministic, snapshot‑stable output across malformed binaries
-- Stronger section, entrypoint, RVA‑graph, TLS, and signature checks
-- Corrected RVA→file‑offset mapping for overlay detection
-- Improved entropy analysis with clearer, conservative signals
-- Cleaner, consistent `ReasonCodes` across the engine
-- Expanded structural + heuristic test coverage
+- Full **Load Config Directory** parsing and validation
+- Extended Optional Header metadata for downstream heuristics
+- Structural anomaly heuristics (GuardCF, unmapped cookie, SEH issues)
+- Faster, more resilient PE Analysis
+- Raw IOC extraction remains world-class
+- Zero regressions across all workloads
 
 ---
 

README.md

@@ -9,7 +9,7 @@
 
 <p align="center">
   <a href="https://pypi.org/project/iocx/"><img src="https://img.shields.io/pypi/v/iocx?logo=pypi&logoColor=white"></a>
-  <img src="https://img.shields.io/badge/tests-851_passed-brightgreen">
+  <img src="https://img.shields.io/badge/tests-947_passed-brightgreen">
   <img src="https://img.shields.io/badge/coverage-100%25-brightgreen">
   <img src="https://img.shields.io/badge/python-3.12-blue">
   <a href="https://github.com/iocx-dev/iocx/actions"><img src="https://img.shields.io/github/actions/workflow/status/iocx-dev/iocx/ci.yml?label=build"></a>
@@ -166,31 +166,31 @@ Predictable even under worst‑case adversarial load.
 **150–300 MB/s** sustained throughput
 Fast path — no PE parsing.
 
-| Detector | 1 MB Time | Throughput |
-|----------|-----------|------------|
-| Crypto | 0.0037 s | ~270 MB/s |
-| Filepaths | 0.0040 s | ~250 MB/s |
-| IP | 0.0064 s | ~156 MB/s |
-| Domains | 0.0033 s | ~300 MB/s |
+| Detector  | 1 MB Time | Throughput |
+|-----------|-----------|------------|
+| Crypto    | 0.0037 s  | ~270 MB/s  |
+| Filepaths | 0.0041 s  | ~250 MB/s  |
+| IP        | 0.0065 s  | ~156 MB/s  |
+| Domains   | 0.0035 s  | ~300 MB/s  |
 
 ---
 
 ### **2. Typical PE Files (~39 KB)**
-- **0.0132 s** (typical)
-- **0.0153 s** (with heuristics)
+- **0.0122 s** (typical)
+- **0.0145 s** (with heuristics)
 - **6–15 MB/s** throughput
 
 ---
 
 ### **3. Adversarial Dense PE (1.5 MB)**
-- **0.1977 s**
+- **0.192 s**
 - **~7.6 MB/s** throughput
 - Triggers TLS anomalies, structural anomalies, anti‑debug patterns
 
 ---
 
 ### **4. Full Engine (Non‑PE)**
-- **1 MB:** 0.0411 s
+- **1 MB:** 0.038 s
 
 ---
 
@@ -200,6 +200,13 @@ Fast path — no PE parsing.
 <summary><strong>Show Version History</strong></summary>
 <br>
 
+### **v0.7.4 — Advanced Directory Parsing**
+- Full **Load Config Directory** parsing and validation
+- Extended Optional Header metadata for downstream heuristics
+- New GuardCF, cookie, anomaly heuristics
+- Faster PE Analysis
+- 99 PE fixtures in test suite; 45 fully spec-validated
+
 ### **v0.7.3 — Structural Correctness & Deterministic Heuristics**
 - Major hardening of all PE structural validators
 - Deterministic, snapshot‑stable behaviour

docs/performance.md

@@ -226,3 +226,99 @@ IOCX is designed to be:
 - **Fast on malformed inputs**
 
 Performance is a **core contract**, not an optimisation.
+
+---
+
+# **IOCX Performance Delta (v0.7.1 → Current Release)**
+
+This table shows how performance has changed since **v0.7.1**, across all major workloads: raw IOC extraction, PE analysis, adversarial samples, and full‑engine processing.
+
+### **Legend**
+- **↑ Faster** (improvement)
+- **→ Same** (no meaningful change)
+- **↓ Slower** (regression)
+
+---
+
+## **1. Raw IOC Extraction (Text / Logs / Buffers)**
+
+Throughput remains extremely high (150–300 MB/s). Variations are within noise.
+
+| Detector        | v0.7.1   | v0.7.4   | Delta         | Verdict      |
+|-----------------|----------|----------|---------------|--------------|
+| Crypto (1MB)    | 0.0037 s | 0.0037 s | 0             | → Same       |
+| Domains (1MB)   | 0.0033 s | 0.0035 s | +0.0002 s     | → Same       |
+| Filepaths (1MB) | 0.0040 s | 0.0041 s | +0.0001 s     | → Same       |
+| IP (1MB)        | 0.0064 s | 0.0065 s | +0.0001 s     | → Same       |
+
+**Summary:** Raw IOC extraction remains at peak speed with no regressions.
+
+---
+
+## **2. Typical PE Files (~39 KB)**
+
+| Case                    | v0.7.1   | v0.7.4       | Delta         | Verdict      |
+|-------------------------|----------|--------------|---------------|--------------|
+| Typical PE              | 0.0132 s | **0.0122 s** | **–0.0010 s** | **↑ Faster** |
+| Typical PE + heuristics | 0.0153 s | **0.0145 s** | **–0.0008 s** | **↑ Faster** |
+
+**Summary:** Clear improvements despite additional validators and structural checks.
+
+---
+
+## **3. Dense / Adversarial PE (1.5 MB)**
+
+| Case     | v0.7.1   | v0.7.4       | Delta         | Verdict      |
+|----------|----------|--------------|---------------|--------------|
+| Dense PE | 0.1977 s | **0.1921 s** | **–0.0056 s** | **↑ Faster** |
+
+**Summary:** Dense PE analysis continues to get faster — a ~3% improvement.
+
+
+---
+
+## **4. Franken PE**
+
+| Case       | v0.7.1   | v0.7.4       | Delta         | Verdict      |
+|------------|----------|--------------|---------------|--------------|
+| Franken PE | 0.0020 s | **0.0014 s** | **–0.0006 s** | **↑ Faster** |
+
+**Summary:** A substantial improvement (~30%). Franken PEs are now effectively “free.”
+
+---
+
+## **5. Full Engine (Non‑PE)**
+
+| Case       | v0.7.1   | v0.7.4       | Delta         | Verdict      |
+|------------|----------|--------------|---------------|--------------|
+| 1MB buffer | 0.0411 s | **0.0387 s** | **–0.0024 s** | **↑ Faster** |
+
+**Summary:** End‑to‑end throughput improved by ~6%.
+
+---
+
+## **6. Pathological / Adversarial Inputs**
+
+| Case              | v0.7.1   | v0.7.4   | Delta   | Verdict |
+|-------------------|----------|----------|---------|---------|
+| ETH‑like blob     | 0.0012 s | 0.0012 s | 0       | → Same  |
+| Punycode blob     | 0.0126 s | 0.0125 s | –0.0001 | → Same  |
+| Deep UNIX path    | 0.0246 s | 0.0250 s | +0.0004 | → Same  |
+| IPv6 pathological | 0.0004 s | 0.0004 s | 0       | → Same  |
+
+**Summary:** Identical performance — validators do not impact non‑PE workloads.
+
+---
+
+# **Overall Summary**
+
+- **Zero regressions across all workloads**
+- **PE analysis is consistently faster**
+- **Dense and Franken PEs show the largest gains**
+- **Full‑engine throughput improved**
+- **Raw IOC extraction remains at peak speed**
+- **Adversarial inputs remain unaffected by new validators**
+
+IOCX continues to get **faster**, even as the engine becomes more robust, more defensive, and more standards‑compliant.
+
+---