RAG, no provenance

[sroussey]

• Rag (Rag #170)

• [feat] New VectorQuantizeTask, updated VectorSimilarityTask

• [WIP] rework document

• [refactor] Update task input handling and smartClone method for improved input data handling for tests

• Replaced structuredClone and JSON methods with a new smartClone function that deep-clones plain objects and arrays while preserving class instances by reference.

• quick versions of tasks as functions now pass input to run not the constructor which means no defaults and cloning

• [refactor] Removed unnecessary checks for undefined values when copying additional input properties.

• [refactor] Enhance tasks with service registry integration

• Updated IExecuteContext and IRunConfig to include registry support.
• Refactored TaskRunner and TaskGraphRunner to utilize the service registry for improved task execution and model retrieval.
• Ensured backward compatibility while enhancing the overall architecture for better service management.
• Introduced a service registry to manage model repositories and execution contexts in AiTask.

• [feat] Introduce input resolver system for enhanced schema handling

• Added a new InputResolver to manage schema-annotated inputs, allowing for automatic resolution of string IDs to their corresponding instances.
• Implemented repository and model resolution capabilities, improving task input handling and validation.
• Created new schemas for tabular, vector, and document repositories to facilitate input resolution.
• Enhanced AiTask and TaskRunner to utilize the input resolver for better integration with service registries.
• Added comprehensive tests to ensure the functionality of the input resolver system and its integration with tasks.

• [feat] Introduce new AI tasks and enhance document processing capabilities

• Added several new tasks including ChunkToVectorTask, ContextBuilderTask, DocumentEnricherTask, HierarchicalChunkerTask, and others to support advanced document processing workflows.
• Enhanced the input handling for tasks to streamline the integration with the service registry and improve task execution.
• Updated the documentation to reflect the new tasks and their functionalities, ensuring clarity for developers.
• Implemented comprehensive tests for the new tasks to validate their behavior and integration within the workflow system.

• Update packages/ai/src/source/ProvenanceUtils.ts

• Update packages/ai/src/source/StructuralParser.ts

• Update packages/ai/src/source/ProvenanceUtils.ts

• Update packages/ai/src/task/ChunkToVectorTask.ts

• Update packages/ai/src/task/QueryExpanderTask.ts

• Update packages/task-graph/src/task/Task.ts

• Update packages/util/src/vector/Tensor.ts

• Initial plan

• Update packages/ai/src/task/VectorQuantizeTask.ts

• Update packages/util/src/vector/VectorUtils.ts

• Optimize quantizeToUint8 and quantizeToUint16 with single-pass min/max

• Remove unused query variable from InputResolver test (Remove unused query variable from InputResolver test #161)

• Initial plan

• Remove unused query variable from InputResolver test

---

• Fix edge case: return non-zero range for empty arrays in findMinMax

• Fix markdown auto-detection to use header pattern matching (Fix markdown auto-detection to use header pattern matching #157)

• Initial plan

• Improve markdown auto-detection with robust pattern matching

---

• Remove unused task class imports from ChunkToVector.test.ts (Remove unused task class imports from ChunkToVector.test.ts #160)

• Initial plan

• Remove unused imports ChunkToVectorTask and HierarchicalChunkerTask

---

• Fix inconsistent vector/tensor terminology in Tensor.ts (Fix inconsistent vector/tensor terminology in Tensor.ts #167)

• Initial plan

• Update Tensor.ts to use consistent "tensor" terminology throughout

---

• Optimize VectorQuantizeTask min/max calculation for large vectors (Optimize VectorQuantizeTask min/max calculation for large vectors #168)

• Initial plan

• Optimize quantizeToUint8 and quantizeToUint16 to use single loop for min/max

• Add empty array guard to quantization methods

---

• Replace unsafe type assertions with type-safe field extraction in Document.addVariant (Replace unsafe type assertions with type-safe field extraction in Document.addVariant #158)

• Initial plan

• Use extractConfigFields for type-safe provenance handling

• Add comprehensive tests for type-safe provenance handling

---

• Add test coverage for VectorSimilarityUtils functions (Add test coverage for VectorSimilarityUtils functions #165)

• Initial plan

• Add comprehensive tests for VectorSimilarityUtils

---

• Extract magic number to named constant in ProvenanceUtils (Extract magic number to named constant in ProvenanceUtils #159)

• Initial plan

• Extract magic number 512 to DEFAULT_MAX_TOKENS constant

---

• Add support for Float16Array in normalize function of VectorUtils.ts

• Update packages/ai/src/source/StructuralParser.ts

• Fix naming inconsistency between Vector type and TensorSchema (Fix naming inconsistency between Vector type and TensorSchema #169)

• Initial plan

• Fix naming inconsistency: rename Vector to Tensor in Tensor.ts

---

• Enhance jaccardSimilarity function to handle negative values by normalizing inputs to a non-negative range. This includes calculating the global minimum across both vectors and adjusting values accordingly.

• [test] Add tests for VectorUtils, covering magnitude, inner product, normalization, and handling of various TypedArray types. Update normalize function to support an additional parameter for Float32Array conversion.

• Add circular reference detection to smartClone method (Add circular reference detection to smartClone method #162)

• Initial plan

• Add circular reference detection to smartClone method

• Fix circular reference detection to handle shared references correctly

• Refactor TaskEvents to import TaskStatus from TaskTypes and add unit tests for smartClone method

• Updated TaskEvents to import TaskStatus from the correct module.
• Added comprehensive unit tests for the smartClone method, including cases for circular reference detection and handling various data structures.

---

---

• Enhance StructuralParser to include title in document nodes

• Enhance DocumentSchema to include title field and update required properties

• [refactor] Document and InputResolver modules

• Removed re-export of schemas and types from Document.ts for cleaner module structure.
• Enhanced AiTask's getDefaultQueueName method to handle single model inputs and throw an error for multiple models.
• Cleaned up InputResolver by removing unnecessary re-exports, streamlining the module for better clarity.
• Added comprehensive tests for Document functionality, ensuring robust handling of variants and provenance.

• Refactor Document class to use optional chaining and nullish coalescing in getChunks method for improved safety. Update README to clarify vector metadata structure and enrich metadata fields for hierarchical documents.

• Refactor DocumentEnricherTask to utilize ModelConfig for summary and NER model parameters, enhancing type safety and clarity in method signatures.

• Refactor provenance handling to support array structure

• Updated the Provenance type to be an array of ProvenanceItem, allowing for multiple provenance entries.
• Modified extractConfigFields and related functions to handle provenance as an array, enhancing type safety and flexibility.
• Adjusted Document and task classes to utilize the new provenance structure, ensuring consistent handling across the codebase.
• Updated tests to reflect changes in provenance structure and validate functionality.

• Refactor ModelRegistry and InputResolver for improved type handling of model arrays

• Updated setGlobalModelRepository parameter name for clarity.
• Enhanced resolveModelFromRegistry to support both single and array of model IDs.
• Modified resolveSchemaInputs to handle string values and arrays of strings more effectively, ensuring proper resolution of inputs.

• [refactor] Remove ArrayTask from between JobQueueTask and Task. Refactor AI task schemas to simplify model handling

• Updated various AI task schemas to replace array-based model definitions with single model references, enhancing clarity and type safety.
• Adjusted input schemas for tasks such as BackgroundRemovalTask, ImageClassificationTask, and others to reflect these changes.
• Removed unnecessary type handling for model arrays in AiTask and AiVisionTask classes, streamlining the codebase.
• Enhanced the GraphAsTask and JobQueueTask classes to support the new model structure, ensuring compatibility across the task framework.

• [refator] Remove Provenance from task and task graph

• Removed the Provenance type and related handling from various classes, including Task, TaskRunner, and Dataflow, to streamline the codebase.
• Updated Document and HierarchicalChunkerTask to directly use VariantProvenance, enhancing clarity and type safety.
• Adjusted method signatures and removed unused provenance-related methods across the task graph framework.
• Updated tests to reflect changes in provenance structure and validate functionality.

• [refactor] Simplify Document handling by removing Provenance and variants

• Removed Provenance-related functionality from the Document class, including the handling of variants and associated methods.
• Updated Document methods to manage chunks directly, enhancing clarity and reducing complexity.
• Adjusted related schemas and tests to reflect the removal of Provenance and the shift to a chunk-based structure.
• Ensured compatibility across the codebase by updating references and method signatures accordingly.

---