Clickhouse integrtion to Ion#3
Open
neerajvipparla wants to merge 16 commits into
Open
Conversation
…ntation This commit introduces a comprehensive design and implementation plan for a new ClickHouse log sink within the Ion observability library. The document outlines the context, architecture, package structure, and component specifications for integrating ClickHouse as a logging sink, enabling efficient analytical queries over log data. The new package will be built independently and will not affect existing functionality until validated.
This commit adds several indirect dependencies to the Go module, including updates for ClickHouse libraries, Brotli compression, and various utility packages. The changes enhance the project's capabilities and ensure compatibility with the latest versions of these libraries.
This commit introduces a new configuration package for the ClickHouse log sink, including a Config struct that holds connection details, default values, and validation methods. The package also includes a Ping function to check the connectivity to ClickHouse. Comprehensive tests have been added to ensure the correctness of configuration validation and default value application.
…ouse This commit introduces a new schema management system for ClickHouse, including the implementation of DDL generation and schema validation. The `buildDDL` function constructs the SQL create table statement based on a provided table name, while the `EnsureSchema` function ensures the schema is created if it does not exist. Additionally, comprehensive unit tests have been added to validate the DDL generation and schema enforcement logic.
This commit introduces a new batch writing mechanism for ClickHouse logs, allowing for efficient handling of log rows. The `batchWriter` struct manages a buffered channel for log rows, with configurable batch size and flush intervals. It includes methods for sending log rows, flushing batches, and tracking dropped rows. Additionally, unit tests have been added to ensure the functionality of the batch writing process.
…lity This commit introduces a comprehensive suite of unit tests for the `batchWriter` and `Core` components of the ClickHouse logging implementation. The tests cover various scenarios including batch flushing, error handling, and synchronization, ensuring robust functionality. Additionally, the `Core` tests validate the zapcore interface and configuration handling, enhancing overall reliability and correctness of the logging system.
…ation This commit updates the `logRow` struct to include ClickHouse-specific tags for each field, improving schema mapping. Additionally, it introduces a regex validation for table identifiers in the configuration, ensuring that invalid table names trigger a panic during initialization. The validation logic is expanded to include checks for various configuration parameters, enforcing stricter rules on their values. New error messages are added to provide clearer feedback on configuration issues.
This commit adds new unit tests for the `extractRow` function, covering scenarios for large and small uint64 values, time fields, and complex numbers, ensuring correct handling and storage in the appropriate fields. Additionally, it introduces a new configuration builder with setter methods for various parameters, improving the configuration management process. The validation tests have been updated to ensure stricter checks on configuration values, enhancing overall robustness.
This commit updates the ClickHouse configuration to include an `Enabled` field, allowing for conditional activation of the log sink. It also improves validation logic to ensure that the DSN is provided when the ClickHouse sink is enabled. Additionally, new unit tests are added for the `extractRow` function to handle byte strings and stringer types, ensuring accurate logging of these fields. The batch writing tests have been refactored for clarity and consistency.
This commit introduces the ClickHouse logging functionality into the Ion framework. It adds a new `chCore` field to the `Ion` struct for managing ClickHouse interactions and updates the logger setup to include ClickHouse-specific logging capabilities. A new `DroppedCount` method is implemented to track log entries dropped by the ClickHouse sink. Additionally, integration tests are added to validate the logging behavior and ensure correct data handling in ClickHouse. Configuration validation is enhanced to support ClickHouse logging levels.
This commit refactors the ClickHouse logging implementation by moving the ClickHouse configuration package to an internal directory. It updates import paths accordingly to reflect the new structure. Additionally, new test files for batch writing and core functionality are introduced, enhancing the test coverage for the ClickHouse integration. This change aims to improve the organization and maintainability of the codebase.
…ouse configuration This commit removes the deprecated 'warning' log level from the valid levels in the ClickHouse configuration. Additionally, it introduces default values for the ClickHouse configuration parameters, ensuring a more robust setup for the logging functionality. The changes aim to streamline the configuration process and improve overall clarity in the logging levels supported.
Add/clickhouse
…tion This commit updates the README.md to include comprehensive documentation for the ClickHouse configuration options. It details the parameters available for the ClickHouse log sink, including their types, defaults, and descriptions. Additionally, it provides the SQL schema for the ClickHouse table and guidelines for monitoring back-pressure, enhancing the clarity and usability of the documentation for users integrating ClickHouse logging.
docs(readme): add ClickHouse configuration details and schema informa…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.