Improve NULL handling for NOT NULL integer columns #65

[monozoide]

Fix: Prevent NULL values for NOT NULL integer columns in SQL export

🐞 Bug Summary

The SQL import process was failing with a NOT NULL constraint error on the asn_int column in the maillogsentinel_events table. The root cause was the SQL exporter generating NULL values for empty strings in CSV data, which were then passed to NOT NULL integer columns during import, causing constraint violations.

Severity: high

🔁 Reproduction

1. Prepare a CSV export with an empty string value in an ASN field
2. Export the data to SQL format using the SQL exporter
3. Attempt to import the generated SQL file into the database
4. The import fails with a NOT NULL constraint violation on the asn_int column

Actual result:
The SQL exporter generates NULL for empty string values, which violates the NOT NULL constraint and creates an invalid SQL file that cannot be imported.

Expected result:
The export process should abort with a clear error when encountering invalid data for a NOT NULL column, preventing the generation of an invalid SQL file.

🌍 Runtime Context

• Affected version(s): 5.15.x
• Environment(s): Production
• OS: Debian 13.1 - Trixie
• Python version: 3.13.x
• Relevant file: /usr/local/bin/lib/maillogsentinel/sql_importer.py (line 467)
• Affected module: sql_exporter.py (SQL export logic)

🕵️ Analysis & Root Cause

The issue originates in the format_sql_value function in sql_exporter.py. The function was converting empty strings to NULL values without checking the column's NOT NULL constraint status. This logic was then passed directly to the database import process, causing constraint violations.

Confirmed root cause: The format_sql_value function lacked proper validation to distinguish between nullable and non-nullable columns when handling empty string values.

✅ Applied Fix

The fix modifies the format_sql_value function in sql_exporter.py:

1. Simplified nullability check: The logic for determining if a column is nullable now only checks for the absence of "NOT NULL" in the column definition, making it more robust and maintainable.

2. Explicit empty string validation: Within the integer conversion block, an explicit check now raises a ValueError when encountering an empty string before attempting integer conversion. This ValueError is caught by the existing error handling logic.

3. Conditional error propagation:

   • For nullable columns: The ValueError is caught and NULL is returned (legacy behavior preserved for valid cases)
   • For NOT NULL columns: The ValueError is re-raised, causing the export process to fail and preventing the creation of an invalid SQL file

Affected modules:

• sql_exporter.py (format_sql_value function)
• tests/test_sql_exporter.py (new regression test)

Risks/side effects:

• Exports will now fail earlier when encountering invalid data for NOT NULL columns (this is the intended behavior to prevent silent data corruption)
• Users will receive clearer error messages instead of import failures
• No breaking changes for valid data; only affects error handling for edge cases

🧪 Tests (including regression)

• Unit test covering the case
• Integration/E2E test (SQL export → import workflow)
• Non-regression test (bug does not reappear)

Manual validation procedure:

1. Create a test CSV file with an empty string value in an integer field that maps to a NOT NULL database column (e.g., ASN)
2. Run the SQL export process on this CSV
3. Verify that the export process fails with a clear error message instead of generating an invalid SQL file
4. Confirm that valid data continues to export successfully without regression
5. Expected result: Export process aborts with error when encountering invalid data for NOT NULL columns; valid data exports without issues

🔗 Links

• Closes [BUG] SQL import fails with NOT NULL constraint error on asn_int column #65

🖼️ Evidence

Code changes summary:

• sql_exporter.py: Enhanced format_sql_value function with improved nullability validation and explicit empty string handling
• tests/test_sql_exporter.py: Added regression test case verifying export fails for empty strings in NOT NULL integer columns

Test output:

• All existing tests pass
• New regression test validates the fix

---

✅ Project checklist

• All my commits are signed (GPG/SSH)
• lint passes locally and in CI
• No sensitive data exposed (credentials, tokens)
• Manual validation completed
• The added/covered tests are relevant for the feature
• I have described a clear manual validation procedure

📝 Summary

This fix resolves a critical bug where empty string values in CSV exports were incorrectly converted to NULL, violating NOT NULL constraints in the database. The solution improves data integrity by failing fast during export rather than generating invalid SQL that fails at import time. The fix is complete, well-tested, and production-ready.