feat: Add transport-level compression, remove application-level is_compressed#210
feat: Add transport-level compression, remove application-level is_compressed#210
Conversation
…mpressed Add Accept-Encoding (zstd, gzip, deflate) to all API requests so the server can compress response bodies at the HTTP transport layer. This replaces the application-level is_compressed=true query param which caused base64(gzip(data)) encoding inside JSON fields. - New transport_compression module handles Accept-Encoding negotiation and Content-Encoding decompression (zstd via optional zstandard package, gzip, deflate) - Client._rest_call() sends Accept-Encoding on every request and decompresses response bodies (including error responses) before JSON parsing - Remove is_compressed=true from 5 SDK callsites (get_events, get_children_events, get_detections, get_audit_logs, get_jobs) and replace unwrap() calls with direct JSON field access - Deprecate Client.unwrap() (kept for external backward compat) - zstandard is an optional dependency: pip install limacharlie[zstd] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move zstandard from optional to required dependency - pre-built wheels are available for all supported platforms (Linux/macOS/Windows, x86_64/ARM64, glibc/musl). Add GitHub Actions CI workflow that runs unit tests and wheel install verification across Python 3.10-3.13 on ubuntu, macos, and windows runners. Each job verifies zstandard installs and zstd decompression works end-to-end. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
os.chown and os.getuid are Unix-only. Guard with hasattr check so config file writing works on Windows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- limacharlie version -> limacharlie --version (correct CLI flag) - Use pip install --find-links instead of shell glob for wheel install (glob doesn't work on Windows PowerShell) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Windows NTFS uses ACLs, not Unix permission bits. os.chmod(0o600) doesn't restrict access the same way on Windows, so the 0o777 mask assertion fails. Only assert file permissions on Unix platforms. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
zstandard is kept as a hard dependency (pre-built wheels for all major platforms), but the runtime now handles ImportError gracefully. If zstd can't be imported (exotic platform, no C compiler), Accept-Encoding falls back to "gzip, deflate" and zstd-encoded responses pass through as-is. This prevents pip install failures from making the entire SDK unusable. - Restore try/except ImportError guard in transport_compression.py - Add _HAS_ZSTD flag for runtime feature detection - Add tests for fallback path (mock zstandard away, verify behavior) - Add no-zstd-fallback CI job that uninstalls zstandard and verifies the SDK still works with gzip/deflate only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract inline python -c snippets from ci.yml into standalone scripts under scripts/ci/ for readability and easier local debugging. Add edge case tests for transport_compression: - Raw deflate (no zlib header) vs zlib-wrapped deflate - Case insensitivity for all encodings (zstd, gzip, deflate) - Whitespace around Content-Encoding header values - Empty data with/without encoding - _HAS_ZSTD flag assertion - zstd passthrough case insensitivity when lib unavailable Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Top-level `import zstandard` in test_transport_compression.py caused pytest collection to fail when zstandard was uninstalled (no-zstd CI job). Tests that need zstandard now use local imports with pytest.mark.skipif(_HAS_ZSTD) or pytest.importorskip(). Tests that verify the no-zstd fallback run in both environments. - With zstandard: 50 passed, 2 skipped (no-zstd-only tests) - Without zstandard: 44 passed, 8 skipped (zstd-specific tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Let's add it to the new SDK/CLI that way we can focus our efrorts on what we need to test anyway. |
dzimine-lc
left a comment
dzimine-lc
left a comment
There was a problem hiding this comment.
@tomaz-lc the change is good, but it also mixes the compression change with intro of GH action based CI, I recommend we separate the two, and disuss/design/do CI in a diff PR?
|
|
||
| on: | ||
| pull_request: | ||
| branches: [cli-v2, master] |
There was a problem hiding this comment.
I am -1 on any backward support to old python, we leave it on v1 (v3 to be precise), propose we don't carry any backward compatibiltiy stuff to cli-v2
Suggest you modify it to 1) only act on cli-v2 branch and remove all backward compatibility part - matrix-python-version.
There was a problem hiding this comment.
I wasn't aware until very recently that cli-v2 branch will be long running for a while.
Having said that, since this branch is already targeted towards cli-v2 it should work fine. Once it's merged / takes over the old master, it should work just fine.
In short - it won't run against old version, just against the new one.
I can split it, yeah, but I needed it to test the changes in this branch + it also detected some existing Windows related bugs :) Will split CI into a separate branch targeted towards this branch. So the flow is / will be |
|
consider doing CI changes first. |
3 participants
Summary
Accept-Encoding: zstd, gzip, deflateheader to all API requests for transparent HTTP transport-level compressionis_compressed=true(base64-encoded gzip inside JSON) now return plain JSON fields, with compression handled transparently at the transport layer insteadtransport_compressionmodule handlesAccept-Encodingnegotiation andContent-Encodingdecompression (zstd, gzip, deflate)zstandardadded as a hard dependency with runtime graceful fallback - pre-built wheels available for all major platforms, but if unavailable the SDK still works with gzip/deflate onlyClient.unwrap()deprecated but kept for backward compatibilityWhy
The previous
is_compressed=trueapproach gzip-compressed individual data fields and base64-encoded them inside JSON responses (base64(gzip(data))). This required explicit client-side decompression (unwrap()) and prevented efficient transport-level compression since compressing already-compressed data is wasteful.Transport-level compression is better in every way - it's transparent to the caller, handled by standard HTTP content negotiation, covers the entire response body (not just specific fields), and supports modern algorithms like zstd which offer better ratios and faster decompression than gzip.
Backward compatibility
This change is fully backward compatible:
is_compressed=truefor older clients that haven't upgraded yetClient.unwrap()is deprecated but still functional for any external callers using it directlyContent-Encodingheader) pass through unchangedThere is a companion API-side PR that adds transport compression support on the server. Old clients sending
is_compressed=truewill continue to work as before - the server skips transport compression for those requests to avoid double compression.zstandardcompatibilityzstandardis listed as a hard dependency inpyproject.tomlso pip installs it automatically. It ships pre-built wheels for all major platforms - no C compiler or build toolchain needed at install time.Runtime fallback: If
zstandardcan't be imported (exotic platform without a wheel or C compiler), the SDK still works -Accept-Encodingfalls back to"gzip, deflate"and all transport decompression continues to work via stdlibzlib. The zstd code path is guarded withtry/except ImportErrorat module load time.Changes
pyproject.tomlzstandard>=0.22.0as hard dependencylimacharlie/transport_compression.pylimacharlie/client.pyAccept-Encodingheader, transport decompression, deprecateunwrap()limacharlie/sdk/sensor.pyis_compressed+unwrap()fromget_events,get_children_eventslimacharlie/sdk/organization.pyis_compressed+unwrap()fromget_detections,get_audit_logs,get_jobslimacharlie/config.pyos.chowncrash on Windows (Unix-only API).github/workflows/ci.ymltests/unit/test_transport_compression.pytests/unit/test_client.pytests/unit/test_config.pytests/unit/test_sdk_sensor.pytests/unit/test_sdk_organization.pyAdditional fixes
config.py: Guardos.chown()withhasattr(os, "chown")-os.chownandos.getuiddon't exist on Windowstest_config.py:os.chmod(0o600)doesn't enforce Unix-style permissions on NTFS - skip permission assertion on WindowsTest plan
Accept-Encoding: zstd, gzip, deflatesent on all requestsunwrap()still works for external callers🤖 Generated with Claude Code