Add support for collection export#325
Merged
Merged
Conversation
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Secrets | View in Orca |
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 10, 2026
Introduces a dedicated COLLECTION_EXPORT env var (default false) that enables collection export via the new collectionExport Helm values added in weaviate/weaviate-helm#325. When enabled: - MinIO is started and the weaviate-export bucket is created automatically - collectionExport.enabled=true and EXPORT_DEFAULT_BUCKET=weaviate-export are set in Helm - If ENABLE_BACKUP is not also set, the backup-s3 module is automatically configured to point to MinIO (collection export uses it as its S3 backend); backups.s3.secrets are omitted when S3_OFFLOAD is active to avoid the awsSecret.yaml multi-source credential guard in weaviate-helm - action.yml gains a collection-export input - CI all-params job enables COLLECTION_EXPORT=true and verifies a collection export can be created via POST /v1/export/s3 - Operating skill updated with COLLECTION_EXPORT=true usage, deployment pattern, and env-var table entry Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merged
4 tasks
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 10, 2026
Introduces a dedicated COLLECTION_EXPORT env var (default false) that enables collection export via the new collectionExport Helm values added in weaviate/weaviate-helm#325. When enabled: - MinIO is started and the weaviate-export bucket is created automatically - collectionExport.enabled=true and EXPORT_DEFAULT_BUCKET=weaviate-export are set in Helm - If ENABLE_BACKUP is not also set, the backup-s3 module is automatically configured to point to MinIO (collection export uses it as its S3 backend); backups.s3.secrets are omitted when S3_OFFLOAD is active to avoid the awsSecret.yaml multi-source credential guard in weaviate-helm - action.yml gains a collection-export input - CI all-params job enables COLLECTION_EXPORT=true and verifies a collection export can be created via POST /v1/export/s3 - Operating skill updated with COLLECTION_EXPORT=true usage, deployment pattern, and env-var table entry Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 10, 2026
- Backup job: use latest Weaviate version instead of hardcoded 1.26.3 (collection export requires a recent version) - Both backup and all-params jobs: set HELM_BRANCH to jose/collection-export since the collectionExport Helm values are not yet released (weaviate/weaviate-helm#325) - Added TODO comments to revert HELM_BRANCH to 'main' once the helm chart is released Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new option, collectionExport, in the values which allows enabling collection export. Also, exposes the env var EXPORT_DEFAULT_BUCKET which points at the right bucket to export collections in.
A new env var got added to specify the PATH. Adding support on the helm charts.
0ca17a0 to
2f3be9e
Compare
antas-marcin
approved these changes
Apr 16, 2026
dirkkul
reviewed
Apr 16, 2026
2f3be9e to
80e0e82
Compare
- Add EXPORT_PARALLELISM commented-out env var to collectionExport config - Add tests for EXPORT_PARALLELISM (absent by default, settable via envconfig) - Fix pre-existing test failures for TRANSFORMERS_PASSAGE/QUERY_INFERENCE_API by passing --namespace default so .Release.Namespace resolves to "default" instead of a random UUID generated by helm template - Make EXPORT_DEFAULT_PATH optional and adapt tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
80e0e82 to
c6d90d3
Compare
|
|
||
| # Required setting. Bucket path in which to save exports. Defaults to empty string. | ||
| # Set this option if you want to save exports to a given path inside the bucket. Must be a valid bucket path. | ||
| EXPORT_DEFAULT_PATH: "" |
Collaborator
There was a problem hiding this comment.
by default we will be setting EXPORT_DEFAULT_PATH empty? shouldn't this be commented as EXPORT_PARALLELISM?
Contributor
Author
There was a problem hiding this comment.
I think the env var is required now. I did try enabling the feature the other day without the env var set and it was faling with this error.
That's what Dirk mentioned by saying that the env var is now required, but it can be default to "". If we don't pass it like that, the cluster will error and won't enable the feature.
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 17, 2026
- Backup job: use latest Weaviate version instead of hardcoded 1.26.3 (collection export requires a recent version) - Both backup and all-params jobs: set HELM_BRANCH to jose/collection-export since the collectionExport Helm values are not yet released (weaviate/weaviate-helm#325) - Added TODO comments to revert HELM_BRANCH to 'main' once the helm chart is released Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 20, 2026
- Backup job: use latest Weaviate version instead of hardcoded 1.26.3 (collection export requires a recent version) - Both backup and all-params jobs: set HELM_BRANCH to jose/collection-export since the collectionExport Helm values are not yet released (weaviate/weaviate-helm#325) - Added TODO comments to revert HELM_BRANCH to 'main' once the helm chart is released Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 20, 2026
- Backup job: use latest Weaviate version instead of hardcoded 1.26.3 (collection export requires a recent version) - Both backup and all-params jobs: set HELM_BRANCH to jose/collection-export since the collectionExport Helm values are not yet released (weaviate/weaviate-helm#325) - Added TODO comments to revert HELM_BRANCH to 'main' once the helm chart is released Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 20, 2026
Introduces a dedicated COLLECTION_EXPORT env var (default false) that enables collection export via the new collectionExport Helm values added in weaviate/weaviate-helm#325. When enabled: - MinIO is started and the weaviate-export bucket is created automatically - collectionExport.enabled=true and EXPORT_DEFAULT_BUCKET=weaviate-export are set in Helm - If ENABLE_BACKUP is not also set, the backup-s3 module is automatically configured to point to MinIO (collection export uses it as its S3 backend); backups.s3.secrets are omitted when S3_OFFLOAD is active to avoid the awsSecret.yaml multi-source credential guard in weaviate-helm - action.yml gains a collection-export input - CI all-params job enables COLLECTION_EXPORT=true and verifies a collection export can be created via POST /v1/export/s3 - Operating skill updated with COLLECTION_EXPORT=true usage, deployment pattern, and env-var table entry Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jfrancoa
added a commit
to weaviate/weaviate-local-k8s
that referenced
this pull request
Apr 20, 2026
- Backup job: use latest Weaviate version instead of hardcoded 1.26.3 (collection export requires a recent version) - Both backup and all-params jobs: set HELM_BRANCH to jose/collection-export since the collectionExport Helm values are not yet released (weaviate/weaviate-helm#325) - Added TODO comments to revert HELM_BRANCH to 'main' once the helm chart is released Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's being changed
Adds Helm chart support for Weaviate's collection export feature, introduced in weaviate/weaviate#10958.
values.yaml— new top-levelcollectionExportsection (mirrors the pattern ofbackupsandoffload):collectionExport.enabled— gates the feature; whentrue, injectsEXPORT_ENABLED=trueinto the podcollectionExport.envconfig.EXPORT_DEFAULT_BUCKET— required bucket name (defaults toweaviate-export); the bucket must exist before enabling, otherwise exports will failcollectionExport.envconfig.EXPORT_DEFAULT_PATH— optional path prefix inside the bucket (commented out, defaults to empty)collectionExport.envconfig.EXPORT_PARALLELISM— optional number of concurrent scan workers per export (commented out, defaults to0= GOMAXPROCS at runtime)templates/weaviateStatefulset.yaml— renders the env vars whencollectionExport.enabled=true; all othercollectionExport.envconfigkeys are forwarded as-is, so future env vars require no template changes..cicd/test.sh— test cases covering:EXPORT_ENABLEDandEXPORT_DEFAULT_BUCKETare absent by defaultEXPORT_DEFAULT_PATHandEXPORT_PARALLELISMare absent by default but can be set viaenvconfig