1430 migrate hermes broker to data service#1515
Open
phycodurus wants to merge 24 commits intoversion-3-0-alphafrom
Open
1430 migrate hermes broker to data service#1515phycodurus wants to merge 24 commits intoversion-3-0-alphafrom
phycodurus wants to merge 24 commits intoversion-3-0-alphafrom
Conversation
…nd fix up sending hermes-received spectra files out over hermes
Fix hermes ingestion so it ingests targets and spectroscopic data too. This is being merged in to branch 1430-migrate-hermes-broker-to-data-service. Parts will be refactored into `tom_hermes` and other parts left 1430. Additionally 1430 will get a new `AppConfig.sharing_backends()` integration point. Briefly, - `tom_dataproducts.alertstreams.hermes_ingester` -> `tom_hermes.alertstreams` - `tom_dataproducts.alertstreams.hermes_publisher` -> `tom_hermes.sharing` The data processors remain in `tom_base.tom_dataproducts`
- Adds integration point for `SharingBackend` subclasses
- Adds `tom_common.sharing.SharingBackend` abstract base class
- tom_hermes will define a subclass
- tom_common.sharing.TomToolkitSharingBackend defined for
TOM-TOM sharing
- The `SharingBackend` ABC defines `get_destination_choices`,
`share`, `validate_credentials`), `get_sharing_backends()` /
`get_sharing_backend()` helper methods
- `TomToolkitSharingBackend` (supports both DRF API_KEY and HTTP
Basic auth for TOM-to-TOM sharing), and a `sharing_backends()`
integration point on `TomCommonConfig`.
- this is part of the tom_hermes refactor; to remove HERMES
related code from `tom_base` to `tom_hermes`
- These are breaking changes going in the v3:
- Removed `share_data_with_hermes`, `share_target_list_with_hermes`,
`share_data_with_tom`, the internal `_dispatch_to_backend`, and the
`check_for_save_safe_datums` placeholder. Callers should use
`get_sharing_backend(name)().share(...)` directly.
- Removed `tom_dataproducts/tests/test_sharing.py` to `tom_hermes`
`TargetShareView.form_valid`, `TargetListShareView.form_valid`, `share_target_and_all_data`, and `continuous_share_data` now all use the `TomToolkitSharingBackend` (via `get_sharing_backend(name)().share(...)` `share_target_with_tom` now supports DRF API_KEY auth (sent as `Authorization: Token <key>`) as well as the existing HTTP Basic (username/password) path.
These have been refactored into tom_hermes. See `tom_hermes.alertstreams` and `tom_hermes.sharing` sub-packages.
DataService instances need access to the request user so they can
authenticate against per-user credentials. HermesDataService's
`/api/v0/topics/` and `/api/v0/query/` endpoints both reject anonymous
requests. So, the form (when populating topic choices) and the service
(when running queries) derive an `Authorization: Token <key>` header
from the logged-in user's `HermesProfile.hermes_api_key`.
- `DataService.__init__` accepts `user=None` and stashes it on
`self.user`.
- `BaseQueryForm.__init__` pops a `user` kwarg into `self.user`
(silently ignored by concrete forms that don't use it).
- `DataServiceQueryCreateView.get_form_kwargs` and
`DataServiceQueryUpdateView.get_form_kwargs` inject
`user=self.request.user`.
- `RunQueryView` and `CreateTargetFromQueryView` instantiate the
DataService class with `user=self.request.user`.
Also fixes `CreateTargetFromQueryView.post()` crashing with
`'NoneType' object has no attribute 'save'` when a DataService's
`to_target` returns `(None, {}, [])` — which can happen legitimately
when a second-stage fetch fails (HermesDataService's archive-query →
full-message-body flow). The view now records a per-row error and
continues to the next result.
Brings the guide in line with the new tom_base/tom_hermes split: - Top note reframed from "deprecation shims will keep old paths working for one release" to "this release is a major version bump; old import paths are removed; see Migration summary." - Subscribe-to-Kafka section's TOPIC_HANDLERS guidance no longer promises a backward-compatible shim — the old dotted path no longer resolves and must be updated. - Migration summary expanded with full lists of names that moved out of tom_dataproducts.alertstreams.hermes_publisher (now in tom_hermes.sharing) and tom_dataproducts.alertstreams.hermes_ingester (now in tom_hermes.alertstreams.ingester), plus the share_*_with_* functions that were removed entirely from tom_dataproducts.sharing in favor of get_sharing_backend(name)().share(...).
The previous filter joined ReducedDatum.message (an M2M to
AlertStreamMessage) on (exchange_status='published', topic=...) to
prevent re-publishing a datum to a topic it had previously been sent
to. That model and that field are being removed.
The new filter reads source_name directly:
queryset.exclude(source_name=f'Hermes:{message_topic}')
"works" means that the testing dependencies get installed and you don't bet Module not found errors when you run the tom_base tests.
Drop the two preload views and their URL patterns; their replacements
now live in tom_hermes. Update three template {% url %}
refs to the tom_hermes: namespace. Removes the last hard import of
tom_hermes from tom_base.
The "Open in Hermes" button's formaction resolves to a URL owned by tom_hermes. So, hide the button when tom_hermes is absent.
Member
Author
|
@copilot resolve the merge conflicts in this pull request |
…rate-hermes-broker-to-data-service # Conflicts: # poetry.lock # tom_dataservices/views.py Co-authored-by: phycodurus <431350+phycodurus@users.noreply.github.com>
Resolved by merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There are two categories of changes here:
remove HERMES facing code from the various places (
tom_targetsandtom_dataproductsmainly) where it lived.Add the
SharingBackendbase class and infrastructure which allowstom_baseto dispatch sharing functionality toINSTALLED_APPS(liketom_hermes) that implement a concrete subclass of theSharingBackendABC. The TOM-TOM sharing backend is intom_common.