BUG: flyscan() master file missing /entry/flyscan_data/ group#13
Merged
Conversation
Add the convention to AGENTS.md: source comments and docstrings describe the code as it is now, for the target audience. Historical narration (used to / formerly / no longer) belongs in issues, PRs, and commit messages, not the source.
The per-frame correlation is now written solely from the authoritative area-detector HDF1 file. When that file is not openable, no group is written and a clear WARNING explains the cause (missing/mis-shaped image-files symlink) and the recovery path. - Merge the per-frame correlation into the single /entry/flyscan_data NXdata group: image_number, frame_index, and timestamp join the image substack and position arrays; the separate /entry/positions group is no longer produced. - Stamp provenance attributes (source, n_frames_paired, n_frames_expected) on /entry/flyscan_data. - Add _expected_frame_count() helper (AD-file count, else start-doc num_frames) with unit tests. - Update the frame-count explanation page to match.
Name the image-files symlink per detector ({det.name}_files, e.g.
eiger2_files) so a beamline with several area detectors can give each
its own root. flyscan() now creates the symlink next to the master
file when the workstation mount (hdf1.read_path_template) is known and
exists; it falls back to the descriptive manual-fix warning otherwise
and never raises.
- Replace fixed AD_FILES_DIRNAME/AD_FILES_ROOT with per-detector
ad_files_dirname()/ad_files_root_for() helpers.
- Add _ensure_ad_files_symlink() (auto-create, safe fallback) and
_read_path_template(); _external_link_target() and
_check_ad_files_symlink() use the per-detector name.
- Tests updated for the per-detector name; new
test_ensure_ad_files_symlink.py.
Make a flyscan master file self-repairable when /entry/flyscan_data is missing (area-detector file unreachable at run end). - Stamp ad_read_path_template / ad_write_path_template (the IOC->workstation path mapping) into the start metadata so a reader can locate the data files from the master alone. - Extract write_flyscan_data() into flyscan_3idc_analysis so the live plan and the offline tool write an identical /entry/flyscan_data. - Add the id3c-flyscan-repair console script: recompute the pairing from the authoritative area-detector file and write the group. - Add the how-to page and tests for the writer, the CLI, and the path-template helpers.
Remove dates, session/notes/strategy-doc references, Phase-N decision labels, and 'empirically/formerly/revised' framing from source comments and docstrings, keeping the present-tense technical facts. Per the AGENTS.md convention; history lives in issues/PRs/commits.
- Reduce the flyscan() signature to a clean kwarg list; move the per-kwarg detail to the docstring Parameters section (and document _force_hdf_nonblocking there, which had been inline only). - Convert module-level constant comments to following docstrings (_CONSUMER_TICK_DEFAULT, _CLEANUP_DRAIN_TICK, _FRAME_QUEUE_SIZE, _ACCL_FALLBACK_SECONDS, _AD_FILES_WARNED, _CAM_ERROR_STATES). - Split dense multi-topic paragraphs into one topic per paragraph (flyscan() summary, read_motor_field, _FRAME_QUEUE_SIZE).
Coverage Report for CI Build 28118953070Coverage increased (+8.9%) to 64.75%Details
Uncovered Changes
Coverage Regressions7 previously-covered lines in 1 file lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12.
What
A flyscan master file's primary analysis group,
/entry/flyscan_data, was silently going missing (or being written from a lossy source) when the area-detector file was unreachable at run end — most commonly because the image-files symlink next to the master was missing or mis-shaped.This PR fixes that and makes the situation recoverable.
Changes (phased)
A — stop silent degradation
/entry/flyscan_datais written only from the authoritative area-detector file. If that file is not openable, no group is written and a clearWARNINGexplains the cause and recovery path (the lossy CA-monitor fallback is removed).image_number,frame_index,timestamp) is folded into/entry/flyscan_dataas subordinate datasets, with provenance attrs (source,n_frames_paired,n_frames_expected); the separate/entry/positionsgroup is no longer written.B — prevent recurrence
{det.name}_files, e.g.eiger2_files), so multiple area detectors can coexist.flyscan()auto-creates the symlink next to the master when the workstation mount (hdf1.read_path_template) is known and exists; it falls back to the manual-fix warning otherwise and never raises.C — make masters self-repairable
ad_read_path_template/ad_write_path_template) is stamped into the run start metadata.id3c-flyscan-repairrecomputes the pairing from the area-detector file and writes/entry/flyscan_datainto an existing master. The writer is extracted (write_flyscan_data) so the live plan and the tool produce identical layout.Docs / cleanup
docs/source/how_to/repair_flyscan_master.md; updatedflyscan_frame_count.md.flyscan()signature (details moved to the docstring Parameters section), per the AGENTS convention.Also included: an unrelated one-line config fix (
eiger_yPV reassignment indevices.yml) that was already staged locally.Tests
New:
test_expected_frame_count.py,test_ensure_ad_files_symlink.py,test_write_flyscan_data.py,test_flyscan_repair.py,test_path_templates.py; updatedtest_check_ad_files_symlink.py,test_external_link_target.py. Full suite: 119 passed.Off-network note
Developed on a host without beamline EPICS access; tests exercise construction/file I/O and pure helpers, not live CA. The repair tool and writers are file-only and fully tested.