BUG: flyscan() master file missing /entry/flyscan_data/ group

AGENTS.md

@@ -161,6 +161,17 @@ methods, or plan methods on the bundle itself.  Example:
 - D-series docstring rules are enabled (`D100`-`D107`).  Every public
   module, class, and function (including `__init__`) must have a
   docstring.
+- **Comments and docstrings describe the code as it is now**, for the
+  target audience (beamline staff and users who will maintain and
+  extend it).  Do **not** narrate history in the source: no "used to",
+  "formerly", "no longer", "the older X", "renamed from", changelog
+  notes, or migration commentary.  That history belongs in GitHub
+  issues, PRs, and commit messages, which are the repositories for it.
+  When you change behavior, rewrite the affected comments/docstrings to
+  describe the new state plainly and delete the obsolete narration
+  rather than appending to it.  Rare exceptions (a non-obvious gotcha a
+  future reader must know to avoid re-introducing a bug) should be
+  stated as a present-tense caution, not as a story.
 
 ## Documentation conventions
 
@@ -196,6 +207,12 @@ These are imported from the user's host-wide notes
 
 - Do **not** push or open PRs without an explicit request.  Commit
   freely when asked.
+- **Issue work happens on a feature branch, never directly on
+  `main`.**  Use an existing feature branch when one already covers
+  the work (a single branch may span several related issues), or
+  create one before the first commit.  The user may override in rare
+  cases, but that is not typical; when committing to `main` appears to
+  be requested, verify first rather than assume.
 - Issue and PR titles must be **brief** and written for the target
   audience (beamline staff and users), focusing on the observable
   problem or change rather than internal implementation detail.  Use a

docs/source/explanation/flyscan_frame_count.md

@@ -20,57 +20,62 @@ completes, the plan writes the frame-to-position pairing **into the
 NeXus master HDF5 file** (the `*.hdf` file in your run directory).
 Look here:
 
-`/entry/positions` (an `NXdata` group)
-: The per-frame correlation table, with **one row per in-scan frame**
-  (the pre-roll and tail frames are already excluded -- so this group
-  has your 301 rows, not 306).  Datasets:
-
+`/entry/flyscan_data` (an `NXdata` group, the **default plot** and the
+single primary product)
+: A ready-to-plot view of the in-scan data, with **one row per in-scan
+  frame** (the pre-roll and tail frames are already excluded -- so this
+  group has your 301 rows, not 306).  Its `data` is a *virtual* dataset
+  (no bytes copied) that already slices the in-scan 301-frame substack
+  out of the full image stack, with `position_start_acquire` as its
+  plot axis.  The plan also sets `/entry@default = "flyscan_data"`, so a
+  NeXus-aware viewer (e.g. NeXpy, h5web) opens this group by default --
+  in-scan images vs. omega position, which is the useful default view of
+  a flyscan.  Datasets:
+
+  - `data` -- the in-scan image substack (virtual dataset).
+  - `position_start_acquire`, `position_end_acquire`,
+    `position_end_period` -- the motor (omega) position at three
+    phases of each frame's exposure period (the plot axes).
   - `image_number` -- IOC frame counter (1-based) for each in-scan
     frame.
   - `frame_index` -- `image_number - 1`; the **0-based index into
-    `/entry/images/data`** for that frame.  Use it to pull the in-scan
-    image substack out of the full (306-frame) image stack:
+    `/entry/images/data`** for that frame.  `data` is already this
+    substack; use `frame_index` only to map back to the full
+    (306-frame) image stack:
 
     ```python
     import h5py
     with h5py.File("20260618-...-S00031-....hdf", "r") as f:
-        images = f["/entry/images/data"]           # all captured frames
-        idx = f["/entry/positions/frame_index"][:] # in-scan frames only
-        in_scan_images = images[idx, :, :]         # the useful 301
-        omega = f["/entry/positions/position_start_acquire"][:]
+        images = f["/entry/images/data"]               # all captured frames
+        idx = f["/entry/flyscan_data/frame_index"][:]  # in-scan frames only
+        in_scan_images = images[idx, :, :]             # the useful 301
+        omega = f["/entry/flyscan_data/position_start_acquire"][:]
     ```
 
-  - `position_start_acquire`, `position_end_acquire`,
-    `position_end_period` -- the motor (omega) position at three
-    phases of each frame's exposure period.
   - `timestamp` -- per-frame timestamp.
 
-`/entry/flyscan_data` (an `NXdata` group, the **default plot**)
-: A ready-to-plot view of the same in-scan data.  Its `data` is a
-  *virtual* dataset (no bytes copied) that already slices the in-scan
-  301-frame substack out of the full image stack, with
-  `position_start_acquire` as its plot axis.  The plan also sets
-  `/entry@default = "flyscan_data"`, so a NeXus-aware viewer
-  (e.g. NeXpy, h5web) opens this group by default -- in-scan images
-  vs. omega position, which is the useful default view of a flyscan.
+  Provenance group attributes record that the data came from the
+  authoritative area-detector file: `source = "ad_file"`,
+  `n_frames_paired`, and `n_frames_expected`.
 
 `/entry/images` (an external link)
 : An HDF5 external link to the detector IOC's frame file at its
   `/entry/data`.  This is the **full** stack -- all captured frames,
   including the pre-roll and tail frames.  This is the dataset whose
   length is 306, not 301.
 
-So: read `/entry/flyscan_data` (or `/entry/positions`) for the paired,
-trimmed result; read `/entry/images` only if you specifically want the
-raw superset.
+So: read `/entry/flyscan_data` for the paired, trimmed result; read
+`/entry/images` only if you specifically want the raw superset.
 
 :::{note}
-These groups are written best-effort at the end of the run.  If the
-Tiled catalog hadn't ingested the run yet, or the IOC frame file
-wasn't readable from the master-file host, you may see only
-`/entry/images` (the external link) with a `WARNING` in the log and no
-`/entry/positions` / `/entry/flyscan_data`.  In that case re-run the
-pairing yourself with the analysis functions linked below.
+`/entry/flyscan_data` is written best-effort at the end of the run, and
+**only** from the area-detector file.  If the Tiled catalog hasn't
+ingested the run yet, or the IOC frame file isn't readable from the
+master-file host (most commonly: the per-detector image-files symlink
+next to the master is missing or mis-shaped), you will see only
+`/entry/images` (the external link) with a `WARNING` in the log and
+**no** `/entry/flyscan_data`.  In that case fix the symlink and re-run
+the pairing yourself with the analysis function linked below.
 :::
 
 :::{admonition} Keep this page in sync with the code
@@ -93,11 +98,10 @@ page in the same commit:
   `hdf_num_capture = int(num_frames * 1.5) + 20` in `flyscan`
   ([`flyscan_3idc.py`](../../../src/id3c/plans/flyscan_3idc.py)).
 - Frame-to-position pairing (the thing that selects the useful
-  frames): `pair_frames_to_positions` and
-  `pair_frames_to_positions_from_ad_file` in
+  frames): `pair_frames_to_positions_from_ad_file` in
   [`flyscan_3idc_analysis.py`](../../../src/id3c/utils/flyscan_3idc_analysis.py).
-- Master-file HDF5 layout written at run end (`/entry/positions`,
-  `/entry/flyscan_data`, `/entry/images`, and `/entry@default`):
+- Master-file HDF5 layout written at run end (`/entry/flyscan_data`,
+  `/entry/images`, and `/entry@default`):
   `update_master_file` in
   [`flyscan_3idc.py`](../../../src/id3c/plans/flyscan_3idc.py).  If you
   rename or restructure any of those groups/datasets, update the
@@ -194,15 +198,15 @@ Do **not** expect the raw `/entry/data` dataset to equal `num_frames`.
 The raw file is the superset; the *useful* frames are selected by
 pairing each frame to the omega position it was exposed at:
 
-- `pair_frames_to_positions(run)` -- from a Bluesky run / Tiled.
 - `pair_frames_to_positions_from_ad_file(...)` -- directly from the
-  area-detector HDF5 file.
+  area-detector HDF5 file (the authoritative, lossless source used by
+  the plan).
 
-Both live in
+It lives in
 [`flyscan_3idc_analysis.py`](../../../src/id3c/utils/flyscan_3idc_analysis.py).
-They use the IOC-timestamped frame stream together with the motor's
-position stream to drop the pre-roll and tail frames and attach the
-correct omega value to each remaining frame.  The result is the 301
+It uses the area-detector file's per-frame timestamps together with the
+motor's position stream to drop the pre-roll and tail frames and attach
+the correct omega value to each remaining frame.  The result is the 301
 position-paired frames that span `p_start -> p_end`.
 
 ## What this does *not* mean

docs/source/how_to/index.md

@@ -8,6 +8,7 @@ Task-oriented recipes.
 run_a_scan
 inspect_data
 visualize_hdf5
+repair_flyscan_master
 add_a_device
 add_a_plan
 edit_and_build_docs

docs/source/how_to/repair_flyscan_master.md

@@ -0,0 +1,82 @@
+# Repair a flyscan master file missing `/entry/flyscan_data`
+
+A flyscan NeXus master file (`*.hdf`) holds its primary product in the
+`/entry/flyscan_data` group: the in-scan image substack plus the
+per-frame motor positions.  That group is written at run end **only**
+when the area-detector image file is reachable from the master file's
+directory through the image-files symlink (`{detector}_files`,
+e.g. `eiger2_files`).
+
+If the symlink was missing or mis-shaped at run end, the master will
+have `/entry/images` (the external link) but **no**
+`/entry/flyscan_data`, and a `WARNING` will be in the run log.  Once
+the symlink is fixed, recover the group with the repair tool.
+
+## Step 1 -- fix the image-files symlink
+
+From the directory that contains the master file, create the symlink
+pointing at the workstation mount where the detector's image files are
+visible.  For an `eiger2` detector whose files are mounted at
+`/net/s3data/export/sector3/s3ida/XRD/`:
+
+```bash
+cd /path/to/your/run/directory
+ln -s /net/s3data/export/sector3/s3ida/XRD/ eiger2_files
+ls -l eiger2_files          # should show 'eiger2_files -> .../XRD/'
+ls eiger2_files/ | head     # should list at least one entry
+```
+
+The symlink must map directly to the image-file root.  A common
+mistake is creating a directory `eiger2_files/` that *contains* a
+child symlink (e.g. `eiger2_files/XRD -> .../XRD`); that adds an extra
+path level and the external link will not resolve.
+
+## Step 2 -- run the repair tool
+
+```bash
+id3c-flyscan-repair /path/to/your/run/20260618-...-S00024-....hdf
+```
+
+The tool:
+
+1. reads the run uid from the master's `/entry/entry_identifier`;
+2. locates the area-detector file from the master's `/entry/images`
+   external link (or, if absent, composes it from the `ad_file_path` /
+   `ad_file_name` start metadata);
+3. recomputes the per-frame pairing from the authoritative
+   area-detector file; and
+4. writes `/entry/flyscan_data` (replacing any existing copy) and sets
+   `/entry@default = "flyscan_data"`.
+
+### Preview without writing
+
+```bash
+id3c-flyscan-repair --dry-run /path/to/.../master.hdf
+```
+
+### Point at a specific area-detector file
+
+If the master cannot resolve the area-detector path (or you want to
+override it), pass it explicitly:
+
+```bash
+id3c-flyscan-repair --external-file /net/.../XRDS4_..._000001.h5 \
+    /path/to/.../master.hdf
+```
+
+Add `-v` for INFO-level logging.
+
+## When repair is not possible
+
+The tool exits non-zero and explains the cause if:
+
+- the master has no run uid (`/entry/entry_identifier`);
+- the area-detector file cannot be located or does not resolve (most
+  often the symlink from Step 1 is still wrong);
+- the area-detector file is not openable; or
+- pairing produces zero in-scan frames.
+
+The repair reads only the area-detector file (the lossless,
+authoritative per-frame source).  It never falls back to a lossy
+source, so a repaired `/entry/flyscan_data` is always the full-count
+result.

pyproject.toml

@@ -50,6 +50,9 @@ line-length = 88
 indent-width = 4
 target-version = "py311"
 
+[project.scripts]
+id3c-flyscan-repair = "id3c.utils.flyscan_repair:main"
+
 [project.optional-dependencies]
 dev = [ "build", "isort", "mypy", "pre-commit", "pytest", "ruff",]
 doc = [ "babel", "ipykernel", "jinja2", "linkify-it-py", "markupsafe", "myst-nb", "pydata-sphinx-theme", "pygments-ipython-console", "pygments", "sphinx-autoapi", "sphinx-copybutton", "sphinx-design", "sphinx-tabs", "sphinx",]

src/id3c/configs/devices.yml

@@ -63,7 +63,7 @@ apstools.devices.motor_factory.mb_creator:
   class_name: DetectorStage
   motors:
     det_x: "3idc:m69"  # translation
-    eiger_y: "3idc:m35"  # translation Eiger2
+    eiger_y: "3idc:m70"  # translation Eiger2
     eiger_z: "3idc:m36"  # translation Eiger2
 
 # laser_optics axes are interlocked against sample_stage.omega (both