Conversation
…ase task class and preproc tasks to conform to engine, tested the preproc pipeline flow
Yashvi-Sharma
requested review from
mikelangmayr,
prkrtg,
scizen9 and
weatherhead99
weatherhead99
left a comment
weatherhead99
left a comment
There was a problem hiding this comment.
I think this all looks really nice.
Haven't run anything to test yet though, we should probably work on getting an example script ready to go on real DEIMOS data just so we have something to run
| run: | ||
| params: | ||
| input_source: "/Users/yashvi/Desktop/Detector Characterization Tools/DTU_dettest/DTU_singledet_acceptance/PTC/SCI/20250812-101350/*_bias_*.fits" | ||
| identifier_func: tasks.custom.guess_image_type_from_filename_DEIMOS |
There was a problem hiding this comment.
this all looks great so far. I'm a little concerned about the absolute paths here. Hopefully that's just a way for you to get this running, but do we need to think about a way of having these configurations be transplantable?
(one way would be to use environment variables here.. shudders). Another way would be to allow e.g. jinja template variables and allow a local per-computer config to be loaded as well on the command line for variable substitutions.
| elif isinstance(data, xr.DataArray): | ||
| return data.values | ||
| else: | ||
| raise TypeError("data must be an xarray.DataArray, or numpy.ndarray") |
There was a problem hiding this comment.
this might be a place to use a match: case rather than if else
2 participants
Major upgrades to the pipeline engine:
Added a base class for preprocessing tasks to reduce redundant code, updated outputs to conform to dict structure.
Minor updates and fixes to utils, data models, and tasks